matt4711

A paper [1] we wrote in 2015 (cited by the authors) uses some more sophisticated data structures (compressed suffix trees) and Kneser–Ney smoothing to get the same "unlimited" context. I imagine with better smoothing and the same larger corpus sizes as the authors use this could improve on some of the results the authors provide.

Back then neural LMs were just beginning to emerge and we briefly experimented with using one of these unlimited N-gram models for pre-training but never got any results.

[1] https://aclanthology.org/D15-1288.pdf

LADWP (LA power provider) has a similar opt-in program for Nest owners with the following conditions:

How the program works

* We’ll adjust your connected thermostat(s) when summer electricity demand is at its highest to help decrease stress on the grid

* Prior to an event, we may lower your thermostat(s) by a few degrees to help maintain comfort during the actual event

* if the temperature in your home feels uncomfortable, you can opt out of an event at any time by adjusting your thermostat.

* You’ll receive a $60 e-gift card at the end of the season for participating.

Signing up nets you another $125 gift card.

I find the requirement of using a laptop without being allowed to use an external monitor to view potentially very long documents (half screen!) for hours on a tiny screen to be ridiculous.

Looking at the paper, comparing methods that use k bits per hash to a method that uses k*log(m) bits per hash seems unfair and misleading.

Funny enough "learning characters/words in the context of the vocabulary they are in" is exactly what NLP machine learning models use to learn "rich" word/text representations based on the "distribution hypothesis" which states that the choice of words in the same context share a common meaning.

Like the requirement that you have to delete tweets in datasets that have been deleted on twitter?

I'm pretty sure all the twitter datasets violate the twitter TOCs.

From the first link: "Comparing OFDM to LTE today we find a better scalability to a much lower latency (an order of magnitude lower round-trip time [RTT] than LTE today) in OFDM."

Doesn't LTE already have quite good latency properties?

Hetzner is involved in many of the non-related entity transfers.

I thought the NVIDIA drivers for the more fancy cards (TITAN etc) are the same as for the gforce cards. Wouldn't this restriction apply to those cards as well? Doesn't make much sense to me...

I find it hard to believe this. Their main index is certainly not all-RAM (there must be some flash and maybe even disk), and the throughput would just not be enough for something like BitFunnel.

From looking at the github repo it does look like the system runs entirely in main memory.

I was at the SIGIR'17 presentation of this paper (won best paper award btw) and have some comments in general:

- They mentioned (from what I remember) that they now use BitFunnel as they core of the complete Bing search engine not just the fresh parts.

- When I read the paper and looking at the code, it looks like their index doesn't include frequency information whereas your PEF code does. It is unclear what was counted in the experiments.

- If you look at the code, they are actually doing much more complicated stuff than just regular bloom filters by "bin packing" the hash positions for each term to reduce false positive rates (see https://github.com/BitFunnel/BitFunnel/issues/278 ). I'm nor sure if it is "fair" to compare a system developed by 10+ engineers over many years to a "phd student" code base developed over short period of time. I think the PEF code is excellent but I'm more talking about that engineering efforts can have a large impact on performance.

- I'm fairly sure you are right regarding the lack of URL-sorting. However, this can have another cause. If you consider Figure 4 in the paper which shows how "higher ranking rows" group documents together to allow faster intersection. URL sorting causes clusters in document-ids. Say, in the example in Fig. 4 there might be a cluster for that specific term for documents 0,1,2,3. This would mean the "higher ranking" row approach becomes worse (more false positives) when clustering occurs in the collection. So while URL-sorting helps PEF, it will most likely make BitFunnel worse.

Is there a binary version of this tool that can be downloaded somewhere? Seems a bit of a pain to install.

There are much faster SA construction algorithms than skew (check out divsufsort). The O(n) algorithms using induced sorting are also likely much faster than this work. The constants of recent O(n) algorithms are very low.

here a link for some state-of-the-art benchmarks of non-parallel SA construction algorithms: https://github.com/y-256/libdivsufsort/blob/wiki/SACA_Benchm...

there also now exist linear speedup parallel SA construction algorithms.

I think china blocks zh.wikipedia.org but all other languages are not blocked.

There is a paper describing such a method (not sure if that is what was implemented):

"Zhongjun Jin, Michael R. Anderson, Michael J. Cafarella, H. V. Jagadish: Foofah: Transforming Data By Example. SIGMOD Conference 2017: 683-698"

In general the techniques in zstd/lzma are much faster than using compressed text indexes.

As a somewhat expert in the field I doubt that this paper has any practical implications.

Say you want to translate a sentence from language A to language B. You have a system that generates 10 possible translations in language B. Now you use a language model of language B to figure out which is the best translation by asking the language model which of the translations has the highest probability of being generated by the model.

There was a post on HN a while ago where somebody wrote a server side tool which can detect curl -> bash piping and delivered different content if that happens.

What about AVM products such as a Fritzbox (https://en.wikipedia.org/wiki/Fritz!Box) ? Primarily a german product but also available internationally (e.g. Australia)

Interestingly, there is a discussion in comments of this paper claiming the authors got it wrong.

But as you said, under the assumption that the test perfectly detects, the results are correct.

Indeed. I was never aware that he was working on his own text editor.

Too bad there are no minimum size restrictions and all apartments in those 30-story towers are essentially shoe boxes.

Lots of Australian ISPs use this model. For example: TPG Broadband Off-Net 50GB -> 10GB peak / 40GB off peak (2am-8am)

We are working on VS support. Last time we tried to port to VS some of the constexpr stuff we use was not available yet. The next release (in 1-2 month) should have full VS2015 support.

Thanks for clarifying this.

The library is very template heavy and thus most of the code is in the header files so I guess it wouldn't change much in this case?

Additionally, we (the main authors) have lots of contributions from other researchers which we would ask for permission to the change the license (I think?).

Most IBM employees in Australia use Toshiba laptops instead of Lenevo.

What is your opinion on this LZ scheme: http://zola.di.unipi.it/rossano/wp-content/papercite-data/pd... which seems to have very good compression effectiveness and decompression time.

besides the sony z1 compact which is actually quite nice. there will probably be a z2 compact soon.