tadkar

ahead of time

I think the important thing is that the first approach to converting complete sentences to an embedding was done by averaging all the embeddings of the tokens in the sentence. What ColBERT does is store the embeddings of all the tokens before then using dot products to identify the most relevant tokens to the query. Another comment in this thread says the same thing in a different way. Feels funny to post a stack exchange reference, but this is a great answer!

[1] https://stackoverflow.com/questions/57960995/how-are-the-tok...

Here’s my understanding. It is intimidating to write a response to you, because you have an exceptionally clear writing style. I hope the more knowledgable HN crowd will correct any errors in fact or presentation style below.

Old school word embedding models (like Word2Vec) come up with embeddings by using masked word predictions. You can embed a whole sentence by taking the average of all the word embeddings in the sentence.

There are many scenarios where this average fails to distinguish between multiple meanings of a word. For example “fine weather” and “fine hair” both contain “fine” but mean different things.

Transformers are great at producing better embeddings by considering context, and using the words in the rest of the sentence to produce a better representation of each word. BERT is a great model to do this.

The problem is that if you want to use BERT by itself to compute relevance you need to perform a lot of compute per query because you have to concatenate the query and the document vector to produce a long sequence that can then be “embedded” by BERT. Figure 2c in the ColBERT paper [1]

What ColBERT does is to use the fact that BERT can use context from the entire sentence and its attention heads to produce a more nuanced representation of any token in its input. It does this once for all documents in its index. So for example (assuming “fine” was a token) it would embed the “fine” in the sentence “we’re having fine weather today” to a different vector than the fine in “Sarah has fine blond hair”. In ColBERT the size of the output embeddings are usually much smaller than the typical 1024 you might expect from a Word2Vec.

Now, if you have a query, you can do the same and produce token level embeddings for all the tokens in the query.

Once you have these two contextualised embeddings, you can check for the presence of the particular meaning of a word in the document using the dot product. For example the query “which children have fine hair” matches the document “Sarah had fine blond hair” because the token “fine” is used in the exact same context in both the query and the document and should be picked up by the MaxSim operation.

[1] https://arxiv.org/pdf/2004.12832

I have a theory that organizations that grow fast and scale well all have this “cellular model” at their core.

Investment bank trading desks in the pre-2008 era, partnership at the big strategy consulting firms and even “multi-strategy hedge funds” now are actually all collections of very incentive aligned businesses. They share the Creo quality of making lots of millionaires and people looking back on their time there as one of great freedom and achievement.

In all these places, employees are paid according to the revenue they generate, with seemingly no ceiling to what you can take home. It is true that the size of any one cell doesn’t scale beyond a small number of people. But all the organisations I mentioned above scale by having units tackling small pieces of vast markets.

The main lesson I took away from reading “Barbarians at the Gate” is that big companies hugely suffer from the principal agent problem, where management is mostly out to enrich themselves at the expense of shareholders and employees (sometimes). This looting is however only possible at a company that was established by a founder with a deep vision and passion for the product and has set up systems and culture that generates sufficient cash for the professional management to leech off.

What I have not read yet is a systematic study of these “cellular organizations” and what the common features are that make them successful. My guess is that the key is that each “unit” or “cell” has measurable economics that makes it possible to share the economic value over a sustained period of time. A bit like why sales people get paid a lot.

This is a great blog to give you things to get started. https://easyperf.net/

As with all things, practice is an essential part of improving!

Then, there's learning from some real achievements. Fast inverse square root, or the 55GB/s Fizzbuzz example: https://codegolf.stackexchange.com/questions/215216/high-thr...

I came here to say exactly the same thing. There are also a couple of other options: the MIR project from RedHat [1], libjit [2], lightning [3] and Dynasm [4] 1. https://github.com/vnmakarov/mir 2. https://www.gnu.org/software/libjit/ 3. https://www.gnu.org/software/lightning/manual/lightning.html 4. https://corsix.github.io/dynasm-doc/tutorial.html

But in general it seems to be very hard to beat the bang for buck from generating C and compiling that - even with something simple like tcc

I suspect that for most Bloom filters, the most commonly used hash functions are “good enough”. There’s also some literature to suggest that using just 2 hash functions and recombining the results is plenty. See kirsch-mitzenmacher [1] and [2]

[1] https://www.eecs.harvard.edu/%7Emichaelm/postscripts/tr-02-0... [2] https://stackoverflow.com/questions/70963247/bloom-filters-w...

I think JetBrains have something where you can do something like what you’re looking for. https://www.jetbrains.com/mps/

I think this is the paper https://arxiv.org/abs/1204.6079

How does preflight deal with selectors used to identify elements changing? This is the critical piece to solve before systems like this are useful. As identified in the paper discussed here: https://blog.acolyer.org/2016/05/30/why-do-recordreplay-test...

From an employer’s perspective, I’d be super interested in hearing about the experience of applying for this visa too! Does the UK government make it easy to apply? What are the interviews like? Anything your employer did that made the whole process easier for you?

The paper has a great figure where they illustrate areas of the overhead vs false positive trade-off space where each filter type performs best. Cuckoo filters make an appearance there

A hyperLogLog is for counting distinct elements. This and Bloom filters are more about checking whether an element has been seen before; a very different use case.

There is a similar great project here [1] with the Hungarian Wikipedia corpus. Great workout for non English and maybe non-ascii operations.

The performance of Java there is super impressive. It should port relatively quickly to this file too...

[1] https://github.com/juditacs/wordcount

Clickhouse is not something you use for a CRUD backend.

The obsession with Clickhouse is the phenomenal performance for the OLAP use case, a scenario where there were not many open source, easy to install and maintain options. For the most part you can treat it as a “normal” database, insert data into it and query it without messing about with file format conversions and so on. The fact that it is blindingly fast is a big bonus!

I am going to massively over simplify, but here goes. The really big idea from control theory is the idea of negative feedback. This basically boils down to measuring the output of your system and making the input to the system some function of the difference between the input signal and the output. The parent post refers to PID control. This refers to the three types of commonly used things to do with the difference (or error) signal.

P is for proportional - where you just multiply the error signal with a constant. What this does is to encourage the system to track the level of the input signal (but potentially with some lag)

I is for integral. This is where you integrate the difference signal over time. What this does is to reduce the lag between the input and output signal D is for derivative. This is where you feed back in the derivative of the error signal What this does is to damp down the swings in the system especially those that come from being too aggressive with the above two knobs.

Good controller design often comes down to picking the right weights for each of the three types of feedback functions you can input into the system. So in this example, it might be that you distribute your requests to servers based on how over or under loaded the servers are...

The observation about consumer inflation is very true, the massive inflation is actually in asset values (house prices, share prices, etc) I also wonder how I reconcile the rest comment with others pointing out the vast amounts of money available, especially in the US, for startups in general. When seed rounds can be more than $3M, I’d say there was plenty of capital available for growth investments as well?

Thanks for the loss function reference! I wonder if there’s something waiting to be discovered here about doing gradient descent but only taking steps with some probability. Definitely something to think about, I can’t imagine this idea hasn’t been explored before. Thanks a lot for the insightful comments, I’ve definitely seen that work in a very new light after knowing about it for years!!

And would the 2 memory algorithm be equivalent to a gradient descent with momentum?

I used to know what a sub gradient was, but I think there must be something more to the ideas in the paper because I’m struggling to see the analogy between gradient descent where you take steps probabilistically and the algorithm described. Perhaps I need to think about how you could potentially recast the quantile estimation problem as an optimisation problem and then apply what is effectively the machinery developed the train neural nets. Very interesting connection!

Related to this, one of my favourite articles [1] suggests that it’s sufficient to only use one or two pieces of memory to get good estimates. Here’s a pretty amazing result from the paper. To estimate the median (loosely) on a stream, first set the median estimate m to 0 prior to seeing any data. Then as you observe the stream, increase the median estimate m by 1 if the current element s is bigger than m. Do nothing if the current element is the same as m. Decrease the estimate by 1 if the element is less than m. Then (given the right conditions) this process converges to the median. You can even extend this to quantiles by flipping a biased coin and then updating based on the result of the flip as well as the element comparison.

The unfortunately now deceased neustar research blog had an interactive widget as well an outstanding write up[2]

[1]https://arxiv.org/abs/1407.1121

[2] http://content.research.neustar.biz/blog/frugal.html

That is definitely changing in credit risk and underwriting as well. There are several companies like [1] applying deep neural nets to the credit risk problem. This on top of a lot of in house work in the big banks to “supplement” what is available on the open market

[1] https://zest.ai/

Some quick comments about the iOS app.

1) it seems quite unintuitive to not be able to drag the cursor along the calculation and to rely on the arrow keys

2) the list is functions is long and not having text search (much like excel) is a major pain for usability. I don’t want to be scrolling ages to find the average function.

3) I also hope that the “commonly used” function list is adaptive, so it adapts to the functions that are used most often by the user

There's also this wonderful book [1] covering similar ground for Ruby. An extract from a review gives a flavour:

"Ruby Under a Microscope" does something fairly ambitious. It attempts to write a system internals book in a language that non computer scientists can readily understand. While there are numerous code snippets and examples to try and examine, the ability to look at the various Ruby internals and systems and see how they fit together can be accomplished by someone with general skills and basic familiarity with programming at the script level (which for many of us is as far as we typically get). An old saying says you can’t tell where yo hare going if you don’t know where you’ve been. Similarly, we can’t expect to get the most out of languages like ruby without having a more clear idea what’s happening under the hood. It’s entirely possible to work with Ruby and never learn some of this stuff, but having a guide like "Ruby Under a Microscope” opens up a variety of avenues, and does so in a way that will make the journey interesting and, dare I say it, even a little fun.

[1] https://www.amazon.com/Ruby-Under-Microscope-Illustrated-Int...

So, I am specifically talking about the scenario where all your labelling functions are highly correlated and there is little or no ground truth data to come up with empirical weights for each of the labelling functions. An example is the scenario where you have the label functions: x>5, x>4.99, x>5.01 for some feature x. I am really struggling to see how Snorkel can correct for the correlation, especially given the relatively simple generative model in section 2.2 of the paper. https://arxiv.org/pdf/1711.10160.pdf

One of the things I’m really curious about is how Snorkel deals with poor labelling functions. More generally, labelling functions are another data source for the model and are just as susceptible to corruption and other real world issues like completeness, bias, counter factual issues and repetition. Perhaps even more so because these are manually constructed. For example, you can imagine that the person writing labelling functions writes effectively the same rule many times over. My understanding of the paper is that Snorkel would then weight this repeated labelling very heavily. I think weak supervision techniques (at least the ones under the Hazy Research umbrella) require a degree of skill in machine learning that is easy to underestimate if you just think about the problem as an issue of domain understanding (or writing labelling functions in Snorkel terms)

I think it’s interesting to see the way these systems evolve. I lived in Singapore at the time of SARS and the speed of response was impressive to see. What is clear now almost 20 years later is the permanence of the knowledge and the improvement of what I thought was an already impressive system.

To summarise a lot of the responses here, Clickhouse is extremely fast on very modest hardware, very easy to set up, very easy to get started (mostly normal SQL) and free. For our workloads and scale of data, nothing comes close in terms of performance (redshift, BQ, spark) and especially TCO. It is super simple to try out, why not give it a go? Or is the question borne from a previous bad experience?

Hi Piotr, great to see your work make the front page of HN!

Will there also be an option to generate a combined pdf of the links for when you’re out of signal (like on the tube)?

You should probably also have some special handling for arxiv links. Those are the bulk of what we send on slack at Featurespace

Basically use the web APIs to issue two spend requests almost simultaneously, and the fact that it takes time for the database to synchronize means you can double spend. As the posters above are saying, it’s very similar to a race condition.

Second the “stupidly simple to setup and get running”. My company works with billion row datasets on client sites where we get super locked down accounts. Clickhouse is a single binary that you can run with no actual “install” needed.

Also echo the comments in the rest of the discussion about it being blazing fast. On our beefier machines we get querying in the 100s of millions of rows per second when the data is not in cache.