svcrunch

The grandparent is definitely wrong on (3). Yes, coding is a killer product, I agree with you.

On (2), I agree with you for local models. BUT, there are also the open source Chinese models accessible via open-router. Your argument ("don't hold a candle to SOTA models") does not hold if the comparison is between those.

On (1), I agree more with the grandparent than with your assessment. Yes, OpenAI and Anthropic are killing it for now, but the time horizon is very short. I use codex and claude daily, but it's also clear to me that open source is catching up quickly, both w.r.t. the models and the agentic harnesses.

I generally don't waste time with patents. I think most patents in deep learning can be overturned by prior art.

My current approach to IP is trade secrets. If we publish, we are careful to avoid details that would make the techniques easy to productionize.

Thanks for your interest. The rerankers are external, GoodMem is a unified API layer that calls out to various providers. There's no model running inside the database or the GoodMem server.

We support both commercial APIs and self-hosted options:

  - Cohere (rerank-english-v3.0, etc.)
  - Voyage AI (rerank-2.5)
  - Jina AI (jina-reranker-v3)

  - TEI - https://github.com/huggingface/text-embeddings-inference
  - vLLM - https://docs.vllm.ai/en/v0.8.1/serving/openai_compatible_server.html#rerank-api

  # Cohere
  goodmem reranker create \
    --display-name "Cohere" \
    --provider-type COHERE \
    --endpoint-url "https://api.cohere.com" \
    --model-identifier "rerank-english-v3.0" \
    --cred-api-key "YOUR_API_KEY"

  # Self-hosted TEI (e.g., BAAI/bge-reranker-v2-m3)
  goodmem reranker create \
    --display-name "TEI Local" \
    --provider-type TEI \
    --endpoint-url "http://localhost:8081" \
    --model-identifier "BAAI/bge-reranker-v2-m3"

  goodmem memory retrieve \
    --space-id <your-space> \
    --post-processor-interactive \
    "your query"

Hi there, thanks for writing and sharing your experiences. I'm one of the builders of GoodMem (https://goodmem.ai/), which is infra to simplify end-to-end RAG/agentic memory systems like the one you built.

It's built on Postgres, which I know you said you left behind, but one of the cool features it supports is hybrid search over multiple vector representations of a passage, so you can do a dense (e.g. nomic) and sparse (e.g. splade) search. Reranking is also built in, although it lacks automatic caching (since, in general, the corpus changes over time)

It also deploys to fly.io/railway and costs a few bucks a month to run if you're willing to use cloud-hosted embedding models (otherwise, you can run TEI/vLLM on CPU or GPU for the setup you described).

I hope it's helpful to someone.

Here's a problem that no frontier model does well on (f1 < 0.2), but which I think is relatively easy for most humans:

https://dorrit.pairsys.ai/

This benchmark evaluates the ability of multimodal language models to interpret handwritten editorial corrections in printed text. Using annotated scans from Charles Dickens' "Little Dorrit," we challenge models to accurately capture human editing intentions.

This is really cool.

Various frontier LLMs were evaluated on their ability to interpret handwritten proofreading marks in printed literary text, using a small benchmark based on Charles Dickens's "Little Dorrit". Results are modest at best, and surprisingly variable across repeated runs, even on the same pages, underscoring the challenge in building reliable, structured-document systems with current multimodal LLMs.

Curious to hear thoughts from others working on similar problems.

No.

But to your point, note that in 2020 neuroscientists introduced the Tolman-Eichenbaum Machine (TEM) [1], a mathematical model of the hippocampus that bears a striking resemblance to transformer architecture.

Artem Kirsanov has a very nice piece on TEM, "Can we Build an Artificial Hippocampus?" [2] The link is directly to the spot where he makes the connection to transformers, although you should watch the whole video for context.

Because I wasn't clear on the chronology, I went back and asked one of the "Attention" authors whether mathematical models of the hippocampus inspired their paper? His answer was "no". If TEM was developed without pre-knowledge of transformers, then it's a very deep result IMHO.

[1] https://www.sciencedirect.com/science/article/pii/S009286742...

[2] https://www.youtube.com/watch?v=cufOEzoVMVA&t=1254s

While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title.

As others have pointed out, self-attention was already a known concept in the research community. They don't claim to have invented that. Rather, the authors began by looking at how to improve the power of feed-forward neural networks using a combination of techniques, obtained some exciting results, and then, in the course of ablation studies, discovered that attention was really all you needed!

The title is a play on the Beatles song, "All You Need Is Love".

In terms of expository style, the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238) by Phuong and Hutter. Written for clarity and with an emphasis on precision, the motivation section (Section 2) of the paper does a great job of explaining deficiencies in the original paper and subsequent ones.

Take a look at the BEIR benchmark, which has served as one of the main drivers for development of neural IR systems since its introduction in 2020.

BM25 presents a challenging cross-domain benchmark, and it wasn't till ~2022 that neural methods overtook it. If memory serves, it was the sparse neural methods like Splade, although recent dense models can also beat it.

The caveat is that BEIR is suffering from overfitting at this point.

but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store.

In my experience, HNSW indexes are very expensive to build, relative to indexes like IVF. They also have a larger memory footprint. IVF, on the other hand, is pretty trivial to parallelize across multiple machines, and while I'm aware there are techniques for doing that with HNSW, I don't know the details well enough.

Also, if you review papers like "SOAR: Improved Quantization for Approximate Nearest Neighbor Search", they hint at some of the throughput barriers faced by graph-based methods like HNSW.

I think your comment is accurate, but regarding your last point:

"However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated."

How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data?

Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.

While transformer-based AI is very powerful, and its potential uses in the business world nearly limitless, the issue of hallucination is holding back adoption. Here, Simon Hughes of Vectara introduces an open-source model, HEM, that can automatically detect hallucinations with fairly low latency.

Disclosure: I am one of the founders of Vectara and head research there.

The metrics presented in the blog post are those of our production model. When designing Boomerang, we tried to balance latency and search relevance in a manner that strikes the right balance for most use cases.

On the other hand, GTR-XXL is an example of a research model that biases in favor of search relevance, at the expense of latency. It's not really practical to deploy in production environments as a result.

I believe that retrieval-augmented generation is the right path to generative AI within organizations, at least for the next few years. Trying to directly fine-tune an LLM on your data also runs into issues with enforcing access permissions.

However, instead of simply being a post-processing step at the end of an IR pipeline, LLMs will eventually sandwhich the IR system, along the lines of the [Demonstrate, Search, Predict framework](https://arxiv.org/abs/2212.14024) by Khattab et al.

I'm curious, is HSNW the only option? Do you support IVF-style indexes? Also, FAISS is nice because it supports a pluggable storage layer. Is this something that's easily supported in USearch?

Great work, and thank you for your contributions.

Thank you for this!

I know that Cruikshank was the original illustrator of many of Dickens's novels, but I prefer the artwork of James Mahoney. As a point of comparison, the same scene by both artists:

1. "Oliver Rather Astonishes Noah", https://imgur.com/a/DWeblXT, as illustrated by James Mahoney.

2. "Oliver Plucks up a Spirit", https://www.charlesdickensillustration.org/oliver-twist?pgid..., as illustrated by George Cruikshank.

I scanned and vectorized all the artwork for "Oliver Twist" and typeset it at http://ahmadsoft.org/downloads/Oliver%20Twist,%20or,%20The%2... (warning, it's a large PDF due to the high resolution vector imagery), but then I started a company in 2020, and haven't had time to finish "Little Dorit", which, compared to "Oliver Twist", I was able to scan at a much higher resolution and get better quality.

This guide reviews 13 prominent instruction-following LLMs given constraints like commercial or non-commercial usage, and self-hosted versus API access.

The rise of neural retrieval systems has diminished the need to define stopword lists.

This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017.

To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5].

Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques, benchmarks like BEIR [7], released a few years later, in 2020, showed that the best keyword retrieval algorithms still outperformed neural techniques in general settings.

The landscape since then has shifted very rapidly: In 2022, for the first time, neural search methods outperformed BM25 on BEIR. This includes late interaction [8], sparse encoding [9], and, most challengingly, dense encoding [10] systems.

The second technical challenge is scalability. After decades of infrastructure optimization, keyword systems scale well to very large corpora, while semantic systems struggle to achieve the same scale. The k-d tree approach presented in the article, for example, while good for experimentation, would be difficult to productionize, as-is, in a large-scale system.

However, research into scaling dense vector retrieval has received a lot of focus recently [11], so I'm confident this will change.

I'll close by saying your observation about being stuck with keyword search in a lot of apps is accurate, but I expect that to change soon. It's becoming easier to embed neural models everywhere, and I think that distilled models in the 5-50mb size range can feasibly power semantic search everywhere you press Ctrl-F today.

[1] https://blog.google/products/search/search-language-understa...

[2] https://blog.google/products/search/introducing-mum/

[2] https://arxiv.org/abs/1706.03762

[3] https://arxiv.org/abs/1810.04805

[4] https://arxiv.org/abs/1907.04307

[5] https://books.google.com/talktobooks/

[6] https://ai.googleblog.com/2018/04/introducing-semantic-exper...

[7] https://arxiv.org/abs/2104.08663

[8] https://arxiv.org/abs/2112.01488

[9] https://arxiv.org/abs/2109.10086

[10] https://arxiv.org/pdf/2112.09118.pdf

[11] https://www.microsoft.com/en-us/research/uploads/prod/2021/1...

JDBC definitely. I didn't mention it because I didn't think of it as part of the Java EE spec (JDBC preceded Java EE if I'm not mistaken).

As far as Jetty handlers, they might be a good alternative to servlets. I admit I'm not familiar with the details.

I think that's a bit harsh :-) They do mention DIN 66399 [1], which I didn't even know is a thing. It makes me wonder why Trump had to resort to flushing documents down the toilet?

[1] https://www.shreddingmachines.co.uk/securitylevels.asp?id=1&...

My advice would be to focus on servlets deployed into an embedded Jetty instance within your server, and that's about it!

The other parts, like Session and Entity EJBs, JMS, Java Server Pages, SOAP, SAR and WAR packaging formats, JNDI etc. etc., have long ago been supplanted by much better technologies.

I agree with many aspects of your assessment, but when you say, "The concept of an easy to use, fire-and-forget, scalable search engine that gives you great relevance out of the box isn't new. Yet the bag-of-words model remains the standard across the sector ...", I think you have to weigh the emergence of transformer-based neural networks around 2017 more strongly.

In 2018, BERT, the first demonstration of a pretrained large language model (LLM), exceeded human performance on Stanford's Question Answering Dataset [1]. Nobody in 2010 predicted such rapid progress.

Between 2017 and 2020, I worked with several teams managing very complex search systems. In one case a single LLM obviated dozens of hand-tuned relevance signals developed over the better part of a decade.

One of the main effects of neural search adoption will be raising the baseline quality of search; a second will be reduction in the overall cost and complexity of search impementations.

For example, it's not easy to configure a keyword system to find "works fine, We have two Roku's [sic] in other televisions which are working fine" in response to "does it work with different tvs?". But neural search finds this result directly, without any tuning or configuration [2].

Thank you for sharing the video and the article!

[1] https://www.nytimes.com/2018/11/18/technology/artificial-int...

[2] https://www.youtube.com/watch?v=Tn7AqmY9yaY&t=112s

Also take a look at Semantris:

https://research.google.com/semantris

You give it a query and it returns a value that it has learned to associate with the query during training.

The zero-shot scenario they describe does not work like this. They explicitly mention that it's not trained with any queries (which is what makes it a very promising technique).

This is fascinating, as far as pushing the creative boundaries of how to accomplish IR with neural networks, and showing the potential headroom that's available.

It seems wildly impracticable to productionize at the moment (excepting Google, perhaps). If I'm not misunderstanding, the index is actually built by training a neural network (they use networks ranging from 250M-11B parameters, i.e. 500MB-22GB in size). Still, for an important collection of documents, this might be how it's done in a few years.

In a similar vein, we showed how to make Rasa search for answers using ZIR Semantic Search:

https://blog.zir-ai.com/semantic-search-helps-chatbots-answe...

There was a dramatic jump in recall (13->74%) and F1 (20->63%). I believe that neural search is the future for most search problems.

Agree with your sentiment.

I cofounded ZIR AI to provide ML (vector)-based search as a PaaS solution, similar to what Algolia or Elasticsearch do for keyword searching.

We have a demo (https://zir-ai.com/demo) running over Quanta Magazine articles. Not only can it outperform the keyword search embedded on quantamagazine.org, but one some queries, it even outperforms Google with a site restrict (e.g. how old is the universe site:quantamagazine.org).

Granted, it's rare, but if you manage it, it's great.

ML-powered search will make this commonplace, once the tech goes mainstream in the next 2-3 years.

There must be a problem with the mp3. I'm listening online and there's no such issues.