Wikipedia search-by-vibes through millions of pages offline

https://www.leebutterman.com/2023/06/01/offline-realtime-embedding-search.html

by gardenfelder • 3 years ago

325 58 3 years ago

This is a browser-based search engine for Wikipedia, where you can search for “the reddish tall trees on the san francisco coast” and find results like “Sequoia sempervirens” (a name of a redwood tree).

wrs• 3 years ago

“Vibes” is a way more relatable term than “sentence embeddings”. I may need to start using that. :)

xhevahir• 3 years ago

If it manages to be "relatable," it does so at the expense of a great deal of precision. OP doesn't explain his choice of wording, and it does not match any usage of "vibe." that I'm familiar with. Was "gist" not trendy enough, I wonder?

lsb8• 3 years ago

“Gist” to me implies accuracy, and has a meaning in GitHub, whereas the averaged embedding of 512-character chunks of text is more, uh, impressionistic.

janalsncm2• 3 years ago

The correct term would be “semantic search” but for a guy hand writing ONNX code I think I can assume they know that.

jiofj• 3 years ago

Low-hanging fruit: make article names clickable!

1-62• 3 years ago

Low hanging fruit could be a mountain of effort for those who'd otherwise continue to focus on improving the major feature.

hk__22• 3 years ago

Low hanging fruit could be a mountain of effort for those who'd otherwise continue to focus on improving the major feature.

"https://en.wikipedia.org/wiki/" + encodeURIComponent(title)

Here you have it. The major feature is in the title: you can hardly call it a Wikipedia search engine if you can’t access the articles.

brianpan• 3 years ago

Even lower-hanging fruit: put a space between the word and the rank so I can word select the title to copy-paste it.

googlryas• 3 years ago

I like the concept, but I'm not having much luck. I entered "weird looking monkey", hoping to get Proboscis, golden snub nosed, etc, but I just end up with the articles "Pet monkey","List of individual monkeys", "Ethnoprimatology", "Monkey".

Whereas when I type the same query into google, I get exactly what I expected. Which is kind of disappointing, I was hoping to find out about some weird looking monkeys that I didn't know about.

kikokikokiko• 3 years ago

Wikipedia editors/guidelines are generally not in favor of "opinionated" adjectives, and the use of "weird looking" in your query sounds a lot like something that would be frowned upon on a wikipedia article. This makes it hard for your search to retrieve good results on this corpus of knowledge.

znagengast• 3 years ago

It's also largely dependent on the embeddings model as others have mentioned. Even if wikipedia doesn't have any words specifically referring to that monkey as "weird", the model itself would know to correlate this monkey's embeddings with the "weird" concept. The main issue with this particular implementation is the model used (all-minilm-l6-v2) which is designed for speed and efficiency over accuracy.

kristopolous• 3 years ago

Couldn't someone pretty easily merge it with something more informal?

For instance, Reddit data dump (https://academictorrents.com/details/7c0645c94321311bb05bd87...), filter for Wikipedia links, include a context of the thread, combine that with the contents of the article

lsb8• 3 years ago

Yeah it’s an off the shelf sentence-transformer model from over a year ago. The demo was more to show off the embedding database, but the embeddings themselves are slightly useful too.

I don’t keep any analytics on the page about what people find and don’t find, so I haven’t set myself up to improve the search results :/

pizza2• 3 years ago

FWIW sentence-transformers truncates the input to at most 256 tokens by default; you might just be embedding the first paragraph or so.

marginalia_nu2• 3 years ago

It's incredibly impressive for what it does, but the results don't seem very good.

Although I know from experience it's really difficuly to assess search result quality by hand, you can be very close to something great and return far worse matches than this does.

lsb8• 3 years ago

Yes! The quality probably isn’t as good as Similar Website Finder https://explore2.marginalia.nu/ ;) and I bet using a more recent sentence embedding would lead to better results, I gotta collect more data

marginalia_nu2• 3 years ago

Haha, SWF is indeed unreasonably good (although it does something very differently and with a lot more processing power). Although I think in large part because I do no dimension reduction and brute force the cosine similarity calculation with raw 10,000,000 dimension vectors like the programming caveman I am.

1-62• 3 years ago

This is great news for those who suffer from memory recall problems. Hope to see more edge devices handle this inferencing locally.

Vt71fcAqt7• 3 years ago

Are diacritics supported? Searching "écorché" gave no relevant results. Cf. google.[0]

[0]https://www.google.com/search?q=%C3%A9corch%C3%A9+site%3Aen....

crazygringo• 3 years ago

This is certainly very interesting.

Unfortunately, I tried describing a few terms across philosophy and psychology and for all of them, the entry I was aiming for was only around the ~20th rank. (Far more popular but less accurate items were populated above it -- e.g. no matter what I typed trying to define a specific modality of psychotherapy, "psychotherapy" was always the #1 result.)

In contrast, I've used ChatGPT to identify the names of certain niche subfields when I couldn't remember what they were called, and it was right every time.

I love the idea of an AI service specifically designed to identify the names of things from descriptions. But I don't think restricting it to Wikipedia (or Wikipedia page titles) is the right approach, and it seems like general-purpose LLM's are doing a great job.

Still, as a proof of concept and as something you can run locally in the browser, this is extremely cool.

PartiallyTyped• 3 years ago

I have found myself describing ideas and goals, and getting back a field, or rather the name of it and certain keywords to look for. It seems that LLMs are the best fuzzy search engines and work in a rather unique though possibly complementary way to traditional search engines.

lsb8• 3 years ago

Thanks! The goal was to demo the database engine and show off how everything can work airgapped (after the browser downloads everything). I think there’s a lot of parameters to tune (Use just the first paragraph of the article, or everything? Search within some short distance of a particular article?) and I haven’t yet.

Wikipedia is a great demo dataset, and I’m definitely up for adding more datasets. Specifically, just like iPhoto lets you search “mountain” and you can get pictures with mountains, might be cool to search with some multi modal models like CLIP on various datasets

bagels• 3 years ago

It's really cool, but why doesn't it link to the wikipedia article?

atombender• 3 years ago

I don't know, I wanted to like this, but I didn't get any relevant matches for any of the searches I tried:

* "The wizard in The Lord of the Rings": No Gandalf or Saruman, only books about LOTR and such.

* "Protagonist of Scorsese's Taxi Driver": No Travis Bickle.

* "A person that plants trees for a living": Somehow a gardener isn't on the list.

* "Curly-haired painter on TV": No Bob Ross anywhere.

* "Unusually shaped modern art museum in Spain": Bilbao does show up as number 4, but none of the others are unusually shaped.

* "Dog shaped like a sausage": Surely a dachshund should be in the top results.

thewakalix• 3 years ago

It's worth noting that every result you wanted here does have a Wikipedia article. (If they hadn't, then their absence wouldn't be as strange.)

rgbrgb• 3 years ago

Love this demo but as others noted it's really easy to find queries where it performs poorly (e.g. typos).

Looks like the embedding model used (all-minilm-l6-v2) currently ranks 35th on the hugging face leaderboard [0]. I'd love to try with other models if anyone wants to +1 this demo :). This feels like a nice dataset to build intuition around embeddings used for RAG etc.

[0] https://huggingface.co/spaces/mteb/leaderboard

lsb8• 3 years ago

Lemme try a few other embedding models after the weekend :)

janalsncm2• 3 years ago

A production-ready search engine runs off of a lot more than embeddings. They will have special logic to handle all sorts of special cases, as well as reranking models to show the most relevant results at the top. To me this is more of a demo of client-side vector search, which can be useful for other things.

lovasoa• 3 years ago

The tech is very impressive but the results are not.

I searched "pointy building in Paris", and got :

Tourism in Paris, Bourse de commerce (Paris), Grands Projets of François Mitterrand, List of tallest buildings and structures in the Paris region, List of tourist attractions in Paris, Palais des congrès de Paris, Landmarks in Paris, Palais de la Bourse, Lyon, Outline of Paris, Architecture of Paris

no mention of the most famous pointy building in Paris...

Maybe sentence embedding of the entire article is not the best thing for this kind of application.

sp3322• 3 years ago

At least 5 of those would have the answer to your question.

scubbo• 3 years ago

But are not _the_ answer to the question.

extraduder_ire2• 3 years ago

If you mean the Eiffel Tower, it's not a building.

I just checked the article, and of the 19 times the word "building" appears, it's mostly a verb, followed by "Chrysler Building"

Unless there's some other famously pointy building I'm not thinking of.

yorwba• 3 years ago

https://www.wikidata.org/wiki/Q243 (the Eiffel Tower, which is so famous it gets an extremely small ID) is an instance of https://www.wikidata.org/wiki/Q1440476 (lattice tower), a subclass of https://www.wikidata.org/wiki/Q12518 (tower) a subclass of https://www.wikidata.org/wiki/Q41176 (building).

jasonthorsness• 3 years ago

The quality of the embeddings is a limiting factor for this sort of search - OpenAI text-ada embeddings are great but that removes the local aspect, and the better huggingface models are too big. With the model sizes increasing it’s hard to see what the path will be for local/offline.

vikp• 3 years ago

There are plenty of great embedding models that are on the order of a few hundreds megs (even outperforming ada-002). See the leaderboard here - https://huggingface.co/spaces/mteb/leaderboard. Local/offline is only growing.

jasonjmcghee• 3 years ago

Wow gte-small feels like a pretty great balance of size and quality (all-MiniLM-L6-v2 has been my go-to)

nullc• 3 years ago

Good offline search could be a major advance in personal privacy.

dougb5• 3 years ago

Really nice implementation! And it's so cool to be able to do this offline. The embeddings aren't quite there yet.

One trick that might be helpful is to embed only the defining (usually the first) sentence or paragraph of the Wikipedia article, rather than the whole document -- not clear to me which portion you're using now.

My own site, OneLook, has had a similar feature (https://onelook.com/thesaurus/) since '03 that lets you find words and concepts by description. It was a pure reverse-dictionary search back when I started, but over the past two decades I've explored word embeddings, then sentence embeddings, and more recently LLMs. Nowadays it uses GPT to generate some guesses for inputs that it can't answer itself.

LLMs are so much better than earlier methods at this task, it's taken some of the wind out of my sails on improving this aspect of OneLook. I frequently hear from people for whom reverse-definition lookups are the main reason they use ChatGPT!

gandalfff• 3 years ago

Having this integrated into Kiwix would be great!

kemayo• 3 years ago

The page is currently failing to work for me, because `model_quantized.onnx` isn't loading -- I'm watching it and it has currently managed to get 19.2MB downloaded (at ~50KB/sec) as I type this, so if every visitor is triggering that...

I think we may be doing awful things to Lee Butterman's bandwidth bill.

lsb8• 3 years ago

It’s static files on one t2.nano! Who knows.

mcint• 3 years ago

Looks like it would help to use a (free) CDN for the static files. You could set up a subdomain that provides caching access to the base site.

lsb8• 3 years ago

Wow, so cool to see this here. Happy to answer any questions

jpp• 3 years ago

Neat to see your stuff show up here!

nonbirithm• 3 years ago

I searched "wabisabi tastes like legos" and got back "Kiwiana" so I dug into a bunch of New Zealand jangle pop bands.

I think it's worth it.

markdown• 3 years ago

Headsup: don't start typing into the form expecting to get a result. It's actually just a screenshot in a blog post.

hamilyon2• 3 years ago

Part of search engine magic is mixing in embeddings (or traditional information retrieval keywords) of the pages that link to the page weighted by clicks and authority score, for example.

Without this signal, a lot of useful information is ignored and the result doesn't feel as magical.

Still impressive, fascinating demo

jxmorris12• 3 years ago

A little late to the party here, but text embeddings (at least the ones used in this blog post) generally aren't very good at "searching by vibes": they more compare by overlapping words or look for similar content to the search query.

However, there is a recent paper that actually does try and do this: "Retrieving Texts based on Abstract Descriptions" (Ravfogel et al., 2023) https://arxiv.org/abs/2305.12517.

They give many examples of searching by vibes: "an architect designing a building", "a company which is part of another company", "a book that influenced the development of a genre", etc. etc. Their embeddings apparently facilitate this type of search much better. Would be interesting to retry the offline Wikipedia search from the linked post with this new type of embeddings.

nborwankar• 3 years ago

Meta comment about the comments saying “I didn’t find what I was looking for” - core fact is it was using Wikipedia as a knowledge base. So if your topic is not represented well enough in Wikipedia it’s not going to return good results. Secondly, comparing something that runs on your mobile device or laptop to Google is like comparing an apple to a container ship full of oranges.

YounesDz• 3 years ago

Thank you for making this.

What's the number means near words in the results? For example "book 70k" what's 70k refer to? The number of discussions on Wikipedia about it? #s of Edits? Or articles that mentioned it?

Wikipedia search-by-vibes through millions of pages offline

Related Stories