Semantic search engine for ArXiv, biorxiv and medrxiv

by 0101111101 • 1 year ago

149 31 1 year ago

elliotec2• 1 year ago

This is really cool, and very relevant to something I'm working on. Would you be willing to do a quick explanation of the build?

010111110110• 1 year ago

Sure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved!

I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...

heisenburgzero• 1 year ago

So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?

elliotec2• 1 year ago

Thank you!!

cluckindan2• 1 year ago

That method can break when author names and subject matter collide.

madars2• 1 year ago

Looks great! Could you add eprint.iacr.org (Cryptology ePrint Archive)?

010111110110• 1 year ago

Do they have a public API/dataset?

madars2• 1 year ago

They have RSS feeds for new/updated papers: https://eprint.iacr.org/rss/

bbor• 1 year ago

Oh god, there's a medrxiv?? TIL...

Don't forget chemrXiv!

010111110110• 1 year ago

Sadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!

BioGeek• 1 year ago

Here it is:

https://chemrxiv.org/engage/chemrxiv/public-api/documentatio...

Fomite• 1 year ago

medrxiv was very useful for keeping the various COVID-19 related preprints from completely swamping biorxiv, especially once biorxiv started aggressively rejecting them.

sitkack4• 1 year ago

https://en.wikipedia.org/wiki/List_of_preprint_repositories

sitkack4• 1 year ago

embedding search via https://searchthearxiv.com/ takes either a word vector, or an abs or pdf link to an arxiv paper.

https://news.ycombinator.com/item?id=42519487

I just did a spot check, I think searchthearxiv search results are superior.

010111110110• 1 year ago

Looks cool! You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`

sitkack4• 1 year ago

That is neat I like that.

It would be cool if the "More Like This" had a + button that would append the arxiv id to the search query.

masterjack• 1 year ago

There’s also the search and browsing on https://sugaku.net, it’s more focused on math but does also have all of the arxiv on it

nblgbg• 1 year ago

Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.

forrestp• 1 year ago

My understanding is that your levers are roughly better / more diverse embeddings or computing more embeddings (embed chunks / groups / etc) + aggregating more cosine similarities / scores. More flops = better search w/ steep diminishing returns

Colbert being a good google-able application of utilizing more embeddings.

Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in subsequent passes on the previous result set.

010111110110• 1 year ago

Exactly! A near property of the matryoshka embeddings is that you can compute a low dimension embedding similarity really fast and then refine afterwards.

itzlambda2• 1 year ago

Using "+" and "-" in search is interesting idea.

I've built similar thing for github stars[1], might implement the same for it.

[1] https://starscout.xyz/

sitkack4• 1 year ago

I assume you have seen https://anvaka.github.io/map-of-github/#2/0/0

itzlambda2• 1 year ago

I had not, thank you for the link!

Semantic search engine for ArXiv, biorxiv and medrxiv

Related Stories