Is there a way to run an LLM as a better local search engine?

https://news.ycombinator.com/item?id=44307466

by oblio • 1 year ago

9 19 1 year ago

news.ycombinator.com

Basically, I was thinking that a way I could actually use LLMs would be to point them at my hard drive, with hundreds of images, PDFs, XLS' and other random files, and start asking it questions to easily find things in there. Can a local LLM run OCR software on its own?

I'm on Windows, if it matters. Is there anything like that out there, already (mostly) built?

DHRicoF• 1 year ago

You need to provide more information.

Is your data organized or is just a dump of unrelated content?

- If you have a bag of files without any metadata the best option is to create something like a RAG, with a pre OCR step for image files (or even some multimodal model call).

- If the content is well organized with a logic structure an agent could extract information with a little look around.

Is static or varies day by day?

- If is static you could index all at once, if not, an agent that pick what to reindex would be a better call.

I'm not aware of a solution like this, but seems doable as an MCP server. But the cost will scale quiclky.

oblio6• 1 year ago

I want the LLM to search my hard drives, including for file contents.

I have zounds of old invoices, spreadsheets created to quickly figure something out, etc.

I'd also want the tool to run in the background to update the index.

I've found something potentially interesting:

https://anythingllm.com/

cranberryturkey2• 1 year ago

local as in on your filesystem?

yen223• 1 year ago

What other forms of "local" are there?

Don't mean to be snarky, apologies if it comes across like that. I'm genuinely curious

cranberryturkey2• 1 year ago

like my neighbors server lol

oblio6• 1 year ago

Filesystem, yes.

Iolaum• 1 year ago

Have you tried asking this question at an LLM? :p

oblio6• 1 year ago

I did and there isn't anything out of the box out there.

I'd basically want Everything as an LLM: https://www.voidtools.com/support/everything/, but also with file content indexing.

aliasmaya• 1 year ago

Seems that you're looking for a RAG System, and you may have a try RAGFlow

oblio6• 1 year ago

Interesting, I had found Paperless:

https://github.com/icereed/paperless-gpt

https://docs.paperless-ngx.com/#features

These options seem far from... user friendly. Another concern is resource usage, I wonder how low LLMs will go (especially as far RAM and GPU requirements are concerned).

Agraillo• 1 year ago

What you described would be a great solution for plenty of tasks, but maybe solving some fallacies one at a time would also be great. For example, when we're sure by placing a properly named file at a directory location, we will find it by recalling the folder name or the name of the file itself while in reality we're often surprised that after months or years this won't work, the expected path to the file either not exists or doesn't contain what we're looking for. The same fallacy is true also for different hierarchical notes organizers.

In this case LLMs with their ability to find semantic equivalence might be a great help. And with the current state of affairs I even think that an LLM with a sufficiently large context window might absorb some kind of the file system dump with directory paths and file names and answer a question about some obscure file from the past.

oblio6• 1 year ago

I think by far the biggest barrier to entry for this is actually resource usage.

Let's say there's a HDD somewhere that has thousands of files: text files, PDFs, XLS, PPT, DOC, etc. That doesn't sound like a huge amount of data to me.

However, there doesn't seem an out-of-the-box solution to ingest this into an LLM and run it to ask it simple stuff like "can you list all the invoices from 2023 and their paths?" without requiring stuff like 16GB of RAM and 8GB of VRAM, which basically puts this "search" solution out of reach for the average laptop, especially Windows laptop, in the last 5 years and probably for the next 5-10 years, too.

It's a shame, but, oh well...

antinomicus• 1 year ago

I personally wouldn’t care if shitty windows laptops can’t run this….my machine has 64gb unified memory and Apple should be working hard on these types of features. Instead we get custom emoji…

msgodel• 1 year ago

Multimodal Qwen is pretty good at OCR although it's pretty slow without a GPU.

For pure search you're almost certainly better off building an index of CLIP embeddings and then doing cosine similarity with a query embedding to find things. I have gigabytes of reaction images and memes I've been thinking about doing this with.

cylinderthought• 1 year ago

i been working on something like this for my own stuff. my drive got screenshots pdfs md files invoices and random logs and i always forget what i named stuff from years ago

what helped me was

- ran ocr on images with tesseract (slow but it works)

- used unstructured and langchain to parse and chunk stuff even spreadsheets and emails

- embedded chunks with sentence-transformers and indexed it with faiss

- then built a local llm agent (used a quantized mistral model) to rerank results smartly

its rough but works like a semantic grep for your whole disk

if you want less diy paperless-ng plus anythingllm plus a lightweight embed model could work or wait some months and someone will wrap it all in an electron app with stripe on the homepage lol

funny how much time we spend trying to find stuff we already wrote

didgetmaster2• 1 year ago

Local hard drive are big enough these days to hold hundreds of millions of files. That means that you could have many millions of each type of file (documents, pictures, spreadsheets, videos, etc.)

We need better ways to properly classify all that data with accurate metadata and quick ways to pick out a small subset of data for A.I. to analyze.

Databases were designed to do this with relational tables (e.g. find all customers from New York who bought our product in 2024); but file systems were not designed to do this with files (e.g. find all pictures I took with my phone in 2020).

A.I. can be a great tool to find patterns in files to answer important questions, but it will be incredibly slow if it has to analyze too many files for every query.

oblio6• 1 year ago

The low level, file system, solution will probably not be widely implemented within my lifetime.

didgetmaster2• 1 year ago

Unfortunately, I tend to agree. I created a great solution (I think), but I am not Microsoft or Apple so getting any traction for it is incredibly hard.

android521• 1 year ago

first , have a good file search system. Then LLM is just the middle that translate human language into commands to search the files and then translate the result back in human langauge.

Is there a way to run an LLM as a better local search engine?

Related Stories