MemoryCache: Augmenting local AI with browser data

https://future.mozilla.org/blog/introducing-memorycache/

by NdMAND • 3 years ago

468 101 3 years ago

future.mozilla.org

Jayakumark• 3 years ago

Was just talking about this on reddit like two days ago

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

While there are many OSS for Loading personal data, they dont do images or videos. In the future everyone may get their own Model but for now tech is there but product/OSS is missing for everyone to get their own QLORA or RAG or Summarizer.

Not just messages/docs: What we read or write, and our thoughts are part of what makes an individual unique. Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today.

Was just watching this yesterday https://www.youtube.com/watch?v=zHLCKpmBeKA and thought, why we still don't have a computer secretary like her after almost 30 years, who is one step ahead of us.

butz2• 3 years ago

I assume that training LLMs locally require high-end hardware. Even running a model requires a decent CPU or, even better, a high end GPU, but it is not so expensive as training a model. And usually you have to use hardware that is available on the cloud, so not much of privacy here.

cjbprime• 3 years ago

You don't need to train the model on your data: you can use retrieval augmented generation to add the relevant documents to your prompt at query time.

smcleod3• 3 years ago

As someone else said you don't need to train any models, also - small LLMs (7b~) can run really well even on a base M1 Macbook air from 3 years ago.

simonw• 3 years ago

"While there are many OSS for Loading personal data, they dont do images or videos"

Local models for images are getting pretty good.

LLaVA is an LLM with multi-modal image capabilities that runs pretty well on my laptop: https://simonwillison.net/2023/Nov/29/llamafile/

Models like Salesforce BLIP can be used to generate captions for images too - I built a little CLI tool far that here: https://github.com/simonw/blip-caption

orbital-decay2• 3 years ago

CogVLM blows LLaVA out of the water, although it needs a beefier machine (quantized low-res version barely fits into 12GB VRAM, not sure about the accuracy of that).

csbartus• 3 years ago

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

That's the most important idea I've read since ChatGPT / last year.

I'll wait for this. Then build my own private AI. And share it / pair it for learning with other private AIs, like a blogroll.

As always, there will be two 'different' AIs: a.) the mainstream, centralized, ad/revenue-driven, capitalist, political, controlling / exploiting etc. b.) personal, trustworthy, polished on peer networks, fun, profitable for one / a small community.

If by chance, commercial models will be better than open source models, due to better access to computing power / data, please let me know. We can go back to SETI and share our idle computing power / existing knowledge

jakderrida• 3 years ago

Our browsing history tells a lot about what we read but no one seems to make use of it other than google for ads.. Almost everyone has a habit of reading x news site, x social network, x youtube videos etc.. Ok, here are the summary for you from these 3 today.

I was imagining something a little more ambitious. Like a model that uses our search history and behavior to derive how to best compose a search query. Bing Chat's search queries look like what my uncle would type right after I explained to him what a search engine is. Throw in some advanced operators like site: or filetype: or at least parentheses along with AND/OR. Surely, we can fine tune it to emulate the search processes of the most impressive researchers, paralegals, and teenagers on the spectrum that immediately factcheck your grandpop's Ellis Island story, with evidence he both arrived at first and was naturalized in Chicago.

amelius• 3 years ago

Local compute is so 80s, when people moved away from dumb terminals and mainframes, to PCs.

simondotau• 3 years ago

Yes but this time we call it “distributed computing”or “edge computing” instead.

gpderetta• 3 years ago

remote computing is so late '90s when people moved away from PCs to servers (the dot in dot com).

Turns out this sort of stuff is cyclical.

conradev3• 3 years ago

Instead of data going to models, we need models come to our data which is stored locally and stay locally.

We are building this over at https://software.inc! We collect data about you (from your computer and the internet) into a local database and then teach models how to use it. The models can either be local or cloud-based, and we can route requests based on the sensitivity of the data or the capabilities needed.

We're also hiring if that sounds interesting!

gardenhedge• 3 years ago

Wow, nice domain. I'd work there for the name alone haha.

thepra• 3 years ago

As fare as I can see it's just a MacOS image, nothing is happening

herval• 3 years ago

site's pretty funny, but would likely be more useful with more information and less clicking-around-nostalgia 8-)

bjord• 3 years ago

your site is not loading at all for me on firefox (emulator error) and is totally non-functional on chrome (TCPCreate failed)

might be worth having some sort of automatic fallback to a static site after a certain amount of failed loading or an error

just saw the link to your html version in another comment and it took literally five minutes to load on firefox

pradn• 3 years ago

Yes, should have local models in addition to remote models. Remote ones are always going to be more capable, and we shouldn't throw that away. Augmentation is orthogonal - you can augment either of these with your own data.

nullc2• 3 years ago

Just having an archiver that gives you a tradition search over every webpage you've loaded-- forget the AI stuff, would be a major advance.

I don't know about everyone but a majority of searches are for stuff I've seen before, and they're often frustrated by things that have gone offline or are downranked by search engines (e.g. old documentation on HTTP only sites) or burred by SEO.

mclightning• 3 years ago

you will be shocked when you try Rewind then...

timenova• 3 years ago

I believe that's exactly what GitHub Copilot does. It first scans and indexes your entire codebase including dependencies (I think). So when it auto-completes, it heavily uses the context of your code, which actually makes Copilot so useful.

You're absolutely right about models coming to our data! If we could have Copilot-like intelligence, completely on-device, scanning all sorts of personal breadcrumbs like messages, browsing history, even webpage content, it would be a game-changer!

diceduckmonk• 3 years ago

Google already tried this 15 years ago

https://en.m.wikipedia.org/wiki/Google_Search_Appliance

bloopernova• 3 years ago

Regarding PrivateGPT, if I have a 12GB Nvidia 4070 and an 11GB 2080ti, which LLM should I run?

Edited to add: https://www.choosellm.com/ by the PrivateGPT folks seems to have what I needed.

SkyMarshal2• 3 years ago

There's a big community discussing exactly that over at https://www.reddit.com/r/LocalLLaMA/.

smcleod3• 3 years ago

+1 r/localllama, 23GB should allow you to run 30b~ models, but honestly some of the new smaller models such as Mistral & friends (Zephyr etc..) are really interesting. You could also Give Mixtral a try if you get a low quant format such as this q3 - https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-G...

avallach• 3 years ago

PrivateGPT repository in case anyone's interested: https://github.com/imartinez/privateGPT . It doesn't seem to be linked from their official website.

nightski• 3 years ago

It's linked from the MemoryCache repo listed at the bottom of the article: https://github.com/Mozilla-Ocho/Memory-Cache

edsu3• 3 years ago

Does anyone know what PrivateGPT is using for its local model, and where it came from?

Update:

Answering my own question it looks like it uses llamacpp in local mode? https://github.com/imartinez/privateGPT/blob/main/private_gp...

linsomniac• 3 years ago

I would sure love a way to "chat" with my browsing history and page content. Is there any way to automatically save off pages that I've visited for later processing? I looked a decade or more ago and didn't really find a good solution.

kaynelynn• 3 years ago

Rewind.ai is pretty much this - I just installed it and am very happy so far.

Alifatisk• 3 years ago

Isn't it Apple devices only?

solarkraft• 3 years ago

I think WorldBrain (https://github.com/WorldBrain/Memex) promises this. While I'm also excited by the idea, I think there was some reason I ended up not using it.

wizardwes• 3 years ago

Zotero might work, but only as a highly imperfect solution, since it is more focused on research

rhn_mk1• 3 years ago

Check out Recoll with Recoll-WE https://addons.mozilla.org/en-US/firefox/addon/recoll-we/

Dwedit2• 3 years ago

Very misleading name. The word "Memory" has a distinct meaning in relation to computing, but this is more about human memories.

Sai_• 3 years ago

I was going to ignore this as a troll comment because computer memory has its antecedents in human memory but the commenter is right - the combination of memory and cache to talk about human memory seems misleading.

edsu3• 3 years ago

I kind of like the association since it speaks to how text collected while browsing the web can be used to generate new text, which is similar, at least metaphorically, to how human memory is reconstructive and transformative, not perfect recall. https://en.wikipedia.org/wiki/Reconstructive_memory

CollinEMac• 3 years ago

I'm confused by the example they gave.

What is the meaning of a life well-lived?

Is the response to this based on browser data? Based on the description I was expecting queries more like:

What was the name of that pdf I downloaded yesterday?
What are my top 3 most visited sites?
What type of content do I generally interact with?

ipaddr• 3 years ago

That information is already available. You want a better search interface.

ape4• 3 years ago

They could private a local url called "about:wrapped" that gives a summary of your usage like Spotify Wrapped. The top 100 sites, you can click on a site for more info like what pages did you visit, when, how often, etc.

lacker2• 3 years ago

Yes, exactly, I want a search interface that's an LLM instead of a bunch of menus.

atomicUpdate• 3 years ago

One thing you'll see from a lot of these LLM examples and demos are intentionally subjective queries, so they can't be judged on pass/fail criteria.

For example, you'll see things like "where should I visit in Japan?" or "how should I plan a bachelor party?", because they are a huge variety of answers that are all "correct", regardless of how much you disagree with them. There is also a huge number of examples from them to draw from, especially compared to something as specific as your browsing history.

reqo• 3 years ago

What I would love to see is this model being able to learn ti automate some tasks that I usually do! e.g. sign up for events/buy tickets etc! If this has access to your login details, and could log in, it could be a great assistant!

redblacktree• 3 years ago

Or shoe bot

candiddevmike• 3 years ago

Teach it to press the skip ad button

lacker2• 3 years ago

Or it could click "hide" on cookie banners for me!

jml7c5• 3 years ago

I hope this encourages Mozilla to focus more on page archiving support on the web. I feel as though they missed a huge opportunity by not making it easy to archive pages with DOM snapshots, or easy to snag videos or images. (Go to Instagram and try to right-click -> download the image; you can't.) Would have been a very good way to differentiate from Chrome, as Google wouldn't want that available for Youtube. And "our browser can download videos and images from anywhere" is pretty easy to sell for potential users.

eigenvalue2• 3 years ago

Agree, it seems like it’s insanely hard to back up a modern JS-enabled web page in a usable way that results in a single file which can be easily shared.

nekitamo• 3 years ago

Have you tried SingleFile? It sounds like what you’re looking for:

https://github.com/gildas-lormeau/SingleFile

BlueTemplar• 3 years ago

I'm baffled that the support of single file, offline HTML is still so bad today :

https://www.russellbeattie.com/notes/posts/the-decades-long-...

(I'm suspecting because this goes against the wants of some of the biggest players who have the incentive of making us leave as many online footprints as possible ?)

Even here, Mozilla recommends converting to PDF for easier (?!?) human readability. Except PDF is a very bad format for digital documents, with no support for reflow and very bad support of multimedia. (PDF is perhaps good for archival of offline documents, even despite its other issues).

Dwedit2• 3 years ago

"Save Page WE" will capture a DOM snapshot to a single HTML file. The only problem is that Data URLs encoded using Base64 are highly bloated.

dralley• 3 years ago

Isn't that basically Pocket, the service that people complain about endlessly as "bloat"?

yeukhon• 3 years ago

Maybe it is just me, since I lived through the Firefox OS era as a past intern: this feels like a possible re-entrance of offering a Mozilla-built OS in the future. They said Internet was born to connect people - but building everything into a browser is not the most optimal way of adding all these fancy stuff. Firefox OS was basically a small linux kernel plus Gecko plus HTML5 for rendering. So much like iOS and iPadOS Mozilla could offer similar OS for devices/platform. I mean, for the past 5 years they have been invested in AR and VR. So I won’t be surprised if they eventually bet on another Firefox OS…

orbital-decay2• 3 years ago

Classic bookmarks have failed because mnemonic organization doesn't scale. This kind of interface does, and can replace it entirely if done right.

Thinking of it, something like this can be used for all your local files as well, acting as a better version of the old filesystem-as-a-database idea. Or for a specific knowledge base (think LLM-powered Zotero).

AureliusMA• 3 years ago

Something like Orbit would be perfect

https://withorbit.com/

wintogreen74• 3 years ago

Sounds like you just invented the modern version of Windows Longhorn

ElectricalUnion• 3 years ago

Wasn't this the idea behind "Networked Environment for Personal, Ontology-based Management of Unified Knowledge" (Nepomuk) Semantic Desktop?

Assuming that you can coerce the LLM to fill in the RDF correctly, and that we now have much more memory and faster storage, it might work.

danielovichdk• 3 years ago

My usage with browsing is not relevant for this. I don't want to "chat" with my browsin g history. I would simply love my browser would index my bookmarks on my OS so I could search the actual content of those bookmarks.

The feedback loop coming gained from chatgtp will I assume always be way better than my local gpt equivalent.

But often I bookmark pages where I know the information on there are important enough for me to come back to more than once.

So I have started crafting out a solution for this. It crawls your bookmarks on your local browser storage, downloads those pages and adds them to your search index on your OS.

That's been an itch for me for years.

groestl• 3 years ago

Small data sets suffer from bad recall in full text search. So a bit of smart fuzzyness added to the search by AI could improve the experience on locally indexed bookmarks quite well.

jval432• 3 years ago

Didn't Chrome do this at the very beginning, when it was initially released? I faintly remember that being a feature.

Personally I would already be content if my browsers didn't forget their history all the time, both Firefox and Safari history is way too short-lived.

ayewo2• 3 years ago

You were probably thinking of Google Desktop which could search almost anything on your machine.

https://en.wikipedia.org/wiki/Google_Desktop

overstay8930• 3 years ago

Isn't this just Safari? *using a modern chip

politelemon• 3 years ago

Did you mean to link to a forked repo?

https://memorycache.ai/developer-blog/2023/11/30/we-have-a-w...

links to https://github.com/misslivirose/Memory-Cache

but did you mean https://github.com/Mozilla-Ocho/Memory-Cache

no_time• 3 years ago

Good idea. Mozilla gets a lot of rightful hate for their mishandling of FF and their political preaching, but I believe they are still capable of developing tech that is both privacy preserving and user friendly at the same time.

I use the offline translator built into FF regularly and It's magic. I would've never thought something like that can run locally, without a server park worth of hardware thrown at it.

Here's hoping this experiment turns out the same way.

pixxel• 3 years ago

Well said; I agree wholeheartedly.

lofaszvanitt3• 3 years ago

What is happening at Firefox is quite strange. Like they are walking backwards.

smcleod3• 3 years ago

This seems like a sensible step in the right direction IMO, (optional) features such as local, privacy respecting LLMs will help to augment peoples online research, bookmarking, contextual search etc....

It's important that we have Firefox working on such experiments otherwise as Google adds more of their privacy invading features to chrome / chromium it will likely impact negatively on peoples desire to find alternative browsers.

lofaszvanitt3• 3 years ago

Yeah, but maybe, if you are constantly losing market share... maybe you should work on things that appeals to a wider audience. Except if you have a trump card and intend to use it as a deus ex machina to suddenly show people you are THE browser, the way forward.

ath3nd• 3 years ago

They might be onto something here.

Instead of doing lots of back-n-forth with the giants, enriching them with each prompt, you get a smaller local model that's much more respectful of your privacy.

That's an operating model I am willing to do some OSS contributions to, or even bankroll.

Gotta love the underdogs, even if admittedly, I am not a big Mozilla org fan.

altairprime• 3 years ago

It’s what Apple’s been doing for a few years, though it remains unclear how much of that is “AI”. So it makes sense that someone else would enter that niche.

visarga• 3 years ago

In the future their AIs are going to talk to our AIs. Because we need protection.

SpaceManNabs• 3 years ago

This seems completely overkill.

I don't even like having to clear my history and wtv regularly. I use incognito mode most times.

Now I have monitor what my local AI collects?

"through the lens of privacy" my ass, man.

Why would I ask my browser what the meaning of a life well lived is?

nektro• 3 years ago

you're better than this mozilla. hopping on the ai trend is disgusting given your alleged morals

tesdinger• 3 years ago

I wish they would fix basic features such as downloading pictures on Firefox for Android. Often long pressing the image on your screen opens a context menu that does not allow download, only following the link associated with the image.

huy77• 3 years ago

so this is how growth hacking look like, building a landing page for a imaginary product to test market-fit idea?

ChrisArchitect• 3 years ago

Could barely get a sense of what any of this meant from the shared link.

Went back a bit further/to the official site:

MemoryCache is an experimental development project to turn a local desktop environment into an on-device AI agent.

Okayy...

And this from November

Introducing Memory Cache

https://memorycache.ai/developer-blog/2023/11/06/introducing...

stainablesteel• 3 years ago

this is a pretty cool idea, i'd like to be able to choose which pages i want to cache

Ringz• 3 years ago

Doing this for a long time.

https://news.ycombinator.com/item?id=38421121

The FF solution is just more automated.

emsign• 3 years ago

Does it ingest the ads containing in the web pages as well? This would be a major concern, ads will pollute the model at least with unwanted information. At worst they'd be a security concern when used for indiscriminate or even targeted manipulation of the model. Advertisers do that to our brains, it's all they do. So why shouldn't they try that with LLMs scraping/being fed from the web?

MemoryCache: Augmenting local AI with browser data

Related Stories