Memary: Open-Source Longterm Memory for Autonomous Agents

by james_chu • 2 years ago

216 66 2 years ago

CuriouslyC2• 2 years ago

While I'm 100% on board with RAG using associative memory, I'm not sure you need Neo4J. Associative recall is generally going to be one level deep, and you're doing a top K cut so even if it wasn't the second order associations are probably not going to make the relevance cut. This could be done relationally, and then if you're using pg_vector you could retrieve all your rag contents in one query.

verdverm5• 2 years ago

My initial thought was "building the knowledge graph is what LLMs and the embedding process does implicitly", why the need for a graphdb like Neo4j?

throwaway11460• 2 years ago

So your solution would be to fine tune the LLM with new knowledge? How do you make sure it preserves all facts and connections/relations and how can you verify during runtime it actually did, and didn't introduce false memories/connections in the process?

dbish7• 2 years ago

I dont know if you need a graphdb in particular but there are likely explicit relationships or entities to resolve to eachother that you’d want to add that aren’t known by a general model about your use case. For example if you are personalizing an assistant maybe you need to represent that “John” in the contacts app is the same as “Jdubs” in Instagram and is this person’s husband.

snorkel• 2 years ago

LLMs have a limited context size, i.e. the chat bot can only recall so much of the conversation. This project is building a knowledge graph of the entire conversation(s), then using that knowledge graph as a RAG database.

kingJulio6• 2 years ago

Exactly! With memary only relevant information is passed into the finite context window.

gaogao3• 2 years ago

I think there's a lot of cases where you don't want to just RAG it. If you're going for tool assisted, it's pretty neat to have agent write out queries for what it needs against the knowledge graph. There was an article recently about how LLMs are bad at inferring B is A from A is B. You can also do more precise math against it, which is useful for questions even people need to reason out.

I need to dig into what they're doing here more with their approach, but I think using an LLM for both producing and consuming a knowledge graph is pretty nifty, which I wrote up about a year ago here, https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... .

I will say figuring out how to actually add that conversation properly into a large knowledge graph is a bit tricky. ML does seem slightly better at producing an ontology than humans though (look how many times we've had to revise scientific names for creatures or book ordering)

dbish7• 2 years ago

Yes, but this doesn’t seem to be an actual knowledge graph which is part of the issue imho. If you look at the Microsoft knowledge graph paper linked in the repo it looks like they build out a real entity-relationship based knowledge graph rather then storing responses and surface form text directly.

jjfoooo6• 2 years ago

I think it's relatively unlikely that having an agent write graph queries will outperform vector search against graph information outputted into text and then transformed into vectors.

The related issue that I think is being conflated in this thread is that even if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db

abrichr4• 2 years ago

Very interesting, thank you for making this available!

At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt) we are looking into using pm4py (https://github.com/pm4py) to extract a process graph from a recording of user actions.

I will look into this more closely. In the meantime, could the authors share their perspective on whether Memary could be useful here?

oulipo2• 2 years ago

Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/

Perhaps it could make sense to add this to your effort?

abrichr4• 2 years ago

Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood — can you please clarify your suggestion?

dbish7• 2 years ago

What’s the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but haven’t found value in adding structure beyond representing them semantically in a list with the context around them of what happened.

abrichr4• 2 years ago

The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task.

There's a lot of literature around process mining, e.g.:

- https://en.wikipedia.org/wiki/Process_mining

- https://www.sciencedirect.com/science/article/pii/S266596382...

- https://arxiv.org/abs/2404.06035

falcor84• 2 years ago

This is a really cool project, but is it just me that feels slightly uncomfortable with its name sounding so similar to "mammary"?

Mtinie• 2 years ago

Discomfort noted, but I think it can work in either case. Pronounced your way, it’s the proverbial teat of knowledge for LLMs.

jprete• 2 years ago

Argh, that only makes it worse.

wholinator2• 2 years ago

Yeah, they could've gone with "Memury" which is pronounced much closer (at least for me) to the original "Memory".

kingJulio6• 2 years ago

The a is for agents :)

altilunium• 2 years ago

Sounds promising. Can this system be integrated with the Wikidata knowledge graph instead?

kingJulio6• 2 years ago

Yes! You can easily swap knowledge graphs under the same agent. Would love to see this happen!

TrueDuality4• 2 years ago

This seems like its overloading the term knowledge graph from its origins. Rather than having information and facts encoded into the graph, this appears to be a sort of similarity search over complete responses. It's blog style "related content" links to documents rather than encoded facts.

Searching through their sources, it looks like the problem came from Neo4j's blog post misclassifying "knowledge augmentation" from a Microsoft research paper with "knowledge graph" (because of course they had to add "graph" to the title).

This approach is fine, and probably useful but its not a knowledge graph in the sense that its structure isn't encoding anything about why or how different entities are actually related. A concrete example in a knowledge graph you might have an entity "Joe" and a separate entity "Paris". Joe is currently located in Paris so would have a typed edge between the two entities of something like "LocatedAt".

I didn't dive into the code but what I inferred from the description and referenced literature, it is instead storing complete responses as "entities" and simply doing RAG style similarity searches to other nodes. It's a graph structured search index for sure but not a knowledge graph by the standard definitions.

dbish7• 2 years ago

Exactly. Glad to see this. I do think knowledge graphs are important to AI assistants and agents though and someone needs to build a knowledge graph solution for that space.

The idea of actual entities and relationships defined like triples with some schema and appropriately resolved and linked can be useful for querying and building up the right context. It may even be time to start bringing back some ideas from the schema.org back the day to standardize across agents/assistants what entities and actions are represented in data fed to them.

gaogao3• 2 years ago

Yeah, one of the specific things I'd love to do is collaboratively bulking up WikiData more. It's missing a ton of low hanging fruit that people using an ML augmented tool could really make some good progress on, similar to ML assisted OpenStreetMapping work

TrueDuality4• 2 years ago

Yeah precisely. Knowledge graphs are simple to think about but as soon as you look into them you realize all the complexity is in the creation of a meaningful ontology and loading data into that ontology. I actually think LLMs can be massively useful for building up the ontology but probably not in the creation of the ontology itself (far too ambiguous and large/conceptual task for them right now).

esafak3• 2 years ago

Are graph databases really relevant during retrieval? Does anyone use them to augment a vector store as a candidate source?

theolivenbaum• 2 years ago

We're using in a large engineering use-case, 100s of millions of objects - and it almost doubled the nDCG@10 score vs pure vector search

Der_Einzige2• 2 years ago

If this paper overloaded the term, than I did the same in my recent EMNLP paper where I used th term “Semantic Knowledge Graph” to refer to what I think you’re talking about.

https://aclanthology.org/2023.newsum-1.10/

Automatically created knowledge graphs using embeddings is a massively powerful technique and it should start to be exploited.

You can blame peer review for letting the definition of knowledge graph get watered down.

Also note that my coauthor. David, is the author of the first and still the best open source package for creating or working with semantic graphs.

https://github.com/neuml/txtai/blob/master/examples/38_Intro...

TrueDuality4• 2 years ago

"Semantic Knowledge Graph" is even worse! The term is intended for the design of semantic networks with edges restricted to a limited set of relations. A knowledge graph is already about semantics!

Gotta say txtai seems like a useful tool to throw in my toolbox!

CyberDildonics• 2 years ago

How many times are people going to reinvent, rename and resell a database?

392• 2 years ago

Log4j was so unbelievably slow to load data, bloated, and hard to get working on my corporate managed box that I wasn't too sad when it turned out unable to handle my workload. Then the security team asked me why it was phoning home every 30 seconds. Ugh.

I have since found Kuzu Db, which looks foundationally miles ahead. Plus no jvm. But have not yet given it a shot for rough edges. At the time, it was easier just to stay in plain application code.

Hopefully the workload intended by this tool won't notice the bloat. But it would be nice to be able to dump huge loads of data into this knowledge graph as well, and let the GPT generate queries against it.

ec109685• 2 years ago

These new systems would do well to have a compelling “wow, this solves a hard problem that can’t be solved in another straightforward way”.

The current YouTube video has a query about the Dallas Mavericks and it’s not clear how it’s using any of its memory or special machinery to answer the query: https://www.youtube.com/watch?v=GnUU3_xK6bg

kingJulio6• 2 years ago

If you search about the Mavericks again (not included in the video) the agent will query the knowledge graph for results from prior executions.

anoy8888• 2 years ago

How does it compare with zep ai ? Anyone knows ?

kingJulio6• 2 years ago

It's open source :)

BirbSingularity• 2 years ago

I hate when I find a cool AI project and I open the github to read the setup instructions and see "insert OpenAI API key." Nothing will make me loose interest faster.

throwup238• 2 years ago

Most projects also give you the option of providing an base url for the API so that people can use Azure's endpoints. You can use that config option with LiteLLM or a similar proxy tool to provide an OpenAI compatible interface for other models, whether that's a competitor like Claude or a local model like Llama or Mistral.

muratsu• 2 years ago

Is the expectation for the lib (or project) to work with various vendors or you expect to just pay for tokens

kingJulio6• 2 years ago

You can easily incorporate llama 3 or other OS models

api_or_ipa• 2 years ago

Unconstructive comment. OpenAI is the golden standard for an llm: if you cared to dig deeper you’d realize that that you really could incorporate another llm with little effort.

JabavuAdams• 2 years ago

Looks cool. This is similar to what I'm doing for long-term memory in AISH, but packaged up nicely. Others have pointed out that they're somewhat abusing the term KG. But ... you could imagine other processes poring over the "raw" text chunks and building up a true KG from that.

Memary: Open-Source Longterm Memory for Autonomous Agents

Related Stories