Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs

https://allenai.github.io/lumos/

by milliondreams • 2 years ago

102 10 2 years ago

allenai.github.io

Looks promising approach to Agentic AI systems.

antupis• 2 years ago

I think that to get truly working agents we need huge working context windows like 1M - 100M tokens everything before that will be kinda hackish.

maCDzP• 2 years ago

I believe Gemini has a 1M context window.

CGamesPlay• 2 years ago

Random, off-topic observation: it seems like I see lots of open-source LLM prompts have prompts that aren't proofread. For example [0], "ounch", "he spend", general lack of punctuation, and so on. But the big model providers don't do this [1], so I'm curious how much impact proper grammar in prompts makes (and if it's a positive or negative impact).

[0] https://github.com/allenai/lumos/blob/main/data/incontext.py...

[1] https://twitter.com/AmandaAskell/status/1765207842993434880

Eisenstein• 2 years ago

It depends. LLMs are text completion bots. If you want it to chat with you, you prompt it with 'The following is a chat log between user and bot: User: Hi\n\nBot:' and it will keep writing as if it got to the end of an actual chat log and is completing what it thinks could be the next part of it, starting with Bot:. In this case it will take cues from the writing before, so any stylistic or misspelling issues could be imitated.

However, instruct finetunes (like GPT-4) will generally not do this. They are trained on a set of data with 'Instruction' and 'Ideal completion', and are filling the 'ideal completion' part, and thus any errors on the instruction should not be detrimental, as long as it can understand what has been written.

Of course this also depends on the 'unseen' parts of the prompting. An example of this is given above -- you don't write the initial part of the prompt as a user, nor do you add the start and end sequences. If you don't use the type of prompting that a model was trained on, you can get strange results.

lmeyerov• 2 years ago

A few bits we've learned from building louie over the last year+:

* Training on the internet means fault tolerance out-of-the-box

* Rule-of-thumb: Sub-gpt4 models are derpy, while gpt4/claude are reasonable. The dumber the model, the more it gets tripped up on basics and the more you need to engineer.

* In areas it's already confused, prompt errors can push it over the edge: When asking for JSON, will it output JSON strings, or Python literals?

* Even GPT4 grade will get tripped up on things like negations. So simpler grammar helps. As does emphasizing where to pay attention. REALLY.

* They're simple-minded. So asking GPT4 to do multiple things, especially over a long duration, gets weird. (Long context is a bit of a myth.)

* They're sensitive to persona. Instructing it to answer carefully as a professional would helps improve quality. The opposite is less clear - the base model is just ok, and typos seem to be similarly ok.

* Agents can easily 2-100X quality, so after basics in the prompt, better to focus on the code outside the prompt.

zzzzzzzzzz10• 2 years ago

How can I run this locally does anyone know? Can't find any instructions.

milliondreams2• 2 years ago

Guess you are looking for this - https://github.com/allenai/lumos/blob/main/README.md

jondwillis• 2 years ago

All of these papers and projects need less jargon, and more examples and proof of performance.

dvt• 2 years ago

The paper[1] seems promising, is there a fine-tuned model available? Or will we have to fine tune Llama-7b or Mistral-7b ourselves?

[1] https://arxiv.org/pdf/2311.05657.pdf

Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs

Related Stories