HN user
Seems to me that integrations will be the most important component of tools like this. As an engineer I get my context from video calls with customers and other engineers, slack messages, emails, docs online, using the product myself, etc. So an auto-engineer should do the same.
I've already merged my first Engine PR! Being able to review PRs like normal and it updates its work is very cool.
I believe we'll see two main types of AI/LLM libraries/frameworks emerge like has happened for ORMs (at least in Python). - The "Sqlalchemy" level: just bridge the gap between code and LLM - The "Django" level: additionally handle how data is indexed for RAG, standard patterns of connecting the pieces, possibly GUI, support for plugins.
It seems langchain and llamaindex are aiming to be the latter, and I'm building https://magentic.dev to be the former. I'd be interested to get your take on whether these abstractions would allow you stray from the narrow path while still being helpful!
I would love some feedback on my library magentic which aims to have the best features of both approaches! https://github.com/jackmpcollins/magentic
If you are using Python, check out the package I've been building, magentic https://github.com/jackmpcollins/magentic It supports structured outputs and streaming, and aims to avoid making unnecessary abstractions (but might require some more understanding of LLM patterns as a result).
Also recently released is pydantic-ai, which is also based around pydantic / structured outputs, though works at level of "agents". https://github.com/pydantic/pydantic-ai
I'm building magentic for use cases like this!
https://github.com/jackmpcollins/magentic
It's based on pydantic and aims to make writing LLM queries as easy/compact as possible by using type annotations, including for structured outputs and streaming. If you use it please reach out!
I'd be really interested to hear what abstractions you would find useful for RAG. I'm building magentic which is focused on structured outputs and streaming, but also enables RAG [0], though currently has no specific abstractions for it.
I've been building magentic [0] around similar ideas! Structured outputs and keeping control flow in python is definitely the way to go to keep things lightweight and easy to integrate. Having state management built in is cool! I'd be interested to hear what kinds of projects this has enabled you to build.
I'm working on a very abstraction light package for LLMs called magentic. The goal is to handle the main boilerplate of building with LLMs, but leave the abstractions up to the user. I'd be interested to hear your opinion on it vs others.
I'm building magentic, which aims to be a middle ground between the current "no framework" and "too much framework" options. It handles retries, logging/tracing, concurrency with asyncio, and other boilerplate, without mandating any prompts or particular patterns. Please check it out for the next comparison!
Pydantic also have support for parsing partial JSON. https://docs.pydantic.dev/latest/concepts/json/#partial-json...
from pydantic_core import from_json
partial_json_data = '["aa", "bb", "c'
result = from_json(partial_json_data, allow_partial=True)
print(result)
#> ['aa', 'bb']
That gif is really cool! I built a Python package magentic [0] which similarly parses the LLM streamed output and allows it to be used before it is finished being generated. There are plenty of use cases / prompts that can be refactored into a "generate list, then generate for each item" pattern to take advantage of this speedup from concurrent generation.
It's a great start but there's a little more work to do for full OpenAI API compatibility, namely streaming support and the tool_choice parameter. Making it fully compatible would allow it be swapped in directly to frameworks like langchain and magentic [1] that I am building).
Thanks for the explanation! Do you see auto-optimization as something that is useful for every use case or just some? And what determines when this is useful vs not?
Looks quite like magentic [0] that I've been building, though broader in scope? I'm (clearly) a huge advocate of pydantic, structured outputs, and keeping control flow in python code (rather than inside abstractions / "chains") as much as possible, so it's great to see those values here too! I'd be interested to hear what you consider in vs out of scope for mirascope and the long-term vision. Also would be cool for one of us to do a mirascope vs magentic comparison blog post.
Is AdalFlow also focused on automated prompt optimization or is it broader in scope? It looks like there are also some features around evaluation. I'd be really interested to see a comparison between AdalFlow, DSPy [0], LangChain [1] and magentic [2] (package I've created, narrower in scope).
[0] https://github.com/stanfordnlp/dspy
I've built a lightweight package that provides a standard interface to the LLM providers, as well as taking care of boilerplate around structured outputs, function calling, and opentelemetry/tracing. It's hopefully a good compromise between ease-of-use and complexity.
I haven't used LangGraph myself, but the latest magentic release is compatible with it if you'd like to check out the examples here https://github.com/jackmpcollins/magentic/issues/287
Please try out https://magentic.dev/ ! It is a light wrapper that is standard across LLM providers and handles the boilerplate code related to structured outputs and function calling. It doesn't include data/vector stores or integrations so it's a little lower-level than langchain, but for a lot of use cases it gives you the flexibility needed.
I completely agree, and built magentic [0] to cover the common needs (structured output, common abstraction across LLM providers, LLM-assisted retries) while leaving all the prompts up to the package user.
I'm building magentic https://github.com/jackmpcollins/magentic which has basically this syntax, though it queries the LLM to generate the answer rather than writing + running code.
from magentic import prompt
from pydantic import BaseModel
class Superhero(BaseModel):
name: str
age: int
power: str
enemies: list[str]
@prompt("Create a Superhero named {name}.")
def create_superhero(name: str) -> Superhero: ...
Does the dashboard/UI support traces? I would love a tool in which to view opentelemetry traces, that can neatly display full prompt and response for the spans that represent LLM queries. I'm planning to add opentelemetry instrumentation to magentic [1] and looking for a UI that is easy to run locally that makes it easy to see what an agent is doing (via OTEL traces). I have more of my thoughts on the github issue: https://github.com/jackmpcollins/magentic/issues/136
Update: I've added the ability to add chat messages using a new decorator `@chatprompt` in v0.7.0. See https://github.com/jackmpcollins/magentic/releases/tag/v0.7....
Ellipses is actually used in quite a few places. See the answers and comments on this stackoverflow post[0]. The usage most similar to what I have in the magentic examples is with the `@overload` decorator in the typing module[1].
With that said, you are free to put any code in the function body including `pass` or just a docstring or even `raise NotImplementedError` - it will not be executed. Using Ellipses satisfies VSCode/pyright type checking and seemed neatest to me for the examples and docs. I have some additional notes on this in the README[2].
[0] https://stackoverflow.com/q/772124/9995080
[1] https://docs.python.org/3/library/typing.html#typing.overloa...
Right now we just pass a single user prompt to the chat model. Setting the system prompt could also be done in the `@prompt` decorator. I've added a github issue to track https://github.com/jackmpcollins/magentic/issues/31
Yes, I'm working on allowing few-shot examples to be provided as part of defining the prompt-function, which should help in cases like this. Unfortunately from my testing just now it appears that OpenAI ignores examples added to the model config.
In the meantime, have a look at the ValidationError traceback which might highlight a specific field that is causing the issue. Some options to resolve the issue might be: the type for this field could be made more lenient (e.g. str); the `Annotated` type hint could be used to give the field a description to help correct the error [0]; the field could be removed. You could also try using gpt-4 by setting the env var MAGENTIC_OPENAI_MODEL [1].
If none of these help resolve it or it appears to be an issue with magentic itself please file a github issue with an example. Comments on how to improve error messages and debugging are also welcome! Thanks for trying it out.
[0] https://docs.pydantic.dev/latest/concepts/fields/#using-anno...
Thanks for sharing! LocalAI supports function calling[0] so this should work for most or all features of magentic - I'm interested to see if concurrent requests work. I will test this out.
At the moment only those that support the OpenAI Chat API, with function calling for the structured outputs. For example you can use LocalAI[0][1] to run models locally.
With magentic you could do chain-of-thought in two or more steps: one function that generates a string output containing the chain-of-thought reasoning and answer, and a second that takes that output and converts it to the final answer object. I agree though that this is not encouraged or made obvious by the framework.
The approach I'm encouraging with this is to write many functions to achieve your goal. So in the case of your email writing example you might have some of the following prompt-functions - write key bullet points for email about xyz -> list[str] - write email based on bullet points -> str - generate feedback for email to meet criteria abc -> str - update email based on feedback -> str - does email meet all criteria abc -> bool And between these you could have regular python code check things like blacklist/whitelist of keywords, length of paragraphs, and even add hardcoded strings to the feedback based on these checks.
Thanks! Currently magentic just uses OpenAI function-calling; it provides it a function schema that matches the structure of the output object. So it fails in the same ways as function-calling - struggles to match complex schemas, occasionally returns empty arrays, ...