HN user
FastMCP author here, surprised and excited to see this hit the front page! Certainly not an announcement, we’ve been around since the beginning, but we will be launching full generative apps support shortly so stay tuned.
Sure! There will not be any contention between their roadmaps, in fact FastMCP will continue to evolve somewhat independently of FastMCP Cloud. We'll be announcing our first external maintainer later this week.
Our goal is for FastMCP to be the most full-featured framework for building production-ready MCP servers. FastMCP Cloud is an opinionated way to spin up infrastructure, auth, observability, etc. automatically for those servers. Nonetheless, FastMCP 2.12 (probably releasing this week) will include a completely new auth layer that supports Google, GitHub, WorkOS, and Azure as integrated OAuth providers. It'll also include a new portable deployment configuration that will make it easy to spin up FastMCP servers -- and all their dependencies -- from a declarative JSON file. The objective is to make MCP accessible, whether you want to use our platform or roll your own!
FastMCP author here -- (maybe surprisingly) I agree with many of the observations here.
FastMCP exists because I found the original spec and SDK confusing and complicated. It continues to exist because it turns out there's great utility in curating an agent-native API, especially when so many great dev tools have adopted this client interface.
But the spec is still so young and at such risk of being co-opted by hype (positive and negative). I would invite everyone to participate constructively in improving it.
Lastly: this article is plainly AI generated, as `from mcp import tool` is completely hallucinated. Just some food for thought for the "AI should be able to figure out my complex REST API" crowd that seems well represented here.
One of the first features we added to FastMCP 2.0 [1] was server composition and proxying (plus some niceties around trying to reuse servers whenever possible) for exactly this reason, so that you can ideally run a single instance of your preferred servers without replicating the setup over every application. In some ways, MCP standardized how agents talk to APIs but introduced a new frontier of lawless distribution! This is something that has to improve.
Nice! Would love to collaborate
There was a tutorial at the Prefect Summit this week, video should be available soon, and lots more coming. Probably our most pressing request!
Thanks! Let us know how it goes!
Yes, this version uses OpenAI LLMs.
I bet it would work if you invert the control image!
The 100k context was originally released only via API, but I just noticed that it's now available in the Claude web UI.
Nice! Is there a story behind the name Frogmouth?
Unfortunately multiline inputs are still tricky but when Textual supports them, we’ll add them.
Anytime! Thanks for checking out our work!
I see Langchain has support for Azure chat models, and Marvin is built on Langchain so it may not be so difficult! Tracking issue here: https://github.com/PrefectHQ/marvin/issues/189
We’ve done some work on a full UI for Marvin! Some things work great in a terminal, others really need the flexibility of the web.
No quota here! If you like Marvin, you’ll probably like Prefect. Both are designed to be clean, Pythonic interfaces to a complex and hard-to-observe system (a data stack; an AI stack).
I think one of the key differences between Prefect and Dagster is that Prefect views orchestration as coordination, while Dagster views orchestration as reconciliation. The data stack is a complex system whose state is frequently mutated by forces outside our users’ control. Therefore, our product is focused on letting our users understand and react to those events, no matter where they come from. That could include everything from scheduling fully-orchestrated Prefect pipelines, to setting an SLA for database maintenance that Prefect doesn’t have anything else to do with. Reconciliation, in contrast, requires users to define a digital twin of their stack, in order to serve as ground truth and become the reconciliation target. Philosophically, we view Prefect as one piece of an ever-changing stack. We are focused on being as flexible as possible to fit into the stack, rather than the other way around.
Sure! We wrote a little bit about the origins in an announce post a couple weeks ago (https://news.ycombinator.com/item?id=35366838).
Marvin (https://www.github.com/prefecthq/marvin) powers our AI efforts at Prefect (https://www.github.com/prefecthq/prefect).
The first version of Marvin was an internal framework that powered our Slackbot. There are close to 30,000 members of our open-source community and we rely heavily on automation to deliver support. Then, as more of our customers started building AI stacks, we began to view Marvin as a platform to experiment with high-level UX for deploying AI. We have a few internal use cases, but it was the diversity of customer objectives that gave us confidence.
Historically, we've always focused on data engineering, but the more we worked with LLMs, the more we saw the same set of issues, basically driven by the need to integrate brittle, non-deterministic APIs that are heavily influenced by external state into well-structured traditional engineering and pipelines. We started using Marvin to codify the high-level patterns we were repeatedly deploying, including getting structured outputs from the LLM and building effective conversational agents for B2B use.
The lightbulb moment was when we designed AI functions, which have no source code and essentially use the LLM as a runtime. It's one of those ideas that feels too simple to actually work... but it actually works incredibly well. It was the first time we felt like we weren't building tools to use AI, but rather using AI to build our tools. We open-sourced with AI functions as the headline and the response has been amazing! Now we're focused on releasing the "core" of Marvin -- the bots, plugins, and knowledge handling -- with a similar focus on usability.
Hope that's what you were looking for!
Thanks! And I agree - as soon as Textual has multi-line inputs, we'll include them.
Exactly! Threads in Marvin are designed to support multiple bots and users. Two key user stories:
- multiple users in a Slack thread talking to the same bot. This is something we want to deliver soon, as Marvin powers our existing Slack bots
- one user addressing multiple bots, each of which is designed for a specific purpose (because bots do way better with reduced scope than when you have one bot try to do everything)
It does need an API token - it will ask you for it when you start if you haven’t stored one already.
We have a related issue open (https://github.com/PrefectHQ/marvin/issues/64) but haven't designed anything yet.
Check out this example from the docs to see how to take a URL as argument and then pass content to the LLM: https://www.askmarvin.ai/guide/concepts/ai_functions/#sugges...
(The previous example is also good)
A few things you could consider:
1. We have a utility for getting content out of HTML at marvin.utilities.strings.html_to_content. That would probably significantly compress it.
2. Chunk the HTML into batches that fit in context, send each over with an AI function that summarizes it (you could instruct the AI function to optimize the summary to help with title generation), then send all the resulting summaries to a title generator
3. We have a suite of HTML loader classes that will probably be ready for production in a couple releases (see https://github.com/PrefectHQ/marvin/blob/main/src/marvin/loa...) but you could try them out now (note: these use parts of Marvin beyond just AI functions, so I'm not recommending it as a drop-in right now). Our loader classes are (ideally) designed to do more than just chunk the input; depending on the nature of the input we do different preprocessing steps to help with insight.
4. Experiment and let us know what you learn - we can incorporate it into a loader class if its effective
Thanks!
Caching is highly requested! We have an issue open (https://github.com/PrefectHQ/marvin/issues/102) and expect to tackle it soon.
You can set temperature as a setting today (sorry we haven't documented all the settings yet) by setting the env var `MARVIN_OPENAI_MODEL_TEMPERATURE=0.2` or at runtime with `marvin.settings.openai_model_temperature=0.2`. Note the temperature is set when a bot / ai_fn is created, not when it's called, so you need to do this early.
Good question and actually a great illustration of how unexpectedly the "flagship" feature of a library can change!
At its core Marvin isn't just for AI functions, but a high-level library that makes it easy to interact with LLMs in a programmatic way. In fact the first version was written to make it easier for us to upload public and private knowledge into our customer service Slackbot. This happens through the `Bot` class, which is designed to help users / programs explore more complex problems. In particular, bots can use plugins to access proprietary knowledge.
Our loader classes are designed to get the knowledge into the bots. Why build "yet another LLM loader library?" We've had enough real-world use cases to know that just taking a document, chunking it, and throwing it in a vector store gives pretty bad results over large enough datasets (especially if the documents are relatively homogeneous). You have to preprocess documents in a particular way, and we wanted to take our learnings and codify them for future use.
So AI functions are definitely the on-ramp to the library, but the real power is in utilizing it to extract insight from data. Since AI functions are "just" bots under the hood, they can use plugins (pass `plugins=[...]` to the `@ai_fn` decorator) and will benefit from this as well. This is supported right now, but we are rapidly improving loader integration more broadly.
Honestly, it's hard! We try our best through a few strategies:
- At this time Marvin is only tested with GPT-3.5 and GPT-4 to reduce the surface area of LLM differences. You're welcome to try others, but the prompts are optimized for performance of those two models
- I expect that prompts optimized for one family of models will not automatically work with others, so as time allows we may end up with branching prompts based on your model choice. Marvin does a lot of work to manipulate the user prompt before sending it to the LLM so I think there's an excellent chance that we could deliver similar results for the same user input.
- Even with GPT-3.5 and GPT-4 we see a remarkable difference between them, with 4 needing far less complex prompts than 3.5 to get the same result. However, given both the cost and availability of GPT-4, we decided to make 3.5 our default to make sure everyone can use the library. Therefore we do our best to make sure prompts work with 3.5 and expect that that is a good bet of compatibility with 4.
Well, that falls in the category of "yes it would go fast but no, I'm not sure you'd want to trust it."
LLMs aren't great at answering questions that involve precise math. What you might do instead is ask Marvin (or ChatGPT, or your LLM of choice) to write the source code you need to compute the CI, and then execute that directly if you accept it.
However, AI functions (and all Marvin bots) can use plugins to solve more complex problems. Another option would be to write a function that computes CIs and make it available to the AI as a plugin.
Correct, it's the latter. The code is neither generated nor executed; everything is actually a "prediction" from the LLM. There's a little more detail in the concept docs (https://www.askmarvin.ai/guide/concepts/ai_functions/) and I'll take your comment as a suggestion we should discuss this aspect even more.
One cool thing buried in the advanced section is if you do write source code for your function, it's executed and the result is also sent to the LLM. This way you can write functions that e.g. take only a URL as argument, load the content from that site, and then summarize it with the LLM.
Edit: forgot to mention I opened an issue to explore a mode where the LLM DOES generate and send back source code, but this would have to be opt-in and VERY careful because most people would not be comfortable blindly executing that code. (https://github.com/PrefectHQ/marvin/issues/64)
Sure! This might be even lighter than what you're expecting.
I'm not sure how to get HN to format my code correctly, so here's a gist. It has two functions, one that generates a list of dicts with fake people data, and another that does sentiment analysis of tweets. Both are two lines of code. https://gist.github.com/jlowin/ae22fb7ac1788f066f809d2b8f573...
You can get more complex than this but hopefully this shows the idea: define your inputs, define your outputs, give it a descriptive docstring (as descriptive as you want!), and call it.
Hi! Prefect CEO here. We made Prefect freely available for exactly this reason - so you can use Prefect and its UI to ensure that your business's processes are running smoothly. Broadly speaking, your internal use won't violate our license at all (unless your SaaS is a workflow orchestration platform, in which case please check out our open jobs because you're the sort of person we'd love to talk to :) ). If you have any questions at all, always happy to help.