rchaves

Hello HN!

tl;dr: We built Scenario, an open-source testing library for AI agents. It simulates real conversations with your agent, its code-driven, and lets you assert anything mid-dialogue. Repo: https://github.com/langwatch/scenario Docs: https://scenario.langwatch.ai/

I'm Rogerio, founder of LangWatch, I've been helping many customers building LLM applications in this past two years and worked with Alex on this.

Most of the efforts for LLM quality so far were about evaluations, single-turn, there was nothing actually good to test agents, it all felt forced, but we believe we cracked it now, we have built an agent testing library that test your agent by simulating a user and playing a conversation back and forth with it.

One of the key challenges there was that we had to make it compatible with all the 273+ AI frameworks (and counting) there are. Luckliy AG-UI protocol popped up recently, standardizing agents frameworks and UI interactions, this is perfect, because at the end of the day, we want our user simulator to "see" just the same that the user sees.

So we made Scenario in a way that is really easy to connect to any agent no matter the tech stack, from a simple string <-> string connection, to openai standard messages format, to AG-UI.

The other key challenge was to balance testing the open-endedness of agents vs having reliable cases you want to test, so we worked a lot on thinking through the autopilot simulation vs the fully scripted one, and here again, the goal was complete interoperability. At the end of the day, the design we achieved was simply having lambdas, that you can call at any point of the test, so it's just code, where you can connect any other evaluation or assertion tool you want, we are not restrictive.

Check out the repo and the docs, we would love to get some feedback in here!

Repo: https://github.com/langwatch/scenario Docs: https://scenario.langwatch.ai/

well I think hype is not bad per se, I'd do it even if not trying to make a buck, it's okay (up to a point) to hype up something so that eventually it finds a problem where it fits well, but yeah, I'm still waiting on this one

same here, but I would even avoid "strong arguments" because that's what we all have been doing so far

what I want is real use cases, show me real-world production examples from established companies where multi-agent collaboration helped them better than a simple agent + tools and deterministic workflows

is this multi-agent collaboration though, or is it just a workflow? All examples you listed seem to have pretty deterministic control flows (write then validade, context exceeded, after each response, etc)

when I think of multi-agent collaboration I think of also the control flow and handover to be defined by the agents themselves, this is the thing I have yet to see examples of in production, and the premise that I also don't buy yet

Nah it's just a marketing problem, "GPT" and "ChatGPT" names is the biggest asset OpenAI has, people have expectations so high for GPT-5 that they cannot burn this name unless it's something truly majestic, bordering AGI at the very least. Until they are confident enough that people will be blown off by it, it's better to continue building up the hype

Erm, he wrote the article with “you” to invoke the feeling of the reader thinking about their own use case, which I did

Different because I ran without good practices before, got more and more messed up over time, grinded to a halt, and it was all terrible. Then worked for 10 years, doing TDD for most of it, as well as pairing and other good practices, everything was better, started condemning people that don’t do it similar to uncle bob. Kept moving forward, entered an environment where everything was “wrong” but still somehow worked very well. Now I’m mature enough that I can have both: skip “good practices” and still not break things

Finally, yes of course the types fucking matter

Yes exactly, from the experience he had as Facebook massively scaling up, while all good practices were thrown down the window (except for foundation and critical parts) and extreme go horse php being written as fast as possible, until it was too big and a migration was needed

I had the exact same experience at Booking, but with terrible Perl instead of php, still, built massively successful business until one day the movement to better practices becomes inevitable

The opposite of what uncle bob says

I’ve seen people spending 10 minutes to test things by hand, would have taken them less to write and run a test, specially with AI now

When writing test actually makes it faster to code, THEN it’s worth it. You can even throw the tests away later, doesn’t matter

Not true. Kent Back’s 3X is a much better take, test and good practices for what is high risk and hard to change, move fast for most of it on the rest to try to find that black swan as soon as possible.

Yes, I do feel this time is different and that I am a top-notch coder (I feel more comfortable sounding like a jerk when on hackernews), 1 year later into my startup, codebase is big, but I’m actually coding faster than ever, as foundation code is more and more complete.

One of the huge reasons for it beyond the right architecture is type safety. Someone that was well seasoned in strongly-typed FP but is now pragmatic can move incredibly fast with enormous safety by just adding the most cost-benefit type strictness, and being flexible on where it doesn’t pay off.

yeah I guess base models without built-it CoT are not going away, exactly because you might want to tune it yourself. If DSPy (or similar) evolves to allow the same or similar than OpenAI did with o1, that will be quite powerful, but we still need the big foundational models powering it all

on the other hand, if cementing techniques in the models becomes a trend, we might see various models around with each technique for us to pick and choose beyond CoT without need for us to guide the model ourselves, then what's left for us to optimize is the prompts on what we want, and the routing the combination of those in a nice pipeline

still the principle of DSPy stays the same, have a dataset to evaluate, let the machine trial an error prompts, hyperparameters and so on, just switch around different techniques (possibly automating that too), and get measurable, optimizable results

Nope, the trains are not often late, this is just in Germany

If you look closely it actually does give multiple instructions per screenshot! However it cannot get too far, because the screen changes under it. For example when it starts typing a tweet, the tweet box expands and the send button moves, so it tries to click it but it's not longer there, it needs to take another screenshot to see because it's kinda executing those steps "in the dark"

we could try to patch an "interpolation" kinda of thing for change, but also, I'm curious to see if the multi-modal models that are coming out supporting video would be able to actually just "watch the video" in real time, this would be the ultimate solution

indeed! Ideally I want it to have very real time human-machine feedback, so you can interrupt it in the middle, point at things, then ask new things, and so on, kinda like if there is someone else pairing with you, and you are telling them to do stuff and course correcting

need to figure out the right UX to do that, and I think the multi-modal models also need to get a bit faster

thanks! I took my inspiration from Vim browser plugin (https://chromewebstore.google.com/detail/vimium/dbepggeogbai...), they have a shortcut F that allows you to choose any element on the website to navigate from

thanks vim!

yes actually, but I only saw it after I've implemented it, I had actually searched for something like that before but I guess Google is worse and worse those days

however, I tried self-operating-computer, and it could not find the right x,y positions on the screen executes the task as effectively

Hey, I was working on something to allow GPT-V to actually do stuff on the screen, click around and type, I tested on my Mac and it’s working pretty well, do you think it would be cool to integrate? https://github.com/rogeriochaves/driver

great naming innit

Hey there everyone, now that AI can "see" very well with GPT-V, I was wondering if it can interact with a computer like we do, just by looking at it. Well, one of the shortcommings of GPT-V is that it cannot really pinpoint the x,y coordinates of something in the screen very well, but I solved it by combining it with simple OCR, and annotating for GPT-V to tell where it wants to click

Turns out with very few lines of code the results are already impressive, GPT-V can really control my computer super well and I can as it to do whatever tasks by itself, it clicks around, type stuff and press buttons to navigate

Would love to hear your thoughts on it!

Yeah I never know where memory goes exactly in langchain, it's not exactly clear all the time. But sure, the main insight I remember is this, take a look at their MULTI_PROMPT_ROUTER_TEMPLATE: https://github.com/hwchase17/langchain/blob/560c4dfc98287da1...

It's a lot of instructions for an LLM, they seem to forget an LLM is an auto-completion machine, and which data it is trained on. Using <<>> for sections is not a normal thing, it's not markdown, which probably the thing read way more often on the internet, instead of open json comments, why not type signatures, instead of so many rules, why not give it examples? It is an autocomplete machine!

They are relying too much on the LLM being smart because they probably only test stuff in GPT-4 and 3.5, but with GPT4All models this prompt was not working at all, so I had to rewrite it, for simple routing, we don't even need json, carying the `next_inputs` here is weird if you don't need it.

So this is my version of it: https://gist.github.com/rogeriochaves/b67676977eebb1936b9b5c...

It's so basic it's dumb, yet it is more powerful, as it does not rely on GPT-4 level intelligence, it's just what I needed

hmmm, just had a chat with GPT-4, it didn't seem convinced that ETLs would do well the same things that LiteChain is trying to achieve: https://chat.openai.com/share/88961bd1-8250-45f0-b814-0680ba...

I'd be happy to see some more examples of LLM application building on ETLs like the video you shared though

this inspired me on writing a new section in my project "Prompts on the outside" (https://github.com/rogeriochaves/litechain#prompts-on-the-ou...)

100% this! What is worse is that LangChain hides their prompts away, I had to read the source code and mess with private variables of nested classes just to change a single prompt from something like RetrievalQA, and not only that, the default prompt they use is actually bad, they are lucky things work because GPT-3.5 and GPT-4 are damn smart machines, with any other open LLM, things break. I was hoping for good defaults, but they are not, the prompt I wrote over 6 months ago little after the launch of ChatGPT to do some of the same things work much better.

Would you have anything you can share with us about those "several features using highly sophisticated LLM chains that do all manner of reasoning", I'm really curious about the challenges, the process and insights there

Right now Arc browser

Gitless folks did some actual user research years ago, getting feedbacks on various tasks difficulty, and they have a nice theory of api misfit, I really recommend the read for more structured speculation of why git is hard: https://spderosso.github.io/oopsla16.pdf

thank you very much for the comments jaggs, it's really encouraging and help me keep going! Indeed, there is a lot to be done, gamification is a great idea, at this point I haven't even setup email sending yet, I'm trying to balance out chasing after NightCafe and others vs differentiation and innovation

thanks again!

added email sign up now :D

done! Took a while but I added email sign up now :)

yeah... the first version I built actually had no sign in, but users would just try it once and go away, no way to save their images or publish it, but also, being totally open make it harder to throttle or block bots

I really wanted to be able to have a totally open and maybe get viral like dalle mini, but I don't have enough money to pay for lots of GPUs

thanks! yeah, I didn't want to deal with all account management shaneningans yet (I'm planning to add sign in with apple next if that helps anyhow)

One point I struggle with is when to write a decision record. There are many decisions that go through without us even realizing it was a decision, sometimes it was just a reply to a comment, or something said on a meeting, which you put on the code and boom, a decision, usually logged only on the “why” of a commit message

What would be the rule of thumb, which decision size is big enough to worth a DR, what triggers “I have to write a DR for this”?