_andrei_

it's langchain, what did you expect

claude invent me revolutionary text editing method for agents and write paper, must make me big money i patent

"artificial intelligence" chuckled

ha, nice - had the same need, i leveraged fumadocs for the ui part https://github.com/3rd/mdreader

What's the purpose of this? There is no replacement for MCP. We need a protocol for calling tools that works with structured outputs, this is what we have.

you're a slop maker

Almost all the points are not about what DSPy is mainly supposed to offer. What's supposedly great at is automatic optimization, for everything else... who the hell puts Python in production just to make some API calls? There are "frameworks" available in all the better languages, but the constructs behind are not that complicated. And why does DSPy even try to compete with LangChain/Graph/crap?

yes, it's an agent

All vendors will have to implement test time code execution, solution exploration, etc. as it's a low hanging fruit with huge gains, so I see it as a great hire. Love Bun, happy for you guys!

Hi Rob, thanks for your thoughts! We're not planning on expanding to watching expiring domains yet, we're focused more on finding domains that were never registered. But what you're interested in can be done by watching all the registered domains that you're interested in and either catching them when they expire (ex. https://www.expireddomains.net) or by registering a backorder (higher success rate but you have to pay, ex. https://www.gname.com/backorder)

what are we gonna dooo?

https://www.reddit.com/r/cscareerquestions/comments/1ojbebq/...

pydantic is a _validation_ library, it does not do any kind of constraints by itself

we need to recognize that the felt necessity of sophisticated type systems is a signal that we’re using the wrong architectural foundations

I'd be forever ashamed if I wrote that.

this is coming to Claude Code

So my argument is that removing the context that is entirely irrelevant for the agent improves performance dramatically.

100% agree on building the optimal context, just have not seen parallel agents do better at sequential tasks. the documentation agent may have better initial context about how to write documentation, but it doesn't have the context of the changes, apart from what is passed to it / can explore. if we don't spawn a new session, and instead throw it a /document command - that would still get all the guidelines and rules for writing documentation, and it should have the same weight since it's at the bottom of the context.

for me the highest 'model jump level' performance booster is externalizing context and controlling the process - having claude initialize a plan file with a pre-defined template that makes it use that as its to do list and documentation place, and getting it to use that as its primary working area

If you don't discuss hooks and subagents, i'm not sure what you're doing right now.

Agents have their own context and can be useful for tasks that can be parallelized, which is a minority of tasks. How are they critical to better performance for you?

ah, deep agents = agents with planning + agents as tools => so regular agents.

i hate how LangChain has always tried to make things that are simple seem very complicated, and all the unnecessary new terminology and concepts they've pushed, but whatever sells LangSmith.

very cherry picked

Funny of this to come out from LangChain.

go install github.com/sinclairtarget/git-who@latest

Do they have the solution (I'd assume Supermaven which they recently acquired does) and are they just giving out an interesting challenge to readers? Or... what is this?

Clinic.js was promising but it's not maintained anymore, would be great if we had a modern alternative: https://clinicjs.org/

That's a prompting issue though.

Hey, yes it draws a lot of inspiration from mitata, but it's much tinier and with less features. I remember I was getting some less accurate measurements in browsers when I tried it (~2 years ago) and I decided to build my own and got the results I expected.

Mitata has added a ton of things since then and it's very mature, and it's built by evan so it's definitely the right choice over benchmate. I used benchmate because I can easily extend it to support the upcoming features in BenchJS.

BenchJS is for easily building shareable benchmarks, and I want to evolve it into a nice code analysis tool. It uses Babel and esbuild to transform and bundle the code before it gets evaluated, and I want to use this to extract more useful insights like the number of operations of a given type (ex. array methods), code path analysis (coverage-like), inline indicators of how much % time a line took, etc.

Beware that you are spreading malware through https://git.makr.io/

Pretty underwhelming, I've been using on Kagi's assistant [0] for the past few months and it's much better. I can `!chat what i want to search for` in my address bar any time, and Kagi will do the search and then open a chat with the LLM of my choice (3.5 Sonnet) and the results in context. It can also do further searches.

[0] https://kagi.com/assistant

made one as well with interactive selection and token counting https://github.com/3rd/promptpack

Thank you for the reply and the video. I like the code + flow UI combo.

It might be that the new Structured Outputs feature is subpar when compared with "structured output through function calling", I'll try to compare them.

`/api` contains the openai/anthropic clients builder, using llm-api (on npm) as a wrapper. There are no external tools, the LLM is just forced to call a shim "returnResult" function to make it fill each field according to the schema.

You're comparing apples to oranges - structured output (a capability) with structured output + CoT (a technique), saying that structured output isn't good for reasoning. Well, it's not supposed to "reason", and you didn't apply CoT to it!

I didn't like it overall and I think it can confuse people and it shouldn't mention best practices:

1. To ensure a reasonable amount of variation, its temperature was set to 1.0.

Why would you use any other temperature than 0 when you are asserting the correctness of the data extraction and the "reasoning" of the LLM? You don't want variation.

2. true_answer = (5029) + (1.7509)

Why are you using the LLM to do math? If the data was extracted correctly (with structured output or function calling) let it write the formula and evaluate it.

3. The new Structured Outputs in the API feature from OpenAI is a significant advancement, but it's not a silver bullet. / there's more to this story than meets the eye

The new API is just a nicer built-in way to extract structured data. Previously (still valid) you had to use function calling and pass it a "returnResult" function that had its payload typed to your expected schema. This is one of the most powerful and effective tools we have to work with LLMs, if used for what it's supposed to do, we shouldn't avoid this because it doesn't "reason" as well.

John Doe is a freelance software engineer. He charges a base rate of $50 per hour for the first 29 hours of work each week. For any additional hours, he charges 1.7 times his base hourly rate. This week, John worked on a project for 38 hours.

And the sample scenario is something I wouldn't use LLMs for. Nevertheless, CoT can still be applied with structured output, I'd like to see your structured-output-reasoning-cot to figure out why it didn't work.

Structured outputs allow you to extend the prompts by providing descriptions for the fields you expect, making it much more effective than other solutions, and you can also implement features like self-healing loops (fx. to get rid of the 1% chance of gpt-4o to not reply with data following the schema), etc. The paper's authors used the plain "JSON mode" that is useless, glad to see you did it better.

---

Anyway, here's GPT-4o with function calling (not even structured output) solving the issue correctly every time: https://gist.ro/gpt-reason.mp4

As you can see it's super consistent with GPT-4o and temp 0, with a silly simple prompt. If someone worked on the prompts / split it into a "multi-step pipeline" (come on, is this what we call fn(fn(x)) now?) they would achieve the same result with 4o-mini.