faxmeyourcode

I've definitely noticed 5.6 sol being extremely trigger happy in ways other models, even 5.5, we're not. I would definitely categorize a few small incidents at work where it performed "actions a reasonable user would likely not anticipate and strongly object to." Just my anecdotal experience.

For example discussing driver upgrade and subsequent password rotation and it didn't stop and ask me if I wanted to restart the service or install the driver or anything, it immediately took action. It feels like a side effect of pushing more "agency."

I feel like another comparison worth looking at is purely cost.

Capability per dollar is something I care about:

    Opus API    $5/$25
    Sonnet API  $5/$15
    Haiku API   $1/$5

    GLM 5.2 API $1.4/$4.4

A town I grew up in went absolutely crazy with roundabouts over the last 10 years or so. They built them everywhere. Most are pretty straightforward and they simplify traffic a ton but there were some bad design choices made.

Take this similarly confusing double-roundabout for example https://maps.app.goo.gl/VTSrSWsBGnsYyzKU6

There are accidents here almost every week and when an out of towner comes off the interstate to get some gas at Sam's Club (where I used to work) I had to try and explain to them how to get back onto the interstate...

I might try to clone this project idea, it was fun to play!

They also proposed a peanut style roundabout with a *train track going through it* that thankfully has not been built yet https://media.conwayarkansas.gov/media/documents/project/85/...

I work a lot with databases, but only a couple times have I really messed with time series stuff. I've enjoyed reading your list of tricks and it's gotten me interested in playing around with writing my own toy database to just experiment with some of these ideas.

Somebody else in the thread brought up the benefit of snapshotting a database at a point in time stores not only the state of execution but also the code, etc. That is a unique benefit I'd be interested in exploring over storing your orchestration outside of the database.

Not trying to dismiss the project - it looks like a lot of hard work has gone in and somebody has a use for it. I just come from an airflow style external orchestrator frame of mind that manages durability state in postgres but keeps the control flow out. Sorry if I came off as a bit snarky

I aggree - I'm not understanding the value of the project either if you look at the example here https://github.com/microsoft/pg_durable/blob/main/examples/i...

It's an interesting technical achievement I guess, but it's very bizarre to try and read this

    SELECT df.start(
        @> (
            ($$SELECT ... FROM demo.invoices WHERE status = 'pending'$$ |=> 'inv')
            ~> df.if_rows('inv',
                $$UPDATE ... SET status = 'processing'$$
                ~> (df.http(...) |=> 'resp')
                ~> df.if($$SELECT $r.ok$$,
                    -- classify, branch, wait for signal ...
                ),
                df.sleep(5)
            )
        ),
        'invoice-approval-pipeline'
    );

This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now.

Why would I want to store my control flow in the database and not in code? It feels strange.

Not trying to dismiss the project, I'm just not getting it yet I think.

I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true

    model                 total_claims  hedged_count  hedged_pct
    claude-opus-4-7       1000          451           45.1
    sonar-pro             1000          391           39.1
    gpt-5.4               1000          277           27.7
    gemini-3-retrieval    1000          129           12.9
    gemini-3-pro          1000          60            6.0

https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...

An HP 50g was my calculator of choice, and the whole RPN style really rubbed off on me. Plus it had more advanced symbolic algebra capabilities than a ti83 equivalent. I enjoyed learning common lisp, scheme, racket, etc through high school and college and still am fond of them today because of this calculator.

How does it compare to mythos?

Wow, I didn't realize some RFID could reach 15 feet out - that's good to know. I naively thought you essentially had to be touching the surface of the tag.

If you treat LLMs as generic transformers, you can fine tune with a ton of examples of input output pairs. For messy input data with lots of examples already built, this is ideal.

At my day job we have experimented with fine tuned transformers for our receipt processing workflow. We take images of receipts, run them through OCR (this step might not even be necessary, but we do it at scale already anyways), and then take the OCR output text blobs and "transform" them into structured receipts with retailer, details like zip code, transaction timestamps, line items, sales taxes, sales, etc.

I trained a small LLM (mistral-7b) via SFT with 1000 (maybe 10,000? I don't remember) examples from receipts in our database from 2019. When I tested the model on receipts from 2020 it hit something like 98% accuracy.

The key that made this work so well is that we had a ton of data (potentially billions of example input/output pairs) and we could easily evaluate the correctness by unpacking the json output and comparing with our source tables.

Note that this isn't running in production, it was an experiment. There are edge cases I didn't consider, and there's a lot more to it in terms of accurately evaling, when to re-train, dealing with net new receipt types, retailers, new languages (we're doing global expansion RN so it's top of mind), general diversity of edge cases in your training data, etc.

Especially for super constrained applications. I don't care if the language model that I use for my extremely specific business domain can solve PhD math or remember the works of Shakespeare. I'd trade all of that for pure task specific accuracy.

Labeling or categorization tasks like this are the bread and butter of small fine tuned models. Especially if you need outputs in a specific json format or whatever.

I did an experiment where I did very simple SFT on Mistral 7b and it was extremely good at converting receipt images into structured json outputs and I only used 1,000 examples. The difficulty is trying to get a diverse enough set of examples, evaling, etc.

If you have great data with simple input output pairs, you should really give it a shot.

Filip also told me that he asked Claude to continue on the even case after the odd case had been resolved. “But there after a while it seemed to get stuck. In the end, it was not even able to write and run explore programs correctly anymore, very weird. So I stopped the search.”

Interesting snippet towards the end. I wonder if they were using claude.ai or claude code. Sounds like they ran out of context and entered the "dumb zone."

Yea, I agree. The dataset is < 100MB... so duckdb can very easily handle this on an old macbook air.

https://duckdb.org/2025/05/19/the-lost-decade-of-small-data

Neel, this is really cool. How long have you been working on this, and where did you guys get inspiration from? Did you work on vlms earlier or something like that? Just curious.

Also, thanks for choosing a technical blog post for presenting this information.

I love the idea behind MyVisualRoutine as a father with a disabled kiddo, thanks for sharing.

The app is beautiful - much better than I could build - what tech is it using if you don't mind me asking? Is it flutter, react native, something else? Just want to get better at mobile dev.

Love this. An example of complete and total dominion over the machine. Great quote here too lol

Prometheus stole fire from the gods and gave it to man. For this he was chained to a rock and tortured for eternity.

Everybody is different, I simply cannot stand the sight of chatgpt styled writing. Give me paragraphs.

I've lobbied to replace our internal tool with a django admin panel. I prototyped it and it showed that it would reduce our code by > 15k lines.

Any internal webapps I need to build like this will 100% be set up with django in the future due to this. I don't need it to be pretty, I just want the UI, database migrations, users, roles, groups, etc for free

One trick to get out of this scenario where you're writing a ton is to ask the model to interview until we're in alignment on what is being built. Claude and open code both have an AskUserQuestionTool which is really nice for this and cuts down on explanation a lot. It becomes an iterative interview and clarifies my thinking significantly.

LLMs dont care about the story, they just care about the current state of the code

You have to tell it about the backstory. It does not know unless you write about it somewhere and give it as input to the model.

I'm sure the same could be said about tractors when they were coming on the scene.

There was probably initial excitement about not having to manually break the earth, then stories spread about farmers ruining entire crops with one tractor, some farms begin touting 10x more efficiency by running multiple tractors at once, some farmers saying the maintenance burden of a tractor is not worth it compared to feeding/watering their mule, etc.

Fast forward and now gigantic remote controlled combines are dominating thousands of acres of land with the efficiency greater than 100 men with 100 early tractors.

You should check out plain text accounting software Beancount. I've built a skill for this here https://github.com/ouachitalabs/skills/blob/main/beancount/S... and I use it for my custom personal finance agent.

The TL;DR was that old open source was the cathedral of exclusive developers and groups. Then the Bazaar showed up (which was the Linux Kernel for example) and that freed us from the shackles of the cathedral.

I didn't make it past the tldr lol is this some kind of poisoned data for GPT 6?

I ran a T450s all through college. My late grandmother gifted me the money after high school graduation to buy my first laptop. I'd learned to code on the old gateway pentium 4 (Windows XP) machine with 512 MB of RAM from the factory. I'd resurrected it as a teenager and completely gutted it. I scoured the web endlessly looking for something that was _not_ touchscreen (hard to find at the time), linux-compatible, and rugged enough for the neglect of a college student's backpack.

It was a linux user's dream. I swapped out the spinny HDD with an SSD. It had hotswappable batteries, a spill resistant keyboard with a drain system, and I ran every linux distribution I could get my hands on, eventually settling on the xfce flavor of Ubuntu.

I still have that machine, and will keep it for my son when he's older. I plan to replace the battery and SSD with something more reliable. These days I use an m2 macbook air for portability... but that thinkpad is still something I will treasure for years.

https://ouachitalabs.com/

I use claude exclusively at my day job with legacy codebases. I'm not using swarms or fancy multi-agent setups. Just sticking to plan mode, giving enough context, and iterating on a spec before starting the actual build.

I'm also not building webapps. I work in data engineering on a large legacy airflow project, internal python libraries, infrastructure with terraform, etc.

Unfortunately this is a skill issue. And it's a totally different skill than reading and writing code, building solid systems, and general software engineering at large. That is annoying, but where we're currently at.

Assume you're writing code manually, and you personally make a mistake. It's often worthwhile to create a mechanism that prevents that class of mistake from cropping up again. Adding better LSP or refactoring support to your editor, better syntax highlighting, better type checking, etc.

That same exact game of whack a mole now has to be done for you and whatever agent you're building with. Some questions to ask: What caused the hallucination? Did you have the agent lay out its plan before it writes any code? Ask you questions and iterate on a spec before implementation? Have you given it all of the necessary tools, test harnesses, and context it needs to complete a request that you've made to it? How do you automate this so that it's impossible for these pieces to be missing for the next request? Are you using the right model for the task at hand?