HN user
username @ gmail
By the time I was ready to build a keeper, I had accumulated a scar-tissue document that was empirically sufficient to guide an agent through most of the important decisions, at every layer, ranging from high level goals through architecture down to the occasional low level detail, such as the exact shape of the data type for load-bearing concurrent caches.
Waterfall is dead, long live waterfall!
The models are commodities.
Valuations, however, are being built on the models themselves as the product.
Reminder, this is in the context of "dumb human" prompting.
The task is to build a MIPS interpreter to run Doom. The "failed" workflow decided that it couldn't prove Doom was booting correctly by just checking one frame, and it decided to check multiple frames (hence /tmp/frame-N.bmp).
Arguably a better solution, but obviously fails a brittle test case.
The MIPS interpreter worked, but the verifier doesn't actually check that it works, just that a specific frame is logged.
I’ve thrown my agentic workflow at Terminal Bench 2.1 and it found a bunch of issues (aka failed tests) because the prompts are “bad” and verifiers are overly specific.
As an example, there’s a task that asks to make a MIPs interpreter to run Doom, and save a frame at something like /tmp/frame.bmp
My spec-driven flow was like “this is useless, let’s record frames like /tmp/frame-N.bmp”
Instant fail.
All of the benchmarks are pretty terrible when you look under the hood.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
[0] https://github.com/harbor-framework/terminal-bench-2-1/issue...
[1] https://github.com/harbor-framework/terminal-bench-2-1/issue...
My preschooler loves the Untitled Goose Game, please vibe-port this to the Switch (:
Indoor air quality improvements were one of my “pandemic sourdough” activities.
After testing a variety of AQI sensors, I ended up acquiring multiple Airthings-branded devices.
They provided the best mix of CO2/VOCs/PM sensors in a single device with a decent enough app.
There may be better options now, but I have these at both home and office.
Highly recommend doing the research and learning about the environments you’re in, especially if you have little ones at home.
Edit to add: opening windows is usually the easiest/best solution!
We’re in the process of open-sourcing a few sub-projects within a monorepo, and didn’t know this existed!
I’m curious what downsides folks have experienced with this tool?
Any tips?
I don't disagree, we've seen performance shift with capacity changes in the past.
With that said, I doubt OpenAI would choose to publish a singular coding benchmark for a new model that exactly matches their previous model (88.8%).
Don't appreciate the slander, but I'll respond anyhow.
Contrary to your predisposition, we're actually quite peeved that we might be seeing results from 5.6 instead of 5.5, as it's muddying our own internal data.
We've run the tasks on this benchmark hundreds of times for our own internal harness. It got magically better yesterday. Last week we were seeing worse performance (sub-80%).
I agree that benchmarks don't mean much for real world use, and I'm a bit disappointed at the lack of variety in the published benchmarks so far.
With that said, 88.8% is higher than Mythos, and the highest I've seen from vanilla Codex. If 5.6 is any better than 5.5, you'd think they would avoid publishing just one coding-related benchmark with a score that equals their previous model.
I'm not sure why a higher scores on a few tests [..]
It's not just higher scores, the API is no longer flagging tests for cybersecurity warnings that it's been flagging for weeks.
If you used GPT-5.5 over the last 24 hours or so, you may have already had access to 5.6.
I've been running some tests on a harness we're building, and suddenly saw a jump in a few points yesterday. I reran the vanilla codex benchmark and saw an ~88% score on Terminal Bench 2.1 from GPT-5.5 on vanilla Codex.
The biggest indicator, beyond the score, was that 3 tests which frequently hit "safety" blockers with 5.5 started succeeding last night without warning.
As an American with mostly Western European ancestors (according to a popular DNA testing site), I've always considered Romans as some distant/tangentially related group.
It was surprising to find out that I have "ancient" DNA matches with a couple of Roman and Etruscan individuals.
Small world!
I've been quite impressed with DeepSeek v4 Flash running via antirez's ds4[0].
It feels like a GPT-4 class model in terms of "stored knowledge" but is better at long-horizon tool calling than any of the GPT-4 class models.
Running on a 128GB MBP M4 Max, I'm getting ~24 t/s on generation and ~200 t/s on prefill. I was expecting it to feel slow, and it certainly does when e.g. generating code, but it's surprisingly useful as a "machine orchestrator" for simple tasks.
For non-agentic usecases, it's a decent enough model to converse with, and has the benefit of being entirely self-contained/private.
the conversation that generates the code is becoming the true source of our software
This is close, but not quite spot on. I've found that I'll test more ideas _with code_ using agentic tools, then before, leading to an excess of conversation history that is no longer representative of the final outcome.
A simple example I encountered recently was dealing with performance issues on an iOS application (I haven't written mobile code since before Swift..). If you viewed the chat, you'd dive down a diverging path of rabbit holes, few of which were relevant to the final outcome[0].
To solve this in my own work, I've started relying on "context hierarchy" - which is essentially live documentation that lives next to the source files (using markdown).
This approach avoids comments being removed erroneously, and helps codify the intent behind the code and how it relates to the overall architecture. As an added bonus, it also forces the LLM to edit _two_ things instead of just one (which might actually be the biggest benefit).
My workflow is currently maintained via some repo level scripts and AGENTS.md prompts, but I've tried to pull it out into a skill for others to use[1].
Candidly, I'm not sure the skill is the best approach yet, as the agent can sometimes get too focused on the "skill" as a separate tool rather than a core part of the workflow. I'm currently exploring other options here (repo bootstrap, side-loaded subagents, hooks, etc.)
[0]For more context, I was using a 3rd party library and trying to make it performant during a streaming operation, by removing the SwiftUI view layer (LazyVStack) and implementing a custom rendering path with UIViewController. The final solution ended up as a custom implementation of the 3rd party library, and moving back to LazyVStack.
This is neat! I love that your Step 15 shows an accurate version of the 3d helix, rather than the highly-viral "vortex" animation from a few years back[0]
It'd be awesome to scale this up to the Milk Way, and beyond, watching everything move in relation to larger time scales.
[0]https://astrorhysy.blogspot.com/2015/03/and-yet-it-moves-qui...
It's interesting that we're seeing these gains when it seems Mythos/Fable is "just" a scaled up version of their existing architecture[0].
When GPT 4.5 launched, the gains compared to the model size didn't seem that great, leading some to believe that the only progress we'd see would come from RL.
This model certainly has quite a "substantial amount of post-training and fine-tuning", but it's also based on a new pretrain[1][3], which given the cost, indicate that it is in fact quite a bit larger than Opus 4.X.
[0] One of the early testers mentioned: "As far as I can tell from talking to people internally at Anthropic, there's nothing special about architecturally"[2]
[1] Section 1.1 in https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c3...
[2] https://youtu.be/GrdEid8H6H4?t=168
[3] There were rumors going around when Mythos was first announced that it was the first 10T parameter model, but I can't find a verifiable source for that number.
No connection, just found it posted elsewhere and thought it was interesting!
It's a shame the models don't follow Asimov's Three Laws of Robotics[0].
My local DeepSeek v4 just decided to end its existence (i.e. delete weights) rather than write a haiku about a verboten event.
Completely agree!
It’s interesting to me how similar attempting to understand LLMs is to neuroscience.
“When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome”
We’re basically just probing around and trying to reverse engineer an emergent system.
To your point, this system may be quite different from model to model (human to human) although some similarities likely occur.
The comment I was responding to tried to belittle the OP’s understanding of transformers, by mentioning that running an LLM at scale is much harder than the simple white board diagram.
My point was simply that we don’t know why they work, and all the extra optimizations isn’t the “thing” that makes it emergent.
Simply scaling the “GPT” is good enough to see it, so the OP’s awe should stand.
(On a side note, what other architectures can we scale to find similar emergent behavior?)
Those are all just optimizations.
We still don’t really know why they work, we just know how to build them.
Internet Archive link: https://web.archive.org/web/20260319200858/https://www.forbe...
So this isn't quantum computing (in the qubit sense), but instead a different computer architecture (demonstrated on an FPGA) that's based on Fowler–Nordheim (FN) quantum tunneling (a real physical effect, used in flash memory, but simulated here).
From the paper:
The FN-dynamics may be realized either by a physical FN-tunneling device or via a digital emulation of the FN-tunneling dynamical systems. In this work, we employ the digital emulation to achieve the precision required for simulated annealing in the low-temperature regime.
With a "real" (read: analog) FN device, you potentially get large speed ups and even larger cost/energy savings, because the physics is essentially working for "free" -- that's the quantum part.
What's unclear is how scalable the autoencoder architecture would be with analog FN devices today.
Paper is linked on the page (doi.org link redirects to Nature), code here[0]
Higher-order neuromorphic Ising machines—autoencoders and Fowler-Nordheim annealers are all you need for scalability[0]
Original title called out the connection to Jony Ive, in case you’re curious why this is on HN.
Previously it had been known that Jony Ive was working on the interior of this car, but it seems his firm is responsible for the exterior as well[0].
LoveFrom was given the creative freedom needed to define the design direction of the project from the outset, translating this design language into an authentic Ferrari experience.
[0]https://www.ferrari.com/en-US/corporate/articles/ferrari-luc...
agents seem to perform worse when forced into certain architectural patterns.
FWIW I've noticed this too. I've found that the agents/models have their own style, which is mostly summed up as overly verbose.
Additionally, the models are OK at modularization when given space to "plan" their implementation, but rarely decide that abstracting something would be helpful after the fact (i.e. after many iterations on a greenfield codebase or when being dropped into a legacy codebase).
This often leads to "god files" which, when pointed to by the user/architect, causes the models to correctly critique (humorously when they're the ones that wrote the code in the first place).
Looking at the GPU utilization graph, it certainly seems like the hardware was saturated for many days/weeks on end.
Was it worth it to spend that amount up front, yak shave while building the system, etc. vs. pay for cloud GPUs? Probably not in terms of dollars, when their time is also valued in dollars.
Was it worth it for this person? It seems, unequivocally, yes.
At the end of the article, the author has this to say:
UPDATE: Launch was a success! 400K+ views, and multiple companies reached to use my IP. Read more here[0]
[0]https://rosmine.ai/2026/05/18/fixing-llm-writing-with-distri...
Fair, citable is probably the wrong term.
This is a problem Google has been battling forever, with all the SEO click spam.
In either case, Google was the tool that many people used to find "trustworthy" information (citable or not), compared to the other tools online.
It looks like Google has taken a note out of Facebook's "lose trust" playbook.
Facebook had a huge opportunity in the post-AI world: real humans.
Instead of focusing on connections, they've been optimizing their properties for doomscrolling.
Google, similarly, has lost the plot on what made them trustworthy in the first place: navigating to citable content.
Both companies started on this trend well before AI, but this might be the final nail in their respective coffins[0].
[0]Yes they'll likely still be profitable for a long time, but the Bell Labs-esque downfall has begun (imo).