Harness Engineering

https://github.com/lopopolo/harness-engineering

79 32 4 days ago

lopopolo8• 4 days ago

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

jpitz• 4 days ago

This is interesting. I'd already had a conversation with my harness ( pi ) about incorporating continuous improvement. This is a great deal better than what I came up with.

lopopolo8• 4 days ago

Glad to hear it! Good luck, have fun. The agents tend to do a pretty good job incorporating these ideas. This was an unexpected thing we learned when publishing the initial harness engineering post.

mips_avatar• 4 days ago

One challenge/opportunity I've had is harnessing really wide running cheap agents. Any thoughts on how to move really cheap agents beyond basic summarization so we can go broader than the pricing of frontier llms allows?

lopopolo8• 4 days ago

Your “really cheap” agents can’t be so cheap that they do not have good tool calling skills. But! Using bigger models to put guardrails in place as static verifiers allows lower complexity changes to “self steer” as tests fail, which means coming down on the cost curve is more effective.

hankbond2• 4 days ago

How do you view harness engineering as an organic development that emerges from its use within a specific domain? Basically the meta-loop that allows an agent to tailor its harness to improve outcomes based on performance feedback. I use Pi a lot and I'm very interested in "self-assembling software".

One concrete example might be maintaining a conventions document per-project that covers how to name things semantically from a list of nouns and verbs. The idea is that LLMs are often not very globally aware, but it's important to maintain coherence across a code base in order for it to scale (in size and over time). Sometimes an LLM might call the same concept a Materialization, sometimes a Projection, and its not useful if its using two terms interchangeably without purpose.

Basically, how are you maintaining coherence when there isn't a human steering the code beyond providing requirements and validation directives?

I see you have relevant context in the repo like https://github.com/lopopolo/harness-engineering/tree/trunk/d... but I'm curious what exists beyond context. Do you use any tooling to steer this type of thing more consistently?

lopopolo8• 4 days ago

Your example is super amenable to vibing some tests. As an example, I’ve been able to ban `number` from representing a duration by walking the AST in a linter to fail if var or param names that look like the end in millis or ms or sec appear. This is largely good enough. If you see that “drifting” behavior appear more than once, you have enough to stop and force the agent to write some static verifiers that reject all but the option you want. For a closer example, we did this with zod schemes and their corresponding inferred types to be universally ZPascalCase and PascalCase instead of camelCaseSchema and CamelCase

slopinthebag• 4 days ago

What motivated you to quote your own quote (??) in your readme claiming to boost productivity by 100x, and where did you derive that number from?

"When a quote sounds profound enough, reality usually nods out of politeness, without echoes is just a sentence wearing pajamas." - slopinthebag

kubb• 4 days ago

I would also love to know. Sounds dubious to say the least.

esafak• 4 days ago

He's reached the singularity. Just kept pointing the agent at its own output, and leveled up...

burnto• 4 days ago

Don’t worry, the 100x boosted harness wrote it

hahahaa• 4 days ago

Thanks. I am guessing you have to try stuff and build tacit experience. No other way, just get stuck in and try stuff, then try and learn bits from others?

lopopolo8• 4 days ago

Basically yea. It is the only way to learn how to outrun your priors on what “high ambition” looks like. The labor that goes into implementation is an uncapped resource now.

lopopolo8• 4 days ago

The models are very good now so the feedback cycle on these meta adjustments is much tighter. Yesterday I was able to one shot a Liquid Glass, HIG-compliant and localized DICOM image viewer (frame by frame and looping video) with Apple Intelligence for de-jargoning the series details. Took 30 minutes. But the app had 60% CPU because it was not caching the decoded JPEGs. I can do a point in time fix for that of course, but the more interesting thing is why that misaligned code was permitted to be generated in a “done” artifact in the first place. What other misaligned code from a perf perspective might there be? And how do I intervene into the system that produced this software to make these misalignments statically not meet acceptance criteria?

zwaps• 3 days ago

You call your own blog post seminal and quote yourself in the repo.

Are you … alright?

DenisM• 3 days ago

In my little ai bubble the original blog post was very influential. It does sound immodest, but could also be true.

otekengineering• 3 days ago

Let me know how I can help you all let it rip.

i'd love to see a way (forum, competition, ?) for people to compare harnesses in different domains. folks like accountants, retail store owners, electrical engineers, and all sorts of niches are building personal harnesses/toolkits around claude/codex. those toolkits fit neither their domain communities nor SWE-heavy harness spaces, and a generic home for small niches could help them flourish.

a current problem is nomenclature. i suspect that many people have organically grown toolkits substantial enough to call harnesses, but are not close enough to the SWE bubble to be familiar with the word harness. ai has made it easier than ever to build bridges between domains, but it's also made it easier than ever to get domain tunnel vision and reinvent a super great wheel.

bagels• 4 days ago

The mother of all prompt injections.

Do others at OpenAI use this?

ricardobeat• 4 days ago

What percentage of this was written with AI?

A command can feel ambient without making its credential model-visible.

I'd be wary of following guidelines on how to build human-computer interaction systems that weren't written with full human oversight. This kind of wording makes me wonder if these are actual recommendations or just the AI pattern-matching and hallucinating something that looks coherent.

lopopolo8• 4 days ago

You can read the receipts in the linked source material yourself. I’ve also read and signed off on every word in this anthology. Try it before you knock it?

But yes, 100% of the content of this repository was written by GPT-5.6 Sol Ultra.

morbicer• 3 days ago

It might be readable to agents but to me it reads like a manic babble.

It's a topic that interests me a lot, I am not a blind AI hater and I can usually stomach LLM prose without issues but I am lost and repulsed here.

chrisweekly• 4 days ago

"weary" means "tired", you meant "wary" (guarded against or cautious and skeptical)

segmondy• 4 days ago

I don't even know how to feel about this after glancing through the repo.

N_Lens• 3 days ago

Just another framework, dime a dozen these days.

turing_complete• 4 days ago

Starting a repo readme with a quote of yourself...

richardfey• 4 days ago

Serious question: how do we verify claims like these on the effectiveness of a harness?

K3UL• 3 days ago

Prompt engineering, context engineering, now harness engineering. I guess in a few months we'll have looped back to software engineering ?

All these buzzwords for basically what's glorified tweaking and configuration. All these years I should have mentioned I was doing Debugger Engineering and IDE Engineering

112233• 3 days ago

I am getting my hair stood up by all the Comment Engineering going on in this thread. If all of the participants are organics, who are you people? If most is bots, the prospects are grim.

Harness Engineering

Related Stories