alexhans

If you actually are interested in understanding what I said, you can look at these links:

- https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt

- https://www.theregister.com/software/2001/06/02/ballmer-linu...

Every Linux user or FOSS enthusiast knows the acronym FUD: Fear, Uncertainty and Doubt, which were a set of techniques commonly used to disparage efforts of open source communities. Linux was evil and anticapitalist and we needed to use "CorporateTool" and ban/restrict Linux.

The same companies later would be running their entire infrastructures on it and on open source.

With AI, open weights and local models, we will see the same claims, even if the named fears change.

The end users and humanity are better served by collaboration and openness than by creating oligarchies.

Many of us were saying it a year ago but now with model restrictions (e.g. fable) and pricing changes it should be obvious to people that part of the economics is avoiding vendor lock-in with evals (so you can easily switch providers/models/optimize) and increasing control by investing in local models which could be good enough for your tasks, at whatever the price for your cloud compute is. Eventually consumer hardware will also be able to run good enough.

You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change.

- [1] https://alexhans.github.io/posts/find-the-loop-story-first.h...

Got it. Different audience. Cool stuff. I'll take a better look as soon as I can.

I've helped people get into programming face to face and also in a site I liked called exercism which also had a multi language track unit test passing style which I really value and it was purely command line, and I can't stress enough how important the command line is for me for people who want to dabble. Nowadays it's easier to get people into the command line because of Claude/codex.

I only have browsed your site from a phone and looks interesting but I wanted to ask if you had particular insights around getting people to approach learning, design through tests, breaking down problems, without having someone to guide them. Have you had a chance to observe people using your tool and adjust or it's been mostly dog fooding something you would've loved to have.

Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.

The hard thing is always keeping complexity low and being ZeroOps.

I can understand someone being overwhelmed and not wanting to configure and "build your own pi" which is really one of the beautiful points of pi, but like with vim, I do recommend that after playing with this for a while you go back to pure pi and then decide what do you really need and incrementally add it.

The power of the incremental in control approach is huge. It allows you to keep moving in whatever direction you want instead of taking yet another dependency.

What's the consensus today on codex vs claude code, does it really matter anymore?

Consensus is probably the wrong word for the popular opinions reflected in HN that you might get.

I would recommend that you have 2 of each at all times when it comes to AI so you don't necessarily become overly locked to quirks of one thing. You'll soon realize that things move so fast that you just start internalizing common patterns instead of depending on one specific vendor.

I recommend that you try pi and codex besides claude, to get your own feel for it.

Touch typing essentially. It's such a comfortable way to work. Remapping mode switching to something like jk instead of Esc is vital to stay comfortably in the home row.

I always liked this site to grok some of those vim fundamentals [1] and the touch typing part was going to touch typing exercise webpages and getting pure practice.

- [1] http://learnvimscriptthehardway.stevelosh.com/

Fun, responsive and intuitive. Congrats.

I guess it depends on what your exact hypothesis is Vs mine.

For at least 6 years we've had AC worthy temps.

The way eval startup is defined here is very specific and doesn't cover successful eval farmwork/SaaS vendors like Arize, Promptfoo, deepeval, etc

The author does have a point around generic benchmarks not being super valuable for companies. But evals should be seen as verifying design/behaviour constraints and can greatly aid product building, golden dataset creations and good software practices.

It's just that the aim should be "how to generate your own good evals, even if it's hard" as not so much "here's some generic evals about models".

I'd argue that these temperatures are not a new thing. It's just a contradiction that is almost a tradition at this point, imho.

Living in London and Dublin, what I've observed is that we get the following contradictory statements:

- "We don't need AC, It's only hot a few times during the year." - "Oh what a terrible heat, global warming is getting worse every year."

Pair to that the fact that in many places windows don't open all the way due to bureocratic regulations and many interior designs are very questionable in terms of air flow and you get some unpleasant scenarios.

Exactly. I'm very happy the discourse has moved on from "but X model is the best" to "you can use open models".

Whether you're using SDK or harness based agents, having evals means you're able to modify any part of your agent and still know what satisfies your "good enough".

It's great for designing products that are easy to change as well.

It's always satisfying when customer rights stories have a known positive outcome. The timeline is unfortunately quite slow and bureocractic but I'm glad OP managed to find out about it.

I think the criticism is constructive. It's really not about hating. I'd wager many of the people who convey this criticism do use AI to aid their writing as well.

It's just that this one in particular lacks one more edit pass removing some of the AI noise on branding-speak and needless repetition (AI tends to list things and beat the point).

Agreed. The post looks great. The story is great but the AI style in this case does distract.

I'm not against using AI for writing at all but you want to be careful that the output doesn't contain too much of this noise over signal type of wording that repeats and wants to just sell you something.

I think it's important to be able to do both so you can stay in control of the price to value created relationship.

In last year, some people were publishing aider /ollama/open router [1] and now thankfully people are publishing all around about pi/qwen/llama.cpp/openrouter. It's widespread.

[1] https://alexhans.github.io/posts/aider-with-open-router.html

The Foil Arms & Hog 12 year old skit [1] is still relevant.

- [1] https://youtu.be/Id-zzOGnN6A (Website part at 1:42 calling out the insurance example).

I was a fan of Seven Languages in Seven Weeks [1] because it exposed you to different paradigms which you could then try to apply where they made sense on whatever tools you were using or building: prototype based, fault tolerante, funcional, logical. Very fun book when used right.

The point being that sometimes the tools themselves don't need to survive because you take the lessons from one thing to another (e.g. move semantics and rust/modern c++)

[1] https://pragprog.com/titles/btlang/seven-languages-in-seven-...

Congrats. This seems like a great prompt to ensure a useful default experience. People should not confuse this with "anti cheating" and instead helping people learn how to learn.

Do you have further insights on AI and education since?

A good tip for side projects is to build stuff that is actually useful to you and to make them ZeroOps (no maintenance required). If you find yourself waiting to add a feature and it's not easy to change (ETC) [1] then it's a good moment to stop and how did you lose control of the software.

This applies to all software projects except for the "useful to you" part, which turns into valuable to "your/the business"

[1] https://youtu.be/c8AzqMr87gQ

When I saw this post I immediately thought of WordPerfect.

It had such a pleasant interface for that time.

But yeah, one of the things made me go for vim over Emacs a long time ago was its relationship with touch typing and not leaving the home row with the vim modes.

Learn Vim script the hard way [1], even if you didn't end up writing any actual vim script, was a game changer in terms of understanding the semantics.

[1] https://learnvimscriptthehardway.stevelosh.com/ was

Hey, sorry for the late reply.

I looked superficially at your site/repo and based on that initial impression:

- Your approach of comparing different parts of the "black box" which affects agent behaviour (Harness, foundation model, skills, context (in your case the loaded on AGENTS.md context) is closely aligned with how I both think and operate. - You're both tackling the "regression" and the "answer hypothesis easily" problems.

Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?

It depends on the level of automation and risk profile. For skills I use this framework of thinking [1] and encourage evals/ground truth as soon as possible so that you can have automatic feedback loops for the markdown part and for the deterministic part (scripts). Once you have the eval/ground truth pair, you're almost doing TDD or Eval Driven Development (which is quite hard the first times you try and realize you actually need to think about intent). The scripts should definitely have their own unit tests for the "skill iteration" in the event that a mutation is desired to cover new behaviour/fix wrong behaviour.

On Agent Skills, it may seem tempting to want more "openness" for the AI to solve the problem creatively but, more often than not, you've described a repeatable workflow and you want predictability and stability instead of novelty so it's really about 1) How can I freeze it to keep being good enough as much as possible 2) How can I know if something happened somewhere which changed the black box (e.g. coding harness auto model picking screws things up 3) How can I make the skill itself ETC (Easy to change), to keep control. Local Models can be a great tool for stability in some scenarios.

In particular, I prefer pass/fail (binary) outcomes instead of scoring which doesn't help regression decisions. Defining "good enough" should be very clear. Flakiness is not a good thing to accept, if the outcomes are consequential.

Anything actually risky should be solid RBAC/policy which doesn't really depend on the LLM.

I had a site that I didn't manage to make visible in HN to create a community for ai-evals.io. I've since interacted with a few people, developed further insights and given some private talks but need to get back to publishing outfacing and trying to contact more people interested in this space because it's absolutely critical. There's a lot of nuance in how different environments think about the eval problem differently: It's all about tracing and course correcting after launch, it's about simulations, sandboxing, security, automatic eval generation, etc.

In any case, I'll try to be more present from now on, and especially from June onwards to try to exchange insights in the open with people who are exploring different solutions in this space.

[1] https://alexhans.github.io/posts/series/evals/building-agent...

Sorry. I'm not sure about the specific part you don't agree with. You prefer people to just use skills instead of building them?

That's fair but I think this is similar to power tools like vim, obsidian or others. There's the path of grabbing other people's workflows and not being able to modify them to really tailor the tool to your needs and there's the minimal incremental path that empowers you and gives you control all the way through. It gets you to understand the tools and you'll be able to think possibilities that match your exact problems.

I'm not dogmatic about it but I do really recommend it. You can see the transformative shift once people start "skill building" instead of "skill consuming".

Edit: The approach I mention works with non engineers/developers. So there's no different technical bar.

Hey, it's awesome that you mention evals. May I ask what you currently use, or look for? Do you roll your own or use an existing framework?

Skills are just a good standard to describe repeatable workflows saving context through progressive disclosure, prompt sharing and, very underused feature, also bound the non deterministic parts with determism (which could be scripts).

Conceptually, you should treat them as incremental software instead of magic you grab from others [1]

The killer feature is that coding harnesses tend to have SkillBuilder agent skills so creating them becomes very easy and you can evolve them.

I recommend you build your own for your particular pain points.

Very simple example [2] showing what another user mentioned around "evals" so that you can really achieve good enough correctness for your automation.

- [1] https://alexhans.github.io/posts/series/evals/building-agent...

- [2] https://alexhans.github.io/posts/series/evals/sketch-to-text...

To me, design files living in the repo are what captures these high and low level decisions. They should express intent which is paramount for automation.

Where are you keeping your design? If it's a WYSIWYG doc like word, what would it take for you to switch to a more text driven world?

I saw https://news.ycombinator.com/item?id=47906942 and thought that the competing perspective of meetings as a resource that is inherently wasteful deserved dedicated discussion, only justified in cases where synchronous coordination is necessary.

If you don't force yourself to see it as a scarce resource it's very easy to end up in situations where it takes up most of your time.