sshine

I've clarified; it's per month.

I also run Kubernetes on Hetzner and created my nodes prior to two price hikes.

Here's how expenses look and would look if I recreated it today:

  Control Plane   3xCX23  @ €3.99  = €11.97    @ €6.49  = €19.47
  Worker nodes    3xCX33  @ €6.49  = €19.47    @ €9.99  = €29.97
  NAT gateway     1xCPX22 @ €7.99  = €7.99     @ €30.99 = €30.99
  Load Balancers  2xLB11  @ €7.49  = €14.98    @ €9.36  = €18.72
  Hetzner Volumes 240GB   @ €0.044 = €10.56    @ €0.044 = €10.56
  ------------------------------------------------------------
  Total per month                    €64.97               €109.71

---

Only the NAT gateway and the LBs have public IPv4 addresses.

The CPX22 NAT gateway is a little overkill, but I was experimenting with making it both the NAT gateway and the load balancer, and I figured I needed dedicated CPU cores to handle inbound traffic. It was an attempt to cut the cost of Hetzner's own LBs, if it proved to scale. I haven't got around to that part, but today the economics wouldn't add up: LBs are cheaper and less work.

This is Talos Linux: https://talos.dev

Talos can be upgraded via talosctl, so even less maintenance than k3s.

The NAT gateway effectively is a bastion host and would serve fine as a VPN gateway.

The nodes themselves can join a tailnet,

Not only did your cost spike, it changed currency and went from postfix to prefix!

I understand people complaining about large bills, but this is over the top!

That's 87 trillion, 967 billion, 679 million, and so on.

Even though it's just a bug, being charged $595B on a platform that is known to cost spike, reminds us that we're not in control of the platform, or our company's expenses.

Are you doing this for client-facing production code?

What's "this"? We're talking about multiple things here.

But presumably yes: I'm working through agents, and all I do is read and prompt. I don't manually write code except Markdown and occasionally Nix code because it's sufficiently high-level, and I rarely read the Rust code. The Nix code is based on templates I wrote by hand, and the Rust code is based on crate selection and some loose principles I've derived from years of programming Rust by hand.

It seems you believe painting over APIs with some amount of documentation will guarantee the implementation is correct and well designed. Am I misreading that?

I don't believe that, you're misreading.

I'm saying:

  - Making and documenting APIs well is just as necessary as before I wrote to files.
  - Correctness is achieved through strong types and extensive testing of all kinds.

There are different tests, and I know they work for different reasons.

I'll list a few kinds that I've found actually valuable in the recent past:

Manual user tests: While expensive, they're most truthful in revealing bugs and confirming if they still exist. We do this by dogfooding our own product, and by running manual user tests before major deployments. User tests help uncover problems like delays that don't appear as errors, but can be major UX bugs (like waiting ten seconds for the effect of a button press).

TDD regression tests: A user experiences an error in the application, this gets reported, the agent analyses the code and writes one or more unit tests to confirm the problem, and writes a fix to see the light go green. I know this test works because it fails first. There's an odd chance the agent either uncovered a different bug, or wrote a test that fails synthetically. I haven't tried the latter, but I've tried many times that it wrote "a test" that passes but the app still crashes. So TDD order matters. And I've tried that it found several more bugs by analysing the code. (Most recent example: https://github.com/atuinsh/atuin/issues/3603 ) Another example of a TDD regression test is: Some async code prematurely exited, I scanned for the class of bugs and found one other case and fixed it, TDD. There's a very simple consideration to make that could be expressed as a linter rule, but is more easily expressed as a single sentence in CLAUDE.md, since it will appear again (it's a pattern I and former colleagues have made several times).

End-to-end integration tests: Wire up all parts of the system in either a semi-authentic (say, docker compose-based with some mocks) or in a fully authentic staging environment (say, Kubernetes); I do both, and it catches both bugs and performance regressions by simulating real workloads. I haven't automated the performance regression testing, only workload simulation, and then I assess the metrics myself.

Algorithmic pareto-optimality: I have an algorithm that can be measured in number of steps by measure of API calls, and number of successful calls; correctness would be "the algorithm gives the same result", and pareto-optimality would be "it doesn't use more steps in the refactored version". Computing time isn't really the cost here, the cumulative size of API payloads are. The API part can actually be mocked, or "made golden" (I've just now learned that it's called a "characterization test", https://en.wikipedia.org/wiki/Characterization_test, but I always called it a golden test from having read that.)

"Full coverage unit tests": I don't actually know how valuable these are. But for example, I went with "Find all functions that return Result<T> or Option<T> and test all their edge cases." Some ~87 functions, some ~150 tests. No failures. But I'd be surprised, because the type of mistake these tests generally fix for me, as a human, are typos and copy-paste errors, and big models generally don't make those.

You say you still read code when code is « algorithmically hard ». How do you define that?

I don't have a good definition. I recently deployed a service that runs something my colleagues call "the algorithm". It's some progressive audio transcription thing. My colleagues feel uncomfortable going live with something that appears to work that nobody read, so I read what was made.

I think the value of reading algorithmically hard code is not to assess correctness, but to regain cognitive debt in case something breaks and you don't know how it's made. I've consulted for people whose main programmer left and they were stuck with a machine they couldn't comprehend, start, or start to comprehend.

Point being: Code review isn't ideal for catching hard logical bugs, it's for understanding code so that you may eventually understand hard logical bugs when they surface.

Do you refactor code or is that not needed anymore?

The need for software architecting and refactoring are exactly the same.

Only maybe more and sooner because the amount of code increases.

I don’t understand your answer to the question of why anyone would publish a library if no one reads code anymore. « For the exact same reasons as before ». How so? This isn’t making sense.

I don't understand why it doesn't make sense.

Agents will read code. Humans do occasionally read some code.

But the value of a well-polished, tested library is not just for humans to read.

A good library will make your code better, because vibing on the spot delivers less guarantee that it's good. You'll miss some effort spent on proper data modelling and edge case and error handling. Sometimes that's the right call.

Say you build a parser for a small language. Using a parser combinator library, you get nice error messages and some very non-trivial thoughts on resuming parsing from a failed state. A vibed parser will give you neither of those.

There are some points related to supply-chain minimization.

But it's the exact same tradeoff, where the choice-points are pushed around.

With guidance, it kind of is happening.

While, simultaneously, an abundance of slop is being made.

Anthropic's entire goal is to extinguish software development as a profession

They’re killing “programmer”, not “code”.

I don't see how they wouldn't take out open source with it as a consequence

An analogy: the automobile industry sought to make working horses redundant, not to go door-to-door and kill horses. Horses getting chopped was an indirect economic consequence.

If nobody even looks at code why would anyone bother to publish a library, much less care about making it maintainable?

For the exact same reasons as before. Agentic programming still integrates well with the existing ecosystem; I’ll tell agents which libraries to use, so I know what to expect.

While I don’t read the implementation of anything any more unless there’s a hard algorithmic problem, I do make an effort to read and document APIs thoroughly.

Interfacing is exactly the same, it’s just agents doing it.

if everybody is vibe coding how long before your "average" dev has no idea what a library even is or why you'd want one?

That is a very good question.

extinguish the ecosystem since open source libraries are essentially a competitor. (If there were no libraries you'd have to ask the AI to do everything!)

That is neither the incentive of AI companies nor the truth.

Availability of Open Source where stealing and illegal relicensing is not being litigated, is a perfect ecosystem for AI to work in.

Maintainer exhaustion is totally a secondary effect, not intended. The maintainer economy was already not working out, AI amplified the asymmetry at play.

I recently set up a Forgejo instance for a personal infrastructure project, and because of security, it’s read-only. So I just disable all the “issues”, “pull requests”, “wiki”. It’s a little sad, but also gratifying to know that I’m making stuff available, but I am not looking for any and every chance to have a conversation about it.

All these comments about AI Slop are underestimating how good these builders have gotten and AI lets you iterate really fast.

Both of these are true: we’re witnessing an unprecedented amount of slop, while also the tools get better and better.

So when talking about Open Source maintainer exhaustion, it’s because of the slop, not because of the great tooling.

AI is an amplifier, and in this case it amplifies the great asymmetry between contributor and maintainer.

not mostly the test, but the real world production scars

Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.

SQLite is a good example to bring up because its extensive closed-source tests are what’s often cited as being what keeps people from forking it. (Turso did it, though, but it takes a company to deliver some guarantee of equivalent diligence.)

And yes, years and years of running.

a downside of this approach is that you sacrifice late binding, as the module language and "core" language are stratified into different layers

https://people.mpi-sws.org/~rossberg/1ml/1ml-extended.pdf

https://github.com/rossberg/1ml

might think it's a good idea to just run with a wildly out-of-scope idea with too much hubris

I'll say that using git as a database instead of simply SQL leads to a lot of corner cases.

For example, you are inside a transaction from the moment you make a single change to anything.

But the filesystem is conventionally thought of as something that can be shared between sessions.

So I need a git worktree per user.

In both my use-cases there's very few users, but if there were two, and they decide to change things independent of one another, then

  - their worktrees will become stale, and I need to add somewhat automated checking how far behind they are
  - their stale worktree will have conflicts, and dealing with merge conflicts in a pedagogical way is a big task
  - working on multiple things at once is possible, but introducing "committed" on top of "draft" + "publish" adds mental gymnastics
  - using the filesystem as a database makes deployment to Kubernetes less trivial because you need persistent storage

The nice thing about this is that if you remove the admin dashboard, it's just another static site.

It can be maintained perfectly fine with regular git and e.g. VSCode.

Add Nix + Claude and you have an entirely different mode of operation.

Which is what I like about plaintext (Markdown). It's not tied to any one program.

Triple backticks also let you specify the syntax very conveniently.

I think "yolo" gets overloaded here to mean "not thinking".

I've described it as static HTML.

The admin dashboard isn't static HTML, but it also isn't public-facing.

The admin dashboard lives behind an oauth2-proxy authenticated via keycloak, and only accessible to the corporate wifi / vpn. The dashboard is really a completely separate application.

The probability that you can hack an admin panel because you know the public URL assumes either there's a "Login" button on the front page, or that the admin panel depends on the source code in the same way as the public-facing pages.

I get what’s you’re saying, and if I couldn’t justify making the best alternative I can imagine in my free time, because I’ve wanted it for a long time, I’d install “a CMS” (not WordPress).

If there's a question about why money was spent on implementing a bunch of stuff nobody knows for a reason nobody cares about, especially for a very short-lived thing like a landing page, then it's a sticky situation if the answer is basically novelty.

The economy behind a decision like is this: alternative SaaS website builders are $20-60/mo./seat. We’ve historically paid $720/mo. for the ability to edit a single website that doesn’t look great but is dead simple to modify.

So if I can make something that scales up to any amount of sites and any amount of editors with ~10 hours on landing the design (which isn’t included in “a WordPress” either way), at ~$700, then I can justify making ten sites per year at the cost of our first.

Or more realistically: The total operating cost of the current website gives me 125 hours in a year to make something better.

Then the question is not “Can I make something better?” (Yes.) Or “Is it affordable to make something from scratch?” (It is.) But rather: Could I make more money doing something else? (I could halve the Azure budget in less than a month by optimizing and cleaning up.)

Thank you.

How do you hack a static HTML page?

The point is that most WordPress pages don’t warrant the dynamic code execution on every page load.

When you use a static site generator and make content creation convenient behind the scenes, you move the entire attack surface to, in my case, nginx, the load balancer, and OpenSSL.

I’ll say “proper docs” has shifted for me for two reasons

I used to insist on commenting code richly, so I could better read it. But comments lie, while code is truth. Read the code, that’s what it does.

With AI, the cognitive overhead of getting a human-worded explanation of what’s true, is one prompt away and is never a stale leftover.

So the purpose of docs: Specs for implementing and getting an architectural overview, and API documentation for exploring the interface of something new.

What I find great is that people still don’t test their code when it became practically free to do so.

I’ll try to tell the story in a more responsible way: My boss asked me to install a WordPress, to which I advised against it; while it’s easy to set up, it doesn’t align with our tech stack (his main team won’t be able to support it easily, woohoo army of juniors!), and the convenience of a quick start is outweighed by having a thing that needs CVE patching when, guess what never got hacked: pure, static HTML.

Since my wife had asked me twice the same week to set up a website with a design mock she’d sent me, I thought: what’s holding me back in both cases from giving them a Claude Design’ed Hugo theme is that they need to edit Markdown on their filesystem and run terminal commands.

So I picked an item out of my infinite backlog, which was very well-defined: a web dashboard that acts as the equivalent of the WordPress admin page that lets you manage a Hugo static site, use a rich editor on top of Markdown, and commit to git instead of a database. I spent the better part of a weekend making this, with my wife as the customer, and when it got good enough, I presented it to my boss. He was happy with the choice, but mostly because of the vibed design, he ultimately didn’t care about the technology.

When someone wants “a WordPress” they’re asking for convenience of an easily updated website.

You don’t have to actually give them a WordPress.

My boss asked me to set up a WordPress for a product landing page.

I naturally won't do this; it's no more than a couple of weeks ago that some SQL injection landed in the search query function of this monstrosity.

WordPress always was and always will be terrible.

So I set up the landing page with a Hugo static site, and I've been vibe-coding a WordPress-like dashboard that operates on git repositories containing Hugo sites.

I call it WorbPress (not released yet), and I'm sure that's what my boss told me to install, or I might've misheard.

And yes, it's written in Rust (with Axum and Alpine.js), because why not?

Linq-to-SQL (not Entity Framework) and I had to write the linq query in the reverse order

I remember those times! Had to write the LINQ, see what it compiled to, redo, until the query was efficient. Abuse LINQ subtleties in how it generated JOIN predicates since it only supported equality. Something about finding an equivalent way of expressing a query with sub-selects that is also computationally equivalent. All so I can get my efficient SQL without writing SQL. So silly.

Rewriting Redis in Rust sounds genuinely useful when you want to rely on it as a library.

Maintaining byte-for-byte parity with newer versions of Redis sounds like being in debt; why not rely on your fork instead?

I miss the days when HTML injection bugs were considered a feature.

The Internet was a lot more innocent before normies and money got involved.

I second this; even switching between minor versions of a model, you need to adjust prompts: the new model is better by implying a bunch of things that, when included in the prompt, will overdo that thing.

Assessing quality of output is often not trivial, either. Typically, problems that are solved by offloading something to an LLM are super subjective, and customers “feel” something is different is vulnerable.

We try to quantify output differences by many different similarity metrics. But a lot of energy goes into subjectively evaluating if something still works.

Some quick research:

Denmark to Iceland is ~2300 km which has a theoretical one-way signal propagation of ~11.5ms.

Real-world round-trip time would be 25-35ms though.

So for certain AI workloads, network is probably not the bottleneck.

60% of the time, it works every time!

I’m not sure how you can say something general about the quality of tests unless you mean by simply prompting “make tests” or similar.

Yes, I’ve experienced that those tests succeed, and the app still breaks trivially on first run.

What I mean is: you design the tests. You analyse patterns. You insist on making testable code (average code by humans isn’t, so neither is average code by LLMs unless you specify testability as a design constraint.

One way to get testable code is to mock all interfaces. This is usually expensive, but not difficult for an AI, because you can set the success criterion to be interface exactness of your mock for a series of plausible and somewhat extensive interactions.

The tests you can make with AI are as good as you can make them otherwise, you just save time doing them, which should justify making more extensive testing.

it feels like software development is going from a precise occupation that requires high degree of understanding to something probabilistic and offloaded understanding

To me it felt like there were always engineers and vibers.

Vibers don't work systematically, never test, accept unknown regression, don't use git, and if they do, treat it like Dropbox, use terrible languages, have terrible habits. Vibers got a new tool, too, and it vastly increases the amount of slop. But the slop was always there.

The slop actually got better after vibers stopped writing their own slop, I have to say.

And vibers are less defensive about the particular slop they didn't write themselves.