gsgreen

Totally agree. The highest benchmark number is easy to chase. We're more interested in answering a different question: "If I deploy this same VM again next week, what should I expect?" That's the whole reason we keep rerunning the same clusters over time.

Small OCI terminology note that I hadn't realized before adding OCI to the dataset: Oracle uses "OCPU" differently than the vCPU language most AWS/GCP/Azure users are used to. In OCI, 1 OCPU generally maps to 1 physical core / 2 hardware threads, so this 8 OCPU Flex VM reports as a 16 vCPU instance in the benchmark data. That makes it closer to the 16 vCPU / 64 GB class when comparing across providers.

Still an early observation window (3 runs per cluster), so we're not drawing concrete conclusions just yet. We start doing that starting at 6 runs+ with temporal spacing between clusters but these patterns are def interesting so far...

6 fresh GCP Seoul deployments across 2 observation windows...

6 fresh AWS Seoul deployments across 2 observation windows...

Same Nuremberg - EU West endpoint. CPU/storage very stable...

We've been venturing into some new regions lately. Nice to see one that's just quietly stable.

Maybe nothing. Maybe something. Not sure. Fun to ponder either way.

Jun 6: $60.09/mo (first cluster) Jun 15: $153.59/mo (second cluster)

Same provider/region/plan - benchmark data almost identical. But that sizey bump.....

True story. Lot of cheaper providers vs. the biggies (and mediums) and we're slowly documenting all this 'behavior'.

For sure. At some point the math starts favoring bare metal over rented compute.

Especially if VPS prices keep rising faster than the underlying infrastructure.

Yer not wrong. Hetz bang/buck pretty nice for the most part. We'll see how/if that changes going forward with additional clusters we'll be doing.

Nine days apart! Maybe weird timing but pretty insane.

Jun 6: $60.09/month ---> Jun 15: $153.59/month -Same provider/region/plan.

In that span we reran the benchmarks and found the infrastructure behavior was nearly identical despite the 156% price increase.

We've run nearly 500 benchmark runs: provider selection matters but region does too big time.

Still early (3 runs each), so we'll keep adding depth with temporal buffering...

Hundy %.

Totally. Availability in US regions seems much tighter than Europe. Ashburn in particular. We'll see if they ramp up (hopefully) sooner than later.

One thing we've learned doing this: When a result surprises you, the correct response is usually "run it again." Looking forward to running again in later June...

Def wasn't a performance story - geography explained it. 2X? Wow.

Most of the interesting things we've found lately happened because repeated deployments changed and this one def didn't (networking did just a bit).

3 fresh runs of AWS m6i.4xlarge - almost identical. CPU, storage, agentic runtime behavior...everything.

When first allocating that vm I was utterly shocked.

Thought this would be a speed story.

Ended up being a predictability story.

Three fresh deployments looked almost boring. Which is probably exactly what you want from a $512/month VM.

What’s weird is the “normal” benchmark layer barely changed - CPU/storage stayed extremely consistent across all 6 runs.

The sustained workload behavior was the oddball.

Not every repeated deployment story ends in weirdness.

6 fresh GCP deployments over 2 weeks.

CPU: 702, 704, 703, 702, 704, 703.

Storage barely moved. No failures.

Boring. In a good way.

Run #1 looked boringly dependable. Almost no steal time. Very little drift, etc.

Then runs #2 and #3 immediately started behaving differently.

Same region. Same plan. 3 fresh deployments.

Curious what runs #4–#6 are going to do - will wait about a week to do for a time buffer.

For sure — the interesting part to me isn’t that burstable infra varies. It’s that the variance barely showed up in the normal benchmark layer at all while the sustained workload behavior started separating underneath.

Testing for diff scenarios and building a dataset to show perf deeply and broadly to help with decision-making. Trying to dial in the true bang/buck.

Fair if it reads that way but we've been working hard on documenting this data.

It's based on repeated runs (3 clusters / 9 runs) on different days — not generated output.

Worth noting: this only shows up with REPEATED runs.

Single benchmarks often look completely normal.

That’s why a ton of comparisons/benchmarks miss it. Doing this because that drives me a little nuts.

Even when you do control the environment, infra isn’t as stable as people think.

Same VPS, same config, but under sustained load you’ll see latency creep or throughput drift depending on the host / routing / neighbors.

Short tests almost never show it — only shows up after a few minutes.

Interesting site - going to delve. Especially 4.7 stuff....