Apache Iceberg Is Brilliant (and Your 12-Person Startup Doesn't Need It)

https://medium.com/fika-ventures/ducklake-why-early-stage-startups-should-stop-cosplaying-as-netflix-a5484cf320cd

by emmawirt • 5 months ago

2 4 5 months ago

waldopat• 5 months ago

Here's a founder/product perspective. This maps well to the skateboard => scooter => bicycle => motorcycle => rocket ship product metaphor that's often used. Each phase teaches different design patterns, constraints, and failure (and success) modes for different inflection points of a startup's journey.

But here's the reality. What got you technically to PMF may hold you back from your Series A and next steps. Technical debt is just the natural cost of growth, but (here's the kicker) optimizing tech stacks too early can lead to slower execution time. Most startups never reach exponential scale anyways. Put another way, starting with "rocket ship" does not immune the startup from rewrites, refactoring or throw away code.

The real systems and management challenge is building architectures that are intentionally temporary or modular. Simple enough that throwing them away later isn’t traumatic and rebuilds aren’t a sign of failure but success.

emmawirt2• 5 months ago

yeah I really love the 'intentionally temporary' framing, that's actually a way better way to articulate what was I was getting at than how I wrote it. The trauma of throwing things away is real, I see it constantly in DD. People treat migration like an admission of failure when it's usually the opposite, you outgrew something, which is a good problem to have.

Someone• 5 months ago

FTA: The design assumes S3 eventual consistency is your enemy

S3 has done better than that since December 2020. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...:

“Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent. What you write is what you will read, and the results of a LIST will be an accurate reflection of what’s in the bucket. This applies to all existing and new S3 objects, works in all regions, and is available to you at no extra charge! There’s no impact on performance, you can update an object hundreds of times per second if you’d like, and there are no global dependencies”

emmawirt2• 5 months ago

yeah you're totally right, good catch. S3 has been strongly consistent since 2020. That weakens the specific framing but the broader point about Iceberg's design assumptions still holds: the architecture was built for a world of eventual consistency and massive concurrent writes, and a lot of that complexity doesn't disappear just because S3 improved. But I should have been more precise there, appreciate the correction.

Apache Iceberg Is Brilliant (and Your 12-Person Startup Doesn't Need It)

Related Stories