AWS to bare metal two years later: Answering your questions about leaving AWS

https://oneuptime.com/blog/post/2025-10-29-aws-to-bare-metal-two-years-later/view

by ndhandala • 9 months ago

727 485 9 months ago

alyxya• 9 months ago

With AI making it possible to use natural language to modify code, bare metal can make things easier to use with your own code and customization. Abstractions tend to be harder to reason about and have more limited functionality in exchange for being easier to get started on some standard setup.

JCM95• 9 months ago

For smaller operations I’d still go with a rent-a-server model with AWS. Theirs is a critical mass though where roll your own makes sense.

The long term app model on the market model is shifting much more towards buying services vs renting infrastructure. It’s here where the AWS case falls apart with folks now buying Planet Scale vs RDS, buying DataBricks over the mess that AWS puts for for data lakes, working with model providers directly vs the headaches of Bedrock. The real long term threat is AWS continues to whiff on all the other stuff and gets reduced to a boring rent-a-server shop that market forces will drive to be very low margin.

Yes a lot of those 3rd party services will run on AWS but the future looks like folks renting servers from AWS at 7% gross margin and selling their value-add service on top at 60% gross margin.

cmiles82• 9 months ago

A bunch written about this recently by analysts. That is the “bear” outlook on AWS

ramon156• 9 months ago

This doesn't really explain why you wouldn't just get a hetzner. I don't have much experience with either, but if you know how to setup your infra then hetzner seems like a no-brainer? I do not want to be tied to AWS where I have no idea what my bill will be

JCM95• 9 months ago

Depending on the use case you very much could just use Herzner. A simpler and more transparent customer experience than trying to navigate the mass complexity of AWS for basic stuff.

nik7362• 9 months ago

It's an interesting article, thanks for that.

What people forget about the OVH or Hetzner comparison is that for those entry servers they are known for, think the Advance line with OVH or AX with Hetzner. Those boxes come with some drawbacks.

The OVH Advance line for example comes without ECC memory, in a server, that might host databases. It's a disaster waiting to happen. There is no option to add ECC memory with the Advance line, so you have to use Scale or High Grade servers, which are far from "affordable".

Hetzner per default comes with a single PSU, a single uplink. Yes, if nothing happens this is probably fine, but if you need a reliable private network or 10G this will cost extra.

jammo2• 9 months ago

Yes, but there are options for dedicated server providers who offer dual PSU and ECC ram etc. It's more expensive though for e.g a 24 Core Epyc with 384GB RAM dual 10G netowork is like $500/month (though there's smaller servers on serversearcher.com for other examples)

vjerancrnjak• 9 months ago

Is there software that works without ECC RAM ? I think most popular databases just assume memory never corrupts .

torginus12• 9 months ago

I'm pretty sure they keep internal internal checksums at various points to make sure the data on disk is intact - so does the filesystem, I think they can catch when memory corruption occurs, and can roll back to a consistent state (you still get some data loss).

But imo, systems like these (like the ones handling bank transaction), should have a degree of resiliency to this kind of failure, as any hw or sw problem can cause something similar.

lossolo• 9 months ago

These concerns are exaggerated. I've been running on Hetzner, OVH and friends for 20 years. During that time I've had only two issues, one about 15 years ago when a PSU failed on one of the servers, and another a few years ago when an OVH data center caught fire and one of the servers went down. There have been no other hardware issues. YMMV.

hedora9• 9 months ago

They matter at scale, where 1% issues end up happening on a daily or weekly basis.

For a startup with one rack in each of two data centers, it’s probably fine. You’ll end up testing failover a bit more, but you’ll need that if you scale anyway.

If it’s for some back office thing that will never have any load, and must not permanently fail (eg payroll), maybe just slap it on an EC2 VM and enable off-site backup / ransomware protection.

torginus12• 9 months ago

I never understood the draw of 'server-grade hardware'. Consumer hardware fails rarely enough that you could 2x your infra and still be paying less.

hedora9• 9 months ago

Their current advance offerings use AMD EPYC 4004 with on-die ECC. I can’t figure out if it’s “real” single correction double detection, or if the data lines between the processor and dimms are protected or not though.

nik7362• 9 months ago

It's only on-die ECC not real ECC

montecarl• 9 months ago

I can't believe how affordable Hetzner is. I just rented a bare metal 48 core AMD EPYC 9454P with 256 GB of ram and two 2 TB NVME ssds for $200/month (or $0.37 per hour). Its hard to directly compare with AWS, but I think its about 10x cheaper.

titanomachy• 9 months ago

Wow. Probably performs better too, with a recent CPU and non-"elastic" disk. What about ingress/egress?

aetherspawn• 9 months ago

Ok but what about a dedicated OVH for example? Those are about 70% cheaper than AWS, so is it still worth it to colo?

bilekas3• 9 months ago

Did you read the article ? The main point of this and the prior article is that YES colocation/baremetal IS a better option for this company (and I would argue the majority of AWS users)

reference : https://news.ycombinator.com/item?id=38294569

bilekas3• 9 months ago

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ?

We have 730+ days with 99.993% measured availability and we also escaped AWS region wide downtime that happened a week ago.

This is a very nice brag. Given they are using their ddos protection ingress via CloudFlare there is that dependancy, but in that case I can 100% agree than DNS and ingress can absolutely be a full time job. Running some microservices and a database absolutely is not. If your teams are constantly monitoring and adjusting them such as scaling, then the problem is the design. Not the hosting.

Unless you're a small company serving up billions of heavy requests an hour, I would put money on the bet AWS is overcharging you.

JCM95• 9 months ago

As the author points out AWS can provide a few things that you wouldn’t want to try and replicate (like CloudFront) but for most other things you’re very much correct. AWS is ultimately very expensive for what it is. The complicated billing that’s full of surprises also makes cost management a head-banging experience.

tyingq• 9 months ago

Fair, though using AWS solely for CloudFront would mean you should compare to Cloudflare, Akamai, Fastly, etc. I'm not sure if the value prop for it looks so great if you don't include the "integrated with your other AWS stuff" benefit.

jagged-chisel• 9 months ago

Forget? You have to hire people for that. We are a software organization. We build software. If we rent in the cloud, there is less HR hassle - hiring, raises, bonuses, benefits, firing … none of that headache involved with the cloud.

Technically? Totally doable. But the owners prefer renting in the cloud over the people-related issues of hiring.

bilekas3• 9 months ago

Forget? You have to hire people for that. We are a software organization. We build software.

You don't need to hire dedicated people full time. It could even be outsourced and then a small contract for maintenance.

It's the same argument you could say for "accounting persons", or "HR persons" - "We are a software organisation!" - Personally I don't buy the argument.

chasd002• 9 months ago

Every company I’ve consulted for has hired a team dedicated to just setting up and monitoring AWS for the software devs. Hell, you’d probably reduce headcount running on bare metal.

9cb14c1ec02• 9 months ago

This is the fallacy that Amazon sold everyone on: that the cloud has no headache or managment needed. This is manifestly untrue. It's also untrue that bare metal takes lots of management time. I have multiple Dell rack servers colocated in several different datacenters, and I don't spend any time at all managing them. They just run.

jsiepkes• 9 months ago

This is exactly the rhetoric Microsoft used in the 00's with it's "Get the facts" marketing campaign against Linux and open-source: "Never mind the costs, think about the people hours you are saving!".

It wasn't as simple as that then, at it's still not as simple as that now.

shishcat• 9 months ago

Don't you have cloud architects and similiar figures already?

qaq• 9 months ago

Just because AWS abstracted something doesn't mean you don't need people who understand all the quirks of the black box you supposedly don't have to worry about. Guess what those people are expensive. You also have to deal with a ton of crap like hard resource account limits that on any meaningful size project will push complexity up by forcing you to use multiple accounts.

vidarh23• 9 months ago

I help people run their systems.

Clients that use cloud consistently end up spending more on devops resources, because their setups tends to be wastly more complex and involve more people.

Aldipower2• 9 months ago

Forgot? Driving something on AWS needs also a lot of people. In my experience even more. The term SRE did not exist before.

embedding-shape13• 9 months ago

We build software

Right, doesn't that include figuring out the right and best way of running it, regardless if it runs on client machines or deployed on servers?

At least I take "software engineering" to mean the full end-to-end process, from "Figure out the right thing to build" to "runs great wherever it's meant to run". I'm not a monkey that builds software on my machine and then hands it off to some deployment engineer who doesn't understand what they're deploying. If I'm building server software, part of my job is ensuring it's deployed in the right environment and runs perfectly there too.

rcxdude• 9 months ago

I really dislike the fallacy that just because you're buying something it means that you're not building anything. In practice this is never true: there's always some people-in-your-org time cost of buying something just as much as there's some giving-money-to-other-orgs cost to building something. So often organisations wind up buying something and spending way more time in the process than it would cost for them to build it themselves.

With AWS I think this tradeoff is very weak in most cases: the tasks that you are paying AWS for are relatively cheap in time-of-people-in-your-org, and AWS also takes up a significant amount of that time with new tasks as well. Of the organisations I'm personally aware of, the ones who hosted on-prem spent less money on their compute and had smaller teams managing it, with more effective results than those who were cloud-based (to various degrees of egregousness from 'well, I can kinda see how it's worth it because they're growing quickly' to 'holy shit they're setting money on fire and compromising their product because they can't just buy some used tower PCs and plug them in in a closet in the office')

base698• 9 months ago

Until you factor in the legions of devops writing terraform, iam, and cicd scripts.

canucktrash6694• 9 months ago

Ultimately these owners hire me to cut their 6-figure AWS bill by 50%. It's mostly rearchitecting mistakes. Amongst them is taking AWS blog propaganda at face value. Those savings could be 80% if they chose managed bare metal (no racking and stacking).

ericd• 9 months ago

You can just set up your own cloud on leased machines, and pocket the huge difference in cost. Devops languages are pretty easy to learn, IME, and the infra stuff takes less maintenance than the AWS proponents seem to think. I guess it depends on your usage profile, but like bandwidth especially is ruinously expensive compared to what you get with leased machines.

fabian2k• 9 months ago

A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect.

It's also that the requirements vary a lot, discussions here on HN often seem to assume that you need HA and lots of scaling options. That isn't universally true.

nicce• 9 months ago

A large part of the different views on this topic are due to the way people estimate the amount of saved effort and money because you're pushing some admin duties to the cloud provider instead of doing this yourself. And people come to vastly different conclusions on this aspect

This applies only if you had an extra customer that pays the difference. Basically argument only holds if you can’t take more customers because upkeeping the infrastructure takes too much time or you need to hire extra person which takes more money than AWS bill difference.

tstrimple2• 9 months ago

discussions here on HN often seem to assume that you need HA and lots of scaling options.

Funny how our perceptions differ. I seem to mostly see people saying all you need is a cheap Hetzner instance and postgres to solve all technical problems. We clearly all have different working environments and requirements. That's I roll my eyes at the suggestions in threads I see of going all in on colo. My last two major cloud migrations were due to colo facilities shutting down. They were getting kicked out and had a deadline. In one of the cases, the company I was working with was the second largest client at the colo but when the largest client decided to pull out the owners decided the economics of running the datacenter didn't make sense to them anymore. Switching colo facilities when you have a few servers isn't a big deal. It's annoying but manageable. When you have hundreds to thousands of servers, it becomes a major operational risk and is enormously disruptive to business as usual.

fulafel2• 9 months ago

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments.

("Shall we make the app very resilient to failure? Yes running on multiple regions makes the AWS bill bigger but you'll get much fewer outages, look at all this technobabble that proves it")

And of course AWS lock-in services are priced to look cheaper compared to their overpricing of standard stuff[1] - if you just spend the engineering effort and IaC coding effort to move onto them, this "savings" can be put to more AWS cloud engineering effort which again makes your cloud eng org bigger and more important.

[1] (For example implementing your app off containers to Lambda, or the db off PostgreSQL to DynamoDB etc)

Hilift4• 9 months ago

The direct cost is the easy part

I don't think it is easy. I see most organizations struggle with the fact that everything is throttled in the cloud. CPU, storage, network. Tenants often discover large amounts of activity they were previously unaware of, that contributes to the usage and cost. And there may be individuals or teams creating new usages that are grossly impacting their allocation. Did you know there is a setting in MS SQL Server that impacts performance by an order of magnitude when sending/receiving data from the Cloud to your on-premises servers? It's the default in the ORM generated settings.

Then you can start adding in the Cloud value, such as incomprehensible networking diagrams that are probably non-compliant in some way (guess which ones!), and security? What is it?

vidarh23• 9 months ago

I was about to rage at you over the first sentence, because this is so often how people start trying to argue bare metal setups are expensive. But after reading the rest: 100% this. I see so many people push AWS setups not because it's the best thing - it can be if you're not cost sensitive - but because it is what they know and they push what they know instead of evaluating the actual requirements.

anal_reactor2• 9 months ago

My manager wants me to make this silly AWS certification.

Let me go on a tangent about trains. In Spain before you board a high-speed train you need to go though full security check, like on an airport. In all other EU countries you just show up and board, but in Spain there's the security check. The problem is that even though the security check is an expensive, inefficient theatre, just in case something does blow up, nobody wants to be the politician that removed the security check. There will be no reward for a politician that makes life marginally easier for lots of people, but there will be severe punishment for a politician that is involved in a potential terrorist attack, even if the chance of that happening is ridiculously small.

This is exactly why so many companies love to be balls deep into AWS ecosystem, even if it's expensive.

torginus12• 9 months ago

Unfortunately it's not, and it gets more difficult the more cloud-y your app gets.

You can pay for EC2+EBS+network costs, or you can have a fancy cloud native solution where you pay for Lambda, ALBs, CloudWatch, Metrics, Secret Manager, (things you assume they would just give you, like if you eat at a restaurant, you probably won't expect to pay for the parking, toilet, or paying rent for the table and seats).

So cloud billing is its own science and art - and in most orgs devs don't even know how much the stuff they're building costs, until finance people start complaining about the monthly bills.

hinkley5• 9 months ago

My last team decided to hand manage a Memcached cluster because it cost half as much as an unmanaged service versus AWS’s alternative. Don’t know how much we really saved versus opportunity cost on dev time though. But it’s close to negative.

jmaker4• 9 months ago

It’s a marketing trap. But also a job guarantee since everyone’s in the same trap. You got a couple cloud engineers or "DevOps" that lobby for AWS or any other hyperscaler, NaiveDate managers that write down some decision report littered with logical fallacies, and a few years in the sink cost is so high you can’t get off of it, and instead of doing productivity work you’re sitting in myriads of FinOps meetings, where even fewer understand what’s going on.

Engineering mangers are promised cost savings on the HR level. Corporate finance managers are promised OpEx for CapEx trade-off, the books look better immediately. Cloud engineers are embarking on their AWS journey of certification being promised an uptick to their salaries. It’s a win/win for everyone, in isolation, a local optimum for everyone, but the organization now has to pay way more than it—hypothetically—would have been paying for bare metal ops. And hypothetical arguments are futile.

And it lends itself well to overengineering and the microservices cargo cult. Your company ends up with a system distributed around the globe across multiple AZs per region of business operations, striving to shave off those 100ms latency off your clients’ RTT. But it’s outgrown your comprehension, and it’s slow anyway, and you can’t scale up because it’s expensive. And instead of having one problem, you now have 99 and your bill is one.

wkat4242• 9 months ago

Yeah our company has recently decided to go all cloud (we're always so late with this stuff, especially now that a moment back from cloud has already started emerging).

But for me it's totally ruining my job to be honest. I like technology because it enables me to make things. I don't want to become an aws or azure specialist and learn what tickboxes the overlords at Amazon and Microsoft allow me to click. Screw that. That's nothing to do with technical knowledge, it's just about being a corporate drone. In my particular case it's Microsoft. Another problem with this is that they know they own everything in our company now so they're starting to treat us as employees, giving us things to do. Like promoting their features inside our company. I mean, they're a vendor FFS. They should answer to us.

A lot of my colleagues are really motivated with this shit and doing all the certs. Many are even becoming Microsoft evangelists and get pissed if I criticise someone. I'm looking for other options in the company now where I can actually do something technical again.

I understand there might be no bare metal work left in this company but in that case I'll just want to do something else. I don't want to be some goon that links their entire career to using the products of one big tech company. And also I think Microsoft and Amazon are horrible companies to work with as a customer. So me as a techie I just don't want to work at that anymore. What will remain are a lot of yes men who know how to click boxes.

And yeah we're not even doing anything smart or taking advantage of what the cloud offers. We just lifted all the physical stuff to compute instances that run 24/7.

ownagefool• 9 months ago

The consequence of running ingress and DNS poorly is downtime.

The consequence of running a database poorly is lost data.

At the end of the day they're all just processes on a machine somewhere, none of it is particularly difficult, but storing, protecting, and traversing state is pretty much _the_ job and I can't really see how you'd think ingress and DNS would be more work than the datastores done right.

Now with AWS, I have a SaaS that makes 6 figures and the AWS bill is <$1000 a month. I'm entirely capable of doing this on-prem, but the vast majority of the bill is s3 state, so what we're actually talking about is me being on-call for an object store and a database, and the potential consequences of doing so.

With all that said, there's definitely a price point and staffing point where I will consider doing that, and I'm pretty down for the whole on-prem movement generally.

vidarh23• 9 months ago

I'm generally strongly in favour of bare metal (not so much actually on prem) but your case is one of the rare cases wher AWS makes sense. Even for cheap setups like that, bare metal could likely be cheaper even factoring in someone on call to handle issues for you, but the amounts are so small it's a perfectly reasonable choice to just pick whatever you're comfortable with.

That's the sweet spot for AWS customers. Not so much for AWS.

The key thing for AWS is trying to get you locked in by "helping you" depend on services that are hard to replicate elsewhere, so that if your costs grow to a point where moving elsewhere is worth it, it's hard for you to do so.

neves• 9 months ago

It's always nice to remember that AWS is responsible for 70% of Amazon profits.

vidarh23• 9 months ago

As Jeff Bezos has been quoted as saying "your margin is my opportunity"...

The biggest difficulty in eating into AWS market share is that believing it is cheap has become religion.

mberning• 9 months ago

It’s expensive and the “design” of the services, if you could call it that, is such that you are forced to pay a lot, or play a lot of games to get around it. If you are going to spend your engineering time working around their ridiculous pricing schemes, you might as well spend the money on building things out yourself.

Perfect example - MSK. The brokers are config locked at certain partition counts, even if your CPU is 5%. But their MSK replicator is capped on topic count. So now I have to work around topic counts at the cluster level, and partition counts at the broker level. Neither of which are inherent limits in the underlying technologies (kafka and mirrormaker)

vb-84482• 9 months ago

Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ?

It's a way to "commoditize" engineers. You can run on premise or mixed infra better and cheaper, but only if you know what you are doing. This requires experienced guys and doesn't work with new grad hired by big cons and sold ad "cloud experts".

calgoo• 9 months ago

Also, when something breaks, you are responsible. If you put it in AWS like everyone else and it breaks, then its their problem not yours. We will still implement workarounds and fixes when it happens, but we are not responsible. Basic enterprise rules these days is to always pay someone else to be responsible.

esskay2• 9 months ago

I'm so surprised there is so much pushback against this

I'm not. It seems to be happening a lot. Any time a topic about not using AWS comes up here, or on Reddit there a sudden surge of people appearing out of nowhere shouting down anyone who suggests other options. It's honestly starting to feel like paid shilling.

7thaccount• 9 months ago

I think some of that is a certain group of people will do anything to play with the new shiny stuff. In my org it's cloud and now GPU.

The cloud stuff is extremely expensive and doesn't work any better than our existing solutions. Like a commentator said below, it's insidious as your entire organization later becomes dependent on that. If you buy a cloud solution, you're also stuck with the vendor deciding to double the cost of the product once you're locked in.

The GPU stuff is annoying as all of our needs are fine with normal CPU workloads today. There are no performance issues, so again...what's the point? Well... somebody wants to play with GPUs I guess.

TheCondor• 9 months ago

It’s the current version of CCIE or some of the other certs. People pay money to learn how to operate AWS, other thing erode the value of their investment.

Spooky232• 9 months ago

I don’t think it’s paid shilling, it’s dogma that reflects where people are working here. The individual engineers are hammers and AWS is the nail.

AWS/Azure/GCP is great, but like any tool or platform you need to do some financial/process engineering to make an optimal choice. For small companies, time to market is often key, hence AWS.

Once you’re a little bigger, you may develop frameworks to operate efficiently. I have apps that I run in a data center because they’d cot 10-20x at a cloud provider. Conversely, I have apps that get more favorable licensing terms in AWS that I run there, even though the compute is slower and less efficient.

You also have people who treat AWS with the old “nobody gets fired for buying IBM” mentality.

BirAdam2• 9 months ago

I'm not either. I used to do fully managed hosting solutions at a datacenter. I had to do everything from hardware through debugging customer applications. Now, people pay me to do the same but on cloud platforms and the occasional on-prem stuff. In general, the younger people I've come across have no idea how to set anything up. They've always just used awscli, the AWS Console, or terraform. I've even been ridiculed for suggesting people not use AWS. Thing is, public cloud really killed my passion for the industry in general.

Beyond public cloud being bad for the planet, I also hate that it drains companies of money, centralizes everyone's risk, and helps to entrench Amazon as yet another tech oligarchic fiefdom. For most people, these things just don't matter apparently.

mrits2• 9 months ago

I think people that lived through the time where their severs are down because the admin forgot to turn them back on after he drove 50 miles back from the colo might not want to live through that again

indymike• 9 months ago

A lot of people here's careers have been made by moving into AWS. A lot of people's future careers will be made by moving out of AWS. That's just the tech treadmill in action.

Do what works best for your situation.

sneak• 9 months ago

If your spend is less than a few thousand per month, using cloud services is a no-brainer. For most startups starting up, their spend is minimal, so launching on the cloud is the default (and correct!) option.

Migrating to lower cost options thereafter when scaling is prudent, but you "build one to throw away", as it were.

red-iron-pine• 9 months ago

It's honestly starting to feel like paid shilling.

the companies selling Cloud are also massive IT giants with unlimited compute resources and extensive online marketing operations.

like of fucking course they're using shillbots, they run the backend shillbot infrastructure.

they literally have LLM chatbot agents as an offering, and it's trivially easy to create fake users and repost / retweet last weeks comments to create realistic looking accounts, when then shill hard for whatever their goals are.

dumbledoren8• 9 months ago

Possible. However what is more likely is that a lot of long-time tech workers have vested stocks or investments in Amazon and they dont want the cash cow (AWS) to get hampered. And similarly a lot of tech workers have invested in AWS skills, so they cant risk those skills becoming less valued in the marketplace due to alternatives.

parliament32• 9 months ago

I don't think it's paid shilling, I think it's people who got bamboozled into learning cloud-provider-clickops over actual systems work and feel threatened when you suggest hyperscalers aren't the future.

realitysballs• 9 months ago

For my org. I don’t have budget for a dedicated in-house opsec team, so if I on-prem it triggers additional salary burden for security . How would I overcome this?

Msurrow• 9 months ago

Familiarize yourself with your company’s decision process on strategic decisions like this. Ensure you have a way to submit a proposal for a decision on making the change (or find someone who has that access to sponsor your proposal), build a business case that shows cost of opsec team, hardware and everything else is lower than AWS (or if cost is higher then some other business value is gained from making the change — currently digital sovereignty could be a strong argument if you are EU based).

If you cant build a positive business case then its not the correct move. Cash is king. Sadly.

Ensorceled2• 9 months ago

You can't. That's the use case FOR AWS/GCP. Once the differential between having a in-house team and the AWS premium becomes positive is when you make the switch.

A lot of the discussion here is that the cost of the in-house team is less than people think.

For instance: at a former gig, we used a service in the EU that handled weekends, holidays and night time issues and escalated to our team as needed. It was pretty cheap, approximately $10K monthly fee for availability and hourly rate when there were any issues to be resolved. There were a few mornings I had an email with a post-mortem report and an invoice for a hundred euros or so. We came pretty close to 5 9's uptime but we didn't have to worry about SLA's or anything.

vidarh23• 9 months ago

If you don't have budget for someone to handle this for you, you can't afford AWS either, as you still need to handle the same things and they're generally more complex when you use AWS.

izacus2• 9 months ago

Use the same people who are now maintaining your complex AWS setup. It's not like that doesn't need maintenance or oncall.

yomismoaqui• 9 months ago

Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ?

We should coin the term "Cloud Learned Helplessness"

vidarh23• 9 months ago

There is this belief that it is not extremely expensive and/or that the ops cost of bare metal will outpace it. It is a belief, and it is very rarely supported by facts.

Having done consulting in this space for a decade, and worked with containerised systems since before AWS existed, my experience is that managing an AWS system is consistently more expensive and that in fact the devops cost is part of what makes AWS an expensive option.

steelegbr• 9 months ago

AWS may be overcharging but it's a balancing act. Going on-prem (well, shared DC) will be cheaper but comes with requirements for either jack of all trades sysadmins or a bunch of specialists. It can work well if your product is simple and scalable. A lot of places quietly achieve this.

That said, I've seen real world scenarios where complexity is up the wazoo and an opex cost focus means you're hiring under skilled staff to manage offerings built on components with low sticker prices. Throw in a bit of the old NIH mindset (DIY all the things!) and it's large blast radii with expensive service credits being dished out to customers regularly. On a human factors front your team will be seeing countless middle of the night conference calls.

While I'm not 100% happy with the AWS/Azure/GCP world, the reality is that on-prem skillsets are becoming rarer and more specialist. Hiring good people can be either really expensive or a bit of a unicorn hunt.

PenguinCoder2• 9 months ago

I'm proudly 100% on prem Linux sys admin. There are not openings for my skills and they do not pay as well as whatever cloud hotness is "needed".

hibikir2• 9 months ago

And don't forget the real crux of the problem: Do I even know whether a specialist is good or not? Hiring experts is really difficult if you don't have the skill in the topic, and if you do, you either not need an expert, or you will be biased towards those that agree with you.

It's not even limited to sysadmins, or in tech. How do you know whether a mechanic is very good, or iffy? Is a financial advisor giving you good advice, or basically robbing you? It's not as if many companies are going to hire 4 business units worth of on prem admins, and then decide which one does better after running for 3 years, or something empirical like that. You might be the poor sob that hires the very expensive, yet incompetent and out of date specialist, whose only remaining good skill is selling confidence to employers.

canucktrash6694• 9 months ago

Managed servers reduce the on-prem skillset requirement and can also deliver a lot of value.

The most frustrating part of hyperscalers is that it's so easy to make mistakes. Active tracking of you bill is a must, but the data is 24-48h late in some cases. So a single engineer can cause 5-figure regrettable spend very quickly.

mhitza• 9 months ago

It's a chicken and egg problem. If the cloud didn't become such a proeminent thing, the last decade and a half would have seen the rise of much better tools to manage on-premise servers (= requiring less in-depth sysadmin expertise). I think we're starting to see such tools appear in the last few years after enough people got burned by cloud bills and lockin.

bcrosby95• 9 months ago

It depends upon how many resources your software needs. At 20 servers we spend almost zero time managing our servers, and with modern hardware 20 servers can get you a lot.

Its easier than ever to do this but people are doing it less and less.

dumbledoren8• 9 months ago

AWS may be overcharging but it's a balancing act. Going on-prem (well, shared DC) will be cheaper but comes with requirements for either jack of all trades sysadmins or a bunch of specialists

Much easier to find. Even more, they are skills much easier to learn for existing engineers. What's better, they are fundamental skills that will never lose their value as those systems are what everything else is built on.

speleding3• 9 months ago

The complexity of AWS versus bare metal depends on what you are doing. Setting up an apache app server: just as easy on bare metal. Setting up high availability MySQL with hot failover: much easier on AWS. And a lot of businesses need a highly available database.

spwa43• 9 months ago

A high availability MySQL server on AWS is about the same difficulty as on your own kubernetes instance (I've got a play one on one of those $100 N100 machines, got one with 16G mem). Then:

    helm repo add mariadb-operator https://mariadb-operator.github.io/mariadb-operator
    helm install mariadb-operator mariadb-operator/mariadb-operator

And then you can just provision MariaDB "kind", ie. you kubectl apply with something specifying database name, maximum memory, type of high availability (single primary or multimaster) and secret reference and there you go: new database, ready to be plugged into other pods.

PenguinCoder2• 9 months ago

Most businesses really don't need that complexity. They think they do. Premature optimization.

izacus2• 9 months ago

A lot of people here have built their whole professional careers around knowing AWS and deploying to it.

Moving away is an existential issue for them - this is why there's such pushback. A huge % of new developer and devops generation doesn't know anything about deploying software on bare metal or even other clouds and they're terrified about being unemployed.

goalieca• 9 months ago

meanwhile skills in operating systems, networking, and optimization are declining. Every system i've seen in the last 10 years or so has left huge cash on the table by not being aware of the basics.

Aurornis2• 9 months ago

I'm so surprised there is so much pushback against this.. AWS is extremely expensive.

I see more comments in favor than pushing back.

The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into a big waste of time and headcount that they should have been using to grow their businesses instead.

The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. Everyone goes through a honeymoon phase where the servers arrive and your software is up and running and you’re busy patting yourselves on the back about how you’re saving money. The real test comes 12 months later when the person who last set up the servers has left for a new job and the team is trying to do forensics to understand why the documentation they wrote doesn’t actually match what’s happening on the servers, or your project managers look back at the sprints and realize that the average time spent on self-hosting related tasks and ideas has added up to a lot more than anyone would have guessed.

Those stories aren’t shared as often. When they are, they’re not upvoted. A lot of people in my local startup scene have sheepish stories about how they finally threw in the towel on self-hosting and went to AWS and got back to focusing on their core product. Few people are writing blog posts about that because it’s not a story people want to hear. We like the heroic stories where someone sets up some servers and everything just works perfectly and there are no downsides.

You really need to weigh the tradeoffs, but many people are not equipped to do that. They just think their chosen solution will be perfect and the other side will be the bad one.

DrewADesign2• 9 months ago

The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely.

What the modern software business seems to have lost is the understanding that ops and dev are two different universes. DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator. Having someone that helps derive the requirements for your infrastructure, then designs it, builds it , backs it up, maintains it, troubleshoots it, monitors performance, determines appropriate redundancy, etc. etc. etc. and then tells the developers how to work with it is the missing link. Hit-by-a-bus documentation, support and update procedures, security incident response… these are all problems we solved a long time ago, but sort of forgot about moving everything to cloud architecture.

mjr002• 9 months ago

I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into a big waste of time and headcount that they should have been using to grow their businesses instead.

Funnily enough, the article even affirms this, though most people seemed to have skimmed over it (or not read it at all).

Cloud-first was the right call for our first five years. Bare metal became the right call once our compute footprint, data gravity, and independence requirements stabilised.

Unless you've got uncommon data egress requirements, if you're worried about optimizing cloud spend instead of growing your business in the first 5 years you're almost certainly focusing on the wrong problem.

You really need to weigh the tradeoffs, but many people are not equipped to do that. They just think their chosen solution will be perfect and the other side will be the bad one.

This too. Most of the massive AWS savings articles in the past few days have been from companies that do a massive amount of data egress i.e. video transfer, or in this case log data. If your product is sending out multiple terabytes of data monthly, hosting everything on AWS is certainly not the right choice. If your product is a typical n-tier webapp with database, web servers, load balancer, and some static assets, you're going to be wasting tons of time reinventing the wheel when you can spin up everything with redundancy & backups on AWS (or GCP, or Azure) in 30 minutes.

tetha• 9 months ago

To me it feels like nuance has been lost.

Personally, I would never self-host some B2C or B2B application if you have less than 50 - 100 techies in a healthy org. You can get just too much from a few VMs and/or a few dedicated servers at like Hetzner, OVH, or AWS managed services. At least for the average web rest thingy with a DB and some file storage. I'm sure it's possible to find counter-examples.

On the other hand, we are about 120 devs at work now, couple thousand B2B customers, 10 Platform Ops, 7 HW & DC Ops. I guess we have more ops-people than a startup may have people. Once we get rid of VMWare licensing, our colos are ridiculously cheap when amortized across 5 years compared to AWS or cloud hosting. Once EOL, they'll also reduce cloud-costs on cheaper providers for test systems and provide spontaneous failover and disaster recovery tests.

We're now also getting good cross-team scaling processes going and at this point the big barriers are actually getting enough power and cooling, not buying/racking/maintaining systems. That will be a big price tag next year, but we've not paid that money to AWS the last two years, so it's fine.

As I keep saying internally, self-hosting is like buying a 40 ton excavator, like Large Marge or a 40 ton truck. If you have enough stuff to utilize a 40 ton truck, it's good. If you need to move food around in an urban environment, or need to move an organ transplant between hospitals, a 40 ton truck tends to be rather inefficient and very expensive to maintain and run.

chickensong2• 9 months ago

All valid and important points, but missing a painful one, also rarely represented in threads like this: flaky hardware.

Almost every bare metal success story paints a rosy picture of perfect hardware (which thankfully is often the case), or basic hard failures which are easily dealt with. Disk replacement or swapping 1u compute nodes is expected and you probably have spares on hand. But it's a special feeling to debug the more critical parts that likely don't have idle spares just sitting around. The raid controller that corrupts it's memory, reboots, and rolls back to it's previous known-good state. The network equipment that locks up with no explanation. Critical components that worked flawless for months or years, then shit the bed, but reboot cleanly.

Of course everyone built a secure management vlan and has remote serial consoles hooked up to all such devices right? Right? Oh good, they captured some garbled symbols. The vendor's first tier of support will surely not be outsourced offshore or read from a script, and will have a quick answer that explains and fixes everything. Right?

The cloud isn't always the right choice, but if you can make it work, it sure is nice to not deal with entire categories of problems when using it.

rustystump• 9 months ago

Rarely do startups fail because of a decision like self hosting or not. In many cases it isnt even a few bad decisions but a long series of them plus outside factors which are uncontrollable.

In my experience the aws problem isnt so much that aws is that costly relative to bare metal but that most people do not execute well on aws over provisioning like mad to solve design issues.

There is a perverse benefit to sales at aws to push nonsense product too because of incentives. But people forget that aws isnt even as bad as a ton of other company spend. I have seen a fortune 100 add a few mm to their annual salesforce contract “for funnzies” because at their scale it wasnt that much money.

zjaffee• 9 months ago

AWS (along with the vast majority of B2B services in the software development industry) is good because it allows you to focus on building your product or business without needing to worry about managing servers nearly as much.

The problems here are no different than using SaaS anywhere else in a business, you can also run all your sales tracking through excel, it's just that once you have more than a few people doing sales that becomes a major bottleneck the same way not having an easier to manage infrastructure system.

maccard3• 9 months ago

I work for a small company owned by a huge company. We are entirely independent except for purchasing, IT, and budget approval. We run our CI on AWS, and it’s slow and flaky for a variety of reasons (compiling large c++ projects combined with instance type pressure). It’s also expensive.

We planned a migration to move from 4OD instances to one on prem machine and we guessed we’d save $1000/mo, our builds would be faster and we’d have less failures due to capacity issues. We even had a spare workstation and a rack in the office that so the capex was 0.

I plugged the machine into the rack and no internet connectivity. Put in an IT ticket which took 2 days for a reply, only to be told that this was an unauthorised machine and needed to be imaged by IT. The back and forth took 4 weeks, multiple meetings and multiple approvals. My guess is that 4 people spent probably 10 hours arguing whether we should do this or not.

On AWS I can write a python script and have a running windows instance in 15 minutes.

wredcoll5• 9 months ago

This is the root success of aws, it lets internal teams bypass sysadmin departments.

UltraSane2• 9 months ago

I'm not going to argue that AWS can be expensive but in my experience its biggest advantage is SPEED. In every company I worked for that ran their own data centers ever damn thing took FOREVER. new servers took months to buy and rack. any network change like a new VLAN took days to weeks. It was so annoying. But in AWS almost anything is just an API call and a few minutes at most from being enabled. It is so much more productive.

SJC_Hacker• 9 months ago

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ?

Long term yes you can save money rolling your own.

But with cloud you can get something up and running within maybe a few days, sometimes even faster. Often with built in scalability.

This is a much easier sell to the non-tech (i.e., money) people.

If the project continues, the path of least resistance is often to just continue with the cloud solution. At a certain point, there will be so much tech debt that any savings from long term costs from the traditional on-premises, co-location or managed hosting, are vastly by the cost of migration.

comprev• 9 months ago

I'm on a Platform team of <8 people and only 3 of us (most experienced too) come from sysadmin backgrounds. The rest have only ever known containers/cloud and never touched (both figuratively and literally :-) bare metal servers in their careers.

They've never used tools like Ansible (or Anaconda) or been in situations where they couldn't destroy the container and start afresh instantly.

eek2121• 9 months ago

I once moved a small site from AWS to Digital Ocean + Cloudflare.

$100-$300 on AWS -> $35/mo for DO + CF. Coincidentally, AWS had an outage soon after, which was avoided thanks to the move.

I have used DO for both clients and myself, and have not had any huge problems with them.

rdtsc• 9 months ago

I'm so surprised there is so much pushback against this.. AWS is extremely expensive.

Basic rationalization. People will go to extraordinary lengths to justify and defend the choices they made. It's a defense mechanism: if they spent millions on AWS they are not going to sit idly while HN discusses saving hundreds of thousands with everyone nodding and agreeing. It's important for their own sanity to defend the choice they made.

j45• 9 months ago

The cloud is incredibly profitable for the efficiencies and improvements its introduced and held onto.

Easy to push back against what is now the unknown (bare metal), when the layers extending bare metal to cloud service have become better and better, as well as more accessible.

dumbledoren8• 9 months ago

when did people forget how to run a baremetal server ?

Bigger question: When did people forget that doing that is much easier than AWS...

citizenpaul• 9 months ago

The "value add" of AWS has never been what it can do or does. It has always appealed to weak/incompetent/sociopathic managers and execs desire to not have to deal with capable employees.

As far as they are concerned AWS is taking care of computing AND hiring for them.

I've never worked anywhere that at least some sort of power holder would instantly go to consultants or outsourcing rather than in house because they believe that if you work for the company you must be incompetent, dumb or below average. If you don't work for them you must be exceptional.

_ea1k• 9 months ago

I'm so surprised there is so much pushback against this..

Same, this trend towards "AWS all the things" has really amazed me.

We've all mocked small companies copying big companies by trying to make their app super-duper scalable from the very start. After all, everyone things they are the next google, despite their 5 total users right now.

But this is really the opposite. AWS is phenomenal for the startup that would readily trade high opex for lower capex. Servers aren't the cheapest things in the world to buy and they depreciate. It makes total sense for startups to start this way.

But why are big companies, with an actual budget for staff, copying the behavior of their favorite startups?

guax2• 9 months ago

Opex looks nicer on the sheets than capex for large deployments. Incredible high investment from AWS on luring in C level with "white-papers" and promises of cost and governance magical revolutions. I've heard the promise of cheaper, faster where you can focus on "innovation". I am yet to see any of it become a reality.

mk89• 9 months ago

How would you do multi-region deployments with your own DC?

This is an issue for several companies that start small and within 5 years they find the need to expand abroad. Be it for data sovereignty or so, which is becoming more important than ever in the last 10 years.

Duplicating a region is "a few clicks away" on AWS. This is what the provider enables you to do.

This and a lot of other things. And for such things, yes, you gotta pay.

Hikikomori• 9 months ago

I mean its not that complicated. Rent space in another location, get separate fibers/wavelength between them, redundant internet connection.

But if you're in a growth/startup phase it doesn't make much sense to spend engineering time on this, not that multi region setups in Aws is one button either. Once you're past that and paying aws a million per week or so I think it can make sense to offload expensive services to your own hardware.

axegon_• 9 months ago

I am not - I hate AWS(and cloud in general) with a passion - overpriced, you are getting locked in by a closed ecosystem the moment you say "hey this feature is neat it will save me so much work", only to realize that you are stuck paying for it for years if you decide to move away from it. But people are inclined to jump on a hype train and become evangelists for life. Truth is AWS(or GCP or Azure or anything else) is a viable option in two cases:

1. You are making a product with 3 friends on evenings and you want to ship asap without having the capacity to invest and setup infrastructure. 2. You are a huge corporation with tens of thousands of employees and hardware needs that you simply cannot source yourself easily or sort out the collocation of the hardware.

Everyone else - get a dozen second-hand servers, shove them in a rack in a data center and you will own the hardware and everything associated with it at half the price of what you'd be paying AWS in a year.

kelnos3• 9 months ago

when did people forget how to run a baremetal server ?

I don't think people have forgotten, but I think Amazon has done an amazingly excellent job of marketing and developer relations over the years, to the point that they've convinced most developers that doing your own thing is 1) a lot of expensive, specialized work, and 2) actively dangerous for your business, whether it's because of security, uptime, or some other ops bogeyman of the day.

Note that I said "developers". Most developers are not sysadmins or IT operations people. Most of them have never set up Linux on a desktop or laptop, let alone on real server hardware (that they've also set up themselves). Most of them never had the chance to forget how to run a bare-metal server; they never knew how in the first place. (Hell, I've been running desktop Linux for 25+ years, and I don't think I've ever set up Linux on actual server hardware. Closest I've come is bare-metal Solaris, but that was like 25 years ago.)

"DevOps" today usually means that you know how to run a CLI tool or drive a web interface to deploy your automatically-built container artifact to some cloud-based production system that someone else manages, hiding the details from you. (This bit also can be true for shops that run on bare metal, depending on how advanced their own sysadmin/ops team is.) While developers are often not decision-makers in a larger org, they can be at smaller orgs, and once those developers get on the cloud, you probably will stay on the cloud (companies like OneUptime are the exception, not the rule), even if you've gotten much larger and it's stupid expensive to continue running that way.

hoppp• 9 months ago

AWS spends a lot in educating developers to use their service and it's a working strategy.

People simply don't believe bare metal is better because of this conditioning towards the cloud.

Every company should strive for self-sufficiency, but this idea is not that widespread in the software industry.

rasjani• 9 months ago

when did people forget how to run a baremetal server ?

My opinion on this: docker sort of changed the game here. It sort of enabled a lot of people to get a "new and fresh" level of abstraction to not bother about bare metal.

As an example, I work in company where most consultants are doing DevOps and k8 is big part of that.

What made me consider that? I've been told multiple times that "you know your stuff" when I mention some kernel or userland feature that container approach provides.

seidleroni• 9 months ago

As someone who works with firmware, it is funny how different our definitions of "bare metal" is.

embedding-shape13• 9 months ago

As someone who does material science, it's funny how our definition of "bare metal" is so different.

onionisafruit• 9 months ago

As someone who listens to loud rock and roll music …

amluto2• 9 months ago

Ask an astronomer what a “metal” is.

andrewl-hn• 9 months ago

In similar way I once worked on a financial system, where a COBOL-powered mainframe was referred to as "Backend", and all other systems around it written in C++, Java, .NET, etc. since early 80s - as "Frontend".

embedding-shape13• 9 months ago

Had somewhat similar experience, the first "frontend" I worked on was a sort of proxy server that sat in front of a database basically, meant as a barrier for other applications to communicate via. At one point we called the client side web application "frontend-frontend" as it was the frontend for the frontend.

pgwhalen• 9 months ago

I don't work in firmware at all, but I'm working next to a team now migrating an application from VMs to K8S, and they refer to the VMs as "bare metal" which I find slightly cringeworthy - but hey, whatever language works to communicate an idea.

ghaff6• 9 months ago

I'm not sure I've ever heard bare metal used to refer to virtualized instances. (There were debates around Type 1 and Type 2 (hosted) hypervisors at one point but haven't heard that come up in years.

Joeboy4• 9 months ago

Wikipedia still thinks it means the thing I (and presumably you) do.

https://en.wikipedia.org/wiki/Bare_metal

Edit: For clarity, wikipedia does also have pages with other meanings of "bare metal", including "bare metal server". The above link is what you get if you just look up "bare metal".

I do aim to be some combination of clear, accurate and succinct, but I very often seem to end up in these HN pissing matches so I suppose I'm doing something wrong. Possibly the mistake is just commenting on HN in itself.

embedding-shape13• 9 months ago

Seems there is a difference between "Bare Metal" and "Bare Machine".

I'm not sure what you did, but when you go to that Wikipedia article, it redirects to "Bare Machine", and the article contents is about "Bare Machine". Clicking the link you have sends you to https://en.wikipedia.org/wiki/Bare_machine

So it seems like you almost intentionally shared the article that redirects, instead of linking to the proper page?

cs7022• 9 months ago

In the early days of cloud service providers, they offered a handful of high-value services, all at great prices, making them cost-competitive with bare metal but much easier. That was then.

Things today are different. As cloud service providers have grown to become dominant, they now offer a vast, complicated tangle of services, microservices, control panels, etc., at prices that can spiral out of control if you are not constantly on top of them, making bare metal cheaper for many use cases.

embedding-shape13• 9 months ago

they offered a handful of high-value services, all at great prices, making them cost-competitive with bare metal but much easier

That was never the case for AWS, the point was never "We're cheap" but "We let you scale faster for a premium".

I first came across cloud services around 2010-2011 I think, when the company I worked at at the time started growing and we needed something better than shared hosting. AWS was brought up as a "fresh but expensive" alternative, and the CTO managed to convince the management that we needed AWS even if it was expensive, because it'll be a lot easier to tear up/down servers as we need it. Bandwidth costs I think was the most expensive part of the package, at least back then.

When I look at what performance per $ you get with AWS et al today, it looks the same, incredibly expensive for the performance you (don't) get. Better off with dedicated instances unless you team is lacking the basic skills of server management, or until the company really grown so it keeps being difficult dealing with the infrastructure, then hire a dedicated person and let them make the calls for what's next.

everfrustrated2• 9 months ago

I'd agree that AWS never sold on being cheaper, but there is one particular way AWS could be cheaper and that is their approach to billing-by-the-unit with no fixed costs or minimum charges.

Being able to start small from a $1/mth bill without any fixed cost overheads is incredibly powerful for small startups.

If I wanted to store bytes in a DC it would cost $10k/mth by the time I was paying colo/ servers/ disks before I stored my first byte. Sure there wouldn't be any incremental costs for the second byte but thats a steep jump. S3 would have cost me $0.02. Being able to try technology and prove concepts at the product development stage is very powerful and why AWS became not just a vendor but a _technology partner_ for many companies.

cs7022• 9 months ago

the point was never "We're cheap" but "We let you scale faster for a premium"

Actually, it was more like "Scale faster, easier, more reliably, with proven hardware and software infrastructure, operated by a proven organization, at a price point that is competitive with the investment you'd have to make to get comparable hardware, software, and organizational infrastructure." But that was then. Today, things are different. Cloud services have become giant hairballs of complexity, with plenty of shoot-yourself-in-the-foot-by-default traps, at prices that can quickly spiral out of control if you're not on top of them.

ActorNightly• 9 months ago

Yep.

AWS is a money printer because it enables companies to make a lot more money without hiring multiple 200k a year engineers to manage infrastructure that comes with scale. And that is also considering that the company will actually have to find competent people to do the work.

With AWS, if you wanna scale, you just use your credit card without worrying about anything.

JCM95• 9 months ago

This. When AWS was 10 solid core services it made sense and was exciting. It’s now a bloated mess of 200+ services (many of which almost nobody uses) with all that complexity starting to create headaches and cracks.

AWS needs to stop trying to have a half-arsed solution to every possible use case and instead focus on doing a few basic things really well.

genidoi• 9 months ago

Imo the fact that an "AWS Certified Solutions Architect" is yet another AWS service/thing that is attainable, via an actual exam[0] for $300, is indicative of just how intentionally bloated the entire system has become.

[0] https://aws.amazon.com/certification/certified-solutions-arc...

cmiles82• 9 months ago

Word on the street is that Amazon leadership basically agrees with this and recognizes things have gotten off course. AWS is a small number of things that make money and then a whole bunch of slop and bloat.

AWS was mostly spared from yesterday’s big cuts but have been told to “watch this space” in the new year after re:Invent.

jrochkind1• 9 months ago

(Real question, not meant to be sarcastic or challenging!) -- What are the challenges in trying to use just the ~10 core services you want/need and ignoring the others? What problems do the others you don't use cause with this use case?

rossdavidh3• 9 months ago

"Embrace, extend, extinguish". It was a Microsoft saying, but it explains Amazon's approach to Linux. Once your customers are skilled in how to do things on your platform, using your specialized products, they won't price-comparison (or compare in any other way) to competing options. Whether those countless other "half-arsed solutions" actually make money is beside the point; as long as the customer has baked at least one into their tech stack, they can't easily leave.

hinkley5• 9 months ago

I don’t think I’ve seen a menu as hilariously bad as the AWS dashboard menu. No popup menu should consume the entire screen edge to edge. Just a wall of cryptic service names with ambiguous icons.

rco8786• 9 months ago

Anytime I have to go into the AWS control panel (which is often) I am immediately overwhelmed with a sense of dread. It's just the most bloated overcomplicated thing I could possibly imagine.

antonkochubey• 9 months ago

You're lucky not to have dealt with Azure and GCP control panels, in that case :-)

rob74• 9 months ago

...while on the other side, the "traditional" hosting/colocation providers feel the squeeze and have to offer more competitive prices to stay in business?

__alexs2• 9 months ago

AFAICT no AWS service has ever had a price increase. This is nonsense.

raincole• 9 months ago

Cloud has been generally getting cheaper if you take inflation into account. But hating AWS is the fad so...

torginus12• 9 months ago

Considering you get exponentially more compute/hardware for the same money every 2 years or so, they haven't been getting that much cheaper.

myroon5• 9 months ago

Most years, AWS increases some prices:

https://github.com/SummitRoute/aws_breaking_changes/blob/mai...

dgemm• 9 months ago

This is the right take - there is a huge variation in "value per dollar" across AWS services. The base ones that solve hard problems like durable persistent state can be very much worth it. They tend to be the older ones.

didgeoridoo• 9 months ago

I’ve heard people talk about using “raw AWS” (instead of a PaaS like Heroku) to “save money.”

It’s like dude if you think AWS is raw, sushi restaurants must really confuse you.

doctorpangloss2• 9 months ago

Microk8s has common, catastrophic performance bugs. There are also catastrophic problems with microk8s Ceph addons. So is this post true? Microk8s, for people who know stuff, is a canary for clusters / applications that don’t really work.

ndhandala2• 9 months ago

We havent found those bugs in our cluster, but we're also moving to Talos (but for diff reasons)

acejam• 9 months ago

Source? Links?

doctorpangloss2• 9 months ago

https://www.google.com/search?q=microk8s+dqlite+site%3Agithu...

Click on the various catastrophic issues. Observe how many are closed with no resolution. Canonical is great but Microk8s is not.

jammo2• 9 months ago

Equinix Metal is now EOL, so worth bearing that in mind..

darkwater3• 9 months ago

The core of this success is this, IMO:

  > Our workload is 24/7 steady. We were already at >90% reservation coverage; there was no idle burst capacity to “right size” away. If we had the kind of bursty compute profile many commenters referenced, the choice would be different.

Which TBH applies to many, many places, even if they are not aware of it.

marcinzm2• 9 months ago

I'd say the core of their success is running everything in a single rack in a single datacenter at first (for months? a year?) and getting lucky. Life is simple when you don't need the costs and effort of reliability upfront.

darkwater3• 9 months ago

They mention having a second half-rack in a different DC.

In any case, not everyone need five nines, and usually it's just much easier to bring down a platform due to some bug in your own software rather that the core infrastructure going down at a rack level.

Hardwired8976• 9 months ago

They mentioned having a backup AWS cluster that would spin up when something happens.

jdsully• 9 months ago

Even if you have that you'll find AWS is "out of stock" and wants you to create reservations that essentially cost the same as just having the machine 24/7.

Nemo_bis• 9 months ago

Reminds me of https://www.specbranch.com/posts/one-big-server/

dumbledoren8• 9 months ago

Nah. They could have just overprovisioned to hell for much cheaper. Boxes at Hetzner cost up to 10 times less than equal level of AWS compute. Just overprovision for cheaper. You have to overprovision on the cloud anyway - you cant risk your users waiting 1-2 minutes until your new nodes/pods come up. So 'cloud is good for spiky load' argument is just a lie we tell ourselves.

darkwater3• 9 months ago

Well in cloud you do over provision a bit by setting autoscaling rules in a way that you still have spare capacity while the new resources are bootstrapping.

deanCommie• 9 months ago

YES, but let's also not ignore that the market is cold right now and the tech industry isn't really growing like crazy like it did during ZIRP.

So yeah in a hot economy anything you launch grows crazy. Then you do the thought leader talk circuit.

And in a cold economy, you can stop growing, optimize for your 24/7 steady workload, and also do the thought leader talk circuit.

cornfieldlabs3• 9 months ago

Equinix Metal got the closest, but bare metal on-demand still carried a 25-30% premium over our CapEx plan. Their global footprint is tempting; we may still use them for short-lived expansion.
The Equinix Metal service will be sunset on June 30, 2026.

https://docs.equinix.com/metal/

mythz• 9 months ago

Several years off AWS, the only thing I still prefer AWS for is SES, otherwise Cloudflare has the more cost effective managed services. For everything else we use Hetzner US Cloud VMs for hosting all App Servers and Server Software.

Our .NET Apps are still deployed as Docker Compose Apps which we use GitHub Actions and Kamal [1] to deploy. Most Apps use SQLite + Litestream with real-time replication to R2, but have switched to a local PostgreSQL for our Latest App with regular backups to R2.

Thanks to AI that can walk you through any hurdle and create whatever deployment, backup and automation scripts you need, it's never been easier to self-host.

[1] https://docs.servicestack.net/kamal-deploy

sondr3• 9 months ago

Cloud makes sense when elasticity matters; bare metal wins when baseload dominates.

This really is the crux of the matter in my opinion, at least for applications (databases and so on is in my opinion more nuanced). I've only worked at one place where using cloud functions made sense (keeping it somewhat vague here): data ingestion from stations that could be EXTREMELY bursty. Usually we got data from the stations at roughly midnight every day, nothing a regular server couldn't handle, but occasionally a station would come back online after weeks or new stations got connected etc which produced incredible load for a very short amount of time when we fetched, parsed and handled each packet. Instead of queuing things for ages we could instead just horizontally scale it out to handle the pressure.

marcinzm2• 9 months ago

They were running for a long time (months? over a year?) on a single rack in a single datacenter. Eventually they scaled out but the word is eventually. I think that summarizes both sides of this debate in a nutshell. You can move off of AWS but unless you invest a lot you will take on increased risk. Maybe you'll get lucky and your one rack won't burn down. Maybe you won't. They did get lucky.

athrowaway3z• 9 months ago

From the story, they seem to have kept the option to fallback on AWS.

gizzlon2• 9 months ago

Hm.. I wonder what the risk of a rack going offline is? Maybe 5% in a given year? Less? More?

Compared to all the other things that can and will go wrong, this risk seems pretty small, but I have no data to back that up.

shakow• 9 months ago

Maybe you'll get lucky and your one rack won't burn down

Given the rates of fires in DCs, you'd rather need to be quite unlucky for it to happen to you.

cornfieldlabs3• 9 months ago

Managed DB costs a lot.

Is there a simple safe setup that we can run on an Ubuntu server?

We self-host the Postgres db with frequent backups to s3 but just in case the site takes off, we need an affordable reliable solution.

Does anyone here run their own db servers? Any advise?

Backups, security, upgrades etc

lofties• 9 months ago

I love the argument that Managed DBs cost a lot, but they're supposedly safer. Meanwhile people can't figure out the IAM permission models so they give the entire world access with root:root.

ndhandala2• 9 months ago

If you're running k8s cluster. Check out cloudnative pg. That thing is a beast.

cornfieldlabs3• 9 months ago

We have hosted on everything on a tiny Hetzner. The site barely has any users apart from our friends:) :(

Info noted

film422• 9 months ago

Maybe look at R2 or Wasabi instead of S3. That would cut your storage bill by 3x and take your cloud network bill to zero. IMO self-managing DBs always sucks no matter what you do.

blindriver• 9 months ago

Have they done a complete failover to their second data center? It wasn’t clear how committed of a failover it was during the tests.

aeve890• 9 months ago

We're now moving to Talos. We PXE boot with Tinkerbell, image with Talos, manage configs through Flux and Terraform, and run conformance suites before each Kubernetes upgrade.

Gee, how hard is to find SE experts in that particular combination of available ops tools? While in AWS every AWS certified engineer would speak the same language, the DIY approach surely suffers from the lack of "one way" to do things. Change Flux with Argo for example (assuming the post is talking about that Flex and no another tool with the same name), and you have a almost completely different gitops workflow. How do they manage to settle with a specific set of tools?

zppln• 9 months ago

If you're that much of a slave to your tool chain you don't get to call yourself an engineer.

film422• 9 months ago

Or you have PTSD after 10 years of being on-call 24/7 for your company's stack. I've built my next chapter around offloading the pager. Worth every penny.

63stack• 9 months ago

Argocd and flux are "almost completely different"? The last time I looked was about a year ago, and there seemed to be only minor differences.

What are the major differences?

amluto2• 9 months ago

I would not want to hire an engineer who claimed to be proficient with any cloud Kubernetes stack but couldn’t learn Talos in a week.

Ragnarork• 9 months ago

Gee, how hard is to find SE experts in that particular combination of available ops tools?

You find expert in Ops, not in tools. People that know the fundamentals, not just the buttons to push in "certain situations" without knowing what's really going on under the hood.

ecshafer5• 9 months ago

AWS is extremely expensive, and I think I have to agree with DHH's assessment that many developers are afraid of computers. AWS is taking advantage of that fear of actually just setting up linux and configuring a computer.

However to steelman AWS use. Many businesses are STILL running mainframes. Many run terrible setups like Access as a production database. In 2025 there are large companies with no CICD platforms or IAC, and some companies where even VC is still a new concept or a dark art. So not every company is in the position to actually hire competent system administrators and system engineers to set up some bare metal machines and configure Ceph, much less Hadoop or Kubernetes. So AWS lets these companies just buy this capabilities while forcing the software stack to modernize.

faxmeyourcode• 9 months ago

I worked at a company like this, I was an intern with wide eyes seeing the migration to git via bitbucket in the year ... 2018? What a sight to see.

That company had its own data center, tape archives, etc. It had been running largely the same way continuously since the 90s. When I left for a better job, the company had split into two camps. The old curmudgeonly on-prem activists and the over-optimistic cloud native AWS/GCP certified evangelist with no real experience in the cloud (because they worked at a company with no cloud presence). I'm humble enough to admit that I was part of the second camp and I didn't know shit, I was cargo culting.

This migration is still not complete as far as I'm aware. Hopefully the teams that resisted this long and never left for the cloud get to settle in for another decade of on-prem superiority lol.

ecshafer5• 9 months ago

I was a at a company that was doing their SVN/Jenkins migration to Git/Bitbucket/Bamboo around 2016/2018. But they were using source control and a build system already, so you have to hand it to them. But I have an associate that was at one of the large health insurance companies in 2024, complaining that he couldn't get them to use git and stop deploying via FTP to a server. There is danger with being too much on the cargo cult side, but also danger with being too resistant to change. I don't know how you can look at source control, a CICD pipeline, artifacts, IaC, and say "This looks like a bad idea".

iLoveOncall2• 9 months ago

This is a completely meaningless article if they don't provide information about their technical stack, which AWS services they used to use, what TPS they are hitting, what storage size they're using, etc.

The story will be different for every business because every business has different needs.

Given the answer to "How much did migration and ongoing ops really cost?" it seems like they had an incredibly simple infrastructure on AWS, and it was really easy to move out. If you use a wider-range of services the cost savings are much more likely to cancel themselves.

globular-toast2• 9 months ago

TFA begins with a link to the original article with those details.

iLoveOncall2• 9 months ago

If you called "We used EKS" details, then yeah they provide those details.

Assuming this is indeed all they used, this was admittedly nonsense, they were essentially using cloud-based bare-metal.

tuhgdetzhh• 9 months ago

Quite recently I made a TCO analysis between AWS and bare metal Hetzner including salary. https://beuke.org/hetzner-aws/

TYPE_FASTER• 9 months ago

It depends on your workload.

Very much this.

Small team in a large company who has an enterprise agreement (discount) with a cloud provider? The cloud can be very empowering, in that teams who own their infra in the cloud can make changes that benefit the product in a fraction of the time it would take to work those changes through the org on prem. This depends on having a team that has enough of an understanding of database, network and systems administration to own their infrastructure. If you have more than one team like this, it also pays to have a central cloud enablement team who provides common config and controls to make sure teams have room to work without accidentally overrunning a budget or creating a potential security vulnerability.

Startup who wants to be able to scale? You can start in the cloud without tying yourself to the cloud or a provider if you are really careful. Or, at least design your system architecture in such a way that you can migrate in the future if/when it makes sense.

mr_toad• 9 months ago

This is a tech company and it’s adjacent to their core competency. Most companies wouldn’t know MicroK8s from a brand of cereal, they’d only create a mess if they tried this themselves.

gizzlon2• 9 months ago

Sure, but they also create a mess in AWS

stuff4ben• 9 months ago

Never heard of Talos before now. That looks pretty cool and I might start playing with that on my home lab. Can't use it at work for reasons, but good to keep on top of tech (even if I am a little behind)

globular-toast2• 9 months ago

This dude did a complete walkthrough setting up a Talos cluster on bare metal: https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talo... It's a nice read. I have my own Talos cluster running in my homelab now for over a year with similar stuff (but no Ceph).

roschdal• 9 months ago

Bare metal is the best metal.

Aldipower2• 9 months ago

Never ever. True metal it is!

ksec• 9 months ago

Many other points. When the Cloud Started, they offered great value in adjacent product and services. Scaling was painful, getting bare metal hardware have long lead time, provisioning takes time. DC was not of as high quality, Network wasn't as redundant. A lot of these today are much less of an issue.

In 2010 you could only get 64 Core Xeon CPU coming in 8 Sockets, or maximum or 8 Core per socket. And that is ignoring NUMA issues. Today you could get 256 Core per socket that is at least twice as fast per core. What used to be 64 Server could now be fitted into 1. And by 2030, it would be closer to 100 to 1 ratio. Not to mention Software on Server has gotten a lot faster compared to 2010. PHP, Python, Ruby, Java, ASP or even Perl. If we added up everything I wouldn't be surprised we are 200 or 300 to 1 ratio compared to 2010.

I am pretty sure there is some version of Oxide in the pipeline that will catch up to latest Zen CPU Core. If a server isn't enough, a few Oxide Rack should fit 99% of Internet companies usage.

submeta• 9 months ago

There is so much hidden cost in maintaining your own bare metal infrastructure. I am always astounded by how people overlook the massive opportunity cost involved in not only setting up, securing, and maintaining your bare metal infrastructure, but also make it state of the art, including best practices, making sure you have required uptime, monitoring and intervening if necessary. - I work in a highly regulated market with 700 coworkers, our IT maintains an endless amount of VMs. And you cannot imagine how much more work they have to do compared to a setup where you spin up services in AWS or Azure. And destroy it when you don’t need it. No updates, no patches. No misconfiguration. Not every company uses automation either (chef, ansible and whatnot)

saxenaabhi• 9 months ago

I agree, I have a restaurant POS system and I think self-hosting would easily kill the product velocity, and if we screw up bad, even the company.

However, I do get the point about cost-premium and more importantly vendor-risk that's paid when using managed services.

We are hosted on cloudflare workers which is very cheap, but to mitigate the vendor risk we have also setup up replicas of our api servers on bunny.net and render.com.

pingoo101010• 9 months ago

Many startups and companies couldn't exist if there was only AWS (or GCP / Azure) due to how much they overcharge.

For example, we couldn't offer free GeoIP downloads[0] if we were charged the outrageous $0.09 / GB, and the same is true for companies serving AI models or game assets.

But what makes me almost sick is how slow is the cloud. From network-attached disks to overcrowded CPUs, everything is so slooooow.

My experience is that the cloud is a good thing between 0-10,000 $ / month. But you should seriously consider renting bare-metal servers or owning your own after that. You can "over-provision" as much as you want when you get 10-20x (real numbers) the performance for 25% of the price.

[0] https://downloads.pingoo.io

hedora9• 9 months ago

I’ve seen cloud slowness create weird Stockholm syndrome effects, especially around disk latency.

It always makes sense to compare to back of the envelope bare metal numbers before rearchitecting your stack to work around some dumb cloud performance issue.

ed_mercer• 9 months ago

Talos is great until it's not. We ran into Ceph IO speed bottlenecks and found it was impossible to debug ("talosctl cgroups —preset=io" is a mess) because the devs didn't want to add an SSH escape hatch into their black box OS. Our Talos nodes would also randomly become unhealthy and you have no way of knowing why. Switched to PXE booted Alpine linux with vanille k8s, and we had a much more stable experience with no surprises, and the ability to SSH whenever we want has been hugely helpful.

dev_l1x_be2• 9 months ago

AWS is extremely expensive.

I really like how people throw around these baseless accusations.

S3 is one of the cheapest storage solutions ever created. The last 10 years I have migrated roughly 10-20PB worth of data to AWS S3 and it resulted in significant cost saving every single time.

If you do not know how to use cloud computing than yes, AWS can be really expensive.

Aurornis2• 9 months ago

The implicit claims are more misleading, in my opinion: The claim that self-hosting is free or nearly free in terms of time and engineering brain drain.

The real cost of self-hosting, in my direct experience with multiple startup teams trying it, is the endless small tasks, decisions, debates, and little changes that add up over time to more overhead than anyone would have expected. Everyone thinks it’s going to be as simple as having the colo put the boxes in the rack and then doing some SSH stuff, then you’re free of those AWS bills. In my experience it’s a Pandora’s box of tiny little tasks, decisions, debates, and “one more thing” small changes and overhauls that add up to a drain on the team after the honeymoon period is over.

If you’re a stable business with engineers sitting idle that could be the right choice. For most startups who just need to get a product out there and get customers, pulling limited headcount away from the core product to save pennies (relatively speaking) on a potential AWS bill can be a trap.

marcosdumay5• 9 months ago

The claim that self-hosting is free or nearly free in terms of time and engineering brain drain.

Free? No, it's not free. It only costs less engineering time than AWS.

rafabulsing• 9 months ago

The implicit claims are more misleading, in my opinion: The claim that self-hosting is free or nearly free in terms of time and engineering brain drain.

Not only is that not an implicit claim in the post, they explicitly say the that it's not free, it's actually just around the same amount of time they used to spend with AWS:

Total toil is ~14 engineer-hours/month, including prep. The AWS era had us spending similar time but on different work: chasing cost anomalies, expanding Security Hub exceptions, and mapping breaking changes in managed services. The toil moved; it did not multiply.

As for the following:

If you’re a stable business with engineers sitting idle that could be the right choice. For most startups who just need to get a product out there and get customers, pulling limited headcount away from the core product to save pennies (relatively speaking) on a potential AWS bill can be a trap.

You're just agreeing with them:

Cloud-first was the right call for our first five years. Bare metal became the right call once our compute footprint, data gravity, and independence requirements stabilised.

MontyCarloHall3• 9 months ago

Assuming those 20PB are hot/warm storage, S3 costs roughly $0.015/GB/month (50:50 average of S3 standard/infrequent access). That comes out to roughly $3.6M/year, before taking into account egress/retrieval costs. Does it really cost that much to maintain your own 20PB storage cluster?

If those 20PB are deep archive, the S3 Glacier bill comes out to around $235k/year, which also seems ludicrous: it does not cost six figures a year to maintain your own tape archive. That's the equivalent of a full-time sysadmin (~$150k/year) plus $100k in hardware amortization/overhead.

The real advantage of S3 here is flexibility and ease-of-use. It's trivial to migrate objects between storage classes, and trivial to get efficient access to any S3 object anywhere in the world. Avoiding the headache of rolling this functionality yourself could well be worth $3.6M/year, but if this flexibility is not necessary, I doubt S3 is cheaper in any sense of the word.

torginus12• 9 months ago

How the heck does anyone have that much data? I once built myself a compressed plaintext library from one of those data-hoarder sources that had almost every fiction book in existence, and that was like 4TB compressed (but would've been much less if I bothered hunting for duplicates and dropped non-English).

I suspect the only way you could have 20PB is if you have metrics you don't aggregate or keep ancient logs (why do you need to know your auth service had a transient timeout a year ago?)

matwood• 9 months ago

Like most of AWS, it depends if you need what it provides. A 20PB tape system will have an initial cost in the low to mid 6 figures for the hardware and initial set of tapes. Do the copies need to be replicated geographically? What about completely offline copies? Reminds me of conversations with archivists where there's preservation and then there's real preservation.

dev_l1x_be2• 9 months ago

Does it really cost that much to maintain your own 20PB storage cluster?

If you think S3 = storage cluster than the answer is no.

If you think about S3 what it actually is: scalable, high throughput, low latency, reliable, durable, low operational overhead, high uptime, encrypted, distributed, replicated storage with multiple tier1 uplinks to the internet than the answer is yes.

spprashant• 9 months ago

The thing I find counter intuitive about AWS and hyper-scalers in general is, they make so much sense when you are starting out a new project. A few VMs, some gigs of data storage, you are off to the races in a day or two.

As soon as you start talking about any kind of serious data storage and data transfer the costs start piling up like crazy.

Like in my mind, the cost curve should flatten out over time. But that just doesn't seem to be the reality.

shadowgovt• 9 months ago

Sounds like they did the right thing for their business model.

I think as AWS grows and changes the curve of the target audience is changing too. The value proposition is "You can get Cloud service without having a dedicated Cloud team," but there are caveats:

- AWS is complicated enough that you will still need a team to integrate against it. The abstractions are not free and the ones that are leaky will bite you without dedicated systems engineers to specialize in making it work with your company's goals.

- For small companies with little compute need, AWS is a good option. Beyond a certain scale... It is worth noting that big companies build their own datacenters, they don't rely on someone else's Cloud. Amazon, Google, and Microsoft don't run on each other.

- Recently, the cost model has likely changed if a company pokes their head up and runs the numbers, there's, uh, quite a few engineers with deep knowledge of how to build a scalable cloud infrastructure available to hire now for some reason. In fact, a savvy company keeping its ear to the ground can probably snap up some high-tier talent very soon (https://www.reuters.com/business/world-at-work/amazon-target...).

It really depends on where your company's risk and cost models are. Running on someone else's cloud just isn't the only option.

yanslookup2• 9 months ago

FD: I work at Amazon, I also started my career in a time where I had to submit paper requests for servers that had turn around times measured in months.

I just don't see it. Given the nature of the services they offer it's just too risky not to use as much managed stuff with SLAs as possible. k8s alone is a very complicated control plane + a freaking database that is hard to keep happy if it's not completely static. In a prior life I went very deep on k8s, including self managing clusters and it's just too fragile, I literally had to contribute patches to etcd and I'm not a db engineer. I kept reading the post and seeing future failure point after future failure point.

The other aspect is there doesn't seem to be an honest assessment of the tradeoffs. It's all peaches and cream, no downsides, no tradeoffs, no risk assessment etc.

AndroTux• 9 months ago

Managing a complex environment is hard, no matter whether that’s deployed on AWS or on prem. You always need skilled workers. On one platform you need k8s experts. On the other platform you need AWS experts. Let’s not pretend like AWS is a simple one-click fire and forget solution.

And let’s be very real here: if your cloud service goes down for a few hours because you screwed something up, or because AWS deployed some bad DNS rules again, the world moves on. At the end of the day, nobody gives a shit.

yanslookup2• 9 months ago

Maybe I've drank the koolaid but I've done both a lot of systems level work and AWS work (I don't actually use any AWS stuff in my role here interestingly) and I think for a business that needs a handful of hosts in 2 AZs I can't imagine the ROI and risk profile being better to self host.

AWS truly does let you focus on your business logic and abstracts a TON of undifferentiated work and well beyond the low hanging fruit of system updates and load balancing.

I guess put another way, providing a SaaS you need to have an SLA, those SLAs flow from SLO and SLIs and ultimately a risk profile of your hw and sw. The risk of a bad HBA alone probably means a day of downtime if you don't do things perfectly. AWS has bad HBAs, CPUs, memory, disks etc all day long every day and it's not even a blip for customers, never mind downtime. And if you don't model bad HBAs in your SLAs then your board is going to be pissed when that outage inevitably happens.

Now if you don't have SLAs and you like sysops, networkops, clusterops, dbops work then sure, YOLO.

hedora9• 9 months ago

At another big-4 hyperscaler, we ended up with substantial downtime and a lossy migration because they didn’t know how to manage kubernetes.

Microk8s doesn’t use etcd (they have their own, simpler thing), which seems like a good tradeoff at single rack scale: https://benbrougher.tech/posts/microk8s-6-months-later/

The article’s deployment has a spare rack in a second DC and they do a monthly cutover to AWS in case the colo provider has a two site issue.

Spending time on that would make me sleep much better than hardening a deployment of etcd running inside a single point of failure.

What other problems do you see with the article? (Their monthly time estimates seem too low to me - they’re all 10x better than I’ve seen for well-run public cloud infrastructure that is comparable to their setup).

yearolinuxdsktp2• 9 months ago

I agree that a business should use Kubernetes only if there is a clear need for that level of infrastructure automation. It's a time and money mistake to use K8s by default.

dumbledoren8• 9 months ago

Variants like k3 are not as complicated and problematic as k8.

fulafel2• 9 months ago

Indeed Kubernetes is problematic for the complexity and fragility reasons. It's a scalability problem, it's designed for big scale with kube staffs and situations where cost savings from bin packing etc outweigh costs of the resulting complexity.

But, SLAs are no good (who cares about getting refunded).

thelastgallon4• 9 months ago

These are the features that AWS provides

(1) Massive expansion of budget (100 - 1000x) to support empire building. Instead of one minimum-wage sysadmin with 2 high-availability, maxed-out servers for 20K - 40K (and 4-hour response time from Dell/HPE), you can have 100M multi-cloud Kubernetes + Lambda + a mix-and-match of various locked-in cloud services (DB, etc.). And you can have a large army of SRE/DevOps. You get power and influence as a VP of Cloud this and that and 300 - 1000 people reporting to you.

(2) OpEx instead of CapEx

(3) All leaders are completely clueless about hiring the right people in tech. They hire their incompetent buddies who hire their cronies. Data centers can run at scale with 5-10 good people. However, they hire 3000 horrible, incompetent, and toxic people, and they build lots of paperwork, bureaucracy, and approvals around it. Before AWS, it was VMware's internal cloud that ran most companies. Getting bare metal or a VM will take months to years, and many, many meetings and escalations. With AWS, here is my credit card, pls gimme 2 Vms is the biggest feature.

torginus12• 9 months ago

The problem with those 5 people, is you can't hire a 6th - your stack is custom and probably even if you find the guy, he'll need months of ramp-up.

In contrast, you could throw a stone into a bush and hit an AWS guy.

rikafurude21• 9 months ago

If your 6th needs months to understand how the basic blocks in your system are arranged then he might not be one of the "good" guys

thelastgallon4• 9 months ago

Why would you need more people? Don't treat the 5 people like shit.

sgarland6• 9 months ago

Bare-metal management isn’t custom, it’s just not cloud. A TFTP server, your choice of confirmation management, your choice of image builder, probably - but not necessarily - a hypervisor, and that’s pretty much it.

This stuff isn’t hard, and it was solved well over a decade ago. Probably two at this point. It’s just been shoved down by the hyperscalers (who themselves are busily hiring for those skills).

dilyevsky3• 9 months ago

You say cloud allows massive expansion like it's a negative but it can be boon for a pre-pmf startup or a scaleup. You simply don't have to worry much about capacity planning in cloud and that can be a huge time/effort saver.

Sure, if you're only growing <30% YoY and already paying several millions for cloud and bandwidth/storage are large fraction of that, by staying in cloud you're proving your incompetence as an engineering org.

thelastgallon4• 9 months ago

You say as if every startup scales to facebook size within months. Most startups are hilariously small and their workloads won't stress out a mac mini. Of course, you could overengineer and spend too much and hire too many people, but thats on the people running the startup.

Slothrop99• 9 months ago

one minimum-wage sysadmin

The internet assures me there are loads of these underemployed Unix/networking experts just sitting around waiting to set up your infrastructure. But in my experience, these people are actually really difficult to hire, and not at all cheap. (Possibly the sharp ones have 'sold out' and gone the SRE route and are now one of those '3000' people.)

So I wonder if there's a certain amount of wishful thinking on both sides here, like "I wish a 'clueful' company would hire me to be their head sysadmin...", while companies who have tried to do this on the cheap usually just have terrible ops. ("Whoops, the backups haven't worked in 2 years...")

agoodusername63• 9 months ago

Yeah I'm one of them. Started as an on-site Linux sysadmin. Moved to cloud SRE because remote is plentiful and it pays better.

I get crap recruiters in my inbox and LinkedIn every other week with the worst offers to go back to on-site bare metal admin. 30% less pay, on-site requirements, and it's a contracted position?

I need that Futurama "oh you're serious, let me laugh harder" gif

If companies want to whine that good Linux datacenter ops doesn't exist anymore, laugh in their faces.

Sohcahtoa82• 9 months ago

(2) OpEx instead of CapEx

Someone please explain to me why this matters. I'd think that expenditures are expenditures, and that if the outright purchase of hardware would see an RoI compared to renting it in the cloud in under a year, it'd be a no-brainer to just buy the hardware.

guax2• 9 months ago

OpEx means that if demand for your service goes down, cost goes down, your hardware does not become a capital liability since it depreciate fast. Way easier to justify changes to it too, you don't need a purchase project to get new instances, you're already "approved" and the contract was already signed with fluctuating costs. Needs more hardware? press a button, no need to research vendors, get contract negotiations in place.

AWS makes the life of finance and leadership a lot easier because they spend a lot of money justifying their superiority in ways that you don't have to think too hard to use and be taken seriously. They're to CTOs what think tanks and lobbyist are for lawmakers.

"No one got fired for buying ibm" for the new era.

There is a lot of truth in AWS propaganda, they're great for many things. But some of it is built on lies, cost being one, performance another.

rglover• 9 months ago

I really dislike how this industry oscillates between various states of epiphany that things that are overcomplicated and expensive are overcomplicated and expensive. As an industry, we must look like utter clowns to the world. It's really sad that saying "own or control your own servers" seems to be a sword in the stone moment for far more people than it should. Things that used to be a "duh" are now a "wow" and it's deeply unsettling to watch.

dimitrios1• 9 months ago

One thing I can say definitively, as someone who is definitely not an AI zealot (more of an AI pragmatist): GPT language models have reduced the barrier of running your own bare metal server. AWS salesfolk have long often used the boogeyman of the costs (opportunity, actual, maintenance) of running your own server as the reason you should pick AWS (not realizing you are trading one set of boogeymen for another), but AI has reduced a lot of that burden.

sema4hacker• 9 months ago

Anycast, Argo Rollouts, Aurora Serverless, AWS, BGP, Ceph, ClickHouse, Cloudflare, CloudFront, DWDM, Flux, Frankfurt, Glacier, Helm, Kinesis, Kubernetes, Metabase, MicroK8s, NVMe, OneUptime, OpenTelemetry Collector, Paris, Postgres, Posthog, PXE, Redis, Step Functions, Supermicro, Talos, Terraform, Tinkerbell, VM's.

I wish you started out by telling me how many customers you have to serve, how many transactions they generate, how much I/O there is.

rossdavidh3• 9 months ago

I had a problem figuring out why the place I was working wanted to move from in-house to AWS; their workload was easily handled by a few servers, they had no big bursts of traffic, and they didn't need any of the specialized features of AWS.

Eventually, I realized that it was because the devs wanted to put "AWS" on their resumes. I wondered how long it would take management to catch on that they were being used as a place to spruce up your resume before moving on to catch bigger fish.

But not long after, I realized that the management was doing the same thing. "Led a team migration to AWS" looked good on their resume, also, and they also intended to move on/up. Shortly after I left, the place got bought and the building it was in is empty now.

I wonder, now that Amazon is having layoffs and Big Tech generally is not as many people's target employer, will "migrated off of AWS to in-house servers" be what devs (and management) want on their resume?

whstl15• 9 months ago

Devs wanting to put AWS on their resume push for it, then the next wave you hire only knows AWS.

And then discussions on how to move forward are held between people that only know AWS and people who want to use other stuff, but only one side is transparent about it.

ahel• 9 months ago

with "dev wanting X" nothing happens. "leadership deciding X" then it needs to get done.

rossdavidh3• 9 months ago

That was what I thought, but when the middle management also wants it, then it can become the 'obvious choice', a la 'nobody ever got fired for choosing IBM'. It seems that middle management + devs can make it seem inevitable to the people above them, especially if those people are non-tech.

hedora9• 9 months ago

Reason to use AWS from the article:

You do not have the appetite to build a platform team comfortable with Kubernetes, Ceph, observability, and incident response.

Has work been using AWS wrong? Other than Ceph, all those things add up to onerous half time jobs for rotating software engineers.

Before gp3 came out, working around EBS price/performance terribleness was also on the list.

electroly• 9 months ago

I put our company onto a hybrid AWS-colocation setup to attempt to get the best of both worlds. We have cheap fiddly/bursty things and expensive stable things and nothing in between. Obviously, put the fiddly/bursty things in AWS and put the stable things in colocation. Direct Connect keeps latency and egress costs down; we are 1 millisecond away from us-east-1 and for egress we pay 2¢/GB instead of the regular 9¢/GB. The database is on the colo side so database-to-AWS reads are all free ingress instead of egress, and database-to-server traffic on the colo side doesn't transit to AWS at all. The savings on the HA pair of SQL Server instances is shocking and pays for the entire colo setup, and then some. I'm surprised hybrids are not more common. We are able to manage it with our existing (small) staff, and in absolute terms we don't spend much time on it--that was the point of putting the fiddly stuff in AWS.

The biggest downside I see? We had to sign a 3 year contract with the colocation facility up front, and any time we want to change something they want a new commitment. On AWS you don't commit to spending until after you've got it working, and even then it's your choice.

jcalvinowens2• 9 months ago

I have seen multiple startups paying thousands of dollars a month in AWS bills to run a tiny service which could trivially run on an $800 desktop on a residential internet connection. It's absolutely tragic.

hedora9• 9 months ago

That’s like $24K a year. Assuming they have working failover and business continuity plans, it’s actually a really good deal (vs having a 10-20% time employee deal with it).

whstl15• 9 months ago

AWS doesn't get magically expensive just because you put your website there.

You don't get to an overcomplicated AWS madness without having a few engineers already pushing complexity.

And an overcomplicated setup also means it needs maintenance. There are no personnel savings there.

kaydub2• 9 months ago

Then they architected and built it out wrong.

It's pretty simple to run low cost services in AWS. If you're small enough that an $800 desktop on a home internet connection will handle it, surely you could run it completely serverless for much less.

I'm surprised how many people I see wanting to go on-prem vs AWS/public cloud. Feels penny smart and pound foolish to me. Lots of people too deep into the technical side of things that they don't even understand the business side.

winrid2• 9 months ago

Hard to say. A $800 desktop could easily equal the compute of $100k/mo in lambda costs.

jcalvinowens2• 9 months ago

You vastly underestimate what you can do on prem. Equivalent elastic compute for a $800 desktop that uses $100/mo in electricity is >$2000/mo. As sibling points out, it could easily be much much more depending on what you're doing.

Bandwidth is usually the killer with AWS. Bandwidth on prem is free. Some cloud vendors are much better, in fairness.

debarshri• 9 months ago

Recently i learned that orgs these days want to show software and infrastructure spend as capex as they can shown it as depreciating asset for tax purposes.

I understand that with AWS you cannot do that as it is often seem as opex.

I guess thats a good enough motivation to move out of AWS at scale.

kyledrake• 9 months ago

The article mentions Equinix Metal but if you look it up they are shutting down the service https://docs.equinix.com/metal/hardware/standard-servers

Doesn't make me want to be a Equinix customer when they just randomly shut down critical hosting services.

I'm pretty sure that it's just the post-merger name for Packet which was an incredible provider that even had BYO IP with an anycast community. Really a shame that it went away, it was a solid alternative to both AWS and bare metal and prices were pretty good.

There's a missing middle between ultra expensive/weird cloud and cheap junk servers that I would really love to see get filled.

dilyevsky3• 9 months ago

Fwiw equinix metal was an acquisition (Packet). Seems like it didnt go too well

jameson• 9 months ago

Curious to know how's the development experience been post-migration? Was there additional friction due to lack of tooling in on-prem that would otherwise available in the cloud env for example?

carlgreene• 9 months ago

Ok so this may be a dumb question...but now do you handle ISP outages due to storms and stuff with on prem solutions? I'd imagine large datacenters have much more sophisticated and reliable internet connections than say an Xfinity business customer, but maybe that's wrong.

neuronflux• 9 months ago

Much more sophisticated and reliable than Xfinity.

Good datacenters have redundant and physically separated power and communication from different providers.

Also, in case something catastrophic happens at one datacenter, the author mentions they are peered to another datacenter in a different country, as another layer of redundancy. Cloudflare handles their ingress, so such a catastrophic event wouldn't likely to be noticed by their customers.

flufluflufluffy• 9 months ago

Right! I can’t believe they decided to ditch the OS entirely and maintained availability like that!

yearolinuxdsktp2• 9 months ago

Running EKS on AWS was their problem. If they didn't run EKS on AWS, they would've had a considerably simpler setup running Amazon Linux, not having to upgrade Kubernetes every 3 quarters, managing network security using security groups instead of having open internal networking, and running in a single AZ would've eliminated intra-AZ costs. In large data centers like us-east-1, an individual AZ is actually internally striped for extra redundancy, and you are much more likely to experience regional downtime than single AZ downtime, especially if you have a stable workload and do not rely on tech beyond rock-solid basics (EC2, VPC, ELB, S3, EBS). If you're willing to operate a single bare metal rack in a DC, you should be willing to run in a single AWS AZ.

I don't know how much time they spend configuring/dealing with Kubernetes, but I bet it's a large chunk of the 24 hour engineer-hours per quarter. But this is not a required expense: "EKS had an extra $1,260/month control-plane fee". Running EKS adds a massive IAM policy maintenance overhead, whereas a non-EKS (EC2 w/ golden AMIs) setup results in drastically simpler IAM policies.

NAT gateways are ~$50 a month, plus data transfer. Setting up a gateway VPC endpoint to S3 will avoid having to pay transfer charges to S3.

They were at 90% reservation capacity, so they should be using reservations for greater savings and in fact, running stable workloads with reservations is something that AWS excels at. Reservation means that you will be able to terminate and re-launch instances even when there's a spike in demand from other users--your instance capacity is guaranteed.

Running the basics on VMs also effectively avoids vendor lock-in. Every cloud provider supports VMs with a RedHat clone, VPCs, load balancing, networked storage, access controls, object storage and a fixed size fleet with auto-relaunch on instance failure.

With a consistent workload, they would have very likely escaped the downtime from AWS a week ago as well, because, as per AWS, "existing EC2 instances that had been launched prior to the start of the event remained healthy and did not experience any impact for the duration of the event".

With Terraform and automation for building launchable images, you can stand up a cluster quickly in any region with secure networking, including in a separate AWS account, in the same region, for the sake of testing.

With AWS, you can set up automatic EBS backups of all your data to snapshots trivially, and even send them to a 3rd locked-down account, so they can't be accidentally wiped.

ZebusJesus• 9 months ago

Thank you for the share this is really good information for making expensive decisions!

nemothekid• 9 months ago

We now save over $1.2M / yr and we expect this to grow, as we grow as a business.

Am I just naive? How is a uptime SaaS product saving over a million year on managed colo vs AWS? Was every API route in it's own EC2 instance?

AWS is expensive sure, but over a million dollars a year? For this product specifically?.

I got some clarification from their earlier posts and it looks like they were intentionally avoiding any AWS platform features:

Our goal was to avoid reliance on AWS or any proprietary cloud technology.
When we were utilizing AWS, our setup consisted of a 28-node managed Kubernetes cluster. Each of these nodes was an m7a EC2 instance. With block storage and network fees included, our monthly bills amounted to $38,000+. This brought our annual expenditure to over $456,000+.

I just think if you are going to deploy on AWS, then treat it AWS like managed-colo, then your bill is going to be high. I understand how that seems unfair, but AWS isn't really in the business of selling virtual machines. If you sit down and ask yourself how you got here, it just seems like you committed yourself to wasting money. If I knew I just needed some linux boxes from the start, there are better choices than AWS.

nostrebored7• 9 months ago

In almost 100% of cases I've seen this, people are convinced that they are going to go for a multi-cloud situation and using any value added service is "lock-in". The amount of times I've seen the migration scenario play out favorably for people has been absolutely ZERO.

Meanwhile, I've seen huge companies successfully complete cloud->cloud migrations in less than a year, as long as they use the value added services of the other cloud.

pjdesno• 9 months ago

I'm involved in a fairly large academic cloud deployment, sited in a 15MW data center built and shared by a few large universities.

There are huge advantages of scale to computer operations in a few areas:

- facility: the capital and running cost of a purpose-built datacenter is far cheaper per rack than putting machines in existing office-class buildings, as long as it's a reasonable size - ours is ~1000 racks, but you might get decent scale at a quarter of that. (also one fat network pipe instead of a bunch of slow ones)

- purchasing: unlike consumer PCs, low-volume prices for major vendor servers are wildly inflated, and you don't get decent prices until you buy quite a few of them.

- operations: people come in integer units, and (assuming your salary ranges are bounded) are only competent in small number of technical areas each. Whether you have one machine or 1000s you need someone who can handle each technology your deployment depends on, from Kubernetes to network ops; multiply 4x for those requiring 24/7 coverage, or accept long response times for off-hours failures.

That last one is probably the kicker. To keep salary costs below 50% of your total, assuming US pay rates and 5-year depreciation since machines aren't getting faster as quickly as they used to, you probably need to be running tens of millions of dollars in hardware.

Note that a tiny deployment of a few machines in a tech company is an exception, since you have existing technical staff who can run them in their spare time. (and you have other interesting work for them to do, so recruiting and retention isn't the same problem as if their only job was to babysit a micro-deployment)

That's why it can be simultaneously true that (a) profit margins on AWS-like services are very high, and (b) AWS is cheaper than running your own machines for a large number of companies.

kshacker• 9 months ago

the capital and running cost of a purpose-built datacenter is far cheaper per rack than putting machines in existing office-class buildings, as long as it's a reasonable size - ours is ~1000 racks, but you might get decent scale at a quarter of that.

Just want to confirm what I am reading. You are talking about ~1000 racks as the facility size, not what a typical university requires.

Frannky• 9 months ago

I only use bare metal—super cheap and very easy to switch. No worries about crazy bills or handling the crazy complexity of their systems. So far, so good. When/if problems start, I'll try them

bhewes• 9 months ago

Yes to this keep core base load in your own bare metal systems, use the clouds for what they do best.

unixhero• 9 months ago

I went bare metal too. Not because of AWS, but because of being frozen out by Hetzner because of a debt of 0.02eur with no way of paying it.

StratusBen• 9 months ago

Co-Founder and CEO of https://vantage.sh/ here - I've been pretty impressed by the rate that repatriation is happening off of public cloud. It rarely ever came up and in the last year it's been popping up more and more -- and especially just for getting access to GPU workloads.

I thought there would be a greater unbundling to AWS or to cheaper providers but it seems like a good-sized portion of the market is just going back to managing their own hardware.

Naklin• 9 months ago

We spent a week of engineers time (and that is the worst case estimate) on the initial migration, spread across SRE, platform, and database owners.

I’m sorry but I don’t believe this for one second.

And unfortunately that makes me distrust the entirety of the article.

jiggawatts• 9 months ago

Something I've noticed about all public clouds is that they promise a unique value proposition, and then entirely fail to deliver it for almost all customers.

Specifically, they promise to provide a "small, rapidly growable slice of a very big thing". I.e.: You can create an empty S3 account for $0.00, fill it with a few megabytes of initial data for $0.00001, and then if you suddenly need petabytes then it scales smoothly and beautifully up past any reasonable scale. You get billed for that at an exorbitant rate, but the point is that you can do it without having to rearchitect anything.

"This Works Great(tm)" for three specific categories of customers:

- Tiny organisations that expect to grow big suddenly: Startups, and maybe the few orgs that have rare or annual events and nothing in between. Think electronic voting systems and the like.

- Small international companies that need global presence on the cheap. SaaS vendors, IoT, and a few niche organisations are pretty much the only ones in this category.

- Enormous organisations that need large scale but can't be bothered (or can't afford) managing that. There was an AWS talk about a customer that needed about 1 PB of storage which these days is "just" 500 x 20 TB disks, but they needed burst IOPS far in excess of that, on the order of 100,000 disks. The "thin slice" model of the cloud works great for this, because S3 has millions of disks behind it, and each customers' data is spread out over those disks.

Everybody else in the "medium" category is sold a bunch of bullshit.

Cloud VMs are between 5x and 10x as expensive as the on-prem equivalents, all costs factored in. I've seen the numbers from CIOs and CTOs, they all got told "the cloud is cheaper", and their costs skyrocketed as soon as they went to the cloud. You still need engineers. You still need "deployments". You still need sysops. You still need to update your VMs and their software somehow. Nothing changes really, except suddenly VMware starts looking cheap in comparison.

The "You can scale, the cloud is flexible, you can..." marketing is a load of bollocks.

First, Pay-as-you-Go pricing is on average 7x on-prem VM pricing. The only way to bring this down to merely 3x the on-prem cost is to LOCK IN the compute using "reservations" of some sort, typically for 3-year periods. This is NOT FLEXIBLE BY DEFINITION!

The whole marketing of the cloud revolves around the flexibility, but all of their pricing and cost optimisation revolves around getting customers to lock in spending for years and years.

Similarly, Spot-priced anything is so unreliable that it is completely unusable for almost all "enterprise" customers, even for non-production use.

Seriously, can you imagine telling a developer that costs $200/hour that they can't do their work because their DEV instance is gone for a day because some other tenant needed it more!?

This underlying issue with VM pricing means that any service that is built on top of VMs inherits the same pricing model with lock-in contracts, totally negating the scalability benefits and global presence of the public cloud. If you want S3 in every region with 10 MB each... that's cheap. If you want just one VM in every region... no longer cheap. If you want any VM-based service in every region... also not cheap. Oh... you wanted DEV/TST/UAT/GREEN/BLUE? With high availability? Get the CFO on the line, he'll need to approve your budget!

"Just engineer your software to be cloud native!" is what you inevitably hear from apologists. Sure, sure... I'll get right on that. I mean sure, the public cloud vendors failed to do so for like two thirds of their own first-party products, but I'm sure I'll have better luck! Let's see.. my "medium sized org" has... checks notes... about 1,000 unique pieces of software deployed on VMs, of which 800 are CotS vendor products, 600 of which require Windows Server and have a GUI configurator. This will go... smooth.

For 90% of the potential customers out there, the big businesses, the enterprises, the universities, governments, and the like... it's just a more expensive data centre that someone else runs for them.

The only significant advantages to public cloud vs on-prem I've seen are:

- Faster networking, with a well-engineered 100 Gbps or 200 Gbps core in most clouds. This includes Internet uplinks of similar spec. These are rare in private hosting.

- Three-way zone redundancy instead of the typical two-way.

- Zone redundant services that are just a "checkbox".

- Separation of duties where the layer 2 network and hypervisor are managed by a vendor instead of internal staff. (This can be difficult to arrange internally for medium sized orgs needing high security, there aren't enough IT admin staff for true separation.)

... that's about it.

twodave3• 9 months ago

I feel like it just depends on what you’re trying to do.

* data driven website with some internal api integration, maybe some client-side application or tooling? Put a server rack in a closet and get a fiber line.

* trying to serve the general public in a bursty, not-cacheable way? Probably going to have to carry a lot of machines that usually don’t do much, cloud might make more sense

* lots of ingress or DNS rules? A hybrid approach could make sense.

In general once you start thinking about scaling data to larger capacities is when you start considering the cloud, because just the storage solution ends up on a long amortization schedule if you need it to be resilient, let alone the servers you’re racking to drive the DB.

layoric• 9 months ago

In general once you start thinking about scaling data to larger capacities is when you start considering the cloud

What kind of capacities as a rule of thumb would you use? You can fit an awful lot of storage and compute on a single rack, and the cost for large DBs on AWS and others is extremely high, so savings are larger as well.

twodave3• 9 months ago

Well, if you want proper DR you really need an off-site backup, disk failover/recovery, etc. And if you don’t want to manually be maintaining individual drives then you’re looking at one of the big, expensive storage solutions with enterprise grade hardware, and those will easily cost some large multiple more than whatever 2U db server you end up putting in front of it.

nodesocket• 9 months ago

“ $600/month for NAT gateways”

I build my own NAT instances from Debian Trixie with Packer. The configuration is literally a few lines:

    sudo iptables -t nat -A POSTROUTING -o ens5 -j MASQUERADE
    sudo iptables -F FORWARD
    sudo iptables -A FORWARD -i ens5 -m state --state RELATED,ESTABLISHED -j ACCEPT
    sudo iptables -A FORWARD -o ens5 -j ACCEPT

    sudo iptables-save | sudo tee /etc/iptables/rules.v4 > /dev/null

prabhatjha• 9 months ago

This writeup is very informative. Learned about few OSS frameworks that I was not aware of. Amazing engineering work. Kudos to you all.

suralind• 9 months ago

Love this write up. But also love the fact that the company pretty much uses the best solutions that they can (that definitely helps with hosting, regardless of cloud/self-host, but is impressive anyway). I’ve been a big Talos fan for couple years, but never worked with a client that would use something different than managed offering from their cloud provider.

Similarly, the fact that they are regularly updating Kubernetes, Talos and presumably other things means that these things are a normal flow and thus can optimize and automate. If you’re doing that because AWS is forcing you to upgrade… then it’s stressful.

stack_framer• 9 months ago

My question is: How did you get your entire team on board with this decision? My team disagrees on even the most trivial technicalities, so I can't imagine doing something on the scale you're doing (even though I wish we could leave AWS for our own hardware).

znpy• 9 months ago

37% off instances still leaves you paying list price for bandwidth, which was 22% of our AWS bill.

This is the most infuriating pain point when using AWS.

We have premium support at work and the engineers basically read the documentation back to us, ignoring our complaints about cross-az bandwidth and the utter and complete lack of az-awareness in their services.

Example: we had some redises that we wanted to migrate to ElastiCache... All the aws engineers did was reading back the marketing pages and pushing for the serverless offering (where bandwidth was the main cost driver). The reader/writer endpoints are totally az-unaware.

Cross-AZ bandwidth for ElastiCache was costing us MORE than ElastiCache itself.

In the end we had to engineer AROUND ElastiCache in order to get it working, and working with predictable pricing.

"Invent and simplify" my ass...

insaneisnotfree• 8 months ago

“You lean heavily on managed services (Aurora Serverless, Kinesis, Step Functions) where the operational load is the value prop.”

Not viable even when your core belongs to AWS. Why? Ask prime video

AWS to bare metal two years later: Answering your questions about leaving AWS

Related Stories