Half of Organizations Spend More with Kubernetes, Mostly Due to Overprovisioning

https://www.infoq.com/news/2024/03/cncf-finops-kubernetes-overspend/

by rbanffy • 2 years ago

43 24 2 years ago

rapind• 2 years ago

I would wager that more than half of all Kubernetes deployments are completely unnecessary yak shaves.

acdha6• 2 years ago

I think there is a solid argument for it as a cloud portability layer – I’m literally in the middle of a GKE->EKS migration where that’s paying off as hoped - but I would fascinated to see the numbers for what percentage of places ever do that or the total cost incurred avoiding services which couldn’t be used interchangeably between providers. I think people want ORM-level portability but it’s more like how you can technically have a Java desktop app support Windows, macOS, and Linux.

jpgvm2• 2 years ago

The portability of your stack is one aspect but there are other more important factors:

1. The tools you want to use work with the other k8s components or things commonly available in k8s environments. Say you want a cloud-portable messaging system. You can deploy Apache Pulsar on k8s and if you use the operator, helm charts (eww) or templates it will integrate out of the box with Prometheus, cert-manager, whatever volume provisioner you are using etc. You can't get that level of cloud-agnostic deployability or integration with tools like Terraform/Pulumi because all the underlying infrastructure resources are different on a per-cloud basis. Having such a powerful generic API that properly describes like 99% of workloads is a massive game changer.

2. The skillset is transferable. If you are comparing k8s to something like ECS, Cloud Run or friends you will end up with a bunch of proprietary knowledge that is hard to transfer. Similarily your hires won't come with that knowledge either but in this day and age they probably have robust k8s skills if they are infra folk or are aware of it if above that part of the stack.

3. Flexibility. Perhaps it's greatest strength and greatest weakness is that the set of primitives chosen to be exposed by k8s (with the exception of things like Ingress that needed to get re-done) are by and large skewed towards flexibility. This means doing all manner of 'very difficult things' are now considerably easier. Conversely however this flexibility means you still need to fully describe more common patterns which simpletons mistake for being "over-complicated" when what they really mean is "less easy". Don't fall into the trap of equating "easy" to "simplicity", k8s once understood is by far one of the simplest but as a result is usually more work to get started. Good example is say Arch Linux vs MacOS, the former is simple the latter is easy.

acdha6• 2 years ago

I suspect a big chunk of this is simply visibility: if you deploy an application using your cloud provider’s native features, you get an itemized bill telling you where your money is going. That’s not perfect (you still need to tag resources) but it’s mature and familiar. If you run everything in Kubernetes - which, to head off a popular tangent, is not just following a fad but a trade off - you have to build and operate the equivalent infrastructure to have the same level of visibility, but since the entire Kubernetes cluster is overhead it’s very easy to get stuck in the IT shared services pitfall where there’s a constant pressure to direct limited resources towards application-visible work.

The survey’s reported breakdown feels right – you need solid leadership to resist the urge to skimp on indirect infrastructure costs and I’d bet most of the 80% know this is a problem but have been too busy to spend time on it:

just under a fifth (19%) had access to accurate information, while 40% said they estimated their Kubernetes costs. There was no monitoring in place for 38%.

ncrmro• 2 years ago

kube-prometheus-stack will setup Prometheus and grafana and once you get the hang of it makes sizing resources a bit better

acdha6• 2 years ago

Sure, but that’s something new to install, configure, and maintain. My point is simply that a lot of places struggle to fund those kind of internal services because spending time on them doesn’t directly produce application-level benefits.

Traubenfuchs2• 2 years ago

I wonder if there are still companies out there running tomcat 10 clusters on their dedicated servers and furiously laughing at the depressing state of the software engineering world.

rixed2• 2 years ago

"Can't be blamed for using IBM" companies are very likely to be using Kubernetes these days, though.

Traubenfuchs2• 2 years ago

Even in pure java shops, you‘d definitely get laughed out of most rooms for proposing a tomcat cluster. Majority of java devs nowadays probably never even experienced working with it directly. There is little knowhow of it left.

Everything must be dockerized and dockerized stuff must run in the cloud.

ckdarby• 2 years ago

Ugh, these are always set up to be nothing more than click bait. They're always surface level and never actually gather real financials. Most of the folks who fill out the information don't even have access to their P&L to be fully educated on the spend.

For most folks here on HN, you should not care about your cloud opex cost as long as it is time benefiting. The most costly thing to waste is time.

Can you cut your cloud bill by 35% with optimal provisioning? Most likely. Can you do this cut with a greater return given all the inputs, time, wages, opportunity cost, AND what level of confidence in completion to that exact or less deadline?

The only times I've seen the numbers work out has been, growth is flattening, +$3M/yr vendor spend or $45k/month specific service spend.

hedora• 2 years ago

Agreed about opex while the business is ramping.

Kubernetes has universally been a productivity killer in the organizations I’ve seen deploy it. It’s one of the bigger threats to the company at my current gig.

Maybe it’s just stuck in the trough of disillusionment this year, but I’d love to see a non-terrible replacement for it take over.

inhumantsar3• 2 years ago

what would you say are the biggest sources of pain? k8s is definitely a beast, and I'm curious what parts in particular that drag down productivity for you.

llama0522• 2 years ago

It's been the opposite for us. No more waiting for VMs to get spun up and it actually enforces a consistent layer across infrastructure.

I'd be curious what the pain you've seen is.

rixed2• 2 years ago

The Hashicorp suite (Nomad as the orchestrator) has the reputation to be simpler and more capable of scaling but I've never tried it yet. Better hurry since the industry has apparently decided to flock around kubernetes.

bradybd• 2 years ago

"Respondents shared that they see team and individual awareness and self-discipline as the best ways to help get the overspend under control (68%)"

Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks. Think this will naturally shift down as larger orgs make switch to small, 8-12 person autonomous teams.

inhumantsar3• 2 years ago

autonomy will absolutely lead to bloat if there isn't an accountability mechanism in place.

at my last job as head of eng at a small (15-20 devs), I pushed teams to build their dashboards with business metrics in addition to the usual operational metrics. things like costs, customer frustration signals and support ticket volumes, product metrics, etc would show up there.

integrating these initially is non-trivial, but once the data is flowing, it's easy for teams to pick them up. gives a good sense of ownership, helps teams address these proactively (or build a business case for addressing them), and gives the team+stakeholders a common dataset so everyone has the same complete picture to work from.

acdha6• 2 years ago

Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks

I’ve found the opposite to be true unless autonomy is paired with accountability. Ultimately the people making decisions need to pay the costs for their choices and I’ve seen that be especially bad where someone had the autonomy to spin up a ton of resources which someone else had to pay for. This can be especially bad if that was resume-driven development where they were playing cloud service bingo trying to list experience with things they really couldn’t have justified based on the application’s actual usage.

llama0522• 2 years ago

I wouldn't say this is a problem specific to kubernetes.

Most places over-provision their virtual machines also, if not more so due to the less dynamic state of the lower layers.

Half of Organizations Spend More with Kubernetes, Mostly Due to Overprovisioning

Related Stories