HN user
As a maintainer: some issues take longer to triage than others. Especially if they are not CRITICAL, and there's a huge holiday season in the midst of it. :)
I know I have been involved in a couple which took time to agree on a "best" solution and to find people to tackle them.
There's a perception that this is true, and like all perceptions it is based in reality, but it is not the reality itself.
We have plenty of data that shows that people DO get promoted at all levels based on OSS work.
Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes.
Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project?
Of course it is never ENOUGH. I'd happily consume hundreds more people. :)
This was, literally, one of the arguments for building and releasing kubernetes. The rise of Hadoop made it much harder to justify MapReduce being different.
If we just talked about Borg, but didn't ship code, someone else might have set the agenda, rather than Kube.
(Building on my own tweets)
Autopilot and Fargate are VERY different solutions to similar problem statements.
Autopilot puts compatibility & transparency at the front. It IS GKE. It is integrated in all the same ways GKE is integrated. There's no black box between you and kubernetes, but you are absolved of the need to manage nodes, which most people REALLY don't want to care about.
Like it or not, nodes are part of the k8s API in many ways. Rather than swim upstream against that, I think Autopilot strikes a very good balance.
Since this started by citing me, I feel somewhat obligated to defend my guidance.
I stand by it.
In an ideal world where apps are totally regular and load is equally balanced and every request is equally expensive and libraries don't spawn threads, sure. Maybe it's fine to use limits. My experience, on the other hand, says that most apps are NOT regular, load-balancers sometimes don't, and the real costs of queries are often unpredictable.
This is not to say that everyone should set their limits to `1m` and cross their fingers.
If you want to do it scientifically:
Benchmark your app under a load that represents the high end of reality. If you are preparing for BFCM, triple that.
For these benchmarks, set CPU request = limit.
Measure the critical indicators. Vary the CPU request (and limit) up or down until the indicators are where you want them (e.g. p95 latency < 100ms).
If you provision too much CPU you will waste it. Maybe nobody cares about p95 @50ms vs @100ms. If you provision too little CPU, you won't meet your SLO under load.
Now you can ask: How much do I trust that benchmark? The truth is that accurate benchmarking is DAMN hard. However hard you think it is, it's way harder than that. Even within Google we only have a few apps that we REALLY trust the benchmarks on.
This is where I say to remove (or boost) the CPU limit. It's not going to change the scheduling or feasibility. If you don't use it, it doesn't cost you anything. If you DO you use it it was either idle or you stole it from someone else who was borrowing it anyway.
When you take that unexpected spike - some query-of-doom or handling more load than expected or ... whatever - one of two things happens. Either you have extra CPU you can use, or you don't. When you set CPU limits you remove one of those options.
As for HPA and VPA - sure, great use them. We use that a LOT inside Google. But those don't act instantly - certainly not on the timescale of seconds. Why do you want a "brick-wall" at the end of your runway?
What's the flip-side of this? Well, if you are wildly off in your request, or if you don't re-run your benchmarks periodically, you can come to depend on the "extra". One day that extra won't be there, and your SLOs will be demolished.
Lastly, if you are REALLY sophisticated, you can collect stats and build a model of how much CPU is "idle" at any given time, on average. That's paid-for and not-used. You can statistically over-commit your machines by lowering requests, packing a bit more work onto the node, and relying on your stats to maintain your SLO. This works best when your various workloads are very un-correlated :)
TL;DR burstable CPU is a safety net. It has risks and requires some discipline to use properly, but for most users (even at Google) it is better than the alternative. But don't take it for granted!
Free (zonal) cluster per account, regardless of size, should cover a lot of this, no?
Try KIND. Much better devex.
This is a fair point. We don't have an HA (multi-master) zonal offering either, because mostly people don't want that.
EKS has always had a fee.
AKS, well, I don't have any insight into their business, but I have my suspicions.
That is not absolute truth. If it were you would eschew kubernetes altogether and just use VMs.
Everything is a tradeoff. If you want total isolation, you pay for it. If you don't want to pay for it, you make more value-based tradeoffs.
Concretely, Google runs "a handful" of "pretty reliable" services on a relatively small number of clusters.
Those empty clusters that you get for free cost Google money. Perhaps it never should have been free, because that skewed incentives towards models like this.
I don't think that is at all a fair characterization, you just don't have the same data available to you.
Thanks for the props. It means a lot to me personally.
Many small clusters just do not deliver on a lot of the value of Kubernetes. Clusters are still hard boundaries to cross (working to fix that). Utilization and efficiency are capped. OpEx goes up quickly.
There are reasons to have multiple clusters, but I think the current trend takes that too far.
TO BE SURE - there's more work to do in k8s and in GKE.
I understand the emotional response here, but I don't think it's rational. GKE has to work as a business, or else the whole thing is in trouble.
I think GKE provides tons of value, but people tend to under-estmate that. In order to keep providing that value, we need to make sure it is sustainable.
I'm really, truly sad that you perceive it as bait-and-switch, but I disagree with that characterization. If you want to move off GKE, I'll go out of my way to help you, but I urge you to take a big-picture look at the TCO.
Google and Google Cloud are largely different businesses, though I understand it's hard to keep that in mind in the context of things like this.
I encourage everyone to always stay nimble and keep your eyes on portability. I also encourage you to try to assess the REAL costs of doing things yourself. It's rarely as cheap as you think it is.
As a Kubernetes maintainer, I am fanatical about portability.
As part of the GKE team, I think we provide tremendous value that people tend to under-estimate.
NOTE: I was NOT involved in this decision, but I understand it, and I want to help other people understand it.
If the main value of GKE over DIY is $73, you should totally DIY.
I mostly try not to be too Google-focused here, but I have to say...
I'm pretty proud of GKE, and I think it offers a lot of value other than just being cheap. Managing clusters is not always easy. GKE handles all of that for you - including integrations, qualifications, upgrades, and patching clusters transparently BEFORE public security disclosures happen.
We have a large team of people who deal with making GKE the industry-leading Kubernetes experience that it is. They are on-call and active in every stage of the GKE product lifecycle, adding value that you maybe can't see every day, but I promise you is there. When things go sideways, there isn't a better team on the planet to field the tickets.
I don't understand the anger here - you're literally saying you'd rather pay more for a service of lower quality because... why? Because they will continue to charge you more? Does not compute.
For those people who use a large numbers of small clusters, I understand this may make you reconsider how you operate. As a Kubernetes maintainer, I WANT to say that a smaller number of larger clusters is generally a better answer. I know it's not always true, but I want to help make it true. GKE goes beyond pure k8s here, too. Things like NodePools and sandboxes give you even more robust control
GKE is the best managed Kubernetes you can get. And we're always making it better. Those clusters actually DO have overhead for Google, and as we make GKE better, that overhead tends to go up. As someone NOT involved in this decision, it seems reasonable to me that things which are genuinely valuable have a price.
Also, keep in mind that a single (zonal) cluster is free, which covers a notable fraction of people using GKE.
I think this trend is not good overall, and people will eventually be very unhappy with it. I'd rather help you figure out how to use fewer clusters.
For folks just trying it out, 1 cluster is still free.
LOL. Thanks for the shout out. I am sorry you had a bad experience. Please be assured that we are working on it, and that we take this seriously.
Why did the people who created Kubernetes all leave Google?
Umm, we didn't?
created omega but it never did replace borg
It did have material impact on Borg.
Sometimes the strategy is simply "we know how to do this, and we'd prefer not to see you go through all the same pain to figure it out".
Probably not, in truth. It's a great idea and not just for this code, but lots of tricky stuff.
As one of the authors of the original code here, this was the result of several days of intense works by a half dozen people working through every corner case we could dream up, and a bunch we thought of on the spot.
It is in no way a guarantee that we got them all, but after spending so much time reason in through why those 'else' clauses were correctly empty, we thought it would be rude not to write it down.
In truth it was as much for future-me as anyone. My memory is know. To be spotty. :)
I'm agreeing and saying that is part of the plan :)
Appreciate the feedback. From my end, I have been both accused of name-dropping (ooh, look at Mr special, he works at Google!) and of having an unstated agenda (hey, he's a phony, he works there!).
I found that simply stating is the compromise. I am not using my employer.ent to justify my opinion, just making my alignment clear.
And in fact, I align with Google AND with Kubernetes.
I just want to be transparent. Everyone has an agenda, I don't want to be perceived as "sneaky" (have been accused in the past, even changed my username)
We could also just link to Google's blog.
https://cloud.google.com/blog/products/gcp/google-cloud-gran...
That's better. I really see this as "Google opens administration of Kubernetes infra, donates $9M in credits to keep it running".