justinsb

I also started (non-trivially) contributing to OSS with OpenStack, and later switched to contributing to Kubernetes. I whole-heartedly agree with the points made here; these same observations really shaped how I chose to contribute to Kubernetes.

Trying to get all the AWS early adopters to abandon their investment and move to something else seemed like it was a huge barrier to adoption, so I started off contributing to the AWS support for kubernetes and making sure that kubernetes worked well there. But the pains of installing OpenStack from upstream were fresh in my mind, so I also contributed to make sure that kube-up worked well and then (as we outgrew that architecture) started the kOps project. My goal is that you should always be able to run OSS kubernetes from upstream, even if you just treat that as an insurance policy that gives you the confidence to use a managed service.

I think the miracle of kubernetes is that I was just one of a large number of people here, each of us bringing our past experiences to make kubernetes better in some way. And the community continues to grow with people each addressing their own painpoints, which does create a lot of churn/progress (depending on your perspective!) I'd say that the people that organized the community were the unsung heroes here, and I suspect they learned a lot from OpenStack as well.

we'd like the scheduler to be aware of which node has which image

The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-ap....

It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589

Maybe it could be read that way, but I think it's a good indication of the problem that a strict reading doesn't include that. The article advocates that you should be using Heroku or Vercel or Netlify or Fly as not doing so is an antipattern, but it then says that instead of k8s "Most organisations should consider some of the higher-level building blocks available via cloud providers", which seems contradictory.

I quite liked the actual message of the article (which I take as "be mindful of your technology choices") - but I think that it is really devalued by the less coherent but more click-baity arguments (e.g. "javascript is the only rational language choice")

Edit: Actually, re-reading the article yet again, I think I can see your reading (though I wish you had written the article, as your writing is much clearer). As I understand it, the article says you should use a PaaS-style solution until you outgrow it, when you should move to something from the cloud providers. For me, that would be CloudRun -> GKE-Autopilot -> GKE (skipping CloudRun if you don't want to learn two systems), but there may be some google-bias there!

The problem with this sort of argument is that it sets up a straw-man, without describing what you should do instead. Kubernetes could be better - of course - but what better approach is the author recommending to use instead? It would be better if we could use one language on frontend and backend - of course - but what is this one language that works everywhere? It would be better - of course - if there was a SaaS that could do everything for us, is cheap and is open source - but what is that SaaS?

It's very easy to say something could be improved, but much harder to propose something to replace it that doesn't have its own shortcomings.

It's a pity because this article actually makes a good point, that we should think about what the actual user/business goal is before making technical decisions, and be careful lest we spend all our time on technology. I've heard this idea described as "innovation tokens". It's disappointing that the article chose to wrap this idea in clickbait, but I guess we wouldn't be talking about it otherwise!

Disclaimer: I have no end of personal biases in favor of kubernetes!

The only nuclear plant under construction in the US is at Plant Vogtle, in Georgia. Regulators set up a system (CWIP) whereby the companies building the plant earn a 10% return on their costs, until the plants come online. I think it's not surprising therefore that costs keep increasing and the delays keep coming. I don't think we can infer that nuclear power plants cannot be built at reasonable cost, rather that we need to consider "regulatory capture" as a significant construction risk.

(Some admittedly one-sided background on CWIP: https://stopcwip.com/ )

Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters.

The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts!

We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.

Yes, the reality is probably somewhere in between. We've heard that users like the idea that kops releases are more ready, where the .0 releases of k8s might have more issues particularly with ecosystem components. So we do lag a little behind - ideally one release.

1.12 has been a particularly tricky release getting everyone from etcd2 -> etcd3, but we've finally turned the corner on that one, so that should now let us catch up a bit.

Finally, we've also heard that users want more of a choice, so we're going to start doing e.g. 1.13-alpha and 1.14 alphas much sooner. We'll still wait to do kops 1.14.0 until everything is ready, but for users that want to run k8s 1.14 sooner, they will have an option that isn't building from source. And hopefully this also gets more people using pre-release versions of kops (in non-production environments) and also helps stabilize the releases more rapidly.

Oh sorry - yes that's me! Disclaimer: I work for Google on GKE & Kubernetes! I thought this was more of a community question and forgot...

Not the cause. The release team asked for KEPs for all new features in the next release and the deadline was today.

Yes, we're waiting till 1.11.6 lands - some important fixes for clouds in there. We've debated releasing at the same time as Kubernetes, but the feedback we've got is that users prefer that a kops release is ready to run, but we also need to do a better job of releasing alphas & betas earlier.

Obviously the current delay is not great - I'd much prefer for us to be on 1.12 (which actually has fixed the main issue already). I'm hoping we can catch up once we're (finally) done with the etcd3 migration.

The general page is at https://status.cloud.google.com/; you can scroll down to see GKE, and my (unofficial) belief is that https://status.cloud.google.com/incident/container-engine/18... should have closed out https://status.cloud.google.com/incident/container-engine/18...

_If_ that's the case, something else is causing the error messages other people are seeing

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy (https://cloud.google.com/terms/), whose intent is to give you some assurances.

(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

The incident with the UI (where we suggested using gcloud temporarily) was opened in https://status.cloud.google.com/incident/container-engine/18..., but then what sure looks to me like the same incident was closed in https://status.cloud.google.com/incident/container-engine/18....

My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now!

It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

This looks very well done - I've been hoping for something like this for a long time!

For code review (pull requests) there's also https://gerrit-review.googlesource.com/Documentation/note-db... which may be of interest.

I think the GPL3 license may be problematic for some potential contributors, but that's entirely up to you!

The article suggests that the difference is the water recirculation - that the "waste" water is cleaned rather than being removed. Is that how your farm's raceway worked?

I believe you forgot to divide by the $100

Which company is that from, and did you include the surcharges? I'm on Georgia Power on the R-22 residential tariff, and I pay about twice as much as the "headline rate", once I've added them all in: http://www.psc.state.ga.us/calc/electric/GPcalc.asp

Ah - sorry, probably should have disclaimered that! I did write the original kops code, but now there's a pretty active set of contributors working on kops (and contributions are always welcome and appreciated!)

kops 1.9 is very close to ready now, but this is a longer lag than normal. We've historically released kops 1.x when we consider that k8s 1.x is stable, including all the networking providers and ecosystem components. That's typically about a month after release.

User feedback has been that that we want to keep that, but that we should also offer an alpha/beta of 1.10 much sooner, so that users that want to try out 1.10 today can do so (and so we get feedback earlier). So watch for kops 1.10 alpha very soon, and 1.11 alpha much earlier in the 1.11 cycle.

Thanks for the note on the renewals (and the caveat!) - that does make it less concerning as it would only impact new services.

By "when moving domains" I mean that I'm enabling a production domain (project.com), where previously I was working on a development domain (project-dev.com). More recently, I was deploying gitlab and exhausted my quota simply because of mistakes I made along the way. This was simply my mistake, so I don't think it would be appropriate to request an official rate limit increase. If (say) a $100 donation to the EFF allowed me to specify a domain on which the rate-limit-counters would be reset to zero (as a one-off), I'd likely have done that - it's easier than changing the process of how I'm managing certificates.

I love LetsEncrypt - the only problem I've hit is the rate limits (in particular 20 new certificates per registered domain per week, when moving domains). I guess I could have avoided this by combining names into a single certificate, but I wonder why there's not an option to pay/donate to bump the limit temporarily: when I hit the limit I would gladly pay $50 or $100 instead of going through the manual process with a legacy CA. Even though wildcard certs will make this better, the fear of accidentally hitting a limit and being unable to renew certificates remains a concern for me.

Is any of this enabled yet (even if only in the dev channels)? Being able to run arbitrary containers locally would make a Pixelbook a lot more powerful.

I think it's a huge positive development that there's a reference implementation as well, in go. At least helpful for evaluation, possibly for a production implementation also.

Are there things that can cost-effectively be done in existing houses? That seems a good way to create demand for greater efficiency - demonstrate the value on a smaller scale first.

Ron was one of our investors in FathomDB, and that turned out to be a bad financial investment, much to my personal dismay & regret.

However, something that I think the essay modestly overlooks is the non-financial elements. The investors made a huge difference in my life & that of the others that worked for FathomDB. I like to think that we moved the industry forward a little bit in terms of thinking about modular services (vs a monolithic platform-as-a-service) although it turned out that the spoils went mostly to the cloud vendors. Many of the ideas developed live on in open-source today.

Of course, this all serves Ron's point in that it doesn't make for a good investment. But that doesn't mean that no good came of it - and it makes me want to work harder next time so that it is both a good outcome and a good investment.

So: thank you to Ron and all our investors. It is no accident that you are called angels.

We should update that issue. There is support for private route53 domains, which means you don't need a DNS domain name, and there is early experimental support for gossip discovery (i.e. no Route53 at all) - just create a cluster with a name of `<name>.k8s.local`.

100% agreed and looking forward to that - kops is a big part of that effort, actually. If you look under the hood you'll see that kops uses a lot of the kubernetes apimachinery. There's obviously a tricky chicken & egg situation with first creating a cluster specifically, but the hope is that a lot of the post-installation activities you do with kopscould be done through kubectl (e.g. adding groups of nodes of a different instance type) , talking to those same kops API objects on the k8s apiserver. The apimachinery team has been doing great work to enable this, that seems to have to fruition in k8s 1.7.

kops can set up an ELB for HA apiservers, but I think you're primarily talking about Ingress for your own services. We don't actually take an opinion on which Ingress controller you use, so you can just choose e.g. https://github.com/zalando-incubator/kube-ingress-aws-contro... when your cluster is up.

Maybe kops should use ALBs for the apiserver, and maybe k8s should support ALB for Services of Type=LoadBalancer. Neither of those are supported at the moment, if they should be then open an issue to discuss. (Even if you're going to contribute support yourself, which is always super-appreciated, it's best to start with an issue to discuss!)