The full-time job of keeping up with Kubernetes

https://gravitational.com/blog/kubernetes-release-cycle/#

by twakefield • 8 years ago

484 221 8 years ago

tw1010• 8 years ago

There aught to be a name to the tendency that as tools get better and better, the more your time goes from having your mind in technical-space to social and news-space. It's like the authority to create goes from the individual first-principles (by necessity) maker, to the control over development being in the hands of an external group, and then all your time is spent keeping up with what they're doing. A similar thing happened with a lot of javascript frameworks. It also happened with the transition from building servers from the ground up, to it all being managed by AWS.

lkrubner• 8 years ago

I wish I could give you more upvotes. You are describing a somewhat hidden psychology, which I think provides a rational basis for much of "Not Invented Here" psychology. We tend to think that "Not Invented Here" psychology is irrational, but in fact, the loss of control over possibly crucial technology is an important cost, which makes all of us stop and re-consider whether we really want to use some software developed by an external team.

jstimpfle13• 8 years ago

And it's not only that - Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding.

It's a sad state when otherwise very intelligent people think it's bad practice to use plain C and a clean OS API that has been stable for decades (because that was somehow "magic" und impossible to understand and error-prone), and advise you to use the Boost filesystem module just to concatenate two paths.

enraged_camel• 8 years ago

Sure, NIH syndrome can be rational for technology that is crucial/central to your system.

The problem is it is often used to justify re-inventing even mundane stuff. I once worked with a client who wanted to implement their own bug tracking system. The client’s main product was something totally unrelated.

cryptonector• 8 years ago

I don't think that's the whole picture. You could copy an interface (and even implementation) and fork it. But often with NIH we see a reinvention of a technology without even looking at alternatives; often this ends badly (in Linux land much more often than not it seems).

cottonseed2• 8 years ago

I also like to ask myself, "Do I want to become an expert at working with external software X, or do I want to become the kind of person who can build software like X?"

simonebrunozzi• 8 years ago

I gave him mine on your behalf :)

mooreds3• 8 years ago

Worth dropping a link to this Joel Spolsky article, where he discusses this concept and talks about the fact that the Excel team (in the 1990s) had their own c compiler:

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

Analemma_• 8 years ago

I think a lot of this is that once a tool reaches a certain level of size and complexity, it’s impossible to know all the technical details unless you’re actually a developer on the project (and after another threshold, not even then). So at that point you have to use news and social information to find out what’s going on, and just trust that people know what they’re doing.

It can be frustrating, but it’s also unavoidable. While I strive to know as much as possible about my tools, if I had to know every part of the stack backwards and forwards, I’d never get anything done. You do have to cede to the abstractions at some point.

hueving4• 8 years ago

The important distinction here is the pace. I can use a Ubuntu server LTS version and not have to stay on top of weekly development release announcements from upstream.

With k8s lacking LTS, it forces you to drink from the firehose.

drdaeman3• 8 years ago

It can be frustrating, but it’s also unavoidable.

Why unavoidable? It's not that one is forced to use some tech... Even though the hyped crowd chants the names so loudly.

A lot of things that complex technology offers are not really needed (not to mention one has to constantly fight with that new complex technology when it doesn't yet do something). And complex tech can be frequently replaced with a small set of tiny scripts and small and simple standalone services.

Domain and deployment-specific scripts and services that are unique, sure. But still easy to grasp mentally (and much smaller than any mature project's codebase).

papaf• 8 years ago

There aught to be a name ..

I think that "vendor lock-in" comes close to describing the situation. However, it does not describe the way the dependence changes the behaviour of the people locked in. I think that "addiction" fits in this case.

zapita• 8 years ago

That's not lock-in, it's just a dependency.

It would be lock-in if the vendor made it unreasonably difficult to remove the dependency.

StillBored2• 8 years ago

Some of this is caused by lack of backwards compatibility too. In the past once you picked a library you could be pretty sure that the API's you depend on wouldn't change much, and then only with a major release which would come with adequate documentation detailing the important changes.

These days the throw away and rewrite/refactor crowd is in such power that if you don't spend 1/2 your time tracking the commits your likely to find that your dependencies are entirely incompatible with whatever you have built on them and you have no idea how to fix that short of reading the last 6 months of mailing-list (whatever) postings just to change the "runlevel" or get your service to start...

ehnto• 8 years ago

The dependency trees in many projects are out of control too.

It was part of the reason I wanted to move on from my last position. We were spending so much time chasing infrastructure dependency changes that it began to feel like a treadmill that was slowly increasing speed. At some point I got tired and couldn't find the heart to debug yet another vagrant up failure, knowing that fixing it will probably break something else, and the actual project work was just as likely to suffer from the same issue.

At home I keep it simple, and my advice to a sole developer or small team looking at some of these infrastructure platforms and tools thinking they need them: these tools are not for your use case.

You don't need vagrant, docker, kubernetes to manage a 4 person dev team and you are just burning hours and building the chain of brittle tooling that will be your biggest pain point until you eventually hire a sadist/devops guy.

munin• 8 years ago

The short story version of this was written by Arthur C Clarke in 1951, "Superiority" http://www.mayofamily.com/RLM/txt_Clarke_Superiority.html

emmelaich4• 8 years ago

Fantastic, thanks.

zaf• 8 years ago

Thanks for that! Good story!

reacharavindh• 8 years ago

I was just thinking the other day about "finished software". These days, those Unix philosophy tools of doing one thing well and leaving small solved problems alone are becoming seemingly fewer and fewer.

crankylinuxuser3• 8 years ago

Indeed. The cathedrals won.

I would posit(x !) that it's because it is easier to form a community and 'cost of entrance' around a megalith like Kubernetes, than around individual tools that do 1 or a few very similar things well.

Then again, I would say it's time to start looking away from "handling text streams", to something that can handle data streams of various formats, including transcoders that can convert from one type to another.

toast0• 8 years ago

I don't necessarily mind relying on unfinished software; but it is tiring to rely on something that doesn't appear to be on the path to being finished. Projects with a complicated upgrade cycle and a rapidly moving upgrade treadmill are definitely a no-go.

bonesss8• 8 years ago

There are definitely client app trends away from the Unix philosophy... I would argue that's a product of the success of the Unix philosophy, though. New apps are developed in a world where 'curl' or 'grep' exist, they can move on to more specific needs.

On the platform front I believe this philosophy has recently 'won the war': Microsoft was forced to create Windows Subsystem for Linux (WSL) as a compatibility layer to access exactly that rich tool ecosystem and it's server & production oriented workflow... Cross platform means "not windows", even on windows.

Up a few abstraction levels though, and we can see that philosophy dominating in cloud space... "Micro-services" are API enabled single-use tools focused on doing 'one thing well' and yielding coordination responsibility to higher level applications and not assuming as little as possible about their end-use. Tool silos to support new unforeseen use-cases.

Even more emblematic of the Unix philosophy in cloud space is the emergence and growing popularity of "serverless" solutions: hyper focused single-use tools directly integrated into the computing environment. A single function, pumping between cloud services or transforming some text, "freed" from infrastructure.

In days of old we had to build up the foundations -- text manipulation, process stats, diff capabilities -- but based on that amazing ecosystem the new generations of those tools are freed to focus on new problems and speak a more abstract and higher level language -- APIs, event queues, NLP services... Just like how some primates started grunting, and then they grunted numbers, and then they made satellites and then conquered mars with robots. Shoulders of giants, and all that :)

nlawalker• 8 years ago

Joel Spolsky talks about this (as a somewhat adversarial tactic, even if it's not purposely so) in "Fire and Motion"[1]

"Watch out when your competition fires at you. Do they just want to force you to keep busy reacting to their volleys, so you can’t move forward?

Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That’s probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP. The companies who stumble are the ones who spend too much time reading tea leaves to figure out the future direction of Microsoft. People get worried about .NET and decide to rewrite their whole architecture for .NET because they think they have to. Microsoft is shooting at you, and it’s just cover fire so that they can move forward and you can’t, because this is how the game is played, Bubby. Are you going to support Hailstorm? SOAP? RDF? Are you supporting it because your customers need it, or because someone is firing at you and you feel like you have to respond? The sales teams of the big companies understand cover fire. They go into their customers and say, “OK, you don’t have to buy from us. Buy from the best vendor. But make sure that you get a product that supports (XML / SOAP / CDE / J2EE) because otherwise you’ll be Locked In The Trunk.” Then when the little companies try to sell into that account, all they hear is obedient CTOs parrotting “Do you have J2EE?” And they have to waste all their time building in J2EE even if it doesn’t really make any sales, and gives them no opportunity to distinguish themselves. It’s a checkbox feature — you do it because you need the checkbox saying you have it, but nobody will use it or needs it. And it’s cover fire.

[1] https://www.joelonsoftware.com/2002/01/06/fire-and-motion/

paganel• 8 years ago

This post made be nostalgic for that period which started in 2002-2003 (just after the dot-com crash ) up to 2008-2009 (when FB and Twitter emerged and when Google "changed" its skin to a full ad company), when everything worth reading was being published on blogs, when people (myself included) still believed in the open web (we were all very busy bashing SOAP) and when it wasn't all about IPOs and earning obscene amounts of money (it's also the period when this website was put online and when its founder was just a respected blogger, LISP-er and former Yahoo employee). Good times.

brudgers• 8 years ago

A description of -- not a name for -- is corporate control of an open source project. Trying to keep up with Angular or Kubernetes or Go is not a problem at the Google from whence they come because the technologies are primarily used by and hence primarily designed for use by teams. Not individual developers.

A team's brain can schedule a time where part of it is training or studying while the rest of it is making progress on the task at hand. A team's brain can resolve ambiguity and complexity more easily because it has multiple human brains and those brains have different strengths and experiences.

A single developer can't duplicate a brain. A small team of developers can't replicate the larger multi-team brain of a Google org-chart. Google org-charts are the context for which its tools are designed. The same is true of other corporate open source technology bundles like React and AWS.

People kick themselves for not grokking technologies like Kuberentes without realizing that it wasn't really designed for their use case, isn't documented to be easy to pick up, and isn't managed with the cognitive load of individuals in mind. The time scales around which these technologies are designed are FTE-months or years.

If a two pizza team can get up to speed in a month, it means an individual will probably take about a year. Given equal cognitive efficiency.

mancerayder• 8 years ago

These are excellent points and very interesting food-for-thought, the 'organizational thinking' that leads to the tools being terribly complex and steep to learn for individuals.

But don't these tools have product managers, and don't product managers create tools to serve customers?

The only cynical viewpoint that I can think of for artificial (or lazy) complexity is consulting dollars.

jweir• 8 years ago

I disagree on Go.

Go is well designed and takes upgrading into consideration. Upgrades very rarely break anything. You should just be able to recompile that 1.3 code on 1.8, etc.

digi_owl2• 8 years ago

Been my building worry as i have watched the ongoing changes is the Linux ecosystem in recent years.

It feels like thereis a echo chamber happening, where a small group of peope working for maybe 2-3 companies have decided that their vision is the correct vision.

And everyone that says otherwise are haters and fossils.

m_ke• 8 years ago

Same thing in deep learning. I already had to dump half of my code because of theano, as well as api changes in keras and tensorflow that made it a pain to load some of my old models in newer versions of the frameworks.

Now I'm rewriting a bunch of it in pytorch while anxiously waiting for 0.4 to come out (no idea when) and break all of my models again.

avaer• 8 years ago

I just call it an ad. It is after all the core competency of the prominent stewards of open source.

The software is the ad, the tangential service is the product; keeping up is the infinite sales call that requires no calls.

This is the price of free.

riazrizvi• 8 years ago

It's usually framed as the Build Vs Buy question. When do you stop your search for an adequate ready-made solution and just build it yourself?

jacques_chester12• 8 years ago

It's like the authority to create goes from the individual first-principles (by necessity) maker, to the control over development being in the hands of an external group, and then all your time is spent keeping up with what they're doing.

This is a fundamental economic question: buy or build? Buying involves search costs ("all your time is spent keeping up"), building involves the cost of ... building.

Often the line shifts because of gains from trade/specialisation and the deepening structure of production. As products become more featuresome, it becomes more economical for producers to specialise in part of the problem. They become better at that part than others are.

It quickly becomes impossible for any single producer to out-produce the combined output of specialists.

As a side note, the tension between the cost of searching and the cost of doing it yourself is believed to be why firms can emerge out of "pure" markets. It also suggests an economic reason for why software projects grow at their margin and another reason for why NIH is so attractive.

drawkbox• 8 years ago

There aught to be a name to the tendency that as tools get better and better, the more your time goes from having your mind in technical-space to social and news-space.

It seems like a form of bike shedding mixed with busy work. Nearly all of it takes away from the actual product being built.

bringtheaction• 8 years ago

Is there anything that can be done about this other than to just accept it I wonder.

hueving4• 8 years ago

This reads like a giant ad for GKE. It emphasizes several times to just use GKE for pretty lame reasons (Google has good SREs and Google started the project).

The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code.

I don't have an issue with GKE, but this article uses little evidence to recommend it when it seems the conclusion should have been "maintaining a k8s cluster requires a full time sysadmin. If your company has a culture of pretending sysadmins are pointless, then you should pay another company offering k8s sysadmin-as-a-service hosted on their hardware."

jrochkind15• 8 years ago

I dunno, if it's an ad for GKE, then "you typically have only about nine months until the latest version goes out of support" is indeed pretty persuasive marketing to me to let someone else deal with it.

bigiain• 8 years ago

This is a showstopper for me.

All our best clients are on a 12 or 24 month upgrade cycles. I need to be confident I can deliver them the latest version of their project knowing there's only regular security updates required over then next 1-2 years.

While I'd love to be doing continuous deployment and multiple production feature deliveries a day - that's not how the business I work at makes their money.

I can live with Ubuntu's 5 year LTS policy, I can't work with a "you might have to do significant rework after just 9 months" platform.

bonesss8• 8 years ago

I think it's important to judge the support model in context of what kubernetes is...

As an API based cloud-management platform Kubernetes has a robust, stable, core topped with multiple layers of abstractions that build on, and use, one another. There are relatively few dead-ends in the projects history, and those have all tend to be superseded by a much nicer, much smarter, much easier abstractions.

An operating system going out of support after 9 months is a total show stopper, naturally. But Kubernetes runs on top of those 'stable' layers. It's stability is in its primitives and the physical API contracts. The primitives being introduces today don't magically impact my production cluster, and because kubernetes is built like a russian nesting doll of APIs there is very little incentive for the project to make any changes, much less hasty changes, much less poorly thought out changes, to the core APIs. Unlike many other dependencies the kubernetes changes impact deployment and management, but will rarely impact application-level concerns. If it worked in old kubernetes, it's generally gonna work in new kubernetes.

I feel absolutely no support-pressure to upgrade my on-prem installations when my deployments are cross-compatible with the updated versions on my cloud providers.

I feel a lot of developer-giddiness-pressure to upgrade my on-prem installations because me and my devs want the cool new things they're baking into the platform...

From my experience with k8s the issues are all related to running a linux cluster that uses docker and iptables -- a pretty unavoidable pain to run docker containers on a linux cluster, IMO. Support in this context broadly means a stable API, and a stable core, and Kubernetes has had those for a while now.

aberoham3• 8 years ago

Author here. I do have a bias toward GKE but that is after trying to use everything else -- DIY, kubespray, OpenShift, Juju, Tectonic, Azure, Fargate, kops, Rancher v2 preview, kubeadm, etc. In my experience nothing (yet) has been as nice and clean as Google Cloud.

iooi• 8 years ago

Have you tried EKS?

lugg2• 8 years ago

Curious why swarm isn't in that list.

How many services / nodes are you managing?

I found the docker swarm learning curve to be very sharp but plateaued super quick, maybe a month of reading docs and bug reports then I was up to speed and don't really run into day to day issues.

Once your prod is setup and you've worked out all the hard decisions around stack and how to do things, getting Devs up to speed is really easy because all they need to know is how to read the docker compose reference.

Even less of any issue learning everything now too because they redesigned their documentation (wording etc as well) and it's much clearer, gotchas are explained etc.

jacques_chester12• 8 years ago

CFCR[0], née Kubo, might be worth adding to the list if you're doing rounds.

Disclosure: I work for Pivotal, we work on CFCR with Google and VMware.

[0] https://docs-cfcr.cfapps.io/

gnufied• 8 years ago

I am not sure if GKE can be directly compared with Openshift/Kops/Tectonic. One is hosted service and other is packaged software with DIY setup. Surely you are paying extra money for what you are getting.

rpbo2• 8 years ago

Full disclosure: I do a lot of work with cloud providers and GCloud is one of my partners.

I think the article covers a very real issue with the pace of k8s. GKE is particularly good at staying updated. There are other providers that are good like Stackpoint Cloud.

If I were going to make an ad for GKE I'd point out that they probably have the most robust internal fiber network of any cloud. AWS tries to push packets onto the public internet as fast as possible.

The load balancer you get with GKE is seriously overpowered and under priced (which makes me love it).

Maybe EKS will be amazing (but we don't have evidence yet). Azure is getting pretty good but they're a bit late to the party. IBM isn't bad but they've struggled the most with staying up-to-date.

bonesss8• 8 years ago

Haven't more recent versions of k8s (say 1.9+), dropped some self-upgrading capabilities to smooth the pain of keeping klusters kurrent?

I'm a few versions back on my clusters, but my understanding was that some decent work was being done to smooth the particular pain point of keeping pace with the project :)

crb2• 8 years ago

The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE.

I happen to be in the building so I sat in on an engineering review for a proposed new GKE feature today. Tim was there, as was Brian Grant, and pretty much everyone else you've heard of (in the room or on the VC). There were also a good number of PMs, SREs, and the leads/engineers working on the GKE machinery.

Thus I have to respectfully disagree.

rpbo2• 8 years ago

Can confirm, this is my experience as well.

hueving4• 8 years ago

Engineering review is not the same thing as reviewing and writing code. I've seen countless projects with sound designs riddled with implementation bugs due to people not fully understanding the intent of certain systems.

Sitting in a room during a high level engineering review falls squarely under "limited overlap".

dward• 8 years ago

The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code.

This statement is absolutely false. There is no upstream team separate from Google Cloud team. There is only the GKE team.

tyler_larson• 8 years ago

The GKE reference is a lot more relevant than you think, it's just a bit deeper below the surface.

Google's relationship with k8s and GKE and in particular their motivation for creating an open-source orchestration service is well documented and frequently retold, but ironically appears to not be well-understood.

With any sufficiently complex compute solution, you're going to need both a platform and some sort of management. Typically vendors sell you their management service (for which you often pay a recurring fee) by locking you into a proprietary platform. What's more, the management service can be dismal but you'll pay anyway because your app is built on their platform. It happens everywhere, the industry is full of examples, and anyone could see that it was going to hit Cloud sooner rather than later. And the result would be ... bad ... for everyone (customers and providers alike) but the dominant platform.

So Google took an exceptionally gutsy gamble. Google excels at platform management. They invented SRE. They posited that in a level playing field, all else being equal, Google would have the best platform management service, and people would pay for it based on its merits alone.

By making k8s open source, they leveled the playing field. It was by far the best tool for the job, and everybody could have it at no cost; neither money nor burden. So there was no business model or justification for building a lock-in sub-par alternative. The platform became a commodity; nobody could own it.

Which is where we are now, and what this article depicts. You still need a management strategy, k8s doesn't change that. But now everyone's management service has to compete on its own merits. Nobody gets to play the lock-in game.

Google's management service is GKE. It's good, but not because Google started Kubernetes. Rather, Google started Kubernetes because Google has the expertise and experience to make GKE good.

bonesss8• 8 years ago

I see it, somewhat cynically, as an advanced competitive strategy.

At some point something will become "the standard". Up to that point Google will have to be making huge investment in their own tooling to operate at their scale...

If "the standard" is someone elses, they're playing second fiddle in the big picture. If "the standard" is different than their own tooling they're paying a premium in dollars and a premium in talent-time to train new hires.

Giving away their tooling and supporting it to become a 'best of breed' solution outsources their training costs and 'onboarding' time to the greater industry. Facebook, MS, and VMWare are paying people to get good at Google tech and selling Googles tech in the Entprise. Their open source strategy also ensures, as you said, a 'level playing field'. A level playing field where they are guaranteed to not get locked out, and where where their size, strengths, and deep competencies of product and domain give them a massive competitive advantage.

It's smart on a lot of levels.

hueving4• 8 years ago

They invented SRE.

Let's not get dramatic and drink too much of the kool-aid. SREs existed well before Google albeit under a different name.

halayli3• 8 years ago

This is a problem I faced using ansible, webpack + js modules, and more. it's a moving ground and you always need to keep up to date with the latest changes often which are breaking.

I tend to design my systems so that they work even if I haven't touched a line in a year but when using such tools it's always a pain.

I wish things were as stable as a bourne shell and unix environment in general. Not that they achieve the same but I just miss the stability I get from raw unix tools.

SteveNuts• 8 years ago

I wasn't around for it, but I bet you'd say the same thing about Bourne Shell when it was new.

The tools will stabilize as they mature, they're still very new comparatively.

wwweston• 8 years ago

Or... as they "mature", people will start talking about how kubernetes is for old crusty engineers over 27 who just don't want to learn anything new, why don't they just learn ganymandias which is much more modern.

halayli3• 8 years ago

Just to be clear I am not insinuating that new tools and their developers are inferior. But backward compatibility is also very important.

When I am using version 1.9 stable release of a product i’d expect more stability

deckard12• 8 years ago

Github is already littered with the dead carcasses of software using shiny new obsolete tools like gulp, grunt, and browserify to name just a tiny few.

You have to have a strong nose for bitrot and ruthlessly filter garbage out of the firehose, to work in web development today. If that package or tool hasn't been updated in 8 months, do you really want to learn it (and force your team to learn it), since it's practically abandonware already?

It's a self-reinforcing cycle. You don't want to put your organization at existential risk by betting on the wrong horse. But there is no choice. Unless you want to get stuck having to hire Perl guys in an ocean of node ninja rockstars. Your company will be the ugly girl that didn't get asked to prom night. The pariah of Silicon Valley.

halayli3• 8 years ago

true but you don’t have to go that far. React, webpack, and sometimes well maintained packages that depend on unmaintained packages suffer from this. I am not picking on web dev, it’s a general problem.

cookiecaper8• 8 years ago

You have to have a strong nose for bitrot and ruthlessly filter garbage out of the firehose, to work in web development today.

This is a big test for the development community, and I don't know how it resolves. Are discerning coders who've seen this fad cycle before just going to have hold their noses and pretend to be gung-ho about the latest fad, or do they stick to their principles and say "Yeah, I'd use that if we had a good reason, but I just don't know many..."?

Bear in mind, doing the latter just means some idiot who's read six lines of a blog post from TechCrunch is going to come in there repeating "blockchain" and "Kubernetes" non-stop until they get hired on the spot.

We have a lot of people who've been misled into believing "copy Google and magical fairy unicorns will fly out of your butt" crawling all over this community, including a lot of under-experienced developers themselves who apparently don't realize that they're reaching outside of their comfort zone with this.

It's very hard to find a company that isn't trying to get Kubernetes-ified as fast as they can. If you go into an interview and say that you're so-so on Kubernetes, it'd be like saying you don't like Node in 2014 or that you don't like Mongo in 2012 -- you're out the door. Every company needs to be like the cool kids and get the magic of the Great Googly Kubernetes smeared on themselves.

This is of course very silly and bad from a technical standpoint, but it seems to primarily be about social standing. That's why something like EKS is so good. They just fork over some cash to Amazon and then they can tell everyone that they too use Kubernetes, and all of Google's hard-fought marketing effort goes down the tubes, since users are still glued to Amazon's platform.

mbrumlow4• 8 years ago

Every time a new framework or tool comes out and everybody jumps on it. I always wonder if somebody will realize that you are trading one set of problems and work for another.

As engineers we really need to stop supporting these sort of effort and take the time to help each other become better engineers that write and maintain our own code. We need to promote learning and mastering the underlying concepts that things like kubernetes tries to hide and shield engineers from.

In most cases tools like kubernetes are so vast and huge so they can be the solution looking for many problems.

It is also curious how once kubernetes became big how many small shops needed "Google" level orchestration to manage a hand full of systems. And how hard people ripped their software stack apart into many many micro services just to increase the container count.

I think if most engineers took a step back and said "I don't know" and took some time to truly understand the requirements of the project they are working on they would find a truly elegant and maintainable solution that did not require tweaking your problem to fit a given solution.

Every tool and library / dependency you add to your solution is only adding more code and complications that you will not be a expert in and one day will find your self at the whim of the provider.

Far to often do we include tens of thousands of lines of code of somebody else's work all for a handful of lines of code that if somebody would have had the confidence and support from other engineers around to try and truly understand the problem domain could have implemented and owned the solution.

The general trend I see as I get older is that we are valuing the first to a solution rather than a more correct solution. Only to be stuck with a solution that requires constant care and work around.

So I plead to all engineers, devlopers, programmers or whatever you call your self. Please stop and take a moment and think hard about how you would solve any given problem without the use of external code first. Then compare your solution to the off the shelf "solution looking for a problem". You might surprise your self.

I will also like to point out that if when solving a problem your solution looks like a shopping list of third party tools libraries and services; you might not fully understand the problem domain.

-- sorry for the rant --

gouggoug4• 8 years ago

I don't think your comment applies at all to Kubernetes.

K8s truly simplifies dev-ops and even the smallest team and website can greatly benefit from it.

I speak from 7 years of experience managing my company's infrastructure's website:

Before kubernetes, I ran my company's stack on very cheap bare metal from OVH. It was great while it lasted, but as the company grew and my team grew, it's become harder and harder to maintain this infrastructure. And I'm not only talking about our production servers. In reality you have to maintain your prod, your staging, and, worst of all, your local development environment.

Over time, your production staging and dev all become entirely heterogeneous. Each environment ends up running totally different/incompatible versions of all your stack's softwares, and no amount of Ansible/SaltStack/Puppet script will save you from this. All those scripts become a nightmare to maintain and your infrastructure as a whole becomes brittle with, for example, bugs happening only in production, but never on your local dev environment.

K8s came as a savior to all my issues: I burned all my old ansible scripts and rewrote all my infrastructure in k8s. Now my prod, staging and dev env are 99% the same. It saves me a tremendous amount of time and headaches. I taught my team how to use minikube and create a replica of our production with one command line on their local computer.

K8s is far from being just a trendy buzzwordy shiny new cool toy to play with. It solves real world problems that dev ops have. I am so glad this technology exists and I hope to never have to go back to writing ansible/puppet/whatever scripts.

mbrumlow4• 8 years ago

and no amount of Ansible/SaltStack/Puppet script will save you from this.

And I am afrade you missed my point entirely if you are bring up yet again more tools and frameworks.

Owning your stack from the top to the bottom - with very few exceptions is what I am suggesting.

I too also speak from experience, and have a few years on you -- not that any of that matters.

If your environments diverged then you did not own your stack properly. It is within every engineers ability to build out a stack that runs the same on each environment. It just means understanding that problem domain and taking the time to write the code that fits.

Far too many times have I smacked the hand of a coworker suggesting using any of these bail wire and duct tape solutions you just mentioned. The end result is we have a wonderful system that we own and can ensure meets our needs. The same binaries, configurations and images are pushed from alpha to staging and finally production with full control of evey artifact at the start of the pipeline.

botskonet• 8 years ago

I hadn't considered the benefits for dev/local instances...

I've known about Kubernetes for some time, but my current job never deploys anything that Kubernetes could improve, so I put it aside and hoped to someday get a chance to toy with it.

I've setup vagrant images pre-loaded with our app for several non-developers to use locally but it sounds like Kubernetes would be a far better way to manage those as well as staging servers.

Unfortunately my current company is 100% against third-party hosting/involvement so I couldn't even use it for staging - our stagings servers are Windows-based, ancient, and internal...

cookiecaper8• 8 years ago

I don't think your comment applies at all to Kubernetes.

It does. I've deployed and maintained our company's infrastructure on Kubernetes for over a year now.

K8s came as a savior to all my issues: I burned all my old ansible scripts and rewrote all my infrastructure in k8s.

This doesn't make any sense.

Ansible scripts out changes to make to a system. Kubernetes deals only with opaque images and does not change them at all, it simply runs them.

K8s is far from being just a trendy buzzwordy shiny new cool toy to play with. It solves real world problems that dev ops have. I am so glad this technology exists and I hope to never have to go back to writing ansible/puppet/whatever scripts.

I'm not really sure how to reply to this without being accused of bad faith, but I'll just reiterate again, your post does not make sense because it is talking about tools that do different things. It's like saying you love hammers and hope to never use wrenches again.

That said, Kubernetes is a shiny buzzword that is much overvalued.

nitwit005• 8 years ago

Over time, your production staging and dev all become entirely heterogeneous.

I've found that this is simply the reality, excepting test-only environments. Developers need to be able to run the applications out of an IDE, compiled with special instrumentation, or whatever else they need to do. It's not possible to support every combination of what they need.

Also, there's some significant developer overhead sometimes. I had another team give me a set of VMs for their product, so I could test a new config. It was great in that it was just like production, but not so great in that it was just like production. How do I log into these VMs? How does authentication work? Where are things located on the VMs? How do I generate an updated VM? You end up having to train everyone in dev ops, because you've handed them a copy of production.

bpicolo3• 8 years ago

even the smallest team and website can greatly benefit from it

I don't think that's true. You have to be at least large enough to justify reasonable investment in microservices.

nova22033• 8 years ago

We need to promote learning and mastering the underlying concepts

Should we do the same thing for programming languages? Make sure people are learning and mastering assembly?

Only to be stuck with a solution that requires constant care and work around.

I've seen this attitude from people who didn't want to use ansible/chef/puppet because their shell scripts were "good enough". Like the shell scripts didn't require constant care..

mbrumlow4• 8 years ago

Well it is a good thing I did not suggest shell scripts, ansible or puppet. I suggested owning it stack. Which means wiring code and tools to suite your needs.

Deployment and upgrades should be built in - as in it is part of your product. Not some afterthought.

It's odd how the two replies have read have jumped to conclusions about what one might use if they were not using kubernetes.

mbrumlow4• 8 years ago

You should master any subject you were hired for to do work in.

I find it hard to equate deploying and maintaining your production environments to assembly.

bonesss8• 8 years ago

I've seen this attitude from people who didn't want to use ansible/chef/puppet because their shell scripts were "good enough".

That attitude is also quite myopic.

Not to condescend, but if one isn't working on cloud-native apps in federated environments then maybe maybe maybe one should be intellectually humble enough to allow that there are some challenges people face that make "good enough" a non-starter, prohibitively expensive, or require direct competition with AWS/MS/Google in their core competencies.

Everything is a spectrum. Serious work can be done on a single server with no backups... Shell scripts can carry serious work far... Ansible or Docker Swarm rock for what they are... but the multi-cloud orchestration platform we're discussing rocks a oodles of things those solutions aren't.

jacques_chester12• 8 years ago

-- sorry for the rant --

It's not unreasonable, but there is a gradient. Linux was a toy, now it's probably involved with handling more real commerce than any other software ever written. Javascript-land was a riot of colour and decay, but basically React and Webpack have won the dominant position.

Evolutionary explosions don't last forever. Deciding to not pick winners is smart early on. But once the ecosystem settles down and there are clear winners, it's time to accept the new normal.

If I sit down to write a web app, I do not first create my own programming language. I do not write an OS. I do not develop a database, or a web server, or a transport-layer protocol. I take the ones that exist off the shelf and use those.

This is only possible because some options are so dominant that they have driven nearly everything out. By doing so they attract to themselves the overwhelming share of effort and attention. They grow a rich cast of supporting systems, they get bug fixes sooner, they serve as the jumping-off points for the most exciting new possibilities.

We are past the stage where "I will write my own cloud platform" is a defensible position. I believe it past about 2 years ago, actually.

justicezyx2• 8 years ago

K8s is different though.

It's an incarnation of a highly-effective & efficient infrastructure paradigm, verified in over a decade long serving of entire google's compute tasks.

It's not really new.

Disclaimer: I work in Google's compute infrastructure team.

etaioinshrdlu• 8 years ago

There is clearly a lot of unfinished tooling and features, as evidenced by the extremely high commit rate.

Compare that to an old piece of infrastructure like "mount" or "chroot".

Is it too much to ask for base infrastructure to be boring?

Granted, I have nothing against the Kubernetes project and it might be fantastic. I don't think it's wise to encourage people to use/learn it at this point, at least for production projects.

SrslyJosh• 8 years ago

I think if most engineers took a step back and said "I don't know" and took some time to truly understand the requirements of the project they are working on they would find a truly elegant and maintainable solution that did not require tweaking your problem to fit a given solution.
The general trend I see as I get older is that we are valuing the first to a solution rather than a more correct solution. Only to be stuck with a solution that requires constant care and work around.

This, 1000 times over. Right now I'm trying to figure out how to turn a project around that started without me and has gone down the path of using an inadequate (but mildly popular) open source tool instead of building something that's actually designed to solve our problem, not someone else's.

maxxxxx2• 8 years ago

I think this is a symptom of the "release often" philosophy. With yearly or longer releases you could actually keep up and read the release notes. With stuff being released several times a year it's too much work to keep up unless you are deeply into it at the moment.

I notice the same with my Android apps. I used to read release notes of new versions but now I have it on auto update and am sometimes surprised that an app I have been using all the time has completely changed and I don't know how to use it anymore.

praseodym• 8 years ago

No, it is the symptom of a high velocity project. If Kubernetes were to have yearly releases, the list of changes would be four times as long and upgrade path would be a major leap instead of four minor steps.

maxxxxx2• 8 years ago

I think a major leap is much easier to handle and to plan for.

peterwwillis3• 8 years ago

trigger warning: bitter jaded ops person working in a real company

"[...] users are expected to stay “reasonably up-to-date with versions of Kubernetes they use in production.” [...] the upstream Kubernetes community only aims to support up to three “minor” Kubernetes version at a time. [...] if you deployed a Kubernetes 1.6 soon after it came out last March, you were expected to upgrade to 1.7 within roughly nine months or by the time 1.9 shipped in mid-December."

Jesus christ this is so annoying.

Businesses don't have a couple hundred billion dollars sitting around to spend on engineers to look at release notes, compare changes, write new features, write new test cases, fix bugs, and push to prod, every 3 months, just to keep existing functionality for orchestrating their containers.

We have LTS because businesses (and individuals) don't want to have to do the above. They just want a reliable tool. They want the ability to say that if a bug is found in 3 years, it will be fixed, and they can just keep using the tool.

We don't give a crap about "Kubernetes’ domination of the distributed infrastructure world". We don't want to use Kubernetes. We just want an orchestration tool - commodified tooling. We want to stop caring about what we're running. We just want the fucking thing to work, and to not have to jump through hoops for it to work.

"Moving Kubernetes Workloads to New Clusters instead of Upgrading"

UGH. We only do this for bastardized unholy stupid shit like OpenStack. Not only is this not fun, it takes forever (you try moving 50 different clients off the service they've been using for three years), and you have to have duplicate resources. What the fuck is the point of cloud computing and containers and all this bullshit if I have to have double the infrastructure and juggle how it's all used just to upgrade some fucking software?!??!?!

"The Kubernetes-as-a-Service offerings, particularly Google Cloud’s Kubernetes Engine (GKE), are the well-polished bellwethers of what is currently the most stable and production-worthy version of Kubernetes."

Oh. We're supposed to pay Google to run it for us.

....I'm just going to use AWS.

kev0092• 8 years ago

Sounds like full employment theorem at work. I haven't kicked the tires on kubernetes yet but I don't really see what all the fuss is about. I liked AMPLab and Mesos but I guess that doesn't have the branding power of big G.

jacques_chester12• 8 years ago

I work for Pivotal, which with IBM wrote a container scheduler (Diego) of similar vintage.

Nobody has the branding power Google has amongst developers.

cookiecaper8• 8 years ago

100% the case. k8s is not a terrible thing and there are good uses for it, but the vast majority of people who are using it don't understand what it's doing, and don't understand that their application is not equipped to handle that type of execution model.

Just yesterday I had someone tell me "Kubernetes was like magic and it made everything easy". This is absolutely not the thing a serious/honest Kubernetes user would say. Kubernetes is a big heavy thing, and it's a lot of trouble to maintain it; generally, far more trouble than using sane configuration management on VMs would be (you need sane configuration management in k8s too, so you aren't swapping one complexity for another). We're reaching "MongoDB is web scale" levels of hysteria around Kubernetes.

Example of the insanity: my account on HN has been censured for the "tedious" nature in which I would assert that Kubernetes is not a good platform for databases.

There is a serious, concerted effort by Google to put k8s at the forefront, and they are not playing games with it.

outworlder• 8 years ago

I was going to reply to the general point, but I got interested on this:

you try moving 50 different clients off the service they've been using for three years

This sounds like a major red flag.

Are they on-prem air gapped servers? Do you manage the infrastructure? Because if you do and they aren't, they are not supposed to see it. And if it is automated, what does it matter if there are 50 or 50 thousand clients?

The point of the cloud is that the customers don't have to care. And you also shouldn't care about individual servers. Or even clusters.

On K8s, it does a lot of really neat stuff that would keep me awake at night otherwise. Oh a K8s worker VM died in AWS. I don't care. There's a machine that's misbehaving. Kill it, another one will come, containers will spawn there. I don't care. Need more container instances, done. Need to deploy a new service? Push the new yaml to the server, done.

Upgrade twice a year and you'll be ok, unless your dev team insists on new features.

peterwwillis3• 8 years ago

they are not supposed to see it

There's a whole lot of things that are not supposed to happen in the tech world, that happen all the time. Most people do not run brand new stateless microservices in K8s. They run stateful apps on VMs in three year old badly configured OpenStack clusters.

For what it's worth, moving the client traffic itself is seamless and totally fine. But first you have to get them to stop hard-coding the names of region and zone specific load balancers into their apps. You switch them over... traffic falls over... and it's your fault. Hence, big moves are coordinated with clients, not to mention feature tested, and then tested by sending some real traffic to the new cluster, before you shift all the traffic. The cloud is not magic, and if you treat it that way, it will bite you.

maherbeg• 8 years ago

I also wouldn't be surprised if an external vendor will build LTS releases for a fee. That's a pretty common and successful model that would solve this problem easily.

shruubi2• 8 years ago

This article concerns me, especially considering the first thing you see is "There is no such thing as Kubernetes LTS (and that’s fantastic)".

What is so great about running your infrastructure on a platform that has no intention of ensuring long-term stability? Irregardless of how well backward-compatibility is maintained, the idea that we should all move our infrastructure to something that lacks the fundamental promise of "updating won't break everything" seems downright irresponsible.

flyinghamster• 8 years ago

Just trying to keep up with the shifting sands of web programming is a headache in itself. And now we have people wanting to build our whole infrastructure on yet more shifting sand. Argh.

wmf• 8 years ago

LTS releases aren't really about stability or upgradability; in practice I think they're more often used as an excuse to never upgrade because the risk of applying 2-5 years of changes at once is too high. It sounds like k8s is trying to nudge people into a more continuous deployment model.

DougWebb• 8 years ago

The last thing I want from an external supplier of something I depend on is to have them driving my development and release cycle. I'll expend the resources to upgrade to your latest version when it's worth my time and cost.

Edit: I'm speaking to the vendor here, not you, of course...

aberoham3• 8 years ago

Author here. The point is that the stable APIs are rock solid, the design of its API versioning and the community's pace of delivery is a huge part of why it's all maturing so quickly.

shruubi2• 8 years ago

That's all well and good until it isn't anymore. And to top it off, your recommendation is to either hire someone to manage k8s full-time or be at the mercy of your cloud vendor and hope that one day you don't turn up to work to find that the new update you weren't aware was being rolled out has just broken everything.

You can't ask a large-scale company to take a dice-roll on k8s for their infrastructure when the only promise of long-term reliability is "trust us not to make any breaking changes".

jrochkind15• 8 years ago

I think you should have leaned on that point harder and more explicitly for the readers that were not already k8 users. :)

gtirloni• 8 years ago

What is so great about running your infrastructure on a platform that has no intention of ensuring long-term stability?

Kubernetes uses API versioning and takes backward compatibility very seriously for stable APIs.

Regarding LTS, these discussions are happening and increasing in number as time passes. Example: https://www.youtube.com/watch?v=fXBjA2hH-CQ

jrochkind15• 8 years ago

That's a really important review that is not showing up in the rest of this lengthy discussion.

The only way frequent upgrades are possibly sane is if you take backwards compat very very very seriously, and succeed at doing so. If users of k8 think it's doing that, that's important info.

(I think many of us are lately getting burned by a lot of open source software that _doesn't_ succeed very well at backwards compat (whether they're trying or not, sometimes it's not clear), _and_ expects you to upgrade all the time).

cookiecaper8• 8 years ago

No LTS is fantastic -- as long as you don't expect me to put it in production. ;)

k8s is a tale of woeful, hilarious levels of platform immaturity jammed down the public's throat by the Google PR machine. I believe Kubernetes is a centerpiece in Google's cloud play against Amazon (I don't think it will be successful), and that this consideration drives a lot of the particular weirdness and distortion that exists around it.

It'd be a whole different ball game if they discouraged production use and told people "We can't even commit to keeping this up to date for one single year, whereas all other enterprise vendors offer 5 years or more, you'd be silly to think this was ready for prime time." But they don't, because then you don't have starry-eyed brogrammers saying "DUDE, we gotta get on k8s so we can be like my homies in Mountain View, aw man, it's so awesome! It says Google Cloud is the best way to get Kubernetes, call 'em up!"

I feel sorry for anyone who was naive enough to say "Well, if Google does it, so do I!" and jumped headfirst into this bear trap, making themselves collateral damage in a pissing contest between two megacorps.

Source: I work with people who decided to move our entire infrastructure to Kubernetes. We have now run virtually everything on k8s for over a year. I'd feel sorry for them, but they left the company some time later, so now I just feel sorry for me. :)

dralley• 8 years ago

"There is no such thing as Kubernetes LTS (and that’s fantastic)"

OpenShift?

smarterclayton• 8 years ago

Technically OpenShift isn’t LTS yet, it’s 1 year and 3 year (the latter is special extended support). Right now we’re planning on 1.9 being our first LTS with the longer support commitment (3yr?). Still being sorted out.

We won’t be doing the 10 year support like for RHEL 4/5/6 yet.

macNchz• 8 years ago

The absolute safest place to run Kubernetes application is still Google’s GKE.

Interesting to read given I recently had a GKE cluster auto-upgrade its master version from 1.6.x to 1.7.x at 8pm one night (however foolish it was to not be subscribed to the release notes RSS Feed[1]), which somehow caused a cascading nightmare of things breaking.

Logs stopped appearing in the GCP logging interface and in our own log parsing pipeline, a bug related to a change in the format of the yaml specifications meant that all the containers got stuck in a broken limbo state as we tried to upgrade the nodes (yay for googling the error and finding open github issues!), and then once we'd manually fixed all of our deployments and were finally able to get our nodes rolled over to the new k8s version, all of our load balancers started intermittently timing out until they were deleted and recreated.

Surely we need to keep a closer eye on the release cycle, and we're guilty of bandwagonning onto this cool new tech, but boy does it suck to get auto-upgraded at night only to discover several breaking changes while in emergency mode trying to fix things.

(1) https://cloud.google.com/kubernetes-engine/release-notes

yeukhon• 8 years ago

Kubernetes’ governance is becoming like Openstack and (I know this is controversial), I hate Openstack, especially because it tried so hard to be “AWS” compatible, and APIs are so awkward to use.

Cloudfoundry is better in terms of governance and project’s direction. Many of the main developers work full time at Pivtoal. But it is hard to run your own CF without significant investment like access management and “painless” upgrade (etcd is a pain in the entire CF stack in my experience). Though I have to admit the project is moving in the right direction in the past year or so.

jacques_chester12• 8 years ago

Cloudfoundry is better in terms of governance and project’s direction.

The Cloud Foundry Foundation rules are very different from CNCF's and intentionally take DNA from Pivotal Labs. It has strengths and weaknesses.

etcd is a pain in the entire CF stack in my experience

It's either been removed from CFAR, or is close to it, I lost track. A lot of time was spent before it was decided that etcd doesn't play nicely with BOSH.

It's come back into Cloud Foundry land via CFCR, as a Kubernetes dependency. Very nostalgic.

caniszczyk• 8 years ago

Kuberentes/CNCF governance is completely different than Openstack. There's a reason every major cloud provider is involved in CNCF versus Openstack. You can see all stats for CNCF projects here, i.e., https://k8s.devstats.cncf.io/dashboard/db/contributing-compa...

CFF was setup in a completely different way, giving Pivotal a lot of control in the beginning by allowing related entities to have votes and than relinquishing that over time. It leads to a more single vendor controlled ecosystem IMHO.

There's pros and cons to both approaches.

Disclosure: I help run CNCF.

jacques_chester12• 8 years ago

Pivotal's strong influence over CFF decisions comes from the fact that votes are assigned according to how many fulltime engineers you devote to Foundation work. Pivotal has more fulltime engineers on CFF projects than anyone else.

There are, as you note, pros and cons. It made sense at the time, as I think there were concerns about vendor politics getting in the way of developing the thing.

I guess one of these days we'll smash the CFF and CNCF together and stagger out with some kind of bicameral system, just for laughs.

Disclosure: I work for Pivotal. Trolling is just my hobby.

atulatul• 8 years ago

Dr Dobb's articles below reflect a somewhat similar feeling

Just Let Me Code http://www.drdobbs.com/tools/just-let-me-code/240168735

Getting Back To Coding http://www.drdobbs.com/architecture-and-design/getting-back-...

scarface74• 8 years ago

Combining my brief time trying to put together a proof of concept with Kubernetes and reading articles like this, I'm so glad I chose Hashicorp's Nomad. It's simpler to configure, more versatile (shell scripts, executables, and Docker containers) and a decent third party UI - HashiUI. With Consul, configuration is dead simple.

The full-time job of keeping up with Kubernetes

Related Stories