Kubernetes Failure Stories

https://github.com/hjacobs/kubernetes-failure-stories

by yankcrime • 7 years ago

320 154 7 years ago

dullgiulio• 7 years ago

Nice, exactly what I needed today, after firefighting a latency spike because of a compute node after a separate ingress node was restarted.

I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...)

In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn't bother to have capable admins setting things up the right way).

What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs.

All have failure modes you need to learn: it takes a dedicated team. And once you have that, you'll need another team and cluster to perform an upgrade.

But hey, when it works, it is really cool to deploy software on it.

ben_jones• 7 years ago

I feel like their needs to be a sanity check. Realistically what does kubernetes do to "not make software fail"? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?

theflyinghorse• 7 years ago

Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).

clhodapp4• 7 years ago

If anything, this enhances your actual point but... if you are using self-signed certs with Kubernetes, then your cluster isn't secure.... so you shouldn't do that.

geggam6• 7 years ago

Arguably self signed certs eliminates possible state funded middle man attacks making it more secure

vorpalhex• 7 years ago

Self signed certs are absolutely secure and more secure than non-self signed assuming you are using proper key management.

Now, whether or not that works for your users...

jayd16• 7 years ago

I think he's implying a whitelisted set of self signed certs, not "trust any cert".

tialaramex• 7 years ago

a sea of self-signed TLS certificates that can expire

I would like to know more about what's going on here. Is this just a sloppy description and in fact Kubernetes uses a private PKI, so that the certs you're using aren't in fact self-signed but signed by a private CA?

raesene93• 7 years ago

There's (IMO) a mix.

Kubernetes and cloud native software make a lot of use of TLS for mutual auth.

A standard Kubeadm cluster (very vanilla k8s) has 3 distinct Certificate authorities, all with the CA root keys online.

On top of that things like Helm, FluentD and Istio will make use of their own distinct TLS certs.

One of the most "fun" pieces is that k8s does not support certificate revocation, so if you use client certs for AuthN, then a user leaving/changing job/losing their laptop can lead to a full re-roll of that certificate authority :)

ownagefool4• 7 years ago

k8s tends to use a private PKI. I do wonder if we're comparing secure verses insecure solutions though. Nobody ever has TLS issues with swarm, puppet, or aws, or are they just trusting the network?

darkarmani3• 7 years ago

so that the certs you're using aren't in fact self-signed but signed by a private CA?

I find that most people confuse or combine "self-signed" with "signed by a private CA". For a lot of uses, the configuration pains are the same to the user: "I have to load this cert into the CA root trust store". They don't realize how much better a private CA really is.

And of course, PKI would be so much more useful with "name constraints" so you don't have to trust a private CA for all domains just the one domain you care about.

peterwwillis• 7 years ago

Some people, when faced with a problem, think: I know, I'll use distributed systems!

Now they have a new career.

hinkley4• 7 years ago

navigate a sea of self-signed TLS certificates that can expire

When certificates become a logistics category of their own, you really do need some monitoring software to warn you when certificates are winding down.

Years ago I worked on a complex code signing system and one set of users were adamant we have alerts starting at 90 days out. It took some convincing to get me to agree it was a priority, but some of my coworkers were never convinced.

manishsharan• 7 years ago

A long time back , I had gone down a rabbit hole to get Kubernetes manage my Mongodb. Thankfully , I got bitch slapped into reality within a few days and gave up because that rabbit hole goes deep. Now my mongodb stands outside of the K8 and I have no issues with it that cant be solved with a rebuild of a replica node.

ravedave5• 7 years ago

A very similar story where I work except an additional addendum of a migration to DocumentDB.

elsonrodriguez5• 7 years ago

Glad I'm not the only one who almost got white-whaled into running stateful services on K8s.

The API is so slick that it makes you feel you can do anything. But there's can, and there's should.

pas3• 7 years ago

The API is there, but is there a stable MongoDB Operator? ( https://github.com/Ultimaker/k8s-mongo-operator ? https://github.com/mongodb/mongodb-enterprise-kubernetes ? )

allochthon4• 7 years ago

Looks like there's a mongodb helm chart that uses stateful sets: https://github.com/helm/charts/tree/master/stable/mongodb

pas3• 7 years ago

Don't use anything that doesn't come with a very-heavily battle-tested Operator (that does periodic backups and does at can restore). Simple StatefulSets are ... cute, but are very far from a reliable solution. Because when the shit hits the fan you won't be able to efficiently fiddle with containers, persistent volumes, secrets and configs (tons of YAML and so on) to try to somehow salvage the setup.

StavrosK4• 7 years ago

Databases are the one thing I will run as close to the OS as possible (usually on a VPS) so I can reason about them as simply as possible.

cookiecaper16• 7 years ago

After a couple of years of massive frustration with the entire direction of the "devops" segment, I think I'm resolved to get out of it and either move further up back to ordinary application development or further down into the actual kernel, preferably working on FreeBSD or some other OS that's more sane and focused than Linux.

Kubernetes represents the complete "operationalization" of the devops space. As companies have built out "devops" teams, they've mostly re-used their existing ops people, plus some stragglers from the dev side. These are the people you hear talking about how great Kubernetes is, because for them, they see it as "run a Helm chart and all done!". Which makes sense, since they were, not too long ago, the same guys fired up about all the super-neato buttons to click in the Control Panel. 90% of "devops" people at non-FAANG companies are operations people who just think of it as a new name for their old job.

Among this set, there's no recognition of the massive needless complexity that permeates all the way through Kubernetes, no recognition of the tried-and-tested toolkit thrown away and left behind, no recognition of the fact that we're working so hard to get things that we've had as built-in pieces of any decent server OS for decades. No recognition that Kubernetes exists so it can serve as Google's wedge in the Rent-Your-Server-From-Me Wars, and no awareness that just in general, there's no reason it should be this hard.

Of course, to them, it's not hard. They have an interface with buttons, they can run `helm install`, they get pretty graphs via their "service mesh". That's what I mean by "operationalized"; Kuberenetes is meant to be consumed, not configured. You don't ask how or why. You run the Minikube VM image locally and you rent GKE or EKS and go on your merry way. The intricacies are for the geniuses who've blursed us with this death trap to worry about! Worst-case, you use something like kops. Start asking questions or putting pieces together beyond this, and you're starting to sound like you don't have very much of that much-coveted "devops mindset" anymore.

"What happens if there's a security issue?" Oh, silly, security issues are a thing of the past in the day and age where we blindly `FROM crazy-joes-used-mongodb-emporium:latest-built-2-years-ago`. Containers don't need updates, you goose. They're beautiful, blubber-powered magic, and the Great Googly Kubernetes in the sky is managing "all that" for us. Right on.

I'm picking on Kubernetes specifically here because it's the epitome of all this, but really everything in "devops" world has become this way, and combined with the head-over-heels "omg get us into the cloud right now" mentality that's overtaking virtually every company, it's a bad scene.

Systems have gotten so much more convoluted and so much dumber over the last 5 years. The industry has a lot to be embarrassed about right now.

geggam6• 7 years ago

IMO .... AWS has been eating everyones lunch. K8s was a play by Google to sell GKE / GCP and it worked.

Problem being not everyone has a team of Google SREs to manage it so when k8s blows up the skillset to figure out the issues simply doesnt exist in the team managing it.

cookiecaper16• 7 years ago

Yes, there's certainly a landgrab occurring in this space, and Google is certainly pouring a massive heaping of G-Juice on k8s in order to get in front of the user. In software, controlling the user interface is controlling everything.

Ultimately, it's to their credit, because in some small measure, it counteracts an Amazon-controlled dystopia headed by Galactic Emperor Bezos, so perhaps we should be glad for it. It's just sad that so many systems have to be the collateral damage.

pjmlp5• 7 years ago

They are still the 3rd cloud provider.

It is hard to state that it worked, specially when AWS and Azure also offer Docker and Kubernetes deployments.

On Microsoft's case it is even supported directly from Visual Studio.

Rapzid• 7 years ago

..people at non-FAANG companies are operations people who just think of it as a new name for their old job. Among this set, there's no recognition of the massive needless complexity that permeates all the way through Kubernetes.

Kubernetes was born out of ideas from Google's own internal systems. I think this discounts the complexity of operational platforms at the large companies. Companies where they build operational APIs straight into their services. It may not be the right tool for every job, but being so overly dismissive of complex operations platforms comes off as extremely pretentious.

defen• 7 years ago

The implicit message is that there are 5 about companies in the world who need Kubernetes' (and friends) inherent complexity. If your company is not named "Amazon", "Google", "Facebook", or "Netflix", it's probably not one of them.

cookiecaper16• 7 years ago

I'm not sure I communicated that clearly -- I meant that people at NOT Google just upgraded the job title. There's no question that some extremely qualified people work on the Kubernetes platform itself.

The issue is that they've thrust it upon the rest of us lowly mortals as a general toolkit, when it's only potentially-appropriate for companies at Google scale, in terms of both traffic and talent.

ownagefool4• 7 years ago

Most large orgs I'm looking at pretty much just have guys logging into tin, on a "secure" network, configuring things based on written instructions.

Is that better than k8s? Is is VMware? Is it Ansible, Puppet or Chef?

How are we dealing with node failures, are you paging and waking people up or just running everything in super HA? How are you making good useage of your compute, VMs?

What is your better system?

cookiecaper16• 7 years ago

Is i[t] VMware? Is it Ansible, Puppet or Chef?

The funny thing being that all of these things are applicable in a k8s-backed environment anyway. k8s is a container orchestrator and scheduler. You still need a platform (VMWare, AWS, GCloud) and configuration management separately. So, I guess the answer is "yes"?

How are we dealing with node failures

First, in an ordinary system, "node failures" are rare. This thing that k8s encourages where the first response to a problem is "just kill the pod and hope it comes back OK on its own" is distasteful in many ways, and only further cements why MS-centric ops people are gung-ho. In my day, when there's a problem with a system, you divert traffic, take snapshot, analyze to figure out the reason it failed, and then prevent failures. Even in k8s-world, failures cost something, and it's bad practice to allow them to occur regularly in production (note that making "just reboot^H^H^H^Hschedule it" a routine is separate from being able to tolerate failures transparently).

Second, you use the same systems that k8s ingresses use internally: load balancers with health checks like HAPROXY, nginx, Envoy, etc.

How are you making good useage of your compute, VMs?

First, not dedicating a bunch of it to redundant and unnecessary distributed systems mechanisms. k8s is not cheap or easy to run.

Second, there are robust toolkits and monitoring options for managing all kinds of workloads, including controlling scale. Depending on your platform and use case, there are a plethora of options. These problems are not new.

The crux of the issue is that k8s is misunderstood as a generalized toolkit for every common-but-not-trivial operational task, because that's how Google promotes it to maximize adoption and thus wedge themselves into a position of more control v. AWS. k8s is a complex multi-node container orchestrator and scheduler. If you didn't need one of those before you knew what k8s was, you probably don't need one now.

lifty2• 7 years ago

I would really like to hear what's your proposed alternative for managing software. The world is digitising at a staggering pace and we have to deal with ever increasing density and complexity in software deployments. How would this look in your ideal world?

darkarmani3• 7 years ago

A better understanding of the hardware+software stack and a return to fundamentals instead of piling frameworks sky-high on top of each other making it very hard to impossible to debug which layer is buggy when things go sideways.

complexity in software deployments

Reduce complexity

cookiecaper16• 7 years ago

In my ideal world, it'd look a lot like SmartOS + NixOS, but that's an ideal. There is a massive middle ground between k8s and some hypothetical ideal, and k8s is completely on the "horrifying monstrosity that you shouldn't touch with a ten-foot pole unless you really have no other options" side of things.

Most server-grade operating systems include facilities that are robust, mature, compact, performant, and reasonably well-integrated, and for the things that aren't part of the OS, there is a long and glorious lineage of applications that can lay claim to those same virtues. Kubernetes makes use of many of them to do its work.

Those of us who've configured a router, load balancer, or application server independently are just perplexed when someone acts like k8s is the only way to handle these very common concerns. We're left asking "Yeah, but... why all this, when I could've just configured [nginx/Apache/haproxy/iptables/fstab]?"

The naive admin will say "because then you just have to configure Kubernetes!", but unfortunately, stacking more moving parts on top of a complex system typically hurts more than it helps. You'll still need to learn the underlying systems to understand or troubleshoot what your cluster is doing -- but then, I think part of what Google et al are going for here is that instead of that, you'll just rent a newer and bigger cluster. And I guess there's no better way to ensure that happens than to keep the skillset out of the hoi polloi's hands.

I assume that many "devops" people are coming from non-nix backgrounds, and therefore take k8s as a default because they're new to the space and it's a hot ticket that Google has lavishly sworn will make you blessed with G-Glory. But systems have been running high-traffic production workloads for a very long time. Load balancing, failover, and host colocation have all been occurring at most shops for +/- 20 years before k8s released in 2014. These aren't new problems.

Alan Kay has called compsci "half a field" because we're just continually ignoring everything that's been done before and reinventing the wheel rather than studying and iterating upon our legacy and history. If anything is the embodiment of that, it's Kubernetes.

pjmlp5• 7 years ago

Docker and Kubernetes are the new NoSQL. Just give it two or three years.

elsonrodriguez5• 7 years ago

The desire and use case for a universal application package/runtime and an API to run that packaged app are not ever going away.

threeseed4• 7 years ago

NoSQL products like DynamoDB, Cassandra, Redis, MongoDB etc have all been growing massively over the last few years as companies move to an increasingly multi-technology mix.

So I also agree that Docker and Kubernetes are going to significantly grow as well.

brugidou• 7 years ago

What makes this even weirder is that all of the FAANG companies are using their own in-house-built platform instead of Kubernetes. Even Google.

pojzon• 7 years ago

And after they are done/switch to something better, they release papers about old tech. Thats how Kubernetes was born...

Tho you really cant stop people from jumping the hype.

kinghajj• 7 years ago

Isn't Google using Kubernetes internally on green field projects?

hinkley4• 7 years ago

I gotta say, I feel like I was having a much better time of it when Continuous Delivery was a logical outgrowth of Continuous Integration.

You just keep building farther and farther out from compilation through testing and packaging and installation and deployment. Everyone on the Dev team can keep up with the changes and understands where there code goes and when. That's what I thought DevOps was about.

Now it's a separate division that treats me like a mushroom: kept in the dark and fed bullshit.

tepidandroid3• 7 years ago

I actually agree with most of this. However, one must recognize that Google is not solely responsible for Kubernetes at this point. Under the banner of the CNCF, K8s is worked on by literally hundreds of companies and thousands of developers whose interests are not necessarily all aligned with the "Rent-Your-Server-From-Me" lock-in doctrine espoused by the large cloud providers. Many companies have an interest in making K8s easier to use from an operational perspective, which gives me hope for the platform's future.

antod• 7 years ago

After a couple of years of massive frustration with the entire direction of the "devops" segment,...
Kubernetes represents the complete "operationalization" of the devops space.

It's funny that I have all the exact same concerns and misgivings as you, but the exact opposite feeling (ie reactionary bias) of where it has come from :)

To me in my less charitable curmudgeonly cynical moments (using when tracking down a Docker or Kubernetes problems), it feels like "ops" was overrun by talented but inexperienced devs adding too many layers of abstraction too quickly CADT style funded by large valley tech companies and VC money. And constantly changing them as fashions come and go. It's like the React ecosystem having a go at Go.

For all the faults of the culture of slower moving traditional systems software written in C etc, we didn't run into so many issues with the level of bugginess or with tooling that gets abandoned/replaced before we've even finished evaluating it. It's like Go has done for systems software quality what PHP did for web software.

To be fair Docker is maturing now, and Kubernetes itself has too now, but the gap between the now relatively stable low level functionality of core kubernetes and what app developers need at a higher level is a gulf of churning crap.

indalo• 7 years ago

my goodness, breathe.

In my experience I haven't had that much difficulty finding problems that k8s solves and what problems it doesnt, for me. I eased into k8s using rancher and groked stuff as I went along, a little ansible and you can go a very long way if you want to use your own hardware.

Not everything has to be run like dedicated servers in a closet named after lion king characters that you SSH into every day "just to check", and keeping software up to date is the same its ever been, pay attention and apply patch, lol.

miff2000• 7 years ago

Some really interesting issues that hopefully, as a new Kubernetes Cluster maintainer, I'll now be able to avoid!

hjacobs4• 7 years ago

Thanks, feel free to reach out on Twitter (https://twitter.com/try_except_/) --- we are always happy to share experiences on running K8s.

cryptica• 7 years ago

This is just propaganda to drive companies away from Kubernetes so that they use expensive hosted 'lock-in' PaaS solutions instead of free, open source, no lock-in solutions.

elsonrodriguez5• 7 years ago

The tone of the repo doesn't really scream propaganda to me. It just looks like a solid list of failures and what people can learn from them.

eeZah7Ux3• 7 years ago

No lock-in in Kubernetes? Hah, try forking it.

hjacobs4• 7 years ago

As the author of the repo, I made it pretty clear that it's not about driving people away from Kubernetes. Please see my posts and talks about it, e.g. https://kubernetespodcast.com/episode/038-kubernetes-failure... and https://srcco.de/posts/why-kubernetes.html

tthisk• 7 years ago

In my opinion many of these issues with Kubernetes arise because it is not the right abstraction for application developers. You are forced to think about too many low level details to get a stable application running.

Many companies forget that they are incharge of getting way more intricate details of the application runtime right when they shift their application to Kubernetes.

In some of my recent experience I have seen a company shift an application from app engine to Kubernetes. However they forgot that appengine is using an optimized jvm to run your application. When they deployed to Kubernetes their app grinded to a halt. They forgot about all the observability that comes out of the box in app engine, thus resulting in an unobservable badly performing application.

Hopefully more domain specific runtimes build on top of Kubernetes can help application developers deploy to a kubernetes cluster in a sane way. I am putting some of my money on knative, however it is still quite hard to convince clients to invest in this area.

ma2rten• 7 years ago

What do you mean by "observability"?

FridgeSeal2• 7 years ago

Generally, things that answer questions along the lines of:

* is my app running * how long for? * if there are associated jobs/worker processes, did they start/complete successfully and how long did they run for * what’s the state of the pod(s) + host instanced my application is running on. * are all the necessary sub components up and running and communicating properly (and are they staying up).

It’s not quite monitoring/logging, although they are related.

threeseed4• 7 years ago

Not sure what any of this means.

Kubernetes does not prevent you from using an optimized JVM to run your application. In fact it doesn't even involve itself at that level rather it focuses just on managing containers which can have anything inside them.

And Kubernetes actually has far more observability than most distributed applications since you can get plugins which trace/monitor every connection between containers as well as monitor the health of each container.

I think what you're after is a PaaS on top of Kubernetes ? Well there are many of those around as well.

glennpratt• 7 years ago

Yes, you can get a bunch of pieces that need a team to evaluate and operate. You are saying the same thing and somehow making it a disagreement?

grantlmiller• 7 years ago

I'm torn on this list. It is important to learn from the mistakes from others, so I like it (postmortems are great for this reason), BUT it feels like folks are using these examples as reasons to stay away from Kubernetes. There could be a significantly larger list of system failures where K8s is not involved. Similarly there could probably be a list of "Encryption Failure Stories" but that doesn't mean we shouldn't encrypt things.

As an industry, one of the things we do pretty well is identify the most viable patterns to solve a problem and then develop and adopt the best primitives of those patterns. This is what Kubernetes is for creating reliable, scalable, distributed systems.

eeZah7Ux3• 7 years ago

k8s does not solve any problem around making your system distributed, scalable and reliable.

You still have to implement your own architecture to implement quorum, leader election, failover etc where needed.

k8s will only redeploy containers when a node fail.

geezerjay3• 7 years ago

k8s will only redeploy containers when a node fail.

That's not true at all. Kubernetes monitors containers' health and automatically scales deployments according to the demand. Thus quite obviously kubernetes does help make your system distributable, scalable and reliable. In fact, that was the design goal of kubernetes.

pepemon3• 7 years ago

There are distributed primitives in Kubernetes (such as leader lock), so you won't have to roll your own "architecture". For example, look at Patroni, it can be run natively on Kube, using its resources for all distributed and redundancy goodness.

msoad• 7 years ago

You know what's worse than Kubernetes complexity? A shallow abstraction of it by some mediocre team to get promotion and leave everyone else struggle with something that you can't get help from anywhere else. This happened in past two unicorns that I worked for. They template the templates and wrapped the `kubectl` with some leaky abstraction.

At least raw Kubernetes is Google-able!

hjacobs4• 7 years ago

We try to keep it simple at Zalando, i.e. just using Mustache templating and wrapping kubectl to add extra commands. See https://www.slideshare.net/try_except_/developer-experience-...

erikrothoff• 7 years ago

I got enough of random 503’s, network errors, slowdowns on mid-to-high traffic and dropped TCP connections yesterday. I setup dedicated VPS’s for our main app servers using Ansible and good ol’ fashion Capistrano. It feels almost old-school, but so much more stable.

odiroot• 7 years ago

These are all very informative and even amusing. But this is to be expected. Kubernetes is an enormous system (or ecosystem of systems actually). It's really hard to understand all the pieces and components, even the built-ins.

I still struggle (after having set up 2 clusters from scratch) with understanding networking, especially Ingresses and exposing things to the public internet.

neya• 7 years ago

In one of my ex-companies I worked with, we had a newbie coder who was just getting fascinated by Kubernetes. Unfortunately, every project this person touched would be left half-way by him and he'd move on to another project.

This company invested around $1 million into a very new Saas product that had a lot of potential. Now, I'm an old school guy. I like systems where I don't touch devops. The cost for such Paas is usually high, but it saves a lot of my time, which I value more than money (and it is valuable, in many cases). I advised the management to use something like AppEngine which is freakin' awesome for such kind of large scale projects.

Unfortunately, this newbie coder was management's pet. They bit the bullet and went on with his advice to use Kubernetes instead. The system randomly failed, they spent tons of time doing devops and microservices on what should have been a simplified monolith. The development time elongated to almost 2 years. By this time, there were competing offerings in the market for much cheaper.

This costed an entire team's morale, which lead to missed deadlines and product launches. This lead to many missed opportunities. For instance, there was a HUGE client we were demo'ing this Saas product. It failed spectacularly on a relatively light load (I've hosted Rails apps on AppEngine that can handle FAR better) and we lost that product and the client.

It was still not too late and I insisted on management identifying the root cause to switch to a managed Paas stack given how resource constrained we were. But this newbie coder turned it into a political situation ('mine vs yours'). I tried various ways to resolve this, but it didn't work out.

As a result, I said 'fuck this, I'm out of here' and I quit. In less than 3 months of my departure, about 10 people quit including one of the senior managers. In about 6 months, the company shrunk from a double digit to single digit company. The company almost went into bankruptcy.

The company lost its $1 million investment. Everyone left. The product development was put on indefinite hold. Finally, this newbie coder left as well. The founders had to rebuild their companies from scratch.

Co-incidentally, I went on to become an AppEngine consultant. I'm able to run unbelievably large monoliths in production, with almost zero downtime for 3 years in a row simply because I chose to avoid devops, and most importantly infrastructure complexity. I pay more, but it easily is worth my ROI. Most of the time, microservices aren't required and we can get away with a simplified monolith. And if you can get away with monoliths, you probably don't need Kubernetes.

orthecreedence3• 7 years ago

if you can get away with monoliths, you probably don't need Kubernetes

Right, especially if you're doing microservices the "right way" which means, each service has not only its own app setup but its own data store. I am responsible for (among other things) maintaining and scaling my company's infrastructure, and while we do have some vague concept of microservices (app, api (monolith), queuing system, various ML apis, headless browser, etc) it is always always ALWAYS "what's the simplest way to do this?" Microservices are a ton of moving parts, and adding in their own data stores is another can of worms.

We have a need for a scheduler. We are using Nomad internally. I chose this over Kubernetes. Why? I know that Kubernetes has more muscle behind it, can do more "things" we might need in the future, etc. I chose it because of its operational simplicity. It strikes a perfect balance of "scale these three things up" without needing a dedicated ops team to run it.

So far it has performed great, and I really think it comes down to limiting the number of moving pieces. Don't get me wrong, microservices are great if you have an ops team and a 50+ dev team all working on different things. But I often see microservices being pushed without warning people of the inherent costs involved.

I'm sure kubernetes is great, and I'm continuously re-evaluating where we are and what we need, but for now simpler is always better. If we do go with Kube, it will likely be a managed version so I don't have to touch the internals...not because I don't want to learn them (I definitely would want to know all the grimy details) but because I just don't have the fucking time to deal with it.

ellius• 7 years ago

I think programming as a profession provides three main services:

1. Implementing user/client features (meeting demand)

2. Designing and maintaining an architecture flexible enough to continue adding features at a reasonable pace

3. Reducing complexity, which primarily involves picking the right primitives and reducing the number of variables as much as possible

2 and 3 are really just in service of 1, but the fact is that if you don't invest in flexibility/leverage/optionality and proactively keep yourself from getting swamped by an overwhelming number of "moving parts," your productivity will creep to a halt and formerly cost-effective features will no longer be economical, destroying your capability to provide customer value.

StavrosK4• 7 years ago

How do you use Nomad? Do you run Docker containers with it? I don't need the scaling, I need the part where new versions of code are autodeployed automatically, services are connected together, and basically things update without having to run ops checklists every time.

Kubernetes feels too heavyweight and deploying machines feels too snowflakey...

StreamBright5• 7 years ago

Devops is not really what you just described. Using the wrong tools is the problem here and tool driven development and some blind-sighted management.

pepemon3• 7 years ago

I can't see how Kubernetes belongs to this problem. It's just the tool for getting specific things done, not some kind of silver bullet. You can build your application layer around Kubernetes, leveraging its primitives and integrating your services tightly, or you can leave it as be (wow). I currently maintain few clusters with managed control plane for hosting some monolith apps. Just a couple of fat worker nodes for failover, ingress with nice metrics and benefits of containerization... and that's it. No need to push for fully decoupled microservice architecture with service mesh that eats more CPU than your app (sarcasm, but not so far from the reality). Just use your tool right.

emmelaich• 7 years ago

Awesome. I'd like to see a list for other related products.

Such as Openstack. I bet there's a few stories there.

rurban• 7 years ago

So the most used tags are: OOMKill, CoreDNS, ndots:5, NotReady nodes.

I'd seriously consider disabling the OOMKiller. Not sure about the DNS issues.

hjacobs4• 7 years ago

As the owner of the linked GitHub repo (also rendered on https://k8s.af --- thanks to Joe Beda), I highly encourage everyone to contribute their failure stories (I'm still looking for the first production service mesh failure story..).

Also be aware of availability bias: Kubernetes enables us to collect failure stories in a (more or less) consistent way, this was previously not easily possible (think about on-premise failures, other fragmented orchestration frameworks, etc) --- I'm pretty sure there are much more failure stories in total about other things (like enterprise software), but we will never hear about them as they are buried inside orgs..

BTW, I also have a small post on why I think Kubernetes is more than just a "complex scheduler": https://srcco.de/posts/why-kubernetes.html

Kubernetes Failure Stories

Related Stories