How we migrated onto K8s in less than 12 months

https://www.figma.com/blog/migrating-onto-kubernetes/

by ianvonseggern • 2 years ago

298 385 2 years ago

figma.com

jb1991• 2 years ago

Can anyone advise what is the most common language used in enterprise settings for interfacing with K8s?

JohnMakin7• 2 years ago

IME almost exclusively golang.

roshbhatia2• 2 years ago

++, most controllers are written in go, but there's plenty of client libraries for other languages.

A common pattern you'll see though is skipping writing any sort of code and instead using a higher level dsl-ish configuration usually via yaml, using tools like Kyverno.

angio2• 2 years ago

I'm seeing a lot of custom operators written in Rust nowadays. Obviously biased because I do a lot of rust myself so people I'm talking to also do rust.

gadflyinyoureye• 2 years ago

Depends on what you mean. Helm will control a lot. You can make the yaml file in any language. Also you can admin it from command line tools. So again any language but often zsh or bash.

cortesoft• 2 years ago

A lot of yaml

yen223• 2 years ago

The fun kind of yaml that has a lot of {{ }} in them that breaks your syntax highlighter.

mplewis2• 2 years ago

I have seen more Terraform than anything else.

akdor1154• 2 years ago

On the platform consumer side (app infra description) - well schema'd yaml, potentially orchestrated by helm ("templates to hellish extremes") or kustomize ("no templates, this is the hill we will die on").

On the platform integration/hook side (app code doing specialised platform-specific integration stuff, extensions to k8s itself), golang is the lingua franca but bindings for many languages are around and good.

bithavoc• 2 years ago

If you’re talking about connecting to Kubernetes and create resources programmatically, Pulumi allows you to interface with it from all the languages they support(js, ts, go, c#, python) including wrapping up Helm charts and inject secrets( my personal favorite).

If you want to build your own Kubernetes Custom Resources and Controllers, Go lang works pretty well for that.

JohnMakin7• 2 years ago

I like how this article clearly and articulately states the reasons it gains to benefit from Kubernetes. Many make the jump without knowing what they even stand to gain, or if they need to in the first place - the reasons given here are good.

nailer2• 2 years ago

I was about to write the opposite - the logic is poor and circular - but multiple other commenters have already raised this: https://news.ycombinator.com/item?id=41194506 https://news.ycombinator.com/item?id=41194420

JohnMakin7• 2 years ago

I don't really see those rebuttals as all that valid. The reasons given in this article are completely valid, from my perspective of someone who's worked heavily with Kubernetes/ECS.

Helm, for instance, is a great time saver for installing software. Often software will support nothing but helm. Ease of deployment is a good consideration. Their points on networking are absolutely spot on. The scaling considerations are spot on. Killing/isolating unhealthy containers is completely valid. I could go on a lot more, but I don't see a single point listed as invalid.

samcat1162• 2 years ago

They're quite specific in that they mention that teams would like to make use of existing helm charts for other software products. Telling them to build and maintain definitions for those services from scratch is added work in their mind.

dijksterhuis2• 2 years ago

When applied, Terraform code would spin up a template of what the service should look like by creating an ECS task set with zero instances. Then, the developer would need to deploy the service and clone this template task set [and do a bunch of manual things]
This meant that something as simple as adding an environment variable required writing and applying Terraform, then running a deploy

This sounds less like a problem with ECS and more like an overcomplication in how they were using terraform + ECS to manage their deployments.

I get the generating templates part for verification prior to live deploys. But this seems... dunno.

wfleming• 2 years ago

Very much agree. I have built infra on ECS with terraform at two companies now, and we have zero manual steps for actions like this, beyond “add the env var to a terraform file, merge it and let CI deploy”. The majority of config changes we would make are that process.

dijksterhuis2• 2 years ago

Yeah.... thinking about it a bit more i just don't see why they didn't set up their CI to deploy a short lived environment on a push to a feature branch.

To me that seems like the simpler solution.

roshbhatia2• 2 years ago

I'm with you here -- ECS deploys are pretty painless and uncomplicated, but I can picture a few scenarios where this ends up being necessary, for ex; if they have a lot of services deployed on ECS and it ends up bloating the size of the Terraform state. That'd slow down plans and applies significantly, which makes sharding the Terraform state by literally cloning the configuration based on a template a lot safer.

freedomben2• 2 years ago

ECS deploys are pretty painless and uncomplicated

Unfortunately in my experience, this is true until it isn't. Once it isn't true, it can quickly become a painful blackbox debugging exercise. If your org is big enough to have dedicated AWS support then they can often get help from engineers, but if you aren't then life can get really complicated.

Still not a bad choice for most apps though, especially if it's just a run-of-the-mill HTTP-based app

ianvonseggern2• 2 years ago

Hey, author here, I totally agree that this is not a fundamental limitation of ECS and we could have iterated on this setup and made something better. I intentionally listed this under work we decided to scope into the migration process, and not under the fundamental reasons we undertook the migration because of that distinction.

Aeolun2• 2 years ago

Honestly, I find the reasons they name for using Kubernetes flimsy as hell.

“ECS doesn’t support helm charts!”

No shit sherlock, that’s a thing literally built on Kubernetes. It’s like a government RFP that can only be fulfilled by a single client.

Carrok• 2 years ago

We also encountered many smaller paper cuts, like attempting to gracefully terminate a single poorly behaving EC2 machine when running ECS on EC2. This is easy on Amazon’s Elastic Kubernetes Service (EKS), which allows you to simply cordon off the bad node and let the API server move the pods off to another machine while respecting their shutdown routines.

I dunno, that seems like a very good reason to me.

watermelon03• 2 years ago

I assume that ECS Fargate would solve this, because one misbehaving ECS task would not affect others, and stopping it should still respect the shutdown routines.

aranelsurion• 2 years ago

To be fair there are many benefits of running on the platform that has the most mindshare.

Unless they are in this space competing against k8s, it’s reasonable for them if they want to use Helm charts, to move where they can.

Also, Helm doesn’t work with ECS, so doesn’t <50 other tools and tech from the CNCF map>.

cwiggs6• 2 years ago

I think what they should have said is "there isn't a tool like Helm for ECS" If you want to deploy a full prometheus, grafana, alertmanager, etc stack on ECS, good luck with that, no one has written the task definition for you to consume and override values.

With k8s you can easily deploy a helm chart that will deploy lots of things that all work together fairly easily.

JohnMakin7• 2 years ago

It's almost like people factor in a piece of software's tooling environment before they use the software - wild.

liveoneggs• 2 years ago

recipes and tutorials say "helm" so we need "helm"

vouwfietsman2• 2 years ago

Maybe its normal for a company this size, but I have a hard time following much of the decision making around these gigantic migrations or technology efforts because the decisions don't seem to come from any user or company need. There was a similar post from Figma earlier, I think around databases, that left me feeling the same.

For instance: they want to go to k8s because they want to use etcd/helm, which they can't on ECS? Why do you want to use etcd/helm? Is it really this important? Is there really no other way to achieve the goals of the company than exactly like that?

When a decision is founded on a desire of the user, its easy to validate that downstream decisions make sense. When a decision is founded on a technological desire, downstream decisions may make sense in the context of the technical desire, but do they make sense in the context of the user, still?

Either I don't understand organizations of this scale, or it is fundamentally difficult for organizations of this scale to identify and reason about valuable work.

WaxProlix2• 2 years ago

People move to K8s (specifically from ECS) so that they can use cloud provider agnostic tooling and products. I suspect a lot of larger company K8s migrations are fueled by a desire to be multicloud or hybrid on-prem, mitigate cost, availability, and lock-in risk.

timbotron• 2 years ago

there's a pretty direct translation from ECS task definition to docker-compose file

zug_zug• 2 years ago

I've heard all of these lip-service justifications before, but I've yet to see anybody actually publish data showing how they saved any money. Would love to be proven wrong by some hard data, but something tells me I won't be.

OptionOfT• 2 years ago

Flexibility was a big thing for us. Many different jurisdictions required us to be conscious of where exactly data was stored & processed.

K8s makes this really easy. Don't need to worry whether country X has a local Cloud data center of Vendor Y.

Plus it makes hiring so much easier as you only need to understand the abstraction layer.

We don't hire people for ARM64 or x86. We have abstraction layers. Multiple even.

We'd be fooling us not to use them.

fazkan• 2 years ago

This, most of it, I think is to support on-prem, and cloud-flexibility. Also from the customers point of view, they can now sell the entire figma "box" to controlled industries for a premium.

teyc• 2 years ago

People move to K8s so that their resumes and job ads are cloud provider agnostic. Peoples careers stagnate when their employers platform on a home baked tech, or on specific offerings from cloud providers. Employers find Mmoving to a common platform makes recruiting easier.

samcat1162• 2 years ago

I have a hard time following much of the decision making around these gigantic migrations or technology efforts because the decisions don't seem to come from any user or company need

I mean the blog post is written by the team deciding the company needs. They explained exactly why they can't easily use etcd on ECS due to technical limitations. They also talked about many other technical limitations that were causing them issues and increasing cost. What else are you expecting?

Flokoso• 2 years ago

Managing 500 or more VMS is a lot of work.

Aline the VM upgrade, auth, backup, log rotation etc.

With k8s I can give everyone a namespace, policies, volumes, have automatic log aggregation due to demon sets and k8s/cloud native stacks.

Self healing and more.

It's hard to describe how much better it is.

ianvonseggern2• 2 years ago

Hey, author here, I think you ask a good question and I think you frame it well. I agree that, at least for some major decisions - including this one, "it is fundamentally difficult for organizations of this scale to identify and reason about valuable work."

At its core we are a platform teams building tools, often for other platform teams, that are building tools that support the developers at Figma creating the actual product experience. It is often harder to reason about what the right decisions are when you are further removed from the end user, although it also gives you great leverage. If we do our jobs right the multiplier effect of getting this platform right impacts the ability of every other engineer to do their job efficiently and effectively (many indirectly!).

You bring up good examples of why this is hard. It was certainly an alternative to say sorry we can't support etcd and helm and you will need to find other ways to work around this limitation. This was simply two more data points helping push us toward the conclusion that we were running our Compute platform on the wrong base building blocks.

While difficult to reason about, I do think its still very worth trying to do this reasoning well. It's how as a platform team we ensure we are tackling the right work to get to the best platform we can. Thats why we spent so much time making the decision to go ahead with this and part of why I thought it was an interesting topic to write about.

felixgallo4• 2 years ago

I have a constructive recommendation for you and your engineering management for future cases such as this.

First, when some team says "we want to use helm and etcd for some reason and we haven't been able to figure out how to get that working on our existing platform," start by asking them what their actual goal is. It is obscenely unlikely that helm (of all things) is a fundamental requirement to their work. Installing temporal, for example, doesn't require helm and is actually simple, if it turns out that temporal is the best workflow orchestrator for the job and that none of the probably 590 other options will do.

Second, once you have figured out what the actual goal is, and have a buffet of options available, price them out. Doing some napkin math on how many people were involved and how much work had to go into it, it looks to me that what you have spent to completely rearchitect your stack and operations and retrain everyone -- completely discounting opportunity cost -- is likely not to break even in even my most generous estimate of increased productivity for about five years. More likely, the increased cost of the platform switch, the lack of likely actual velocity accrual, and the opportunity cost make this a net-net bad move except for the resumes of all of those involved.

Spivak• 2 years ago

we can't support etcd and helm and you will need to find other ways to work around this limitation

So am I reading this right that either downstream platform teams or devs wanted to leverage existing helm templates to provision infrastructure and being on ECS locked you out of those and the water eventually boiled over. If so that's a pretty strong statement about the platform effect of k8s.

vouwfietsman2• 2 years ago

Hi! Thanks for the thoughtful reply.

I understand what you're saying, the thing that worries me though is that the input you get from other technical teams is very hard to verify. Do you intend to measure the development velocity of the teams before and after the platform change takes effect?

In my experience it is extremely hard to measure the real development velocity (in terms of value-add, not arbitrary story points) of a single team, not to mention a group of teams over time, not to mention as a result of a change.

This is not necessarily criticism of Figma, as much as it is criticism of the entire industry maybe.

Do you have an approach for measuring these things?

wg03• 2 years ago

If you haven't broken down your software into 50+ different separate applications written in 15 different languages using 5 different database technologies - you'll find very little use for k8s.

All you need is a way to roll out your artifact to production in a roll over or blue green fashion after the preparations such as required database alterations be it data or schema wise.

javaunsafe2019• 2 years ago

But you do know which problems the k8s abstraction solves, right? Cause it has nothing to do with many languages nor many services but things like discovery, scaling, failover and automation …

imiric2• 2 years ago

All you need is a way to roll out your artifact to production in a roll over or blue green fashion after the preparations such as required database alterations be it data or schema wise.

Easier said than done.

You can start by implementing this yourself and thinking how simple it is. But then you find that you also need to decide how to handle different environments, configuration and secret management, rollbacks, failover, load balancing, HA, scaling, and a million other details. And suddenly you find yourself maintaining a hodgepodge of bespoke infrastructure tooling instead of your core product.

K8s isn't for everyone. But it sure helps when someone else has thought about common infrastructure problems and solved them for you.

mplewis2• 2 years ago

Yeah, all you need is a rollout system that supports blue-green! Very easy to homeroll ;)

friendly_deer• 2 years ago

Here's a theory about why at least some of these come about:

https://lethain.com/grand-migration/

lmm9• 2 years ago

For instance: they want to go to k8s because they want to use etcd/helm, which they can't on ECS? Why do you want to use etcd/helm? Is it really this important? Is there really no other way to achieve the goals of the company than exactly like that?

I'm no fan of Helm, but there are surprisingly few good alternatives to etcd (i.e. highly available but consistent datastores, suitable for e.g. the distributed equivalent of a .pid file) - Zookeeper is the only one that comes to mind, and it's a real pain on the ops side of things, requiring ancient JVM versions and being generally flaky even then.

tedunangst• 2 years ago

How long will it take to migrate off?

codetrotter• 2 years ago

It’s irreversible.

hujun• 2 years ago

depends on how much "k8s native" code you have, there are application designed to run on k8s which uses a lot of k8s api and also if you app already micro-serviced, it is also not straight forward to change it back

wrs2• 2 years ago

A migration with the goal of improving the infrastructure foundation is great. However, I was surprised to see that one of the motivations was to allow teams to use Helm charts rather than converting to Terraform. I haven’t seen in practice the consistent ability to actually use random Helm charts unmodified, so by encouraging its use you end up with teams forking and modifying the charts. And Helm is such a horrendous tool, you don’t really want to be maintaining your own bespoke Helm charts. IMO you’re actually better off rewriting in Terraform so at least your local version is maintainable.

Happy to hear counterexamples, though — maybe the “indent 4” insanity and multi-level string templating in Helm is gone nowadays?

smellybigbelly• 2 years ago

Our team also suffered from the problems you described of public helm charts. There is always something you need to customise to make things work on your own environment. Our approach has been to use the public helm chart as-is and do any customisation with `kustomize —enable-helm`.

BobbyJo4• 2 years ago

Helm is quite often the default supported way of launching containerized third-party products. I have works at two separate startups whose 'on prem' product was offered this way.

freedomben2• 2 years ago

Indeed. I try hard to minimize the amount of Helm we use, but a significant amount of tools are only shipped as Helm charts. Fortunately I'm increasingly seeing people provide "raw k8s" yaml, but it's far from universal.

cwiggs6• 2 years ago

Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not.

I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places.

Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.

wrs2• 2 years ago

I agree, I wouldn’t generate k8s from Terraform either, that’s just the alternative I thought the OP was presenting. But I’d still rather convert charts from Helm to pretty much anything else than maintain them.

stackskipton2• 2 years ago

Lack of Variable substitution in Kustomize is downright frustrating. We use Flux so we have the feature anyways, but I wish it was built into Kustomize.

tionate• 2 years ago

Re your kustomizen complaint, just create a complete env-specific ingress for each env instead of patching.

- it is not really any more lines - doesn’t break if dev upgrades to a different version of the resource (has happened before) - allows you to experiment with dev with other setups (eg additional ingresses, different paths etc) instead of changing a base config which will impact other envs

TLDR patch things that are more or less the same in each env; create complete resources for things that change more.

There is a bit of duplication but it is a lot more simple (see ‘simple made easy’ - rich hockey) than tracing through patches/templates.

3np• 2 years ago

You can deploy your Helm charts through Terraform, even. It's been several years sinc so the situation might have improved but last I worked this way it was OK except of state drift due togaps in the Helm TF provider. Still found it better then either by itself.

gouggoug• 2 years ago

Talking about helm - I personally have come to profoundly loathe it. It was amazing when it came out and filled a much needed gap.

However it is loaded with so many footguns that I spend my time redoing and debugging others engineers work.

I’m hoping this new tool called « timoni » picks up steam. It fixes pretty every qualm I have with helm.

So if like me you’re looking for a better solution, go check timoni.

roryokane• 2 years ago

Link: https://timoni.sh/

JohnMakin7• 2 years ago

It's completely cursed, but I've started deploying helm via terraform lately. Many people, ironically me included, find that managing deployments via terraform is an anti pattern.

I'm giving it a try and I don't despise it yet, but it feels gross - application configs are typically far more mutable and dynamic than cloud infrastructure configs, and IME, terraform does not likey super dynamic configs.

solatic5• 2 years ago

My current employer (BigCo) has Terraform managing both infra + deployments in Terraform, at (ludicrous) scale. It's a nightmare. The problem with Terraform is that you must plan your workspaces such that you will not exceed the best-practice amount of resources per workspace (~100-200) or else plans will drastically slow down your time-to-deploy, checking stuff like databases and networking that you haven't touched and have no desire to touch. In practice this means creating a latticework of Terraform workspaces that trigger each other, and there are currently no good open-source tools that support it.

Best practice as I can currently see it is to have Terraform set up what you need for continuous delivery (e.g. ArgoCD) as part of the infrastructure, then use the CD tool to handle day-to-day deployments. Most CD tooling then asks you to package your deployment in something like Helm.

chucky_z• 2 years ago

You can setup dependent stacks in CDKTF. It’s far from as clean as a standard TF DAG plan/apply but I’m having a lot of success with it right now. If I were actively using k8s at the moment I would probably setup dependent cluster resources using this method, e.g: ensure a clean, finished CSI daemon deployment before deploying a deployment using that CSI provider :)

mnahkies• 2 years ago

Whilst I'll agree that writing helm charts isn't particularly delightful, consuming them can be.

In our case we have a single application/service base helm chart that provides sane defaults and all our deployments extend from. The amount of helm values config required by the consumers is minimal, and there has been very little occasion for a consumer to include their own templates - the base chart exposes enough knobs to avoid this.

When it comes to third-party charts, many we've been able to deploy as is (sometimes with some PRs upstream to add extra functionality), and occasionally we've needed to wrap/fork them. We've deployed far more third-party charts as-is than not though.

One thing probably worth mentioning w.r.t to maintaining our custom charts is the use of helm unittest (https://github.com/helm-unittest/helm-unittest) - it's been a big help to avoid regressions.

We do manage a few kubernetes resources through terraform, including Argocd (via the helm provider which is rather slow when you have a lot of CRDs), but generally we've found helm chart deployed through Argocd to be much more manageable and productive.

brainzap2• 2 years ago

Helm charts are a bit painful but they have a few critical features. Atomic deploy that rolls back onfailure. Ability to generate the full kubernetes definition with helm template. Ability to print out all configuration values with description.

At our company we have all deployments wrapped into a flat helm chart with as little variables as possible. (I always have to fight for that because devs like to abstract helm 100 levels and end up with nothing)

andrewguy9• 2 years ago

I look forward to the blog post where they get off K8, in just 18 months.

surfingdino• 2 years ago

ECS makes sense when you are building and breaking stuff. K8s makes sense when you are mature (as an org).

xiwenc• 2 years ago

I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run their apps on VM’s?

elktown3• 2 years ago

I think some are just pretty sick and tired of the explosion of needless complexity we've seen in the last decade or so in software, and rightly so. This is an industry-wide problem of deeply misaligned incentives (& some amount of ZIRP gold rush), not specific to this particular case - if this one is even a good example of this to begin with.

Honestly, as it stands, I think we'd be seen as pretty useless craftsmen in any other field due to an unhealthy obsession of our tooling and meta-work - consistently throwing any kind of sensible resource usage out of the window in favor of just getting to work with certain tooling. It's some kind of a "Temporarily embarrassed FAANG engineer" situation.

cwiggs6• 2 years ago

I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA.

In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it.

I blame this logic on the low interest rates for a decade. I could be wrong.

bobobob420• 2 years ago

Any software engineer who thinks K8 is complex shouldn’t be a software engineer. It’s really not that hard to manage.

methodical2• 2 years ago

Fair point but I think the key point here is unnecessary complexity versus necessary complexity. Are zero-downtime deployments and load balancing unnecessary? Perhaps for a personal project, but for any company with a consistent userbase I'd argue these are a non-negotiable, or should be anyways. In a situation where this is the expectation, k8s seems like the simplest answer, or near enough to it.

darby_nine2• 2 years ago

It's some kind of a "Temporarily embarrassed FAANG engineer" situation.

FAANG engineers made the same mistake, too, even though the analogy implies comparative competency or value.

maayank• 2 years ago

It’s one of those technologies where there’s merit to use them in some situations but are too often cargo culted.

caniszczyk• 2 years ago

Hating is a sign of success in some ways :)

In some ways, it's nice to see companies move to use mostly open source infrastructure, a lot of it coming from CNCF (https://landscape.cncf.io), ASF and other organizations out there (on top of the random things on github).

tryauuum2• 2 years ago

For me it is about VMs. Feel uneasy knowing that any kernel vulnerability will allow a malicious code to escape the container and explore the kubernetes host

There are kata-containers I think, they might solve my angst and make me enjoy k8s

Overall... There's just nothing cool in kubernetes to me. Containers, load balancers, megabytes of yaml -- I've seen it all. Nothing feels interesting enough to try

stackskipton2• 2 years ago

vs the Application getting hacked and running lose on the VM?

If you have never dealt with, I have to run these 50 containers plus Nginx/CertBot while figuring out which node is best to run it, yea, I can see you not being thrilled about Kubernetes. For the rest of us though, Kubernetes helps out with that easily.

moduspol4• 2 years ago

For me personally, I get a little bit salty about it due to imagined, theoretical business needs of being multi-cloud, or being able to deploy on-prem someday if needed. It's tough to explain just how much longer it'll take, how much more expertise is required, how much more fragile it'll be, and how much more money it'll take to build out on Kubernetes instead of your AWS deployment model of choice (VM images on EC2, or Elastic Beanstalk, or ECS / Fargate, or Lambda).

I don't want to set up or maintain my own ELK stack, or Prometheus. Or wrestle with CNI plugins. Or Kafka. Or high availability Postgres. Or Argo. Or Helm. Or control plane upgrades. I can get up and running with the AWS equivalent almost immediately, with almost no maintenance, and usually with linear costs starting near zero. I can solve business problems so, so much faster and more efficiently. It's the difference between me being able to blow away expectations and my whole team being quarters behind.

That said, when there is a genuine multi-cloud or on-prem requirement, I wouldn't want to do it with anything other than k8s. And it's probably not as bad if you do actually work at a company big enough to have a lot of skilled engineers that understand k8s--that just hasn't been the case anywhere I've worked.

drawnwren3• 2 years ago

Genuine question: how are you handling load balancing, log aggregation, failure restart + readiness checks, deployment pipelines, and machine maintenance schedules with these “simple” setups?

Because as annoying as getting the prometheus + loki + tempo + promtail stack going on k8s is —- I don’t really believe that writing it from scratch is easier.

angio2• 2 years ago

I think you're just used to AWS services and don't see the complexity there. I tried running some stateful services on ECS once and it took me hours to have something _not_ working. In Kubernetes it takes me literally minutes to achieve the same task (+ automatic chart updates with renovatebot).

mountainriver2• 2 years ago

I hear a lot of comments that sound like people who used K8s years ago and not since. The clouds have made K8s management stupid simple at this point, you can absolutely get up and running immediately with no worry of upgrades on a modern provider like GKE

archenemybuntu• 2 years ago

Kubernetes itself is built around mostly solid distributed system principles.

It's the ecosystem around it which turns things needlessly complex.

Just because you have kubernetes, you don't necessarily need istio, helm, Argo cd, cilium, and whatever half baked stuff is pushed by CNCF yesterday.

For example take a look at helm. Its templating is atrocious, and if I am still correct, it doesn't have a way to order resources properly except hooks. Sometimes resource A (deployment) depends on resource B (some CRD).

The culture around kubernetes dictates you bring in everything pushed by CNCF. And most of these stuff are half baked MVPs.

---

The word devops has created expectations that back end developer should be fighting kubernetes if something goes wrong.

---

Containerization is done poorly by many orgs, no care about security and image size. That's a rant for another day. I suspect this isn't a big reason for kubernetes hate here.

solatic5• 2 years ago

I don't get the hate for Kubernetes in this thread. TFA is from Figma. You can talk all day long about how early startups just don't need the kind of management benefits that Kubernetes offers, but the article isn't written by someone working for a startup, it's written by a company that nearly got sold to Adobe for $20 billion.

Y'all really don't think a company like Figma stands to benefit from the flexibility that Kubernetes offers?

BobbyJo4• 2 years ago

Kubernetes isn't even that complicated, and first party support from cloud providers often means you're doing something in K8s inleu of doing it in a cloud specific way (like ingress vs cloud specific load balancer setups).

At a certain scale, K8s is the simple option.

I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.

JohnMakin7• 2 years ago

I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.

Maybe - people seem really gungho about serverless solutions here too

eutropia• 2 years ago

I guess the ones who quietly ship dozens of rails apps on k8s are too busy getting shit done to stop and share their boring opinions about pragmatically choosing the right tool for the job :)

bamboozled2• 2 years ago

Agreed, we're a small team and we benefit greatly from managed k8s (EKS). I have to say the whole ecosystem just continues to improve as far as I can tell and the developer satisfaction is really high with it.

Personally I think k8s is where it's at now. The innovation and open source contributions are immense.

I'm glad we made the switch. I understand the frustrations of the past, but I think it was much harder to use 4+ years ago. Now, I don't see how anyone could mess it up so hard.

MrDarcy2• 2 years ago

Kubernetes isn't even that complicated

I’ve been struggling to square this sentiment as well. I spend all day in AWS and k8s and k8s is at least an order of magnitude simpler than AWS.

What are all the people who think operating k8s is too complicated operating on? Surely not AWS…

chrischen• 2 years ago

There are also a lot of cog-in-the-machine engineers here that totally do not get the bigger picture or the vantage point from another department.

dorianmariefr• 2 years ago

ruby on rails is all you need

logifail3• 2 years ago

it's written by a company that nearly got sold to Adobe for $20 billion

(Apologies if this is a dumb question) but isn't Figma big enough to want to do any of their stuff on their own hardware yet? Why would they still be paying AWS rates?

Or is it the case that a high-profile blog post about K8S and being provider-agnostic gets you sufficient discount on your AWS bill to still be value-for-money?

jeffbee• 2 years ago

There are a lot of ex-Dropbox people at Figma who might have learned firsthand that bringing your stuff on-prem under a theory of saving money is an intensely stupid idea.

ozim• 2 years ago

They are preparing for next blog post in a year - „how we cut costs by xx% by moving to our own servers”.

hyperbolablabla• 2 years ago

I work for a company making ~$9B in annual revenue and we use AWS for everything. I think a big aspect of that is just developer buy-in, as well as reliability guarantees, and being able to blame Amazon when things do go down

NomDePlum• 2 years ago

Much bigger companies use AWS for very practical well thought out reasons.

Not managing procurement of hardware, upgrades, etc, and a defined standard operating model with accessible documentation and the ability to hire people with experience, and have to hire less people as you are doing less is enough to build a viable and demonstrable business case.

Scale beyond a certain point is hard without support and delegated responsibility.

tayo42• 2 years ago

There must be a prohibitively expensive upfront cost to buy enough servers to do this. Plus bringing in all the skill that doesn't exist that can stand up and run something like they would require.

I wonder if as time goes on that skill to use hardware is dissappearing. New engineers don't learn it, and the ones that slowly forget. I'm not that sharp on anything I haven't done in years, even if it's in a related domain.

j_kao• 2 years ago

Companies like Netflix with bigger market caps are still on AWS.

I can imagine the productivity of spinning up elastic cloud resources vs fixed data center resourcing being more important, especially considering how frequently a company like Figma ships new features.

manquer• 2 years ago

A valuation is a just headline number which have no operational bearing.

Their ARR in 2022 was around $400M-450M. Say the infra budget at a typical 10% would be $50M. While it is a lot of money, it is not build your hardware money, also not all of it would be compute budget. They also would be spending on other SaaS apps like say Snowflake etc to special workloads like with GPUs, so not all workloads would be in-house ready. I would surprised if their commodity compute/k8s is more than half their overall budgets.

It is lot more likely to slow product growth to focus on this now, especially since they were/are still growing rapidly.

Larger SaaS companies than them in ARR still find using cloud exclusively is more productive/efficient.

sangnoir2• 2 years ago

Why would they still be paying AWS rates?

They are almost certainly not paying sticker prices. Above a certain size, companies tend to have bespoke prices and SLAs that are negotiated in confidence.

ijidak• 2 years ago

It's a fair question.

Data centers are wildly expensive to operate if you want proper security, redundancy, reliability, recoverability, bandwidth, scale elasticity, etc.

And when I say security, I'm not just talking about software level security, but literal armed guards are needed at the scale of a company like Figma.

Bandwidth at that scale means literally negotiating to buy up enough direct fiber and verifying the routes that fiber takes between data centers.

At one of the companies I worked at, it was not uncommon to lose data center connectivity because a farmer's tractor cut a major fiber line we relied on.

Scalability might include tracking square footage available for new racks in physical buildings.

As long as your company is profitable, at anything but Facebook like scale, it may not be worth the trouble to try to run your own data center.

Even if the cloud doesn't save money, it saves mental energy and focus.

cwiggs6• 2 years ago

k8s is complex, if you don't need the following you probably shouldn't use it:

* Service discovery

* Auto bin packing

* Load Balancing

* Automated rollouts and rollbacks

* Horizonal scaling

* Probably more I forgot about

You also have secret and config management built in. If you use k8s you also have the added benefit of making it easier to move your workloads between clouds and bare metal. As long as you have a k8s cluster you can mostly move your app there.

Problem is most companies I've worked at in the past 10 years needed multiple of the features above, and they decided to roll their own solution with Ansible/Chef, Terraform, ASGs, Packer, custom scripts, custom apps, etc. The solutions have always been worse than what k8s provides, and it's a bespoke tool that you can't hire for.

For what k8s provides, it isn't complex, and it's all documented very well, AND it's extensible so you can build your own apps on top of it.

I think there are more SWE on HN than Infra/Platform/Devops/buzzword engineers. As a result there are a lot of people who don't have a lot of experience managing infra and think that spinning up their docker container on a VM is the same as putting an app in k8s. That's my opinion on why k8s gets so much hate on HN.

Osiris• 2 years ago

Those all seem important to even moderately sized products.

maccard5• 2 years ago

For anyone who thinks this is a laundry list - running two instances of your app with a database means you need almost all of the above.

The _minute_ you start running containers in the cloud you need to think of "what happens if it goes down/how do I update it/how does it find the database", and you need an orchestrator of some sort, IMO. A managed service (I prefer ECS personally as it's just stupidly simple) is the way to go here.

gunapologist992• 2 years ago

It's worth bearing in mind that, although any of these can be accomplished with any number of other products as you point out, LB and Horizontal Scaling, in particular, have been solved problems for more than 25 years (or longer depending on how you count)

For example, even servers (aka instances/vms/vps) with load balancers (aka fabric/mesh/istio/traefik/caddy/nginx/ha proxy/ATS/ALB/ELB/oh just shoot me) in front existed for apps that are LARGER than can fit on a single server (virtually the definition of horizontally scalable). These apps are typically monoliths or perhaps app tiers that have fallen out of style (like the traditional n-tier architecture of app server-cache-database, swap out whatever layers you like).

However, K8s is actually more about microservices. Each microservice can act like a tiny app on its own, but they are often inter-dependent and, especially at the beginning, it's often seen as not cost-effective to dedicate their own servers to them (along with the associate load balancing, redundant and cross-AZ, etc). And you might not even know what the scaling pain points for an app is, so this gives you a way to easily scale up without dedicating slightly expensive instances or support staff to running each cluster; your scale point is on the entire k8s cluster itself.

Even though that is ALL true, it's also true that k8s' sweet spot is actually pretty narrow, and many apps and teams probably won't benefit from it that much (or not at all and it actually ends up being a net negative, and that's not even talking about the much lower security isolation between containers compared to instances; yes, of course, k8s can schedule/orchestrate VMs as well, but no one really does that, unfortunately.)

But, it's always good resume fodder, and it's about the closest thing to a standard in the industry right now, since everyone has convinced themselves that the standard multi-AZ configuration of 2014 is just too expensive or complex to run compared to k8s, or something like that.

drdaeman• 2 years ago

For what k8s provides, it isn't complex, and it's all documented very well

I had a different experience. Some years ago I wanted to set up a toy K8s cluster over an IPv6-only network. It was a total mess - documentation did not cover this case (at least I have not found it back then) and there was a lot of code to dig through to learn that it was not really supported back then as some code was hardcoded with AF_INET assumptions (I think it's all fixed nowadays). And maybe it's just me, but I really had much easier time navigating Linux kernel source than digging through K8s and CNI codebases.

This, together with a few very trivial crashes of "normal" non-toy clusters that I've seen (like two nodes suddenly failing to talk to each other, typically for simple textbook reasons like conntrack issues), resulted in an opinion "if something about this breaks, I have very limited ideas what to do, and it's a huge behemoth to learn". So I believe that simple things beat complex contraptions (assuming a simple system can do all you want it to do, of course!) in the long run because of the maintenance costs. Yeah, deploying K8s and running payloads is easy. Long-term maintenance - I'm not convinced that it can be easy, for a system of that scale.

I mean, I try to steer away from K8s until I find a use case for it, but I've heard that when K8s fails, a lot of people just tend to deploy a replacement and migrate all payloads there, because it's easier to do so than troubleshoot. (Could be just my bubble, of course.)

YZF• 2 years ago

There are other out of the box features that are useful:

* Cert manager.

* External-dns.

* Monitoring stack (e.g. Grafana/Prometheus.)

* Overlay network.

* Integration with deployment tooling like ArgoCD or Spinnaker.

* Relatively easy to deploy anything that comes with a helm chart (your database or search engine or whatnot).

* Persistent volume/storage management.

* High availability.

It's also about using containers which mean there's a lot less to manage in hosts.

I'm a fan of k8s. There's a learning curve but there's a huge ecosystem and I also find the docs to be good.

But if you don't need any of it - don't use it! It is targeting a certain scale and beyond.

tbrownaw3• 2 years ago

k8s is complex, if you don't need the following you probably shouldn't use it:

I use it (specifically, the canned k3s distro) for running a handful of single-instance things like for example plex on my utility server.

Containers are a very nice UX for isolating apps from the host system, and k8s is a very nice UX for running things made out of containers. Sure it's designed for complex distributed apps with lots of separate pieces, but it still handles the degenerate case (single instance of a single container) just fine.

st3fan2• 2 years ago

If you don't need any of those things then your use of k8s just becomes simpler.

I find k8s an extremely nice platform to deploy simple things in that don't need any of the advanced features. All you do is package your programs as containers and write a minimal manifest and there you go. You need to learn a few new things, but the things you do not have to worry about that is a really great return.

Nomad is a good contender in that space but I think HashiCorp is letting it slowly become EOL and there are bascially zero Nomad-As-A-Service providers.

tracerbulletx• 2 years ago

Even on a small project it's actually better imo than tying everything to a platform like netlify or vercel. I have this little notepad app that I deploy to a two node cluster in a github action and its an excellent workflow. The k8s to get everything deployed, provision tls and everything on commit is like 150 lines of mostly boilerplate yaml, I could pretty easily make it support branch previews or whatever too. https://github.com/SteveCastle/modelpad

saaspirant• 2 years ago

Unrelated: What does _TFA_ mean here? Google and GPT didn't help (even with context)

solatic5• 2 years ago

The Featured Article.

(or, if you read it in a frustrated voice, The F**ing Article.)

osigurdson4• 2 years ago

Kubernetes is the most amazing piece of software engineering that I have ever seen. Most of the hate is merely being directed at the learning curve.

otabdeveloper46• 2 years ago

No, k8s is shit.

It's only useful for the degenerate "run lots of instances of webapp servers running slow interpreted languages" use case.

Trying to do anything else in it is madness.

And for the "webapp servers" use case they could have built something a thousand times simpler and more robust. Serving templated html ain't rocket science. (At least compared to e.g. running an OLAP database cluster.)

spmurrayzzz• 2 years ago

I agree with respect to admiring it from afar. I've gone through large chunks of the source many times and always have an appreciation for what it does and how it accomplishes it. It has a great, supportive community around it as well (if not a tiny bit proselytizing at times, which doesn't bother me really).

With all that said, while I have no "hate" for the stack, I still have no plans to migrate our container infrastructure to it now or in the foreseeable future. I say that precisely because I've seen the source, not in spite of it. The net ROI on subsuming that level of complexity for most application ecosystems just doesn't strike me as obvious.

hylaride4• 2 years ago

Not to be rude, but K8s has had some very glaring issues, especially early on when the hype was at max.

* Its secrets management was terrible, and for awhile it stored them in plaintext in etcd. * The learning curve was real and that's dangerous as there were no "best practice" guides or lessons learned. There are lots of horror stories of upgrades gone wrong, bugs, etc. Complexity leaves a greater chance of misconfiguration, which can cause security or stability problems. * It was often redundant. If you're in the cloud, you already had load balancers, service discovery, etc. * Upgrades were dangerous and painful in its early days. * It initially had glaring third party tooling integration issues, which made monitoring or package management harder (and led to third party apps like Helm, etc).

A lot of these have been rectified, but a lot of us have been burned by the promise of a tool that google said was used internally, which was a bit of a lie as kubernetes was a rewrite of Borg.

Kubernetes is powerful, but you can do powerful in simple(r) ways, too. If it was truly "the most amazing" it would have been designed to be simple by default with as much complexity needed as everybody's deployments. It wasn't.

globular-toast2• 2 years ago

I don't get the hate even if you are a small company. K8s has massively simplified our deployments. It used to be each app had it's own completely different deployment process. Could have been a shell script that SSHed to some VM. Who managed said VM? Did it do its own TLS termination? Fuck knows. Maybe they used Ansible. Great, but that's another tool to learn and do I really need to set up bare metal hosts from scratch for every service. No, so there's probably some other Ansible config somewhere that sets them up. And the secrets are stored where? Etc etc.

People who say "you don't need k8s" never say what you do need. K8s gives us a uniform interface that works for everything. We just have a few YAML files for each app and it just works. We can just chuck new things on there and don't even have to think about networking. Just add a Service and it's magically available with a name to everything in the cluster. I know how to do this stuff from scratch and I do not want to be doing it every single time.

shakiXBT4• 2 years ago

if you don't need High Availability you can even deploy to a single node k3s cluster. It's still miles better than having to setup systemd services, an Apache/NGINX proxy, etc. etc.

roydivision• 2 years ago

One thing I learned when I started learning Kubernetes is that it is two disciplines that overlap, but are distinct none the less:

- Platform build and management - App build and management

Getting a stable K8s cluster up and running is quite different to building and running apps on it. Obviously there is overlap in the knowledge required, but there is a world of difference between using a cloud based cluster over your own home made one.

We are a very small team and opted for cloud managed clusters, which really freed me up to concentrate on how to build and manage applications running on it.

smitelli2• 2 years ago

I can only speak for myself, and some of the reasons why K8s has left a bad taste in my mouth:

- It can be complex depending on the third-party controllers and operators in use. If you're not anticipating how they're going to make your resources behave differently than the documentation examples suggest they will, it can be exhausting to trace down what's making them act that way.

- The cluster owners encounter forced software updates that seem to come at the most inopportune times. Yes, staying fresh and new is important, but we have other actual business goals we have to achieve at the same time and -- especially with the current cost-cutting climate -- care and feeding of K8s is never an organizational priority.

- A bunch of the controllers we relied on felt like alpha grade toy software. We went into each control plane update (see previous point) expecting some damn thing to break and require more time investment to get the cluster simply working like it was before.

- While we (cluster owners) begrudgingly updated, software teams that used the cluster absolutely did not. Countless support requests for broken deployments, which were all resolved by hand-holding the team through a Helm chart update that we advised them they'd need to do months earlier.

- It's really not cheaper than e.g. ECS, again, in my experience.

- Maybe this has/will change with time, but I really didn't see the "onboarding talent is easier because they already know it." They absolutely did not. If you're coming from a shop that used Istio/Argo and move to a Linkerd/Flux shop, congratulations, now there's a bunch to unlearn and relearn.

- K8s is the first environment where I palpably felt like we as an industry reached a point where there were so many layers and layers on top of abstractions of abstractions that it became un-debuggable in practice. This is points #1-3 coming together to manifest as weird latency spikes, scaling runaways, and oncall runbooks that were tantamount to "turn it off and back on."

Were some of these problems organizational? Almost certainly. But K8s had always been sold as this miracle technology that would relieve so many pain points that we would be better off than we had been. In my experience, it did not do that.

ahoka3• 2 years ago

What would be the alternative?

breakingcups• 2 years ago

I feel so out of touch when I read a blog post which casually mentions 6 CNCF projects with kool names that I've never heard of, for gaining seemingly simple functionality.

I'm really wondering if I'm aging out of professional software development.

renewiltord• 2 years ago

Nah, there’s lots of IC work. It just means that you’re unfamiliar with one approach to org scaling: abstracting over hardware, logging, retrying handled by platform team.

It’s not the only approach so you may well be familiar with others.

twodave• 2 years ago

TL;DR because they already ran everything in containers. Having performed a migration where this wasn’t the case, the path from non-containerized to containerized is way more effort than going from containerized non-k8s to k8s.

_pdp_2• 2 years ago

In my own experience, AWS Fargate is easier, more secure and way more robust then running your K8S even with EKS.

watermelon03• 2 years ago

Do you mean ECS Fargate? Because you can use AWS Fargate with EKS, with some limitations.

_pdp_2• 2 years ago

Yes, ECS Fargate.

ko_pivot3• 2 years ago

I’m not surprised that the first reason they state for moving off of ECS was the lack of support for stateful services. The lack of integration between EBS and ECS has always felt really strange to me, considering that AWS already built all the logic to integrate EKS with EBS in a StatefulSet compliant way.

datatrashfire• 2 years ago

https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ec...

This was actually added beginning of the year. Definitely was on my most wanted list for a while. You could technically use EFS, but that’s a very expensive way to run anything IO intensive.

ko_pivot3• 2 years ago

This adds support for ephemeral EBS volumes. When a task is created a volume gets created, and when the task is destroyed, for whatever reason, the volume is destroyed too. It has no concept of task identity. If the task needs to be moved to a new host, the volume is destroyed.

julienmarie• 2 years ago

I personally love k8s. I run multiple small but complex custom e-commerce shops and handle all the tech on top of marketing, finance and customer service.

I was running on dedicated servers before. My stack is quite complicated and deploys were a nightmare. In the end the dread of deploying was slowing down the little company.

Learning and moving to k8s took me a month. I run around 25 different services ( front ends, product admins, logistics dashboards, delivery routes optimizers, orsm, ERP, recommendation engine, search, etc.... ).

It forced me to clean my act and structure things in a repeatable way. Having all your cluster config in one place allows you to exactly know the state of every service, which version is running.

It allowed me to do rolling deploys with no downtime.

Yes it's complex. As programmers we are used to complex. An Nginx config file is complex as well.

But the more you dive into it the more you understand the architecture if k8s and how it makes sense. It forces you to respect the twelve factors to the letter.

And yes, HA is more than nice, especially when your income is directly linked to the availability and stability of your stack.

And it's not that expensive. I lay around 400 usd a month in hosting.

maccard5• 2 years ago

Figma were running on ECS before, so they weren't just running dedicated servers.

I'm a K8S believer, but it _is_ complicated. It solves hard problems. If you're multi-cloud, it's a no brainer. If you're doing complex infra that you want a 1:1 mapping of locally, it works great.

But if you're less than 100 developers and are deploying containers to just AWS, I think you'd be insane to use EKS over ECS + Fargate in 2024.

epgui4• 2 years ago

I don’t know if it’s just me, but I really don’t see how kubernetes is more complex than ECS. Even for a one-man show.

mountainriver2• 2 years ago

This just feels like a myth to me at this point. Kubernetes isn’t hard, the clouds have made is so simple now that it’s in no way more difficult than ECS and is way more flexible

belter• 2 years ago

I run multiple small but complex custom e-commerce shops

How do you handle the lack of multi tenancy in Kubernetes?

xyst• 2 years ago

Of course there’s no mention of performance loss or gain after migration.

I remember when microservices architecture was the latest hot trend that came off the presses. Small and big firms were racing to redesign/reimplement apps. But most forgot they weren’t Google/Netflix/Facebook.

I remember end user experience ended up being _worse_ after the implementation. There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown. There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1). Company didn’t have an easy way to trace logs across different services (way before distributed tracing was a thing). Turns out only a specific condition would trigger the dependency loop (maybe, 1 in 100 requests?).

Good times. Also, job safety.

api2• 2 years ago

This is a very fad driven industry. One of the things you earn after being in it for a long time is intuition for spotting fads and gratuitous complexity traps.

makeitdouble• 2 years ago

I think that aspect is indirectly covered, as one of the main motivation was to get on a popular platform that helps hiring.

I agree on how it's technically a waste of time to pursue fads, but it's also a huge PITA to have a platform that good engineers actively try to avoid, as their careers would stagnate (even as they themselves know that it's half a fad)

keybored• 2 years ago

1. You have to constantly learn to keep up!

2. Fad-driven

I wonder why I don’t often see (1) critiqued on the basis of (2).

teleforce• 2 years ago

There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1).

The classic dependency loop example that you thought will never encounter again for the rest of your life after OS class

pram2• 2 years ago

The best part is when it’s all architected to depend on something that becomes essentially a single point of failure, like Kafka.

bamboozled2• 2 years ago

Not saying you're wrong, but what is your grand plan ? I've never seen anything perfect.

intelVISA• 2 years ago

Shhh, you're ruining the party!

arctek• 2 years ago

Isn't this somewhat better, at least when it fails it's in a single place?

As someone using Kafka, I'd like to know what the (good) alternatives are if you have suggestions.

lmm9• 2 years ago

At least Kafka can be properly master-master HA. How people ever got away with building massively redundant fault-tolerant applications that were completely dependent on a single SQL server, I'll never understand.

vergessenmir• 2 years ago

I'm sorry,but I can't tell if you're being serious or not since you commented without qualification.

One of the most stable system archictectures I've built was on Kafka AND it was running with microservices managed by teams across multiple geographies and time zones. It was one of the most reliable systems in the bank. There are situations where it isn't appropriate, which can be said for most tech e.g( K8S vs ECS vs Nomad vs bare metal)

Every system has failure characteristics. Kafka's is defined as Consistent and Available and your system architecture needs to take that into consideration. Also the transactionality of tasks across multiple services and process boundaries is important.

Let's not pretend that kubernetes (or the tech of the day) is at fault while completely ignoring the complex architectural considerations that are being juggled

jamesfinlayson• 2 years ago

There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown.

Yep - saw that at a company recently - something in AWS was running a little slower than usual which cascaded to cause massive failures. Dozens of people were trying to get to the bottom of it, it mysteriously fixed itself and no one could offer any good explanation.

spyspy• 2 years ago

Any company that doesn’t have some form of distributed tracing in this day and age is acting with pure negligence IMO. Literally flying blind.

jiggawatts6• 2 years ago

Even if the microservices platform is running at 1% capacity, it's guaranteed to have worse performance than almost any monolith architecture.

It's very rare to meet a developer who has even the vaguest notion of what an RPC call costs in terms of microseconds.

Fewer still that know about issues such as head-of-line blocking, the effects of load balancer modes such as hash versus round-robin, or the CPU overheads of protocol ser/des.

If you have an architecture that involves about 5 hops combined with sidecars, envoys, reverse proxies, and multiple zones you're almost certainly spending 99% to 99.9% of the wall clock time just waiting. The useful compute time can rapidly start to approach zero.

This is how you end up with apps like Jira taking a solid minute to show an empty form.

throwaway485403• 2 years ago

Are you talking about cloud Jira? I use it daily and it's very quick, even search results appear immediately...

randomdata• 2 years ago

It's very rare to meet a developer who has even the vaguest notion of what an RPC call costs in terms of microseconds.

To be fair, small time units are difficult to internalize. Just look at what happens when someone finds out that it takes tens of nanoseconds to call a C function in Go (gc). They regularly conclude that it's completely unusable, and not just in a tight loop with an unfathomable number of calls, but even for a single call in their program that runs once per day. You can flat out tell another developer exactly how many microseconds the RPC is going to add and they still aren't apt to get it.

It is not rare to find developers who understand that RPC has a higher cost than a local function, though, and with enough understanding of that to know that there could be a problem if overused. Where they often fall down, however, is when the tools and frameworks try to hide the complexity by making RPC look like a local function. It then becomes easy to miss that there is additional overheard to consider. Make the complexity explicit and you won't find many developers oblivious to it.

lmm9• 2 years ago

This is how you end up with apps like Jira taking a solid minute to show an empty form.

Nah. Jira was horrible and slow long before the microservice trend.

otabdeveloper46• 2 years ago

It's actually worse than what you said. In 2024 the network is the only resource we can't upgrade on demand. There are physical limits we can't change. (I.e., there are only so many wires connecting your machines, and any significant upgrade involves building your own data center.)

So really eventually we'll all be optimizing around network interfaces as the bottleneck.

beeandapenguin• 2 years ago

Extremely slow times - from development to production, backend to frontend. Depending on how bad things are, you might catch the microservice guys complaining over microseconds from a team downstream, in front of a FE dev who’s spent his week optimizing the monotonically-increasing JS bundles with code splitting heuristics.

Of course, it was because the client app recently went over the 100MB JS budget. Which they decided to make because the last time that happened, customers abroad reported seeing “white screens”. International conversion dropped sharply not long after that.

It’s pretty silly. So ya, good times indeed. Time to learn k8s.

dangus• 2 years ago

This article specifically mentions that they are not running microservices and has pretty clearly defined motivations for making the migration.

ec1096852• 2 years ago

Why would moving to Kubernered from ECS introduce performance issues?

They already has their architecture, largely, and just moved it over to K8s.

They even mention they aren’t a micro service company.

trhway• 2 years ago

there’s no mention of performance loss or gain after migration

to illustrate performance cost i usually ask people what ping they have say from one component/pod/service to another and to compare that value to what ping they'd get between 2 linux boxes sitting on that gorgeous 10Gb/40Gb/100Gb or even 1000Gb network that they are running their modern microservices architecture over.

ellieh• 2 years ago

I imagine because the article mentions:

More broadly, we’re not a microservices company, and we don’t plan to become one

dgb23• 2 years ago

It seems the "micro" implies that services are separated by high level business terms, like "payment" or "inventory" with each having their own databases instead of computational terms like "storage", "load balancing" or "preprocessing" etc.

Is this generally correct? How well is this term defined?

If yes, then I'm not surprised this type of design has become a target for frustration. State is smeared across system, which implies a lot of messaging and arbitrary connections between services.

That type of design is useful if you are an application platform (or similar) where you have no say in what the individual entities are, and actually have no idea what they will be.

But if you have the birds-eye view and implement all of it, then why would you do that?

rco87862• 2 years ago

Or cost increase or decrease!

crossroadsguy• 2 years ago

I see this in Android. Every few years (sometimes multiple times in a year) a new arch. becomes the fad and every TDH dev starts hankering for one way or the other on the lines "why are you not doing X..". Problem is, at immature firms (specially startups) the director and senior manager level leaders happily agree for the re-write i.e re-archs because they usually leave every season for the next place and they get to talk about that new thing which probably didn't last beyond their stay, or might have conked off before that.

And the "test cases" porn! Goodness! People want "coverage" and that's it. It's a box to tick, irrespective of whether those test cases and they way those are written are actually meaningful in anyway. There are more like "let's have dependency injection everywhere" charade.

alexpotato• 2 years ago

As often, Grug has a great line about microservices:

    grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too

    seem very confusing to grug

https://grugbrain.dev/#grug-on-microservices

kiesel• 2 years ago

Thanks for the link to this awesome page!

Escapado• 2 years ago

Thanks for that link, I genuinely laughed out loud while reading some of those points! Love the presentation and what a wonderful reality check I can’t agree with more.

tsss• 2 years ago

single micro service called by all of the other micro services

So they didn't do microservices correctly. Big surprise.

zbentley• 2 years ago

I mean, that pattern is pretty common in the micro service world. Services for things like authz, locking, logging/tracing, etc. are often centralized SPOFs.

There are certainly ways to mitigate the SPOFiness of each of those cases, but that doesn’t make having them an antipattern.

jknoepfler• 2 years ago

One of the points of a microservice architecture (on k8s or otherwise) is that you can easily horizontally scale the component that was under pressure without having to scale out a monolithic application... that just sounds like people being clueless, not a failure of microservice architecture...

okr• 2 years ago

And if it was a single application, how would that solved it? You still would have that loop, no?

Personally, i think, it does not have to be hundreds of microservices, basically each function a service. But i see it more as the web in the internet. Things are sometimes not reachable or overloaded. I think that is normal life.

jmdots• 2 years ago

Please just use it as a warehouse scale computer and don’t make mode groups into pets.

jokethrowaway• 2 years ago

In which universe migrating from docker containers in ECS to Kubernetes is an effort measured in years?

strivingtobe• 2 years ago

At the time we did not auto-scale any of our containerized services and were spending a lot of unnecessary money to keep services provisioned such that they could always handle peak load, even on nights and weekends when our traffic is much lower.

Huh? You've been running on AWS for how long and haven't been using auto scaling AT ALL? How was this not priority number one for the company to fix? You're just intentionally burning money at that point!

While there is some support for auto-scaling on ECS, the Kubernetes ecosystem has robust open source offerings such as Keda for auto-scaling. In addition to simple triggers like CPU utilization, Keda supports scaling on the length of an AWS Simple Queue Service (SQS) queue as well as any custom metrics from Datadog.

ECS autoscaling is easy, and supports these things. Fair play if you just really wanted to use CNCF projects, but this just seems like you didn't really utilize your previous infrastructure very well.

kachapopopow6• 2 years ago

It appears to me that people don't really understand kubernetes here.

Kubernetes does not mean microservices, it does not mean containerization and isolation, hell, it doesn't even mean service discovery most of the time.

The default smallest kubernetes installation provides you two things: kubelet (the scheduling agent) and kubeapi.

What do these two allow you to do? KubeApi provides an API to interact with kubelet instances by telling them to do with manifests.

That's all, that's all kubernetes is, just a dumb agent with some default bootstrap behavior that allows you to interact with a backend database.

Now, let's get into kubernetes default extensions:

- CoreDNS - linking service names to service addresses.

- KubeProxy - routing traffic from host to services.

- CNI(many options) - Networking between various service resources.

After that, kubernetes is whatever you want it to be. It can be something that you can use to spawn few test databases. Deploy an entire production-certified clustered databases. A full distributed fs with automatic device discovery. Deploy backend services if you want to take advantage of service discovery, autoscaling and networking. Or it can be something as small as deploying monitoring (such as node-exporter) to every instance.

And as a bonus, it allows you to do it from the comfort of your own local computer.

This article says that figma migrated necessary services to kubernetes to improve developer experience and clearly said that things that don't need to be kubernetes aren't. For all we know they still run their services in raw instances and only use kubernetes for their storage and databases. And to add to all of that, kubernetes doesn't care where it runs, which is a great way to increase competition between cloud providers lowering the costs for all.

tbrownaw3• 2 years ago

Is it very common to use it without containers?

otabdeveloper46• 2 years ago

It is impossible.

darby_nine2• 2 years ago

You can use VMs instead. I don't think the distinction matters very much though.

kachapopopow6• 2 years ago

I run it that way on my windows machines, the image is downloaded and executed directly.

This ties into a funny example: k8s manages my vm's via kubevirt, those then have a minimal k8s version installed that runs my jobs. The implementation simply mounts the extracted image to a virtual fs and executes it there, then deletes the file system.

lmm9• 2 years ago

Kubernetes absolutely means containerisation in practice. There is no other supported way of doing things with it. And "fake" abstraction where you pretend something is generic but it's actually not is one of the easiest ways to overcomplicate anything.

kachapopopow6• 2 years ago

If you disable security policy you and remount to pid 1 you escape any encapsulation. Or you can use a k8s implementation that just extracts the image and runs it.

But that's assuming you're running containerd or something similar. There are dozens of k8s implementations some as light as only providing you with manifests which then you have external schedulers which subscribe (called controllers) that execute on these manifests.

consteval8• 2 years ago

To be fair, at the small scales you're talking about (maybe 1-2 machines) systemd does the same stuff, just better with less complexity. And there's various much simpler ways to automate your deployments.

If you don't have a distributed system then personally I think k8s makes no sense.

kachapopopow6• 2 years ago

I did say with few machines it can be overkill, but when you have more than a dozen of 2-3 machines or 6+ machines it gets overwhelming really fast. Kubernetes in it's smallest form is around 50MiB of memory and 0.1cpu.

Too2• 2 years ago

How do you deploy to systemd? How do you run a container in systemd? Now you need a second and third system, perhaps Ansible and docker-compose, which is simple on the surface but quickly grows in complexity with home made glue to keep all the loose components together.

I agree that for a handful of pet-servers for a team with more existing Linux experience than k8s experience, this is a better starting point, because of the shorter learning curve. Just not kid ourselves that the end product has any less complexity, it's only a different skill set.

ec1096852• 2 years ago

At high-growth companies, resources are precious

Yeah, at those low growth companies, you have unlimited resources /s

datadeft• 2 years ago

Migrating onto Kubernetes can take years

What a heck am I reading? For who? I am not sure why companies even bother with such migrations. Where is the business value? Where is the gain for the customer? Is this one of those "L'art pour l'art" project that Figma does it just because they can?

xorcist• 2 years ago

It solves the "we have recently been acquired and have a lot of resources that we must put to use" problem.

kevstev• 2 years ago

FWIW... I was pretty taken aback by this statement as well- and also the "brag" that they moved onto K8s in less than a year. At a very well established firm ~30 years old and with the baggage that came with it, we moved to K8s in far less time- though we made zero attempt to move everything to k8s, just stuff that could benefit from it. Our pitch was more or less- move to k8s and when we do the planned datacenter move at the end of the year, you don't have to do anything aside from a checkout. Otherwise you will have to redeploy your apps to new machines or VMs and deal with all the headache around that. Or you could just containerize now if you aren't already and we take care of the rest. Most migrated and were very happy with the results.

There was plenty of services that were latency sensitive or in the HPC realm where it made no sense to force a migration though, and there was no attempt to force them to shoehorn in.

rayrrr• 2 years ago

Just out of curiosity, is there any other modern system or service that anyone here can think of, where anyone in their right might would brag about migrating to it in less than a year?

jjice• 2 years ago

It's a hard question to answer. Not all systems are equal in size, scope, and impact. K8s as a system is often the core of your infra, meaning everything running will be impacted. That coupled with their team constraints in the article make it sound like a year isn't awful.

One system I can think of off the top of my head is when Amazon moved away from Oracle to fully Amazon/OSS RDBMSs a while ago, but that was multi year I think. If they could have done it in less than a year, they'd definitely be bragging.

therealdrag0• 2 years ago

I’ve seen many migrations take over a year. It’s less about the technology and more about your tech debt, integration complexity, and resourcing.

05bmckay• 2 years ago

I don't think this is the flex they think it is...

ravedave5• 2 years ago

Completely left out of this post and most of the conversation is that being on K8 makes it much, much easier to go multi-cloud. K8 is k8.

syngrog66• 2 years ago

k8s and "12 months" -> my priors likely confirmed. ha

sjkoelle• 2 years ago

The title alone is a teensie bit hilarious

Ramiro3• 2 years ago

I love reading these "reports from the field"; I always pick up a thing or two. Thanks for sharing @ianvonseggern!

jstrong• 2 years ago

enjoyed the article - provides a comprehensive tour of the benefits of deploying in a tmux session (seamless restarts via ctrl-c, up, enter). for super important stuff, a systemd unit is worth considering.

How we migrated onto K8s in less than 12 months

Related Stories