De-cloud and de-k8s – bringing our apps back home

https://dev.37signals.com/bringing-our-apps-back-home/

by mike1o1 • 3 years ago

568 408 3 years ago

dev.37signals.com

acedTrex• 3 years ago

This seems like an application/stack that didn't have a valid need for k8s in the first place. Don't just use K8s because its what people say you should do. evaluate the pros and the VERY real cons and make an informed decision.

bdcravens4• 3 years ago

That's why we've had good results with ECS. Feels like 80% of the result for 20% of the effort, and I haven't found our use cases needing that missing 20%.

sgarland• 3 years ago

With EC2 launch types, probably. Setting up ECS for Fargate with proper IaC templates/modules isn't much easier than EKS, IMO.

birdyrooster2• 3 years ago

“Need” Eh, I do it because it’s awesome for a single box or thousands. Single sign on, mTLS everywhere, cert-manager, BGP or L2 VIPs to any pod, etc and I can expand horizontally as needed. It’s the best for an at home lab. I pity the people who only use Proxmox.

klabb32• 3 years ago

Even k3s used 500MB and 2-5% CPU for a single server node with 0 user space pods. That seems incredibly bloated to me.

guilhas• 3 years ago

How easier is mrsk vs k3s?

bdcravens4• 3 years ago

I'm not sure if anyone other than 37Signals is using it at scale yet, so you may get a better idea by looking at the docs yourself.

https://github.com/mrsked/mrsk

ilyt5• 3 years ago

Oh, YAML-based DSL to deploy stuff, how original!

Now we only need template-based generator for those YAMLs and we will have all worst practices for orchestration right here, just like k8s + helm

turtlebits3• 3 years ago

Looks like an apple vs oranges comparison. They seem to have a low number of distinct services, so there isn't a real need for k3s/k8s (ie orchestration), on the other hand, they need config management.

mike1o12• 3 years ago

To me the really surprising thing is that they still use Capistrano for deploying basecamp!

camsjams2• 3 years ago

Maybe because Capistrano is written in Ruby and the language matches their internal products? That was my only guess.

mbreese• 3 years ago

I was guessing that they kept using Capistrano because it still worked. No need to change something that’s working…

(Somewhat of an ironic comment when talking about an article about ditching K8s…)

NetOpWibby• 3 years ago

That's a name I haven't heard since I was deploying my Jekyll blog during my apprenticeship at Thoughtbot. Wow.

ocdtrekkie3• 3 years ago

De-clouding is going to be a huge trend as companies are pressured to save costs, and they realize on-prem is still a fraction of the cost of comparable cloud services.

This whole cloud shift has been one of the most mind-blowing shared delusions in the industry, and I'm glad I've mostly avoided working with it outright.

dopylitty• 3 years ago

The thing that gets me about it is the very real physical cost of all this cloud waste.

The big cloud providers have clear cut thousands of acres in Ohio, Northern VA, and elsewhere to build their huge windowless concrete bunkers in support of this delusion of unlimited scale.

Hopefully as the monetary costs become clear their growth will be reversed and these bunkers can be torn down

ocdtrekkie3• 3 years ago

For what it's worth, large providers will always need datacenters. But perhaps datacenters run by public cloud providers today will be sold off to larger businesses running their own infrastructure someday at a discount. Most of the infrastructure itself all will age out in five or ten years, and would've been replaced either way.

Heck, datacenters in Virginia are likely to end up being sold directly to the federal government.

adamsb6• 3 years ago

The big cloud providers are likely packing machines more densely and powering them more efficiently than alternatives like colos.

ilyt5• 3 years ago

They ain't going to be unused lmao. If migration happen they will just stop building new ones or have to compete harder on pricing.

rr808• 3 years ago

Our firm started the big cloud initiative last year. We have our own datacenters already, but all the cool startups used cloud. Our managers figure it'll make us cool too.

ocdtrekkie3• 3 years ago

This sort of thing is absolutely insane. Like, sure, small office, no existing datacenter infrastructure, it might make sense to bootstrap your business on someone else's cloud. But if you literally have a cooled room and an existing network infrastructure, it's absolutely silly to spend money on using someone else's.

icedchai• 3 years ago

On-prem has its own issues. Many small applications need little more than a VPS and a sane backup/recovery strategy.

robertcope• 3 years ago

Something I feel like these conversations seem to miss is that it is not binary; you don't have to host hardware on-prem if you don't want to be in AWS. There are other clouds. There are Sungards of the world were you can pay for racks of managed hardware. There are a lot of options between buying and managing your own hardware and AWS.

lloydatkinson2• 3 years ago

I've been following their move to on premise with interest and this was a great read. I'm curious how they are wiring up GitHub actions with their on premise deployment. How are they doing this?

The best I can think of for my own project is to run one of the self hosted GitHub actions runners on the same machine which could then run an action to trigger running the latest docker image.

Without something like that you miss the nice instant push model cloud gives you and you have to use the pull model of polling some service regularly for newer versions.

efxhoy• 3 years ago

They mentioned their mrsk tool sshes to the boxes to deploy so the action probably runs the tool and does just that?

mr_ndrsn6• 3 years ago

Negative. No external tool/company has ssh access. GHA is strictly for CI, which is decoupled from the actual deploy.

If we do decide to tie it in, it will be using the GH Deployment API to inform the local tool on CI status or something.

not_enoch_wise• 3 years ago

It's almost as if...

...kubernetes isn't the solution to every compute need...

klardotsh• 3 years ago

Tell that to the myriad of folks making their money off of peddling it. You'd swear it were the only tool available based on the hype circles (and how many hiring manager strictly look for experience with it).

illiarian• 3 years ago

Cloud Native Landscape.... https://landscape.cncf.io/images/landscape.pdf

It's more than just peddlers at this point. There are peddlers peddling to other peddlers, several layers deep.

ilyt5• 3 years ago

I gotta say from dev perspective it is very convenient solution. But I wouldn't recommend it to anyone that runs anything less complex than "a few services in a database". The tens of minutes you save in writing deploy scrips will be replaced by hours of figuring out how to do it k8s way.

From ops perspective let's say I ran it from scratch (as in "writing systemd units to run k8s daemons and setting up CA to feed them", because back then there was not that much reliable automation around deploying it) and the complexity tax is insane. Yeah you can install some automation doing that but if it ever breaks (and I've seen some breaking) good fucking luck, non-veteran will have better chance with reinstalling it from scratch.

birdyrooster2• 3 years ago

Except it was created to model virtually every solution to every compute need. It’s not about the compute itself, it’s about the taxonomy, composability, and verifiability of specifications which makes Kubernetes excellent substrate for nearly any computing model from the most static to the most dynamic. You find kubernetes everywhere because of how flexible it is to meet different domains. It’s the next major revolution in systems computing since Unix.

ranger207• 3 years ago

I (roughly) believe this as well[0], but more flexibility generally means more complexity. Right now, if you don't need the flexibility that k8s offers, it's probably better to use a solution with less flexibility and therefore less complexity. Maybe in a decade if k8s has eaten the world there'll be simple k8s-based solutions to most problems, but right now that's not always the case

[0] I think that in the same way that operating systems abstract physical hardware, memory management, process management, etc, k8s abstracts storage, network, compute resources, etc

aliasxneo• 3 years ago

Always two extremes to any debate. I've personally enjoyed my journey with it. I've even been in an anti-k8s company running bare metal on the Hashi stack (won't be running back to that anytime soon). I think the two categories I've seen work best are either something lik ECS or serverless and Kubernetes.

Melingo21• 3 years ago

I'm so lost on so many choices this company did.

You de cloud and now use some mini tool like mrskd?

I'm running k8s on azure (small), gke (very big), rke2 on a setup with 5 nodes and k3s.

I'm totally lost why they would de k8s after investing already so much time? They should be able to work with k8s really well at this point.

Sry to say but for me it feels the company has a much bigger issue than cloud vs non cloud: no one with proper experience and strategy.

drewda• 3 years ago

If one of the co-founders/owners is writing the devops tooling from scratch... well, that's a decision.

Not saying it's necessarily a bad decision. But it's potentially driven more by personal interests than a dispassionate and strategic plan.

dilyevsky2• 3 years ago

Means either the company is doing super well or about to fail, no in-between here

chologrande• 3 years ago

It does seem like they just moved all of their infra components, and got rid of autoscaling.

Load balancing, logging, and other associated components are all still there. Almost nothing changed in the actual architecture, just how it was hosted.

I have a hard time seeing why this was beneficial.

slackfan• 3 years ago

Cost, mostly.

Aeolun3• 3 years ago

I feel like this is the reason so many horrible kubernetes stacks exist.

llama0522• 3 years ago

Completely agreed with this. K8s is not that operationally hard. The concepts are all there and it's just giving you a framework for it essentially.

Many k8s operators out there to help you self host it. Abandoning the entire concept is just wild.

cryptonector• 3 years ago

I can understand declouding, but getting rid of Kubernetes seems like a lot of work for little gain.

xwdv• 3 years ago

They’re not thinking strategically, they’re thinking tactically.

jgilias• 3 years ago

This is very amusing to read!

This company invented Ruby on Rails and was in business before ‘cloud’ was a thing. Many things can be said about 37signals and DHH in particular, but lacking proper experience is definitely not one of those.

Melingo21• 3 years ago

I am very curious why they had such a bad k8s experience while mine is the total opposite.

And just inventing known stuff doesn't mean they are infra or platform experts.

I grew into cloud in the same time frame as they did and I'm also a platform expert and k8s expert.

So are they automatically better in deployment and platforms because they invented ruby on rails?

Melingo21• 3 years ago

And don't get me wrong: whatever works for the company but k8s experience alone is already super helpful.

A lightweight k8s stack out of the box + argocd + cert manager is like I fra steroids.

rektide3• 3 years ago

The whole kubernetes section of this writeup is two sentences. They went with a vendor provided kube & it was expensive & didn't go great.

It just sounds like it was poorly executed, mostly? There's enough blogs & YouTube of folk setting up HA k8s on a couple rpi, & even the 2GB model works fine if having not-quite-half the ram as overhead on apiservers/etcd nodes.

It's not like 37signals has hundreds of teams & thousands of services to juggle, so it' s not like they need a beefy control plane. I dont know what went wrong & there's no real info to guess by, but 37s seems like a semi-ideal easy lock for k8s on prem.

pferde3• 3 years ago

Or you could look at it like this:

It's not like 37signals has hundreds of teams & thousands of services to juggle, so it's not like they need k8s and the complexity it brings.

ethicalsmacker• 3 years ago

If only Ruby had real concurrency and the memory didn't bloat like crazy.... you wouldn't need 90% of the hardware.

wmf3• 3 years ago

https://m.signalvnoise.com/only-15-of-the-basecamp-operation...

sys_64738• 3 years ago

Larry Ellison was right.

electric_mayhem• 3 years ago

We used to call this a “sudden outbreak of common sense”

mfer• 3 years ago

I wonder how they are going to handle fault tolerance when machines go offline?

So as to avoid being paged in the middle of the night, I grew to really like automation that keeps things online.

turtlebits3• 3 years ago

Their app is running on at least 2 machines, so the load balancer takes care of it.

rad_gruchalski4• 3 years ago

Good for them. Now they have a one-off to manage themselves. It’s pretty easy to de-cloud using something like k3s. So much value added in Kubernetes to leverage. But they have Chef and they’re a Ruby shop, I guess they’ll be good.

TBH, Kubernetes has some really rough edges. Helm charts aren’t that great and Kustomize gets real messy real fast.

turtlebits3• 3 years ago

The scope of their self-developed tool doesn't seem very large, it looks like it could be a wrapper around SSH. I've done similar things using a SSH library with python to deploy and run docker-compose yamls on multiple servers.

rad_gruchalski4• 3 years ago

There are many of these tools out there. When I was working for Technicolor Virdata some years ago, we’ve been heavily invested in https://github.com/infochimps-labs/ironfan. It was extensible, we had support for SoftLayer and IBM SCE, we had some patches to make the bootstrap and the kick command perform faster. But it was still slow and people didn’t like Ruby (I don’t mind it).

Even back then I wasn’t a fan of doing a proactive ssh connection to the node. I always leaned towards the machine pulling artefacts and deploying them. Like Flux CD does.

klooney• 3 years ago

Chef kind of scares me. It's such a tarpit, and it felt like abandonware five years ago

camsjams2• 3 years ago

Is this cheaper though?

For a medium-to-large app, K8s should offset a lot of the operational difficulties. Also you don't have to use K8s.

Cloud is turn-on/turn-off, whereas on-premises you pay up front investment.

Here are all of the hidden costs of on-prem that folks forget about when thinking about cloud being "expensive":

- hardware

- maintenance

- electricity

- air conditioning

- security

- on-call and incident response

Here are all of the hidden time-consumers of on-prem that folks forget about when thinking about cloud being "difficult":

- os patching and maintenance

- network maintenance

- driver patching

- library updating and maintenance

- BACKUPS

- redundancy

- disaster recovery

- availability

sn0wf1re2• 3 years ago

Every one of your examples in the second list is relevant to both on-prem and cloud. Also cloud also has on-call, just not for the hardware issues (still likely get a page for reduced availability of your software).

vlunkr• 3 years ago

“Just not for the hardware issues” is a huge deal though. That’s an entire skillset you can eliminate from your requirements if you’re only in the cloud. Depending on the scale of your team this might be a massive amount of savings.

tstrimple2• 3 years ago

This is simply not true unless you build in the cloud the same way you build on prem and just have a bunch of VMs. PaaS services get you away from server / network / driver maintenance and handle disaster recovery and replication out of the box. If you're primarily using IaaS, you likely shouldn't be in the cloud unless you're really leveraging the bursting capabilities.

https://robertgreiner.com/content/images/2019/09/AzureServic...

hnlmorg2• 3 years ago

The problem here is “cloud” can mean different things.

If you’re taking about virtual machines running in a classical networking configuration then you’re not really leveraging “the cloud” — all you’ve done is shifted the location of your CPUs.

However if you’re using things like serverless, managed databases, SaaS, then most of the problems in the second list are either solved or much easier to solve in the cloud.

The problem with “the cloud” is you either need highly variable on-demand compute requirements or a complete re-architecture of your applications for cloud computing to make sense. And this is something that so many organisations miss.

I’ve lost count of the number of people who have tried to replicate their on-prem experience to cloud deployments and then came to the same conclusions as yourself. But that’s a little like trying to row a boat on land and then saying roads are a rubbish way to filter traffic. You just have to approach roads and rivers (or cloud and on-prem) deployments with a different mindset because they solve different problems.

andrewstuart• 3 years ago

This is the fiction that CTOs believe - "it's simply not practical to run your own computers, you need cloud".

ilyt5• 3 years ago

We have 7 racks, 3 people and actual hardware stuff is minuscule part of that. Few hundred VMs, anything from "just a software running on server" to k8s stack (biggest one is 30 nodes), 2 ceph cluster (our and clients), and a bunch of other shit

The stuff you mentioned is, amortized, around 20% (automation ftw). The rest of it is stuff that we would do in cloud anyway and cloud is in general harder to debug too (we have few smaller projects managed in cloud for customers.

We did calculation to move to cloud few times now, never was even close to profotable and we woudn't save on manpower anyway as 24/7 on-call is still required.

So I call bullshit on that.

If you are startup, by all means go cloud

If you are small, go ahead, not worth it.

If you have spiky load, cloud or hybrid will most likely be cheaper.

But if you have constant (by that I mean difference between peak and lowest traffic is "only" like 50-60%) load and need a bunch of servers to run it (say 3+ racks), it might actually be cheaper on-site.

Or a bunch of dedicated servers. Then you don't need to bother to manage hardware, and in case of boom can even scale relatively quickly

erulabs3• 3 years ago

Only one sentence about why they chose to abandon K8s:

It all sounded like a win-win situation, but [on-prem kubernetes] turned out to be a very expensive and operationally complicated idea, so we had to get back to the drawing board pretty soon.

It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? Sort of undercuts their main argument that this bit isn't even remotely fleshed out.

benatkin5• 3 years ago

Well to be fair Kubernetes doesn't always pluralize the names of collections, since you can run "kubectl get deployment/myapp". You don't want to do the equivalent of "select * from user" do you? That doesn't make any sense!!! And don't translate that to "get all the records from the user table"! That's "get all the records from the users table". (Rails defaults to plural, Django to singular for table names. Not sure about the equivalent for Kubernetes but in the CLI surprisingly you can use either)

mike1o12• 3 years ago

To be fair, from the article it says that they built the bulk of the tool and did the first migration in a 6-week cycle. mrsk looks fairly straight forward, and feels like Capistrano but for containers. The first commit of mrsk is only on January 7th of this year.

In less than a six-week cycle, we built those operational foundations, shaped mrsk to its functional form and had Tadalist running in production on our own hardware.

bastawhiz3• 3 years ago

They spent a month and a half building tooling _capable of handling their smallest application_, representing an extremely tiny fraction of their cloud usage.

imiric2• 3 years ago

Sometimes there's value in building bespoke solutions. If you don't need many of the features of the off-the-shelf solution, and find the complexity overwhelming and the knowledge and operational costs too high, then building a purpose-built solution to fit your use case exactly can be very beneficial.

You do need lots of expertise and relatively simple applications to replace something like k8s, but 37signals seems up to the task, and judging by the article, they picked their least critical apps to start with. It sounds like a success story so far. Kudos to them for releasing MRSK, it definitely looks interesting.

As a side note, I've become disgruntled at k8s becoming the defacto standard for deploying services at scale. We need different approaches to container orchestration, that do things differently (perhaps even rethinking containers!), and focus on simplicity and usability instead of just hyper scalability, which many projects don't need.

I was a fan of Docker Swarm for a long time, and still use it at home, but I wouldn't dare recommend it professionally anymore. Especially with the current way Docker Inc. is managed.

skor• 3 years ago

wouldn't dare recommend it professionally anymore

please provide more insight, thanks

kobalsky• 3 years ago

and focus on simplicity and usability instead of just hyper scalability

I don't get why people in IT need a boogeyman. Looks like "k8s is hard" is the new "pointers are hard" we use to scare other people now.

mikepurvis• 3 years ago

I'm happy with my usage of k8s, but I think it's unfortunate that current container abstractions are so oriented around imperative assembly in "layers". I want a way to run NixOS in a container and have it feel first class— existing approaches either require installing everything every time with no caching, or pre-building and manually lifecycling your container (streamLayeredImage), or knowing upfront what you're going to need (Nixery).

xp84• 3 years ago

simplicity and usability instead of just hyper scalability

This is such a key phrase here.

If I'm starting a small SaaS company tomorrow, my ideal for setting up infrastructure would be a stack which can for now look similar to what this article sets up (especially with the tremendously lower bills), but with an easy migration path to k8s, should I hit the jackpot and have that 'very good problem to have' of too many customer requests to handle.

My big issue with k8s, and honestly with other big fancy toolsets, is that getting started with it requires you to choose between:

- Hire several seasoned cloud orchestration experts, preferably with the platform you've chosen (AWS, GCP, Azure) who will know how to troubleshoot this beast when you have a mysterious issue, or: - YOLO it! Just follow the basic tutorials to set k8s up, and hope you don't end up sitting up all night with a site that's refusing connections while your customers flee.

The first one is the only responsible choice but it's going to add another half million to your cash burn, and that's on top of the high-margin "managed" service cloud bills like RDS.

So I can see why people are drawn to a system where instead of paying for k8s and "Postgres in a box" they can pay for a simple server and have simple tooling to deploy, back up, etc.

wankle5• 3 years ago

Especially with the current way Docker Inc. is managed.

I was reviewing GCP's Free Tier today, they have the same approach, if they need to change or drop services they agree to give 30 days notice, same as Docker did. It's probably common for other cloud companies offering free stuff as well. All the negative attention Docker received was fully and wholly undeserved.

jameshart5• 3 years ago

I think people overindex on thinking that Kubernetes is about scalability.

Honestly, its inbuilt horizontal scaling systems are pretty lacking. Scaling is not actually K8s's strong suit - sure, you can make it scale, but that takes effort and customization.

But what K8s, at base, is actually useful for is availability.

You tell K8s how many instances of a thing to run; it runs them; if any of them stop running, it detects that and tries to fix it.

When you want to deploy a new version, it replaces the old instances with new ones, while ensuring traffic still gets served.

And it does all of this over a substrate of shared underlying server nodes, in such a way that if any of those servers goes down, it will redistribute workloads to compensate.

All of that is useful even if you don't care about scale.

oxfordmale• 3 years ago

It seems they are migrating for migration's sake.

AWS ECS -> Google GKE -> AWS EKS -> Self hosted Kubernetes -> bespoke solution.

They are pretty much changing their tech stack every few years, never maturing their knowledge of their current tech stack.

I am sure it makes sense to self host, but given their track record I am wondering if they wil be migrating to the cloud again in a few years.

axlee2• 3 years ago

To be fair that served them well in the past: the reason why anyone knows about 37signals is because they reinvented the wheel back in 2004 with Rails, but what a great reinvention it was. Who knows what can come next.

wankle5• 3 years ago

k3s is lightweight and even I have clusters running, I can sync them too if I wish, easily, I agree, it seems odd they didn't go with some kube design on prem.

jameshart5• 3 years ago

We are talking about 37Signals here. This is the company that, when faced with the problem of making a shared to-do list application, created Ruby on Rails. And when they decided to write up their remote working policy, published a New York Times bestselling business book.

This is not a company that merely shaves its Yaks. It offers a full menu of Yak barber services, and then launches a line of successful Yak grooming products.

benatkin5• 3 years ago

They seem to have lost their touch though. I think they peaked with Remote.

After typing that I found that they renamed from Basecamp Inc. back to 37signals and their website is trying to hearken to their past. https://en.wikipedia.org/wiki/37signals

Edit: lol https://37signals.com/22/

Dalewyn• 3 years ago

Wouldn't surprise me if they're doing all that with Yakety Sax[1] blaring in the background.

[1] https://www.youtube.com/watch?v=ZnHmskwqCCQ

madeofpalk4• 3 years ago

...and don't forget the time where they wrote a blog post, kept posting through it, and a significant amount of the company quit.

crabbone2• 3 years ago

It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling?

You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes.

When you start using Kubernetes for real, you will end up replacing almost every component it brings by default with something else. CoreDNS? -- sucks for big systems. Volumes? You aren't going to be using volumes from local filesystem... that's insane! You'll probably set up Ceph or something like that and add some operators to help you use it. Permission management? -- Well, you are out of luck in a major way here... you have, basically, Kyverno, but it really, really sucks (and it's still not Kubernetes!).

Real-life Kubernetes deployments end up being very poorly stitched together piles of different components. So much so that you start wishing you'd never touched that thing because a huge fraction of the stuff you now need to manage is integration with Kubernetes on top of the functionality provided by these components.

benatkin5• 3 years ago

You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes.

When you install Kubernetes, you get a container runtime. That's a distribution I guess. Part of this seems like GNU/Linux.

The other stuff you're listing isn't solved by MRSK either...

dikei2• 3 years ago

I don't know, for small scale, K8S rocks: I just fired up Kubespray and have a 20-node cluster up and running in maybe an hour, CoreDNS haven't gave me any problem so far.

Using local volumes is actually not an insane idea if your stateful service can handle data replication themselves: many modern databases can.

tptacek13• 3 years ago

The whole second half of the article is about why they decided to stop using K8s.

danmaz74• 3 years ago

Whatever their reasons, I'm very happy to be able to use a much simpler alternative to K8s to orchestrate my apps on bare metal servers.

Jnr4• 3 years ago

k8s is an industry standard currently, but it is not great. The lack of available free/open tooling to set up and manage them (the cluster) properly seems to indicate that it is also a way of selling it (cloud). Meaning that if you want to use k8s you have to go with the large cloud providers, otherwise your life will be painful.

I for one am patiently waiting for more innovation in this area and seeing that there are companies that try to disrupt/improve it makes me hopeful and I appreciate it.

asjkaehauisa• 3 years ago

They could just use nomad and call it a day

tptacek13• 3 years ago

They could, but instead they're doing something closer to static scheduling. They have a small set of applications and a lot of visibility into what their needs are going to be, so the complexity of a dynamic scheduler might not pay its own freight in their environment.

I like Nomad a lot and it's what I would use if I were migrating a "halfheartedly" K8s application to on-prem metal, but I couldn't blame someone who felt burned by K8s complexity for not investing in another dynamic scheduler.

dustedcodes• 3 years ago

If I was Netflix I would de-cloud, but if I was a small team like 37signals then de-clouding is just insanity. I think DHH is either very stupid or extremely naive in his cost calculations or probably a mix of both. Hey and Basecamp customers will see many issues in the next few years and hackers will feast off their on-premise infrastructure.

tky• 3 years ago

They’ve had non-cloud infrastructure for a very long time. Their new orchestration methods notwithstanding, reliability and security are unlikely to suffer.

bastawhiz3• 3 years ago

I'm not going to put this down, because it sounds like they're quite happy with the results. But they haven't written about a few things that I find to be important details:

First, one of the promises of a standardized platform (be it k8s or something else) is that you don't reinvent the wheel for each application. You have one way of doing logging, one way of doing builds/deployments, etc. Now, they have two ways of doing everything (one for their k8s stuff that remains in the cloud, one for what they have migrated). And the stuff in the cloud is the mature, been-using-it-for-years stuff, and the new stuff seemingly hasn't been battle-tested beyond a couple small services.

Now that's fine, and migrating a small service and hanging the Mission Accomplished banner is a win. But it's not a win that says "we're ready to move our big, money-making services off of k8s". My suspicion is that handling the most intensive services means replacing all of the moving parts of k8s with lots of k8s-shaped things, and things which are probably less-easily glued together than k8s things are.

Another thing that strikes me is that if you look at their cloud spend [0], three of their four top services are _managed_ services. You simply will not take RDS and swap it out 1:1 for Percona MySQL, it is not the same for clusters of substance. You will not simply throw Elasticsearch at some linux boxes and get the same result as managed OpenSearch. You will not simply install redis/memcached on some servers and get elasticache. The managed services have substantial margin, but unless you have Elasticsearch experts, memcached/redis experts, and DBAs on-hand to make the thing do the stuff, you're also going to likely end up spending more than you expect to run those things on hardware you control. I don't think about SSDs or NVMe or how I'll provision new servers for a sudden traffic spike when I set up an Aurora cluster, but you can't not think about it when you're running it yourself.

Said another way, I'm curious as to how they will reduce costs AND still have equally performant/maintainable/reliable services while replacing some unit of infrastructure N with N+M (where M is the currently-managed bits). And also while not being able to just magically make more computers (or computers of a different shape) appear in their datacenter at the click of a button.

I'm also curious how they'll handle scaling. Is scaling your k8s clusters up and down in the cloud really more expensive than keeping enough machines to handle unexpected load on standby? I guess their load must be pretty consistent.

[0] https://dev.37signals.com/our-cloud-spend-in-2022/

tstrimple2• 3 years ago

First, one of the promises of a standardized platform (be it k8s or something else) is that you don't reinvent the wheel for each application. You have one way of doing logging, one way of doing builds/deployments, etc.

You can also hire people with direct relevant experience with these tools. You have to ramp up new developers to use the bespoke in house tooling instead.

chillfox6• 3 years ago

I think the entire scaling thing is a bit like "manual memory management vs GC", both have advantages and disadvantages.

bastawhiz3• 3 years ago

Yes and no. Different types of memory management essentially accomplish the same thing. The way you build for them and their performance characteristics vary. In that way, scaling is the same.

But scaling is different in that you're physical ability to scale up with on-prem is bounded by physically procuring/installing/running servers, whereas in the cloud that's already been done by someone else weeks or months ago. When you shut off on-prem hardware, you don't get a refund on the capex cost (you're only saving on power/cooling, maybe some wear and tear).

It's not just that you need to plan differently, it's that you need to design your systems to be less elastic. You have fixed finite resources that you cannot exceed, which means even if you have money to throw at a problem, it doesn't matter: you cannot buy your way out of a scaling problem in the short-medium term. If you run out of disk space, you're out of disk space. If you run out of servers with enough RAM for caching, you're evicting data from your cache. The systems you build need to work predictably weeks or months out, and that is a fundamentally different way of building large systems.

wmf3• 3 years ago

Why are there VMs?

mannyv• 3 years ago

I think RoR is the core problem here.

denysonique• 3 years ago

Seems like the name of their container shipping tool Mrsk is inspired by Maersk a leading container shipping company from Denmark.

ivarv• 3 years ago

indeed - the presenter (dhh) is Danish, and there was a photo of a container ship near the beginning of the demo video.

trololo01001• 3 years ago

Plot twist: all of this is just a very clever Maersk marketing campaign.

AlexITC• 3 years ago

I have been saying this to my customers for a long time, most projects do not really benefit from K8s but on the contrary, it is a huge operational/cost compromise to use K8s for a monolith app that does simple CRUD operations where occasional downtime is actually acceptable.

In my last project, I removed the unnecessary complexity that K8s was bringing and went back to ansible scripts, which has worked nicely.

With another customer, we inherited a frontend application that was being deployed with K8s while vercel is a considerably simpler/faster alternative.

K8s certainly has its advantages but I'd bet that many projects using it do not gain much.

rossdavidh• 3 years ago

My impression is that it makes deploying your 100th server much easier, at the cost of making your first several much harder. If you're going to have 100+ servers, that's probably worth it. If you're not (and most companies aren't), then it's like getting your CDL so that you can go to the grocery store in a semi-tractor trailer, when you should have driven there in a compact car.

slyall• 3 years ago

The thing I noticed is that they are not using any other AWS services. No S3, Elasticache, DynamoDB, etc. They are just running applications and databases.

This will not be the case with many people using cloud and a migration to bare metal will be much harder. Each of those services needs an equivalent to be deployed and managed and it's features might be up to what the AWS equivalent has.

Even the stuff that they are moving (databases, load balancers, etc) is significant operational overhead. In AWS database fail-over is an option you tick. Self hosting has whole books written about how to do database high availability.

wmf3• 3 years ago

For their more complex products they do use RDS, Elasticache, etc. so that will be a whole other adventure.

Thaxll2• 3 years ago

They're saying a VM takes seconds to boot up, yeah only because they run static dedicated servers, of course in the cloud if you wait for the VM to come online it's going to take longer, now how long does it takes them to add a new dedicated server and to add it to that pool of servers? days?

The other main issue I see is that they use chef and mrsk to setup applications, how is Filebeat setup? Is it chef that set it up or is it mrsk?

dilyevsky2• 3 years ago

now how long does it takes them to add a new dedicated server and to add it to that pool of servers? days?

Depending on provider and your config it could take minutes to hours these days. It also could take months if you’re ordering your own hw

chillfox6• 3 years ago

I started my career at a company that was excellent at capacity management and prediction. Using physical hardware they never hit a capacity problem, ever, despite growing like crazy. This did require the Marketing department being in close communication with the IT department about upcoming campaigns.

Everywhere else though have been terrible at predicting future capacity needs. As far as I can tell that's because they just use tools that gives a prediction based only on historical growth.

I guess my point is that it's entirely possible to be good at capacity management, and if you are then the lead time disadvantage of physical hardware can be completely negated.

Thaxll2• 3 years ago

From my experience with static physical capacity you're always way over provisionned, because just in case and how long it takes to get servers.

Ameo• 3 years ago

Regardless of the merits or drawbacks of "de-clouding" for this particular company, it seems to me that their ops team is just really bored or permanently unsatisfied with any solution.

They say that they've tried deploying their apps in all of:

* Their own Datacenter

* ECS

* GKE

* EKS

* and now back to their own Datacenter

Even with their new "de-clouded" deployment, it seems like they have created an absolutely immense amount of complexity to deploy what seems to be a variety of generic Ruby CRUD apps (I might be wrong but I didn't see anything different in the post).

They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik... It seems well on par complexity-wise with a full K8s deploy with all the bells and whistles (logging, monitoring, networking, etc.)

Google says that this company, 37signals, has 34 employees. This seems like such a monumental amount of orchestration and infra stuff unless they're deploying some crazy complex stuff they're not talking about.

Idk what the lesson is here, if there is one, but this seems like a poor example to follow.

jrumbut• 3 years ago

This is the company that gave birth to Ruby on Rails. They appear to have a culture of being very opinionated about their tools and unafraid of doing things their own way.

Probably not an example most companies that size ought to follow but I'm glad they were crazy enough to do it!

butt__hugger• 3 years ago

37signals and RoR have a habit of flip flopping on their decisions. See CoffeeScript.

cutler2• 3 years ago

There's an operator for that.

tptacek13• 3 years ago

Each of these "flip flops" probably lasted a good deal longer than the median 20+ person startup, so that seems pretty facile. But the parallel with CoffeeScript seems valid --- people on message boards are really not OK with nonstandard languages, and are never less happy than when a company they've heard something about does actual computer science of any kind. See, for instance, Fog Creek and Wasabi.

mirekrusin• 3 years ago

CoffeeScript impacted ECMAScript more than anything else.

tptacek13• 3 years ago

We're talking about a product that has existed since 2004. They did:

* Their own data center, before Docker existed

* The non-K8s Docker way of deploying in AWS

* The GCP, and then AWS, ways of doing K8s Docker

* Docker in their own data center.

For 20 years of deployment, that doesn't look crazy to me.

The actual components of each environment are pretty standard. They wrote Capistrano, which predates Chef. Filebeat is a tiny part of ELK, which is the de facto standard logging stack. They use a smart reverse proxy, like... everybody else. It's easy to anything sound complicated if you expand the stack to as many acronyms as you can.

andrewf• 3 years ago

* Their own data center, before Docker existed

Also, it might be worth calling out: their product launched in 2004, Linode and Xen were launched in 2003, S3 and EC2 launched in 2007. The cloud as we know it today didn't exist when they started.

ghaff• 3 years ago

Whether DHH is "right" in some philosophical sense, this is a small company with a lot of technical experience in a variety of technologies and with presumably a lot of technical chops, so generalizing their experience to "cloud is good" or "cloud is bad" isn't really possible.

Rapzid2• 3 years ago

Well, they were def at Rackspace in there somewhere.

jb_gericke2• 3 years ago

Let’s write our own container orchestrator though, because control planes are dumb?

bennysonething• 3 years ago

What do 37 signals do that makes money?

klabb32• 3 years ago

I think you’re right that they doing it for fun or because they can, primarily. But I am excited to see them pioneer in this area, both because it’s more open and hacker friendly, but also because they’re moving the needle towards healthy competition amongst the providers.

Our big-three cloud hegemony has already showed it’s ugly sides, both in terms of price (egress, anyone?) and quality (hello, zero interop and opaqueness). I’d argue we’ve seen significant complexity increases in especially server-side tech in the last 5-10 years, with relatively little to show for it, despite massive economic investments. I expect that trend to continue downwards unless we take back the hacker friendliness of infra & ops.

PS. Actually scratch that I’m excited, that’s an understatement. I’m thrilled!

jb_gericke2• 3 years ago

Pioneer? Other than rewriting docket swarm it sounds like a stack from the early 2000s…

Melingo21• 3 years ago

Kubernetes is a poster child on open source and transparent development.

And it pushed the needle tremendously.

I'm lost on how you can compare this even.

spiralpolitik• 3 years ago

From my experiences there are two lessons:

There is usually a sweet spot in terms of size where being on the public cloud makes sense, both from a cost and management perspective. Once you go above that size then having to manage IAM starting becoming a pain. Usually around the same point public cloud costs start becoming noticeable to your finance team and so you have to start dealing with questions around that. Usually that's a good point to do a sanity check before things get even bigger and more expensive.

Similar k8s works well for certain classes of problem, but doesn't work well for all classes of problem. Any form of k8s has an operational overhead and you really need to make sure that you are going to get the ROI from the effort of maintaining the stack for it to be worthwhile.

cyberax• 3 years ago

having to manage IAM starting becoming a pain

Just use multiple AWS accounts. You won't need any complicated IAM policies.

hsn9158• 3 years ago

This is very common in almost all web companies since around 2015.

I've never seen a company with a simple infrastructure, no matter how simple their actual application actually is.

If you choose a slow dynamic language (Ruby/Python) your deployment has to be massively complicated; you have no choice about it.

For one simple reason: you will need a multitude of separate components to be made to work together.

You need many application instances because there's no way one machine can handle all your traffic: Ruby is just too slow.

A sharded database cluster as a source of truth:

You went through the effort of making several applications instances with a load balancer: you don't want a single database server to be a single point of failure.

A distributed redis/memcache index to accelerate queries and lower the pressure on the real database.

You might have several index-type engines for different types of quries. Most people use ElasticSearch in addition to Redis.

You need some system to manage all this complexity: monitor it, deploy new versions, rollback to a previous version, run migrations and monitor their state, etc etc.

This is the bare minimum. Most people have a setup that is way more complicated than this. I don't even know how they come up with these complexties, but they not only come up with frequently: they love it! To them it's a feature, not a bug.

hw2• 3 years ago

So, you’re explaining a stack with:

- application instances

- load balancer

- database

- cache

- search cluster if application search is necessary

Sounds like any cookie cutter application to me, even modern ones. How is that complicated?

bayesian_horse• 3 years ago

You are making a lot of assumptions and many of those are not universal problems or even at all.

Compiled languages eventually need a complicated setup for the very same reasons. There is no such thing as "scales" and "doesn't scale". Even Go or C++ webapps have to be scaled up.

If you can get away without complexity on Go or whatever, good for you. Most companies don'T.

jillesvangurp• 3 years ago

When I read stuff like this it strikes me that probably, by far, their largest operational expense is their staffing cost to orchestrate all of this. I come from a background of running small startups on a shoe string budget. I need to make tough choices when it comes to this stuff. I can either develop features or start spending double digit percentages of my development budget on devops. So, I aim to minimize cost and time (same thing) for all of that. At the same time, I appreciate things like observable software, rapid CI/CD cycles, and generally not having a lot of snow flakes as part of my deployment architecture. I actually have worked with a lot of really competent people over the past two decades and I like to think I'm not a complete noob on this front. In other words, I'm not a naive idiot but actually capable of making some informed choices here.

That has lead me down a path of making very consistent choices over the years:

1) no kubernetes and no microservices. Microservices are Conways Law mapped to your deployment architecture. You don't need that if you do monoliths. And if you have a monolith, kubernetes is a waste of CPU, Memory, and development time. Complete overkill with zero added value.

2) the optimal size of a monolith deployment is 2 cheap VMs and a load balancer. You can run that for tens of dollars in most popular cloud environments. Good enough for zero down time deployments and having failover across availability zones. And you can scale it easily if needed (add more vms, bigger vms, etc.).

3) those two vms must not be snow flakes and be replaceable without fanfare, ceremony, or any manual intervention. So use docker and docker-compose on a generic linux host, preferably of the managed variety. Most developers can do a simple Dockerfile and wing it with docker-compose. It's not that hard. And it makes CI/CD really straight forward. Put the thing in the container registry, run the thing. Use something like Github actions to automate. Cheap and easy.

4) Use hosted/managed middleware (databases, search clusters, queues, etc). Running that stuff in some DIY setup is rarely worth the development time and operational overhead (devops, monitoring, backups, upgrades, etc). All this overhead rapidly adds up to costing more than years of paying for a managed solution. If you think in hours and market conform rates for people even capable of doing this stuff, that is. Provision the thing, use the thing, and pay tens of dollars per month for it. Absolute no brainer. When you hit thousands per month, you might dedicate some human resources to figuring out something cheaper.

5) Automate things that you do often. Don't automate things that you only do once (like creating a production environment). Congratulations, you just removed the need for having people do anything with teraform, cloudformation, chef, puppet, ansible, etc. Hiring people that can do those things is really expensive. And even though I can do all of those, it's literally not worth my time. Document it, but don't automate it unless you really need to and spend your money on feature development.

But when I need to choose between hiring 1 extra developer or paying similarly expensive hosting bills, I'll prefer to have the extra developer on my team. Every time. Hosting bills can be an order of magnitude cheaper than a single developer on a monthly basis if you do it properly. For reference, we pay around 400/month for our production environment. That's in Google cloud and with an Elastic Cloud search cluster included.

Other companies make other choices of course for all sorts of valid reasons. But these work fine for me and I feel good about the trade offs.

jspdown• 3 years ago

I can't agree more on this. Most products out there with medium to low traffic can be handled just fine like this. The cost of automation is often not worth the financial effort.

There's a dangerous trend in putting microservices everywhere. Then having the same level of quality as a monolith requires an infinite amount of extra work and specialized people. Your product must be very successful to justify such expenses!

My rule of thumb; monolith and PaaS as long as your business can afford to.

bsenftner• 3 years ago

Where, oh god where, are there more sensibly thinking people like you! This is pragmatic and straight forward. There is very little room for technical make work nonsense in your described strategies. Most places, and many devs I meet cannot imagine how to do their jobs without a cornucopia of oddly named utilities they only know a single path of use.

datadeft5• 3 years ago

What if I do this with Terraform using AWS Serverless and staying in the free tier for this workload that you are referencing instead of VMs and a load-balancer?

I just don't see why people prefer the VM based approach over serverless.

roncesvalles2• 3 years ago

Agree entirely. I think system design interviews are partly to blame because they select for people who think that the only way to design a system is the cargo cult method that interview prep books and courses preach, which is:

- break everything into microservices

- have a separate horizontally scalable layer for load balancing, caching, stateless application server, database servers, monitoring/metrics, for each microservice.

- use at least two different types of databases because it's haram to store key-value data in a RDBMS

- sprinkle in message-passing queues and dead-letter queues between every layer because every time you break one system into two, there can be a scenario where one part is down but the other is up

- replicate that in 10 different datacenters because I'll be damned if a user in New York needs to talk to a server in Dallas

And all this for a service that will see at most 10k transactions per second. In other words, something that a single high-end laptop can handle.

99.9% of the time your architecture does NOT need to look like Facebook's or Google's. 99% of tech startups (including some unicorns) can run their entire product out of a couple of good baremetals. Stop selecting for people who have no grounding of what is normal complexity for some given scale.

tetha• 3 years ago

This is actually a really interesting post to me. I'm currently working at the opposite of a startup with a shoestring budget. We're a medium-sized company with 100 - 150 techies in there. As a unique problem, we're dealing with a bunch of rather sensitive data - financial data, HR data, forecast and planning data. Our customers are big companies, and they are careful with this data. As such, we're forced to self-host a large amount of our infrastructure, because this turns from a stupid decision into a unique selling point in that space.

From there, we have about 7 - 12 of those techies working either in my team, saas operations, our hardware ops team, or a general support team for CI/images/deployment/buildserver things. 5 - 10% of the manpower goes there, pretty much.

The interesting thing is: Your perspective is our dream vision for teams running on our infrastructure.

Like - 1 & 2 & 3: Ideally, you as the development team shouldn't have to care about the infrastructure that much. Grab the container image build templates and guidelines for your language, put them into the template nomad job for your stuff, add the template pipeline into your repository, end up with CD to the test environment. Add 2-3 more pipelines, production deployments works.

These default setups do have a half life. They will fail eventually with enough load and complexity coming in from a product. But that's a "succeed too hard" kinda issue. "Oh no, my deployment isn't smooth enough for all the customer queries making me money. What a bother" And honestly, for 90% of the products not blazing trails, we have seem most problems so we can help them fix their stuff with little effort to them.

4 - We very much want to standardize and normalize things onto simple shared services, in order to both simplify the stuff teams have to worry about and also to strengthen teams against pushy customers. A maintained, tuned, highly available postgres is just a ticket, documented integrations and a few days of wait away and if customers are being pushy about the nonfunctional requirements, give them our guarantees and then send them to us.

dgroshev3• 3 years ago

The only point I disagree with is Terraform. It is brilliant for this exact scenario because it's self documenting. When you do need to update those SPF records in two years time, having it committed as a Terraform file is much better than going through (potentially stale) markdown files. It's zero maintenance and really simple. Plus its ability to weave together different services (like configuring Fastly and Route53 from the same place) is handy, too.

hw2• 3 years ago

I wonder how much of these movements are them iterating and hunting for ROI in their infrastructure costs. Did GCP and AWS salespeople sell them on the benefits of the cloud, offer discounts, white glove migration help, showed some calculation on how much $$ they will save in the cloud, etc that on paper sounded great, but wasn’t ultimately a good fit?

Their market is probably saturated and perhaps declining that they are reaching for optimizations elsewhere.

Jnr4• 3 years ago

There is no such thing as "saving money in cloud". It is all about convenience and it always costs more than a smart team could achieve elsewhere.

I tend to hear an argument that it is cheaper since you do not have to pay people to maintain those services, but in reality you still need that person to set up and maintain your particular cloud setup. And the services themselves are much much more expensive than maintaining your own servers in a data center.

In my opinion cloud hosting and services are more meant for large corporations where no one wants to take responsibility and is scared of doing anything. Cloud is a nice way to shift the blame if/when things go bad - "but cloud is industry standard, everyone does it".

ExoticPearTree• 3 years ago

If you have a fairly stable traffic pattern, hosting your own stuff tends to be significantly cheaper than any cloud provider.

kodah• 3 years ago

They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik

These tools are pretty stock standard in the Systems Engineering world. I think anyone that's been a Systems Engineer that's over 30 has probably deployed every one of these.

One thing I've learned over my mixed SWE and SE career is that infrastructure is expensive and grows regardless of revenue. I didn't truly appreciate this until I launched Kubernetes on Digital Ocean and began running my personal cloud on it. It was costing me over $100/m for very little. That money was gone whether I pushed a ton of VPN traffic over my mesh or not. It didn't care about how much I stored in the disk I reserved, and frankly, that cost was going to grow as time went on. I pulled the plug, setup servers in my house, wired up Traefik and Docker Compose V2 with a little Tailscale sprinkled on. The servers stay up to date with some scripts, I deploy new apps on select servers with Docker Compose and Docker profiles.

It's possible for companies to do similar things, but not to the extremes I took it to. A really good infrastructure SWE generally goes for $300k. You can pay people with an expertise in these things who can streamline them and create maintainable products out of your infrastructure or you can pay for Legos and glue from a managed service provider like AWS, GCP, or Azure. At some point the latters costs will not scale, you'll pivot and cost reduce many times - maybe even begin rearchitecting. I think there's a lot of companies that are noe realizing the cheap money is gone, and the cloud has somewhat relied on cheap money.

mpweiher• 3 years ago

Idk what the lesson is here

I'd say the lesson is that we, as an industry, haven't figured out this "cloud" stuff yet.

And it looks to me that what we need, roughly, is some sort of "deployment polymorphism" that separates interface from implementation.

datadeft5• 3 years ago

Idk what the lesson is here, if there is one, but this seems like a poor example to follow.

The lesson is not to focus on tech tooling as much and focus more on product instead. Imagine this energy doing into the product....

shafyy• 3 years ago

In addition to what other commentators have stated: 37signals has ~ 80 employees, not 34.

dpkirchner• 3 years ago

They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik

I use a lot of this or similar (terraform instead of Chef, logstash instead of filebeat) and I'm a one person team. If anything these tools make my job a lot easier and less complex.

wankle5• 3 years ago

I so wanted to read the page but after 10 seconds of the high contrast white text on pure solid black background I started developing a headache.

rhizome31• 3 years ago

Yes, very weird, I got kind of dizzy.

pxc4• 3 years ago

Because I have some light sensitivity issues, I use browser extensions including Dark Reader and Midnight Lizard to enforce my own 'dark mode' across the web.

You can also use extensions like that to set the contrast to a more comfortable level on websites that are already dark.

I highly recommend this if you have light sensitivity issues like me.

Also note that when the contrast on a page is higher, you can generally get away with lower brightness. This is pretty convenient on phones, and probably more necessary as well since on a phone you're more likely to have an OLED screen that really surfaces extreme contrast like white on black.

jacooper3• 3 years ago

Have they tried Swarm?

vcryan2• 3 years ago

37signal folks love to put a spin on anything they do as if it's ground breaking or super innovative... but it rarely is. In particular, they love to take a contrarian position. Like their books, there really isn't anything interesting written here.

vcryan2• 3 years ago

I have to imagine part of the reason they need to run so many servers is because they are running Ruby. The same application on say, Elixir, probably would require less hardware, reducing the cost of ECS or similar.

zainhoda• 3 years ago

Omg, they could have literally just solved all of this by choosing Cloud Run (a thin wrapper on top of Knative) rather than running GKE directly.

tptacek13• 3 years ago

Skimming the thread here, it seems like there's some confusion about the goals:

* They've decided to move from EKS to on-prem largely because of cost. That's logical: almost by definition, it costs more to run workloads on cloud machines than on your own hardware. You can't address that problem by moving from EKS back to ECS, like one commenter suggested.

* They've decided to move from K8s to mrsk, a system they developed. They're fuzzier about why they did that, but the two fairly clear claims they made: (1) their deployments under K8s are a lot more complicated, and (2) they slashed their deploy times (because a great deal of their infra is now statically defined).

I feel like there's more productive debating to be done about K8s vs mrsk than there is about EKS vs. mrsk. By all means, make the case that applications like Tadalist are best run on K8s rather than a set of conventions around bare-metal container/VMs (which is all mrsk is).

lmm3• 3 years ago

almost by definition, it costs more to run workloads on cloud machines than on your own hardware.

Why should that be so? I'd expect the all-in cost of a cloud machine to be less than my own hardware, for the same reason that buying electricity from the grid is usually cheaper than generating it on-prem.

You can't address that problem by moving from EKS back to ECS, like one commenter suggested.

If EKS is more expensive (because it's something they see as a value-add) whereas ECS is a commodity service at commodity prices, then moving there could well solve the cost issue.

koolba• 3 years ago

A better analogy for cloud vs on-premises is going to a restaurant vs cooking. The markups are about the same too.

tptacek13• 3 years ago

The major determinant in hosting cost isn't power, it's the cost of the hardware. But I mean, even if you don't buy my axiomatic derivation, you can just work this out from AWS and GCP pricing.

danielheath• 3 years ago

Cloud is reasonably cheap until you need to move data out (eg to serve html to customers). Egress charges are where the big players hit you.

tekno45• 3 years ago

You're paying for someone elses profit monthly with the cloud.

It has to cost more.

ivlad• 3 years ago

AWS’s 24% of operating margin does not appear out of nowhere.

ai_ja_nai2• 3 years ago

Why should that be so? I'd expect the all-in cost of a cloud machine to be less than my own hardware

Because cloud hardware doesn't have all the burdens of physically managing a real server. Replacing SSDs. Upgrading RAM. Logging to a iDRAC to restart a crashed server. All those things don't exist in the cloud and make you loose so much operational time. That's why clouds will ALWAYS cost more than bare metal. The cons is that with cloud you keep paying for the same servers: there are no assets anymore, only costs.

cortesoft• 3 years ago

Yeah, I would love to hear more about why they decided not to go with on prem k8s... the other arguments made logical sense to me, but they don't explain the reasoning for mrsk very well.

mike_d2• 3 years ago

Every company that I have been at that uses k8s at scale ends up having an internal team to manage the complexities and build internal tooling to make it work. It sounds like they left behind a lot of the cruft and just built a tool that does the one thing most people want: put a container on a VM and call it good.

xyzzy_plugh2• 3 years ago

By all means, make the case that applications like Tadalist are best run on K8s rather than a set of conventions around bare-metal container/VMs (which is all mrsk is).

Okay sure I'll bite. An application like Tadalist is best run on k8s.

With any application regardless of how it runs, you generally want at least:

- zero-downtime deploys

- health checks

- load balancing

Google's GKE is like $75/mo, and the free tier is one cluster, which is enough. For nodes, pick something reasonable. We're naive so we pick us-west1 and a cheap SKU with 2 vCPUs 8 GB is ~$30/mo after discounts. We're scrappy so we eschew multiple zones (it's not like the nearby colo is any better) so let's grab two of these at most. Now we're in $60/mo. We could go cheaper if we want.

We've click-opsed our way here in about 25 minutes. The cluster is up and ready for action.

I write a Dockerfile, push my container, install k3d locally, write about 200 lines of painstaking YAML that I mostly cribbed off of stack overflow, and slam that through kubectl into my local cluster. Everything is coming up roses, so I kubectl apply to my GKE cluster. My app is now live and I crack open a beer. Today was a good day.

Later, whilst inebriated from celebration, I make some changes to the app and deploy live because I'm a cowboy. Oops, the app fails to start but that's okay, the deployment rolls back. No one notices.

The next day my app hits the front page of HN and falls over. I edit my YAML and change a 2 to a 10 and everything is good again.

Things I did not need to care about:

- permissions (I just used the defaults and granted everything via clickops)

- ssh keys (what's ssh?)

- Linux distributions or VM images (the Goog takes care of everything for me, I sleep better knowing I'll wake up to patched OS)

- passwords

- networking, VIPs, top of rack switches, hosting contracts, Dell, racking and stacking, parking, using my phone

And all without breaking the bank.

---

Okay so I cheated, you weren't looking for a GKE vs on-prem/Colo case. You asked

make the case that applications like Tadalist are best run on K8s rather than a set of conventions around bare-metal container/VMs

to which I say: that's all kubernetes is.

Did you even read their blog post? virtio? F5? MySQL replication?? How is this a good use of engineering time? How is this cost efficient? On what planet is running your own metal a good use of time or money or any other godforsaken resource. They're not even 40 people for crying out loud. It's not like they're, say, Fly.io and trying to host arbitrary workloads for customers. They're literally serving rails apps.

Want to start small with k8s? Throw k3s or k3d on a budget VPS of your choosing. Be surprised when you can serve production traffic on a $20 Kubernetes cluster.

If you care about Linux distributions, and care about networking, and care about database replication, and care about KVM, and care about aggregating syslogs, and love to react to CVEs and patch things, and if it's a good use of your time, then sure do what 37signals did here. But I'm not sure what that planet is. It's certainly not the one I live on today. 10-15 years ago? Sure. But no longer.

I can't believe just how ridiculous this entire article is. I want to find quotes to cherry pick but the entire thing is lunacy. You can do so so much on a cloud provider before approaching the cost of even a single 48U in a managed space.

At some scale it makes sense, but not their scale. If I never have to deal with iDRAC again it'll be too soon.

You have a horse in this race: apps like Tadalist are best run on something like Fly or knative/Cloud Run or Heroku rest in peace. But a set of conventions around bare-metal containers/VMs? Give me a break.

I don't think you intended it, but I find it disingenuous to separate cloud hosting and kubernetes. The two are connected. The entire premise is that it should be a set of portable conventions. I can run things on my laptop or desktop or raspberry pi or $10/mo budget VPS or GCP or AWS or Azure or Linode or god willing a bunch of bare-metal in a colo. It's fundamentally a powerful abstraction. In isolation it makes little sense, which TFA handedly demonstrates. If you eschew the conventions, it's not like the problems go away. You just have to solve them yourself. This is all just NIH syndrome, clear as day.

Forgive the long winded rant, it's been a long day.

ai_ja_nai2• 3 years ago

Agree, I would never want to go back to the old bad days of managing a real rack at a datacenter, with exactly the same guarantees of a single region deployment inside any cloud. BUT it is true that all the multi region/AZ guarantees + logs + dashboards + network services @ AWS costs tend to skyrocket in a couple of years. And here is where k8s really shines, in my opinion: allowing you to abstract your deployment away from a cloud even on cheap hosting. All the rest outlined in the article is just reinventing the wheel.

datadeft5• 3 years ago

That's logical: almost by definition, it costs more to run workloads on cloud machines than on your own hardware

Only if you are not adjust the workload. Lift+shift is cost more. Re-architecting and making the workload cloud ready cost less.

rednerrus2• 3 years ago

Who's gonna do the rearchitecting work? Are you hiring a whole new team or do you not need to keep the lights on while you're transitioning? Depending on the complexity of your application that rearchitecting is gonna eat up a ton of your cost savings.

mattgreenrocks• 3 years ago

The comments in this thread are quite eye-opening.

It really shows what a sacred cow k8s and cloud has become.

I’m not much of an ops person so I’m not qualified to comment on what 37 signals has created. But I will say I’m glad to see honest discussion around the costs of choosing k8s for everything even if it has significant mindshare.

Perhaps this is the endgame of resume-driven development: cargo culted complexity and everyone using the same tech for similar-ish problems and then wondering why it’s so hard to stand out from both a product and an employee perspective.

leetrout3• 3 years ago

I am much of an ops person and I will say k8s has its place and its in 10-20% of companies max.

It is absolutely ridiculous how many places use k8s and rarely for what it can really do.

Cargo culted complexity, indeed.

mike_d2• 3 years ago

Some people are really good at writing software, other people are really good at running systems. k8s/cloud allowed the former to pretend to be good at the latter.

lmm3• 3 years ago

I hate kubernetes as much as anyone, but building your own container orchestration platform so that you can deploy a handful of CRUD webapps sounds a lot more like resume-driven development than using a well-known and standard (if somewhat overengineered) solution.

mgkimsal• 3 years ago

There are other options between k8s and “build your own”.

tptacek13• 3 years ago

I don't think the authors care about their resumes at this point. There are rational reasons to use a static scheduling regime and a set of conventions around deployment and support services rather than a dynamic scheduler. If it were me, I'd build this with Nomad, but I can imagine not wanting to manage a dynamic scheduler when your workloads are as predictable as theirs are --- you make their case for them when you point out that they just have a "handful of CRUD apps".

smrtinsert• 3 years ago

If there's discussion it isn't a sacred cow

jpgvm• 3 years ago

k8s is misunderstood. Everyone focuses on the complexity/over-engineering/etc arguments when those really don't matter in the grand scheme of things.

It's not about any of that, it's about having a consistent API and deployment target that properly delineates responsibilities.

The value of that then depends on how many things you are running and how many stakeholders you have taking part in that dance. If the answer to both of things are small then k8s value is small, if the answer to either of those is high then the value is high.

i.e k8s is about organisational value, it's technical merits are mostly secondary.

snupples• 3 years ago

The "it's too complex" argument usually reflects more on the commenter than on kubernetes itself. It's actually one of the most very straight forward and thoughtfully designed platforms I've ever worked with.

What I've found in my experience is that applications in general are complex -- more complex than people assume -- but the imperative style of provisioning seems to hide it away, and not in a good way. The inherent complexity hides behind layers of iterative, mutating actions where any one step seems "simple", but the whole increasingly gets lost in the entropic background, and in the end the system gets more and more difficult to _actually_ understand and reproduce.

Tools like ansible and terraform and kubernetes have been attempts to get towards more definition, better consistency, _away_ from the imperative. Even though an individual step under the hood may be imperative, the goal is always toward eventual consistency, which, really only kubernetes truly achieves. By contrast, MRSK feels to be subtly turning that arrow around in the wrong direction.

I'm sure it was fun to build, but one could have spent 1% of that time getting to understand the "complexity" of kubernetes - by the way, which quickly disappears once it's understood. Understandably, though, that would feel like a defeat to someone who truly enjoys building new systems from scratch (and we need those people).

MrBuddyCasino• 3 years ago

Perhaps this is the endgame of resume-driven development: cargo culted complexity and everyone using the same tech for similar-ish problems and then wondering why it’s so hard to stand out from both a product and an employee perspective.

Spot on. Tech is a fashion industry and most people just follow trends. I still sometimes wonder if people are playing the elaborate long-term resume-optimisation game, or if they don't value simplicity highly enough to optimise for it, because the downsides are externalised.

waynesonfire2• 3 years ago

k8s folks get paid big money to keep it running. Not surprised by the comments here at all. As the saying goes, "in complexity, there is opportunity." and the k8s devops team is milking it hard.

Valord• 3 years ago

Agreed. I was having similar feelings reading comments.

throwawaaarrgh• 3 years ago

A lot of people are kind of missing the forest for the trees here. Ignore the fact that what they're doing is probably a terrible idea for most other people. If it works for them, that's fine. It might only work for them, and that's fine.

Don't paint your bike shed orange just because somebody famous painted theirs orange. They have their reasons. Paint yours whatever color works best for you, for your own reasons.

ozim2• 3 years ago

Only if DHH would not be posting marketing pieces where it seems that if you also paint your shed in orange it will save you $8mil in 5 years.

It is also that DHH is somewhat renowned in software world so it is also not like Pedro Pascal is telling you to do that.

Yes it is marketing and boasting but it is not so easy to see forest for the trees here.

motoboi• 3 years ago

On-prems is wherw companies goes to die.

cutler2• 3 years ago

K8s, Docker and AWS/GCP/Azure are to ops what React is to web development, ie. rarely the appropriate tool for the job. Trouble is you now have a generation of devs who have no experience with anything else.

yafbum• 3 years ago

How does the overall cost of infrastructure compare, at the end of something like this? How fast does the move back pay off in terms of ROI?

karakanb• 3 years ago

I find it very interesting that every conversation around k8s turns into a flame war between "just use k8s" and "no you don't need k8s at all". in reality, it is probably more of a spectrum than a boolean value. Also, it seems like people have a different definition on "using kubernetes":

* manage your own k8s cluster on your own hardware: probably pretty hardcore, I've never done this, I'd imagine it'd require me to know about the underlying hardware, diagnose issues and make sure the computer itself is running before managing k8s itself. only when the hardware is running properly I can focus on running k8s, which is also operationally expensive as well. Tbh I don't see a reason for a small/mid scale product to go this route unless they have a very specific reason.

* manage your own k8s cluster on cloud hardware: this seems like a bit simpler, meaning that I don't actually need to know much about running/managing hardware, that's what the provider does for me. I have done this before with k3s for some small applications, I have 2 small scale applications running like this for ~2 years now on Oracle's free ARM instances, I don't really do any active work/maintenance on them and they are running just fine. I'd probably have a lot of trouble if I wanted to upgrade k3s version for large scale applications, or usecases that have tight SLAs.

* use a managed k8s offering from a cloud provider: I've been doing this one the most, and I find it the easiest way to run my applications in a standardized way. I have experience in running applications on this setup for mid-scale as well as multi-national large scale consumer facing applications. Admitted, even though scale has been big, applications themselves have been mostly CRUD APIs, RabbitMQ / Kafka consumers and some scheduled jobs most of the time.

The trick seems to lie in the word "standardized" here: it is probably possible to run any application on any sort of hardware/orchestration combination, and MRSK could be a really nice solution for that as well. However, in my personal experience I have never managed to find an easier way of running multiple full applications, e.g. things that have multiple components such as web APIs, async workers, etc, in a standardized, replicable way.

I run the following components in one of my cloud-managed k8s clusters: - Vault - A few Laravel applications - A few Golang APIs - Grafana & Loki - Metabase

Using k8s for situations like this where the specific requirements from the underlying infra is not very complex actually enables a lot of experimentation / progress simply thanks to the ecosystem. For all of these components there are either ready-made Helm charts that I can simply run a `helm install` and be 90% there, or it is trivial to build a simple K8s deployment configuration to run them. In my experience, I couldn't find anything that comes closer to this experience without having a large engineering team dedicated to solve a very specific problem. In fact, it has been pretty chill to rely on the managed k8s offerings and just rely on my applications.

It's a spectrum: there are a billion cases that don't need k8s, and there are probably a similar amount that could actually benefit from it. There's no absolute truth to it other than the fact that k8s is actually useful for certain cases and it is for sure not always "resume driven development". This doesn't mean that we shouldn't be looking out for better approaches, there's probably a lot of accidental complexity around it as well, but we could also acknowledge that it is actually a useful piece of software.

I don't know, I feel like I have to pick sides every time these sorts of stuff is being discussed as if there is an objective truth, but I am fairly convinced these days that there is a middleground that doesn't involve fanaticism in either direction.

BackBlast• 3 years ago

Online deployments are discussed with some frequency. Tooling is talked about. Always as a "cluster". Why do we need clusters anymore? Scaling containers, scaling functions, scaling, scaling, cluster, cluster. We suffer so much tunnel vision about horizontal scaling when it's just unnecessary anymore for most applications. The cloud products are all about horizontal.

Do you really need more than the 400 threads >12TB ram with PBs of storage found in a reasonable high end server?

ludwigvan• 3 years ago

mrsk introduction screencast by DHH explains the process well, and is pretty impressive as usual: https://m.youtube.com/watch?v=LL1cV2FXZ5I

Joel_Mckay• 3 years ago

Depending on your business use-case, sharing a pool of IPs can have detrimental impact on access. For example, you may find the prior users were doing unauthorized security scans, spamming email, or hosting proxies.

i.e. if you get an IP block with a bad reputation, than you may find running a mail or VoIP server problematic.

If you are running purely user centric web services, than it doesn't matter as long as you are serving under around 30TiB/month.

There is also the issue of future offline decryption of sensitive records without using quantum resistant storage cryptography.

Rule #4: The first mistake in losing a marathon is moving the finish line. =)

kristianpaul• 3 years ago

Neither cloud or on-prem fits everyones requirements, but at the end you need to know your environment well. One think i like from ECS and fargate is that you could use projects like my_init and get a container behave closer to a vm (run ssh, and other daemons at the same time).

clepto2• 3 years ago

Throughout my company’s pursuit of moving everything under the sun into AWS I have done my best to keep everything able to be migrated, we have some systems which are just, simply going to have to be completely rebuilt if we ever needed to move them off of AWS, because there is not a single component of the system that doesn’t rely on some kind of vendor lock-in system AWS provides.

I aim to keep everything I’m working on using the simplest services possible, essentially treating AWS like it’s Digital Ocean or Linode with a stupidly complex control panel. This way if we need to migrate, as long as someone can hand me a Linux VM and maybe an S3 interface we can do it.

I really just have trouble believing that everyone using Kubernetes and a bunch of infrastructure as code is truly benefiting from it. Linux sysadmin isn’t hard. Get a big server with an AMD Epyc or two and a bunch of RAM, put it in a datacenter colo, and maybe do that twice for redundancy and I almost guarantee you it can take you at least close to 9 figures revenue.

If at that point it’s not enough, congratulations you have the money to figure it out. If it’s not enough to get you to that point, perhaps you need to re-think your engineering philosophy(for example, stop putting 100 data constraints per endpoint in your python API when you have zero Postgres utilization beyond basic tables and indexes).

If you still really genuinely can’t make that setup work, then congratulations you are in the 10%(maybe) of companies that actually need everything k8s or “cloud native” solutions offer.

I would like to note that given these opinions, I do realize there are problems that need the flexibility of a platform like AWS, one that comes to mind is video game servers needing to serve very close to a high number of geographic areas for latency concerns.

codethief• 3 years ago

To play the devil's advocate here:

I aim to keep everything I’m working on using the simplest services possible, essentially treating AWS like it’s Digital Ocean or Linode with a stupidly complex control panel.

What's the benefit of AWS then, if you're not using any of the managed services AWS offers, and are instead treating AWS as an (overly expensive) Digital Ocean or Linode?

clepto2• 3 years ago

I’m arguing there’s not a benefit, it’s just the service I have to use for reasons outside of my control.

fbdab1032• 3 years ago

Getting a real dokku vibe from this. Although, currently limited to one container per machine, which is a huge limitation.

richardw• 3 years ago

I see this as an instance of a company discovering value well-described in this article from 2021:

"The Cost of Cloud, a Trillion Dollar Paradox"

https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...

kgeist5• 3 years ago

We use k8s to run the app both on AWS, on our own hardware in a few datacenters (in countries with strict personal data laws) and on clients' own servers as well (something like the banking sector or a jewelry company, i.e. companies which don't trust the cloud).

From what I heard, AWS is the most stable and easiest to work with of all; the servers which run on our own hardware have more outages and our SRE team often needs to make trips in person to the datacenters to replace hardware etc. Clients' hardware is the faultiest (unsurprisingly). Ideally we'd rather host everything on AWS :)

ec109685• 3 years ago

It’s pretty sad that both Google and AWS price their on premise versions of their hardware at the same price as if you were running in their cloud.

Makes it totally a non-starter to be a universal platform.

lmarcos• 3 years ago

I think many have lost the main point of this post: 37signals seems to be a successful tech company, and their employees are having fun.

Very few tech companies can say that. So, it doesn't really matter if they ditch k8s for their own solution. Really, it doesn't.

KingOfCoders2• 3 years ago

Timing is perfect, several of my clients fell through Series B and now look for cutting cloud costs (all way over provisioned for their traffic and customer numbers).

KingOfCoders2• 3 years ago

Cloud is important for those who need it.

We did run a fast growing startup (sometimes 100% MoM jumps), 5M active users with 50k concurrent users (not visitors) with DB writes on 6 machines + 2 DB servers and $100M ARR 10 years back. If you're this size, MRSK makes total sense.

If you're much larger or growing >50% MoM continuously, K8 in the cloud makes more sense.

roboben• 3 years ago

cluster-autoscaler
ingress controllers
storage drivers
external-dns
node termination handlers
complex networking concepts around VPN, peering, route tables, NAT, …
where DNS is handled
It also misses the entire sphere around identity and access management for those resources that also needs to be maintained

Well, how is this all solved with their new tooling? Like they describe a whole huge complicated problem space and then write a tool for the simplest part of it: deploying an app. :shrug:

xyzzy_plugh2• 3 years ago

I've never read such a ridiculous article. I really wanted to give them the benefit of the doubt but good lord. How is any of this simpler or better? It's like they prefer the pain of 2004 mixed with the worst parts of modern infrastructure. The dream of the 2000s really is alive in DHH's head, isn't it?

TFA links to their cloud spend for 2022[0], wherein lies the rub:

In total, we spent $3,201,564 on all these cloud services in 2022. That comes out to $266,797 per month. Whew!
For HEY, the yearly bill was $1,066,150 ($88,846/month) for production workloads only. That one service breaks down into big buckets as follows:

What the actual fuck? THREE MILLION DOLLARS? A million for their email service?? I have seen bills much larger, but for what 37signals does I am shocked. There is surely a ton of low hanging fruit to drop the bill despite the claim that it's as optimized as it can get. No way.

Even then, Hey is $99/year, and they claimed to have 25k users in the first month or so as of 2020, that's nearly $2.5MM. I presume they've grown since then. Another 2020 article[2] mentions 3/4 of their users have the iOS app, and the Android app currently shows "50k+ installs" so let's assume we're talking 200-400k users as a ceiling, ignoring attrition, which would pull $20-40MM. Even if it's half that, the cost doesn't seem unreasonable.

They're spending nearly $90k/mo on Hey. Of that the majority is RDS and OpenSearch. TFA makes it clear they know how to run MySQL, why on earth don't they stop running RDS? Both of these can easily be halved if they ran the services manually.

EKS is practically free so whatever. They state they have two deployments for ~$23k/mo total -- production is likely larger than staging but let's assume they're equal -- or ~$12k/mo each. A middle of the road EC2 instance like m4.2xlarge is less than $215/mo which gets more than enough cores and memory to run a rails app or two per node. That works out to around 55 nodes per environment. This benchmark[3] shows an m4.2xlarge can serve 172req/s via modern Ruby on Rails. At 500k users that works out to over 1600 request/user/day which seems excessive but likely within an order of magnitude of reality. These are the folks who wrote RoR so I would hope they can optimize this down further. <10000req/s for $12k/mo is pretty awful, and I'm being conservative.

Then let's talk about the $1MM/mo S3 bill. I'm not sure how to make 8PB cost that much but even the lightest touch at optimizing storage or compression or caching knocks the cost down.

This is all just nuts. There's no reason this all shouldn't be running on AWS or GKE with a much smaller bill. Their apps are predominantly CRUD, some email. Instead they replaced kubernetes with an in-house monstrosity.

I am bewildered.

[0] https://dev.37signals.com/our-cloud-spend-in-2022/

[1] https://www.theverge.com/2020/6/22/21298552/apple-hey-email-...

[2] https://m.signalvnoise.com/on-apples-monopoly-power-to-destr...

[3] https://www.fastruby.io/blog/rails/ruby/performance/how-fast...

e12e• 3 years ago

After reading tfa - I'm actually a bit confused about MRSK. I'm not sure I'd want to run MRSK and Chef? They seem more orthogonal than complementary?

Ed: at least between Capistrano and Chef it seems that MRSK is redundant?

benreesman• 3 years ago

So as an industry we've been having some version of this debate (at FB we were having it at least as far back as 2014, my org was IIRC the first big one to test-drive our Borg-alike container solution).

These days I think maybe it's just that classic dilemma: over-design and over-build to be ready for contingencies, or build just what we know we need and maybe get caught with our slacks down. This goes by a zillion names WET vs DRY, YAGNI, microservice vs monolith, there are countless variations on the same core idea.

If you start with PHP and MySQL and a chain-smoking sysadmin, and you get hit with hyper-growth then you adapt or die, and you have a mountain of data to figure it out. This is paradoxically an easier decision tree (IMHO) even if maybe some of the engineering is harder or at least higher-stress.

But by far the more common case is that we're building something that isn't huge yet, and while we hope it goes huge we don't actually know if it will: should we build more features and kinda wing it on the operability/economical/automated can of worms, or should we build for the big time from day one?

I think it's a legitimately hard set of questions and reasonable people can disagree. These days I think the only way to fully screw it up is to get ideological rather than pragmatic about it.

liampulles• 3 years ago

I think there is a space between managed K8s on the cloud and e.g Ansible managed docker deploys on-prem. I'm curious to see how it pans out.

greendestiny_re• 3 years ago

Sounds to me like 37signals uses the risk aversion paradigm typical for stagnating businesses — instead of building and refining their strengths they're fixated on mitigating their weaknesses.

zubairq• 3 years ago

37 Signals has many technical people and can afford to de-k8s. But K8s is designed for a totally different use case, for large corps where there are most non IT staff but where IT resources and standards need to be managed in a more central way. Most standard banks or large companies do not want to roll this stuff by hand, they care about STANDARDS!

ignoramous• 3 years ago

That's the thing with technology though, it goes mainstream as adoption grows. RoR started small at 37Signals and eventually became a standard. MSRK might yet be one, there's no telling right now.

smetj• 3 years ago

It's anecdotal but the sentiment I have is that the Kubernetes ecosystem drains an even bigger part of the collective effort required to provide business value. I believe many engineers have a disconnect on what it means to provide real business value. Solutions like Kubernetes are designed to be able to accommodate an endless number of scenarios out of which you probably only need 1 to provide value for your business. The consequence is that there's a disproportionate ratio of Kubernetes possibilities hence complexities vs. the simplicity of your requirements. Once your workload runs on Kubernetes, you cannot afford to ignore the complexities of Kubernetes so you are automatically sucked into the rabbit hole.

Lapsa• 3 years ago

well... in my book, k8s has always been in "Dinosaur" category. somewhat useful, somewhat versatile, perhaps even good. quick glance on documentation eradicates any desire to learn the tech

jwmoz• 3 years ago

How much users/traffic does Tadalist have?

DeathArrow• 3 years ago

At one of my former workplaces we ran Kubernetes on premises and it worked like a charm. I still think that Kubernetes can be a good fit for microservices even if you use your own hardware.

namaria• 3 years ago

Kubernetes is an amazing tool. Cloud computing is a powerful way to leverage a small team and prototype stuff quickly.

Ocean going ships are impressive pieces of kit. CNC machine tools are a powerful way to leverage small teams and manufacture high quality stuff quickly.

Now, telling every repair business in town they need robotic lathes and a fleet of major cargo ship is nonsense.

Why this kind of discourse thrives in software is beyond me.

daitangio• 3 years ago

K8s has some cognitive overhead. For simple deploy a docker client-server with docker-compose is a winner, see misterio[1] which basically leverage docker-compose +ssh

But when you need to guarantee system will auto-restart, and healthchecks and so on, K8s is the de-facto standard.

Helm template language (based on Go) is not ideal, but it difficult to replace K8s nowadays with simpler systems.

[1] https://github.com/daitangio/misterio

zac23or• 3 years ago

Wow. "K8s is simple", it has the same vibes as Linux user vs Dropbox:

'...you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem' https://news.ycombinator.com/item?id=8863

https://www.theregister.com/2021/02/25/google_kubernetes_aut...

I love HN's disconnection from reality.

cmeacham98• 3 years ago

It's not that Kubernetes is simple (it's not), but Kubernetes is relatively simple compared to the task it accomplishes.

If you have containers that need to be scheduled and networked and supplied with storage and exposed to the internet across a large set of machines past the scale where you can easily do so with tools like docker-compose, Kubernetes (might) be for you. There's a good chance it will be simpler to understand and reason about than the homegrown kludge you could make to do the same thing, especially once you understand the core design around reconciliation loops.

That said, you might not need all that, and then you probably shouldn't use Kubernetes.

chriszioutas• 3 years ago

have they thought to just run openstack on their own servers? everything I saw leads to me to use saltstack + openstack as they dont wanna be on cloud.

Cwizard• 3 years ago

I think it is cool they are developing new tooling. I don’t understand all the negativity. Isn’t it good that people keep innovating in this space?

Also how is this different from deploying before k8s and TF etc was a thing? We would write our own scripts to manage and deploy our servers. This is the same no? Just bit more structured and it has a name.

De-cloud and de-k8s – bringing our apps back home

Related Stories