Signals says cloud repatriation has already saved $1M

https://www.theregister.com/2023/09/18/37_signals_cloud_repatriation_savings/

by rcarmo • 3 years ago

31 47 3 years ago

Had a student do a research thesis on this last year and IIRC whether it's a massive cost saver or an expensive disaster boils down to good old fashioned "system analysis".

Know your system, every part. Big cost savings are often made at this stage just discovering a shit-ton of obsolete stuff.

Don't just map the network, map the _function_ of the network. Then look at its logical versus physical topology for anomalies (also a great security exercise).

What you bring back on-prem must be systematically viable.

Factor in on-prem energy costs, they are not trivial.

The biggest overall factor is people and wages. Any significant repatriation project ends up needing to hire back the system administrator you fired a couple of years ago.

blitzar2• 3 years ago

boils down to good old fashioned "system analysis". Know your system, every part

This type of answer has managed to talk me out of some amazing roles (and dodge some horrible people).

Apparently the correct answer is "there is a magic formula whereby you dont need to understand anything or do any analysis and it will deliver you sunshine and rainbows"

thelastparadise2• 3 years ago

Reminds me of a (public) talk by DIRNSA on security a few years back.

The talk boiled down to: know your network. Know the traffic on it. Then you have a chance of being secure.

vidarh8• 3 years ago

Actual on prem is not for most people. Managed hosting in a data centre run by someone else is the halfway house (and what 37Signals is doing). You get most of the savings, and very few of the pain points.

One of the biggest benefits besides egress should be people and wages. When I consulted in this space, clients on AWS were gold dust, because they tended to spend far more.

A significant reason is that you tend to be able to drastically simplify when you can pick hardware that fits your workloads rather than what fits your cloud provider, and can forego all the multi-tenancy complications your cloud provider needs to account for.

coastermug• 3 years ago

Are you able to share that research thesis? Or any key papers on how this should be done? Knowing SOTA on these approaches is always nice.

nonrandomstring2• 3 years ago

Sorry no, I quit teaching at that uni and don't keep contact with the cohort now.

But a little more info I remember:

It arose under a project suggestion three or four students took me up on, and that was "Long term resilience".

The obvious antecedent being "If Google goes down tomorrow....."

"De-clouding" was in the title. And the conclusions, perhaps predictably, were that hybrid systems were the sweet spot across a range of business types, with no panaceas, or even decent maxima to be found in fully on-prem or total cloud.

The students artifact IIRC was a "matrix" (a graph theoretical model) of the edge-costs for traversing well modular systems into different states - something that could be used to plan a "repatriation"

FWIW the student was African and we had a good laugh at the racist undertones of that language and it's subtle implications about "Empires".

mdhb• 3 years ago

I spent over ten years in the Rails community and as a result consider myself fairly well versed in the things that tend to come out of the mouth of DHH.

This particular one is very much in line with some of his other greatest hits where the more you dig into his claims and assumptions the more things don’t seem to always add up in quite the way he is presenting them.

Without going super deep on the topic, he came to cloud with his on-prem architecture choices, made very few adaptions to take advantage of what cloud is good at besides trying to slap K8s around it and then was upset that the benefits never materialised.

I can understand that being a shitty and painful experience but he is now acting as an anti-cloud evangelist of sorts while arguing against a series of constraints and choices that he picked and clearly weren’t appropriate for his business.

But now for him that’s all that cloud can ever be as it’s all he ever knew.

willtemperley• 3 years ago

So in reality they've done two unnecessary migrations, burned a ton of cash on cloud compute and are selling it as a success?

sgarland4• 3 years ago

While I don’t doubt your personal experience, I’ve spent time at a cloud-native company and boy, do they know how to spend money. Just because you’ve bought into every .aaS the cloud has doesn’t mean you’ll save money.

RDS, for example, is stupidly expensive. I maintain that it’s not that hard to use Xtrabackup or any of the other free and open tooling to run regular backups for RDBMS. It’s also not that hard to set up HA. If those things seem hard, your Aurora Serverless bill is going to be even harder to swallow when it hits.

ckdarby2• 3 years ago

Sounds like a great business opportunity for you that will make millions and millions.

flanked-evergl• 3 years ago

I dunno, to me the argument is basically, "We can outcompete our competitors and the cloud provides at once". I doubt it, actually, because it requires 37 Signals to make tradeoffs against their primary business, which is whatever people actually pay them for, in favour of something which is not their primary business, i.e. running a cloud platform.

But sure, if all they buy from AWS is VMs, may be better.

pclmulqdq• 3 years ago

The trick is that 99% of the engineering expense at cloud providers is to enable weird multi-tenancy use cases and scale to obscene size. You don't have to do either of those if you run your own hardware - your infrastructure can be quite simple.

sixhobbits• 3 years ago

"37Signals" , i.e. basecamp, Hey, etc. I know HN strips out numbers automatically to remove clickbait articles like "37 signals you need to watch for when buying a SaaS" or whatever, but pretty annoying when it butchers something like this.

opan• 3 years ago

Thanks, I missed the second S and thought this was about the Signal messenger.

darthrupert• 3 years ago

AWS is always more expensive than owned hardware + good team -- it's after all what they generate their profits from. However, it's also probable that they were using the cloud wrong. Their software (afaik) is a typical 3-tier application coded on Ruby on Rails, which probably implies that they used just EC2s, with probably an orchestrator on top.

Also, if they didn't need any new staff after moving to their own server rooms, this implies that either their staff was having lots of free time or that their staff currently is overworked.

vidarh8• 3 years ago

When I've done contract devops, clients on AWS always tended to need more of my time. The level of devops complexity for AWS is often far higher than self hosted.

It's really hard not to beat AWS on cost, including person hours to manage the setup.

johnklos2• 3 years ago

Of course it saves money. The main selling line of the cloud is you're too dumb to administer, so just trust them and throw money at them.

If you're not too dumb to administer, then running your own hardware is invariably cheaper.

What's surprising is that so many otherwise technical people believe the cloud BS without really thinking things through. Anyone who can do basic maths can see that aside from trivial uses renting is always going to cost more in the long run.

aaronharnly• 3 years ago

I don’t think that’s the main selling point of cloud? The main selling points are that (1) you can scale capacity up and down, so you aren’t purchasing, provisioning, and operating hardware for peak demand (2) with infra-as-code tooling, software teams can configure and deploy significant infrastructural capabilities without scaling specialized operations roles. Dumbness does not play into it.

sgarland4• 3 years ago

IME, much of the “need to easily scale” is fixed by understanding your bottlenecks, being able to understand the metrics being captured, and how to address them. You would be shocked (or not) at how few people raised on cloud can explain what IOPS are, or what IOWAIT is, let alone linking the two up.

Re: IaC, you definitely should understand what it is you’re instantiating (and why), no matter how you’re doing it. AWS Community TF Modules are all well and good until something breaks, and you realize you don’t really understand what’s going on.

vidarh8• 3 years ago

My experience in doing devops on contract for years, though, is that what instead happens is that a bunch of different teams scale things up beyond what they need, don't take things down again, and nobody understands which servers are actually needed or why they're being billed what. There's good money to be made in just helping companies figuring out their cloud billing.

I also don't for a second agree that software teams can deploy significant capabilities without specialised ops roles - if anything my consulting showed me the clients I had that deployed to clouds needed far more help on average.

It may be a selling point, but it's a fiction.

And almost every single colo-provider and managed hosting provider these days also provides cloud services, so you can lease servers or racks, and get cloud instances often in the same data centre tied into your network if you suddenly need to handle a spike. As a result, you can typically load your managed servers far higher today than you used to be able to.

thelastparadise2• 3 years ago

Right, and we're talking computers here.

Sure, they can easily be set up in a complex, failure-prone way. But if we keep it simple and stick to known fundamentals, they can run almost entirely passive.

Think: a basic debian stable server with SFTP vs. a cluster of some complex storage protocol.

The basic debian SFTP server can run for years and years with minimal intervention.

Cloud has become a cargo cult.

johnklos2• 3 years ago

Indeed.

Too often I hear about how people need "features" without even knowing what they think they need them for. It's like the drug companies selling drugs on TV - suddenly people are asking their doctors if they need some random drug that has absolutely nothing to do with anything wrong with them.

Your Debian SFTP server example is apt. Many times I've asked people why they think they need all sorts of complicated things when everything they need could be run off of a Raspberry Pi.

I've even set up Raspberry Pis to show people that they literally don't need more than that.

vidarh8• 3 years ago

A whole lot of people want to believe because hardware is a big unknown they don't understand that often has complicated requisition paths for them. It's tempting to buy all the arguments when it means you can just get more resources with an API instead of having to justify the need for them.

andrewstuart3• 3 years ago

The absolute bottom of the market lowest price AWS GPU spot instance costs 15 cents an hour.

That’s USD$1,314 per annum for an extremely underpowered machine.

The numbers don’t add up. Just buy a physical machine, plop it on a network somewhere, you’ll have dramatically more computing power and you won’t be paying for it again next year, and the year after that etc

“The cloud is where Moore’s law goes to die.”

ckdarby2• 3 years ago

Except the need for managing it securely. Oh, it is very successful? Now the 50 dev team each want one.

Oh, you maxed out the number of ports on the switch.

One of those GPUs have customer information that only select individuals should have.

One of them needs to write only for off-site backups.

Every time this on-prem vs cloud stuff comes up folks just skim over all the other pieces included. Didn't even talk about racking, network and or power.

Yes, there is a spot and time where on-prem makes sense and there is also a time for cloud.

I barely cover some of them in this 30 min video, https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb

Spooky23• 3 years ago

You’re adopting an almost religious dogma about this. At the end of the day, it’s a dollars and cents decision.

During development and growth of services, cloud makes total sense as you abstract the capital costs. If you’re a startup making Uber for cupcake makers, you’d be dumb to do anything else.

Depending on your needs, once things settle down, it’s often cheaper and more effective to capitalize your costs yourself.

It’s akin to a car. If you go somewhere for a day or two, Uber makes sense. For a week or more, rent a car, for a year, you probably want to buy or lease a vehicle.

Again, it depends on you. For me, I have large workloads running in GCP and Amazon. But 80% of the production I’m responsible for is “on-prem”, because it’s about 30% cheaper. YMMV

alberth3• 3 years ago

Lease a Server

You get all of above (mgmt, etc) for the same cost.

For example, you can get for the same $1200/year:

- 16 core (Zen4 current gen)

- 128 GB RAM (ECC/DDR5)

- 2 x 2TB NVME

For just $100/mo.

https://www.hetzner.com/dedicated-rootserver/ax102/configura...

I feel like the greatest marketing move AWS has played is getting people to forget that Leasing Servers is a thing.

meltyness• 3 years ago

Until some junior uses an instance store for customer data or creds.

vidarh8• 3 years ago

I've done racking, wiring up the network, etc. and if that adds up to more than ~3%-5% of your total spend, you're bad enough at it that you should lease spent at a managed hosting facility. That also makes your power predictable - it's baked into the rack rental most places.

In which case most of the steps you list boils down to "go to their website and fill in a support request" or tick some boxes, and if there's a wait, spin up a cloud instance in the same network in the meantime.

Very few people mean actually on prem when they say on prem. Usually the comparison is with managed hosting or at most colo providers where you can but don't have to wire things up yourself.

Sure, there are use cases where cloud is cheaper, but most people are nowhere near them. There are also cases where actual on prem makes sense. But unless you're somewhere where power and real estate is cheap, odds are your cheapest option will be managed hosting, by a large factor over cloud, and a quite a bit smaller factor over on prem.

yjftsjthsd-h• 3 years ago

Except the need for managing it securely.

Putting a server in EC2 doesn't make it secure.

Oh, it is very successful? Now the 50 dev team each want one.

Cool, now you're paying 50x as much.

andrewstuart3• 3 years ago

The tasks you are talking about is not rocket science.

Before the cloud most companies did this themselves.

You assume the only people with the capability to do this stuff are cloud vendors.

alberth3• 3 years ago

Source: https://world.hey.com/dhh/our-cloud-exit-has-already-yielded...

midwit• 3 years ago

How practical is it to run a server at your house? Is fiber a must? What if the power goes out? What about exposing your home ip?

unforeseen9991• 3 years ago

Colo is a much better option, but i've hosted at home before. I have fiber, UPS, and a generator. The home IP wasn't a concern, but you could proxy behind cloudflare or something?

datadrivenangel• 3 years ago

"We’re still storing eight petabytes of data in AWS’s S3 service. We’re leaving that one for next year. " - DHH in July [0]

That means 37 signals probably still have a 7 figure cloud bill

[0] https://37signals.com/podcast/leaving-the-cloud-the-finale/

Signals says cloud repatriation has already saved $1M

Related Stories