11 years of hosting a SaaS

https://ghiculescu.substack.com/p/11-years-of-hosting-a-saas

by ghiculescu • 3 years ago

334 219 3 years ago

ghiculescu.substack.com

I'm sure its interesting but I am making a conscious choice to not read or linger on user hostile articles. As soon as I want to scroll I am interrupted by a page takeover requiring action and subsequently compelling me to lose interest.

Did you want me to read it or click buttons for you? Too bad your UI gets in the way of the U.

SPBS• 3 years ago

Substack is definitely scummy in this regard, it's not really the author's choice (although a conscious decision was made to host it on Substack).

andai2• 3 years ago

I read a Substack article almost every day and still consistently get a little jolt of rage when the overlay fades in.

I guess they do it because it works, and if the average person felt the same way, they wouldn't be doing it?

ValtteriL• 3 years ago

Firefox's reader view gets rid of all that on top of reducing other distractions

jarebear6expepj2• 3 years ago

So does closing the tab.

awayto2• 3 years ago

We started on Heroku, because in 2012 if you did any Ruby on Rails tutorial that included deploying your app, you ended up with a Heroku account.

Couldn't be more accurate. Heroku and Rails is almost as much of a throwback as Node and Express. You just had to be there. And it was great. Web dev was always a hobby for me as a teen, but then I turned to the rails book as a means of learning a professionally designed system, when I wanted to get serious as a dev. It definitely served me well. Heroku, at the time, offered a very streamlined and accessible way to integrate rails. It was a great time to learn.

SenHeng2• 3 years ago

I sometimes wonder how my career (and life) would've differed if 11 years ago, I had chosen to learn rails instead of laravel because of familiarity with php via Wordpress. From there it went on to C# (windows), JS (frontend), scala/go/node.js backend + JS front end. Lots of JS.

Here being Japan, there were a lot of opportunities to pick up ruby/rails along the way but I stuck on the JS trajectory partially because I didn't want to 'start over' with rails _now_. There were only a few times when not knowing ruby/rails meant I was limited to specific tasks so it was never really career limiting. It did mean that I actively avoided working in ruby shops, for better or worse.

If I had picked rails instead then, would I have transitioned into more of a backend engineer with some frontend duties instead of the reverse?

Not that it matters, but there definitely is a tendency in our industry to 'look down' on frontenders as not real engineers and thus not consider them for leadership positions.

erhaetherth2• 3 years ago

Not that it matters, but there definitely is a tendency in our industry to 'look down' on frontenders as not real engineers and thus not consider them for leadership positions.

I think that's funny because there some to be a ton of backenders that can't do frontend at all. And then they want to look down on FE when they can't do it themselves? It's not just basic HTML and CSS if you're building a complex app.

I do both (FE & BE) so... I've seen it all and enjoy it all. Not sure one is easier than the other.

yurishimo• 3 years ago

It's not too late to go back! Rails and Laravel are still insanely popular and productive for getting work done.

I'm also disappointed that real frontend expertise is not valued as highly as a traditional backend engineer. Think about someone like an Architect. You could say that they are glorified artists (yes, I know they do more than that) and they are highly valued for their artistic input as well as their professional advice; but it's also not unheard of to have an architect solely provide artistic direction and a structural engineer provide support.

drawkbox4• 3 years ago

Node and Express. You just had to be there. And it was great.

I agree and Node and Express are still great when going with just js. Node and Koa more now but still great. Socket.io for real-time. All are great for getting things up quickly, simple and shipping things.

awayto2• 3 years ago

For sure. If it ain't broke don't fix it. Node and express are still great. I still use them. Heroku and rails would still be great to this day, too.

replete• 3 years ago

In context of your comment, what became better than Node and Express?

esafak• 3 years ago

So what are the new Herokus? What do you recommended for starting a serious project? AWS AppRunner? How about something open source?

ghiculescu13• 3 years ago

fly.io is the least bad.

But honestly nothing these days is as good as Heroku was 10 years ago, if you just want to put something online and don't care how it gets there.

babyshake• 3 years ago

Vercel for serverless apps is very easy to do.

kkapelon• 3 years ago

Check railway.app, fly.io, render.com

Possibly qovery.com as well.

mstocks• 3 years ago

FL0.com is new and looks v solid

Nicholas_C• 3 years ago

I see Render being mentioned on HN quite a bit as the new Heroku. I’m still using Heroku for my personal projects but have been meaning to check Render out.

JanSt• 3 years ago

Digital Ocean App Platform is really good for what I run :)

jiggawatts2• 3 years ago

Azure App Service is a thing that everyone just skips over, listing competitors with an annual revenue smaller than some individual Azure App Service deployments...

I mean seriously, railway.app has just fifteen employees in total, and revenue in the single-digit millions.

Reminds me of when someone posted a "cloud storage vendors poster" with hundreds of vendors -- but not Azure -- some with revenues smaller than the cost of an Azure Storage Account one of my customers was using... to back up a single server. By accident.

andai2• 3 years ago

I think Railway nails the user experience, but basically every time I use it something goes wrong in a small or a big way (build fails randomly, UI glitches, random 404s in the control panel that fix themselves minutes later).

Used it for a few clients but it's tough to keep using it with all the issues, so I'm looking at Render and other alternatives. But if Railway could smooth out the reliability / stability, I wouldn't have any reason to switch.

flybayer• 3 years ago

https://Flightcontrol.dev is a relatively new one that provides a PaaS-like experience on your own AWS account

*I'm cofounder & CEO

zoomzoom• 3 years ago

I'am a cofounder of withcoherence.com, a PaaS developer experience on top of your own cloud. We are not open-source, but we support both AWS and GCP and provide a free tier of usage. Happy to answer any questions, feel free to get in touch!

atomicnumber3• 3 years ago

Can it be that between docker and digital ocean giving you a gig of ram for $6/mo, people are just DIYing it much more often these days?

irsagent• 3 years ago

A recent observation, and this could be a biased one, you mentioned:

we’ve spent a bit of time learning more about moving off the cloud to a managed data center. But the nice thing is not feeling like we need to.

That SaaS's are moving/thinking/completed to running their own infustructure away from managed services like digital ocean and aws, like 37Signal(Basecamp & Hey).

aetherspawn4• 3 years ago

Would be interested to know what your hosting bill was like?

We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two).

(1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.

ghiculescu13• 3 years ago

It's around 150k USD/month at the moment.

Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.

aetherspawn4• 3 years ago

Our deployment is much smaller than yours! Our upfront hardware cost including carpentry and HVAC was around $80k, our monthly cost in cloud was around $5k/mo and now we pay only about $1k/mo in energy (which includes the existing $500/mo bill from the office). We could reduce this further with solar panels, but we share the tall-ish building with 13 other suites and we couldn't get approval to take up most of the roof.

Sai_• 3 years ago

fyi - tanda.co gives a 403 forbidden error

martinald• 3 years ago

What does spend break down as?

welder• 3 years ago

Holy shit that's more than I expected for your type of business. I'm running a SaaS too with half a million users and a lot of data processing, but my hosting costs are way lower... under $10k/mo

Buttons840• 3 years ago

We expected the HW to fail often and such, but in 2 years ... nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two).

I began my career by scripting Asterisk using Python. I created a daemon using Twisted and that process ran for 3 years without fail (as in, the same Linux OS process was running for 3 years). This was on commodity hardware.

withinboredom4• 3 years ago

IME, if a drive is going to fail, it's going to fail in the first hundred hours or so of use. In my entire life (of using hundreds of disks), I've only had three failures. Two of those were on the same server, at the same time, in the same RAID array and took out the whole array -- the data was gone forever. Thank the gods for backups.

So, just make sure you take backups pretty regularly because you never know.

8organicbits• 3 years ago

since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore

This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually.

[1] https://instances.vantage.sh/aws/ec2/c6a.32xlarge

[2] https://www.hetzner.com/dedicated-rootserver/rx170

cjalmeida• 3 years ago

This is the way. Unless you are huge, self hosting and even colo is not competitive with long term VM commitments in hyper scalers.

They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.

aetherspawn4• 3 years ago

One of our 100 core boxes is Windows Server and it looks like they want $79,949 annually for it.

riku_iki3• 3 years ago

On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually.

plus more expensive network for cloud

olive247• 3 years ago

Does geolocation of the server not matter for your application? I am curious if there are any latency issues with this solution as opposed to a hosted option.

rozenmd4• 3 years ago

Use managed services for as long as possible

Big agree here.

Yes, you can save stupid money by handrolling postgres on an extremely beefy Hetzner server, or you can pay someone else and keep building your product: https://onlineornot.com/self-hosting-vs-managed-services-dec...

This isn't to say, "don't bother learning how to do it yourself", but more "learn to pick your battles".

Radim• 3 years ago

It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine.

I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different teams.

erhaetherth2• 3 years ago

I've been on unmanaged MySQL for ~8 years now. Considered switching to managed but I'm not seeing any performance or stability issues, so I guess I'll just keep this train going until it craps out on me, then restore a backup onto a managed service, say sorry for the downtime, and that'll be that.

raverbashing• 3 years ago

I agree

And more to the point, learning how to "handroll" Postgres could be beneficial. You could have learned about options for limiting the amount of memory, etc

Sure, managed is easier and use it when you can afford it easily. But before that, it's better to see how things are going (mem usage, disk usage, bottlenecks, etc)

speedgoose4• 3 years ago

I'm looking at the ansible playbooks to setup my favourite beefy baremetal Hetzner server (128GB ram, Ryzen 9 5950X 16-Core, 450Gb fast NVME SSD, 3.5TB x2 NVME SSDs, 155€/month):

- Install Debian 11 while booted in rescue mode.

- Setup the root file system encryption using cryptsetup and dropbear (to enter the key during the boot through SSH). Involves chroot and some fun commands.

- Setup ZFS encrypted mirror filesystem for the two additional SSDs.

- OpenSSH hardening and Teleport installation.

- Kubernetes installation (K3S)

- Connecting kubernetes to my Argo CD instance or an existing Kubernetes cluster.

And then through GitOps:

- Installation of openebs-zfspv

- Installation of kube-prometheus-stack helm chart

- Installation of (many) postgresql instances and other craps

I have been playing with Linux servers for 20 years and I find this fun and rewarding. But I do understand people saying that baremetal Hetzner is not for everyone. Especially if you start to have requirements such as "data must be encrypted at rest".

SanderNL• 3 years ago

I don't think "modern" stacks are sane enough to "handroll" anymore. Sure, you can do it, but look at the poster in this thread that details the setup of a debian server.

Kubernetes, "Argo CD", zero-trust, the sheer amount of "management" is off the chart.

"Installation of kube-prometheus-stack helm chart".. "Installation of openebs-zfspv"..

It's not postgres that's the problem here.

tacker20002• 3 years ago

Yea i also dont see the point of having everything “managed”.

RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB.

If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine.

But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.

Ambix2• 3 years ago

Yep, I'd always prefer freedom and power of hosting my own PG instance upon a some robust VM offering to guiggling with clumsy AWS menus.

PaulRobinson• 3 years ago

If you're managing AWS infra through the web application, you're definitely not doing things as per AWS-prescribed best practice.

drawkbox4• 3 years ago

Yes especially data, simplicity is key as well.

Where possible go with simple but abstracted cloud storage, cloud tables and then a cloud db that is managed. We use Azure mostly right now but our storage system works across Azure storage, Amazon S3, Google Cloud and others. For tables, Azure Tables mainly. For database with filtering/paging better performant and ACID compliant cloud db, CosmosDB currently which is a dream with the differing apis (SQL, Mongo, Cassandra, Tables style). The more you can avoid vendor or dev-lockin the better so simple formats/messaging/routing and abstracted specifics/implementations.

When you store data in storage or a cloud db the scaling is "infinite" and you can also snapshot or backup to anther one, you never worry about data.

The front ends and APIs are mostly repos pushed to app/web services and everything else in data storage. Super simple and anywhere you need some special service that can be serverless or a dedicated setup, like maybe a RDBMS, chat server, network server or WebRTC/socket endpoint that interacts with the simple side. These managed as well if possible, though not always. Additionally, build cheap and horizontal scaling on web/real-time servers. Vertical scaling and sharding is for suckers.

Side note: CosmosDB is like a combination of NoSQL, document databases and GraphQL and it is ACID compliant and you can do REST or SQL, it can even wrap MongoDB and Cassandra and make them ACID compliant. It really feels like the best way. Not many have all that and ACID compliant. Not even Amazon Redshift has that, DynamoDB does if specified. Google Firestore does if specified. I used to be big on RDMBS Oracle then MSSQL then PostgreSQL and those are great for backing/reporting etc but CosmosDB combines all the power of RDMBS, NoSQL, document databases, and ACID compliant with little worry about scale. It is vendor lockin to Azure, which you can route around with platform abstraction, but currently can't be beat. As you got that clean API layer you could change later but best way is limited/clean and if possible, non breaking change API layers/signatures.

alsiola2• 3 years ago

Really do not share your experience with CosmosDB, some of it's attributes that currently make me miserable include...

- Scaling is not infinite, it's up to 20gb per partition key (1), which can't be changed after document creation.

- One set of global indices, no equivalent to DynamoDB's secondary indices.

- Still can't run their docker container on mac (2) natively.

- Weird SQL-like dialect that's required for all but the simplest queries. JOINs are spectacularly awkward.

- Tooling is horrific. Based on the previous point, no existing tooling works for it (and nobody is building tooling for a DB with such minimal market share). For example, I needed to manually deleted 30 or so documents/rows yesterday - only way to achieve this is with 30 separate click-to-deletes in their UI.

- Minimises analytics options. There exist a plethora of business intelligence type tools that will happily sit on top of most common DBs. None of them like CosmosDB. So you're stuck with synapse link or whatever MS calls it now.

Overall it seems to combine the worst aspects of both RDS and document stores, with the worst aspects of both traditional and serverless infrastructure.

(1) https://learn.microsoft.com/en-us/azure/cosmos-db/partitioni... (2) https://learn.microsoft.com/en-us/azure/cosmos-db/local-emul...

mamcx• 3 years ago

Where possible go with simple but abstracted cloud storage

This is the total opposite of simplicity. Simple is PG and backup setup. DONE.

Maybe you mean "comfort"?

mejutoco2• 3 years ago

Or you could use sqlite until you need postgres. I have to admit I reach for postgres immediately when in many cases sqlite would have served me just as well.

kristiandupont2• 3 years ago

SQLite seems to be gaining popularity with even larger projects which is surprising to me. As I see it, the big value prop of SQLite is that it runs in-process which, for a webapp, is almost nil?

Other than that, it's not like queries are any simpler and the "simple" type system is, in my opinion, not a feature. I get that some might disagree with that.

Is there some other reason why you would prefer it?

ghiculescu13• 3 years ago

I’m super intrigued by SQLite these days.

11 years ago you used Postgres because that’s what Heroku told you to use.

jordiburgos• 3 years ago

Which one do you recommend?

Those get quite expensive.

rozenmd4• 3 years ago

I personally use RDS

rldjbpin• 3 years ago

having no skin in the game, I feel this is a very interesting view, especially when you are offering a managed service building on top of another.

go through enough levels of dependencies, and someone who could do a few layers of the onion in-house can sweep in with competitive pricing/offerings a la sherlocking.

ksajadi2• 3 years ago

the progression from Heroku to AWS is very common as SaaS businesses grow. However one thing that’s missing from all “alternatives to Heroku” is what I call life after deployment or Day-2 operations. Many solutions show you a sleek demo of how easily you can deploy a Hello World app to their platform but leave you in a straight jacket when it the app is deployed and you need tools to keep it running, like changing an Nginx (or equivalent) setting or using persistent storage or a Postgres contrib.

Making decisions about infrastructure is as much about day-2 as it is about the initial deployment. I see fly.io and render in this camp. Shiny day-1 docs and demos and then wishing you good luck when you need a `rails c` to see something in the DB

ghiculescu13• 3 years ago

That's what I'm struggling with. You're not wrong, but I feel like it's way too easy to go to the other extreme and suddenly you're way in over your head.

ilchalpenl• 3 years ago

The title is correct. But any of you changes or lessons are perfectly fine. (Sure, a bit awkward to do ssh from cafes). May be reading the Google SRE PDF (or equivalent) would have been a bit more useful. At my wife's business one of the team member shall LOGIN to DO/AWS every month send a screenshot that account/CC is in good standing.

wouldbecouldbe2• 3 years ago

vercel is great

ksajadi2• 3 years ago

How’s rails deployment experience with Vercel?

lvangool2• 3 years ago

This precisely the niche Cloud 66 tries to fill, it caters for the growth businesses, providing initial ease but without sacrificing control. This allows companies to grow without being forced out before they are ready. (Disclaimer: I work for Cloud 66)

jmuguy3• 3 years ago

Does Cloud 66 have any resources on migrating a large (~300 GB) Postgres DB off Heroku, with minimal downtime? This has been our biggest sticking point when thinking about leaving Heroku.

timwis• 3 years ago

Can’t you do rails c in fly via the fly ssh console command?

rozhok• 3 years ago

The point is you can do it from Heroku UI.

flippy_flops2• 3 years ago

I hate what Salesforce has done to Heroku but after a year of running our saas on aws, render, and fly - here I am, back on Heroku. It sucks, but slightly less than the alternatives.

Fly has potential, but they've changed/grown so much that most docs are out of date, everything is buggy, support is not very responsive, and their security posture leaves a lot to be desired. Not a fun place to be production issues pop up.

suryao• 3 years ago

I'm building Argonaut to be a layer on top of AWS/GCP to make them less sucky. The primary use case is startups that have a bunch of services and have outgrown heroku etc. I'd love to chat if we can help - you get the stability of AWS and a push to deploy dev experience for your apps.

friism2• 3 years ago

(I work at Heroku) Do you have more details on what sucks? Anything we're not already tracking to fix in our public roadmap? https://github.com/heroku/roadmap/issues

anurag• 3 years ago

(Render CEO) What can Render do better?

domlebo70• 3 years ago

Alex! Great to see you on HN.

ghiculescu13• 3 years ago

Hey mate!!

l5870uoo9y• 3 years ago

We didn’t take advantage of any other platform features, we just treated AWS like any other VPS.

This is a lot of switching around for a plain VPS.

joshstrange• 3 years ago

Yeah, if you aren’t going to use AWS managed services then what are you doing on AWS? EC2 is great but if that’s all your using your probably overpaying.

polote• 3 years ago

Well I do not feel any of the 3 advice he gives, good advice.

First the longer you stay in heroku ,the most complex is it to exit it the time you really need it and the less flexible you are in the time being.

Second, wish he had pay for a pit team sooner, but could this money better used investing in marketing or sales like he probably did ?

The guy has obviously succeeded as a business owner, would it still be the case if he had implemented these advice ? We will never know, but what we know for sure is not implementing these advice made him successful

codetiger• 3 years ago

I think he meant, to stay with managed services as long as possible. Which makes sense, as in the early stage, the focus has to be the functionality rather than cost/performance optimisation.

ghiculescu13• 3 years ago

I think our success is despite a lot of decisions (specifically the ones in the post), not because of them.

kakwa_• 3 years ago

We will never know, but what we know for sure is not implementing these advice made him successful

"Made him successful" implies a causality that is a bit too strong.

Indeed, we will never know for sure.

Maybe implementing these advice would have impeded development of other critical areas of his business.

Maybe it would have would have helped make is business more successful as he would have had a more reliable product.

Or maybe the business impact would have been neutral, but would have resulted in better quality of life/less stress for him and his employees.

But in general, the way I read this article is: they made good decisions overall, but as everything in the world, it was not optimal (switching platform too early, making some big mistakes like the credit card one, etc).

It's a very interesting read nonetheless, with clear take away:

* chose boring tech you know and focus on your product, not the tech, specially in the early days

* grow your infrastructure and complexity with your product needs

* accept you will mess-up but properly learn from it, and grow your organizational knowledge, structure and processes accordingly.

dom_hutton• 3 years ago

If I could have my time again I'd focus on enablement of the broader engineering group through an incident management process, readiness exercises, some aggregate incident analyses for the org to learn from and leaning into observability.

Infrastructure is kind of a solved problem for common use cases today, just requires the expertise.

With our ducks in a row I'd next have look to a GRC function for the compliance bits whilst splitting the platform engineers time between embedding engagements and tooling investments.

You're on the right path man, I'd love to know what I know now back then but unfortunately time doesn't work like that.

fullstackchris2• 3 years ago

Infrastructure is kind of a solved problem for common use cases today, just requires the expertise.

This is the problem however for many (older) companies. They either don't care, or quite literally don't know the infrastructure solutions out there which can save literally thousands of hours per year of headache. Sure, for many companies with legacy systems they have a "dont fix what isn't broken" mindset, but from what I've seen, I always ask, if shipping and modifying new versions of a system takes hours or even days to complete, is the system really not 'broken'? I guess I never realized it, but having automated and clean infrastructure with tests and uptime metrics is a must-have for me on anything I build going forward. Take 2-3 weeks to save months of headache.

ghiculescu13• 3 years ago

Thanks. Means a lot, particularly coming from you.

shri_krishna• 3 years ago

Get a 403 Forbidden when I visit https://www.tanda.co/

rathish_g• 3 years ago

It seems that the new PIT team was unable to manage the HN traffic.

djvu97• 3 years ago

They should switch back to digitalocean when things were manangeable

AndroTux5• 3 years ago

But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to SSH. This wasn’t totally their fault, but at the time it just all felt like a dodgy, shaky setup.

This wasn’t *at all* their fault and there’s nothing dodgy about it. This was entirely your fault, and even mentioning this as a „partly“ negative for DO is imo very wrong. I imagine any other service provider, including Heroku, would have acted the same way. You can’t expect to mooch off a month of service without paying and without consequences. If you don’t pay your bills, service will be interrupted. It’s as simple as that.

withinboredom4• 3 years ago

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service.

DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

onion2k• 3 years ago

DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for.

Email is the expected form of business communication in 2023. I don't think any of the hosts I use even have my physical address. Maybe they could SMS me. I doubt they have my phone number either though.

I would expect a billing warning notice when I SSH into the service though. Maybe that was in place for the 30 days before they shut the customer out. I certainly hope so.

auggierose• 3 years ago

What countries are those?

scott_w3• 3 years ago

These laws are usually for utilities providers to residential customers. The aim is to stop people in poverty freezing to death in winter, not to protect a business with poor operating practices.

huijzer• 3 years ago

physical paper notices

What is this? Germany?

chillfox2• 3 years ago

I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.

AndroTux5• 3 years ago

For a business that makes checks notes 50 bucks a month in revenue with DO? If this were the case, prices would be astronomically high because there would be tons of scammers and bad actors abusing this system, leaving their servers running for months, because apparently DO has to send them a letter via snail mail to Antarctica and wait for their response. All of this would then have to be paid for by legitimate customers as for DO to break even, which means a simple droplet would probably cost around $50 alone. So no, thank you.

For larger businesses with thousands of dollars of revenue per month? Sure. For small $50 charges? No way.

MattGaiser• 3 years ago

The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored.

The using of an unmonitored email is the one that really gets to me.

ghiculescu13• 3 years ago

Yeah. I'm not proud of it.

rocho2• 3 years ago

And then there's Twitter who doesn't pay GCS and AWS for months and keeps operating normally.

AndroTux5• 3 years ago

Yeah, Twitter is totally comparable in that case, $50 a month is exactly the same thing as however many Millions of Dollars Twitter is paying to GCS/AWS with presumably custom contracts. But just to humor you: https://techcrunch.com/2023/06/14/twitter-is-being-evicted-f...

scott_w3• 3 years ago

I can guarantee you they’re not ignoring emails from their providers, if this is the case.

Account management involves a lot of discussion and your account manager will keep the service running if they think the bill will eventually get paid.

Dowwie2• 3 years ago

It's not as simple as that. Do you know how quickly a business loses customers without being flexible? It's also reputation damaging for the market to know that DO draws a hard line.

karamanolev• 3 years ago

What is being flexible? They gave 1) a month of unpaid bills 2) emailed the customer N times 3) gave the option to enter a backup credit card. The customer made it so neither of these worked. I expect that DO deals with a lot of non-paying customers. I wouldn't put it on them to what - figure out where they live and drive up to their house?

pontiacbandit8• 3 years ago

They literally send an email every single day for a month. And they are flexible enough that if you do write them they will happily extend that grace period.

It’s like saying car is a shitty transportation service because it stops working once the gas runs out.

Not paying attention the the reserve light, instrument msg and pings is solely your problem. More so because you are running a business…

nine_k• 3 years ago

I'd say that your service is dead unless proven otherwise. Your bills are unpaid unless proven otherwise. I'd be alerted by absence of regular emails telling me that my card was billed for the hosting service.

dsr_• 3 years ago

It turns out that humans are very bad at noting the absence of a signal.

Computers, on the other hand, are perfectly happy to use an expired timer to generate mail, a dashboard icon, or any other shout for attention that you please -- as long as you actually set it up. Paying attention to it is still a human problem.

everybodyknows• 3 years ago

I'd say that your service is dead unless proven otherwise.

I'd say you're dead right -- this is the only safe assumption. And any email client smart enough to check incoming email against a regexp could be extended to raise a not-easily-ignored alert if no pattern match (indicating success in processing a payment) had arrived by some chosen date of the month. Yet I've never come across such a feature in Gmail, Fastmail, nor Thunderbird. Maybe someone sometime has hacked such a thing into Gnus ...

tommica• 3 years ago

I read the "dodgy, shaky" parts as their own setup was that, not digital oceans

alberth• 3 years ago

But it kind of is.

Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire.

Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.

TheNewsIsHere3• 3 years ago

It’s definitely not the merchant’s responsibility to catch when your card changes.

We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account.

That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.

waboremo• 3 years ago

These are regionally restricted on top of users being able to opt out of such service at both the visa/mastercard level and at their local banks. This means you cannot rely on the service entirely and have to create alternative flows even if the update system works perfectly. Both Visa and Mastercard detail for developers the manual methods they can use, which still require human monitoring/input. They also both warn people about monitoring this at the merchant level so you minimize disruptions, late fees, cancellations, etc.

It's not as seamless as you're making it sound.

dangerboysteve• 3 years ago

Totally agree! Super dick move for dumping on DO when this was all your fault.

kodah• 3 years ago

Digital Ocean has an invoices API: https://docs.digitalocean.com/reference/api/api-reference/#o...

No need to use email at all.

troyvit• 3 years ago

Yeah, Tanda's own terms of services[1] are more restrictive than DO's. They "restrict or suspend" all services (or else terminate the agreement) if the client is 14 days late. This vs ignoring billing alerts for a month and being surprised when the service is shut off?

[1] https://www.tanda.co/terms-conditions/

voakbasda• 3 years ago

Whatever happened to companies actually reaching out with a phone call to an actual human in order to resolve these issues before they become a problem? A provider that simply throws a switch because a credit card expired is not one that I would trust. So, yeah, DO (and every other provider that operates that way) deserves a little bit of shade for actions that clearly demonstrate a lack of care for their customers.

yifanl• 3 years ago

They were paying all of $40 a month, I'd imagine that doesn't buy them very much customer support from any decently sized cloud provider.

alexpls• 3 years ago

This really resonates. It's so easy to get caught up in chasing trends and overlook delivering customer value/stability.

Good luck on the next 11 years :)

nicbou• 3 years ago

This is one of the best ops writeup I've read. People are rarely so vulnerable and relatable. Usually it's all "here's how we do things perfectly at our unicorn company".

The most important lesson was about reducing the alerts and sleeping better. It's important to be able to leave your laptop at the office and not worry about things going wrong. This post gave me flashbacks of when I made websites for others and hosted them myself. Never again!

fullstackchris2• 3 years ago

Great post, I've used all these services once or more at various points in my career. I'm still on the Digital Ocean stage, as I've found AWS is often overly complicated (or at least too much for me as a primarily solo founder) I'm curious about this line:

We were doubling the size of our customer base every 9 months, and pretty soon this meant we needed more servers.

I've found that a moderately sized 4 core 'droplet' from Digital Ocean handles nearly anything I can throw at it (at least in terms of the web application world) - I see you product is doing employee management / scheduling - is there heavy compute / resources associated? Or is it more due to privacy requirements (i.e. you need a siloed instance / server per client?). Not trying to sound smug at all, just genuinely curious where the load comes from.

ghiculescu13• 3 years ago

Great questions.

There is some heavy compute - most of the real value is in our pay calculations and compliance features which are all based on algorithms that calculate correct pay based on legal requirements.

But mostly it's just having lots of concurrent users. For a while a few droplets was plenty, and then one day we hit a tipping point where it was overwhelming.

spiffytech• 3 years ago

It seems like server performance has improved a lot over the last decade. The move from HDD -> SSD -> NVMe alone makes a huge difference for anything that touches disk. And my memory is that it was pricier to buy machines with more cores or RAM vs today.

It wouldn't surprise me if today's budget VPS does the work of several machines from 10/11 years ago, and costs less too!

nologic01• 3 years ago

Something that would be interesting to compare and contrast is actually using your own hardware to host the service(s). Servers are super-reliable these days, it doesn't feel like a huge step from the VPS approach to simply have a bunch of them.

At least one tradeoff is that you'd have to be physically in a location with adequate bandwidth / latency but for various use cases it might be very cost effective during some phase of the startup.

I wonder also how the AI induced focus on specialized hardware might change the calculus

anonyfox• 3 years ago

After 20 years of dev, having devops skills nearly maximized (with a AWS specialization), built/maintained/scaled many products, ... can completely relate. I really do see hostings costs differently now compared to earlier in my career, when I was more junior/midlevel where one still argues about "cool" tech/languages/architectures.

What you want from day one is a completely managed hosting for tech stack you work with, but are not hard tied to forever.

Right now I am starting some new/modern meteor.js stuff again, there is meteor galaxy for hosting that "just works" for that stack, you bring your own mongodb (I prefer atlas itself, they even have serverless now), and everything is taken care of, including CI/CD/Monitoring/... . Its like a few minutes of initial configuration, and never think about it again, and if done correctly you shold have horizontal autoscale of some kind automatically nowadays.

Yes, this is significantly more expensive than directly using AWS (which they use under the hood), and even though I am personally highly trained in this stuff+terraform/cdk/..., I don't want to have all this work anymore when I also can shell out a few hundred bucks per month instead. Just in case Galaxy becomes a problem for some reason, I _still_ can deploy the app stack to some VPS provider, but I'd use some already existing automation (like meteor-up in this case) instead of really digging into typical devops topics for it.

There is a bad feeling in the mouth as an engineer to shell out "more than needed" for infrastructure, but my rule of thumb now is that I am happy to eat that frog as long as the potential cost saving is less than 2 infrastructure engineer FTEs, thats my trigger to _maybe_ discussing it.

nektro• 3 years ago

great writeup and congrats on 11 years!

jwr2• 3 years ago

I read this article with interest, having run a SaaS for the last 8 years or so (solo-founder). I find it intriguing that the OP needed so much compute power. Computers are really powerful these days.

As a counterpoint of sorts, here's my "hosting journey":

* run everything on a single physical server rented at Hetzner (DE)

* [... several years pass, business grows ...]

* switch to ansible, learn it, spend a week or so to write automation for a 3-server setup, also learn terraform and write terraform configs for setting up a Digital Ocean system from scratch

* run production on a 3-server setup at Hetzner DE, run a staging system at Hetzner FI, also serving as a possible quick manual failover, for a total of 6 physical servers, test re-initializing systems from scratch regularly, test setting up a Digital Ocean system from scratch regularly

* [... several years pass ...]

* that's it — I really can't see a need for more in the near future.

But then, my software is not in Ruby on Rails, and I have no experience with that platform. I use Clojure and ClojureScript and I was careful to design everything to be rather client-heavy at the start. I also never wanted to depend on PaaS systems, mostly to avoid lock-in, but also because I don't buy the "just use our magic database offering and forget about database problems forever" selling pitch. You can sweep possible problems and complexity under the rug and hide them, but you can't run away from them. I also do not use Postgres (collective gasp in the audience), because having a single centralized point of failure is not something I want in my setup.

I also never needed significant sysadmin/devops support. Granted, I do have some experience, but these things do not require dedicated teams, unless you are YouTube. A little ansible+terraform goes a long way, so does buying an hour of two of consulting from an experienced sysadmin.

Those physical servers that I use are significantly faster than the over-subscribed cloud VM instances that you usually get from AWS and the like. And they have 64GB of RAM, not some measly amounts. If I need more servers, it takes on the order of hours to get additional ones, but I'm not sure what I'd use them for.

My total hosting bill is on the order of 350€/month and is boringly predictable.

gen2202• 3 years ago

Just curious, if you feel comfortable sharing, what do you use as your database?

I'd agree Postgres is not the right answer to every problem, but the documentation of its failure modes and the mitigations thereof make it a "good" answer to "most" problems.

I'm curious what the "problem" (in access pattern terms, doesn't have to be business terms) and "solution" (i.e. persistence technology) is in your setup!

jwr2• 3 years ago

Sure. I use RethinkDB, which works well, but has been pretty much abandoned at this point. It's a good database, but it wasn't "in fashion" like many worse solutions were.

I am working on replacing it with FoundationDB. I want to have a fully distributed database with strict serializable semantics (see https://jepsen.io/consistency), and there is very little out there that gets the job done. FoundationDB is really impressive and works really well. I'm worried that it isn't "fashionable", though.

As for access patterns, I'm not sure if I understand the question, but I'll offer one thought: if you're writing an app, you don't need a "query language'. You'll quickly learn what your queries are, and the right approach is to restructure your data to fit your access patterns. Your "queries" will be written in your programming language of choice, not in the databases "query language".

I feel that the idea of a "query language" is stuck in our heads back from the days when the boss would come and tell you to produce a custom report from the database. It's just not how app databases are used these days.

RandomWorker• 3 years ago

This story resonates with me — apparently we are on phase two. And we need dedicated team members to take care of this. Thanks for sharing your lessons this was hugely valuable. Don’t let the negative comments get you down, when building things it’s hard and it’s all a big learning game. Congratulations on building a great piece of software.

ghiculescu13• 3 years ago

Thanks! Reach out if you ever need a hand.

lakomen• 3 years ago

Those are such noob mistakes. No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond.

Side remark, that's what I was trying to tell the dgraph guy. If you have an int64 ID for ALL your transactions and all your IDs it will eventually overflow. It's a single incrementing value for ALL actions. Inset a new record, new ID, retrieve this record, new ID etc.

Everything this guy writes about are newbie mistakes, stemming from a lack of "far sight", aka thinking 5 steps ahead. Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these.

What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not. I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part.

I know this article is about Ruby. But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer, I would NOT go AWS lambda and complicate my life and, most importantly, the development process or as I call it the solution delivery process.

This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations.

There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters. The split between backend and frontend happened around 2013. Having a classic website with low performing Ruby, that also has such a bad workflow, e.g. Blog, you can't start with Comment, you have to start with Blog, then Post, then Comment. This is archaic. With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client. If I need SEO I'm back to the scaling problem, but the Go backend won't be the one that needs scaling. And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle. Doing this for the SSR JS frontend is where it gets expensive. So either try to static render with Svelte, if possible, to solve this problem and load balance or get locked into complex auto scaling cloud nonsense. And this is where it gets REALLY expensive.

Your Ruby on Rails app might be good enough for a PoC, but that's about it. If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure. And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent. You can have your cake and eat it too. Just think it through from the start to the end.

btreecat• 3 years ago

Your comment feels like one I would have written in my past, but I think there is some possible context or understanding I might be able to share based on some of my XP.

No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond.

Sane defaults are important. If the DB defaults to a particular value it's likely because it's fine for most use cases. It's an excellent thing to have enough customers and utilization to require something more than int for an ID. That said if there is little to no performance difference, I'd argue that this should be made the default upstream rather than the default recommendation or "best practice."

Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these.

Not every one is able to do that kind of systems level thinking across an entire product stack, and that's OK. What's more important is not building things you will never use, or might use one day but don't need today.

I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part.

I would spend more time thinking about how to market to folks who want to reduce their cloud investment. My current org is way over engineered both in infra and engineering in general. We have been working to reduce complexity and cost and it's had real big wins for us as an org. KIS is an excellent philosophy when applied pragmatically.

I know this article is about Ruby.

The author could have replaced ruby with "Language-X" and the article would have been just as accurate. It's often not the language that's slow, more often it's the DB schema, data models, and business logic.

But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer

If all it has to do is reply "Hello World" plenty of languages and frameworks could hit similar numbers. But if all those frameworks are blocked by a de-optimized external request, it doesn't matter. Here, the authors app was def having DB scaling issues, this could happen with any language, runtime, or framework. It will happen sooner with certain combinations than others yes, but utilization and customer feedback are more important than early optimization.

This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations.

I think that's a harsh assessment. We all buy into some hype, and we all generally try to make the best decision with the information we have at the time. Instead work backwards, put yourself in the authors shoes, assume the idea at the time made sense, and then ask "What had to be true, or seem true at the time, to reach the conclusion they did?"

There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters.

It matters, until it doesn't. https://www.tiobe.com/tiobe-index/ as an example, we see top spot held by a known "slow" language (certainly comparable to Ruby). JavaScript, PHP, and VB all beat out Go in utilization. Performance matters, is a statement that requires context. Sometime the "performance" of your developers matters more than your runtime.

With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client.

And now the current trend is to start to reduce the amount of load on the client because turns out 2Mb of JS to render an SPA is not the best XP on a lot of devices and networks. It's important to understand who your users are, and the operating context they bring.

And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle.

Horizontal scaling isn't dependent on language. You need to design your API and choose your persistent storage correctly to make this task easy. But in reality, most folks could just use caching and be fine.

Your Ruby on Rails app might be good enough for a PoC, but that's about it.

Based on what, your personal opinion? It's a perfectly fine language for a back-end service and I say that as someone who has no interest in learning Ruby.

If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure.

Maybe. We use scala at work, because "it can handle large volumes of streaming data" and yet it's some of the slowest parts of our stack because it's hard for folks to "do right." Sometimes you need a language who's feature is easy to use and hire for.

And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent.

Yes and no. K8s is basically the next generation of LAMP for the cloud. It's fine, just understand what you are investing in and what it isn't. We use aws at work w/o k8s and it's a complex nightmare to manage in some respects. My k8s cluster at home however is fantastic and fun to tinker on, and causes me less stress with more uptime than my AWS clusters at work. I think it's more important to understand the problem space, and how a tool might fit in or not, than to hope it just solves problems magically.

The cloud isn't magic, it's just someone else's computers.

adamckay• 3 years ago

What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not

Perhaps its more to do with your attitude?

This comment comes across as very condescending and needlessly aggressive to someone admitting their mistakes made whilst growing their successful business and graciously sharing them so others can learn or commiserate on similar mistakes.

nathants• 3 years ago

lambda is great for three things:

- scaling to zero for low use services

- reliably managing other infrastructure, like pools of ec2 or ephemeral ec2 spot

- getting 1000 cpu cores for 5 seconds to do latency sensitive data heavy lifting

the elephant in the room with cloud is always egress bandwidth billing. egress heavy apps just shouldn’t live on aws.

for me, the fun with aws is not figuring out how to make cloud lasagna and then write thought leader blogs about how tasty it is, but in understanding which parts of aws are actually better than alternatives, and how to compose them into good systems.

s3, dynamo, ec2, ec2 spot, lambda, r53. egress bandwidth aside, these things are for great good.

add in cf workers+r2 for egress bandwidth heavy components.

oaiey• 3 years ago

At our first meeting our [AWS] account manager brought along his solutions architect. I had never met a solutions architect, so I didn’t really know what they did.
Eventually I realised that I wasn’t the problem.

I had the same experience. These free "Solution Architects" are a disgrace to the architecture profession. They are sales person and are paid/rated/motivated to sell you up. Not what is best for you. Ours went over the in-house architect with zero feedback and told our CTO/CIO how shitty the architecture is. Fun. Stay away from them.

123pie123• 3 years ago

If something is free etc...

I've always taken the advice of any prepaid 'experts' with a huge grain of salt, not just AWS

pdimitar• 3 years ago

It really makes me super bitter how people like the author of the article who basically fumble their way through stuff are still successful, even though they seem to have done a lot of stuff the wrong way.

Calling out D.O. for a fairly transparent process is very wrong. They did what they had to and OP was oblivious to a fairly standard process of "hey, your CC expired, let's do something about it before we turn off your servers" which OP seems to have ignored several times in a row.

All of the story felt like he was just stumbling from one hype to another, and from one easy to avoid mistake to another. Even when I had only 7 years of experience in total (now I have 21.5) I was paranoid enough to look for DB ID turn-over and absolutely would have made sure to upgrade the DB server when I start noticing it hits 80% load consistently.

But that's life for you. A technically excellent guy still lives paycheck to paycheck, meanwhile an absent-minded guy who is easily hyped has a successful business. [sighs deeply] ...Moving on.

Finally, choosing AWS but still only using EC2 is kinda non-intuitive for me; why do that at all? Maybe because they allow transparent upgrading that makes for less sysadmin work? If so then fair, but that's still like buying a Ferrari to drive on long empty roads but never go above 90 km/h.

amake• 3 years ago

The team had several people in every time zone

Kind of hard to believe

ghiculescu13• 3 years ago

Crap sentence, but people in Australia, US, and UK can cover every time zone.

11 years of hosting a SaaS

Related Stories