Reliability: It’s not great

https://community.fly.io/t/reliability-its-not-great/11253

by bishopsmother • 3 years ago

1226 453 3 years ago

community.fly.io

emschwartz• 3 years ago

One of my colleagues keeps repeating “reliability is our number one feature”.

I’m not sure it is for 100% of early stage startups, but I guess it is once you exceed some minimum usage threshold.

That said, definitely appreciate the detailed explanation.

jpdb• 3 years ago

One of my colleagues keeps repeating “reliability is our number one feature”.

I think reliability is the #1 feature at any stage because if you're unavailable, you're at best useless and more than likely you are actively harmful because your users have an expectation.

However, if you're unavailable outside of times customers don't expect you to be there then you're not actually unavailable. This is more likely for an early stage start-up, but you don't typically choose or know when you're expected to be available nor do you always get to choose when you're unavailable.

sa46• 3 years ago

In terms of confidentiality, availability, and integrity: I'll bet LastPass would gladly trade availability right now to regain confidentiality.

ignoramous8• 3 years ago

Our team at AWS had a poster up on the wall that more or less went:

1. Security

2. Durability

3. Availability

4. Speed

jrochkind16• 3 years ago

One of my colleagues keeps repeating “reliability is our number one feature”.
I’m not sure it is for 100% of early stage startups,

I mean, it probably depends on the nature of the startup? Platform-as-service seems particularly sensitive to reliability (whether or not it's "#1 feature"), in a way that might not be true of startups in other spaces.

cschep• 3 years ago

TL;DR -- It's very domain specific if reliability is your number one feature.

For a startup that is hosting other people's production application/data then this is absolutely true. Less than 100% always needs to be addressed.

For a startup that is selling bingo cards then reliability probably isn't nearly as important. I'm guessing there were certain holidays that were more important than others as far as reliability goes though? Maybe patio11 can chime in :)

nprateem3• 3 years ago

That's what crossing the chasm is about. Before the chasm are tinkerers who don't mind things failing as much, but after it, people want reliable things that Just Work.

soperj2• 3 years ago

For django, they should really contribute to 2 scoops django cookie cutter program, so that you can get an out of the box django instance that can just deploy to Fly.io.

mixmastamyk5• 3 years ago

May be wrong, but they seem to be very focused on interesting stacks like Elixir and RoR while building on Go/Rust. The corollary being neglect of the bread/butter stacks with high market share, like Python and Java/JavaScript. Don't think I've seen a blog post discussing those three beyond a passing mention?

Not the end of the world, but mildly disappointing. At least they are all in with Postgres and Linux, a great foundation.

jjtheblunt• 3 years ago

I thought the only thing their tech stack was all in on was docker images being rewritten to run on firecracker as a substrate.

is it not agnostic about things like Elixir etc, at the tech level, though they've got super nice documentation for those tools you mentioned?

ilaksh3• 3 years ago

You can use any Docker image.

tomwojcik2• 3 years ago

I don't disagree with you, but they tweeted

We’ll readily admit our docs still have a Django-shaped hole in them.

https://twitter.com/flydotio/status/1578039196618575874?t=nu...

nilsbunger• 3 years ago

Their problem right now isn't adding more customers - they seem to have more than they can handle!

If I were them I'd focus as many resources as possible on making the stack rock-solid, and away from acquiring more customers or adding more capabilities.

In fact I'd try to down-scope some features if at all possible, like the example they give of disabling app deploys while they're doing platform updates.

We use fly.io at a small scale and it's worked really well for us, but the money is in customers at a larger scale who must have 100% reliability.

llimllib• 3 years ago

https://github.com/ehmatthes/django-simple-deploy will deploy to fly (or platform.sh) out of the box, should be pretty much the experience you're describing

soperj2• 3 years ago

thank you!

tomwojcik2• 3 years ago

I needed a cookie cutter for my side projects so I've created one for sqlite that I actively use and a similar one for postgres. Both are very basic, contributions welcome.

Sqlite https://github.com/tomwojcik/django-fly-sqlite-template

Psql https://github.com/tomwojcik/django-fly-postgres-template

yamrzou• 3 years ago

I'm not a user of Fly.io. I can't help but notice how remarkable the effect of open communication on potential end users like me. I remember reading about their reliability problems on HN some time ago. That biased my view of the company. After reading this, the open communication and transparency restored my trust in them, and would make them again a potential candidate for future projects. Because now I know that they acknowledge the problem and that they are trying to improve things.

leetrout• 3 years ago

mrkurt[1] is also active here and has been very transparent in his comments about scaling issues.

Similar to this post he commented a week ago:

In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former.

I am rooting for Fly! Great team. The company reminds me of early HashiCorp.

[1] https://news.ycombinator.com/user?id=mrkurt

willio58• 3 years ago

Agreed, this is how company communication should be.

I don't use Fly but would consider them in the future even given their recent issues.

I look at this in contrast to Twitter who had/has? an outage today. Their leadership is opaque and doesn't take responsibility for the issues they are causing.

alfalfasprout2• 3 years ago

In fairness, a CEO who has basically been Kanye-ing himself and his company into irrelevance is a low bar.

mbesto• 3 years ago

Me too.

However, this is a double edge sword. Their key value proposition is scale / speed which makes it concerning that they haven't "solved" that yet.

gizmo5• 3 years ago

This post is carefully worded corporate messaging, but because they write for their developer audience it has an informal "oh shucks we messed up bad y'all" vibe to it. But make no mistake, this is 100% corporate messaging.

I get that growing is super hard. And maybe fly will grow up to be a good platform some day. But that's the future. Today, they're flying by the seat of their pants and I mostly feel sorry for people who were tricked into thinking this platform is ready for production use.

spoiler2• 3 years ago

I'm not sure why the cynicism around their candor. Do you think it's not genuine just because it was posted by a company employee?

Your post implies corporate messaging is bad. And anything posted by a company—or at least I don't know where you draw the line—can be considered corporate messaging. Am I just reading too much into your phrasing?

skrtskrt• 3 years ago

professionals don't get tricked into thinking a platform is ready for production use

If you don't have SLOs and SLAs, then you get what you get, essentially. Even a company with a great reputation can completely reverse course with a single bad incident, and you get nothing in return if there's not a contract.

Trufa• 3 years ago

I don’t get what you’re saying, this isn’t a brag disguised as a confession, they are actually admitting to poor performance, of course it’s to eventually make users trust them, but a) I don’t see nothing bad with that b) they are choosing the hard route.

They are being open and transparent (afaik) even if carefully worded, which I also don’t blame them for.

snapetom6• 3 years ago

This is probably therapy, but your message and fly.io's post resonates a lot with what I'm going through. I took a product owner role about 6 months ago, my first, with a company that has turned out to be just a mired mess, and a product universally hated both internally and externally.

Long story short, it's completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight. It ended up not delivering on any promises it made, and there were a lot of them.

I was warned left and right before presentations and meetings, "this customer hates your product because of ...." I started off every meeting with saying, "we're rearchitecting the product, this is how we're doing it, this is the tech we are using." Immediately there was a sense of relief from customers, followed by questions like, "why can't <current product> deliver <feature> that was promised?" I'm completely honest with bad decisions that were made and how it impacted the feature. Sure, there is skepticism on what we are doing, and I tell them they should absolutely be skeptical based on our track record. The result has been customers who have hated my product now offering to work with us on development.

I've also been completely forthcoming on configuration, security, resources, and setup issues I am finding, many of them are absolutely freakin' insane. I've flat out told customers it's frankly embarrassing and never let us do something like this in the future. The best feedback on this was, "At least you're telling us something. We usually get silence from this team."

God, this is the most depressing job ever.

hinkley3• 3 years ago

Architectural astronauts.

mrkurt6• 3 years ago

I feel this. I hope you get over the hump and your job gets fun. We've had flashes, at least, but I do think what we're doing (and probably what you're doing) require some irrational behavior.

zamnos6• 3 years ago

Can you help me in a detailed sense - what did you tell customers? did you literally say there's product is "completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight"? That seems a little over-honest to me but of course I wasn't there.

ndneighbor• 3 years ago

Part of what I hated about Product Management at my last role was the consistent helplessness I felt when I was on calls with our customers. I could tell our product wasn't meeting their needs but all I could do was try my best to give the engineers context on how best to eventually meet them.

I remember my first few days on the job just being ripped to shreds by our customers who (understandably) were slighted. Don't miss those days at all.

claytonjy4• 3 years ago

Your job sure does sound depressing, and it's not one I would succeed at, but if you can power through and turn this product around that's a hell of an accomplishment you'll have to be proud of.

I'm curious what you'd like to do next. You could probably have a great career doing these sorts of turnarounds repeatedly across companies, maybe even as a consultant, but would you want to?

eyelidlessness2• 3 years ago

When I was in an engineering IC version of this role, I longed for someone like you to have my back in management and have customers’ backs too. If I had a time machine and a magic lamp I’d team up with you.

As a future representation of past me, I can tell you:

1. Everything it’s making you feel is valid.

1b. If you’re feeling burnt out, please listen to it. It gets worse if you let it.

2. While I can’t hire you now, I can already tell you’re eminently hireable. If you have any cautious inclination to move, you will probably be better served by greener pastures.

3. Just take care of yourself.

4. When 3 contradicts 2, favor 3.

joshxyz• 3 years ago

it is.

i do enjoy programming, working with other devs, but as soon as i stepped into a product management role it's a hellishly different set of problems. you're in the middle of the tech, the developers, the problems, and the customers. lots of lessons in there. tiring, but worthwhile.

anthlax• 3 years ago

Could you further elaborate on “intellectual engineers”? What mistakes were made? Esoteric languages? Obfuscating design patterns?

Partially so I could learn from mistakes and partially since I’m a sucker for post-mortems :)

alfalfasprout2• 3 years ago

This is huge. Even as a member of a larger company, this stuff matters. If you have a vendor that doesn't bullshit you when things go wrong, you can actually trust. This is how you avoid companies having the "hmmm they seem to be having lots of issues recently, let's consider moving off them" conversation.

iLoveOncall2• 3 years ago

This is how you avoid companies having the "hmmm they seem to be having lots of issues recently, let's consider moving off them" conversation.

Yeah, this is how you get companies to have the "well, it SEEMED they were having lots of issues, now it's clear that they indeed did, moving off of them is priority #1".

bodecker• 3 years ago

Open communication is great when there are incidents, but even better is having no incidents. (of course there are nuances depending on specific context)

eyelidlessness2• 3 years ago

I’m not [directly, at present] in the market for their product but the very frank and real introduction basically moved Fly from my “I think I’ve heard of that but I’ll need my memory jogged for any further recognition” category to my “potentially good stuff to reference when relevant” category.

This kind of frank and human communication is vulnerable, but it’s good for establishing credibility… with me at least!

iLoveOncall2• 3 years ago

Really? For me it's the contrary, posts like this from companies are a signal to say "Jump out of the ship while you still can".

Kinda like when crypto exchanges tweet "Yo we're definitely not blocking withdrawals, we're perfectly healthy".

You know what would make me consider a company? The fact that they don't have a bad reputation to begin with, and don't need to make posts like that to try to save their reputation.

tiffanyh• 3 years ago

Would the simple solve be that Fly.io just mark any new service of theirs “beta” for x-months post launch?

samwillis2• 3 years ago

None of the services that they have had issues with in the recent past are new, they have been running for at least a couple of years. They would need to put a "beta" sticker on the whole platform for that suggestion to work.

But the post makes it clear that the issue isn't that they had problems with new services. It was rapid customer growth before they had time to scale up the infrastructure as they had planned to do.

sergiotapia• 3 years ago

It's been almost a year since I gave Fly a review (https://news.ycombinator.com/item?id=31391116) and it's a bummer that they're still struggling to get things right. Double bummer because I love Phoenix and Elixir and they employ Chris McCord there.

Maybe they were _too_ ambitious at the start? They have a hard road ahead of them, and competition like Render.com and Northflank have provided me with solutions to all of my problems. Great dev ux, great prices and predictable solutions. They also keep pushing out very useful features. A third competitor also sprung up Railway! There's certainly blood in the water.

Will they catch up to others before the competition solves the "global mesh" unique value proposition Fly.io currently has? That's the $1MM question.

zomglings3• 3 years ago

I read your review, and had a question so I thought I'd follow up here. You mentioned render.com as a competitor - does render host its own infrastructure or do they act as a go-between between their users and AWS/GCP/whatever?

cldellow2• 3 years ago

They act as a go-between in that they ultimately host on AWS/GCP. They host their own infrastructure in that they appear to run Kubernetes and have built out their own deployment and service fabric, so they're just using the underlying machines as dumb compute, they're not, eg, building on RDS.

In March 2021, someone asked a question about carbon emissions of their data centres. They said they hosted on both GCP and AWS, but mentioned they were interested in moving to their own bare metal [1].

In April 2021, I asked a question about egress fees to Google, and they walked back a bit the comment about moving to bare metal [2].

As of March 2022, they're still in AWS/GCP [3].

As of September 2022, workloads for new users deploy into AWS, even in regions that were previously served by GCP [4].

[1] https://community.render.com/t/does-render-use-green-energy/...

[2] https://community.render.com/t/is-render-com-hosted-in-googl...

[3] https://community.render.com/t/are-your-servers-owned-by-you...

[4] https://community.render.com/t/which-render-regions-map-to-w...

iamdbtoo• 3 years ago

I'm a big fan of fly.io. From their hiring process to the product itself it's all carried out in a thoughtful manner. I hope they can weather this rough time.

claytonjy4• 3 years ago

Very interesting to see Kurt assert theyre going to "solve managed Postgres", and I'm super curious to know what that means. Does it mean something like RDS, or more like CrunchyData?

I could see them building something RDS-like on their own, but if they're trying to go further than that I wonder if they'll buy or partner with other companies rather than doing it themselves. Neon strikes me as a Postgres-as-a-service that could pair well with Fly.

mmcclure2• 3 years ago

That comment jumped out to me too, my recollection was that they've been pretty vocal about that not being something they wanted to solve themselves as a core competency. I'm not quite parsing if these two blurbs should still be taken together or if the second sentence is refuting the first.

The second problem we have with Postgres was a poor choice on my part. We decided to ship “unmanaged Postgres” to buy ourselves time to wait for the right managed Postgres provider to show up.
We’re going to solve managed Postgres. It’s going to take a while to get there, but it’s a core component of the infrastructure stack and we can’t afford to pretend otherwise.

+1 to Neon seeming like a good fit, but it's also very much a beta (alpha?) both as a product and company (at least from my impression). I'm not sure that's a bet they'd want to make right now given the context of this post.

nikita4• 3 years ago

(Neon CEO)

We are launching our paid tier March 15th and will be production ready shortly after. We are running 20K+ databases and measuring reliability and uptime.

Generally reliability is a function of architecture (we are solid there), good SRE practices, and a long tail of event you live through, fix, and make sure they never happen again. The bigger the fleet the faster the learning.

craigkerstiens• 3 years ago

Craig here from Crunchy Data. Not sure if you mean Crunchy Data is like RDS or isn't, in some cases we're very similar as a managed service provider. But are focused on a better developer experience and quality support.

We've had a number of customers that use us for the database and fly for the app. We had a user benchmark a number of heroku alternatives with various database providers and we were actually better response time than the unmanaged instances on fly themselves in addition to all other providers they tested - https://webstack.dancroak.com/

I won't speak for Fly, but we're big fans of them and think we pair quite well together.

mrkurt6• 3 years ago

Yes we think they pair well together too. I believe the ball is in your court though. ;)

claytonjy4• 3 years ago

I haven't used CrunchyData for work, but I see you as offering what RDS does plus plenty more. RDS does a lot, but after using Timescale Cloud professionally I saw how much RDS doesn't do, like actually-simple upgrades, one-click forks, etc. and Crunchy looks similar in going beyond RDS.

I think the community would really love to see a direct Fly+Crunchy integration!

jrochkind16• 3 years ago

I don't even understand what you mean as the difference between "something like RDS" and "something like CrunchyData" -- they seem like similar products to me?

claytonjy4• 3 years ago

I see RDS as the absolute bare minimum for a managed database; providers like Timescale or Crunchy tend to add some pretty useful stuff on top.

aeyes2• 3 years ago

If I was in their shoes I'd probably aim for a "serverless" Postgres experience where you get a connection string and you know nothing else.

I think RDS, Crunchy, Aiven and others aren't quite there yet.

chime• 3 years ago

They kind of offer that with their Redis (via Upstash). But for our use-case, we needed it to be managed PG and Redis. Going out of the LAN introduces too much latency.

nu11ptr• 3 years ago

Not a client of fly.io, but dang impressive for the company to be this open and honest. Definite respect - wish more companies were like this. It puts them on my short list almost immediately for future needs.

nathell• 3 years ago

Came here to say pretty much this. Technical issues come and go; open communication is a core part of the company culture, and builds up trust.

nprateem3• 3 years ago

Why would you shortlist a company that admits to reliability issues?

macintux• 3 years ago

All companies have reliability issues, especially in this problem space. I’d rather go with someone who acknowledges their failings in public.

jen202• 3 years ago

Because if they admit to them they’re probably in the process of fixing them. When compared to a certain B tier cloud who gaslight users about flaws in the product and API, the amount of effort expended to get to an admission of fault is much lower for companies not inclined to lie by omission.

samwillis2• 3 years ago

Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want.

Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it.

However, at the same time the market is looking for a successor to Heroku. A zero dev ops PAAS with instant deployment, dirt simple managed Postgres, generous free level of service, lower cost as you scale, and a few regions around the world. That isn't what Fly set out to do... exactly, but is sort of the market they find themselves in when Heroku then basically told its low value customers to go away.

It's that slight miss alignment of strategy and market fit that results in maybe decisions being made that benefit the original vision, but not necessarily the immediate influx of customers.

I don't envy the stress the Fly team are under, but what an exciting set of problems they are trying to solve, I do envy that!

vineyardmike3• 3 years ago

I agree - fly is so easy to use (when it works) that it’s hard not to be impressed. BUT what I’ve found is that we don’t need edge compute, since our customers aren’t that latency sensitive, so it’s lost on us. It’s only a few more milliseconds to us-east-1.

I’ve heard (on HN) of a dozen different companies vying for the heroku replacement spots and yet Fly seemed to capture the attention. I couldn’t name another one off hand.

What I truly want and probably lots of other people too is Flyctl (and workflow) for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.

latchkey6• 3 years ago

What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dalles.

Google Cloud. It is painfully easy to spin up managed postgres, super easy to deploy gcp cloud functions or gcp cloud run. It isn't expensive either and just works.

cldellow2• 3 years ago

Render.com is another spiritual successor of Heroku. I'd love a world where Fly and Render are both very successful companies.

manmal2• 3 years ago

I can second that I‘ve seen render.com mentioned very often, maybe even more so than fly.

gen220• 3 years ago

What I truly want and probably lots of other people too is Flyctl for AWS. The same simplicity to run as fly, but give me something cheap in Virginia or the Dallas.

Pardon the ignorance, is this not the Amplify CLI [1] ?

[1] https://docs.amplify.aws/cli/

trunnell• 3 years ago

  > Flyctl for AWS

Have you tried AWS Copilot? I’m having good success with it. Probably not quite as simple as flyctl, but still it’s only one command to deploy a container.

I would really like fly.io to overcome these hurdles. I bet they will.

zoomzoom• 3 years ago

I think this whole category is interesting, from the next-gen PaaS to the cloud-native ecosystem. Totally empathize with how hard what fly is doing in terms of scale and reliability is.

At Coherence (withcoherence.com) we're focused on a developer experience layer on top of AWS/GCP. You might describe it as flyctl for AWS.

trilobyte3• 3 years ago

generous free level of service,

This is likely the biggest culprit for a lot of these companies. Too many of us have grown up in the culture of getting hosting and platform for "free", but at some point the companies providing it still have to pay the bills. There has to be a better pricing model that let's someone deploy their relatively small, low-traffic app for $10s/month or even $200 - $300 / year for the basics (e.g. - Heroku free tier type capabilities). It's not going to save these companies but it would limit excessive growth of their own costs from a free tier while at the same time still being affordable for 1 - 2 person teams who are trying to get something in front of users.

karmelapple3• 3 years ago

I agree. And I know this is unpopular, but I think none of these companies should be expected to have a free tier. A low-cost tier? Certainly. Perhaps even a free trial with a credit card? Great.

But our team, who has used Heroku for over a decade, got bit multiple times by Heroku having a free tier.

Why were we impacted by other apps? Because Heroku’s load balancers are shared amongst all their apps. That includes all the sketchy apps running on the platform.

If Heroku could somehow isolate us from everyone else? Great - and they offered that for awhile with a reasonably-priced Add-On supported by them called SSL Endpoint. It cost about $15/month and put us into a pool that was shared with other folks willing to spend that much per month to run their app.

I understand that’s not great for a hobby project. But for those of us trying to run a large product on Heroku and not have to spend multiple extra thousands of dollars every month for a Heroku Private Space, this was a great way of pooling: put a small fee in place for one pool of resources. Not many malware writers or other misbehaving app creators will probably want to spend that much per month.

But they axed that a few years ago. Only a couple months after when we were thrown back into the load balancer pool with all the other free apps, one of the IPs was marked as spam and we had to figure out a kind of janky solution.

Additionally, Heroku seemingly spent a ton of resources on free tier support, malware fighting, etc. I hope to see more features on Heroku since they’ve dropped that support… but I haven’t seen much evidence of that in roughly six months since they did that. But we’ll see.

mike_hearn• 3 years ago

If you don't need all that much, Oracle Cloud does offer a free tier for VMs. You get 2 AMD VMs or 4 ARM VMs and even a free Oracle DB, object storage, load balancing and monitoring.

https://www.oracle.com/cloud/free/

It's still just a free tier so you can't expect good support, but, it's there.

dcow• 3 years ago

Check out DO app platform. It’s literally exactly what you describe.

mike_d7• 3 years ago

All of the forum posts and howtos on spinning up an AdGuard or similar service on Fly.io instead of a local Raspberry Pi probably didn't help things. While it does drive user growth, it isn't the type of users you want.

One of the hardest lessons every business needs to learn is how to say no to the users they don't need.

mattbillenstein3• 3 years ago

Yeah, distributed systems at the global scale are very very difficult - at least with the Heroku style problem, you'd be looking at scaling in a single datacenter I think - deployments to multiple datacenters wouldn't share dependencies.

I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives.

Simplicity is the key to reliability, but this isn't a simple product, so idk.

lmm• 3 years ago

I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives.

In my experience people who ran Postgres distributed across a WAN tended to use obscure third-party plugins at best, more often a pile of dodgy Perl scripts. Using something designed from the ground up to be clustered seems to have a much better chance of working out than trying to make something that's been built as a single-instance system for decades work across the internet.

zamnos6• 3 years ago

Backend simplicity also means a more shallow moat. It makes it easier for Digital Ocean/Linode (Akamai)/Hetzner to offer a competing service with the same backend knobs to turn, should they decide they want to get into that market.

The goal should be to make the backend as simple as possible, but no simplier. Complexity here leads to operational burden and toil. But that's why you hire good SREs and treat them well. What's more important is frontend complexity, aka how difficult it is for customers to use. Backend and frontend complexity aren't necessarily linked, which, imo, fly.io achieves, downtime aside.

ec1096853• 3 years ago

The CloudFlare folks wrote a good blog post on how they are seeing their customers use Edge compute — latency is far down on the list: https://blog.cloudflare.com/cloudflare-workers-serverless-we...

everybodyknows• 3 years ago

Hmm, that post is almost three years old -- still accurate?

fmajid4• 3 years ago

The US CLOUD Act means a EU customer cannot use a US cloud provider to host PII, even if the server itself is physically in the EU, because US law will still compel the provider to yield the data to US authorities. The European Commission is trying to paper over the cracks with a fig leaf of judicial review, but it's only a matter of time until a Schrems III decision from the CJEU invalidates that polite fiction.

vendiddy3• 3 years ago

I can second this. We were evaluating moving off Heroku and to Fly.io, but we didn't need all of the edge compute stuff. We just want a better Heroku without having to think about infrastructure and having to think about edge compute just got in our way.

I feel like Next.js is in a similar position. While their main vision is SSR, I wonder if they are missing out on a chunk of the market that simply doesn't want to think about infra. We use them because we just don't have to worry about webpack or fiddling with deployment and hosting. We could care less about SSR and in fact we disabled it app-wide.

alexgrover3• 3 years ago

Why would they be missing out? Vercel can host static sites just fine, whether that’s one generated by Next or any other framework or written by hand

leerob• 3 years ago

One of the key design choices of Next.js was to enable granularity on the runtime (Node.js or Edge[1]) and the rendering method (static or dynamic[2]) on a per-route basis. If you want a full SSR site, that's okay. If you want a full static site, that's also okay.

We often see folks wanting a mix of both. For example, maybe the /about page is static, but the home page is dynamic and personalized based on the visitor. You can do all of this with Next.js. Our future direction is adding even further granularity, enabling this decision at the data fetch level, allowing you to cache results across deployments[3].

[1] https://beta.nextjs.org/docs/rendering/edge-and-nodejs-runti...

[2] https://beta.nextjs.org/docs/rendering/static-and-dynamic-re...

[3] https://vercel.com/blog/vercel-cache-api-nextjs-cache

bostik2• 3 years ago

There's a wonderfully blunt saying that applies here (too): you are not in the business you think you are, you are in the business your customers think you are.

If you offer data volumes, the low water mark is how EBS behaves. If you offer a really simple way to spin up Postgres databases, you are implicitly promising a fully managed experience.

And $deity forbid, if you want global CRUD with read-your-own-writes semantics, the yardstick people measure you against is Google's Spanner.

quickthrower23• 3 years ago

In a nutshell if you offer cloud services you need to be better than the MAG clan, Digital Ocean too. And people will want it dirt cheap. It’s still hard to be a profitable web host as it always was (MAG has the advantage that none of them were web hosts at first base)

zamnos6• 3 years ago

Where does the misalignment between what the customer thinks they want, and what they actually want fit in to your philosophy? Google Spanner is a great example of this because who doesn't want instantaneous global writes? It's just that, y'know, there's a ton of businesses, especially smaller ones, that don't actually need that. The smarter customers realize this themselves, and can judge the premium they'd pay for Spanner over something far less complex. What I'm getting to is that sales is a critical company function to bridge the gap between what customers want, and what customers actually need, and for you to make money.

The first releases of EBS weren't very good and took a while to get to where we are. Some places still avoid using EBS due to bad experience back in 2011 when it was first released.

chrisshroba• 3 years ago

if you want global CRUD with read-your-own-writes semantics, the yardstick people measure you against is Google's Spanner.

I’m trying to build more of an intuition around distributed systems. I’ve read DDIA and worked professionally on some very large systems, but I’m wondering what resources are good for getting more on the pulse of what the latest best practices and cutting edge technologies are. Your comment sounds like you have that context so any advice for folks like me?

richardhod• 3 years ago

What are the limitations to heroku that people are going to Fly for? Maybe there's a standard article that would be useful to read about it?

zamnos6• 3 years ago

It's more about Heroku dropping free and low-cost plans, which is them demonstrating that they don't currently care about three low end of the market, more than any specfic feature.

mcintyre1994• 3 years ago

This is an old doc from fly so I’m not sure how much of it is still accurate, but it talks about some of the stuff Heroku didn’t have that they have: https://fly.io/docs/app-guides/speed-up-a-heroku-app/

There's no support for a single dedicated IP address for your application. With Heroku, your application's CPU resources are mostly located in one datacenter. Heroku doesn't support HTTP2 or Brotli compression and it doesn't do Edge TLS termination. And it doesn't run your applications on dedicated MicroVMs. These are all things that Fly's Global Application Platform does.

The other comment that mentions Heroku dropping low cost plans is the reason for the explosion in growth as I understand it though.

simonw2• 3 years ago

People don't trust it any more, because Salesforce have been under-investing in it for years and recently rug-pulled on everyone who had ever used the free tier (after providing that free tier for 15 years already - long enough for people to reasonably expect it to continue).

VWWHFSfQ• 3 years ago

dirt simple managed Postgres

Heroku PostgreSQL is very simple, yes. But once you need non-trivial scale it's expensive and extremely non-performant. Even a medium-sized RDS will outperform Heroku's most expensive database offering by 20x in my experience. My company doesn't even run PG on Heroku anymore. We have a VPC/Private Space connection to AWS Aurora because the cost/performance difference is so extreme.

snacktaster• 3 years ago

I don't know the details of how Heroku implements their hosted postgres service, but I'm _guessing_ that it's just a bunch of PG servers running on EC2 instances. There's probably a lot of CPU stealing "noisy neighbors" going on. But yeah, I've also experienced Heroku's PG databases being dog-slow compared to RDS for the same workloads.

karmelapple3• 3 years ago

I have run the experiment, and Crunchy Data’s Postgres servers are 4X more bang for the buck than Heroku’s.

I let some folks at Heroku know this who are product managers, and they are investigating it… but I would be shocked if Heroku gets a big performance improvement anytime in, say, 2023.

20X seems like a lot for RDS, though I’d be curious to learn more! We are switching to Crunchy because of that clear cost/performance difference you mention.

leishman• 3 years ago

This is spot on. I found myself using Fly for a project because it was super easy, not because I needed edge compute. TBH it's still actually unclear to me who needs edge compute? What apps require this sort of infra? It's not 99% of web apps right?

davnicwil• 3 years ago

Personally I see this as a 'why not, if it works' type thing.

Sure you don't need it for 99% of usecases, but if it just works using familiar architectures then it is also strictly better for 99% of usecases so you might as well, and people will naturally want it.

That 'familiar architectures' part is the hard bit, though.

hinkley3• 3 years ago

I still think that in the next pendulum swing we'll end up with edge computing and (smaller) self-hosted backends. Everything old is new again, and we haven't entirely recreated Akamai from first principles yet.

vineyardmike3• 3 years ago

One of the big benefits of edge compute is that it’s geographically distributed. Doesn’t make a big impact across the US, but globally a lot of nations have specific data laws, so it’s important to host data in the required nation. Keep customer data in its nation of origin, but have a single control plane and platform for ever data center.

quickthrower23• 3 years ago

It is going to be apps that provide rich experiences that need to do a lot of server communication to deliver them. I am thinking of things like collaborative whiteboards, for example. If 2 people are in Europe, working on the same whiteboard, then it should be low latency. The edge nodes will be near each other (or next to each other).

satvikpendem5• 3 years ago

I'm going to plug Coolify, an open source Heroku alternative (with Docker support too) that I'm using on a cheap $5 Hetzner server which is a lot cheaper than the equivalent Fly or Render etc service, and it really doesn't have much upkeep from me even if you add in the time setting up the server initially, which is like an hour, and afterwards, it Just Works™.

https://coolify.io

notpushkin12• 3 years ago

Dokku is also nice and battle-tested: https://dokku.com/

And may I also plug Lunni, a self-hosted Docker Swarm-based PaaS I'm working on right now: https://lunni.dev/

Both work pretty well on $5 servers.

arjvik2• 3 years ago

No experience with either, but how does Coolify compare to Dokku, the OSS Heroku alternative I've been hearing about until now?

erebe__• 3 years ago

You can take a look at www.qovery.com It provides an Heroku like experience but runs on your cloud account (aws, scaleway or digital ocean).

They build on existing tech that is already working, so it is more stable.

ignoramous8• 3 years ago

Save for a few "in-preview" features, Fly was stable too but then they started growing faster than they could keep up (a good problem to have!). Stability isn't a permanent state.

ghiculescu• 3 years ago

I coincidentally tweeted the exact same thing earlier today.

I selfishly hope Fly put all their focus toward becoming Heroku 2.0. I’m sure some people care about all the edge latency stuff but I don’t know many of them.

machiaweliczny2• 3 years ago

Yeah, I guess devs first will need to learn better SQL before it will start to matter.

hiAndrewQuinn• 3 years ago

God, I'm glad someone else sees it as clearly as I do. I learned about Fly from them acquiring freaking Litestream! The SQLite replicator! The canonical database at the edge of the network! Of course that's what they want to do.

decidertm• 3 years ago

I'm a co-founder at Northflank. This is what we've spent 3+ years building. https://northflank.com.

I am sympathetic with much of Kurt's post. We spent a long time building solutions to several of the areas highlighted (managed PG, persistent volumes, secret management and service discovery).

Making radical changes to architecture on a live cloud platform is always a challenge.

On the front-end Northflank is a next-gen PaaS built for high DX, speed, and powerful capability (real-time UI, API, CLI, GitOps, IaC).

Our backend is built using Kubernetes as an OS: providing a huge amount of flexibility on service discovery, load-balancing, persistence/volumes and scale.

The benefit of using Kubernetes is a universal API across all major cloud providers. We can scale clusters and regions across EKS, GKE and AKS in seconds, either in our managed PaaS or inside our customer's own cloud account.

Our managed dataservices: MySQL, Postgres, Redis, Mongo, Minio are all built using Kubernetes Operators with a small but mighty team.

From a generous free tier to autoscaling to managed postgres and other advanced PaaS/DevOps automation workflows Northflank offers something unique.

mrkurt6• 3 years ago

This is, indeed, the exciting part. As Heroku fans, we never really felt like it needed a replacement. And if it did, it seemed like Render was the natural Heroku v.next.

One thing we've noticed, though, is that people do actually want Heroku but close to users. It's not exactly edge compute. In some cases, it's "Heroku in Tokyo". In others it's "Heroku, but running in all the english speaking regions".

I think the thing that ate up most of our energy is also the thing that might actually make this business work. We built on top of our own hardware. That's the thing that made it difficult to build managed Postgres. We put way more energy into the underlying infrastructure than most nü-Heroku companies. The cost was extreme, but I'm like 63% sure this was the right choice.

KRAKRISMOTT• 3 years ago

Just partner with Neon or other similar companies in this space. Scale-to-zero distributed databases is well understood technology.

https://neon.tech/

jrochkind16• 3 years ago

As Heroku fans, we never really felt like it needed a replacement.

If Salesforce kept investing in heroku, it might not. But there is a huge loss of confidence in heroku's future going on among heroku's customers right now, which is part of what you're seeing, as I'm sure you know. (Also I think to some extent you are being political/kind towards heroku... if heroku's owners were still investing in heroku for real, adding 'edge' functionality like fly.io is focusing on is what one would probably expect...)

And frankly... your tool seems more mature and... not to be rude to competitors but seems to have more of that certain `je ne sais quoi` of Developer Experience Happiness that heroku _used_ to have and other potential heroku competitors don't really quite seem to have yet. Does what you expect/need in a polished and consistent way.

I think work you put into the underlying infrastructure definitely shows there, and was the right choice. Tidy infrastructure helps with tidy consistent developer experience.

So I understand why people are looking to you as a heroku replacement. I am too! (And I don't really need the edge compute stuff; although I could potentially see using it in the future, and it shows you folks are on top of things).

And while I kept reading fly staff saying on HN comments that you didn't want to be a heroku replacement, so were unconcerned with the few places people were mentioning where you still felt short of it -- when I saw your investment in Rails documentation and tools (and contribs back to Rails), I thought, aha, i think they've realized this is a market looking for them, which they are only a couple steps from and it would make sense to meet.

When you mention in OP a "heroku exodus" to you... I'm curious if that was all people who left when heroku ended free tier stuff, and they've all come to you for your free tier stuff... becuase that does seem dangerous, such a giant spike in users who are not paying and don't bring revenue with them! I don't personally use very much heroku free tier stuff. I hope if that's a challenge, it's one you can get over. I don't think you are under any obligation to offer free stuff that can be used for real production workloads indefinitely -- although, as I'm sure you know, free stuff is huge for allowing people to try _before_ they buy, and whatever limits you put on it to try to prevent indefinite production use get in the way of someone's "try before you buy" too... and at this point, _reducing_ your free offerings is a lot harder PR-wise than having started out with less in the first place. :(

anurag4• 3 years ago

I shared your post with Render's engineering team and it got a lot of love because we know the struggle and can truly empathize because of our own Heroku-accelerated growth. What Fly and Render are doing is hard, but someone needs to do it.

If the market is big enough to support AWS/GCP/Azure as $N00B businesses each, it’s not a leap to imagine a future where both Fly and Render are incredibly successful, loved, and independent businesses spanning decades. Let's keep at it.

say_it_as_it_is2• 3 years ago

Can you please explain what the Vault related failure is about? Is this about timing out services failing to start within an acceptable time range?

sayrer• 3 years ago

It sounds like you built the right product on the wrong technology stack, at the lower levels. For example, I have never heard a Nomad success story, but this might be colored by interviewing engineers desperate to escape it.

Something like linkerd on Kubernetes would be stronger, I suspect. But I don't know the exact nature of your problems.

kelp• 3 years ago

Your margins are going to end up being a lot better than any other PaaS that's built on top of the big cloud providers.

sirsinsalot• 3 years ago

Digital Ocean gave me the PaaS replacement and managed PG and I couldn't be happier.

If anyone else is looking.

chillfox3• 3 years ago

I tried running a docker app on Digital Ocean's app platform, the UI is nice and I can see it being acceptable for your average CRUD app. But I had to abandon it due to the latency just being too high and their built in monitoring interval not being configurable to the ~5-20ms range (this was for competitive Battlesnake last year).

say_it_as_it_is2• 3 years ago

Edge of what exactly? The software resides within networks they control. There is no edge in this scenario.

MuffinFlavored• 3 years ago

generous free level of service,

Can't you make the argument that Heroku got out of this market on purpose? I know they were bought out by "ye old corporate greedy meanie overlord" or whatever but... I'm sure there is data that showed it made sense from a "make money business" perspective to not be in that market.

natbobc• 3 years ago

I would even go so far as to say free users aren’t customers in a platform like this. There’s no revenue model and if it’s “run without constraint” you get some fun scaling problems but those types of issues can sometimes lead to optimizations that keep the system online in the face of many small apps but doesn’t necessarily cover the case of a large resource app.

milani• 3 years ago

lower cost as you scale

The cost aside, I'm wondering how fly or heroku support their customers when they grow to microservices ecosystems.

The problem shifts from deploying easily to deploying reliably meaning one release of a service should not break the other services. Other problems appear too, like service discovery, peer authentication, gateways, test and staging environments where there are downstream dependencies, etc.

Are customers supposed to leave when they grow to this level? Or are there solutions for these?

amw-zero• 3 years ago

Why does there need to be a successor to Heroku?

chucky_z2• 3 years ago

mrkurt have you considered some of the lower tiers of vault enterprise that allow for performance replicas that just outright solve that problem? might be cheaper than an engineer at this point.

outworlder• 3 years ago

This is a theme. Existing open source is not designed for global deployment

Eh? Unless you are consuming something as a service and it actually advertises it as a feature, nothing is ready for 'global deployment'.

If you have a 'centralized' secret storage, then you have made it tied to a region. Want to have redundancies and lower latency? You'll have to distribute it. Vault has docs about this: https://developer.hashicorp.com/vault/tutorials/day-one-raft...

josegonzalez6• 3 years ago

This one is interesting because Vault has an enterprise product which I assume (hope?) Fly is paying for. That enterprise version includes performance replicas, which allows for cross-region replication of secrets with region-local reads (and slightly lower writes). The OP almost makes it sound like they are using the non-enterprise versions (or at the very least, not taking advantage of this particular functionality).

That said, I'd imagine with large enough scale, these sorts of features break anyways.

theloco• 3 years ago

I love reading stuff like this. I don't use fly, don't plan to, not totally sure everything it does and will check it out. But this is some great raw data on how stressful it is after you launch.

tebbers• 3 years ago

I really feel for Fly, as a potential customer. They are trying really hard. I would still love to use them one day and this post is definitely a step in the right direction. Growing is painful but they have smart people working there so fingers crossed that they sort this ASAP and it doesn't become existential.

jrochkind16• 3 years ago

I remain kind of amazed about how heroku managed to pull off what they pulled off, in the first case.

Also:

The Heroku exodus broke our assumptions. Pre-Heroku, most of the apps we were running were spread across regions. And: we were growing about 15% per month. But post-Heroku, we got a huge influx of apps in just a few hot spots — and at 30% per month.

I hadn't before seen anyone with a big picture view confirm a heroku exodus was happening, although a lot of people suspected it or had anecdotes.

But if fly is seeing a pretty enormous number of customers moving from heroku to fly... oh wait, now I'm wondering, is this mainly a result of heroku ending free services, and those are free customers coming to fly for free services?

If so... that's a pretty big burden to take on without revenue to match, it does seem kind of dangerous for fly.

mike_d7• 3 years ago

I remain kind of amazed about how heroku managed to pull off what they pulled off

Heroku was built on top of AWS. They didn't have to handle many of the hard problems Fly does.

8organicbits• 3 years ago

Do they unpack what the 30% includes? Is that revenue, VMs, accounts?

lopatin• 3 years ago

It sounds like they need more money to scale the shared stack

ec1096853• 3 years ago

I wonder what types of RPS they are seeing that required a gossip based protocol to broadcast state around versus a more traditional data store.

I take it that it’s far more important that the local region know about changes than a remote region, which makes a mastered store in one location as the source of truth problematic.

I also wonder why these companies don’t backstop themselves on the public cloud? Failing into an AWS seems better than running out of capacity and some its services could be used in circumstances where an open source technology isn’t ready.

likecarter• 3 years ago

Yeah, reading the post made it seem like they followed “best practices” without really thinking things through. KISS.

e1g• 3 years ago

This reads like a mea culpa from an indie hacker, but Fly.io had 5+ years and raised $40M to get these basic fundamentals right. And we get promises of a new status page.

ignoramous8• 3 years ago

Well, that's one way to look at it.

Fly's been many things over the course of its lifetime [0], but I believe their latest pivot (on what they call "Machines") is pretty darn good. I've been using Machines since Oct last year, and things have gotten better week-over-week. Like with any platform, Fly has its own idiosyncrasies, which don't take much to get hang of. That said, I am the only person in my tech shop that deals with Fly. Some orgs with larger teams and heavier apps that deploy frequently or run DBs / disks on Fly (I don't) have had a rough few months; so that's there too.

[0] Ex A: https://news.ycombinator.com/item?id=13985940

mbStavola• 3 years ago

Fly Machines, if I understand them correctly, feels like a step backwards. Sure they might work better than "standard" Fly apps, but one of the motivating cases for Fly is being able to effortlessly scale across the world without having a Ph.D. in CS and a fistful of certs for Cloud engineering. That vision for Fly is awesome, game changing even, ignoring their current stability issues.

Machines isn't that. From the documentation, it appears as though it's "just" a VM pinned to a single region and none of the "magic" of Fly really applies. If the server your VM is hosted on goes down, Fly won't redeploy your container. It's just downtime. Spinning up in other regions is something you have to think about and actually do. It seems closer to Heroku than it does Fly.

Maybe I am totally misunderstanding Fly Machines and their use-case, maybe they're aiming to close the gap between Machines and Fly apps. It's just a bit of a bummer to see something that looks like walking back the original "promise" of Fly and makes me question whether or not Fly is going to just become like every other PaaS (even if it's a really good one).

sph• 3 years ago

Big companies fail spectacularly as well, so it is refreshing to read a indie hacker-style mea culpa than a pile of nonsense PR one would expect from a company that raised $40M.

Honesty pays off in the long run, but it's something businesses quickly forget past a certain stage.

phreack• 3 years ago

It is concerning that they feel notifying a problem on the status page hurts their ego. It is under no circumstances something personal, and ideally it should even be automated.

pier254• 3 years ago

I've been using Fly for over two years or so. The sentiment of this post doesn't align with my personal (anecdotal) experience.

The PG issues hit me two times in the previous weeks but other than that it's been working great for me.

With the move to v2 apps (using their new machines infra) things are actually faster and smoother than ever.

About a year or so ago their CLI was quite buggy but I haven't really hit any bugs in months.

I will remain with Fly for the time being. Hopefully they don't close shop!

tptacek7• 3 years ago

We're nowhere even within the line of sight of closing up shop. We just haven't been doing a good job of aggressively communicating (a) when things go wrong and (b) what we're doing to account for it.

The Fly.io of 2023 looks almost nothing like that of 2021 (all for the better), and it's not obvious to our users what's changed. We've been doing a shitty job of communicating, and we're taking our licks for it now.

chris_st• 3 years ago

A lot can happen in 11 years :-)

And thanks a lot for fly.io -- it's working great for my (rather small) use cases.

russellthehippo• 3 years ago

Agree - V2 apps on machines are incredibly slick to launch (create/start/stop), get info on with graphql, and scale up and out. Magic. When the PG administration experience is that good I'll move it all over.

victorbjorklund• 3 years ago

Great post. Love how transparent Fly is. Im a customer but without any life and death important apps. And yea they had some issues lately so good they are addressing it.

lll-o-lll2• 3 years ago

At first I was all like “Ha ha, losers can’t scale”

And then I was “Huh, these technical challenges are actually pretty difficult”

And then I was all “crap, these are a bunch of technologies I was about to add to our stack”

Thanks heaps fly.io people; having the humility to honestly talk about the challenges and failures massively helps people such as myself as we navigate new unfamiliar technologies. If more companies were willing to do this, it’d be a lot easier to avoid common pitfalls.

chucky_z2• 3 years ago

The tech in their stack is still pretty good. Unless you’re supporting tens of thousands of customers and trying to make the promises that fly makes today. Look at the fly engineer replies in this thread.

Also they basically only use OSS versions, they could go give Hashicorp some money to solve their Vault problems. They could probably partner with SecondQuadrant for PG as two examples. That might not make sense for their business though.

Hard problems are hard no matter the choice.

lll-o-lll2• 3 years ago

Sure, I was going for a little humour there. A little riff on the whole “we always judge others until we walk in their shoes”.

The take away I was hoping for is “providing insights into how we struggle helps others”

ChrisMarshallNY3• 3 years ago

Well, I feel for them. Scaling up is a bitch.

I've been lucky, in the past, but a lot of that, is because I have "overengineered," and the tools/frameworks have advanced to meet the new demand.

I am in the middle of a complete, bottom-to-top rewrite of the app we've been developing for the last couple of years. It's going great, but making this leap was a fraught decision.

It's mainly, so I wouldn't have to write a post like that, in a year or two.

We spent all the time refining it, until we had what we wanted, and it worked great on our small test team.

Then, I loaded up a test server with 10,000 fake users, and tossed the app at that. To be fair, we don't think we'll have even that many users for quite a while. It's a very specialized demographic.

* SOB *

It no do so well.

At that point, I had to decide whether to fix the issues (they were quite fixable), or revisit the architecture.

The main issue with the architecture, was that it was an "accreted" app, with changes gradually being factored in, as we progressed. The main reason for this, is because no one really knew what they wanted, until we ran it up the flagpole (sound familiar?).

The business logic was distributed throughout the app. That was ... ugly.

I envisioned myself, a year or two down the road, sucking on a magnum, because the app had turned into a Cruftosaurus, and was coming for me in my nightmares.

So I decided to rewrite, as we hadn't done any kind of MVP or public beta, so we actually had the runway to do this.

I refined the entire business logic of the app into a single, platform-agnostic SPM module, which took just over a month, and have started to develop the app around that. It's pretty much a rewrite, but I am recycling a lot of the app code. We also brought in our designer, and he's looking at every screen I make. It's working well for him.

Like I said, it's going great. Better than I expected.

I know that I have a huge luxury, and I'm grateful. I can credit a lot of that, to doing some stress-testing before we got to a point where we had a bunch of users to support. I was able to go in, and go all Victor Frankenstein on the model.

The result, so far, is that this thing screams, and you don't really even notice that there's that many users on it. The model has already been proven (that SPM module), and all we're doing, is chrome (which is a ton of work).

bbkane• 3 years ago

Hope this isn't a dumb question, but what's an "SPM module"?

ChrisMarshallNY3• 3 years ago

Sorry. Apple-specific.

Swift Package Manager.

https://littlegreenviper.com/series/spm/

The only dumb question, is the one I don't ask.

machiaweliczny2• 3 years ago

Don’t know about expected usage but 10K rows seems like low number to me. If you mean 10K req/s then perf starts to matter but usually it’s SQL queries that fail first if you have 100K+ rows. In general good caching solves most of stuff + read/write separation. That’s all from my poor experience.

What I mean is that these scaling problems don’t have much to do with app logic but having nice core is good so :thumbsup:

ChrisMarshallNY3• 3 years ago

10K is a rounding error, to most folks, around here.

The app (and the demographic that it Serves) are very security and privacy-conscious, so security and privacy are the main coefficients. Most folks would be disappointed in how few features the app boasts, as each feature is a potential compromise. I'm glad to have low usage, as a result.

I wrote the backends, as well as the frontend, and have avoided third-party dependencies, all around.

It seems one of the first casualties of fast scaling is security.

I really didn't want to take any chances. An overly-complex architecture is begging for security compromises.

skywhopper• 3 years ago

Interesting issues. Nothing surprising for anyone who’s run a global SaaS before, especially if growth has been incredibly fast. I find the gripes about Consul, Nomad, and Vault interesting since it sounds like the problems are mainly due to poor architectural decisions. Fly is rewriting those tools rather than invest in deploying them properly and in the process are running into new issues that those tools have already solved, which doesn’t give me confidence that the path forward will be any less bumpy.

ashiban• 3 years ago

One of the key challenges we observe is that if you're small enough, a Heroku like experience works well - and most of your needs would be covered by virtually any combination of techstacks.

It gets significantly more challenging when you grow, either in feature complexity or scale complexity - and then very few services can offer what AWS/GCP/Azure offer - albeit at the increased engineering/monetary cost of using them.

We're building a different kind of approach[0] that aims to absorb the mechanical cost of using public cloud capabilities (that are proven to scale) without hiding it altogether.

[0] https://github.com/KlothoPlatform/klotho

pyentropy2• 3 years ago

Almost half of the issues are caused by their use of HashiCorp products.

As someone that has started tons of Consul clusters, analyzed tons of Terraform states, developed providers and wrote a HCL parser, I must say this:

HashiCorp built a brand of consistent design & docs, security, strict configuration, distributed-algos-made-approachable... but at its core, it's a very fragile ecosystem. The only benefit of HashiCorp headaches is that you will quickly learn Golang while reading some obscure github.com/hashicorp/blah/blah/file.go :)

tptacek7• 3 years ago

We are asking to HashiCorp products to do things they were not designed to do, in configurations that they don't expect to be deployed in. Take a step back, and the idea of a single global namespace bound up with Raft consistency for a fleet deployed in dozens of regions, providing near-real-time state propagation, is just not at all reasonable. Our state propagation needs are much closer to those of a routing protocol than a distributed key-value database.

I have only positive things to say about every HashiCorp product I've worked with since I got here.

pyentropy2• 3 years ago

I respect that. Can you elaborate a bit on the routing protocol thing? I assume you used WAN gossip?

I love the simplicity of fly.io & wish you all the best improving Fly's reliability!

otterley5• 3 years ago

Well, why did you do that? If you’d asked them whether this was a supported configuration or intended purpose, they’d have said no; and anyone who had experience deploying Consul at large scale would have told you the same.

There is truly no compression algorithm for experience.

pcthrowaway2• 3 years ago

Are there any plans to make Corrosion open source? Or are you able to talk at all about the technologies/patterns used to create it? I feel like service discovery is still ripe for disruption

deivid• 3 years ago

I'm a bit sour reading this. I've always liked fly and particularly the engineering blog, so much so that a couple of months ago I decided to apply for an infra position, to work on some of these very topics. Sadly after 4~5 rounds of interviews (including a workday) they just ghosted me.

tptacek7• 3 years ago

If that happened, it absolutely was not on purpose. Shoot me an email at thomas@fly.io.

ignoramous8• 3 years ago

Don't feel too bad nor take it personal. They probably have a lot of applicants, and are looking to grow their team by hiring someone with very specific skills.
I also applied a few months ago while I was in the middle of my job search. For one, I couldn’t really answer their "favorite syscall question" because I’ve never dealt with syscalls :) so maybe I just wasn't a good fit.

Surely, everyone's favourite syscall is exit()

jlokier• 3 years ago

For what it's worth, exit() is not a syscall, but _exit() often is.

revskill• 3 years ago

Fly.io seems like Vercel 1.0 (where you can just deploy docker image and done), but it's more than that, with configurable volumes, secrets,...

I'm bullish on fly.io.

1023bytes• 3 years ago

I get it, I like fly.io, but the last outage made me switch to Railway.app

nathants• 3 years ago

i also wanted a good cli for aws, and built one:

https://github.com/nathants/libaws

companies like fly are fantastic.

they provide a good service, and they put market pressure on aws.

a free tier isn’t important anymore. with usage based pricing for lambda/dynamo/s3, an app with usage approaching zero has no cost.

plasma• 3 years ago

Thanks for sharing!

Would it help to replace Corrosion with a simpler "Here's my local known state" blob that is POST'd to blob storage (for example) on a major cloud provider, and have another service read that at intervals? Just to make it really simple.

There will be a better way than that, but my thought is if you can make it simpler (known state is always just pushed, so missing updates auto-recovers and avoids corruption) then you can be building on top of a more stable service discovery system.

Centralized secret storage, can you keep the US instance read/write, but replicate read-only copies (a side-car tool that copies the database to other regions at various intervals?) so each region can fetch secrets locally?

Or perhaps both can be solved with a general "Copy local state to other regions" service that is pretty simple but gives each region its own copy of other region's information (secrets, provisioning states, ...).

I've needed to do similar things for some of the apps I've built, where a service needed another (simpler) service in front of it to bear the traffic load but was operationally simple (deferred the smarts to the system it was using as the source of truth) and automatically recovered from failure due to its simplicity.

preseinger• 3 years ago

+1 — service discovery _feels_ like something that gossip can solve, but that's only true at global/open/untrusted scale, in a well-scoped and authoritative domain like a company it's really not, it's really just another typical, expected-to-be-consistent state problem

djha-skin• 3 years ago

In response, we’ve shipped a project called Corrosion. Corrosion is a gossip based service discovery system.

I wonder why they didn't try to use Serf[1] for this, since they were so into HashiCorp tools. It also uses the gossip protocol.

1: https://www.serf.io/docs/index.html

swamp40• 3 years ago

"CEO finds novel way to fix Capacity problems..."

They just lost about 40% of their paying users with that blog post.

nomilk2• 3 years ago

What an earnest post, and how damn refreshing it is to see such concern for users, accountability, honesty and openness (quite a contrast to another PaaS)!

I moved one app successfully from heroku to fly and attempted to move a few others. These are my experiences (both good and bad):

Great:

- The load time on the pages is insanely faster on fly than heroku. Sometimes I thought I was on the localhost version of the app, it was that snappy.

- Love that it uses a Dockerfile

- Love paying for what I use (compared to Heroku's rigid minimum of $16/month for hobby dyno w/ postgres for baby apps, or $34/month just to get a second web dyno for toddler apps). The same apps are <$5/month each on fly.

Not great:

- I find the fly.toml file hard to understand and use, and the cycle time slow to fix or tinker with it. It's partly (entirely?) a 'me' problem because I haven't spent a huge amount of time reading the documentation.

- I found scheduling a rake task in a rails app time consuming (~days) the first time, but very easy (15 minutes) the second and subsequent times, once I knew a way that worked (cron didn't work; had to use a tool I hadn't used before 'supercronic').

- Deploys sometimes time out with `Error failed to fetch an image or build from source: error rendering push status stream: EOF`. Most layers copied, but randomly, some layers wouldn't. All I could do is keep trying until it worked, which it did, 2 hours later. Not the end of the world, but an annoying complication when you're already trying to solve complex problems.

- I followed a youtube video on how to move a rails app from heroku to fly, and it worked on a modern app, but I couldn't quite get fly happy when moving the older app - something to do with postgres versions, and I didn't want to spend all day figuring it out. I'm not hugely experienced with docker, it could have been an easy fix for someone more experienced.

On reflection, 3 of the 4 negatives above are solvable by me reading the docs more thoroughly and getting more proficient with docker.

I look forward to continuing using and exploring fly, and can't be happier with the directness, transparency and care from fly staff. A platform with huge potential.

bradgessler• 3 years ago

Did you try migrating with this guide? https://fly.io/docs/rails/getting-started/migrate-from-herok...

The issues you ran into with older versions of Rails was probably because the Dockerfile that `fly launch` generated was for new versions of Rails. We switched to https://github.com/rubys/dockerfile-rails to streamline Dockerfile generation and support older versions of Rails.

If you try it again and run into issues you can open an issue at https://github.com/rubys/dockerfile-rails/issues or post in https://community.fly.io and somebody will help get that sorted out.

The more versions of Rails we can deploy the better!

nomilk2• 3 years ago

You bet, that guide is gold.

I encountered a few issues. One was definitely something to do with older postgres. From memory, I tried downgrading it using apt, but then other things played up and I put the project aside.

Another rails 6 app I tried to move into fly encountered this: https://community.fly.io/t/rails-app-problem-with-node-modul...

I followed Sam's suggestions to regenerate the Dockerfile with dockerfile-rails (btw, thanks for your work on dockerfile-rails, super excited about it) and solved a couple more issues, but I again ran out of steam when new issues kept coming and coming. I'm sure when I'm more comfortable with docker these will become trivial to solve.

These were not super determined attempts by me, more playing around. I look forward to more serious attempts when I'm more capable with docker.

andy_ppp• 3 years ago

How easy is it to deploy things with GitHub actions, AWS and terraform instead? I think I’d like full infrastructure control after having accidentally got a job doing DevOps which I’m starting to get quite into. I should probably write up a blogpost about converting everything over…

none_to_remain• 3 years ago

Three little pigs, but they all live in a house of straw built on top of several houses of sticks. (The foundation is an old house of brick, but the wolf isn't bothered by that.)

Karupan• 3 years ago

Thanks for the honest technical write up - not easy to air one's dirty laundry to users. Given the scalability and stability issues, I am curious to understand the percent of apps deployed to fly are actually used in production/critical to business. Sounds like they have quite a few hobby/free tier users (myself included) who probably won't notice certain issues unlike paid customers.

hinkley3• 3 years ago

We’ve put a lot of work and resources into growing the platform and maturing our engineering organization. But that work has lagged growth.

I fundamentally don't understand why people are in such a big hurry to get 'famous'. I've worked a couple of places where the marketing side was working as hard as they could to make sure that our heads were on fire at all possible moments. At one job I had a (very, very junior) manager come up to me and say great news we landed <big customer> and my immediate reply was, "fuck me". We were already running to stay upright and now we're about to have twice as much scrutiny. Wonderful.

If you push hard enough, eventually everyone looks like an idiot. The number of humans for whom that is not true could fit into a book. Both alive and deceased. They most definitely do not work for the companies I've described, at least not enough of them so you'd notice.

computomatic• 3 years ago

Where does fly.io document their per-account services limits? For example, max apps, databases, etc.

I took a quick look and couldn't find them. Do they have any documented service limits?

A google search turned up [0] which does not inspire optimism.

...there isn’t a limit to number of apps from a billing standpoint...

[0] https://community.fly.io/t/free-tier-limits-and-quota-needs-...

rtpg• 3 years ago

I have two extremely tiny sites (like, "handful of users/1 user" sites) on Fly. I have had multiple incidents despite me not even touching them.

The thing that worries me about these incidents is they haven't been, like, full service outages. A small subset of users talking about issues in forums. This makes me just feel like Fly has an immense amount of issues.

At least if like 50% of fly goes down then it feels like a config fat finger. When it's a bunch of tiny issues now all my ops debugging has to start with going to the fly forums (and it's _always been issues on fly's side_).

The price is "right" (though like with all PaaS the gaslighting about running multiple processes in one container makes me feel bad about the state of cloud computing). And I really like the CLI stuff mostly! But I extremely don't care about edge computing so for me fly is just heroku and I would love to feel more confident on that end.

(EDIT: the nice thing is I get email support with a bit of cash. This is a thing that will go away when they get bigger but it's here while things are still breaking often)

debarshri• 3 years ago

This is a great blog and have so many insights for an SRE or devops person. But this goes to show you how difficult it is self host stuff at scale.

I used to work for a company that built deployment platforms for law firms. All our deployments where on prem and we had the same complexity with kubernetes. We had similar setup with vault and stolon for HA PG. More moving parts you have in infra, more permutations and combinations of failure modes you have.

What these guys are building is something I have seen in many orgs trying to do it internally and fail. PaaS is a hard problem if you want to solve it "reliabily"

anacrolix• 3 years ago

    - Machines seem like a waste of time
    - Access directly to VMs is being removed (and doesn't support TCP over IPv4, or UDP over IPv6)
    - The CDN is nice but should support private networking too.
    - Volume management is deficient: It should be possible to access and fix volumes outside the context of an its app instance.
    - Egress traffic should be free between apps over private networking, at least in the same DC.

KETpXDDzR• 3 years ago

That sounds like a typical problem of unnecessary complexity to me. I wonder how many over engineered (web) applications could run as a single, efficient process on a single machine.

TheDong3• 3 years ago

Do you know what fly.io does?

Like, it cannot possibly run on a single machine by definition because the product they're building is running customer code on edge compute nodes distributed across the globe.

The main selling point of their service is that it's _not_ a single machine.

throwawaaarrgh2• 3 years ago

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability.

Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatened by competing things and literal bugs and weather and the soil it's grown in and more. It needs constant care. Over time it grows stronger, and can eventually fend for itself pretty well. Sometimes you get lucky and it just grows fine by itself. And sometimes 50 different things conspire to kill it. But you have to be there monitoring it, finding the problems, learning how to prevent them. Every garden is a little different.

It doesn't matter what a company like Fly does technology wise. It takes time and care and churning. Eventually they will be reliable. But the initial process takes a while. And every new piece of tech they throw in is another plant in the garden.

So the good news is, they can become really reliable. But the bad news is, it doesn't come fast, and the more new plants they put in the ground, the more concerns there are to address before the garden is self sustaining.

mrkurt6• 3 years ago

This is an excellent description.

IshKebab2• 3 years ago

Maybe so but there are definitely technology choices that have vastly different "initial reliability". What would you expect to be more reliable - a bash script or a Rust program?

I'm not that familiar with setting up global network infrastructure but I imagine there are similar choices that can vastly affect initial reliability.

capableweb• 3 years ago

I'd say it depends more on the person rather than the technology.

A master in bash will build more reliable API (in bash no less!) than a beginner in Rust, simply because of experience and knowing their way around the tools they're using. Newer/different technologies won't simply solve a problem unless the person has some sort of domain knowledge of said problem.

zamnos6• 3 years ago

There are no silver bullets for whole system reliability, but high-availability clustered databases was this wiz bang thing that greatly improved the reliability of your database, back in the day. It didn't come cheap, and there were growing pains, but sometimes the available technology does make a difference.

unxdfa2• 3 years ago

You can make sensible assumptions that result in engineering gains though. Step around the problems not through them.

For example I have learned that the first step to reliability is removing as many hashicorp products from your stack as possible though. Appears I am not the only one.

jen202• 3 years ago

If you’ve been using them in ways clearly explicitly called out as not per the design goals, then sure, removing any piece of technology will help you. I’m guessing that is not your actual problem though.

TheDong3• 3 years ago

The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability.

It's seems true for fly's problem space, but in many problem spaces there really are easy engineering solutions to reliability problems.

For a very easy example, I once worked on a rails app that crashed frequently and managed 5 req/s at best. It turns out the app only loaded static data from hardcoded json files on disk and templated that into stuff. In other words, it was a static site. Replacing it with an actual static site + nginx and a cdn instantly fixed all reliability issues for that website forever, and made it easier to maintain the content to boot.

machinawhite• 3 years ago

I'm actually surprised such a simple app would have such bad performance and crash at all?

bennetthi• 3 years ago

Nicely said. I remember AWS outages (S3, EBS, and RDS) in the early 2010s when their products were younger. But given time to improve each has become more and more resilient.

HPsquared• 3 years ago

Good engineering helps reliability but doesn't guarantee it.

Bad engineering causes bad reliability.

benatkin• 3 years ago

It's too late. They've already misled people.

siliconc0w• 3 years ago

Respect the post, building your own infrastructure provider is playing on hard, the big players have had armies of engineers iterating on their stacks for a decade+

birracerveza• 3 years ago

What is this? A company being open and honest about problems their customers are facing? What is happening? Has the world gone mad??

nprateem3• 3 years ago

Interesting take. As a non customer I now won't consider them for any projects as they've confirmed their unreliability.

vinay_ys• 3 years ago

Corrosion seems over-engineered. Instead of doing a simpler federation of multiple databases (one per datacenter) across the globe, they decided to do gossip amongst every single VM across the globe! You don't really gain much but you do get all the noisy complexity for sure.

clement_b• 3 years ago

I've been with fly.io at small scale and I have always loved their approach to content (docs, blogs, forums), and needless to say, their product. They are very talented and are building something truly great. Their openness, shown in this post, is an example to follow. It's very hard to be that honest and direct when you're meant to be an infallible entity, but it's not a surprise at all to see fly authoring such a post. That's how they operate and that's why I trust them over anyone else.

davedx• 3 years ago

@kurt happy (sometimes paying) Fly user here. Keep up the great work!

gpjanik• 3 years ago

I like Fly's response to the problems - honesty and openness - so I'm going to add to their problems and try to use it ;)

ecmascript• 3 years ago

Wow, posts like these kind of want me to sign up and become a customer. The honesty and openness is something I highly value and it's weird that this is so uncommon that you get surprised by reading stuff like this.

Seems like they have a good understanding what the problems are so they will most likely be solved sooner or later.

Good work and keep honesty as open as you've done so far :)

sidcool• 3 years ago

I have a debilitating impostor syndrome around resiliency and reliability of my systems. I always feel I am doing something wrong compared to the Googles, the Microsofts and the Fly.ios of the world. This does help feel better.

al_be_back• 3 years ago

in my view, a new edge computing entrant needs a niche market e.g low latency gaming or privacy-heavy computing, and to stay away from MAAG cloud territory.

Dave3of5• 3 years ago

I always thought it would be really interesting to work for a company like fly.io.

Solving hard problems like this seems interesting.

On the other hand it could be a giant shit show of micromanagement and toxicity, who knows really.

At the moment they aren't hiring though so that's that.

tonnydourado• 3 years ago

Damn. I don't know, and I guess almost no one can know, how much of this is genuine honesty and how much is calculated messaging, but I barely care. It's refreshing enough that I kinda wanna give the thing a try. Which, ironically, might make their problems just a tiny little bit worse, since I likely won't be a paying customer XD

crabbone• 3 years ago

I'm not in the Web business, and have no idea what fly.io is offering, but whenever I hear anyone trashing Consul, I give them a standing ovation. An application which decided to use Base64-encoded JSON for its communication protocol deserves every bit of mockery it can get.

pwelch2• 3 years ago

Growing pains are never fun. It doesn't mention (at least that I read) if they're using HashiCorp Open Source or Enterprise. Open Source is great and I owe my career to it but they might be hitting the scale when the Enterprise features and support start to be worth the price.

I've only used Fly.io for a personal app but I think it's a great option so I hope they keep growing.

pwelch2• 3 years ago

Ah my question was answered more or less. They're an edge case which makes sense: https://news.ycombinator.com/item?id=35048318

Reliability: It’s not great

Related Stories