How Canva saves Amazon S3 costs

https://www.canva.dev/blog/engineering/optimising-s3-savings/

by kenaqshal • 3 years ago

134 177 3 years ago

canva.dev

andrewstuart3• 3 years ago

Seems crazy to run such large infrastructure on a major cloud.

Tens of millions wasted.

Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible.

S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave.

S3 is 9 cents per GB egress fees.

Cloudflare R2 charges zero egress fees.

At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.

yashkadakia• 3 years ago

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!).

We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of time where the expected load was low to maximize server utilization in those periods.

ssss11• 3 years ago

Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.

onion2k2• 3 years ago

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on.

You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.

username_my1• 3 years ago

usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.

PaywallBuster• 3 years ago

or just consider different providers

and easily lower costs by 50% depending on balance storage/bandwidth

andrewstuart3• 3 years ago

> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops

This is the traditional “sell” for cloud computing and I don’t buy it at all.

The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems.

The clouds sales pitch is the if you go cloud then you don’t need all these specialists.

That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

The old “don’t run your own systems, it’s cheaper and easier to go cloud” is just sales fiction.

Don’t believe it.

amluto2• 3 years ago

And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1)

This is an important factor to remember when evaluating could costs. If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours. And you need to deal with the potentially catastrophic cost of inter-AZ or inter-region traffic, which can be catastrophic and/or cost more developer hours to mitigate.

zokier3• 3 years ago

Many aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.

littlestymaar7• 3 years ago

Millions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.

coldtea• 3 years ago

If you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy.

More of a "good problem to have" (and you can solve it when you need to).

ramraj07• 3 years ago

Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.

littlestymaar7• 3 years ago

I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation.

Also, it seams that it's 230PB in total, not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).

xarope• 3 years ago

let me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime):

- if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs).

- on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately".

Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage.

So long as they are still earning money, why wouldn't they stick to this OPEX model?

Not saying this is good or bad, it depends on your business model, ultimately.

re-thc3• 3 years ago

So long as they are still earning money, why wouldn't they stick to this OPEX model?

The suggestion was to move to a different / cheaper provider. You can stick to OPEX.

ToJans• 3 years ago

I'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.

re-thc3• 3 years ago

Cloudflare R2 charges zero egress fees.

The scarier part is Canva actually does use Cloudflare CDN, so it's not so much the cost of adding a new provider. It's just that the CDN team that manages the Cloudflare account is likely a different team :)

Too much isolation, silos and politics...

sofixa• 3 years ago

The much more likely answer is that R2 is too new compared to when they started (announced in 2021) and switching from S3 to it is probably not trivial at Canva's scale.

Sai_• 3 years ago

They could optimize this by only egressing they most recently used petabytes of data for some value of “recently”.

littlestymaar7• 3 years ago

“Nobody ever got fired for choosing I̵B̶M̶ Amazon”

whycombagator• 3 years ago

What happens when cloudflare changes the price?

ttymck• 3 years ago

Why do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on <new storage>

notyourwork2• 3 years ago

Absolutely agree. There are more nuanced strategies to migrate off a cloud provider. Could even enable a soft export for idle accounts, leave it where it lie and move it when customer re-engages.

pc862• 3 years ago

I would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.

avereveard2• 3 years ago

what's the durability of R2?

re-thc3• 3 years ago

eleven 9's apparently

VHRanger2• 3 years ago

How would you measure that realistically? How durable is Backblaze or Azure?

Cloudflare has been reliable on storage adjacent uses (caching) for a long time. Presumably you're more likely to be the cause of errors than R2 is by a few orders of magnitude.

speed_spread• 3 years ago

Nitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!

electroly3• 3 years ago

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB.

Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.

fakedang• 3 years ago

Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)

dehrmann2• 3 years ago

I worked at a place that did an AWS to GCP migration. I didn't look into the details, but they used a provider that basically gave them a fat, dedicated fiber link between the two (they picked the same GCP region). It was good enough that requests could have data cross clouds. It was expensive to set up, but it saved on egress costs and opened up options for migrating services.

gurchik• 3 years ago

Cloudflare R2 also has a tool in beta that provides an incremental migration from S3. As users request files using your R2 URLs, Cloudflare automatically migrates the files from S3 to R2 on request. Depending on your user's request patterns this may be a way to migrate everything without paying any additional cost. If you have a lot of files that are rarely accessed then they will be slow to migrate, but because they are rarely accessed your S3 costs could be significantly reduced by using different storage classes anyway. Eventually you will have a small enough number of infrequently accessed files remaining in S3 that the additional cost to migrate them all in one go can make financial sense.

xwowsersx• 3 years ago

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.

ignoramous• 3 years ago

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful.

See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/

shadowtree3• 3 years ago

Most businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute.

Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.

betaby4• 3 years ago

Infra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.

dehrmann2• 3 years ago

The real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.

choppaface• 3 years ago

“we store more than 230 Petabytes of data in Amazon S3, with our single largest S3 bucket coming in at a whopping 45 Petabytes”

How did they get so much data?! They say they only have 75 million stock photos. They must have tens of petabytes that are just dark.

jedberg4• 3 years ago

At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.

If you're doing a one-time export you can use a Snowball, which would cost about $70K for 230PB, assuming you had somewhere to load it off to once you got the data so you could give the device back.

alecco• 3 years ago

And now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"

vkaku2• 3 years ago

Cloudflare could build a product that not only caches, but transfers S3 to R2 on usage.

- Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2.

- Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2.

Standard migration procedure.

rat9988• 3 years ago

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

acchow• 3 years ago

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops.

There's tons of OSS to manage all layers of this stuff now.

colechristensen3• 3 years ago

You can have a few petabytes in a single rack.

50-ish drives in a single 4U server is done.

simonjgreen6• 3 years ago

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage.

RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of mirrors" (RAID10) gives you the safest and the highest performance array (at scales where these decisions are being made which most people would see anyway). The common argument against is the cost (usableCapacity = totalCapacity/2) but my counter argument is that RAID5 is a timebomb with a secret countdown. RAID5 is great if you don't need the data on the array.

Source: >20 years running complex large arrays

aljarry• 3 years ago

But that's on a single RAID in a single location. You'd probably need at least one failover, and backups for that as well, if you'd like to target higher availability / durability.

S3 keeps multiple copies in multiple geographic locations.

withinboredom4• 3 years ago

Buy hard drives. Typically the spinning ones for the data because $/GB is really cheap. Then you buy some SSDs for your actual applications to run on.

From there you install Hadoop, maybe a query tool like Spark, Impala, and Presto. Then you install some reporting tools like Apache SuperSet.

At least that's how we did it at my old job with Petabytes of data.

zokier3• 3 years ago

Quickly checking Dell sells you boxes that are advertised as 14PB per rack and have S3 api. You probably can do better by shopping around, but it gives a ballpark what is available in the on-prem market.

pjc50• 3 years ago

Have a look at the Backblaze blogs, where they detail how they built a storage provider on the cheap.

bluedino• 3 years ago

and be prepared to write your own software stack

dx034• 3 years ago

If you don't mind it running in a data center in Europe, you could rent servers from Hetzner for ca. $400 per month for 220TB. I'm sure they can provide a few PB on the spot. Spread them across their locations and you have geo redundancy too. Next to no egress fee also allows to have a copy with another provider.

https://www.hetzner.com/dedicated-rootserver/matrix-sx

jokethrowaway• 3 years ago

Canva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s

More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud.

At Canva's scale you'll spend hundreds of thousands, save literal millions and have way more headroom and stability than with crappy auctioned instances on AWS.

iLoveOncall2• 3 years ago

At Canva's scale you'll spend hundreds of thousands a year, just hiring one of the engineers working on your home-grown solution. Multiple this by 50 or more just for the staff, before talking about hardware, cost of delivery speed, cost of bugs, etc.

andrewstuart3• 3 years ago

Instead you can spend millions on paying top market dollar to AWS experts trying to work out your giant pile of spaghetti cloud infrastructure.

Desperately trying to work out if they can delete this resource or that because who the hell knows which bit of critical code is using it. Or spending days trying to manage a chewing gum ball of IAM policies.

oefrha• 3 years ago

Apple uses AWS for most(?) of their cloud services. TikTok grew to the #1 spot entirely on a public cloud (Alibaba). I’m sure they sure have hired “a few ops people that actually know how to operate a server” instead.

nemothekid• 3 years ago

The anti-cloud sentiment on HN is as old as time; there will always be posters who can claim that they can build X with just dedicated machines on Hetzner for 1/10th of the cost. Yet AWS keeps growing and the people running these companies must be unable to do basic math. My account just crossed 11 years, and I think the whole cloud vs on-prem conversation is probably one of the most least productive conversations on HN. If there was so much waste out there you would think someone would figure out how to just siphon all that waste directly into their pockets instead of Jeff's; and maybe they have, I don't know, but the posts are always the same, handwaving about how the mark ups are insane the engineers at these companies aren't able to balance a checkbook.

betaby4• 3 years ago

Apple operates their own data centers. Apple uses GCP as part of the Google's deal related to search.

orangepanda2• 3 years ago

Offtopic — is it some cardinal sin to include a link to the home page from a documentation? I dont know what canva does, if I stumble on a documentation page first, should I not be allowed to see the product's landing page at some point? This is so prevalent, it has to be a conscious choice.

skilled• 3 years ago

What? There are multiple homepage links on this page, including the Logo, Home from navigation menu, as well as the first word in the article is a link out to Canva's homepage.

orangepanda2• 3 years ago

The logo and "home" do not lead to the product page - they lead to developer docs home page.

Did not think to try the first link though.

Solvency2• 3 years ago

My favorite is when you click on some landing page made by a company.... and there is zero way to get to the rest of the site. Marketing-centric landing pages often completely strip out their global navigation for no reason.

SerCe• 3 years ago

Thank you for raising this! I'll take a look at how we can make the discovery more straightforward.

motoxpro2• 3 years ago

A lot of people in this thread should start consultancies that transition companies off the cloud.

Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.

vasco2• 3 years ago

Couldn't agree more.

One only says this if they don't have a good understanding of the differences between capex & opex, the amount of money spent on tech labor & compliance projects, as well as the level of assurances AWS give you regarding durability of data. All of this in 2023 when we have almost 20 years of data showing more and more business moving to the cloud and staying in the cloud.

I think anti-cloud sentiment aligns with hacker mentality on decentralization=good (which it is), anti big corporation feelings (boo amazon) and so it leads people in these threads to make emotional arguments over something that is clearly going in the other direction. It's an excellent example of confirmation bias, trying to look for any and all justification to say the cloud is worse, when the majority of businesses have decided otherwise.

charcircuit3• 3 years ago

Have you even looked at S3 bandwidth pricing? I don't get how anyone using S3 for storing media or binaries can justify that outrageous price unless every download is connected to something making you money.

jesterson• 3 years ago

It's just a part of natural cycle.

Cloud became a fad a while ago, companies like Amazon/Google/MSFT rushed in burning cash to provide incentives. Now said companies adjust fees to milk businesses due to high exit cost, just because they can. This will inevitably create outflow of customers, however we are far from this point currently.

citrin_ru• 3 years ago

Consultants who help to move to the cloud are not cheap too. And migration to AWS/Azure is still more common than out.

jabroni_salad• 3 years ago

Pretty much the only go-backs that I have ever seen were when someone tried to forklift an onprem application that was never meant to run on a computer whose cpu cycles are metered. One of the most striking things to me in ops:dev relations was the way the conversation around performance and efficiency changed once we started paying for it by the cycle.

paxys2• 3 years ago

They could do it all in a weekend, easy.

llanowarelves• 3 years ago

The thing about economics is that it's physics, with a time delay. We are seeing the time-delayed reaction to over-extension into the cloud play out, not instantly but in the downturn cycle.

hinkley3• 3 years ago

So do you think server hardware companies will do well during a downturn?

AntonCTO4• 3 years ago

I do. Interested? :D On a small project, I managed to reduce costs by 70 times simply by transitioning and eliminating unnecessary features. This happened four years ago, and the project is still running smoothly. However, what I truly enjoy is providing an objective view of the actual cloud transition process to companies before they make any misguided decisions.

motoxpro2• 3 years ago

Just a commentary that if you could save a bunch of companies 70x then that is the easiest sell in the world. I WISH I could do something like that because I would be a billionaire haha.

electroly3• 3 years ago

There is a lot of good money to be made for people who understand cloud cost structures and the alternative options available. You'll want to target mid-size businesses that bet on a cloud and were actually successful but are now drowning in cloud bills. Large companies will hire in-house expertise, and small companies don't spend enough for it to be worth hiring a contractor. But I really think there's a market of mid-size companies who don't know how to reduce their AWS bills. Cloud cost optimization (and perhaps migrations between clouds or to hybrid cloud/on-prem at times) would be a great business for a solo contractor. Being able to pay for your own contracting fee out of the savings is almost a given.

jaza• 3 years ago

I'm really surprised how much anti cloud opinion is being voiced here (but maybe I should have known better, after all these years on HN).

Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.

withinboredom4• 3 years ago

I'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes.

[1] https://garagehq.deuxfleurs.fr/

paxys2• 3 years ago

All the marketing pages in the world mean nothing until they can show who currently uses them and at what scale.

tansan• 3 years ago

Whoa! this is the first time I have seen deuxfleurs. Do french people use this over github?

AntonCTO4• 3 years ago

Thank you, that's amazing! While there isn't an extensive comparison available, I have come across a small benchmark that compares a few features. You can find it here: https://garagehq.deuxfleurs.fr/blog/2022-perf/

activiation• 3 years ago

If you want trouble and exorbitant fees, the cloud is for you...

xnx• 3 years ago

Articles like this are a great example of how you can be a hero for fixing a problem, but get no recognition for preventing one.

soderfoo• 3 years ago

Surely they negotiated a private pricing agreement given the volume of data/usage...one would hope.

Having been a part of these negotiations for services other than S3, DTO discounts can be generous if you have a good forecast (3-5 years) on your data egress.

StratusBen• 3 years ago

This is an interesting post...though I'm surprised to see this kind of optimization occur so late stage/as large as Canva.

There's quite a few anti-cloud opinions in the comments but for anyone who is hosted on S3 and would like this same level of automated analysis for cost savings on S3 (and other AWS services), we basically profile for this out-of-the-box on Vantage and it's free to get an optimization check to see savings: https://www.vantage.sh/

Not meant to be as shameless of a plug, but seems very relevant to the topic at hand for onlookers.

Solvency2• 3 years ago

"I'm surprised to see this kind of optimization occur so late"

Every single day I read the trope on HN that "premature optimization is the root of all evil".

So, these guys literally followed HN's creed.

lowbloodsugar• 3 years ago

Canva is ten years old. They grew to 1 million users in two years. Doing this work 8 years ago wouldn't have been "premature optimization". More likely, this is happening now simply because the financial environment has changed and companies are now giving increased weight to projects that save money over those that add features.

jgalt2122• 3 years ago

100 million monthly active users

Oh, my. That's a lot. What does Figma? 300?

jgalt2122• 3 years ago

Canva now has users across 190 countries, who design in over 100 languages thanks to some amazing localization work done by our Internationalisation team.

How may of those countries are profitable? My sense is Internationlization is a relatively fixed cost, but how does customer support in Burma, for example, not trump any realizable revenues there?

Dunedan• 3 years ago

Something which is often overlooked is that transitioning objects in S3 to another storage class using lifecycle policies is a one-way operation. If you later notice that the chosen storage class doesn't fit or your use case changes, you can only go "forward" using lifecycle policies and not "back" to e.g. "S3 Standard" [1]. Going "back" is therefore a quite costly and cumbersome operation, as you have to apply the change of the storage class to each object individually.

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...

MagicMoonlight• 3 years ago

There is no way they need 230 PB saved. That's javascript-tier design in practice. If you can put almost all of it into glacial storage then that's because you don't actually need to store it.

encodedrose• 3 years ago

If you're at a point of spending $1.6M to transition objects from one class to another and spending millions per month in storage costs -- you need to have a real conversation about if storing your data in a vendor-locked cloud is the right path forward. S3 isn't the only option, MinIO + dense storage is one viable option if your spend is high enough to justify running MinIO. Backblaze is another.

List price of Backblaze:

$0.005/GB - Base Storage

$0.01/GB - Egress (Use a data partner to bring this down to $0)

$0 - PUT API Requests

$0.004/10,000 - GET API Requests

No delete penalties a/k/a minimum storage

List price of Glacier Instant Retrieval:

$0.004/GB - Base Storage

$0.09/GB - Egress

$0.02/1,000 - PUT API Requests

$0.01/1,000 - GET API Requests

Billed for 90 days of storage

DarthNebo• 3 years ago

Yeah no, R2 is a way better choice than being vendor locked like this when burning millions

How Canva saves Amazon S3 costs

Related Stories