Backblaze Storage Pod 4.5 – Tweaking a Proven Design

https://www.backblaze.com/blog/storage-pod-4-5-tweaking-a-proven-design/?hn=1

by slyall • 11 years ago

73 59 11 years ago

backblaze.com

omarforgotpwd• 11 years ago

Why design your own storage pods when services like Amazon S3 offer similar prices, even with bandwidth?

vitovito• 11 years ago

S3 is $0.03/GB/month.

Your own pod is $0.048/GB/once.

ashish01• 11 years ago

To be fair 0.03 also includes operation costs like power and space, replacements. I _feel_ that a backup company will have an operational advantage having its own storage but its more complicated than comparing cost per GB.

e12e2• 11 years ago

How do you figure 0.03 per month is similar to 0.05 per 4? years?

[Ed: Not to mention, from the comments: "our drives [have done] roughly 4 gbps of traffic over the last week." I don't know about you, but I'm not doing 4 gbps sustained to s3 ...]

eCa• 11 years ago

Regarding your edit, if I read it correctly that quote is not from Backblaze.

skuhn8• 11 years ago

It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors.

I personally wouldn't use Backblaze's storage pod, so my numbers are based on equivalent components (and don't reflect what's possible if you negotiate prices and optimize for your particular needs -- Backblaze's costs are surely lower than this):

  11 Supermicro 36 drive 4U server : $4000 each
  1 48-port top-of-rack switch : $5000 each
  374 4TB drive : $400 each
  22 500GB drive : $200 each
  2 CDU : $3000 each
  Rack + integration : $10000 each

Total rack cost is $220,000 for 1.4PB or 700GB with basic 2-copy redundancy. Power draw is around 8kw, so 1 year of power is around $1100. Figure on a drive failure rate of 5%, memory 2%, PSU 3% and you might spend another $18,000 to stock spares. So besides your core datacenter costs, this solution will cost $239,100 to operate for a year. These costs only improve when you amortize the hardware over a 3 year period, which is standard.

Now how much does S3 cost to store 700GB for a year? $248,530.94.

Finally, consider bandwidth costs. I'll crib from a recent comment of mine [1]:

1 gbit/s in a datacenter: $30,000 / year 500 mbit/s in S3: $142,540 / year

And that in a nutshell is why AWS doesn't make financial sense at any significant scale. S3 makes sense if you aren't really going to use all of that space: you can scale your spending up as your needs scale up. That isn't the kind of problem Backblaze or other storage businesses have, there isn't much chance of that storage going unused for any extended period of time. And if they architect their datacenter and service correctly, the down period between rolling in hardware and getting it utilized sufficiently to begin seeing a return on the investment should be minimal.

[1] https://news.ycombinator.com/item?id=9139244

toomuchtodo2• 11 years ago

Thanks for doing the math. I've repeatedly shown people (I do Infrastructure, sometimes AWS sometimes physical) that AWS is awesome for proof of concept. All opex, no commitments. As soon as you start getting to a certain scale though, time to move onto physical equipment when you've got predictable load patterns.

If Stackoverflow can still do hardware colo'd, so can you.

jrs235• 11 years ago

I think you mean 700TB not 700GB

nullc• 11 years ago

1.4PB or 700GB with basic 2-copy redundancy

You mean 700TB.

moe3• 11 years ago

Your cost estimates are outlandish, making Amazon S3 look like a bargain.

In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa.

If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).

vidarh2• 11 years ago

I've done similar calculations for clients, and fully agree.

Cutting the storage costs is easy, but bandwidth in particular is ridiculously priced at Amazon (for the cost of 1TB transfer from AWS, I can rent a server with many times as much transfer and still have money left over), to the point where even if a client for some reason insists on S3 storage for perceived safety, it can still pay to store in S3 but put boxes "in front" with capacity for a single copy of each object, and fall back to S3 in case of local drive failures.

Unless the load is totally write dominated (so not good for backup services), cutting the AWS bandwidth bill can often pay for the extra servers and colo costs many times over.

parasubvert3• 11 years ago

So, why do Instagram and Netflix continue to use S3 for petabytes of video and image data? In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3.

The math is never this simple.

Firstly I assume you mean 700 TB, not GB.

You might as well look at reduced-redundancy S3 which brings the annual cost down to $194,220. With that you still have multiple WAN links, excellent throughput, multiple data centers, and no labour for dealing with failed drives.

And if we're truly trying to do backup / archival, where download throughput doesn't matter, let's look at Glacier, even if we do 50000 monthly uploads and retrieve 20% of the 700TB monthly, we're down to $96,630.

Most sizable businesses also aren't buying bargain basement drive arrays from SuperMicro, they're buying EMC, NetApp, 3PAR, or Hitachi.

As with anything S3 is not perfect, but it really depends on your use case if your own data center or rented equipment is appropriate. For many large businesses, Amazon makes more sense.

kylec• 11 years ago

Backblaze did the math a few years ago and found that Amazon S3 was about 24x more expensive than building their own pods, for an estimated pod lifetime of about 3 years:

https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...

atYevP7• 11 years ago

Yev from Backblaze here -> When we first started producing the pods, we wanted to do two things: 1) provide unlimited backup 2) provide it for $5/month. We did the math at the time and found that for $5/month you'd be able to store about 30GB a year on Amazon S3 (at the time it was about $0.15/gb). So we needed to create our own solution to this problem and designed storage pod 1.0, which provided 67 terabytes for about $7500. We figured the average person would store more than 30GB of data so the Amazon route wouldn't work. Ours did and continues to, based on a variety of factors but the main one being, as Amazon's costs decrease, so do ours (except for the whole Thailand thing...https://www.backblaze.com/blog/backblaze_drive_farming/ and https://www.backblaze.com/blog/farming-hard-drives-2-years-a...). Plus it's kind of nice controlling your own destiny!

mbrameld• 11 years ago

Out of curiosity, did you assume that they just hadn't thought of services like that and didn't run any numbers to see if what they were doing made sense?

fraXis• 11 years ago

Duplicate: https://news.ycombinator.com/item?id=9154273

atYevP7• 11 years ago

Weird how timing is everything, right?

slyall• 11 years ago

I submitted and got that one. Problem is it had already vanished from the "new" page so wasn't going to make the front, ever.

Resubmitted with "Backblaze" in the title to get better traction. I got lucky this time, sometimes I submit something and get nowhere while another submit is the ones that takes off.

Maybe a tweak to the new page so that if somebody resubmits a URL that has fallen off the new page it gets pushed back on there. The current system is a bit of a lottery, just one chance for a URL to make it and then it's gone.

k1t• 11 years ago

  Secure Connection Failed

  An error occurred during a connection to www.backblaze.com.
  Cannot communicate securely with peer: no common encryption algorithm(s).
  (Error code: ssl_error_no_cypher_overlap)

Oh dear, please upgrade your SSL certs.

duskwuff• 11 years ago

This has nothing to do with the SSL certificate. The site is configured to use RC4, which used to be the recommended cipher to avoid certain attacks (particularly BEAST). However, RC4 has other weaknesses, and this is no longer recommended. Current versions of Firefox will actually refuse to negotiate an RC4 connection.

sp332• 11 years ago

Doesn't that mean that the certificate is using the outdated RC4 cipher, and should be updated?

miduil• 11 years ago

@brianwski (or any other backblaze volunteer) We still can't visit your website. https://news.ycombinator.com/item?id=8999036 It's been about a month ago we mentioned this issue.

ricardobeat2• 11 years ago

I really enjoy these posts, and would be a happy Backblaze customer if they didn't have this (unpublicized) policy of deleting external hard-drive backups after 30 days:

    Backblaze works best if you leave the external hard drive attached to
    your computer all the time. [...] If the drive is detached for more
    than 30 days, Backblaze interprets this as data that has been
    permanently deleted and securely deletes the copy from the
    Backblaze datacenter.

This probably reduces the total amount of storage used, making the $5/month economically viable. I'd happily pay more if it weren't for this though, currently paying ~$12/month for Bitcasa.

atYevP7• 11 years ago

Yev from Backblaze here -> we do have that on our website (https://www.backblaze.com/cloud-backup.html and https://www.backblaze.com/remote-backup-everything.html) and in some help articles but I hear you. It's a topic that comes up, but when we've done the calculations in the past, keeping a copy forever as an archive raises our costs to the point where we'd have to raise our prices and we're not too keen on that! We have notification emails so that we let you know ahead of time before data is removed though, so it's rarely become an issue in terms of "accidental deletions".

carterf• 11 years ago

Damn, I didn't know that either. Been a customer for years and wish I had seen that earlier. I've also had many lapses in time. As long as I'm paying why would you ever delete my data?

free652• 11 years ago

You don't have it here -> https://www.backblaze.com/cloud-backup.html. There is no mention of 30 days. I am so what pissed about since I bought the service just to backup my external drive.

DenisM• 11 years ago

So if I go on a 4-week vacation, I will return to an empty account? Uh oh. That should be printed in bold red letters everywhere. It's the opposite to peace of mind.

ricardobeat2• 11 years ago

Hi Yev. It is indeed on the website, but only in the second link you mentioned. I only found out after signing up for the service last year, and then decided to cancel. I wanted to backup photo collections that I might only access once a year - not very keen on having to do a maintenance scanning session once a month. Maybe you could come up with a separate plan, with limited storage, that solves this use? In any case, thanks for taking the time to reply.

ejdyksen2• 11 years ago

This is a deal-breaker for me. I have limited SSD space on my MBP, so my Lightroom catalog is on an external hard drive. I can easily go 30 days between plugging it in.

That drive contains, by a very wide margin, my most important and least replaceable files.

gry• 11 years ago

What is your backup solution?

nonissue• 11 years ago

It actually makes sense. If you think about it, if every external that anyone ever plugged in was backed up and then never seen again, that could be sooo much data. What if that drive was used, wiped, renamed and then used again? You potentially have the same drive on BB twice.

A way to alleviate the issue would be to charge a small fee per drive or per account for persistent external back ups. If it was 1-2$ a month extra for the peace of mind that a drive would always be backed up, I'd gladly pay it.

At the end of the day, you have a choice as a consumer and you went with bitcasa. I'm happy paying $5/m for unlimited backup for my main computer. All said and done it's an amazing deal, even when you consider the external drive issue.

matthewmacleod• 11 years ago

I'm a Backblaze user, and think this is a pretty reasonable policy — Backblaze is a backup system with a 30-day history, and not an archival or "unlimited online storage" system.

In addition, that policy seems to be pretty obvious. I definitely knew about it before I signed up, I'm sure it's signposted in the app itself, and I receive email notifications about a drive that's not plugged in.

All-in-all, seems pretty reasonable.

Veratyr• 11 years ago

I know you might not care about the security and you're only using them for the cheap storage but just keep in mind that Bitcasa aren't as secure as they claim they are as they're able to provide your data to third parties via both their web interface and their API, which means that they possess all decryption keys. i.e. the second section of the following quote from their 'Personal' page is false:

"Bitcasa keeps all of your files secure by applying client-side, AES-256 block-level encryption before they are even uploaded to your Personal Drive. Only you have access to your content. Not even Bitcasa employees can access or determine what files are in any given user’s account."

I was a customer of theirs when they first launched beta but ever since they've been screwing with their consumer customers with seemingly little thought to the consequences and their marketing copy is dangerously deceptive.

I've since moved to SpiderOak, who are quite transparent about their security and have updated their pricing to be more competitive.

Scarbutt• 11 years ago

So they don't offer a linux client to keep people away from running it on servers? ;)

grandinj• 11 years ago

Aaaargh. Yet another idiotic website that disables zoom gestures, making it unreadable on my iphone.

Backblaze Storage Pod 4.5 – Tweaking a Proven Design

Related Stories