Backblaze is now a Terraform provider

https://www.backblaze.com/blog/easy-storage-easy-provisioning-backblaze-is-now-a-terraform-provider/

by caution • 5 years ago

317 112 5 years ago

majormjr• 5 years ago

I was just looking at this new Terraform provider yesterday how timely. Nice to see the quickstart guide this will be helpful for managing the buckets and application keys.

atYevP5• 5 years ago

Yev from Backblaze here -> That's great! Glad the quick-start makes it simple.

vbsteven2• 5 years ago

Terraform is an amazing tool and it’s always great to see more infrastructure providers jumping on the bandwagon.

atYevP5• 5 years ago

Yev from Backblaze here -> Yea, it's pretty great! They're a great tool and we're glad to be involved!

OJFord• 5 years ago

But unfortunately plugins cannot provide remote backends, and it's a little clunky to use the S3 one:

https://github.com/hashicorp/terraform/issues/27304

javajosh• 5 years ago

It's interesting because it feels like a small turf war. The "Amazon Provider Team" wants to own the s3 provider backend, and you want to make it pluggable. Is this to avoid an explicit "b2" provider? I can see how it might be confusing to use terraform and see an s3 provider and then have it...not use s3. :)

ArchOversight• 5 years ago

Terraform doesn't allow any backends other than the ones built-in. So even if you wanted to provide a "b2" backend, it'd have to be built into to terraform.

Using the S3 backend with B2 should work fine since B2 makes a S3 compatible API available, but its made more difficult because the "Amazon Provider Team" is the one that maintains the "s3" backend for Terraform, and they want to do additional validation that matches AWS's expectations.

renewiltord2• 5 years ago

Very cool! Great job, guys.

Do you use a similar technique to https://poweruser.blog/embedding-python-in-go-338c0399f3d5 to embed the Python SDK?

Also, was there a reason it's not against the B2 API? Not a judgment, just curious about the design tradeoffs in a professional-talking-to-professional sense.

AYBABTME• 5 years ago

That design choice prevents their provider from working with official Terraform container images. They should make the API calls directly in Go, it's just weird to do it via Python... They have very few resources exposed on their API so it's not like writing a Go client would be all that hard.

renewiltord2• 5 years ago

Why would the official container images not work? The Python interpreter and libs are bundled into the (massively fat) Go plugin binary, right?

And appropriately linked?

mattjaynes4• 5 years ago

If you'll be using B2 for live production data, a couple points:

First, B2's pricing is pretty amazing, especially compared to S3 and similar competitors: https://www.backblaze.com/b2/cloud-storage-pricing.html

Second, be aware those savings come with some downsides. The major one for us has been their maintenance window every Thursday from 2:00-3:00pm Pacific Time. Usually there's no outage, but sometimes there is. There's no warning - it's just down sometimes during that window. So, if uptime is important for your data, consider the cost of also implementing a fallback solution to cover your production use during those maintenance windows. https://www.backblaze.com/scheduled-maintenance.html

tedmiston• 5 years ago

What a strange time for a company with lots of US customers to schedule a maintenance window.

Twisol• 5 years ago

I'm not so sure. It's better than a late night window in that it forces their customers to actually deal with the engineering window, rather than just cross their fingers.

johnjboren• 5 years ago

Don't push to production on a Friday, but Thursday at the latest, sure...

bretpiatt• 5 years ago

With the B2 target use case of backup and archive data I suspect it's actually a good time to do it for their customer base (and it also then happens to be good for their engineering team too, awake and alert!).

truetraveller3• 5 years ago

Ouch. Does B2 Cloud actually go down during this time. That's a major show-stopper. Was actually counting on B2. Not sure anymore.

mattjaynes4• 5 years ago

Not often, I only remember a few times in the last couple of years. If you're just hosting backups, you're unlikely to even notice. But if you're serving live production data from B2, it can bring down your whole service, which is quite painful especially if you have a large customer base.

atYevP5• 5 years ago

Yev from Backblaze here -> typically not - we've built most of our systems to be keep data up and flowing during those maintenance windows - if we anticipate longer windows or are doing things that can impact performance we typically announce it on our blog and twitter!

seekbeak2• 5 years ago

Same here. We're midway through a migration, and this has us re-thinking the whole move. I wonder if we had a duplicate copy of our data in the EU data center if we could do a fallback during US downtime, or if the entire 'cloud' goes down.

truetraveller3• 5 years ago

Side question: Apart from this maintenance window, is B2 Cloud reliable? I've heard of problems with the S3 API. Is the "native" API more stable? Would love to know your insight, it will potentially save me a lot of time!

mattjaynes4• 5 years ago

Have had live production data on there for a couple of years and it's been very solid outside the maintenance windows.

(The exception being the recent outages GoDaddy caused for them, but since they've moved to using Cloudflare as their registrar, I don't anticipate further issues there: https://news.ycombinator.com/item?id=26119619 )

tedivm3• 5 years ago

Several years ago Backblaze lost all of my wife's data. Their dashboard said it was all there, and we trusted their systems to be accurate. When attempting to download the data it turned out that none of it was there. When my wife contacted support they tried to blame her.

Obviously this was a few years ago, but a backup provider failing at their one job and then blaming the customer left a really bad impression that keeps me from using them.

Dylan168072• 5 years ago

I haven't used a it a lot but it's been a solid endpoint for backups.

I did some medium-intensity benchmarking a while back and decided not to put certain server data on it because I was getting a few 20+ second timeouts per thousand read requests. I can handle server errors, and I have retry logic, but this was something where I needed to be able to access the data within a second or two. Maybe it would have worked better if I set a very aggressive timeout, I'm not sure. Deeper testing is something I'll worry about some other time if the data actually grows past a couple hundred gigabytes.

This was mostly with the S3 API, I don't remember if I ever succeeded in getting the program to use the native one.

preommr3• 5 years ago

Digital Ocean has spaces with unlimited uploads, 250gB + (0.02/gb), 1TB of outbound transfer all at $5/mo.

That seems way cheaper than this.

For backups and large, long term storage, AWS has Glacier, that's really really cheap.

jjeaff• 5 years ago

$0.02/gb compared to $0.005 with b2. So not really in the same ballpark as far as price.

dr-smooth• 5 years ago

Wait until you see how much it costs to retrieve your entire backup from Glacier.

ddorian43• 5 years ago

If you ever need those backups, your company will fail from the receipt to receive them.

For Digital Ocean, please look that their pricing is higher in both bandwidth & storage.

caymanjim• 5 years ago

I'm a big fan of Digital Ocean and run a bunch of droplets. B2 is way cheaper than Spaces for storage (1/4 the price). I tried using Spaces anyway, because I wanted something with faster throughput for streaming video, but Spaces was even slower than B2, even within the same Digital Ocean datacenter. All these S3 clone storage systems are clearly throttled, and there seems to be at least soft collusion to keep the bandwidth about the same between them, and just enough to prevent video streaming. I'll go sit in the corner and adjust my tinfoil hat now.

killingtime74• 5 years ago

Linode has the same and allows using bandwidth pooled across all instances

mekster• 5 years ago

I wonder if this still applies about performance comparison.

https://github.com/gilbertchen/cloud-storage-comparison

mike503• 5 years ago

Spaces was not fully compatible with S3 at one point. It was nearly impossible to download your entire bucket when it was huge. Rclone was able to, but it was horribly slow. AWS CLI would only grab up to 1000 items. It seems like they did finally fix that though.

I’m actually a huge fan of Bunny now. The CDN piece is about as cheap as it gets (for any utility based service), it’s optimizer and other things work well, and it works seamlessly with their storage system too. Which is super cheap itself, allows you to control how much it’s replicated (and where) - just waiting for them to deliver S3 compatibility so all the existing tools that exist work, or some other type of CLI tool.

qw3rty012• 5 years ago

Also keep in mind all the data is in a single datacenter, so if something like OVH happens to backblaze, you'll probably lose the data

I mainly only use them for backups instead of production data

EDIT: As of 2019 there are multiple datacenters, but it doesn't seem like the data is stored redundantly across them

jiofih4• 5 years ago

They have four datacenters, three in the US and one in the EU. Details are not given regarding how the 20 shards that comprise your data are distributed geographically, but they state eight 9s of reliability.

dx0345• 5 years ago

The EU and US data centers are completely separated. You can't even use both from the same account. To change, you need to open a completely new account selecting EU at the start. So it's not really easy to use both at the same time and there's definitely no redundancy across data centers if you're a customer in the EU.

vishnumohandas• 5 years ago

B2 claims to offer 99.9% uptime[1], and seems to have SLAs[2] similar to that of AWS S3[3].

[1] https://www.backblaze.com/blog/cloud-storage-durability-vs-a...

[2] https://www.backblaze.com/company/sla.html

[3] https://aws.amazon.com/s3/sla/

mattjaynes4• 5 years ago

In practical terms, 99.9% uptime means no more than ~44 minutes per month of downtime (~9 hours per year).

busymom0• 5 years ago

A limitation I ran across when using B2 was that their pre-defined url generation doesn't allow you to set file-size limits nor does it allow you to set the file name in the pre-defined url. It simply gives you a url to upload it to. So if you are using b2 for storage for lets say image uploads from browser, some malicious user has the ability to modify the network request with whatever file name or file size they want. Next thing you know, you have a 5gb sized image uploads happening....

This pretty much prevents me from using B2 for now.

SmellTheGlove• 5 years ago

I ran into the same limitation! IIRC, there also wasn't a way to expire a signed upload URL sooner than whatever the default was, which was hours or maybe a day. I had the exact use case you mentioned, too - image uploads bypassing my backend server. I didn't want the generation of a signed url to, say, upload a profile photo, give carte blanche to create a hidden image host when combined with the limitation that you highlighted. All sorts of bad things could come of that. I ended up just going back to S3 - costs more, but still worth it.

fukmbas• 5 years ago

Lol who has a maintenance window in the middle of the day?

nikisweeting• 5 years ago

I love B2's free ingress and egress when you use CloudFlare, they utterly destroy the competition on price.

But my true dream would be for backblaze to someday offer ZFS as a service.

I want to `zfs send -i my_pool@2020-03-11 | b2zfs recv some_bucket_id`, then be able to view my snapshots and files within in the backblaze web UI, and restore with `b2zfs send -i some_bucket_id/my_pool@2020-03-11 | zfs recv my_pool`.

You can already mount B2 as a FUSE filesystem with something like ExpanDrive, then write ZFS raw file vdevs to the B2 FUSE mount, but it's horrifically slow and probably too janky for any real use.

ptomato• 5 years ago

rsync.net's zfs as a service is really great for that, though a bit more expensive than b2 is.

bpye• 5 years ago

Rsync.net looks great but the storage cost is 5x B2 unfortunately

EDIT: As mentioned below this is for the most expensive (lowest capacity) tier. I’d been comparing for my own home use and so I would be unlikely to exceed 10TB but if you’re looking at higher capacity then maybe the calculus is different.

kgog• 5 years ago

maybe you need borg with borgbase.

mhio• 5 years ago

Without b2 adding anything zfs specific, that should be possible with a regular large file upload if you can deal with each upload part being buffered in a file/memory locally before upload.

rattray2• 5 years ago

I sure wish this post had included a few code snippets, just to make it extra-clear what using Backblaze through Terraform might look like.

vbsteven2• 5 years ago

Inside the article is a section "How to get started using Backblaze B2 in Terraform" with a link [1] to the getting started guide which should have everything you need.

I can see why they put it on a separate page to not clutter up the article.

[1] https://help.backblaze.com/hc/en-us/articles/1260803375989

rattray2• 5 years ago

Nice, thanks!

I was less interested in the code sample as a "how-to" and more for skimmability – rather than reading a bunch of words I just wanted to see what it would look like.

Eg, from that link:

    terraform {
      required_version = ">= 0.13"
      required_providers {
        b2 = {
          source  = "Backblaze/b2"
          version = "~> 0.2"
        }
      }
    }

    provider "b2" {
    }

    resource "b2_application_key" "example" {
      key_name     = "test-b2-tfp-0000000000000000000"
      capabilities = ["readFiles"]
    }

    data "b2_application_key" "example" {
      key_name = b2_application_key.example.key_name
    }

    output "application_key" {
      value = data.b2_application_key.example
    }

snicker7• 5 years ago

Terraform is great, but I wish more cloud providers had Guix/Nix integration.

CameronNemo3• 5 years ago

Are there any providers that to have such integration? I have only lightly used nix. Having a hard time understanding what a cloud provider integration would look like.

ethnt• 5 years ago

You could use NixOps[0] for Nix but I'm not sure you can directly compare Terraform and Guix/Nix? My set up involves Terraform for infrastructure and Nix for provisioning, and it's working for me so far.

[0] https://github.com/NixOS/nixops

emptysongglass• 5 years ago

You can deploy and manage NixOS on Big Cloud using Terraform: https://nix.dev/tutorials/deploying-nixos-using-terraform.ht...

aseipp• 5 years ago

I think this too, but at the same time the reality is a tool like Terraform is really complicated to implement well, and, importantly, it always has to work. All the time. There are really high standards for this, and in my personal experience, alternative solutions like NixOps don't quite stack up in reliability or broad utility versus Terraform. The design is good in theory, but it needs just a huge amount of work to be trusted.

For the most part, I provision things with Terraform and then instantiate the servers with NixOS/Nix itself, and this mostly works. For bonus points you can use Nix to generate the HCL that Terraform reads in (because Nix can write JSON, and HCL is just JSON in a trenchcoat) if you want to put some veneer on it.

1vuio0pswjnm7• 5 years ago

For large downloads, does BB support the Range header. If the user is on a connection that that is not suitable for long downloads, could the Range header be used to download a large file in several parts.

0xbkt• 5 years ago

They support byte serving.

satyrnein• 5 years ago

Anyone have any experience with the Backblaze compute partners? I want to put data up on B2, but then I still need to do things with it.

philsnow• 5 years ago

The wording is a little clunky; the "is" in "Backblaze is now a Terraform provider" makes me think that BB is competing with Terraform Cloud.

"Backblaze now has a Terraform provider" or "Backblaze released a Terraform provider" makes more sense to me.

solatic• 5 years ago

Terraform is an open-source infrastructure as code (IaC) tool

Maybe it was in the beginning, but Terraform is far more powerful than that now. Terraform is a monad that neatly separates pure declarative configuration from the I/O (side effects) that are factored out into providers. Terraform used at its most powerful is not limited to infrastructure, it also sets up the platforms and applications running on that infrastructure for you, by separating the configuration of the platforms and applications from the generic API calls that apply that configuration. Terraform's dependency graph ensures that the calls are made in the right order, no matter if they are made to infrastructure APIs, platform APIs, or APIs belonging to layers further up the stack

nhoughto• 5 years ago

Popped in to say backblaze built out a tf provider at our request and they were great about it! Got a quick early build out to test, GA build a few months later and are very responsive to feedback. Pleasure to deal with

atYevP5• 5 years ago

Yev from Backblaze here -> That's really nice of you, glad we could work together!

Backblaze is now a Terraform provider

Related Stories