Borgbase backups have been unavailable for 3 days

https://status.borgbase.com/status/default

by smcleod • 3 years ago

105 85 3 years ago

status.borgbase.com

smcleod2• 3 years ago

Their current ETA for service restoration is another 2-4 days.

Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%.

They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture.

Opshugs to the folks at Borgbase.

box-us10 offline for storage expansion
We are currently experiencing a temporary outage on our box-us10 server due to an unplanned expansion process. We added two new hard drives to the server in an effort to enhance capacity and accommodate future growth. However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.
To prioritize the safety and integrity of your existing data, we have decided to keep the box-us10 server offline until the expansion is successfully completed. It's estimated that about a week of downtime will be needed for the expansion. We understand the inconvenience this may cause and sincerely apologize for any disruption to your services.
Current progress: 26% (Sunday morning UTC) Date Created: 2023-08-11 16:17:36 (3 days ago) Last Updated: 2023-08-13 18:15:53 (15 hours ago)

CamperBob2• 3 years ago

unplanned expansion process

Sounds like an Elon Musk euphemism after a bad day at SpaceX.

bloopernova4• 3 years ago

Someone added disk to an array without understanding what would happen next.

x0x03• 3 years ago

me wonder about their redundancy and recovery architecture

You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

dikei2• 3 years ago

Yeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.

TylerE• 3 years ago

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

jm4• 3 years ago

They messed up and either ran out of disk space or had an array fail. Everything in that statement reads like they are trying to frame a critical production failure as a glitch that occurred during maintenance. This wasn’t about a routine storage expansion. It was an emergency caused by some operational screwup and the only way to fix it is to rebuild the array with new disks.

viraptor2• 3 years ago

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

boring_twenties• 3 years ago

In a storage business no less. They literally had one job: to know how to resize arrays, so the rest of us don't have to.

shrubble• 3 years ago

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

colechristensen3• 3 years ago

It would appear to be something like that.

However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in place to to dry runs in non production or any of the other things that would protect against something like this. Really questionable to trust an organization that makes a mistake like this.

kiririn• 3 years ago

A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given

smcleod2• 3 years ago

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...).

However I agree that it doesn't look good for a storage company.

TheNewsIsHere• 3 years ago

I agree with you.

I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation?

I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming project.

sparrish• 3 years ago

And it's a practical guarantee that something like this will never happen again. The wise man learns from his mistakes.

bartvk• 3 years ago

A white lie? Like what? How would you word it?

You mean to downplay what actually happened?

ddtaylor• 3 years ago

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with.

First I have a local copy of my data living on my actual hard drive.

Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has about 200GB of storage for these backups for about $1/mo added cost.

Next I use the Backblaze Cloud tool to synchronize those encrypted backups to B2 using `b2 sync --delete` command. It runs automatically via cron every night. The costs here are about $0.01 per month since I only get charged what I actually use.

Backups are pruned and I can easily control the schedule, copies, etc. If I need to recover a file it mounts as a file system that I can easily navigate using any tools I want, including the command line. I can also mount older or different snapshots.

willbush• 3 years ago

I do something similar, but with restic. I wrote about it here: https://willbush.dev/blog/synology-nas-backup/ if anyone is interested.

dikei2• 3 years ago

Similarly, for my Linux devices I use `borg` to backup to a local NAS, and use `kopia` to have another backup to Google Cloud Storage, as B2 too slow where I live.

I used to have back up on local external drives too, but stopped doing that, since the process was manual and I often forgot about it.

rexreed• 3 years ago

Which cloud server do you use for your backups for the $1/mo in added costs?

somishere• 3 years ago

I've a similar local / S3 (compat) strategy but use Kopia. I currently only backup remotely to B2 (risk taker!), but could easily and cheaply add redundancy in here .. e.g S3 / R2 / random hosting. It's very cheap ~US$1. UI and default strategies are perfect for what's required. Completely automatic unless I need to browse some files in which case I just open the Kopia UI. I haven't used it, but understand Kopia also supports rsync.

Edit: previous comment on same (not a shill, I swear!): https://news.ycombinator.com/item?id=34152369

bloopernova4• 3 years ago

My borgbase backups have been great, although according to my dashboard I'm on a different server instance.

I don't envy their sysadmin team right now. Good luck, folks.

bomewish• 3 years ago

isn't it one guy??

chanandler_bong2• 3 years ago

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying.

I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

exhilaration• 3 years ago

automated test restore failed

You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

semi-extrinsic• 3 years ago

Not OP, but I would guess it's something like this:

  1. Make a e.g. 30MB file of random data  
  2. Copy it to "_reference" file  
  3. Upload the file to backup service  
  4. Restore the file from backup service  
  5. Diff restored file against reference

adobrawy2• 3 years ago

I don't use borg, but I used duplicity, which offers something like that. The verify operation simulates a backup restore to compare whether the restored file's checksum matches that expected from the metadata and optionally against the local file. I use this routinely, interesting to see that a local S3 provider can sometimes mess up your files silently.

chanandler_bong2• 3 years ago

Pretty simple, really.

Pick a couple random files that should be in the repo, restore them from a random archive, check the md5sums against the source. If the md5sums don't match (or the file can't be found), something is wrong. I am mainly backing up RAW image files, so they should never change.

Basically...

$TEST_FILE=$(ls -p /source_dir | grep -v / | shuf -n1)

$TEST_ARCHIVE=$(borgmatic -c config.file list | shuf -n1)

borgmatic extract yada yada yada

md5sum $TEST_FILE restored_file

ryan292• 3 years ago

I got an email about it right away and I’ve also been getting warnings (that I configured in the dashboard) for inactivity on a repo that’s affected.

kiwicopple• 3 years ago

(off topic) I promised to reply to you in another thread, but I can't because replies are locked. feel free to reach out to me if you want - my contact is in my profile

jnsaff22• 3 years ago

It looks to me that their US10 is not like an AZ but an actual server with bunch of HBA and disks. So very much pets and not only in a single location but possibly in a single rack or even box.

You are (maybe) protected against a few disk failures but that's about it.

This FAQ entry seems to confirm this: https://docs.borgbase.com/faq/#which-storage-backend-are-you...

muppetman• 3 years ago

I've loved BorgBase up until this point. I mean at least they emailed me quickly to tell me, but now I don't have backups. Well that's not true, I backup to a number of locations and BorgBase is just one of them. But it's a chink of my redundancy lost and a huge outage like this for a week makes me wonder what I'm paying for. What if BorgBase WAS my only backup solution and I needed to roll back?

Everyone makes mistakes and I appreciate they were very upfront about it, but yea. Questioning if I'll be an ongoing customer now.

countula• 3 years ago

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development.

rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653

Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup.

Any one have any experience with Hetzner Storage Boxes for this?

exitheone• 3 years ago

Been using hetzner storage boxes for borg backup for >5 years. Works without issue for multiple TB of backups. Every maintenance downtime so far has been announced in advance.

AnonHP• 3 years ago

The main reason I’ve avoided Hetzner storage box is because they don’t accept loading money into the account as a prepaid credit for a longer period. It’s always a month-by-month payment. Ok, so they do allow it, but only through a bank transfer, which is either not easy or possible for those outside the EU. If they allowed this through credit/debit cards or PayPal or some other mechanism, I’d surely try it. Hetzner in general seems to be very conservative in handling (and avoiding) risks.

I’d like to prepay for a year or longer for critical services so that in case something goes wrong (card expired, didn’t notice reminder emails, temporarily off the grid, temporarily incapacitated), things just don’t go poof.

lifeofguenter• 3 years ago

Hetzner Storage just lost a couple of snapshots a while ago…

If you want “perfect architecture” and reliability/availability you just need to pay for it (aws/s3). Or make use of multiple providers.

noAnswer2• 3 years ago

If you want “perfect architecture” and reliability/availability you just need to pay for it (aws/s3).

I find that really funny. AWS has outages regularly. Whole data centers who go dark. Remember the S3 outage because an admin fat-fingerly deleted a large number of servers? ...no, people forget very quickly because of the "No one was ever fired for using AWS" mantra.

ryan292• 3 years ago

Borgbase is pretty good. I’ve used it for 6 years.

The usage graph on each repo is great. I redid my server this year and was accidentally backing up a stale snapshot every night for one of my repos. It would succeed, but without changes. The flat line for usage tipped me off. I’m not sure I would have noticed otherwise since it was a VM image that I could restore and boot, but it wasn’t super obvious it was full of stale data.

This downtime isn’t ideal, but, if downtime is needed to preserve data integrity, I’ll take it over the risk of trying to maintain availability.

rvieira• 3 years ago

I've tried Borg but I settled with Kopia and a[0] cheap S3-like storage.

Everything is automated and runs on a schedule, but when I need some ad-hoc restore, say, of a single snapshot or folder I can always use KopiaUI.

Been quite happy with it.

[0] - actually two providers, for redundancy, plus local physical storage

wiredfool2• 3 years ago

I’ve had a couple of hetzner storage boxes- generally they’re ok.

Though just this weekend there was some screwiness with sub accounts and ssh keys not working that did resolve itself after 24-48 hours. I didn’t notice anything communicated about it.

rsync2• 3 years ago

"rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653"

This is correct - we believed we had a working recipe that properly sandboxed the 'rclone mount' and 'rclone serve' directives but we couldn't make it work in a way that allowed us to sleep at night.

But now that is all changing...

It appears that we can run rclone serve restic --stdio in a way that doesn't actually serve anything or create sockets, etc., and as soon as we finish our tests we will have it in place and people can lock their accounts with something like this:

restrict,command="rclone serve restic --stdio --append-only backups/my-restic-repo" ssh-rsa ...

... in authorized_keys and achieve proper append-only mode at long last.

Stand by ...

btschaegg• 3 years ago

As a customer, that is great to hear (well, read). I was reading about how a restore scenario with Borg would look like if "append only" really matters (so in a ransomware scenario), and what I've seen makes me rather uncomfortable [1]. I don't want to have to do that in a stressful situation.

Is there a particular channel where you intend to post your findings? :)

[1] https://borgbackup.readthedocs.io/en/stable/usage/notes.html...

alex3305• 3 years ago

I used their trial for a bit to test it out with Vorta [1] in a container. Vorta (and Borg) seemed to work fine, until I wanted to restore an archive and I noticed that my recent snapshots were completely empty. Probably because of a misconfiguration on my end though. But it made me look elsewhere. For me backups should be a fire, test and forget solution.

Recently I made the switch to Kopia [2] which seems to have feature parity with Borg (and Restic [3]). It also has a web UI which is way easier to work with than Vorta. And I can easily view, extract and restore individual files or folders from there. This gave me way more confidence about this solution. The only thing I really miss is that I cannot chose different targets for different paths. For instance, with Borg I was able to backup a partial of my Docker appdata to an external source. And I haven't found a way to do this with Kopia. Besides that I'm pretty happy with this solution and I would recommend it.

1. https://vorta.borgbase.com/

2. https://kopia.io/

3. https://restic.net/

Borgbase backups have been unavailable for 3 days

Related Stories