brianwski

Disclaimer: I work at Backblaze.

Do we know if these drives are the same as you would purchase in a retail outlet

Seagate (for instance) won't sell ANYBODY drives directly, so they force us to get bids from various resellers and distributors. So when we pick the lowest price, I don't think Seagate knows who the drives are going to, but there might be a trick in there somewhere I am not aware of.

Disclaimer: I work at Backblaze, and I was here first.

I'd worry they would sue for...

If you are referring to Backblaze, we're not going to "sue" anybody for anything.

We (Backblaze) have dealt with a bunch of frivolous lawsuits (and patent trolls) over the years suing us, and OMG we're not going to instigate any lawsuits over some honest person legitimately reporting some issue and being helpful. It isn't going to happen.

Our reputation is important to us. Not just for Backblaze: I'm saying the individuals that founded Backblaze and those people that work now here base our entire existence and careers and the number one marketing efforts at Backblaze are based around we are trying to be "the good guys" and transparent and acting like it. There is no possibly world where we try to suppress a screwup like this through legal means. That would be a PR debacle of epic proportions.

If something went wrong, let's shine a spotlight on that cockroach and figure it out together. I'm not sure the exact drive we are all talking about, but my first guess would be a customer ordered a $189 "USB Restore" all their data shipped to them on an encrypted drive) and we (Backblaze) shipped the customer a USB restore drive and they are subsequently selling it (after copying their restore off of it) on the open market. If it is above 8 TBytes this is absolutely *NOT* the case and we should get to the bottom of it. Without lawyers mucking up the situation.

I doubt I'd get anywhere close to ROI with SSDs

I think most people assume the SSDs are more expensive, but depending on their application they choose SSDs for the OTHER properties, like SSDs are quieter, and faster, and like you mention use less power.

There was this narrative for a long time that SSDs failed way more often, or beating them up with reads and writes would make them fail while hard drives could take the same number of reads and writes and not fail. So a subset of people chose hard drives for durability regardless of cost. Personally I never saw any evidence of SSDs failing more often than hard drives, and PLENTY of counter evidence. For instance, I wrote a program that unintentionally overwrote the same fixed length 16 byte file in the same location over and over again too often, and it failed more on drives than SSDs. (The failure mode was not whole disk failure, the 16 byte file itself mysteriously stopped being writable on some computers after a long time of working.) I had some friends that worked at Tivo and they said they had the same bug/shortcoming early on. The Tivo would overwrite the same section of disk over and over and it failed LONG before the drive manufacturers said it would fail. The drive manufacturer said they needed to vary the location of where they wrote the data to get anywhere near the published numbers.

I had assumed the file system layer would take care of that kind of management for me, but I guess not so much.

Capitalism does not reward hard work. Smart work, maybe, but not hard work.

I'd say a person successful/rewarded in a capitalistic society (defined by accruing more wealth than they started with and not just spending down some large family inheritance they were born into) probably has some combination of: luck, smart (intelligent) decisions, hard word, probably has a good "setup" going into the working world like parents didn't abuse them growing up and at least helped pay for their education, etc. Maybe that last one is just a subset of luck, we certainly do not choose our parents.

Sure, you can be missing one of the above list and do Ok, but probably not two. Statistically a smart, hard working person who had the world's crappiest upbringing probably won't be that successful compared with somebody who had all three going for them. But if you look at 20 "success stories" it is extremely rare the person didn't work hard. I think it's an important component of being rewarded in capitalism. It does a disservice to people to say "slacking off or working hard results in identical outcomes".

To support at least half of your point, you cannot be stupid and work hard and be successful. But a smart person who works hard will most likely do better than a smart person who doesn't work hard.

Someone with some time could correlate SMART data to failure rate.

We looked into it a little, some notes written up here: https://www.backblaze.com/blog/what-smart-stats-indicate-har...

Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so....

This guy wrote a paper based on the Backblaze SMART data: https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?acc...

These 5 guys wrote another paper based on the Backblaze SMART data to train up a Bayesian network to predict failures: https://ieeexplore.ieee.org/document/8489097

This is another article of predicting hard drive failures using the Backblaze SMART data: https://karthikna.github.io/Prediction-of-Hard-Drive-Failure...

I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.

Disclaimer: I work at Backblaze but more on the backup client side that runs on laptops, not in the datacenter storage side.

Is there a failure list for write heavy drives?

To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter.

Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read.

Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.

Disclaimer: I work at Backblaze. Here is a histogram of backup sizes of our Backblaze Personal Backup Customers if it is useful:

https://i.imgur.com/GiHhrDo.gif

You have to zoom in to see the meaningful information. It can be super surprising to developers and IT people (like us) to see a lot of customers have less data than you might think.

When we started we had no idea what might come of all of this, so it was "stressful". :-) So I welcome another backup client developer to our club, and hope this can help them out.

Disclaimer: I'm the engineer you are commenting on, I just want to straighten out one mis-understanding.

This quote especially seemed childish and immature to me:

> "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"

No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested

Several times you seem to jump to the assumption that I don't understand fsync and disk flushing and that is the core issue. You aren't understanding what I'm criticizing. Here is an example of what bothers me:

You take a picture at your wedding, and you store it on your hard drive. You like the photo, it means a lot to you, and you use it as the background for your desktop FOR FIVE YEARS. You have rebooted hundreds of times, and it's always the background for your desktop. Then one day 5 years after your wedding, you reboot your laptop, and it seems to take a little longer to boot, and then after you sign into the laptop half the image you use for your desktop background is scrambled. The middle of it looks like dirty snow. You didn't get any reports of any issues from the OS manufacturer, but now one of your photos is corrupted.

This isn't because the software that wrote the photo 5 years earlier forgot to flush the picture to disk. It just isn't. Behind the scenes, as your laptop was booting from an aging drive, it probably encountered some issue, and it went about fixing the problem as best it could - which I have no problem with. My issue is the drive lost some data, and if the OS manufacturer would let you know this occurred you could take useful actions like order a new drive, prepare a restore from a few weeks ago before that issue occurred, etc.

No, this is NOT Microsoft's or Apple's fault.

It isn't their fault that the drive is going bad, I agree. Drives go bad, that's why we have backups. My issue is the OS manufacturer try to cover up too much, keep too much hidden from the user, and didn't let the user know data loss has occurred (or might have occurred). And yes, I hold them accountable for not telling customers what is going on. It isn't anybody's "fault" that it occurred, but there is a responsibility to let customers know about it so the customer can take appropriate actions so they don't lose data (or more data).

I try to write incredibly paranoid software. Part of the reason is that is the "average" environment the Backblaze client runs in is more unstable than what most software developers are used to. The whole point of backups is to run when the computer is going sideways, it has bad RAM, it's losing disk sectors, or a customer's cat likes sleeping on the keyboard because it's warm, and the fans are clogged with cat fur. And because the family has teenage children that don't know about computer security problems, they download and install unstable junk from all over the internet because why not? That's the environment my software runs in, and I take my job of trying to protect my customer's data very seriously.

Disclaimer: I work at Backblaze so I'm biased and you should keep me honest.

if you run into big trouble, you can go get the box back for a really fast restore

Backblaze provides this service for our customers! Customers can ask for an 8 TByte encrypted USB attached external hard drive to be prepared in the Backblaze datacenter with all their data beautifully restored on it, then we FedEx this anywhere in the world to them. This is a free service as long as the customer returns the USB drive to Backblaze in a reasonable amount of time (a couple months, and we can work with customers if they need longer). Or customers can pay $189 (which includes the drive, the data, and world wide shipping) and keep the 8 TByte USB drive.

You can read more about the USB hard drive restores here: https://www.backblaze.com/blog/usb-hard-drive-restore/ and why it is a free service here: https://help.backblaze.com/hc/en-us/articles/217665948-Resto...

What we often see is that if a customer's laptop is stolen or crashes, they sign into the Backblaze website to download the 3 or 4 individual files they were working on when the laptop was stolen. Let's say that is a term paper due the next day. That way they are back up and running IN SECONDS. Then the customer orders a free USB drive with 8 TBytes of their data which will show up in a few days. They can live without their wedding photos and their music for 3 days, but that term paper has to be handed in.

useful to add another column for "label",

Yeah, that would work really well. When you look at the "services" control panel in Windows, there are two columns. One column is "Name" of the service, and another column is a longer explanation with the column header "Description". I put a small description in there for bzserv (our service) plus a URL to our company website. I think this is just being polite, customers who don't recognize what "bzserv" is can immediate find more information on it.

and congrats on the IPO

Thanks, that was super exiting for us. After 14 years, I claim (and this is controversial) that we're no longer a startup and now we're just a mid-sized publicly traded company. :-)

process-based firewalls like Little Snitch, because each copy needs to be configured separately

Yeah, that was actually a surprise and unfortunate. What the Mac architect (one of my business partners) and I think is that now that it is nice and stable, we might go down to 1 or 2 bztrans_thread executables, and one bztransmit. That seems like a better tradeoff where we waste much MUCH less disk space, and it is only 3 executables to allowlist in Little Snitch, and it achieves basically what we want now that it's stable and working well.

Originally there were 10 threads MAXIMUM, and we made 10 copies. And each copy was linking with shared libaries so it was only 10 MBytes of disk space which nobody noticed. Then Windows lost their friggin' minds with one of their releases and forced us to link statically which bloated it way up to 5 or 10 MBytes per executable. Then we went to 20 threads maximum and the whole thing was silly. When we went to 100 threads maximum we said "enough" and went to mod 20 for re-using executable names.

By the way, ALL OF THIS could be avoided if Microsoft and Apple provided an API to set the name displayed in Task Manager/Activity Monitor. Maybe that's a security issue, I don't know. But frankly wouldn't it be SUPER TOTALLY USEFUL if chrome displayed the current web page loaded in the process name of each and every chrome process? Then you would know which one to kill when something goes sideways.

Symlinks have been supported since Vista

I'm not aware of even Windows 10 or Windows 11 being able to create Symlinks on FAT32 and exFAT, but I could be wrong? But it doesn't matter, the code as written is cross platform, it works on Apple Journaled File Systems, APFS, exFAT, etc. We then compile it for Macintosh, compile it for Windows, and compile it for Linux.

We can then spend extra time and carefully detect each filesystem and each platform and then make the optimization if we can. And this is a valid criticism that we have not done this yet. But no matter what we need this general code that will always work FIRST, what the links are is a space optimization to save valuable SSD space when it is possible.

> Backblaze ships with 21 identical copies of the same executable

This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.

I'm the programmer at Backblaze that made the copies on purpose, I wrote some extra code to do this, and it's meant to help us debug certain things. Yes they are identical, the installer only ships with one copy of the executable, the installer then makes the copies on purpose. I get to explain this from time to time. :-)

In Windows when you want to know what is going on behind the scenes, you can bring up Task Manager and look at the different names of the different processes that are running. On the Macintosh this is called Activity Monitor, same sort of thing. The different names for the executables are for different "threads" which have different roles. Backblaze is multi-threaded to get higher performance.

The parent coordination process is called "bztransmit". But when doing the actual transmission it spawns the bztrans_thread01, bztrans_thread02, bztrans_thread03, etc.

So BEFORE I made multiple copies of the executables with different names, a customer would say "bztransmit is hung" or "bztransmit is using up too much memory". There was very little visibility into this. But now that I made multiple copies with different executable names, when the same customer says "bztrans_thread03 is hung". Or they say "bztransmit is using too much memory". We immediately have narrowed down what to look at.

Here is a screenshot showing what "Chrome" looks like to me in Windows, and how it compares to how Backblaze's bztransmit looks like to me in Windows: https://i.imgur.com/KOJHJ9Q.jpg In that screenshot, you can see there is the "main thread", and "worker threads". Meanwhile chrome is just one big list of processes all named the same thing (see the screenshot). I prefer the Backblaze system, but I understand it upsets some customers that prefer the chrome experience.

That's it. It's not some huge mystery.

One question asked here was do we know you can launch the same executable twice? Yes, and we do that. The bztrans_thread05 is launched for thread 05, thread 25, thread 45, thread 65, etc. It's THREAD_NUMBER mod 20. Here is what it looks like to hit 500 Mbits/sec upload speeds, this isn't photoshopped, it's a real screenshot on my development computer: https://i.imgur.com/hthLZvZ.gif

Another question is: why not use hard links or symbolic links? That's the only real optimization possible here, everything else was on purpose. The answer is not an excuse, it's just an explanation if you are curious. The software we develop at Backblaze is cross platform, so what we like to do is make the most general form first that will always work, then if customers complain or we want to refine it we special purpose code in per file system or per platform. The most general thing to do is make full copies. We could then go on to make links on the Mac WHEN POSSIBLE and the equivalent on Windows WHEN POSSIBLE, but it never became a large priority. The reason I can't use one technology is we support several file systems, and not all of them are the same or support the same technology for links.

Every feature we have is the result of prioritizing it over working on other things. Until recently, we did not have a lot of funding or an infinite supply of programmers, so we had to choose what order to implement each feature in. I'm not saying we got all the priorities correct, or that we did things in the correct order. I don't really even think there is one correct order. For example, some individual home user customers prefer saving 180 MBytes of their valuable boot SSD space over me implementing single sign on for our corporate customers. On the other hand some corporate customers DEMANDED single sign on or they wouldn't purchase the product at all. They are both correct, but there is only one of me, so we made some judgement calls and left the multiple copies and worked on single sign on. Some customers were happy, some are miserable.

We do have open client recs for both Windows and Mac programmers, so if you would like to make a good salary, full benefits, and help us out, come join us! :-)

... 4 - 6 words ... very conservatively 160 characters is probably going to fit

Gotcha. I'll file a Jira ticket to see if we can easily and quickly raise it to 200 characters (round number). I think we can handle that easily and safely.

standalone hardware token support (like Yubikeys or Nitrokeys).

I'll run the idea by a few groups and see what people say. For feature requests NOT related to security, traditionally it's kind of a customer voting system. If enough customers want a certain feature it sorts to the top of the priority list and gets done sooner.

For security things it is obviously different, like if a big exploit like Heartbleed is released in the world we drop everything else and focus on that.

This is sort of somewhere in the middle, it's security related so it gets some additional boost up the priority list for that, but it's also a security feature not all customers would choose to use, so it is still affected by whether our sales and support teams are hearing requests for it.

Love his web page on dutch bikes in Amsterdam.

Ha! That's my half second of viral fame right there.

You have no idea how blindly I stumbled into making that webpage. I haven't ridden bicycles more than a handful of times since college, and something possessed me to start taking photo after photo of bicycles without much thought. I was just travelling home from Europe to the USA with a short layover in Amsterdam when I took them. When I got home I stared at this unorganized pile of photos of bicycles and almost deleted them. I honestly thought nobody would ever, in a million years, care about a webpage about bicycles. I blasted that web page out in one continuous stream of tongue in cheek caffeine fueled frenzy, threw it up live, and went to bed.

About 3 or 4 months later my website couldn't serve enough pages to keep up with the load. I called my ISP and they said, "looks legit", I said "not possible, my website is at the ass end of the internet, nobody ever goes there". I bought an analytics package from them just to prove them wrong. All the traffic was to one web page: "Amsterdam Bicycles". A page about bicycles. (sigh)

Totally by accident, without intending to or knowing what I was doing, I stumbled upon several groups of people that are just PASSIONATE about bicycles. First of all, unknown to me there are people that are really, really into bicycles. They have mailing lists, they have websites. Who would have known?

But the group that just floored me was the Dutch people themselves. They are beyond passionate about bicycles, and there is a rich and amazing history behind it. For example, in World War 2 the Germans confiscated all the Dutch bicycles, and there is literally still a saying (in Dutch) that loosely translates to "I want my bicycle back" as an inside joke about still being resentful about this affront. And overlay that with the Dutch actually seem to have a great sense of humor and sense of self, and don't mind it if somebody pokes a little fun at them and they laugh right along. I wasn't disrespectful and mostly just expressed confusion over the situation which was completely honest at the time. So the city of Amsterdam sent me a T-shirt, a bicycle bell, and a bicycle light, and clippings from local newspapers about my silly web page. They were just so over the top great about it.

And the best part was the people from the photos reached out to me, like the woman pedaling here: https://www.ski-epic.com/amsterdam_bicycles/pp9b_amsterdam_b... contacted me. In the caption for that photo I asked "Why is the guy riding, and the woman pedaling?" The answer is this: the owner of the bicycle always pedals. OF COURSE I offered to take the photo down if she wanted and she said, "Are you kidding, I love it!"

Well, I'd rather be lucky than good.

Is there a reason there’s a short limit on how long my Backblaze password can be?

The current limit is 50 characters. There are reasons we don't let it get super long like 2,000 characters but we could easily double or triple the maximum length it if it was requested. Is there a number of characters you have in mind? I found a chart here: https://i.imgur.com/trR2u8g.jpg that says a randomly generated 17 character password would take 93 trillion years to crack. The chart doesn't even go up to 50 characters. Backblaze also supports 2-factor authentication via SMS or Google Authenticator codes which personally I would highly recommend to customers also.

who knows what kind of encryption you guys are using

Backblaze has two product lines: "Backblaze Backup" where we wrote the app that encrypts data on your laptop before uploading into our storage cloud, and "Backblaze B2" where you use any third party tool (or write your own) so the encryption in that case is all under your control. You can see a list of 3rd party tools here: https://www.backblaze.com/b2/integrations.html (make sure you scroll down). There are little pictures of penguins by the software that supports Linux, little pictures of an apple for software that support Macintosh, and a little Window icon for the software that support Windows.

For "Backblaze Backup", we use a well known design where we use symmetric AES-128 to encrypt the file, but each file is encrypted with a different AES-128 key, and that key and an Initialization Vector (IV) is then encrypted by a 2048 bit RSA public/private key. You can read about the encryption flow we use in this blog post: https://www.backblaze.com/blog/how-to-make-strong-encryption...

Hey Robbie-

I am touched you saved my email and remembered me, and that at the time you didn't respond negatively to the message. Thank you for being a customer.

The only thing I regret is that comment about version number of 2.3.1 and the claim that it would run flawlessly for a decade. Oh geez, please upgrade to 8.0.1, you won't regret it.

-- Brian Wilson, CTO Backblaze, brianw@backblaze.com

Disclaimer: I work at Backblaze on the backup client.

it looks like two sperm fighting over the egg

In 2017, this was proposed as one of several possible animations in the Backblaze client showing you are uploading files from your laptop to the cloud: https://i.imgur.com/L16xkMj.gifv It is an animated Gif, if you don't see it moving.

When I saw it, I had to send an EXTREMELY uncomfortable workplace email about a very personal activity two people do with something dribbling out. A few minutes later I hear from across the room "Well I CREATED it and now I can't unsee it!"

The decision was made to avoid that animation in favor of something else.

One interesting thing that Backblaze supports (that I don't think many people use) is that you can actually use any S3 API on any bucket, and any B2 API on the same bucket. Like every other call if you want.

So if you HAPPEN to find something is more clear with the B2 API, it's fine to use those calls on a bucket. If you find something is more clear in the S3 API, it's also fine to use those calls. The bucket won't get confused. :-)

how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html

If you want to use Amazon S3 APIs, you do not call ANYTHING that is documented on the Backblaze website, and you especially should not call "b2_get_upload_url" because that is a B2 native API, not an Amazon S3 API. You can always tell if you are using "B2 Native" if the call starts with "b2_" -> then that has literally nothing to do with Amazon S3 compatibility, it is the custom Backblaze protocol.

If you want to find out about Amazon S3 APIs (which you use to communicate to Backblaze's Storage Cloud or Amazon S3) then you can start here: https://docs.aws.amazon.com/general/latest/gr/signature-vers... Make sure you stay ENTIRELY on the Amazon website, and only read Amazon documentation, and use the APIs Amazon talks about (but of course you are doing all of this communicating with the Backblaze Storage Cloud backend). If any of that fails in your application, or in incompatible, PLEASE LET US KNOW!!

Disclaimer: I work for Backblaze so I'm biased.

Have you tried backblaze b2 storage? Requires more work client-side...

Backblaze recently released an S3 compatible API, so I'm hoping it is zero client-side work: https://www.backblaze.com/b2/docs/s3_compatible_api.html

If you try it, and find any issues, please let us know!

Disclaimer: I work for Backblaze so I'm biased and you should keep me honest. :-)

Backblaze as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface

Backblaze released an S3 compatible API recently: https://www.backblaze.com/b2/docs/s3_compatible_api.html

We're ALWAYS curious about any issue customers see, so if there is something specifically missing you use, we both want to hear about it, and we might be able to add it. Even if it doesn't help you right away, maybe it will help somebody else a few months down the road who might need the same feature.

We know we're compatible with Veeam backups (which only go through S3 APIs) for instance, and we continue to maintain that. We added the "S3 Object Lock" specifically for this particular vendor. So if you are missing one or two APIs, let us know!

you refuse to accept mine and others’ arguments about PEK

Not at all! I agree with you COMPLETELY. You want a zero knowledge backup product, Backblaze offers that and makes a large amount of money from it (millions of dollars annually actually), and we think you (csnover) should use that product because that's what you want. I understand the arguments, and you are COMPLETELY correct and I accept your arguments - that's why we built it and sell it, because you are correct.

One of our OTHER product offerings (in addition to the zero knowledge backup offering) is to host public websites. Public websites just can't have zero knowledge. We offer both products SEPARATELY, I've explained this over and over again, but you'll just post in two more days saying "Backblaze thinks zero knowledge is bad" (spectacularly false). Public websites (by definition) cannot be zero knowledge, do you comprehend this?

I don’t tell intentional falsehoods

You keep saying we don't have a zero knowledge backup system (wow, spectacularly false) and say we think zero knowledge is bad (again, spectacularly false since I've stated repeatedly that zero knowledge systems are THE BEST security). Then you say you don't tell intentional falsehoods. Come on, it's obvious something is going on.

I’m trying to kill your company with criticism

Is that it? Why do you care? Did we do something to you? I have a hard time believing you put in all this effort because you flipped a coin and decided you would try to kill Backblaze. Why us? I can't fix what I don't know about.

I look forward to your 200 future posts about how Backblaze doesn't offer zero knowledge backups.

Disclaimer: I work at Backblaze so I'm biased and you should keep me honest.

they probably haven't thought about it in such stark terms

Backblaze takes security incredibly seriously, and I assure you we have thought about it in such stark terms.

Expounding on that: Backblaze never raised any significant VC funding, we survive entirely on sales of our products, and most of that is keeping our customer's data utterly private, confidential, and safe. As this is the business we are in, our reputation is INCREDIBLY important to us. If we have a major breach we'll most likely lose our customer's trust, lose all our customers, and go out of business.

Since this is all I've done for the last 14 years, and my income and life savings is all wrapped up in Backblaze, I take this issue as seriously as a heart attack. So do my business partners (Backblaze was founded by 5 equal partners.) We're also up to around 200 employees who all would suffer greatly if we lose the trust of our customers.

Internally, Backblaze has a "Security Council" of software engineers and technical operations people with something like a combined 150 years of security experience and obsession. One of these council members got his CS degree from MIT and is both one of the smartest people I've ever worked with (we've worked together at 3 separate companies so far over 25 years) and also deeply paranoid and stressed out all the time. Another of the security council members has a PhD in computer science. And so on... They watch over all the design proposals, the APIs, the technical infrastructure, everything at Backblaze. They propose and implement new procedures, new programs like our BugCrowd program where we have external white hat hackers constantly trying to break in. We are also going through an internal security audit right now paying consultants to get yet another perspective.

In addition to the Security Council, all software engineers and all technical operations people are expected to worry about security all of the time. It is quite possibly our most talked about, most worried about, most important thing we do.

I'm sure their development practices will improve

We always, ALWAYS strive to do better and do more.

this response by you reads as quite disingenuous

It wasn't intended as such, I really meant it. I'd like to get to the bottom of this, understand what this person's true issue with Backblaze is.

these seem like very real reasons to get upset, maybe you "can try to fix" those?

What the user is doing is called "Gish gallop". This is a technique where somebody makes a rapid fire list of unrelated half truths or misrepresentations, each of which takes CONSIDERABLY longer to address than to claim. And I've repeated explained why they are invalid, but the user just shows up a day or two later and makes the same exact list of complaints. No edits, no admitting that even one of the complaints is invalid. Gish gallop.

This is not the behavior of somebody that is genuinely interested in having Backblaze address or fix that list of issues. There is something else going on, and I personally would like to know what it is. First of all because I'm curious what the issue is, second of all I hope I can fix whatever the real issue is.

I'm not going through the whole list because I've done that maybe 10 - 15 times so far? But let's take this one, because it's spectacularly false, this person KNOWS it's false, but this person repeatedly makes the claim over and over again:

Backblaze mislead users about PEK. The decryption key is sent to their server, and so is your password. It is not a zero-knowledge system. They don’t care.

Backblaze has 4 security levels, one of which is zero-knowledge, and we ENCOURAGE customers to pick the correct level for themselves. You can read my longer, in-depth answer to this same user just 2 days ago here: https://news.ycombinator.com/item?id=25904473 or you can read my longer, in depth answer 18 days ago here: https://www.reddit.com/r/backblaze/comments/kroqhn/private_e... or you can read my answer TWO YEARS AGO in the link this person supplied you (!!!!) or you can go back to the beginning, 13 years ago, when Backblaze started, where we explained EXACTLY how our encryption worked the same as the Microsoft Encrypted File System ("EFS") here: https://www.backblaze.com/blog/how-to-make-strong-encryption...

Now, despite it being a spectacularly false accusation that has been documented and explained so many times in so many forums, this user will undoubtable show up in another couple days and make this claim again. All the user's claims are like this. Obviously something else is going on.

I just wish that user would tell me what the real issue is. I can't fix what I don't know about.

Disclaimer: I work at Backblaze.

one should stick to SSDs for sound

I think you should stick to SSDs for performance reasons. :-) I can't go back to spinning drives for my desktop computers.

But here is one idea for you... Backblaze started in my dive one bedroom rental apartment in Palo Alto, California. Here is what that looks like with cubical walls setup in my living room: https://i.imgur.com/twu5vKe.jpg

Ok, so my desk in that picture is the one on the far right. I positioned my desk in the living room against a wall carefully so that my gas furnace room (with a door) was DIRECTLY on the other side of the wall. Then I put the computer inside the furnace room, drilled a hole through the drywall large enough to run the monitor cables and USB for the keyboard and mouse, and my desk was SILENT.

I caught SO MUCH grief for putting my computer in a furnace room, but the silence was worth it. I still claim to this day that if the furnace pumped out all of it's heat into the closet surrounding the furnace that would be a terrible design, that's not how it works at all. :-) But if it bothers you deeply, find a wall where there is a coat closet on the other side.

Here are more pictures of the original combination furnace room and machine room in the first Backblaze Corporate Headquarters: https://www.ski-epic.com/2007_small_business_setup/dell_xps_...

noticed that you are a fellow Beaver. Go Beavs! ;-)

Ha! This is a silly novelty "Beaver Baseball Cap" I was given during my internship in 1988 at Hewlett-Packard in Corvallis: https://i.imgur.com/G0rPHGP.jpg For 32 years it has sat on my computer monitor or on a shelf nearby. Nobody really asks about it anymore.

I got pretty lucky on timing at OSU. The year before I was there they taught beginning programming on punch cards on a mainframe computer called a "CDC Cyber". In my freshman year I took Pascal on the brand new 1984 Macintoshes in the Computer Science department, while the engineering students still learned Fortran on the mainframe my freshman year.

In 1992 I got a job as a Software Engineer at Apple in Cupertino, and you can trace that straight back to my blind luck of starting my programming education at OSU on the Mac in the exact correct year. Well, I'd rather be lucky than good.

And I've ALWAYS been lucky in my career.

Whether intentional or not, it's also great word-of-mouth advertising.

Oh, this has really worked out for Backblaze and we know it.

The first time we published our drive failure rates (I think January of 2014?) a few people said, "Uh oh, now Backblaze will get sued by the drive manufacturers." And we cringed and waited. :-) But the lawsuit never came, in fact there were NO repercussions, only increased visibility. People who have never heard of our company before find the data interesting, and then they ask "hey, what does this company do to own this many drives?" And a few (like yourself) sign up for the service.

Existing customers seem to stick with us for a long time, and even recommend us to other friends and family from time to time. So one tech person who stumbles across these stats might ACTUALLY bring us 3 or 4 more customers over the next 5 years. That's real money to us.

Not only have the drive manufacturers not sued us, they are actually NICE to us beyond the scale of our actual drive purchases! In one amusing example, our drive stats were used in a lawsuit as evidence. To be clear Backblaze was not the plaintiff or the defendant in the court case, we had no skin in the game at all and didn't want to be involved, but our drive data (and internal emails) were subpoenaed to be entered into evidence. Before we were served, the drive manufacturer called us and apologized for the inconvenience and made it clear they had no beef with us. Yes, a multi-national company that makes BILLIONS of dollars per year called a 40 person company (at the time) that could barely make payroll each month to apologize for the inconvenience. :-) We thought it was very considerate of them, and a little amusing. I'm proud to be the one that "signed" the papers indicating Backblaze had been "served".

I would expect an automatic process - disable and remove failed drive completely from a tome, add SOME free reserve drive at SOME rack in the datacenter to the tome

We have spec'ed this EXACT project a couple different ways but haven't built it yet. In one design, there would be a bank of spare blank drives plugged into a computer or pod somewhere, let's pretend that is on a workbench in the datacenter (it wouldn't be, but it helps the description). The code would auto-select a spare drive from the workbench and begin the rebuild in the middle of the night. At whatever time the datacenter technician arrives back in the datacenter they can move the drive from the workbench and insert it in place of the failed drive in the vault. It doesn't even matter if it is half way through the rebuild, the software as written today already handles that just fine. It would just continue from where the rebuild left off when it was pulled from the workbench.

The reason we don't really want to leave the drive in a random location across the datacenter in the long run is we prefer to isolate the network chatter that goes on INSIDE of one vault to be on one network switch (although a few vaults span two switches to not waste ports on the switches). Pods inside the vault rebuild the drive by talking with the OTHER 19 members of the vault, and we don't want that "network chatter" expanding across more and more switches. I'm certain the first few months of that would be FINE, maybe even for years. But we just don't want to worry about some random choke point getting created that throttles the rebuilds in some corner case without us in control of it. Each pod has a 10 Gbit/sec network port, and (I think) a 40 Gbit/sec uplink to the next level of switches upstream. The switches have some amount of spare capacity in every direction, but my gut feeling is in some corner case the capacity could choke out. We try to save money by barely provisioning everything to what it needs without much spare overhead.

Plus the time to walk a drive across the datacenter isn't that big of a deal. Some of the rebuilds of the largest drives can take a couple days ANYWAY. That 5 extra minutes isn't adding much durability.

Disclaimer: I work at Backblaze.

as much as I hate it I'm following Backblaze around and posting incorrect information about them

I get the impression Backblaze did something to upset you. Can you let me know what it is so I can try to fix it?

If there wasn't a pandemic on I would invite you to come to our office and I could buy you lunch and I could try to make up for whatever we did to upset you.