ADefenestrator

Yes, this is a reasonably common strategy. It's how Cassandra's batch and group commit modes work, and Postgres has a similar option. Hopefully NATS will implement something similar eventually.

The Samsung consumer drives definitely don't do well under sustained high write workloads. The SLC cache fills up after a while, and write speeds drop drastically. The They also have some variety of internal head-of-line blocking type issue, where read latency goes way up when the writes saturate. I can't say I've ever seen 1s latency out of them, though.

Consumer drives can definitely have some quirks. The 2TB 960 Pro also just had weird write latency, even under relatively moderate load. Like 2-4ms instead of <1ms. It didn't really get much worse with extra load and concurrency, except that if there's writes enqueued the reads end up waiting behind them for some reason and also seeing the latency penalty.

They can also be weird behind RAID controllers, though I'm not sure if JBOD counts there. For whatever reason, the 860 EVO line wouldn't pass TRIM through the SAS RAID controller, but the 860 PRO would.

vB itself isn't FOSS. XenForo is the best alternative by a fair margin, but for FOSS there's good old PHPBB.

Yeah, even in a case like this where they turned out to be right, it might be one of those "predicted 15 out of the last 2 recessions" sorts of things.

Wouldn't have necessarily had to be nearly as bad as IE6 for security, either. Just lock the special browser to facebook.com only, or even just the ads API, and there's not a lot of room for exploit.

It's a bit hard when the temperature deltas are so small, though. It probably works well if you're pre-heating air before the main heat source in a very cold region, but otherwise you'd need to use some sort of heat pump to get it to move around.

I've heard they are actually closer to production, but yeah.. HAMR's been "a couple years away" for over a decade at this point, I think.

I haven't dug too deeply, but it's unclear whether anything but the Asrock Rack boards have full validated ECC implementations for Ryzen (vs just using the memory but ignoring ECC), and I think that may depend some on which Ryzen CPU is used (maybe PRO-only). I'd love a source of better info there, though.

You're right though - since the client's doing the work and they have a lot of redundancy/diversity in the storage it's not as big of a deal for them as it would be for us. I'd be a bit wary because the client-only verification does mean that there's no verification-with-ECC step in the entire chain, but I'm not sure that's significantly worse in terms of actual risk.

Pretty far. Since we cram so many drives into each server, our total server count is actually relatively low for the amount of storage we have. I'm not sure exactly how many units you need to amortize the design costs across to make it worth it for a custom ODM design, but I suspect it's in the tens of thousands.

It definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.

Disclaimer - Backblaze employee here, but just speaking for myself:

It was.. sort of cheaper. They didn't actually build the servers, and as described the server wouldn't work (onboard SATA didn't support port multipliers, lack of ECC would probably cause problems in practice, bit hand-wavey on power/space/network/manpower costs, etc). The goal of the article was to get other people to build cheap storage and put it up for rent on their network. They do have some amount of storage space available for very cheap on the network now, but personally I suspect it's people who figured "what the heck, I'll give it a try!" as opposed to people actually building storage servers and making a profit renting them out.

I was honestly pretty disappointed - I'd hoped they'd found a cheap motherboard with ECC and support for port multipliers, but nope.

Hi! Backblaze employee who did some of the LB stuff here. It's relatively standard/straightforward. There's a L4 load balancing layer using IPVS and ECMP-via-BGP, then a custom application that does the actual proxying/forwarding to the appropriate vault.

The US was too little and too late with testing for contact tracing to work. It might become possible again if the number of cases gets low enough, though.

The exact numbers? Of course not. Especially early on, they were suppressing information in addition to just not having complete information themselves. The cat's out of the bag at this point though, with WHO observers and foreign reporters on the ground. Even if there's some fudging and incompleteness, the overall trajectory of the infection curve is likely about right.

It's not just intubation but varying other protocols for treating sepsis, ARDS, etc depending on the exact set of symptoms. Even the ventilator itself is far from "hook it up and leave it" - there has to be a careful and up-to-date balance between getting enough oxygen into the bloodstream and not damaging the lungs more than absolutely necessary.

The people on a nasal cannula of oxygen probably only need limited skilled intervention and supervision, but once they're intubated it's pretty intensive.

Probably a lot of the other way around as well. Google employees likely use Chrome and Docs/Sheets heavily, so they get a rapid high-quality feedback loop with any bugs or performance issues affecting the Chrome devs and people around them.

"prone to" is putting it mildly. ICMP echo requests tend to be processed by the often relatively anemic control plane CPU rather than ASICS and are at the bottom of the heap in terms priority. If you send a router 1MB of ICMP echo requests, it's virtually guaranteed to drop some or even most of them.

In addition to the SATA port multipliers, they'd need actual SATA PCIe cards. Basically nobody makes a motherboard with onboard SATA that supports port multipliers.

That's the price being paid for the storage, but is it actually covering the cost of providing that storage? Or is it just 100-300 people who thought "Huh, neat, I'll toss a host online and see how it goes" ? I'd lean towards the latter and assume those storage costs are heavily subsidized by a few people satisfying curiosity.

Even if it had slots for the splitter cable, Intel and AMD onboard SATA explicitly doesn't support port multipliers as far as I know. You can buy PCIE SAS cards that do for relatively cheap, but then you have to find board with enough PCIe slots. Easy enough on the "gamer" boards but if you want ECC (and you probably do, for storage) and IPMI (you probably do, if you have more than a few dozen servers) your options get much more limited. Other than 1 or 2 Asrock Rack boards, you pretty much have to move into Epyc 7000-series or Xeon Silver or above. Often dual-socket on the Xeons to get a board with lots of PCIe.

In theory something like an Epyc 3000-series with lots of PCIe or onboard SATA that supports port multipliers would work great, but I don't think anyone actually makes that.

The trick is this: the chances that a specific fund will do well that long via luck are very low, but the chances that there exists a fund among all that exist that has done well via luck are quite high.

I'm not too familiar with JVM internals or Spark, but I know with Cassandra at least there's a cost to off-heap memory. You gain in GC, but eventually you have to move that data in and out of the JVM's heap.

Even for batch processing, long GCs can be bad. It's not just processing the batch that stops, but the whole world. Anything trying to listen for more data, keep track of time elapsed, etc is going to run across more problems. It can also expose some race conditions that would normally be so unlikely that you'd never hit them.

Static JVM memory overhead isn't bad at all, probably even under your 100-200MB guess. The lack of compact primitives adds some additional proportional overhead, but the biggest factor is just the extra "slack" space needed for good GC performance. Depending on the circumstances and requirements that could be 30-400%.

A bit apples and oranges, though. The ASCI Red is a general-purpose computer that ran whatever you wanted via the full x86 instruction set. The Xavier and Tesla chips are basically very specific FMADD modules that can't do much else.

Colo can mean "rent a suite from DRT in one location and a building in another" in addition to space in a rack. Lyft's probably around that size (and it's a good idea to do some colo to gain experience before trying to build out your own anyways)

Depends on the case. Facebook at least generally replaces drives as they go. In a pipelined fashion rather than as they fail, though. So, a given drive failure doesn't lead to immediate emergency replacement, but sometime in the next few days as someone makes a drive-replacement pass through that area.

Modern PHP's quite a bit better than the utter mess that was 4.0 or even the half-ugly 5.0. They've deprecated the worst of the misfeatures, especially by default. Now if only they'd adopted the HHVM/Hack Collections instead of the terrible arrays...

Even in cases like Cassandra it's far from flawless, mostly due to the massive complexity. Failover works great, compaction works great, schema changes work great, repairs work... but what happens if two of those happen at once? Or all of them? There have been quite a few bugs over the years that involve corner cases when the various coordination and deferred-work systems interact.

There's some other downsides of course, but RAID1 basically gets you exactly that. Double price for much lower failure rate of the storage volume. Go with different manufacturers for the two drives to further reduce the failure rate at the probable cost of a little performance and space.

5. Your network isn't partitioned exactly, but someone bumped an ethernet cable and the packet loss has reduced the goodput on that link to a level too low to sustain the throughput you need. With most congestion control algorithms (basically, not-BBR), at 10Gb even 1% packet loss is devastating.

6. Well, basically the hundred other reasons the machine could brown-out enough that things start timing out even though it's sporadically online. Bad drive, rogue process, loose heatsink, etc.

Dead hosts are easy. Half-dead hosts suck.

G1 does a lot better here in my (somewhat limited) experience. There's some overhead, and you have to give it some extra space, but the mixed collections are decent at making their way through medium-aged objects that survived to the old generation. Still nowhere close to 5-10ms though, and with a fair bit of memory bandwidth burned copying it through the survivor spaces.