HN user

menage

496 karma
Posts2
Comments111
View on HN

At Google almost 20 years ago, a bunch of our machines (possibly with slightly bespoke CPUs?) were behaving oddly. These machines were mostly in use for serving Google's web index, so almost the entire RAM was devoted to index data; the indexserver processes were designed to be robust against hardware failure, and if they noticed any kind of corruption they'd dump and reload their data. We noticed that they were dumping/reloading massively more often than they'd expect.

Eventually the cause was narrowed down to that, randomly when the machine was stressed, the second half (actually, the final 2052 bytes) of some physical page in memory would get zeroed out. This wasn't great for the indexservers but they survived due to the defensive way that they accessed their data. But when we tried to use these new machines for Gmail, it was disastrous - random zeroing of general process code/data or even kernel data meant things were crashing hard.

We noticed from the kernel panic dumps (Google had a feature that sent kernel panics over the network to a central collector, which got a lot of use around this time) that a small number of pages were showing up in crash dump registers far more often than would statistically be expected. This suggested that the zeroing wasn't completely random. So we added a list of "bad pages" that would be forcefully removed from the kernel's allocator at boot time, so those pages would never be allocated for the kernel or any process. Any time we saw more than a few instances of some page address in a kernel panic dump, we added it to the list for the next kernel build. Like magic, this dropped the rate of crashes down into the noise level.

The root cause of the problem was never really determined (probably some kind of chipset bug) and those machines are long obsolete now. But it was somehow discovered that if you reset the machine via poking some register in the northbridge rather than via the normal reset mechanism, the problem went away entirely. So for years the Google bootup scripts included a check for this kind of CPU/chipset, followed by a check of how the last reset had been performed (via a marker file) and if it wasn't the special hard reset, adding the marker file and poking the northbridge to reset again. These machines took far far longer than any other machines in the fleet to reboot due to these extra checks and the double reboot, but it worked.

Reddit could offer a premium membership that allows add-free and API access, and require the third-party apps to log in on behalf of the user. The fee might still turn a lot of people off (although the lack of ads would make it so much nicer), but it at least wouldn't be a technical hurdle. That model seems to work for YouTube.

Almost! :-)

I coined the term "Xenoserver" back in 1999, for a paper in IEEE HotOS proposing an architecture for allowing systems in core networks to safely accept and execute code from untrusted users (for a fee, of course!): https://www.cl.cam.ac.uk/research/srg/netos/papers/1999-hoto...

I was inspired by the word "xenos" (meaning both stranger and guest, or in combination a stranger who you invited into your home) rather than the word "xenia" for the general concept of hospitality extended to such a stranger [EDIT: since I wasn't familiar with the word "xenia"].

The Xenoserver project at Cambridge University developed this idea, and eventually focused around the hypervisor, which took on the name Xen.

Celiac disease and Non-Celiac Gluten Sensitivity are very different beasts in terms of their effects. The latter is more likely to triggered in the presence of (as you say) non-trivial amounts of gluten; Celiac disease is an auto-immune disease that, while not normally at the level of anaphylactic shock, can have very unpleasant effects in tiny amounts.

And even within Celiac disease there are a lot of variations. My mother has been diagnosed as Celiac for 50 years (back before most doctors had heard of it, and she almost died from malnutrition pre-diagnosis). She obviously avoids anything that has any mention of wheat, or has been cooked in the same oil as glutenous food; she generally avoids anything that mentions "possible cross-contamination" on the label but doesn't have to be a total stickler. I don't think she's had a serious attack in many years. A friend of mine was diagnosed as Celiac maybe 10 years ago, and is incredibly sensitive - despite his best efforts he seems to end up with horrible symptoms every couple of months or so just through tiny amounts of contamination.

The FDA limit for claiming something is "gluten-free" is 20 ppm; I believe that level exists partly because it's very hard to detect anything less than that anyway, but it also fits in well with what most Celiac sufferers can tolerate.

But cross-contamination in a shared production facility can happen fairly easily (and is important for folks who are sensitive to a few parts-per-million of gluten), hence why lots of packets have that warning in the small print. An obvious "gluten-free" label means that the producer is asserting that this cross-contamination hasn't occurred and you don't have to search the small print.

Ménage Problem 4 years ago

This association with the French word that happens to be my surname is something that I've encountered more since moving from the UK to the US. Possibly due to the proximity to France resulting in a lot more UK people learning French in school than those who grow up in the US?

I think that depends a bit on how many of the folks tainted by the AHP violation are still working at Pfizer. If Pfizer is made up of a bunch of companies that had violations themselves before acquisition, that's probably reflected in the current company culture.

Google bought the seed that became Android. The startup had only been working on the mobile phone OS concept for about a year when Google acquired it in 2005 (originally they'd planned a camera OS). The first public Android phone wasn't released until 2008.

Mill's approach is pretty similar to Rosetta in this regard - binaries are distributed in a relative high-level abstract format that's a bit like LLVM IR, and code generation for a specific chip's concrete ISA is done at/before installation time.

The Tartine Bread book (as of 2018) explicitly recommends the autolyse/rest period (page 52, with more explanation on page 73).

Someone with an option actually paid money to exercise it.

If you still have the option, then you haven't paid to exercise it. Once you exercise it, you have a stock (or the sale proceeds in the case of an exercise-and-sell transaction).

It doesn't affect the chance of any one prisoner finding their number - that's still 50%. But the joint strategy massively reduces the independence of the individual prisoners' success, and hence the distribution of the number of successful prisoners.

Rather than a normal distribution (sum of independent outcomes) with a big peak around 50% of the prisoners finding their number and a vanishingly small chance of them all (or none) finding the right number, you end up with a much more complex pattern - there's an approximately uniform distribution in the 0-50 successes range with ~70% of the overall outcome, and a huge peak at the 100 successes point with ~30%.

https://www.r-bloggers.com/2014/08/update-100-prisoners-100-... has a bunch of distribution plots showing this.

I guess I should clarify that this wasn't the only reason I didn't take the acquisition transfer - it really wasn't clear that the acquirers knew what to do with the large number of smart people that they were basically acqui-hiring. But the heavy concentration of meetings caused by the 3-day week did influence my decision.

I left Facebook shortly after they abandoned their own plans for a 3-day-week pilot scheme which I'd been very eager to join. Someone I'd worked with before was now head of engineering at a reasonable-sized tech company, and was willing to give the 3-day week plan a try for me.

This was right at the start of 2020, and only working 3 days a week definitely helped me avoid pandemic burnout. But unfortunately I ended up working the 3 days that had the most meetings (this was a double-sided effect - it made sense to work those days else I'd miss important meetings, and then important meetings that included me gravitated to those days). So I ended up with 3 days that were stuffed full of meetings.

This was unpleasant enough that when the division was sold to another company, I opted not to take the transfer (and staying with the old company wasn't an option, as part of the terms of the acquisition deal). I'm now doing some flexi-time consulting that, while not as lucrative, gives a lot more latitude for scheduling my time.

no whole-filesystem image

At least in the past, almost all jobs ran in their own private filesystem - it was stitched together in userspace via bind mounts rather than having the kernel do it with an overlayfs extracted from layer tar files (since overlayfs didn't exist back then), but the result was fairly similar.

Most jobs didn't actually request any customization so they ended up with a filesystem that looked a lot like the node's filesystem but with most of it mounted read-only. But e.g. for a while anything running Java needed to include in their job definition an overlay that updated glibc to an appropriate version since the stock Google redhat image was really old.