cletus

I have to disagree with this as someone who worked at Meta (and Google).

WhatsApp was a textbook example of how not to do an acquisition. The story I heard was when it was acquired, a spreadsheet went around and everyone basically decided what level they were in the (then) FB job ladder and all the engineers said they were E7s (Senior Staff SWE). The way PSC worked, WhatsApp at the time was only ever calibrated against themselves (from what I heard). It had become a fiefdom and, as someone who was on a team that tried to get them to do anything, the experience was awful.

IG was handled better but it was also an almost nonexistent team when acquired, which might well explain it. They stuck with their Django/Python codebase and (IMHO) that was a mistake. The amount of duplication that we had to do for IG specifically was embarrasing. The framework and tooling FB had on the product side was light years ahead of what IG had. IG used to have a very good product focus but I think that's long dead now. It was good because IG had a clear vision for their app and ultimately (IMHO) management had a different view to "grow". They briefly tried to launch another app (IGTV) that flopped, hard. There were a bunch of UI/UX changes that clearly showed the focus had become simply following celebrities instead of sharing updates (eg where the post/compose buttons moved to).

I mention Google because I saw the same things happen at Google.

Youtube was (and my guess is, still is) its own entity. Culturally, Youtubers don't see themselves as Googlers. They didn't (AFAIK) use Google3 or any of the other stuff most of the rest of Google did. But Youtube itself was perceived very positively, technically, particularly in relation just general encoding/decoding infrastructure as well as Bandaid (where racks are shipped to ISPs to cache videos).

Android was another acquisition that prided itself in not being Google. This was very much fostered by Andy Rubin while he was still there. Obviously Google needed to write Android apps but I got the sense that it was always Google engineers who solved all the problems whereas Android just didn't care. They cared only about shipping Android. Fuchsia was an Android offshoot.

Docs and Maps were both acquisitions but they went fully Google3 and were different orgs but weren't seen as separate. The engineering director of Docs (Fuzzy) had, from what I can recall, a very positive reputation beyond Docs (now Drive).

Doubleclick was also an acqusition but went fully Google and you'll find a lot of people who don't even know it was an acquisition.

I don't know what org you worked in but they all vary. My own experience was that Infra orgs in comparison to Google were primitive and barely above just running random Docker-like (Tupperware) instances with a godawful variant of C++, probably started by someone who had done C++ at Google and had decided they really wanted mutable function parameters and exceptions for no particular reason.

The thing I really respected about FB product orgs generally was that really did ship things quickly. I used to joke that the smallest unit of time at Google was a quarter. God help you if you eneded another team (under a different VP) to do something. You'd have to spend a quarter arguing with them to get them to add it to their OKRs for the following quarter.

At FB the timeline for launching a new thing to a limited audience was measured in weeks. The biggest barrier usually was the weekly build cycle for the blue app. The release cycle for Web was S-tier and (IMHO) the people who worked on the infra for Web were generally god tier. This was another reason why IG doggedly sticking with Python just created problems.

There are many thigns you can criticize Meta for (eg the stupid crypto, the billions wasted on VR) but the Web Foundation and Ent teams were god tier and I'll die on that hill.

Anyway, even back then the ML teams and infra, not to put too fine a point on it, sucked (IMHO). Newsfeed was OK but the recommendations for a lot of things like videos just sucked. All of this was mainly because it all relied on daily offline jobs. And then Tiktok came along and showed everybody (including Youtube) just what a bad job they were doing at recommendations. And don't get me started on the IG Reels dumpster fire.

Oh ok, I thought of another one: Messenger. I knew some smart people on Messenger but overall the product and the infra were, again, a dumpster fire.

If I were the CTO of any of these companies I would be working my butt off to be making an internal version of Claude. Let me explain my reasoning using Google as an example (disclaimer: Xoogler).

Google has a lot of systems to make a very large monorepo manageable so builds and code search don't take forever. The build system is Blaze (on which Bazel is based), which has a Pythonic syntax and was once Python but that hasn't been the case (AFAIK) for over a decade. This means you build a massive digraph of build artifacts. By "large" I mean somewhere between 100M and 1B vertices (guessing). Loading that became a significant problem for a build so there's heavy caching around that. There's also heavy caching around build artifacts (ie Forge).

So, part of the issue with every developer using Claude is that you have a ton of inefficiency becasue everybody has a significant context. And what is context really? It's not too dissimilar to the build graph and/or code search you already have.

So the infra I would be working on would be some kind of "global context" or "context cache". Now a lot of context changes when you do a local change but a lot doesn't. As an ordinary engineer, you aren't generally modifying /base. You're modifying leaf nodes or branches for very few leaf nodes.

The reasons I see to do this are:

1. Cost-savings by deduplication;

2. Speed if context is partially-cached;

3. You avoid issues of sending out your codes to third-parties. In the case of Google or Amazon, if they use Claude at all, they would probably only be using their own clouds so they avoid this. But Uber doesn't have that luxury;

4. You avoid any issues of people using your prompts for responses for training and leaking any potential sensitie information that way;

5. You can use off-peak resources for a lot of this work;

6. You can control resources within your own pervasive resource management (in the case of Google); and

7. You can more easily integrate into internal tooling.

I also think that expanding compute power is the biggest risk to Anthropic (and OpenAI). There's a vast difference between a model you need a cluster of NVidia's finest to run vs one you can run on a Macbook Pro. We aren't there yet on a Macbook Pro but it'll only be a few years we are.

There's more history than this. Disclaimer: Xoogler (2010-2017).

When I first started the environment you used depended entirely on language. In the C++ and Python space, there was the vim and emacs divide. With Java it was more complicated. Some still used vim/emacs but a lot of people used Eclipse.

Now Eclipse was a real problem at Google because of the source control system. Java IDEs are primarily built to import binaries, specifically jars. In the outside world, these dependencies are managed via Ant (very early days), Maven/Gradle or the like.

At Google there's a mono-repo (Perforce/Piper) and you check out parts of it locally and rely on the rest via a network connection (to SrcFS IIRC, it's been awhile). This was neat because you could edit a file locally and the dependencies would just recompile (via Blaze).

So for Eclipse a whole lot of initialization had to be done and the IDE would fall over. A lot. It had a team of ~10 working on it at one point. Then somebody did a 20% project called magicjar. Magicjar took a Perforce client and built all the dependencies as jars that could be imported directly without parsing the entire source tree (which was usually huge). This made it possible, even preferred, to use IntelliJ, which is what I did. Magicjar was great.

Other people actually made CLion work reasonably well with C++ too. That was nice. This was a much bigger undertaking with many more corner cases just given how C++ works (ie headers and templates).

So checking out a client was relatively heavyweight, even with a minimal local tree. And, if you worked on Google3, you had to do this a lot. You might need to do a config file change. This was the real starting point for Cider because it was way nicer to do config file changes with it.

Obviously I don't know where all this went from there. VS Studio as a Cider frontend? Ok, that was news to me. Engineers being unhappy when things change and when the slightest thing works differently is the least surprising thing I've ever heard.

Oh it's worth adding that in my time many people didn't use Perforce (P4) directly. They used somebody else's project, which was a Git frontend for it, called Git5. I believe it was already being deprecated while I was still there. But Git5 modelled a P4 change as a branch so you could play around with your Git commits locally and then squash them into a single P4 change. I actually liked this a lot.

I worked at Google a long time ago now during the whole Google+ fiasco. One thing that was super controversial internally was the so-called Real Names policy. For those who don't know or remember, it was Vic Gundotra's idea that people should use their real identities. He kept using this weird example that he didn't want it filled with people named "Dog Turd". I don't know why.

So there was this mysterious black box that decided if your name was "real" or not. At first this didn't support pseudonyms or any kind of anonymity and that's actually really important for any social network. Think of someone seeking help coming to terms with their sexual orientation, gender identity, addiction, eating disorder or whatever. Or simply going against their family's religious wishes. I later worked at Facebook and one thing I'll give them credit for is Groups. FB Groups had an identity that actually couldn't be tied by anyone else to your profile or identity in any other group. That was a good product decision.

Anyway, if your name somehow failed the magic real names filter, your account got banned. Your entire Google account was banned and basically there was no recourse other than knowing someone who worked at the company or making a big enough fuss on Twitter.

Many people, myself included, criticized and protested this decision. You should at least segment Google products. There's absolutely no reason to ban your Gmail account because an automated system decided your Google+ account name wasn't "real". But that feedback was ignored and this was well before the public launch. And the public backlash proved this position correct (IMHO).

But the net effect was that I decided I can't use any other Google product. Let's say a system is launched to find offensive photos and there's a false positive on one of my images in Google Photos. Maybe it's just a hash collision with a known image. And then what? I lose my entire Gmail? Are you kidding me?

It's wild to me that this is still an issue ~15 years later. I think my stance actually isn't strict enough anymore. You probably shouldn't use Gmail at all. I should really find a paid email provider hosted entirely in Europe, preferably Switzerland or some other country with strong pro-user regulation.

So I have no idea if this Gemini story is true or not. I say that because 95% of the things on Reddit are completely made up. But it is plausible. I wouldn't be surprised if it's true. It means I wouldn't use Gemini at all if I used Gmail.

Story time. I used to work for Facebook (and Google) and lots of games were played around bugs.

At some point the leadership introduced an SLA for high then medium priority bugs. Why? because bugs would sit in queues for years. The result? Bugs would often get downgraded in priority at or close to the SLA. People even wrote automated rules to see if their bugs filed got downgraded to alert them.

Another trick was to throw it back to the user, usually after months, ostensibly to request information, to ask "is this still a problem?" or just adding "could not reproduce". Often you'd get no response. sometimes the person was no longer on the team or with the company. Or they just lost interest or didn't notice. Great, it's off your plate.

If you waited long enough, you could say it was "no longer relevant" because that version of the app or API had been deprecated. It's also a good reason to bounce it back with "is still this relevant?"

Probably the most Machiavellian trick I saw was to merge your bug with another one vaguely similar that you didn't own. Why? Because this was hard to unwind and not always obvious.

Anyone who runs a call center or customer line knows this: you want to throw it back at the customer because a certain percentage will give up. It's a bit like health insurance companies automatically sending a denial for a prior authorization: to make people give up.

I once submitted some clear bugs to a supermarket's app and I got a response asking me to call some 800 number and make a report. My bug report was a complete way to reproduce the issue. I knew what was going on. Somebody simply wanted to mark the issue as "resolved". I'm never going to do that.

I don't think you can trust engineering teams (or, worse, individuals) to "own" bugs. They're not going to want to do them. They need to be owned by a QA team or a program team that will collate similar bugs and verify something is actually fixed.

Google had their own versions of things. IIRC bugs had both a priority and s everity for some reason (they were the same 99% of the time) between 0 and 4. So a standard bug was p2/s2. p0/s0 was the most severe and meant a serious user-facing outage. People would often change a p2/s2 to p3/s3, which basically meant "I'm never going to do this and I will never look at it again".

I've basically given up on filing bug reports because I'm aware of all these games and getting someone to actually pay attention is incredibly difficult. So much of this comes down to stupid organizational-level metrics about bug resolution SLAs and policies.

So there's some survivor bias here but it's generally not bad advice. You should be focusing on outcomes like improving SLAs, top line metrics and so on. You should be solving user and business problems. That's all good advice. But still this article presumes a lot.

In my experience, managers will naturally partition their reports into three buckets: their stars, their problems and their worker bees. The worker bees tend to be ignored. They're doing fine. They're getting on with whatever they've been told to do or possibly what they've found to do. They're not going to create any problems. The problems are the underperformers. These are people who create problems and/or are at risk of getting a subpar performance rating.

Now there are lots of reasons that someone can be a problem. I tend to believe that any problem just hasn't found the right fit yet and, until proven otherwise, problems are a failure in management. That tends to be a minority view in practice. It's more common to simply throw people in the deep end and sink or swim because that takes much less overhead. You will see this as teams who have a lot of churn but only in part of the team. In particularly toxic environments, savvy managers will game the system by having a sacrificial anode position. They hire someone to take the bad rating they have to give to protect the rest of the team.

And then there are the stars. These are the people you expect to grow and be promoted. More often than not however they are chosen rather than demonstrating their potential. I've seen someone shine when their director is actively trying to sabotage them but that's rare.

Your stars will get the better projects. Your problems will get the worse ones. If a given project is a success or not will largely come down to perception not reality.

The point I'm getting to is that despite all the process put around this at large companies like performance ratings, feedback, calibration, promo committees, etc the majority of all this is vibes based.

So back to the "take my job" advice. If someone is viewed as a star, that's great advice. For anyone else, you might get negative feedback about not doing your actual job, not being a team player and so on. I've seen it happen a million times.

And here's the dirty little secret of it all: this is where the racism, sexism and ableism sneaks in. It's usually not that direct but Stanford grads (as just one example) will tend to vibe with other Stanford grads. They have common experience, probably common professors and so on. Same for MIT. Or CMU. Or UW. Or Waterloo. And so on.

So all of the biases that go into the selection process for those institutions will bleed into the tech space.

And this kind of environment is much worse for anyone on the spectrum because allistic people will be inclined to dislike from the start for no reason and that's going to hurt how they're viewed (ie as a star, a worker bee or a problem) and their performance ratings.

Because all of this is ultimately just a popularity contest with very few exceptions. I've seen multiple people finagle their way to Senior STaff SWE on just vibes.

And all of this gets worse since the tech sector has joined Corporate America in being in permanet layoff mode. The Welchian "up or out" philosophy has taken hold in Big Tech where there are quotas of 5-10% of the workforce have to get subpar ratings every year and that tends to kill their careers at that company. This turns the entire workplace even more into an exercise in social engineering.

I'm going to pick out 3 points:

2. Being right is cheap. Getting to right together is the real work

6. Your code doesn’t advocate for you. People do

14. If you win every debate, you’re probably accumulating silent resistance

The common thread here is that in large organizations, your impact is largely measured by how much you're liked. It's completely vibes-based. Stack ranking (which Google used to have; not sure if it still does) just codifies popularity.

What's the issue with that? People who are autistic tend to do really badly through no fault of their own. These systems are basically a selection filter for allistic people.

This comes up in PSC ("perf" at Meta, "calibration" elsewhere) where the exact same set of facts can be constructed as a win or a loss and the only difference is vibes. I've seen this time and time again.

In one case I saw a team of 6 go away and do nothing for 6 months then come back and shut down. If they're liked, "we learned a lot". If they're not, "they had no impact".

Years ago Google studied the elements of a successful team and a key element was psychological safety. This [1] seems related but more recent. This was originally done 10-15 years ago. I agree with that. The problem? Permanent layoffs culture, designed entirely to suppress wages, kills pyschological safety and turns survival into a game of being liked and manufacturing impact.

18. Most performance wins come from removing work, not adding cleverness

One thing I really appreciated about Google was that it has a very strict style guide and the subset of C++ in particular that you can use is (was?) very limited. At the time, this included "no exceptions", no mutable function arguments and adding templtes had an extremely high bar to be allowed.

Why? To avoid arguments about style issues. That's huge. But also because C++ in particular seemed to attract people who were in love with thier own cleverness. I've seem some horrific uses of templates (not at Google) that made code incredibly difficult to test for very little gain.

9. Most “slow” teams are actually misaligned teams

I think this is the most important point but I would generalize it and restate it as: most problems are organizational problems.

At Meta, for example, product teams were incentivized to ship and their impact was measured in metric bumps. But there was no incentive to support what you've already shipped beyond it not blowing up. So in many teams there was a fire and forget approach to filing a bug and forgetting about it, to the point where it became a company priority to have SLAs on old bugs, which caused the inevitable: people just downgrading bug priorities to avoid SLAs.

That's an organizational problem where the participants have figured out that shiping is the only thing they get rewarded for. Things like documentation, code quality and bug fixes were paid lip service to only.

Disclaimer: Xoogler, ex-Facebooker.

[1] https://www.aristotleperformance.com/post/project-aristotle-...

The mistake was not having nullability be expressed in the type system.

At Facebook I used their PHP fork Hack a lot and Hack has a really expressive type system where PHP does not. You can express nullability of a type and it defaults to a type being non-nullable, which is the correct default. The type checker was aware of changes too, so:

    function foo(?A $a): void {
      $a->bar(); // compile error, $a could be null
      if ($a is null) {
        return;
      }
      $a->bar(); // not a compiler error because $a is now A not ?A
      if ($a is ChildOfA) {
        $a->childBar(); // not an error, in this scope $a is ChildOfA
      }
    }

The real problem with Java in particular is you'd end up chaining calls then get the dreaded NullPointerException and have no idea from the error or the logs what was broken from:

   a.b.c.d();

As an early user of SO [1], I feel reasonably qualified to discuss this issue. Note that I barely posted after 2011 or so so I can't really speak to the current state.

But what I can say is that even back in 2010 it was obvious to me that moderation was a problem, specifically a cultural problem. I'm really talking about the rise of the administrative/bureaucratic class that, if left unchecked, can become absolute poison.

I'm constantly reminded of the Leonard Nimoy voiced line from Civ4: "the bureaucracy is expanding to meet the needs of the expanding bureaucracy". That sums it up exactly. There is a certain type of person who doesn't become a creator of content but rather a moderator of content. These are people who end up as Reddit mods, for example.

Rules and standards are good up to a point but some people forget that those rules and standards serve a purpose and should never become a goal unto themselves. So if the moderators run wild, they'll start creating work for themselves and having debates about what's a repeated question, how questions and answers should be structured, etc.

This manifested as the war of "closed, non-constructive" on SO. Some really good questions were killed this way because the moderators decided on their own that a question had to have a provable answer to avoid flame wars. And this goes back to the rules and standards being a tool not a goal. My stance was (and is) that shouldn't we solve flame wars when they happen rather than going around and "solving" imaginary problems?

I lost that battle. You can argue taht questions like "should I use Javascript or Typescript?" don't belong on SO (as the moderators did). My position was that even though there's no definite answer, somebody can give you a list of strengths and weaknesses and things to consider.

Even something that does have a definite answer like "how do I efficiently code a factorial function?" has multiple but different defensible answers. Even in one language you can have multiple implementations that might, say, be compile-time or runtime.

Another commenter here talked about finding the nearest point on an ellipse and came up with a method they're proud of where there are other methods that would also do the job.

Anyway, I'd occasionally login and see a constant churn on my answers from moderators doing pointless busywork as this month they'd decided something needed to be capitalized or not capitalized.

A perfect example of this kind of thing is Bryan Henderson's war on "comprised of" on Wikipedia [2].

Anyway, I think the core issue of SO was that there was a lot of low-hanging fruit and I got a lot of accepted answers on questions that could never be asked today. You'll also read many anecdotes about people having a negative experience asking questions on SO in later years where their question was immediately closed as, say, a duplicate when the question wasn't a duplicate. The moderator just didn't understand the difference. That sort of thing.

But any mature site ultimately ends with an impossible barrier to entry as newcomers don't know all the cultural rules that have been put in place and they tend to have a negative experience as they get yelled at for not knowing that Rule 11.6.2.7 forbids the kind of question they asked.

[1] https://stackoverflow.com/users/18393/cletus

[2] https://www.npr.org/2015/03/12/392568604/dont-you-dare-use-c...

Imagine you're testing a service to creates, queries and deletes users. A fake version of that service might just be a wrapper on a HashMap keyed by ID. It might have several fields like some personal info, a hashed password, an email address, whether you're verified and so on.

Imagine one of your tests is if the user deletes their account. What pattern of calls should it make? You don't really care other than the record being deleted (or marked as deleted, depending on retention policy) after you're done.

In the mock world you might mock out calls like deleteUserByID and make suer it's called.

In the fake world, you simply check that the user record is deleted (or marked as such) after the test. You don't really care about what sequence of calls made that happen.

That may sound trivial but it gets less trivial the more complex your example is. Imagine instead you want to clear out all users who are marked for deletion. If you think about the SQL for that you might do a DELETE ... WHERE call so your API call might look like that. But if the logic is more complicated? Where if there's a change where EU and NA users have different retention periods or logging requirements so they're suddenly handled differently?

In a mokcing world you would have to change all your expected mocks. In fact, implementing this change might require fixing a ton of tests you don't care about at all and aren't really being broken by the change regardless.

In a fake world, you're testing what the data looks like after you're done, not the specific steps it took to get there.

Now those are pretty simple examples because there's not much to do the arguments used and no return values to speak of. Your code might branch differently based on those values, which then changes what calls to expects and with what values.

You're testing implementation details in a really time-consuming yet brittle way.

My second project at Google basically killed mocking for me and I've basically never done it since. Two things happened.

The first was that I worked on a rewrite of something (using GWT no less; it was more than a decade ago) and they decided to have a lot of test coverage and test requirements. That's fine but they way it was mandated and implemented, everybody just testing their service and DIed a bunch of mocks in.

The results were entirely predictable. The entire system was incredibly brittle and a service that existed for only 8 weeks behaved like legacy code. You could spend half a day fixing mocks in tests for a 30 minute change just because you switched backend services, changed the order of calls or just ended up calling a given service more times than expected. It was horrible and a complete waste of time.

Even the DI aspect of this was horrible because everything used Guice andd there wer emodules that installed modules that installed modules and modifying those to return mocks in a test environment was a massive effort that typically resulted in having a different environment (and injector) for test code vs production code so what are you actually testing?

The second was that about this time the Java engineers at the company went on a massive boondoggle to decide on whether to use (and mandate) EasyMock vs Mockito. This was additionally a waste of time. Regardless of the relative merits of either, there's really not that much difference. At no point is it worth completely changing your mocking framework in existing code. Who knows how many engineering man-yars were wasted on this.

Mocking encourages bad habits and a false sense of security. The solution is to have dummy versions of services and interfaces that have minimal correct behavior. So you might have a dummy Identity service that does simple lookups on an ID for permissions or metadata. If that's not what you're testing and you just need it to run a test, doing that with a mock is just wrong on so many levels.

I've basically never used mocks since, so much so that I find anyone who is strongly in favor of mocks or has strong opinions on mocking frameworks to be a huge red flag.

Story time. This has basically nothing to do with this post other than it involves a limit of 10,000 but hey, it's Christmas and I want to tell a story.

I used to work for Facebook and many years ago people noticed you couldn't block certain people but the one that was most public was Mark Zuckerberg. It would just say it failed or something like that. And people would assign malice or just intent to it. But the truth was much funnier.

Most data on Facebook is stored in a custom graph database that basically only has 2 tables that are sharded across thousands of MySQL instances but most almost always accessed via an in-memory write-through cache, also custom. It's not quite a cache because it has functionality built on top of the database that accessing directly wouldn't have.

So a person is an object and following them is an edge. Importantly, many such edges were one-way so it was easy to query if person A followed B but much more difficult to query all the followers of B. This was by design to avoid hot shards.

So I lied when I said there were 2 tables. There was a third that was an optimization that counted certain edges. So if you see "10.7M people follow X" or "136K people like this", it's reading a count, not doing a query.

Now there was another optimization here: only the last 10,000 of (object ID,edge type) were in memory. You generally wanted to avoid dealing with anything older than that because you'd start hitting the database and that was generally a huge problem on a large, live query or update. As an example, it was easy to query the last 10,000 people or pages you've followed.

You should be able to see where this is going. All that had happened was 10,000 people had blocked Mark Zuckerberg. Blocks were another kind of edge that was bidirectional (IIRC). The system just wasn't designed for a situation where more than 10,000 people wanted to block someone.

This got fixed many years ago because somebody came along and build a separate system to handle blocking that didn't have the 10,000 limit. I don't know the implementation details but I can guess. There was a separate piece of reverse-indexing infrastructure for doing queries on one-way edges. I suspect that was used.

Anyway, I love this story because it's funny how a series of technical decisions can lead to behavior and a perception nobody intended.

People should go to jail for this.

Anyone who has worked on a large migration eventually lands on a pattern that goes something like this:

1. Double-write to the old system and the new system. Nothing uses the new system;

2. Verify the output in the new system vs the old system with appropriate scripts. If there are issues, which there will be for awhile, go back to (1);

3. Start reading from the new system with a small group of users and then an increasingly large group. Still use the old system as the source of truth. Log whenever the output differs. Keep making changes until it always matches;

4. Once you're at 100% rollout you can start decomissioning the old system.

This approach is incremental, verifiable and reversible. You need all of these things. If you engage in a massive rewrite in a silo for a year or two you're going to have a bad time. If you have no way of verifying your new system's output, you're going to have a bad time. In fact, people are going to die, as is the case here.

If you're going to accuse someone of a criminal act, a system just saying it happened should NEVER be sufficient. It should be able to show its work. The person or people who are ultimately responsible for turning a fraud detection into a criminal complaint should themselves be criminally liable if they make a false complaint.

We had a famous example of this with Hertz mistakenly reporting cars stolen, something they ultimately had to pay for in a lawsuit [1] but that's woefully insufficient. It is expensive, stressful and time-consuming to have to criminally defend yourself against a felony charge. People will often be forced to take a plea because absolutely everything is stacked in the prosecution's favor despite the theoretical presumption of innocence.

As such, an erroneous or false criminal complaint by a company should itself be a criminal charge.

In Hertz's case, a human should eyeball the alleged theft and look for records like "do we have the car?", "do we know where it is?" and "is there a record of them checking it in?"

In the UK post office scandal, a detection of fraud from accounting records should be verified by comparison to the existing system in a transition period AND, moreso in the beginning, double checking results with forensic accountants (actual humans) before any criminal complaint is filed.

[1] https://www.npr.org/2022/12/06/1140998674/hertz-false-accusa...

I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause.

The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQL models. The first is normal for microservices (but an indictment at the same time). The second is... weird.

The whole point of GraphQL is to create a unified view of something, not to have 23 different versions of "Movie". Attributes are optional. Pull what you need. Common subsets of data can be organized in fragments. If you're not doing that, why are you using GraphQL?

So I worked at Facebook and may be a bit biased here because I encountered a couple of ex-Netflix engineers in my time who basically wanted to throw away FB's internal infrastructure and reinvent Netflix microservices.

Anyway, at FB there a Video GraphQL object. There aren't 23 or 7 or even 2.

Data storage for most things was via write-through in-memory graph database called TAO that persisted things to sharded MySQL servers. On top of this, you'd use EntQL to add a bunch of behavior to TAO like permissions, privacy policies, observers and such. And again, there was one Video entity. There were offline data pipelines that would generally process logging data (ie outside TAO).

Maybe someone more experienced with microservices can speak to this: does UDA make sense? Is it solving an actual problem? Or just a self-created problem?

The key part here is "machine utilization" and absolutely there was a ton of effort put into this. I think before my time servers were allocated to projects but even early on in my time at Google Borg had already adopted shared machine usage and therew was a whole system of resource quota implemented via cgroups.

Likewise there have been many optimization projects and they used to call these out at TGIF. No idea if they still do. One I remember was reducing the health checks via UDP for Stubby and given that every single Google product extensively uses Stubby then even a small (5%? I forget) reduction in UDP traffic amounted to 50,000+ cores, which is (and was) absolutely worth doing.

I wouldn't even put latency in the same category as "performance optimization" because often you decrease latency by increasing resource usage. For example, you may send duplicate RPCs and wait for the fastest to reply. That could be double or tripling effort.

So I've worked for Google (and Facebook) and it really drives the point home of just how cheap hardware is and how not worth it optimizing code is most of the time.

More than a decade ago Google had to start managing their resource usage in data centers. Every project has a budget. CPU cores, hard disk space, flash storage, hard disk spindles, memory, etc. And these are generally convertible to each other so you can see the relative cost.

Fun fact: even though at the time flash storage was ~20x the cost of hard disk storage, it was often cheaper net because of the spindle bottleneck.

Anyway, all of these things can be turned into software engineer hours, often called "mili-SWEs" meaning a thousandth of the effort of 1 SWE for 1 year. So projects could save on hardware and hire more people or hire fewer people but get more hardware within their current budgets.

I don't remember the exact number of CPU cores amounted to a single SWE but IIRC it was in the thousands. So if you spend 1 SWE year working on optimization acrosss your project and you're not saving 5000 CPU cores, it's a net loss.

Some projects were incredibly large and used much more than that so optimization made sense. But so often it didn't, particularly when whatever code you wrote would probably get replaced at some point anyway.

The other side of this is that there is (IMHO) a general usability problem with the Web in that it simply shouldn't take the resources it does. If you know people who had to or still do data entry for their jobs, you'll know that the mouse is pretty inefficient. The old terminals from 30-40+ years ago that were text-based had some incredibly efficent interfaces at a tiny fraction of the resource usage.

I had expected that at some point the Web would be "solved" in the sense that there'd be a generally expected technology stack and we'd move on to other problems but it simply hasn't happened. There's still a "framework of the week" and we're still doing dumb things like reimplementing scroll bars in user code that don't work right with the mouse wheel.

I don't know how to solve that problem or even if it will ever be "solved".

Sure. First you need to separate buffered and unbuffered channels.

Unbuffered channels basically operate like cooperate async/await but without the explictness. In cooperative multitasking, putting something on an unbuffered channel is essentially a yield().

An awful lot of day-to-day programming is servicing requests. That could be HTTP, an RPC (eg gRPC, Thrift) or otherwise. For this kind of model IMHO you almost never want to be dealing with thread primitives in application code. It's a recipe for disaster. It's so easy to make mistakes. Plus, you often need to make expensive calls of your own (eg reading from or writing to a data store of some kind) so there's no really a performance benefit.

That's what makes cooperative async/await so good for application code. The system should provide compatible APIs for doing network requests (etc). You never have to worry about out-of-order processing, mutexes, thread pool starvation or a million other issues.

Which brings me to the more complicated case of buffered channels. IME buffered channels are almost always a premature optimization that is often hiding concurrency issues. As in if that buffered channels fills up you may deadlock where you otherwise wouldn't if the buffer wasn't full. That can be hard to test for or find until it happens in production.

But let's revisit why you're optimizing this with a buffered channel. It's rare that you're CPU-bound. If the channel consumer talks to the network any perceived benefit of concurrency is automatically gone.

So async/await doesn't allow you to buffer and create bugs for little benefit and otherwise acts like unbuffered channels. That's why I think it's a superior programming model for most applications.

Google has over the years tried to get several new languages off the ground. Go is by far the most successful.

What I find fascinating is that all of them that come to mind were conceived by people who didn't really understand the space they were operating in and/or had no clear idea of what problem the language solved.

There was Dart, which was originally intended to be shipped as a VM in Chrome until the Chrome team said no.

But Go was originally designed as a systems programming language. There's a lot of historical revisionism around this now but I guarantee you it was. And what's surprising about that is that having GC makes that an immediate non-starter. Yet it happened anyway.

The other big surprise for me was that Go launched without external dependencies as a first-class citizen of the Go ecosystem. For the longest time there were two methods of declaring them: either with URLs (usually Github) in the import statements or with badly supported manifests. Like just copy what Maven did for Java. Not the bloated XML of course.

But Go has done many things right like having a fairly simple (and thus fast to compile) syntax, shipping with gofmt from the start and favoring error return types over exceptions, even though it's kind of verbose (and Rust's matching is IMHO superior).

Channels were a nice idea but I've become convinced that cooperative async-await is a superior programming model.

Anyway, Go never became the C replacement the team set out to make. If anything, it's a better Python in many ways.

Good luck to Ian in whatever comes next. I certainly understand the issues he faced, which is essentially managing political infighting and fiefdoms.

Disclaimer: Xoogler.

A lot of the time, a lack of bugfixes comes from the incentive structure management has created. Specifically, you rarely get rewarded for fixing things. You get rewarded for shipping new things. In effect, you're punished for fixing things because that's time you're not shipping new things.

Ownership is another one. For example, product teams who are responsible for shipping new things but support for existing things get increasingly pushed onto support teams. This is really a consequence of the same incentive structure.

This is partially why I don't think that all subscription software is bad. The Adobe end of the spectrum is bad. The Jetbrains end is good. There is value in creating good, reliable software. If your only source of revenue is new sales then bugs are even less of a priority until it's so bad it makes your software virtually unusuable. And usually it took a long while to get there with many ignored warnings.

Xoogler here. I never worked on Fuchsia (or Android) but I knew a bunch of people who did and in other ways I was kinda adjacent to them and platforms in general.

Some have suggested Fuchsia was never intended to replace Android. That's either a much later pivot (after I left Google) or it's historical revisionism. It absolutely was intended to replace Android and a bunch of ex-Android people were involved with it from the start. The basic premise was:

1. Linux's driver situation for Android is fundamentally broken and (in the opinion of the Fuchsia team) cannot be fixed. Windows, for example, spent a lot of time on this issue to isolate issues within drivers to avoid kernel panics. Also, Microsoft created a relatively stable ABI for drivers. Linux doesn't do that. The process of upstreaming drivers is tedious and (IIRC) it often doesn't happen; and

2. (Again, in the opinion of the Fuchsia team) Android needed an ecosystem reset. I think this was a little more vague and, from what I could gather, meant different things to different people. But Android has a strange architecture. Certain parts are in the AOSP but an increasing amount was in what was then called Google Play Services. IIRC, an example was an SSL library. AOSP had one. Play had one.

Fuchsia, at least at the time, pretty much moved everything (including drivers) from kernel space into user space. More broadly. Fuchsia can be viewed in a similar way to, say, Plan9 and micro-kernel architectures as a whole. Some think this can work. Some people who are way more knowledgeable and experienced on OS design seem to be pretty vocal saying it can't because of the context-switching. You can find such treatises online.

In my opinion, Fuchsia always struck me as one of those greenfield vanity projects meant to keep very senior engineers. Put another way: it was a solution in search of a problem. You can argue the flaws in Android architecture are real but remember, Google doesn't control the hardware. At that time at least, it was Samsung. It probably still is. Samsung doesn't like being beholden to Google. They've tried (and failed) to create their own OS. Why would they abandon one ecosystem they don't control for another they don't control? If you can't answer that, then you shouldn't be investing billions (quite literally) into the project.

Stepping back a bit, Eric Schmidt when he was CEO seemed to hold the view that ChromeOS and Android could coexist. They could compete with one another. There was no need to "unify" them. So often, such efforts to unify different projects just lead to billions of dollars spent, years of stagnation and a product that is the lowest common denominator of the things it "unified". I personally thought it was smart not to bother but I also suspect at some point someone would because that's always what happens. Microsoft completely missed the mobile revolution by trying to unify everything under Windows OS. Apple were smart to leave iOS and MacOS separate.

The only fruit of this investment and a decade of effort by now is Nest devices. I believe they tried (and failed) to embed themselves with Chromecast

But I imagine a whole bunch of people got promoted and isn't that the real point?

You're alluding to the double taxation problem with dividends. This is a problem and has had a bunch of bad solutions (eg the passthrough tax break from 2017) when in fact the solution is incredibly simple.

In Australia, dividends come with what are called "franking credits". Imagine a company has a $1 billion profit and wants to pay that out as a dividend. The corporate tax rate is 30%. $700M is paid to shareholders. It comes wiht $300m (30%) in franking credits.

Let's say you own 1% of this company. When you do your taxes, you've made $10M in gross income (1% of $1B), been paid $7M and have $3M in tax credits. If your tax rate is 40% then you owe $4M on that $10M but you have already effectively paid $3M on that already.

The point is, the net tax rate on your $10M gross payout is still whatever your marginal tax rate is. There is no double taxaation.

That being said, dividends have largely fallen out of favor in favor of share buybacks. Some of those reasons are:

1. It's discretionary. Not every shareholders wants the income. Selling on the open market lets you choose if you want money or not;

2. Share buybacks are capital gains and generally enjoy lower tax rates than income;

3. Reducing the pool of available shares puts upward pressure on the share price; and

4. Double taxation of dividends.

There are some who demonize share buybacks specifically. I'm not one of them. It's simply a vehicle for returning money to shareholders, functionally very similar to dividends. My problem is doing either to the point of destroying the business.

Reinvesting in the company is the one thing we should absolutely subsidize. That goes to wages, capital expenditure and other measures to sustain and grow the company.

Paying out dividends and doing share buybacks just strips the company for cash until there's nothing of value left. It's why entshittification is a thing.

I'm all for root cause analysis. A big part of that is that large companies become extremely risk-tolerant because history has shown there is little to no downside to their actions. If the government always bails you out, what incentive is there to be prudent? You may as well fly close to the Sun and pay out big bonuses now. Insolvency is a "next quarter" problem.

I'm aware that TARP funds were repaid. Still, a bunch of that money went straight into bonuses [1]. Honestly, I'd rather the company be seized, restructured and sold.

You know who ends up making sacrifices to keep a company afloat? The labor force. After 2008, auto workers took voluntary pay cuts, gave up benefits and otherwise did what they could to keep the company afloat, benefits it took them ~15 years to fight to get back. In a just world, executive compensation would go down to $1 until such a time that labor sacrifices are repaid.

[1] https://www.theguardian.com/business/2009/jul/30/bank-bonuse...

[flagged]

Speaking as a former Google Fiber software engineer, I'm honestly surprised this is still around.

In 2017, basically all the Google Fiber software teams went on hiatus (mine included). I can't speak to the timing or rationale but my theory is that the Google leadership couldn't decide if the future of Internet was wired or wireless and a huge investment in wired may be invalidated if the future Internet was wired so rather than guessing wrong, the leadership simply decided to definitely lose by mothballing the whole thing.

At that time, several proposed cities were put on hiatus, some of which had already hired local people. In 2019, Google Fiber exited Louisville, KY, paying penalties for doing so [1]. That really seemed like the end.

I also speculated that Google had tried or was trying to sell the whole thing. I do wonder if the resurrection it seems to have undergone is simply a result of the inability to find a buyer. I have no information to suggest that one way or the other.

There were missteps along the way. A big example was the TV software that was originally an acquisition, SageTV [2]. Somebody decided it would be a good idea to completely rewrite this Java app into Web technologies on an embedded Chrome instance on a memory-limited embedded CPU in a set-top box. Originally planned to take 6 months, it took (IIRC) 3.5+ years.

But that didn't actually matter at all in the grand scheme of things because the biggest problem and the biggest cost was physical network infrastructure. It is incredibly expensive and most of the issues are hyperlocal (eg soil conditions, city ordinances) as well as decades of lobbying by ISPs of state and local governments to create barriers against competition.

[1] https://arstechnica.com/information-technology/2019/04/googl...

[2] https://arstechnica.com/information-technology/2011/06/googl...

Sorry but this is just incorrect on many fronts. I can speak to this issue as a former engineer on Google Fiber so I got to see just how the sausage was made.

Existing national ISPs just have inbuilt advantages that a newcomer cannot replicate or can't replicate cheaply. This is the result of decades of lobbying state and local governments.

Take something as simple as where you run cables in the streets. You basically have two choices: you dig trenches or you string up cables on a pole. There is no best answer here as it depends on a lot of factors like weather and climate, local soil conditions, natural disaster risks (eg wildfires, earthquakes), distances involved and existing infrastructure and legislation.

So imagine in a given area trenching is uneconomical. This could be just because there's a lot of limestone rock in the soil so it's difficult, slow and expensive to actually dig the trenches and this may be complicated by local noise ordinances, permitting, surveying, existing trenches and so on. So you end up stringing up cables on poles.

Who owns those poles? Is it the city? Is it AT&T? You may have rights to string up cables on those poles but the devil really is in the details. You might have to apply for a permit for each and every pole separately. They might be approved by the city but then how does the work happen? Can you do it? Maybe. But you might need AT&T (or whoever) to do something first like move their own cables. Maybe several other companies have to move cables first. Maybe each company has 90 days to do that work and this can add up so it can take over a year just to be able to put a cable up on a pole. And you can't really do any work until all the poles are available. That's just how fiber works.

And where do you run the fiber too? Do you run it back possibly several miles to a POP? There are advantages in that but obvious disadvantages like cost and just overall cable size and weight. Or do you use local substations? If so, what kind of building is that? Is it a large building that residents find "ugly" and object to on aesthetic grounds or maybe even environmental grounds that means more delays? How much does that cost? Is AT&T grandfathered in with their substations and nodes?

And then after you've done all that and you have your last mile fiber, how many customers do you get? Roughly 30-40% of houses get fiber by how many companies are you splitting that pool with? You have to amortize your entire network build over your projected customer base and it makes a massive difference if it's 10% of dwellings or 15% or 30% or 40%.

In industry parlance this is called an "overbuild" and is inherently economically inefficient. It'll actually raise the cost of every ISP because each will get a lower overall take up rate.

That's why the best solution is municipal broadband that either provides service or acts as a wholesaler to virtual ISPs.

The cost of running a fiber cable from a POP to a house has only gone up over th eyears and it's the majority of your cost. That's really why Internet costs haven't come down. And also why the best Internet in the US is municipal broadband and it isn't even close.

Network protocls are slow to change. Just look at IPv6 adoption. Some of this is for good reason. Some isn't. Because of everything from threat reduction to lack of imagination equipment at every step of the process will tend to throw away anything that looks weird, a process somebody coined as ossification. You'll be surprised how long-lasting some of these things are.

Story time: I worked on Google Fiber years ago. One of the things I worked on was on services to support the TV product. Now if you know anything about video delivery over IP you know you have lots of choices. There are also layers like the protocls, the container format and the transport protocol. The TV product, for whatever reason, used a transport protocol called MPEG2-TS (Transport Streams).

What is that? It's a CBR (constant bit rate) protocol that stuffs 7 188 byte payloads into a single UDP packet that was (IPv4) multicast. Why 7? Well because 7 payloads (plus headers) was under 1500 bytes and you start to run into problems with any IP network once you have larger packets than that (ie an MTU of 1500 or 1536 is pretty standard). This is a big issue with high bandwidth NICs such that you have things like Jumbo frames to increase throughput and decrease CPU overhead but support is sketchy on a hetergenous network.

Why 188 byte payloads? For compatibility with Asynchronous Transfer Mode ("ATM"), a long-dead fixed-packet size protocol (53 byte packets including 48 bytes of payload IIRC; I'm not sure how you get from 48 to 188 because 4x48=192) designed for fiber networks. I kind of thought of it was Fiber Channel 2.0. I'm not sure that's correct however.

But my point is that this was an entirely owned and operated Google network and it still had 20-30+ year old decisions impacting its architecture.

Back to Homa, three thoughts:

1. Focusing on at-least once delivery instead of at-most once delivery seems like a good goal. It allows you to send the same packet twice. Plus you're worried about data offset, not ACKing each specific packet;

2. Priority never seems to work out. Like this has been tried. IP has an urgent bit. You have QoS on even consumer routers. If you're saying it's fine to discard a packet then what happens to that data if the receiver is still expecting it? It's well-intentioned but I suspect it just won't work in practice, like it never has previously;

3. Lack of connections also means lack of a standard model for encryption (ie SSL). Yes, encryption still matters inside a data center on purely internal connections;

4. QUIC (HTTP3) has become the de-facto standard for this sort of thing, although it's largely implementing your own connections in userspace over UDP; and

5. A ton of hardware has been built to optimize TCP and offload as much as possible from the CPU (eg checksumming packets). You see this effect with QUIC. It has significantly higher CPU overhad per payload byte than TCP does. Now maybe it'll catch up over time. It may also change as QUIC gets migrated into the Linux kernel (which is an ongoing project) and other OSs.

no object allocated as part of a request should survive longer than the request itself

So I've spent a lot of time doing Hack (and PHP) as well as Java, Python and other languages. For me, as far as serving HTTP requests goes, Hack/PHP are almost the perfect language. Why?

1. A stateless functional core. There's no loading of large libraries, which is an issue with Python and Java in certain paradigms. The core API us just functions that mean startup costs for a non-stateful service are near zero;

2. The model, as alluded to the above quote, basically creates temporary objects and then tears everything down at the end of the request. It's so much more difficult to leak resources this way as opposed to, say, a stateful Java or C++ server. PHP got a lot of hate unjustly for its "global" scope when in fact it's not global at all. "Global" in PHP/Hack is simply request-scoped and pretty much every language offers request-scoping;

3. There's no threading. Hack, in particular, uses a cooperative async/await model. Where you'd normally create threads (eg making a network request), that's handled by the runtime to make an async/await call out of non-blocking I/O. You never have to deal with mutexes, thread starvation, thread pools, lock ups, etc. You never want to deal with that in "application" or "product" code. Never.

So this article is specific to Ruby-on-Rails, which obviously still has persistent objects, hence the need for GC still.

How Facebook deals with this is kinda interesting. Most FB product code uses an in-memory write-through graph database (called TAO, backed to MySQL). There is an entity model in Hack on top of this that does a whole bunch of stuff like enforcing privacy (ie you basically never talk to TAO directly and if you do, you're going to have to explain why that's necessary, and you absolutely never talk to MySQL directly).

But the point is that persistent entities are request-scoped as well (unlike RoR I guess?).

This mentions Jupiter generations, which I think is about 10-15 years old at this point. It doesn't really talk about what existed before so it's not really 25 years of history here. I want to say "Watchtower" was before Jupiter? but honestly it's been about a decade since I read anything about it.

Google's DC networking is interesting because of how deeply integrated it is into the entire software stack. Click on some of the links and you'll see it mentions SDN (Software Defined Network). This is so Borg instances can talk to each other within the same service at high throughput and low latency. 8-10 years ago this was (IIRC) 40Gbps connections. It's probably 100Gbps now but that's just a guess.

But the networking is also integrated into global services like traffic management to handle, say, DDoS attacks.

Anyway, from reading this it doesn't sound like Google is abandoning their custom TPU silicon (ie it talks about the upcoming A3 Ultra and Trillium). So where does NVidia ConnectX fit in? AFAICT that's just the NIC they're plugging into Jupiter. That's probably what enables (or will enable) 100Gbps connections between servers. Yes, 100GbE optical NICs have existed for a long time. I would assume that NVidia produce better ones in terms of price, performance, size, power usage and/or heat produced.

Disclaimer: Xoogler. I didn't work in networking though.

At Google, I worked on a pure JS Speedtest. At the time, Ookla was still Flash-based so wouldn't work on Chromebooks. That was a problem for installers to verify an installation. I learned a lot about how TCP (I realize QUIC is UDP) responds to various factors.

I look at this article and consider the result pretty much as expected. Why? Because it pushes the flow control out of the kernel (and possibly network adapters) into userspace. TCP has flow-control and sequencing. QUICK makes you manage that yourself (sort of).

Now there can be good reasons to do that. TCP congestion control is famously out-of-date with modern connection speeds, leading to newer algorithms like BRR [1] but it comes at a cost.

But here's my biggest takeaway from all that and it's something so rarely accounted for in network testing, testing Web applications and so on: latency.

Anyone who lives in Asia or Australia should relate to this. 100ms RTT latency can be devastating. It can take something that is completely responsive to utterly unusable. It slows down the bandwidth a connection can support (because of the windows) and make it less responsive to errors and congestion control efforts (both up and down).

I would strongly urge anyone testing a network or Web application to run tests where they randomly add 100ms to the latency [2].

My point in bringing this up is that the overhead of QUIC may not practically matter because your effective bandwidth over a single TCP connection (or QUICK stream) may be MUCH lower than your actual raw bandwidth. Put another way, 45% extra data may still be a win because managing your own congestion control might give you higher effective speed over between two parties.

[1] https://atoonk.medium.com/tcp-bbr-exploring-tcp-congestion-c...

[2] https://bencane.com/simulating-network-latency-for-testing-i...