Google's infamous internal 2010 “I just want to serve 5TB” video now public

https://www.youtube.com/watch?v=3t6L-FlfeaI

by raldi • 5 years ago

1130 503 5 years ago

youtube.com

raldi4• 5 years ago

Background: https://rachelbythebay.com/w/2021/10/30/5tb/

This video was hugely influential on changing the way Google does internal tools and operations.

justicezyx• 5 years ago

I worked at TI, Planet and later Borg, I did not feel much influence of this video other a chuckle. Or I might be too low level to perceive.

jeffbee4• 5 years ago

I think it was a very common perception among application-level SWEs and SREs that TI, platforms, and Borg did not themselves use the stack enough to perceive its flaws.

quelltext• 5 years ago

How did things change?

compiler-guy6• 5 years ago

The most obvious change that came from this video is that Google abandoned the Borgmon readability requirement. At Google, every change needs approval from someone who has passed a detailed style-guide review process in the given language.

Now over multiple changes, it used to require one fairly big one. It's still a pain in the languages that require it--which is all the main ones, but very few of the niche ones.

Many other things changed as well. Much of what the video complains about got automated and better documented. But the company has grown so much, and the product lines have diversified so dramatically, that there are still plenty of places to complain about the overhead.

mathteddybear2• 5 years ago

Broadly speaking, there are tools to automate this or that, some technologies are getting deprecated and replaced by new ones

Also probably the privacy review could be a bigger bottleneck these days ;-)

scottlamb2• 5 years ago

I think you can read about some of these changes in Google's SRE and SWE books (even if they don't mention this video in particular), at least the ones most likely to be interesting to someone outside Google.

But dropping Borgmon readability was the most immediate and obvious. It was basically true that no one had Borgmon readability. The policy was a catch-22: you couldn't get readability for the simple/formulaic Borgmon macro invocations that were encouraged and often sufficient. You could only get it for doing something "clever". I got it by writing fancy borgmon rules to paper over a problem that (in hindsight) I should have solved elsewhere.

Another was easing quota management. IMHO the most unbelievable thing in the video was that after Broccoli Man told Panda Woman to get quota in two cells, she just said "done". Besides the hassle in transcribing what you needed into the request system [1], various types of quota were chronically unavailable where you needed them, even in tiny amounts. In 2010, I kept a critical infrastructure service running by regularly IMing major clients' on-calls asking them to donate 0.1 cpu(!) of their quota in some cell or another when I didn't have quite enough to grow. There was a "gray market" mailing list where people would trade resources they couldn't get through the primary system. But eventually, they built a system that for small services would make the quota just happen for you.

Overall, it was a kick in the pants for the most basic infrastructure teams that made them see how unnecessarily hard this is for their internal customers, prompting them to make small things just happen while keeping large things possible. In any large organization, it's healthy to get this kind of feedback regularly. The actual specific changes and technologies are pretty specific to Google in 2010...

[1] Many people managed this very very tediously with spreadsheets. I eventually wrote a tool to generate the requests based on comparing your intended production config with your current quota.

vechagup• 5 years ago

There's been a big investment in server platforms that strive to enable SWEs to build a new service that follows Best Practices with as little knowledge and handholding as possible. These consist of conformance tests that yell at you while you're coding if you are trying something generally thought to be bad, and semi-automated workflows that help you bring your code to production. When everything works as intended, the production workflows set up a decent set of alerts, acquire resources, configure CI/CD pipelines, and launch your jobs with just a few button presses on your part. (In practice, one of the steps will probably require debugging, but eh, it seems way better than the broccoli man video.)

jrockway• 5 years ago

I feel like people forgot after about five years. I remember wasting a week filling out various pieces of paperwork and submitting byzantine configuration CLs so that some contractor would have permission to view a certain webpage through the corporate proxy. (I think what made me most mad is that regular employees could view the website with no additional configuration. I can understand if I was filling out tickets to get approval, or a security review, but the actual configuration of the proxy had to change to allow this, in addition to getting all of those approvals!) My team didn't make the website, and the contractor didn't work on my team, so I'm honestly not sure why I was involved. I just remember being annoyed about it. I'm sure there are some memes about it in the archive.

ikiris7• 5 years ago

Contractor access is its own hellscape.

hnov• 5 years ago

This is paradoxically because historically everything was wide open to anyone so access-control and such isn't super fleshed out for most apps behind the proxy. Random internal app X could have been conceived and built with very little oversight and opening it up to a rotating cast of temporary workers is seen as an unnecessary risk. Broadly used apps (e.g. the bug ticket system) tend to have app-level security-controls and are not blocked by the proxy for contractors.

inoffensivename2• 5 years ago

It was hugely influential in identifying the frustration of getting things done at Google. In my experience it's even more true now than it was back then, the number of things you have to deal with has just grown. I've been at Google since 2006 and I feel like I'm losing my mind with all the complexity.

jez• 5 years ago

Out of genuine curiosity, what keeps you at Google for 15 years despite perceived increase in complexity to getting things done? I'm wondering whether the answer is like "there's a lot of complexity, but I like the work I do more" or "I like the people more" or some other reason.

sicromoft• 5 years ago

See also the recent discussion here of "I Don’t Know How To Count That Low": https://news.ycombinator.com/item?id=28988281

throwawayamz27• 5 years ago

This is not true, Google is no longer capable of changing anything internally, it's like a government in that respect, it can make a new process/program (that might on the surface make some things look better) but underneath the old system is still there.

kevin_thibedeau• 5 years ago

Well they can launch new semi-independent initiatives under the Alphabet umbrella. They just need to use that to create a web search engine. Something sorely lacking in their current portfolio.

mikewarot• 5 years ago

I didn't realize this was made by Google in the first place when I saw it a few days ago. I hope things are simpler now, but I doubt it.

xiphias2• 5 years ago

I wish I would have had this video before 2010. I got paged at night every time there was a PCR failover, and I didn’t know what to do with it. This video is better than all the extensive documentation that we had.

Imnimo• 5 years ago

What I don't get is why they wouldn't just use MongoDB. MongoDB is web-scale.

nostrademons4• 5 years ago

That was a major impetus for this video, IIRC. The "MongoDB is web-scale" video went around Google about a month before Broccoli Man and some enterprising Googler figured they could use the same software to make a satire of Google's internal tools.

hedgehog• 5 years ago

Link for the Mongo video: https://www.youtube.com/watch?v=b2F-DItXtZs

And bonus lean startup video: https://www.youtube.com/watch?v=3J9KhpgYVB0

fragmede• 5 years ago

MongoDB is web-scale: https://www.youtube.com/watch?v=b2F-DItXtZs

NSFWish; it gets a bit personal around 3:11

vorticalbox2• 5 years ago

Maybe because mongodb had been out less than a year in 2010?

gnabgib• 5 years ago

I think you missed the /s from GP.

323• 5 years ago

But is it planet-scale?

swalsh• 5 years ago

That's out of date, we're now in the days of IPFS.

hinkley• 5 years ago

/dev/null is also web-scale

DeepYogurt• 5 years ago

Is /dev/null fast? I will use /dev/null if it is fast.

sondr3• 5 years ago

And available as a SaaS: https://devnull-as-a-service.com/

sergiotapia• 5 years ago

Much more infamous.

"Mongo DB Is Web Scale" - https://www.youtube.com/watch?v=b2F-DItXtZs

keymone• 5 years ago

s/in//

shirleyquirk• 5 years ago

famous ternal?

birken• 5 years ago

Hey... those of us that worked on Google's internal Bigtable service worked very hard so you didn't have a file to a ticket to set up replication between your Bigtable cells.

The rest does seem about accurate though.

cletus• 5 years ago

Ah, this takes me back (disclaimer: Xooger, 2010-2017). It's painful and funny because it's true. Or was true.

Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.

leg• 5 years ago

It is true that Borgmon readability went away due to this video. It wasn't Sergey, it was an eng director.

ikiris7• 5 years ago

"no one has borgmon readability". years later and i still die laughing.

DaiPlusPlus• 5 years ago

Pray tell, what is/was Borgmon?

twinge• 5 years ago

A system for alerting based on time-series data, with its own rule language. The language (along with many others) required authors have demonstrated they can adhere to the style guide by going through a process to obtain readability.

https://sre.google/sre-book/practical-alerting/

sleepydog• 5 years ago

It's a language and supporting infrastructure for collecting and querying time series data for monitoring.

It was replaced a long time ago by a new system called monarch, but a few holdouts will probably continue using borgmon until the heat death of the universe.

dekhn8• 5 years ago

prometheus 0.1

taldo• 5 years ago

Ah, the laughs (Xoogler since 2020). It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.

dilyevsky3• 5 years ago

That video came out a few years before flex appeared I think at a time they were having a sort of “resource crunch” on the heels of growth spur following the GFC.

willidiots• 5 years ago

Flex was available for certain things (Colossus IIRC, gave you a ton of flex quota) but for others it wasn't. Because This is Google.

the-rc7• 5 years ago

It wasn't that there was a crunch — that had always existed. There just wasn't all the tooling to implement anything like flex. At least this video was made after "buying Borg quota" was a normal thing. Before it, you had to "buy" regular machines and donate/assimilate them into Borg. Then after X days you'd receive your quota, minus a Borg "tax" of 10% to cover borglet and system daemons' overhead.

dekhn8• 5 years ago

you left out monitoring for reliability which is a major part of this video

packetslave3• 5 years ago

automon is (was?) a magical thing

bhickey• 5 years ago

About six years ago I had a resource manager deny me a database instance the very same day it became available for flex in another product area. I tried to "Hey Mister" resources from someone in that group to no avail. Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.

plaguuuuuu2• 5 years ago

Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.

That's diabolical. I knew Google should never have dropped the "do not be evil" thing

jedberg• 5 years ago

The same conversation at Netflix 10 years ago:

I want to serve 5TB of data.

Ok, spin up an instance in AWS and put it there.

I want it production ready.

Ok, replicate it to a second instance. If it breaks we'll page you to fix it.

The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.

ignoramous2• 5 years ago

I want to serve 5TB of data. Ok, spin up an instance in AWS and put it there... it was amusing when we hired people from Google who were confused by the lack of process and approvals.

Quoting from Velocity in Software Engineering https://queue.acm.org/detail.cfm?id=3352692:

In 2003, at a time in Amazon's history when we were particularly frustrated by our speed of software engineering, we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other, yet he remained deeply impatient and complained loudly and with great clarity about how hard it was to get anything done. He wrote a six-pager that had a great hook in the first paragraph: "To many of us Amazon feels more like a tectonic plate than an F-16."

Matt's paper had many recommendations... including the maximization of autonomy for teams and for the services operated by those teams by the adoption of REST-style interfaces, platform standardization, removal of roadblocks or gatekeepers (high-friction bureaucracy), and continuous deployment of isolated components. He also called... for an enduring performance indicator based on the percentage of their time that software engineers spent building rather than doing other tasks. Builders want to build, and Matt's timely recommendations influenced the forging of Amazon's technology brand as "the best place where builders can build."

...leading up to the creation of AWS.

ryandrake• 5 years ago

The "approval paralysis" thing happens at a lot of companies, large and small, not just GiantTech. It creeps up on you slowly: 1. A big problem happens that gains the attention of leadership. 2. The problem is root-caused to some risky thing an employee did trying to accomplish XYZ. 3. To correct this, a process is put in place that must be followed when one wants to do XYZ, and (critically) gatekeepers are anointed who must approve the activity. 4. These gatekeepers are inevitably senior already-busy people who become bottlenecks. Now we can't do this critical thing without hounding approvers. 5. Some other big problem happens and the above cycle starts all over again.

Before you know it, every even slightly risky task you need to do through the course of your job requires the blessing of approvers who are well-intentioned, but all so overloaded they don't even answer their E-mail or chats. They sometimes need to be physically grabbed in the hallway in order to unblock your project. Progress grinds to a halt and it still has not stopped production problems--just those particular classes of problems that the approval processes caught.

EDIT: Not sure what the right solution is, but it must be one that doesn't rely on a particular overloaded human doing something. Maybe an automated approval system that produces a paper trail (to help with postmortem and corrective action later) and ensuring all changes can be rolled back effortlessly. Easier said than done, obviously.

jll29• 5 years ago

we turned to Matt Round, an engineering leader who was a most interesting squeaky wheel in that his team appeared to get more done than any other

Matt went on to study theology, and he's started a church community in Scotland: https://www.linkedin.com/in/mattround/

"Leader Company Name: Hope City Church Edinburgh Dates Employed: Sep 2017 – Present Driving a new church start-up."

bcoughlan• 5 years ago

What does "platform standardization" mean in this context?

GauntletWizard• 5 years ago

Holy crap! I've been asking for this for forever[1]. Thank you to the leaker!

[1] https://news.ycombinator.com/item?id=21786729

benley• 5 years ago

You're welcome <3

Here's hoping Google doesn't get mad about it - though after 12(ish) years there's really nothing secret in that video.

jazzyjackson• 5 years ago

what am I looking at here

EDIT: it has been explained to me: https://rachelbythebay.com/w/2021/10/30/5tb/

w0mbat2• 5 years ago

When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable. Eventually it all got rewritten by other people to do things properly like the video says.

dekhn8• 5 years ago

I managed to deploy a whole system at google that had the ability take down all of google globally by DoS'ing the network, and ran it casually (IE, starting and stopping it when I felt like it, at the capacity I felt like, with the binary versions I wanted) for 3 years.

In retrospect, this was absolutely crazy! The actual visible outcomes were: 1 cluster drained due to heat rising so fast the alerting thought there was a fire, 1 page to an engineer in the middle of the night (sorry discovery-service) and a whole bunch of complaints about CPU stealing that weren't my fault.

Those were the good old days.

robocat• 5 years ago

Rachel said something similar “My own ‘solution’ to it after far too much thrashing was just to say ‘we cannot get all N types of quota in the same place so we are at the mercy of whatever happens to be available, and if that dries up, we stop running’. Granted, this was for some internal stuff that was seven or eight levels removed from anything that anyone on the outside might ever see, but still, it was stupid and made me feel so dirty. I'm sure my non-solution probably bit someone later. Sorry, whoever.” — https://rachelbythebay.com/w/2021/10/30/5tb/

hiepph• 5 years ago

Isn't that things are supposed to be? Prototype and then polish later. I think teams should work in layer like this, but I guess with big company they're just too lazy and assume you're smart enough and have time to take care of everything.

w0mbat2• 5 years ago

Right. Time was of the essence with this service, and it was still very reliable and scalable in its original simple form, because it leaned on a lot of existing Google infrastructure. Later on it got upgraded to follow the rules better.

dpryden2• 5 years ago

Non-Googler: What do all those words mean?

Noogler: Haha, this video is so funny!

L4 SWE: (Crying because the video is so true)

L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.

L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...

throw12346512342• 5 years ago

Non-Googler: What do all those words mean?

Exactly. This wasn't too relatable, even though I have the GCP Certified Architect cert.

jjtheblunt2• 5 years ago

As an ex Apple person, i'd say it means there's way too much hierarchy at Google? not sure i'm reading it right though

dpryden2• 5 years ago

I can't tell if this comment is implying that my comment is unclear, or if you're agreeing with the first line of my comment.

In either case, though, it's an inside joke precisely because it's more relatable to those who are (or were) inside. In particular, I think it would be most funny to someone who was at Google about a decade ago; when I left Google in 2017 things had already changed enough that this didn't ring quite as true for new hires.

That said, GCP is not very representative of what the internal platform looked like circa 2010. (Or even of what the internal platform looks like now, as far as I know.)

NikolaeVarius• 5 years ago

Why would internal tooling mean anything to you? And why would GCP knowledge be useful in any way?

Its fairly simple to extract the gist of what these systems from the script.

vanderZwan• 5 years ago

Not sure if "SWE" stands for software engineer, or "Sweden" as in Stockholm Syndrome

keville• 5 years ago

Why not both? :sob:

frakkingcylons2• 5 years ago

Oh definitely Sweden.

praptak• 5 years ago

Random synapse activation:

A few years ago there was a Swedish tourist at a hotel where I was on vacation. He had a blue-yellow hat with "SWE" written on it in Courier font. I felt an urge to steal his hat because it looked better than most of the Google-branded swag I got as a Google SWE :)

belter• 5 years ago

L7+ SWE

My life is a waste but the money is too good...

kubb• 5 years ago

This applies to every level, particularly the lower levels.

tandr2• 5 years ago

How good?

cperciva• 5 years ago

Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale?

raldi4• 5 years ago

Or: These are really good points for a visibly-user-facing post-alpha service, but isn't it a bit overengineered for an experimental internal service whose clients can tolerate the risk of occasional downtime?

nostrademons4• 5 years ago

L9+

SilasX7• 5 years ago

Yeah, that was my reaction. I get the need for all this reliability/failover, but it's horrible failure of abstraction/separation of concerns.

There's no reason the serving team should have to learn how to do all of those things on the checklist, since it can be done by anyone who's already learned the infra. You're expecting them to learn all kinds of stuff outside of their specialty, when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated.

nuerow• 5 years ago

Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale?

My guess it fits nowhere because the L5s don't have the ability to automate it, and the L6s think it's trivial and as it's done sparingly then it doesn't justify the work to do things differently.

And this is why we can't have nice things.

omreaderhn• 5 years ago

That would be 'Xoogler' because Google's engineering and broader corporate culture does not reward work like that and so when you realize that, you leave.

In general, Googlers have very little idea how far behind the rest of the industry they are when it comes to tooling.

I am a Xoogler.

GeneralMayhem2• 5 years ago

I think that was more or less the intended response. And ten years on, most of these things are automated. This video was a kick in the pants internally.

cmrdporcupine• 5 years ago

imho the Google interview process selects for people who thrive on organizational challenges.

throwaway20371• 5 years ago

These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.

calmlynarczyk• 5 years ago

I work at a global corporation with 50,000 employees. Even though I've never been at Google I felt every pain point this video was getting at because our company is trying to implement all of this stuff right now.

"Oh you want to go to production? Here's a list from A-XX stating what you need to accomplish that." Thing is I thought they actually handled this gracefully when I started because lots of requirements were tiered with various criteria you had to meet to move up (mostly for brownie points).

But then one day the Tech Execs lose their minds and decide "everything needs to meet all criteria for every single process." You want to create an S3 bucket to store data? That will be a week of submitting paperwork and another month of meetings and approvals from various teams you've never heard of. Plus you have to register your schema, implement data quality checks, unit tests, regression tests, get a PR and CO approved for your central config change, remediate any CVEs in the tooling that you used, and build all of this using our in-house CI/CD platform we created because we're just soooo special. Now you're allowed to launch. Oh wait, NO because we've put the entire corporation on hold from launching new systems for the last calendar year because we're still trying to agree on the final process everyone needs to follow to go to production.

It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.

acdha• 5 years ago

It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.

Followed by the inevitable ranting about “shadow IT”, AKA the requirements gathering they really should have done.

unethical_ban• 5 years ago

In my previous role, the secdevops groups (matrixed teams) were building custom terraform modules for our devs to use in order to easily deploy compliant AWS infrastructure - and devs could only deploy via terraform/CI-CD. While TF specifically states that custom modules are not meant to be used as wrappers, I thought it was a clever way to try getting security "out of the way" while still enforcing best practices.

oblio3• 5 years ago

It's surreal how universally so many orgs makes the same mistake of trying to throw more and more process at problems.

It's hard to find the right balance. You want a bit of process, but not too much.

But it's one of the hardest problems in the existence of humanity and whoever solves it should probably get all the Nobel prizes available (including peace and chemistry!).

tetha• 5 years ago

Well we're small, but our development is currently starting to build new products and new extensions to their products. I'm pretty happy that everyone is pretty much onboard with our situation #1.

There are a few hard requirements, but most of the requirements we as operations put up are tied to the guaranteed service level agreement to the customer and possibly overall user count.

If there is just an entirely lax service level agreement there might be no need to invest time in clustering non-trivial applications, or implementing more monitoring than a simple HTTP check. On the other hand, if you're selling some 99.95 24/7 with penalties to a customer, the list of must-dos suddenly grows a lot.

The nice thing of approaching it like this: It allows a gradual increase in operational rigidity and robustness. A product team doesn't hit a wall of requirements for their first productive customer. They rather have to incorporate more requirements as the service becomes more successful. Or they don't if the idea doesn't work.

brightsim• 5 years ago

Literally every point you made applies also at my workplace. The optimist in me hopes we work at the same place, but I fear that your last statement might just be the truth :-)

ts4z2• 5 years ago

To be fair, they did, and many things have improved. And this video was used as an uncomfortable reminder to make some of those changes.

TideAd• 5 years ago

My team has issues deploying builds to test machines. It's like 15 steps and takes an hour. The tooling is atrocious and recently got even worse.

We eventually found the team responsible for this (the org structure is hard to penetrate because no one answers emails). They said they had no idea anyone was dissatisfied. Then they said that it was a low priority so they didn't care and nothing would be done.

In my experience, you can usually convince an engineer that their stuff has a problem and they need to fix it. But it's often impossible to convince management if they aren't on the hook for user satisfaction.

m0zg2• 5 years ago

At Google back then "leadership" might as well not even show up. It was super bottom-up, and _you_, not "leadership" were supposed to identify and fix issues. No "leadership" would stop you, either, at least in most cases. I don't believe that in all my years there anyone ever told me what to do. It was very easy to start projects, shut down projects, get headcount, get resources (if your business case is sufficiently persuasive to others). Not a complete free for all, but certainly _a lot_ more freedom than you'd normally see in companies of that size. And (IMO) people used that freedom and autonomy pretty well.

That kinda deteriorated over time, culminating with Sundar "McKinsey" Pichai, and then went rapidly downhill from there, and now I flat out reject their recruiters, based on the feedback from friends still employed there.

devnull3• 5 years ago

At 2:05 the green dude asks if you think your users are scum and do you hate them.

The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.

munk-a• 5 years ago

If the users had properly set up a PCR notification about the change and registered it to a bigdata instance then they would never be surprised about service discontinuations. The moral of the story is that you can't fix stupid users. /s

nunez5• 5 years ago

even more ironic given that google+ came out four years later

Spivak2• 5 years ago

I’m so confused, isn’t this just like basic highly available infrastructure mixed with a toxic SRE culture?

I want to serve 5TB!

Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.

And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.

Are people not used to what it really takes to “just run some code?”

gliese1337• 5 years ago

I am used to it, but

1. It is rare for the details of how to actually accomplish each of those steps to be both documented and the documentation made accessible.

2. If you can describe it that succinctly, it really ought to be automated. If it can't be automated... then you left something out of your instructions, which goes back to point (1).

Spivak2• 5 years ago

Like the steps to do all of this are automated, but we can't read your mind. All of this is basically boils down to submit a PR against some repo that says "there shall be two instances in these regions, there shall be a database in this cluster, there shall be a bucket with this name, etc etc" that the SRE team reviews and merges, which triggers an infra deploy.

q3k6• 5 years ago

People with HA production experience can easily vibe with points made by Broccoli Man. Yes, these things make a lot of sense if you actually want to get code running reliably, especially at scale (organizational and userbase).

But we must not forget how this can look from the point of view of someone who hasn't had to deal with a page due to an entire datacenter going offline, who's not aware of all the hundreds of small things that can go wrong by doing the 'obvious' thing. I think the video is more of a way to poke fun at the optics of this (and some of the overly arcane stuff involved), rather than at the idea of high availability being useless. At least that's how I've always felt about it, a way to remind SREs to respect their internal users (simplify! automate! document!) and that what makes sense to them might look ridiculous to others.

svachalek• 5 years ago

It totally makes sense for Gmail, but at Google "serve 5TB" means something like sort your manager's inbox, something that someone somewhere has an interest in doing, or trying, but of no real consequence for failure.

mlindner2• 5 years ago

Wow I saw this somewhere a long time ago. But I don't remember where and in what context.

jamestimmins2• 5 years ago

As an external user who has found Google's services to be incomprehensible, it's nice to know it is (was) equally as painful internally.

anshumankmr• 5 years ago

What is this exactly?

ts4z2• 5 years ago

Xtranormal was a video service that would animate scripts with some stock characters.

Someone made a bit of internal-only snark, and "I just want to serve 5TB" became an in-joke for turning easy problems into exercises in frustration.

Some of these things have, actually, been addressed.

treebog• 5 years ago

What does “serve 5TB” refer to? They expect 5TB of network bandwidth over some time period (a month?)? Or their database takes up 5TB on disk?

metalliqaz• 5 years ago

i think it just means to put 5TB of data online

drjasonharrison• 5 years ago

If you watch the video, it doesn't matter. It's just something they want to serve.

raldi4• 5 years ago

It's a joke that's sort of open to interpretation.

The most straightforward is, "I just want do this incredibly simple thing; why is it so hard?"

But there's also the level of, "Googlers are so engineeringly pampered that they think serving 5 terabytes is the equivalent of Hello World."

And then there's another level of, "Well, isn't it? After all, this is Google and this is $YEAR."

rachelbythebay3• 5 years ago

Imagine it as "I want to have a http://foo/~me/ type path where I can park 5 TB of stuff and other people can fetch from it when they feel like it".

5 TB of data made available, not 5 TB of transfer/bandwidth/etc.

packetslave3• 5 years ago

IIRC, the impetus for Jon Orwant creating the video was him wanting to make 5TB of data publicly available (the US patent dataset? it was before my time) and all the hassles that were involved.

jamestimmins2• 5 years ago

As a friend of mine explained why she left Google a year or so ago, "I got tired of emailing 30 people to try to figure out who owned a single variable."

raldi4• 5 years ago

The best and worst parts of being a Google engineer: Impossible things are merely very hard, but on the other hand, easy things are also very hard.

nostrademons4• 5 years ago

A TL I worked with once had a simple but effective strategy for that:

"Remove it and see who complains."

I did that (with the impenetrably named "PrefetchExperiment", last touched by a branch that lost previous file history in 2007). Turned out it was the source data for Google's DNS to figure out how to route queries to the lowest-latency datacenter, based on their geographic location. In about a month, it would've taken down all Google services. Oops.

It was a very effective way of figuring out who owned the variable and writing a big long comment explaining what it's there for and which team to contact before changing it, though.

joshuamorton8• 5 years ago

Scream tests are always fun. ("break it and see who screams")

AlexanderTheGr8• 5 years ago

LMAO, isn't this very similar to FaceBook's recent DNS problem?

Also I love the idea of removing it and seeing who complains.

ikiris7• 5 years ago

The hilarious thing is I know exactly what file you're talking about here.

bluefox• 5 years ago

BitTorrent existed since 2001. Get on with the times.

gcampos• 5 years ago

What exactly are these "peer bonuses"? Is it real? Is it what I think it is? Do people actually use them as bargain chips?

q3k6• 5 years ago

What exactly are these "peer bonuses"? Is it real? Is it what I think it is?

Each month, you can nominate another employee for a small bonus. This is designed to be given to coworkers who have gone above and beyond what was expected from them.

Do people actually use them as bargain chips?

From my experience it's so over-the-top absurd that it would be difficult to have someone interpret such an offer as anything other than a joke or a meta-joke.

https://blog.bonus.ly/a-look-at-googles-peer-to-peer-bonus-s...

tazjin2• 5 years ago

It's not each month. You can send a lot of them. There's a theoretical limit and a bunch of restrictions but in practice they're unenforced.

advisedwang• 5 years ago

Yes they are real, it's in the low hundreds of bucks range. They must be approved by the recipients manager. Its also limited how many a employee can send (but the limit is fairly high). There is also "kudos" which comes with no money, but has no limits or approvals required.

They are intended to be used for going above and beyond, not for stuff that falls within the scope of ones job. Using them as bargaining chip is explicitly against policy.

nunez5• 5 years ago

you get some money ($150/bonus, IIRC) for helping someone out, assuming manager approval

akin to the usual corporate "thank you" gift card, but more money and generally easier to distribute

compiler-guy6• 5 years ago

People don't use them as bargaining chips most of the time--it is explicitly against policy. I'm sure it happens some times.

What they do do is send one when someone else does something nice (like fix a bug from a project they have left or whatever else). If you ever need something similar again, the person you peer bonused has warm fuzzies about the experience and a hint that they might get it again.

People also use peer bonuses during perf time to demonstrate that the work they are doing impacts other people enough for a somewhat uncommmon thank you.

B-Con• 5 years ago

You can send a small, semi-official "thanks for a job well done" to someone else and it comes with a few bucks attached. People joke about using them nefariously (as people tend to joke), but I've only seen them used appropriately.

SilasX7• 5 years ago

Wow, kind of funny that Xtranormal now lives on in the few viral videos that were made with it.

Here's where the company is now (the original domain is used for something else now):

https://en.wikipedia.org/wiki/Nawmal

shoeshoeshoey• 5 years ago

Facebook had its own meme: "Pusher I need a hotfix"

dang2• 5 years ago

Recent and related:

I don’t know how to count that low - https://news.ycombinator.com/item?id=28988281 - Oct 2021 (259 comments)

especially this comment: https://news.ycombinator.com/item?id=29032656

martini333• 5 years ago

Google: The Sunk Cost Fallacy

nunez5• 5 years ago

still hilarious

nunez5• 5 years ago

this is making me miss memegen

google had its downs, but wasting hours on memegen was not one of them

dmitrygr• 5 years ago

Ask someone still on the inside and you'll hear that the memegen is a mere shadow of its former self

Xoogler 2012-2019

nunez5• 5 years ago

that's really sad

loxias6• 5 years ago

And now, in 2021, Google has inflicted their "clarity" on the rest of the world. I miss jobs from the 2000s, the jobs where you were paid to write software for a living.

You know, engineering! Given a task, or set of requirements, develop software on your computer, software which eventually runs on the customer's computer, where it's used to solve the customer's problem.

My most recent full time employment a year ago was at a great company. Healthy culture, some of the most talented coworkers I've ever had the pleasure to work along side.

Over the year I lasted there, I used for the first time: Docker, Golang, Kubernetes, Terraform, Gitlab, Saltstack, Prometheus, (and probably other middleware that my brain has GCd to free space). I was barely able to get anything done. At least, it always felt that way.

Maybe I'm just an idiot, I don't know. I'd accept it if true! What I do know is that I used to be able to build things for people, be compensated well for it, and get satisfaction from a customer liking what I've built. It was simple.

In this brave new world, with containers, pods and this and that and the other thing, where it can take months before one even understands enough primitives to do a "hello world".... how can anything ever get done?? How can anything inventive, creative, or experimental emerge from our industry when the develop/test/improve cycle has gone from minutes to weeks or months?

I don't know what the future looks like, but the present strikes me as unsustainable in the long run.

(edit: Wow! I expected this to be downvoted to oblivion, not my highest rated comment on the site...)

<tiny>(Forgive this shameless self promotion: if, dear reader-who-is-a-hiring-manager, you have a paid role for a lowly but experienced systems engineer who doesn't know anything about "web" or "apps" or "social" but is quite adroit with C/C++ (and a few others), most "sciencey/mathy" type problems, signal processing, firmware, network protocols, automation/scripting, and more, ... email is welcome!)</>

VHRanger3• 5 years ago

In my team, we often deploy internal "services" as cronjobs on an EC2 service. This hasn't run into any issues in 24 months.

One of these we decided to move to a more serious infrastructure (a set of AWS lambdas). It's failed three times in 6 months since, and we're moving it back to be a good old cronjob on a server.

Simple is good.

SamuelAdams• 5 years ago

Just curious what your cost differences are between a dedicated EC2 instance and lambdas. For our organization an EC2 instance was at best 8-10 times more expensive than lambdas.

cheeze• 5 years ago

What does the cronjob do? Start some service that listens for inbound connections? Or are you talking more about daemons that do some set of work every interval?

ethbr0• 5 years ago

The current landscape (optimizing for hyperscale, at the cost of complexity) seems like a natural extension of relatively few giant corporations funding the majority of programmers. To such an organization, efficiency & time to market are more important than simplicity.

iamstupidsimple6• 5 years ago

But at least in the FAANG example, time to market is much slower because of said complexity.

thrashh3• 5 years ago

Perhaps we need more specialization but I remember the time before these kind of tools and I hated it.

I’m a lazy person and I absolute love tools. Tools like Docker helped me never have to solve other people’s complex environment problems again. I love metric reporting tools like Prometheus because it helps me front problems before they become weekend emergencies. I use a paid Git GUI so I can fix complex Git problems without ever making a mistake.

pm90• 5 years ago

Same. I do not have any nostalgia for when you had to say into machines and run scripts. Please no.

loxias6• 5 years ago

I'm also a lazy person! Which is why tools like this are a PITA to me.

The one exception is Docker. It's not a regular part of my workflow, because of how it makes things both harder to get started (making a working Dockerfile takes a bit of time), harder to debug, and slower to build (I just changed one line! Now I have to rebuild the whole image to see if it fixed the problem... &c).

However, for deployment of the final product? I agree Docker's GREAT. But, consider, in that respect it offers nothing I didn't have at the start of my career 20 years ago. Static linking for interpreted languages. :)

karolist• 5 years ago

Which paid Git GUI friend?

rvnx• 5 years ago

I tend to agree with the green dude :|

It's normal to have a production service replicated on 2 availability regions.

The green guy is annoying, because reality is annoying, and reliability is not about luck, but is about a properly calibrated and tested process.

Yes, you need to write monitoring, you cannot run only with "hope".

Yes it sucks that a DC can go down. Your particular service is not important if it's down, but having a copy of the production data is essential in case of a catastrophe.

Except for the tests that are probably unnecessary, everything else seems to make sense.

The peer bonuses are an issue though.

ridaj• 5 years ago

I think the challenge is you'd expect a company like Google to have more of the setup be automated. If replication and monitoring are such universally good ideas, then why don't they come out of the box?

midasuni2• 5 years ago

Depends what the problem you’re trying to solve is. In my experience the vast majority of business problems do not need that kind of reliability, and if they do they don’t need it deployed in such a Byzantian way.

novok• 5 years ago

Green guy should be making a all of that a one click process to start up a service shell that does all of that for you although. Then as you write it up an automatic linting & rules engine will highlight what is missing before you make a final pull request to get the necessary human approvals, ONCE.

gopher_space• 5 years ago

A lot of it feels like premature optimization. Like I'm laying down a heavy infrastructure to support change but it's already locking me into certain ways of looking at problems.

loxias6• 5 years ago

All of this is true, but I'd wager there's 10, at MOST, entities on the planet that are large enough to warrant this level of ... "architectural overkill". The other 99.99% of us don't need it.

I CERTAINLY don't debate that Google, or Amazon, or Facebook, or Netflix, or the phone system, or anything else that touches a noticeable percentage of the human race needs architecture like this to provide "5 9s".

But, just like when "big data" became a buzzword, and many people thought their problem needed "big data" approaches to solve, the thought that all but a small minority of entities need this is Simply Wrong.

I am reminded of a client doing something with genomics about 9 years ago. They had some over-complicated "new tool" infested approach to solve their "big data" problem, but the run times were taking too long. I was brought in as a consultant to improve it. After I was done, a data processing run that took hours (causing employees to run them overnight) before took minutes, or seconds. What did I do? I got rid of all the complexity. I replaced their expensive cluster with one studly provisioned machine. I replaced their collection of networked Java microservices with 1 non-networked multithreaded C program. I replaced their XML based format for data at rest with something I whipped up, tuned to what they actually needed.

Once their "we need big data!" >10TB data set could fit in a single machine's memory, the rest was easy. What used to "require" a cluster of machines and overnight processing could be done interactively, and quickly enough for the scientists to get into a much more productive "flow", doing dozens of runs per day.

tl;dr: unless you're google (or google scale) you don't need all this crap. :)

mwcampbell2• 5 years ago

software which eventually runs on the customer's computer

I gather then that you're not a fan of SaaS. True, one can cynically explain the rise of SaaS as rent seeking. But there's undeniably value in selling whatever functionality your software provides without burdening the customer with having to run it on their own computer(s). And when we do that, it's our responsibility to make the service reliable, which is what a lot of these tools are trying to do.

loxias6• 5 years ago

I gather then that you're not a fan of SaaS.

I'm neutral, I think? I don't quite see the point of it would be more accurate. I don't think I've used any SaaS in my personal life (other than streaming services. Which I'd prefer as a local app anyway, and I still do, for music, but not video)

I'm sure it's a matter of opinion, not something with an objective answer, but "burden of running software on their own computers" genuinely confused me as I read your comment, I thought "burden? what burden?".

As a user:

If software is designed properly (and most isn't...) you download it once, and it runs. Is the burden the time it takes to do the download? Compared to the noticeable burden of using a webapp, with problems like crappy and frustrating responsiveness, an inability to work without an internet connection, and frequent inability to handle tasks of real complexity, I'd choose a local program any day.

As an employee:

Heck yes SaaS! $/month >>> $$$/customer :D Of course it's rent seeking, and I take (and give) no shame in that.

mathteddybear2• 5 years ago

Reminder that jobs from the 2000s that you were paid to write software for a living include also J2EE and CORBA projects.

loxias6• 5 years ago

True, true. The mention of J2EE did just made me shudder a bit. ;)

But at least, then, I could write code, run it, make changes, and run it again! And see results! Quickly! My main objection to this "future world" is the vast number of layers of abstraction that you need to fight through just to get your first result.

As you can surely guess from my biases and opinions, my happiest engineering projects are those that only require me, my thinkpad, emacs, some man pages, and a C++ compiler. :D And those are the ones I do in my spare time.

Zababa5• 5 years ago

There's almost some kind of irony on uploading that to youtube, a feeling of "why can't I deploy my service as easily as people can upload videos to youtube?".

mseepgood• 5 years ago

This monotonous speech synthesis is annoying to listen to. The delivery of the jokes is awful. Who can sit through a 3 min video like that?

drannex• 5 years ago

That's what makes this even more funny.

zaphar2• 5 years ago

I think it mostly works best when you've lived it. Which I did. And the resurfacing of that video brought back a lot of memories.

pas• 5 years ago

https://m.youtube.com/watch?v=b2F-DItXtZs

zucked• 5 years ago

This was an output of a free (now defunct) service that used to accept transcripts and pump out these videos with TTS audio. It led to some hilarious results, usually within niche communities. Around ~2010 these things were everywhere.

frakkingcylons2• 5 years ago

For anyone else who'd rather read than watch this video, here's the transcript (from YT's auto-generated captions): https://pastebin.com/8UrFftM6

dekhn8• 5 years ago

Thank you, whomever did this! I asked for it in a comment recently.

This video basically is making fun of a common situation of Google at the time, where a person wants to serve up some data for analytics, but the sysadmins expect the person to follow a process intended for much more complex and high availability services run by teams of skilled engineers.

It parodies SRE as a BOFH sysadmin, even though in general SRE are quite easygoing and helpful.

It helped poked fun at a number of overly stuffy processes and also helped push people to make hosting modest datasets (like this 5TB one) easier.

Wonnk13• 5 years ago

One of the few things I miss about my time there...

Never did get Java readability :(

slac• 5 years ago

I have the t-shirt!

jbverschoor• 5 years ago

omg so toxic

sbpayne• 5 years ago

I'm so glad I can see this again. I forgot how much I missed this.

jboggan• 5 years ago

Eagerly awaiting the "GoogFellas" video leak.

smartician2• 5 years ago

It's not much different today. Nowadays you'll also need privacy review, accessibility review, security review, and diversity & inclusion review.

ntaylor• 5 years ago

diversity & inclusion review

cynicalkane• 5 years ago

Assuming this is sarcasm, you realize Google has a massive userbase all over the globe from all walks of life, right? Does it make business sense to accidentally exclude certain people? Or ethical sense?

throw10920• 5 years ago

diversity & inclusion review

Is this tongue-in-cheek, or are you serious? Poe's law and all that.

kevingadd3• 5 years ago

If you're publishing a dataset in the terabytes it does actually make sense to at least do a pass over it and make sure the data you're using isn't skewed in any undesirable way that would cause problems down the road. For example, if you're releasing 5tb of face photos for training facial recognition nets, it would certainly be a problem if all the faces are white women or asian men - the result would probably be over-fit and not perform as well for people in other categories. It would be correct to call that a diversity/inclusion issue.

Privacy and accessibility reviews serve similar purposes there, you're reducing risk by checking for these various problems and ideally they also spot ways to improve the quality of your outcomes.

smartician2• 5 years ago

Partly tongue-in-cheek. These review processes exist, but whether they're required or not depends on the product area and type of project.

rodgerd• 5 years ago

Perhaps Google don't want to be in the news for identifying dark-skinned people as monkeys again?

dekhn8• 5 years ago

having launched some product at Google in my day, I know quite well how to skate through that process (although D&I was not part of it when I filled out my forms). Sadly for my friends in privacy and security, it's not hard for product teams to exploit Google's propensity to launch and override privacy and security concerns.

ikiris7• 5 years ago

Is the super secret process to just have a vp invested in the launch?

danans• 5 years ago

That's only for public launches (and I'd add QA review to the list), and I'd argue that each of them are critical.

For serving analytics data internally, you only need privacy review, for obvious reasons.

Security and auditing is built into the tools used for querying and serving any such data internally.

dang2• 5 years ago

We detached this subthread from https://news.ycombinator.com/item?id=29086292.

cromwellian9• 5 years ago

As a Googler, it's often easier for me to setup a GCP consumer account, AWS, or Heroku account to demo something, compared to using anything internal. I remember the most annoying situation was like 10 years ago when me and other engineers ported Quake 2 to run in Chrome, we were in a time crunch to demo it multiplayer, and I ended up setting up an AWS account to serve it. But then I left it running and forgot and ended up getting a few hundred dollars billed to me because the Quake2 server was chewing CPU.

bamboozled7• 5 years ago

I could imagine you're violating some pretty strict policies doing this?

You're taking proprietary code and running it on a competitors platform?

I like to think I'm pretty open minded about stuff like this, and I've actually done something similar, but I'd be surprised if you didn't get your ass handed to you for that type of thing?

mabbo• 5 years ago

Why? Do you think there's a risk of Amazon stealing code from a customer?

No matter what code they took, the cost would never be worth it for them.

BoorishBears• 5 years ago

This is what separates companies that get things done from companies that pay a lot of people to hopefully maybe get things done.

A server for a multiplayer Quake port...

Who is Google paying to hand their ass to them over that?

Who has both the authority to hand their ass to them, and a lack of discretion to not let it end at "well in general we don't do that, but I see why you did it and there's little to no risk"

At some companies yes, someone is paid to go "I caught someone putting our proprietary code up on a competitors platform!!!!" and no one will actually think critically about what exactly was proprietary, so someone putting up a quake demo might as well have put up the Coke secret formula

And now OP who actually generated some value at little to no risk gets their ass handed to them and someone who simply lacked the skills to realize the risk profile gets a notch on their "I add value and earn my paycheck" badge.

potatoman22• 5 years ago

Can't let the Quake 2 source code escape Google

cromwellian9• 5 years ago

No, no proprietary code was used, the port was done from the Open source Java clone Jake2: https://en.wikipedia.org/wiki/Jake2

We ported it by using Google Web Toolkit Java->JS compiler, and replaced OpenGL with WebGL, and all of the other bits with Web APIs (websocket, pointer-events, fullscreen-api, filesystem api, etc)

The assets (proprietary artwork, levels, etc) were not hosted on AWS, it simply downloaded the EXE file from ID servers and extracted it in the browser.

cromwellian9• 5 years ago

For those interested: The original project https://code.google.com/archive/p/quake2-gwt-port/

GitHub (Stefan Haustein is my genius teammate who did all of the heavy lifting on the OpenGL -> WebGL piece) https://github.com/stefanhaustein/quake2-playn-port

You can still play it here, on AppEngine http://quake2playn.appspot.com/

mgfist2• 5 years ago

You're taking proprietary code and running it on a competitors platform?

Do you not realize how dumb this sounds? Companies run critical code/infrastructure on AWS to the tune of many billions of dollars a year.

didip• 5 years ago

What is "Borgmon readability" and why was it important. I think that's one of the punch line of the video.

yeputons• 5 years ago

If you change source files in language X, someone proficient in language X (aka "has X readability") should approve that it corresponds to Google Code Style in language X.

You start without it and may obtain once you've written a bunch of code in language X.

I'm not sure if there is really a Borgmon readability. But if there is, it seems like Borgmon configuration files are both common (so that there is a readability requirement) and uncommon (so that there are very few people with readability).

GeneralMayhem2• 5 years ago

There isn't Borgmon readability anymore because of this video (the video is about 10 years old).

Borgmon is a monitoring system for Borg (https://sre.google/sre-book/practical-alerting/#the-rise-of-...), and it has its own language for configuring monitoring rules. The Borgmon language is infamously obtuse, and the idea of "clean" Borgmon code has basically always been a punchline. To be fair, the problem domain that it's in (declarative computation and reduction over high-dimensional datasets) is tricky to make clear, and if you use it exactly as intended, it can do some really cool stuff. But the upshot is that it has a high learning curve, a high propensity to degenerate into wallpaper, and a very small number of people who are sufficiently familiar with it to be readability reviewers.

It also doesn't help that "just copy-paste someone else's" actually does cover 90% of use cases, which both reinforced the idea that it was a chore and actually made it harder to gain readability, since being granted "readability" status requires writing a meaningful amount of non-trivial, non-copy-paste code.

silentsea90• 5 years ago

Actively working towards being a Xoogler so I don't have to live in this dystopia.

sunyc• 5 years ago

I actually have Borgmon readability! Peer bonus pls.

ineedasername4• 5 years ago

I think I'm missing something. Does readability have a special meaning in this context or is it really as basic as it sounds, and therefore all the more ridiculous for being such a bottleneck?

goleary• 5 years ago

At Google an employee must have "readability" in a language in order to sign of on a code reviews in that language.

You must pass a test in order to get "readability".

nojvek• 5 years ago

The borgman readability approvers makes me chuckle.

At Stripe, there were language approvers. Only those blessed could approve PRs. Even XML had a set of approvers. I had fun time getting hold of an XML approver.

kgin• 5 years ago

Only if you think your users are scum. Do you think your users are scum? Do you? Why do you hate your users?

opinion-is-bad• 5 years ago

The multiple repetitions of “This is Google” hit home for me. I never worked as a software engineer so much of the rest is out of scope to my experience, but the constant idolization of Google, and by proxy each other for working at such a place, eventually changed from feeling coy to cultish.

metanonsense• 5 years ago

Move fast and break things! And while you are at it, please, don't break anything.

romwell• 5 years ago

Ah, as a xoogler, I can finally enjoy this video without tears :)

Lammy2• 5 years ago

Personally I'm still waiting for a copy of "Pusher, I have a hotfix."

rachelbythebay3• 5 years ago

Maaaaaaybe ID 10150136890464739. Maybe.

Lammy2• 5 years ago

Didn’t work logged-out, but I’ll try again if I ever reactivate my account. Thanks!

CephalopodMD• 5 years ago

Broccoli man is incredible. I hope someone leaks the rest...

b20000• 5 years ago

meanwhile google docs is still a slow piece of shit.

zzt123• 5 years ago

Was this video ever used by Google for anything outside of internal before now? I swear I’ve seen this before (it’s still hilarious!) but I never worked at Google.

codewiz• 5 years ago

10 years ago, on my first week at Google, my manager suggested I watch this video to get a fairly accurate idea of what the team does :-)

ChuckMcM• 5 years ago

Wow, way to trigger my Google PTSD. :-)

I think the only thing I would have added would be using some service that recently was changed so that someone could get promoted but does the same thing the old one did but with different bugs.

Reschivon• 5 years ago

It was a site where you could type in a script and it would do text-to-speech and actually animated some goofy characters to lip-sync the words for you

Does anyone still have the code for generating "Fox girl and Broccoli man" videos? I'm thinking of starting a small revival of this meme.

adminscoffee• 5 years ago

it's funny how you have to have a dictionary in order to understand how to navigate google's infrastructure

Google's infamous internal 2010 “I just want to serve 5TB” video now public

Related Stories