Time for a code-yellow?: A blunt instrument that works

https://nilam.ca/2024/11/02/time-for-a-code-yellow-a-blunt-instrument-that-works/

by BerislavLopac • 2 years ago

64 112 2 years ago

janice1999• 2 years ago

Coming from an old school engineering company, this article frankly reads as a little insane to me. I assumed "Code Yellow" was a catastrophe on par with a tornado hitting a factory or a massive security breach. Instead the examples are not hitting artificial growth metrics and needing to launch advertising. It's bad enough to blow up teams planned work but this is what you demand employees (not founders mind you) sacrifice their personal life for (i.e. get paid less per hour and not see their families)? The author not only acknowledges the time he is stealing from employees but that stressing them out is the point. If you're 8 years into a company and lurching from (artificial) crisis to crisis to "sweat the teams" something is seriously wrong.

monkeydreams2• 2 years ago

I work in public health so our outages are critical to safe treatment of our patients.

We have the concept of a Major Incident/Major Event (because the term Code Yellow is already co-opted to mean any administrative fault across a hospital which might impact the flow of patients).

These are all-hands-on-deck moments. They may be called by any member of staff who discovers an event which will impact our service, though it will be ratified by a Major Incident Manager (MIM). While the event is underway the MIM is God; except when it affects staff welfare. If a member of staff says that they cannot attend a war room for whatever reason, then the MIM will move onto the next person up the chain, even calling in directors if they feel it is required.

What is being described above is tech-bro shittery. Calling a major event because you haven't hit a sales target should keep the C-Suite up at night, sure, but calling in techs and devs and 'sacrificing the L and the B in Work Life Balance'? The C-Suite should be making the strategic decisions to reverse the decline, not suddenly drag everyone into a meeting to fix their lack of foresight and working towards an end that the average tech/dev cannot influence.

ElevenLathe• 2 years ago

I've been a major incident manager. It works great to have a hardass like me giving orders on a conference bridge: stuff gets fixed really fast! People pull out laptops at little league games and get it done. The outage is solved, and business goes on. I look like a genius.

The problem is that, if a major incident is the only process in your organization that results in things getting done in a reasonable amount of time, there is a large temptation to declare them for everything. Somebody wasted 6 months failing to renew our contract with the payment processing vendor because other stuff was being prioritized above it? Open a major incident and we'll get them to call the vendor CEO at home on Sunday. We're a big account, so they'll answer. It's the end of the fiscal year and if we don't decomm this expensive system like was promised, a VP somewhere will lose their bonus? Declare a major incident and ElevenLathe will get it done in a few hours, even if it is Thanksgiving.

I would page on-call people at any hour of the day for real emergencies without hesitation, but always asked for orders in writing to page people (not executives, but real people) who weren't on call, or for nonsense like this. This got annoying so they started assigning other incident managers to the nonsense.

cbsmith3• 2 years ago

It's not called a Code Red.

Not all catastrophes are immediate in nature. Most organizations have incident response protocol which works well enough for an immediate catastrophe. Non-immediate, gradual threats can represent a much greater risk, because there's no inflection point (beyond "way too late") where the threat becomes so imminent as to trigger that response. Code Yellows are a mechanism to artificially force that inflection point before it is too late.

Lammy3• 2 years ago

It's not called a Code Red.

The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow. See Stephen Levy's In The Plex (2011) pg186:

“A Code Yellow is named after a tank top of that color owned by engineering director Wayne Rosing. During Code Yellow a leader is given the shirt and can tap anyone at Google and force him or her to drop a current project to help out. Often, the Code Yellow leader escalates the emergency into a war room situation and pulls people out of their offices and into a conference room for a more extended struggle.”

crote• 2 years ago

Not all catastrophes are immediate in nature

Of course, but if it's not immediate, why does it warrant forcing people into a Zoom meeting on a Saturday while you're at your kid's Tae Kwon Do competition? If it's not immediate, why do you have to "encourage the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance"?

Either it's an immediate catastrophe which requires immediate action, or it can be handled by giving it absolute priority during regular business hours.

AlexandrB3• 2 years ago

There's definitely a "delusions of grandeur" thing that happens with some startups. "Come join our incredible journey to change the world by micro-optimizing ad placement on image boards."

nottorp• 2 years ago

Is this an article about the virtues of mandatory overtime?

DrillShopper4• 2 years ago

It's from a manager / founder so yes, implicitly it is

bawolff• 2 years ago

This feels like a blunt instrument to solve management failures.

Waterluvian• 2 years ago

Any time military language gets used like seen in this blog (eg. “War room”), it’s usually alongside management failure to be addressed by crunch of some sort.

arkh3• 2 years ago

Any time military language gets used

And some military terms are never used. Reserve: the idle resource you have that you can deploy to stop a breach or to profit from an opportunity.

Nope, we're using Human Resources like people used to use machines in the 80s: you have to get them working 100% of the time. And when there's an emergency? Code Yellow. When there's an unforeseen opportunity opening up? Well, too bad so sad your competitor got it.

Lammy3• 2 years ago

2019: Building our fourth senior executive team in six years to lead us to the next plateau of scale.

Sounds like nobody was sticking around long enough to even find out.

cbsmith3• 2 years ago

That seems about right. That's actually helpful. When you have a management failure, trying to solving it by "getting better at management" isn't really a solution.

bayindirh8• 2 years ago

It is alluring because it allows for existing plans to be deprioritized, removes any/all ambiguity around what is most important at the moment, and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.

When I read this, I understand that when management fails, and wants something to be get done, because reasons, they can just pull all stops and declare temporary slavery until the problem is solved.

This should be normally a "once in a decade" event. Not "once in a year" instrument. Being proud of removing "life" and "balance" from employees' reality reads like the worst power trip ever.

I worked in similar environments. Never again. We have our "code yellow"s in my current job, but we know when it's going to come, and prepare ourselves and our lives. Go through it, pat ourselves on the back for the good job, learn our lessons for the things we fail along the way, and continue our journey.

andrewjf• 2 years ago

Also the line about sending this email while on the weekend to his staff at his son's tae kwon do is particularly absurd.

bayindirh8• 2 years ago

The guy comes across as someone who thinks that non-work activities are just inconveniences for his and the company's well-being.

Work is there to support other areas of life. Personal life is not something that we "do" in the time which is left from work. It's not the second class citizen when compared to work.

Did he failed at this notion so badly, so he turned to this workaholic, everything else is secondary lifestyle, I don't know.

roenxi• 2 years ago

It is hard to pick the most absurd line. I quite enjoyed "2018: Scaling our infrastructure to keep the site up on Sunday’s (we were having constant outages every week)". Take a moment to reflect that implies achieving one 9 of reliability required an emergency all-hands response that bypassed their normal processes. I may not be an expert in high-reliability systems but that isn't how I expect the problem to be tackled.

My read is the management team are not very capable and that this gentleman is not a natural leader. Doesn't sound like he knows how to build systems in a planned and thoughtful manner and has embraced a classic lurching-into-crisis strategy for want of understanding any alternatives.

EDIT Also juxtaposes beautifully with the "‘keeping the lights on’ fallacy" he mentions. Those crazy engineers with their belief that they need to try and keep the lights on. Also, when I forcefully pull them away from that, why do the lights go out on Sunday???

mtrovo• 2 years ago

I’ve experienced a Code Yellow myself and can definitely see its value. In large companies, you sometimes need that top-to-bottom rallying cry to overcome the bystander effect when big problems emerge.

After ChatGPT’s initial release, Google famously sounded the alarm and brought Larry Page back into the mix, showing how this kind of all-hands effort can quickly organize everyone around a single goal.

AlexandrB3• 2 years ago

Brought Larry Page back to do what? Google is worse than ever, Gemini is just one more piece of clutter ruining what used to be a useful tool.

ponow2• 2 years ago

So your angle is just to let ChatGPT fill that space?

Blackthorn• 2 years ago

It is alluring because it allows for existing plans to be deprioritized, removes any/all ambiguity around what is most important at the moment, and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.

Why would anyone go along with this unless it came with a FAT bonus?

01HNNWZ0MV43FF• 2 years ago

I also accept PTO credits as payment for voluntary overtime

fn-mote• 2 years ago

Except not as a 1:1 trade!

gregors2• 2 years ago

The idea is that you put in the extra work and he keeps the extra profit. Sounds like a sweet deal.

HL33tibCe7• 2 years ago

When you’re on your deathbed, I’m sure you will be glad that you spent your kid’s taekwondo tournament writing an email so that an e-commerce company could make line go up.

bayindirh8• 2 years ago

Some of these people don't understand that they won't see that tournament again, and they won't be able to make their relationship "line go up" if they don't build the foundations when their children are still children.

DrillShopper4• 2 years ago

Some of these people don't understand that they won't see that tournament again

Just as many don't care or see it as an inconvenience to be there. They might do less damage to their families by just being honest and skipping.

Terr_• 2 years ago

"But you were always a good man of business, Jacob," faltered Scrooge, who now began to apply this to himself.
"Business!" cried the Ghost, wringing its hands again. "Mankind was my business; charity, mercy, forbearance, and benevolence, were, all, my business. The deals of my trade were but a drop of water in the comprehensive ocean of my business!"

-- A Christmas Carol, by Charles Dickens

rdtsc2• 2 years ago

At Beacon, we are not going to wait for the next crisis to sweat our biggest challenges; we will build a culture that makes Sweating The Problem our default.

Brilliant! I love it. It's Code Yellow all the time. If it worked once a year at Instacart, and Google made so much progress with it, we'll just turn "the Yellow" on all day, every day.

Then maybe they can have a Code Red for when it's super-duper important to "sweat" the problem.

you walk away with the confidence that you can handle whatever comes next.

You don't even need a bonus, just enjoy your new found sense of accomplishment. And maybe a subscription to the jelly-of-the-month club [1]

[1] https://kitchychristmas.com/jelly-of-the-month/

DrillShopper4• 2 years ago

Person who closes the most tickets gets a car

Second place gets a set of steak knives

Third place is you're fired

Always Be (C)losing

tanseydavid• 2 years ago

Coffee is for closers.

marcus_holmes• 2 years ago

Yeah I thought the same. I would not want to work for this guy or any organisation that thinks like him.

It's the manager's job to prioritise. There is always more work than can be done, and always more problems than can be solved with the available resource. Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.

To the author: please learn how to actually manage. Honestly, you'll do much better. I look forward to reading your article about how you learned to manage problems instead of sweating your team into the ground.

rdtsc2• 2 years ago

Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.

Indeed. Even the words "sweating the problem" just sounds callous and inconsiderate.

It can even work for Google, where can pull in different team members at different times. But in a smaller company that means everyone's work-life balance is permanently stressed.

dghlsakjg• 2 years ago

... and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.

This manager doesn't get it.

x0x0• 2 years ago

I'm glad I'm not the only one who read:

* I promised myself never again. Never again would I call a code-yellow. Code-yellows suck, drain team morale, and they leave a lingering distaste amongst all those involved. Yet, during my 8-years at Instacart, they were our most effective and consistent weapon in ensuring we made meaningful progress on our hairiest problems.*

to be a really long-winded way of saying, "I'm utter shit at management, refuse to prioritize until it's a company-threatening crisis, and I'm happy to make my team suffer for my incompetence."

I'm sure the employee churn at Instacart was in no way related to this fact.

And now he's operating a PE/rollup firm.

munchbunny• 2 years ago

If thing A is important enough to declare a "code yellow" in order to ignore things B, C, and D to focus on A, then were B, C, and D really that important? Could you have focused your team on A from the start, making a "code yellow" unnecessary? (Hint: yes, you could have, so the question should be why you didn't, and whether you could have seen it coming.)

I've seen this happen a lot with mediocre leaders. "Code Yellow" equivalents happen because they weren't able to understand that A was really the most important thing, typically because B, C, and D were important for optics or politics, but not genuinely important to the customer or the problem at hand.

A "Code Yellow" is a useful political tool to move an organization to focus a bit more on problem solving by saying "I don't care about your politics, your org charts, whatever, just solve the damn problem." In that sense, it really does work.

charles_f2• 2 years ago

The worst in the story is that thing B, C and D were probably decided, and commendeered by the same exec who, 3 months later will decide that thing A doesn't move enough and "they need to act on that". It's a practical tool to correct their indecisiveness and errors, and who's paying the price for it...

SpicyLemonZest• 2 years ago

Political constraints are real constraints! It's very possible to end up in a state where declaring an emergency is genuinely better than any of the options to avoid it. Consider:

* A is clearly important and will have to be "code yellowed" if it's not addressed in the next few months. But you've got to drop one of B, C, or D in order to avoid the code yellow.

* B is one of your best engineers' passion project. She's not a big fan of C or D, and she might run away to Google if you tell her they're more important.

* C is a promise you made to a customer to unblock some big deal. They might not sign if you tell them you changed your mind and would prefer to work on some projects B and D they don't care at all about.

* D is a demand from your VP to address some company priority, and he doesn't think the effort-reward ratio from B or C are very high. If you try to drop D he will probably get mad and tell you that you're not allowed to.

aftbit• 2 years ago

Wow what a toxic attitude. The worst part is acting like this constant crunch time is a normal and reasonable expectation. I'll also call out the derision towards "keeping the lights on" because, of course, keeping the damn lights on is a prerequisite to any kind of growth. Code Yellow doesn't tell people "hey it's not important that the site stay up, just focus on growth instead". It tells people "what you are doing 9-5 to keep the site up is not enough, spend your 5-9 working on this other initiative as well". Evil.

pavel_lishin3• 2 years ago

It doesn't feel like it means the same thing to OP as it does to the rest of us:

The biggest constraint I have seen teams imposing unconsciously is the ‘keeping the lights on’ fallacy. This manifests as the project team giving the project/goal 25-50% of their attention (although saying it is their main priority) because they feel that they need to continue ‘keeping the lights on’ with other work activities or projects.

It sounds like "keeping the lights on" means ... working on other things as well? It doesn't specify that it's to make sure those other things keep working. Maybe it's implied? I have no idea.

dagss• 2 years ago

I take keeping the lights on to mean, e.g., customer support and fixing bugs an incidents. Not necessarily uptime alone.

gregors2• 2 years ago

This guy is bad at his job.

pavel_lishin3• 2 years ago

Well, he may be bad at his job, but at least it also sounds like he's bad at being a father.

MathMonkeyMan• 2 years ago

he did show up

nasseri• 2 years ago

Is this satire?

BeefWellington2• 2 years ago

I've met people like this and they seem to always have a disaster on their hands.

Probably because their companies spend so much time deciding which task should be prioritized and having meetings about it that nobody can get actual work done.

turbojet13213• 2 years ago

If everything is an emergency, nothing is an emergency. I don't understand why so many managers fail to grasp this.

It's the same with prioritization. I've literally had conversations that go:

Manager: I need you to drop everything and do X right now, it's top priority

Me: Ok, well I'm currently doing Y, which was top priority this morning. Which is more important, X or Y?

Manager: Well, they're both equally important!

Me: OK, sure. I can't work on both, which one would you like me to do first?

Manager: [uncomfortable thinking noises]

Manager: Are you sure you can't do both at once?

Me: Yes

Manager: [Pause] Keep going with Y. I'll see if someone else can do X

sigh

nine_zeros3• 2 years ago

This dilbert-style management has become very normalized in tech - to the detriment of everyone.

It fundamentally comes from a place where there are too many layers of managers, none of who want to make crucial decisions that can bite them back.

Aka leadership only in org chart but not in setting direction, leading from the front, or accepting tradeoffs.

Ultimately it all boils down to org chart leadership. If managers could be voted out of the org chart by people below them in the chart, they'd have no choice but to show more spine and help the people below.

But this is not going to happen in corporations. So the best thing to do is quiet quitting. There is a reason this exists.

jcgrillo2• 2 years ago

IME this is much more prevalent in organizations that hire "nontechnical managers". I personally will never work in such companies again, the experience of having your boss have literally no clue what it is you do for a job is not one I'll sign up for again.

hibikir• 2 years ago

It gets worse: A manager in those kinds of organizations that is truly scared also doesn't want people under them to make decisions either, so eventually everyone learns inaction.

At sope point the organization comes up with a code freeze, and a manager is afraid of changes near the freeze. Them said freeze keeps expanding as you get down to ICs, and eventually nobody gets to release anything in December, even if something is wrong and is costing the company users and money.

It's bad when people choose to quiet quit on management, but it's scarier when they basically are told to do it, in not so many words.

trhway2• 2 years ago

I have a highest priority queue where X is going into, and where Y is already comfortably sitting together with the other highest priority items (of course i don't have other queues :). Everybody loves that their X is going to be a highest priority item for me. "a" not "the". Thanks, English.

magicalhippo• 2 years ago

My dad used to flippantly say he had three piles of papers on his desk: "urgent", "very urgent" and "no longer urgent".

As time passed the papers would get moved from one pile to the next.

turbojet13213• 2 years ago

I've found this is only a short-term solution. It's much more effective IME to force the priority call onto the person making the demand.

I've done this as a team lead/scrum master. You want to add something to the sprint? No probs - you just have to choose something the same size to take out. You want to take a person off the team? Sure, you just have to negotiate with our clients about which work gets taken out of the sprint.

It's amazing how many things are suddenly less urgent when someone is forced to make an actual trade-off rather than push the pressure down onto the delivery team.

charles_f2• 2 years ago

Making decisions is by definition cutting off stuff. It is hard, and requires courage, both of which no-one wants to expend. Because of course we want it all. Reducing WIP is probably the most efficient and least liked management tool.

Most teams I've been a part of have the syndrome you describe at every level of the hierarchy. I'm in constant battles to reduce the scope of what I have to build to avoid spending a year before it sees the light of day. I integrated a team a little while ago to help them do better. 7 people, 12 features in progress. I remember an organizational-level presentation we had was presenting the 15 priorities and accompanying 11 focus areas for the next 6 months. In the company wide evals, on top of our "self-defined" personal goals, we also have 3 imposed ones (d&i, security and learning) that were imposed by the corporation over the years as a reaction to something, and no-one will ever have the courage to remove.

This is almost a disease, and I've lost faith in that it will be cured some day

toss1• 2 years ago

Exactly and literally; from the roots of "de"="off" and "caedere"="to cut" [0]

If you can't cut something off, you cannot decide.

decision (n.)

mid-15c., decisioun, "act of deciding," from Old French décision (14c.), from Latin decisionem (nominative decisio) "a decision, settlement, agreement," noun of action from past-participle stem of decidere "to decide, determine," literally "to cut off," from de "off" (see de-) + caedere "to cut" (from PIE root *kae-id- "to strike").

[0] https://www.etymonline.com/word/decision

heresie-dabord• 2 years ago

From TFA, everything is an emergency:

we leaned on Code Yellow’s so aggressively that they became part of our standard operating cadence.

Code Employees-Prep-Their-Résumés

sameoldtune• 2 years ago

Managers are taught to live in a world of emergencies. I believe that someone who takes a measured and thoughtful approach to things is excluded from management out of principal.

tpoacher• 2 years ago

I call this "top of the notes" syndrome, from my years in medicine, where every now and then a consultant physician would print something out and say "this is important, it should be on top of the patient's notes" (i.e. clipped on the cover of the physical folder, as opposed to affixed in it, so as to be the "first" thing one sees on grabbing that folder).

Which sounds great, until you have two of those.

And in reality, quite often you'd see folders where the "top of the notes" stuff had more entries than the normal contents of the folder itself. So, ironically people would skip the top stuff and jump straight into the folder as normal.

nine_zeros3• 2 years ago

strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.

And what is the incentive for the rank and file to do this? Execs get millions of dollars worth of stock every 3 months. The rank and file isn't getting this. Why would the sacrifice their L and B?

To keep their jobs? Naw. For this, they will cut corners and lie.

Management practices have become so terrible in tech. It is no wonder that the best people keep switching jobs every few years. There is zero incentive to work under such poor management practices.

readthenotes1• 2 years ago

"During a code-yellow, a leader can escalate a project/situation to a war room situation, pulling people out of their day-to-day work to focus entirely on the problem at hand. "

In one of those books management gurus right at the end of their workspan, one of them wrote that he did not believe a project was worth doing if it was not worth having a war room in which to focus people on the problem at hand.

Lowering Work In Progress is one of the easy ways to gain success...

jacknews• 2 years ago

"we will build a culture that makes Sweating The Problem our default."

What a douche. So it's always code-yellow, and now they'll need to invent a code-red.

avidiax• 2 years ago

If you are really sweating the problem, you are sweating only one problem.

But an organization usually has more than one problem that needs to be addressed concurrently, unless it wants to constantly be "fire fighting" problems that could have received partial attention earlier but instead have become emergencies.

So this guy is going to want to sweat more than one problem, which really just means doing a normal job of prioritizing things, but with more sweat. Just remove the L & B from WLB, permanently. Sort of like this boss imagines he does by writing emails at his family's events, but without the extreme upside potential afforded to executives.

__turbobrew__• 2 years ago

You exit from the president of Instacart and then spend Saturdays at your son’s tae kwon do practice coercing subordinates to give up their life?

lizzas2• 2 years ago

24/7 oncall to ... be yanked onto something the boss fancies. No thanks. What about... plannning?

habosa• 2 years ago

The apostrophe placement was almost as bad as the content. No thanks.

CalRobert• 2 years ago

“ found myself sitting at my kids Tae Kwon Do competition (which was running 3.5 hours late!) on a Saturday writing an email to the team ”

Maybe if you work hard enough you won’t need to stay at such a bad job

steinuil• 2 years ago

I don't understand which part of this requires the team to "sacrifice the ‘L’ and ‘B’ from Work-Life-Balance". If you want to encourage your team to re-evaluate their decisions and tackle big problems maybe you should build a culture around that rather than forcing everyone to be in emergency mode?

I think that's what this post is saying, but all the posturing about forcing people to working harder to build "grit" is making it hard to read it as genuine. Clearly this is not a management issue, it's the team's fault for not having enough grit to work hard all the time!

singleshot_2• 2 years ago

"2019: Building our fourth senior executive team in six years to lead us to the next plateau of scale"

Valuable lessons from a competent person. Great article!

snapcaster• 2 years ago

Yeah this seems insane to me, I don't understand how 4 executive teams in 6 years is a sign of anything but chaos and dysfunction

AIorNot• 2 years ago

just like layoffs, this just part of the tech culture, as long as we reward managers with promotions for riding their teams (and no doubt I'm sure those middle managers are working their ass of too) this culture doesn't change.

in many cases deep financial bets have already been made before the code is even written or the idea properly sounded out. culture is culture..

human desire to appease the social order is stronger than rational decision making

closeparen• 2 years ago

Under the surface constraints, the ones which are imposed almost unconsciously, are the most pernicious.

This is true and important. For example, no matter how critical a project may be, it is understood that the first duty of any white collar worker is to show up to every meeting on their Google Calendar. When we are are allocating, prioritizing, and estimating resources for projects, what we are really talking about is the time left over after discussing ~anything that ~anyone may want to pull them into. Only an executive whose calendar is kept by an EA subjects meetings to the kind of cost-benefit tests and prioritization that we use for IC project bandwidth.

kittikitti• 2 years ago

Corporate mind games exposed!

Time for a code-yellow?: A blunt instrument that works

Related Stories