US Senate’s email system melts down in face of security test and reply-all chaos

https://www.politico.com/minutes/congress/09-8-2023/senate-reply-all-mess/

131 81 3 years ago

SoftTalker• 3 years ago

I've seen this happen a few times, usually someone accidentally spams their entire address book, or even better a bunch of listserv addresses. Then you get the "reply all" messages to "take me off this list" or "unsubscribe" which just adds fuel to the fire.

JoeOfTexas• 3 years ago

This happened today at work. Like just ignore the email...

jjoonathan• 3 years ago

But that's no fun! ( •̀ᴗ•́ )و

dessimus• 3 years ago

Thank god for Transport Rules to just silently drop emails with matching subject for a few days.

Sylamore• 3 years ago

One of the privileges of previously being an Exchange admin was being able to send the "Do not reply all" message and then setting up a rule on the server to kill all further messages.

But it's also why I had a script to lock down any distribution list with more than 100 members to authorized senders only.

CatWChainsaw• 3 years ago

The last storm I was in had people replying all to tell people to not reply all, followed by people replying all to shame the people who replied telling people not to reply all.

Followed by others replying all asking how to mute the conversation, and others replying all with instructions. Multiple times. Which prompted more people to reply all commenting on the increasing recursion of the absurdity of the whole thing.

So you'd get fed up and mute the chain and think you'd think you'd killed the process, but any time someone changed the email subject line and replied all again it came back like a bad penny. Which caused more people to angrily reply all with vulgarities, which caused supervisors to reply all that their behavior was being noted and disciplinary meetings would be had.

And then one day, someone replied-all to one of the child email chains with the entire script of The Bee Movie. Which launched more of all of the above. And did the same thing to every other child email chain, sporadically, for a few months. I think it took over half a year for the last reply-all to quiet down.

I think at least one of those chains lives on for posterity in one of my folders, and now you tempt me...

mensetmanusman• 3 years ago

MS did it first: https://techcommunity.microsoft.com/t5/exchange-team-blog/me...

8f2ab37a-ed6c• 3 years ago

Amusing, almost 30 years later and this same problem still exists: https://www.linkedin.com/pulse/surviving-bedlam3-email-storm...

nvy3• 3 years ago

Reply-All cascades are classic entertainment in government work.

When I was in the Air Force some poor Warrant Officer started a Reply-All clusterfuck when he replied to a base-wide email about issued Android phones.

After about 30 Reply-Alls some very senior officer replied (to all) something to the effect of "Nobody else Reply, or your ass is grass" to which some joker from Special Operations replied (all) "Copy, Sir".

Some of the most fun you can have in the public sector.

foota• 3 years ago

Is this a copypasta? If not, it should be.

nvy3• 3 years ago

It's not a copypasta; I was there for Reply All 2019.

MarkSweep• 3 years ago

Here is a story of a similar form:

https://devblogs.microsoft.com/oldnewthing/20111014-00/?p=93...

It’s a pretty predictable fact pattern.

chatmasta• 3 years ago

We had this in college where someone sent "M to mute" and then thousands of people started sending emails with "M" in the body. Nobody knows how many of them were being sarcastic, but it became a meme that we found ways to insert into all sorts of unwanted group communication channels.

rakoo• 3 years ago

When the friends groupchat becomes overly active about a topic only few are interested in (not including me), I joke and reply UNSUBSCRIBE. I think I'm the only one to understand, and definitely the only one to laugh

Jtsummers• 3 years ago

USAF doesn't have warrant officers (used to, but the last ones retired last century), but yeah I saw a couple of these in my time, too. Not a lot of fun.

nvy3• 3 years ago

Never said I was USAF.

tomjen3• 3 years ago

These days you can just mute the email in Outlook.

hughesjj• 3 years ago

At my Amazon peers...

"Wallet"

FBISurveillance• 3 years ago

Context:

At amazon in 2010, it started with a meeting invite that was accidentally sent to everyone. The meeting was for the Amazon Wallet team for whatever work they were doing at the time. A reply-all storm ensued, and despite several people asking to stop hitting the reply-all button, it continued for several days. The management had to chime in and sometimes threatened people with punishment if they continued participating to the reply-all storm.

If you ignored the conversation you may not have seen it, but most reply-all storm ended up with someone saying something the line of "we don't care about your wallet".

Anyway, asking people to stop hitting the reply-all button is far from being the latest reply-all on these kind of things...

jrockway2• 3 years ago

These are just as fun at large corporations. A bunch of people you've never met, all very mad at each other. (I always join in on these. I simply cannot resist. One time, someone emailed me privately to say that I should know better. Oh, I do. I just want to watch the email server burn.)

tyingq• 3 years ago

I feel like the sender is more at fault in these cases. BCC exists for a reason.

ryathal• 3 years ago

I really don't know why it's so hard to get basic competency in public sector to use bcc or restrict send access to large groups. That said, I enjoyed the drama when an email devolved into requests to stop replying all and requests to be taken off the list.

araes• 3 years ago

If you would like more info on that topic... Was on HN back in early summer. https://news.ycombinator.com/item?id=36620608

"A recent international research study allows us to quantify the difference between the broad population and the tech elite. The data was collected from 2011–2015 in 33 countries and was published in 2016 by the OECD (the Organisation for Economic Co-operation and Development, a club of industrialized countries).

In total, 215,942 people were tested, with at least 5,000 participants in most countries. The research aimed to test the skills of people aged 16–65."

Summary Page: https://www.oecd-ilibrary.org/education/skills-matter_978926...

Executive Summary: https://www.oecd-ilibrary.org/education/skills-matter/execut...

Key Adult Proficiency: https://www.oecd-ilibrary.org/education/skills-matter/adults...

Nielsen / Normal (focusing on computers): https://www.nngroup.com/articles/computer-skill-levels/

eclipxe• 3 years ago

For Amazonians - Wallet?

jrockway2• 3 years ago

Details on the Wallet thing: https://news.ycombinator.com/item?id=37336088

BLKNSLVR• 3 years ago

Back in the days before ransomware and RATs and APTs and ubiquitous cloud-hosted corporate email, when viruses were accidentally introduced rather than spear-phished, state/criminal-gang-sponsored existential threats to a company, I had the idea (without the ability to manifest it) of a virus that would infect an MS Exchange instance and set everyone's Out of Office on and all incoming email to be forwarded to every other mailbox in the company.

Not sure if it is or was feasible, but I still like the simplicity of an automation-stack doing the really hard work of exponential amplification.

ChrisArchitect• 3 years ago

Related recent thread with lots of reply-all anecdotes:

* @gmail.com (xkcd)

https://news.ycombinator.com/item?id=37333848

chevman• 3 years ago

My other favorite is when corporate comms or HR sends out some calendar invite to a massive number of people, does not use bcc, and then the server crashes when folks go in to the accepted invite on their calendar and you get thousands of calls to 'check availability' flood in.

Have seen that bring the Exchange Server to its knees a few times :)

MBCook• 3 years ago

Or the out of office auto-replies. Heh.

WarOnPrivacy• 3 years ago

In college, I came up with a one-liner to send the entire MAN to every email address. The server outage disrupted classes for a week.

I can't come up with anyone who wasn't >irritated with me.

wolverine8767• 3 years ago

Senate staff were instructed to respond to the email, which outlined a potential threat for the drill, with their status and location on Capitol Hill or off campus. But the thousands of simultaneous reply-alls that ensued first clogged up inboxes before staff were unable to get into their inboxes altogether.

That's why we test! Imagine if that happened during an actual crisis. Give them a break - that's the point of testing, to discover these glitches.

Note that another negative outcome is that high value targets (e.g., Schumer and McConnell) are emailing every Senate employee with their location, not to mention the email chain also provides aggregated information on where crowds of targets can be found.

Also, email ain't so secure. A sophisticated enemy - and I'm pretty sure the US government actually has some of those - would love that information. Heck, you can find locations just by locating mail traffic.

The Senate’s email system, run through Microsoft Outlook

It's 2023 and someone is still writing that?

MR4D• 3 years ago

Imagine if that happened during an actual crisis.

I’m ironically imagining our enemies are coming up with some heinous reply-all scheme that will bring us to our knees.

wolverine8767• 3 years ago

It's called "Social Media".

c0pium2• 3 years ago

I love how out-of-touch HN comments can be sometimes, it’s a breath of fresh air for me. Outlook the mail software (as opposed to the coat of paint applied to hotmail) is used by almost the entire Fortune 500, basically every government at every size, and a decent chunk of the Windows-using western world (which is most of it).

If you’re wondering why you have trouble selling to enterprise and government clients, this type of gap in perception is a big part of it.

xav0989• 3 years ago

I think it was more a comment on the fact that Outlook is the mail client, not the MTA.

coryrc• 3 years ago

The point is mail is run through Microsoft Exchange, not the client software.

wolverine8767• 3 years ago

Thanks for explaining!

poorlyknit• 3 years ago

Gee, I wonder why F500s are getting ransomwared left and right :)

I get that giving your drones software that is widely used is the easiest path but ActiveDirectory, Outlook and Windows are basically three of the horsemen of insecure corporate networks. The fourth is probably adware disguised as virus scanners.

alephnerd• 3 years ago

If you’re wondering why you have trouble selling to enterprise and government clients, this type of gap in perception is a big part of it.

Most HN founders aren't on YC anymore aside from their 1 free Show HN post and the account you make for the YC app.

Speaking from experience, they have much healthier discussions on Bookface, Twitter, and in person.

You're basically stuck talking with salty burnouts on here (to paraphrase a YC friend of mine who has publicly hated on HN)

eviks• 3 years ago

Why would you give them a break if this should've been tested, found, and fixed, ages ago?

wolverine8767• 3 years ago

When did this project start?

Clubber2• 3 years ago

It's 2023 and someone is still writing that?

Journalism isn't the profession it once was (if it ever was) from major corporations. They're just writing information they are fed by a source and spruce it up a little for views, maybe. There's no investigation going on anymore, there's no source checking generally, at least not like there used to be. The current business model doesn't allow for that in large organizations.

Angostura3• 3 years ago

It's Politico, not an IT news source. I think someone on deadline potentially confusing Microsoft's client software and it's MTA is forgivable, particularly given the ever-shifting landscape of MS' branding in this area

Someone1234• 3 years ago

This problem is relatively easy to solve with a Global rule:

- TO: [All]

- Subject: ^(?i)(re|fw):\s

- Action: Drop.

Heck, you could go one step further and block all TO: [ALL] entirely since they should BCC.

tgv• 3 years ago

The latter is best. I think another sensible action is to remove “all” from the reply. If you really want to send to all, you have to enter it yourself.

dataflow• 3 years ago

If you go this route (which isn't great), you probably want to include "Fwd" as well.

furyg3• 3 years ago

Way back in the day when I was doing a lot of Exchange consulting work, we would recommend removing permission to send to all except for one or two specific accounts. This was to avoid reply-all meltdowns, for security (viruses couldn't send to all), and also from an HR / internal communication perspective. We would recommend something similar for department-level groups, but often with a bit more freedom.

kylehotchkiss• 3 years ago

Great way to keep busy on anything but passing any sort of funding bill!

bagels• 3 years ago

Same thing happened at Microsoft, and Facebook. I'm sure it's happened anywhere with large email lists. Servers should probably just reject replies to lists large lists.

langsoul-com• 3 years ago

Surely there's a better way of getting everyone's location in an emergency. Although, long/lat aren't easy enough to say, type or find.

lapsis_beeftech• 3 years ago

Seems trivially easy to solve since all the users are already carrying tracking devices with GPS.

jjk1662• 3 years ago

Presumably they are more interested in information like which room someone is in than their position on Earth. I doubt that indoors the location accuracy of phone GPS would be accurate enough to be useful, particularly in buildings where many locations are designed to be secure both from physical attacks and espionage (probably a lot of thick metal plates in random walls). And even without interference issues, GPS elevation accuracy is pretty bad. Finally, you're only tracking the location of the person's phone, which in an emergency could easily get left behind somewhere.

None of this is to say an email chain is the optimal solution though.

Angostura3• 3 years ago

If there's a cheap, simple way to implement it - I'd quite like to know.

SturgeonsLaw• 3 years ago

"You guys all have phones right?"

The gov could write an app, installed on every senator's phone, that tracks their location as needed.

Perhaps the senators might be against such invasive surveillance? Then I'd hope their voting record on such bills matches their privacy concerns when they're the ones affected.

oersted• 3 years ago

I finally got around to watching The West Wing. There's a lighthearted small subplot of exactly this in S01E19.

By the way, it is so refreshing nowadays to watch a show about extremely competent imperfect people that legitimately strive to do as much good as they can, as they see it. Likewise with Sorkin's writing, while unrealistically witty and fast, it is a delight watching someone so obviously talented give their all to their work. Granted, as in other Sorkin writing, it does sound a bit like all the characters are just versions of him with different voices. But it seems to me like an intentional stylistic choice, a somewhat heightened reality.

22289d• 3 years ago

The movie Malice is one of my all-time favorites. 1993, average reviews. Total random movie. Recently I paid attention to the credits: written by Aaron Sorkin. It all makes sense now. It's been ignored for whatever reason but it's amazing.

developer93• 3 years ago

I call this competence porn - a brief escape to a world where leaders are both competent and benign.

saghm• 3 years ago

Of all things happening in a West Wing episode (although to the White House, not the Senate) to still be topical 25 years after airing, this would not have been high on my list. I'm not sure to consider myself overly optimistic or overly naive...

cvccvroomvroom• 3 years ago

Terminate everyone who pressed reply-to-all as they can't be trusted to follow simple directions and they shouldn't have a security clearance either.

mattbrewsbytes• 3 years ago

And here I thought the oxy-moronic reply-all pavlovian response had been trained out of humans. I've been fortunate enough to work on teams where I haven't witnessed this in at least a decade. Of course its in our technically inept government where this happens.

toyg• 3 years ago

It's 2023, we have systems processing trillions of transactions per minute, and still a few thousand emails can "melt" an email server? That's testament to how little consideration the mail stack gets.

db48x2• 3 years ago

Well, kinda. Due to features like BCC headers, each recipient is supposed to get their own unique copy of every email, so email servers can end up spending a lot of time writing tens of thousands of copies of every mail sent to the list to disk. And then of course they have to sync that to all of the redundant servers, to support HA.

breakwaterlabs• 3 years ago

Symbolic links and deltas are a thing. None of this is hard from a technology standpoint.

Exchange just isn't the best designed or most modern software.

TurkishPoptart• 3 years ago

https://en.wikipedia.org/wiki/Email_storm

treesknees• 3 years ago

This happened to us several years ago, a basic reply-all storm across probably 30k employees. The icing on the cake though - when the storm finally calmed down a few hours later, the original sender then tried to recall the message. The recall notification went out to everyone and sparked it back up again.

kthejoker22• 3 years ago

Got caught in 2 different bad reply all storms at Accenture a few years ago.

The temptation to respond and mock / yell at the people who keep replying is strong.

Feel like it should be a fireable offense at a place presumably hiring for critical thinking and situational awareness.

PeterStuer• 3 years ago

Remember the days before mailing list distributers filtered out of office replies? It was fairly common to open your mail in the morning and have to delete several hunderds of these auto replies and dozens of replies from people complaining about it adding to the bulk.

US Senate’s email system melts down in face of security test and reply-all chaos

Related Stories