Twitter Is DDOSing Itself

https://sfba.social/@sysop408/110639435788921057

by ZacnyLos • 3 years ago

1396 1021 3 years ago

sfba.social

ZacnyLos2• 3 years ago

I find it hilarious Mastodon is more stable than one-instanced Twitter run by some madness tech-maniac.

dredmorbius2• 3 years ago

Mastodon has had its issues, both globally and for specific instances.

Sidekiq falling over is a big one. See: <https://docs.joinmastodon.org/admin/scaling/> and <https://nora.codes/post/scaling-mastodon-in-the-face-of-an-e...>

(I have to email my own admin every few months to ckeck if things are OK.)

And during the October Revolution as hoardes arrived from birdland, things got ssssslllloooowwww globally.

Worked out eventually, but it took a bit.

Individual instances also tend to run into scaling issues, with Jerry Bell's Infosec.Exchange coming to mind. (Mostly because Jerry's discussed this a bit.) And of course individual instances can be shut down or fail in various ways. I've migrated several times myself.

I will say that most of the time things seem fine, and it's exceptionally rare for there to be truly Fediverse-wide issues.

(I've been on Mastodon / the Fediverse since 2017, for the most part quite actively.)

ZacnyLos2• 3 years ago

Archived: https://archive.ph/u5kNK

kvetching• 3 years ago

Could this be the reason for the rate limiting? Elon thinks the server usage is scrapers when in reality it's just bad code causing a self DDOS?

bradknowles• 3 years ago

Ask him whether he wrote the code himself that is DDOS-ing Twitter. I wouldn't be surprised if he had that much hubris.

anigbrowl• 3 years ago

This could explain why my computer fan goes into overdrive when trying to read Twitter, has been going on for a few months but I don't care enough to figure it what's going on in the bg

ttctciyf• 3 years ago

Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in.

It seems an outlandish claim, but then again Muskified Twitter has previous form for this kind of thing with that time when they self-derailed by locking themselves out of their own api, right?[1]

1: https://opuszine.us/posts/when-twitter-fails-itself

arter4• 3 years ago

This is interesting.

Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429.

If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.

Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry.

I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.

bheadmaster• 3 years ago

If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.

The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.

Quarrelsome• 3 years ago

I would guess the front end was written under the assumption that the back end would still work without auth. Perhaps the backend changes (mandatory auth + rate limiting) were pushed without sufficient testing of the front + back?

cactusplant73743• 3 years ago

Did Elon pay the AWS bill? That seems like a likely culprit. Twitter instances are being forcibly shutdown.

amluto• 3 years ago

Twitter operates its own datacenters.

badwolf2• 3 years ago

Well, they haven't paid their GCP bill... https://theconversation.com/twitter-is-refusing-to-pay-googl...

colechristensen2• 3 years ago

This is not remotely a likely culprit.

romseb• 3 years ago

"curlprit" for too many GET's causing a 429 is just the perfect typo.

readyplayernull• 3 years ago

Could someone report the error at press@twitter.com and see what they think about it?

vuln• 3 years ago

Quite the knee slapper. Thanks. I hope you didn’t spend many brain cycles on it.

ineedasername11• 3 years ago

Whoever is in charge of that account went all Oregon Trail on things and caught dysentery

kccqzy• 3 years ago

While exponential backoff is theoretically optimal, I doubt it's actually used that often in practice. I've seen too many cases where someone decides serving user requests with low latency is so important that they'd rather have a constant randomized backoff than exponential backoff. I've been in many design meetings and seen enough documents where the decision not to use exponential backoff is explicitly made, understanding the tradeoff with overloading and system recovery.

klabb3• 3 years ago

I wouldn’t be surprised if this has no back off or limit at all. 10 RPS is fast enough that it may simply be sequential.

colechristensen2• 3 years ago

I’ve had to… uhh… eagerly advocate for exponential backoff for weeks of constant uptime issues before someone listened and actually implemented it and solved the problems.

Like several times in different roles.

People do it, exponential backoff is everywhere in your stack, but it doesn’t end up in your application layer until you have enough traffic that you actually have to manage throughout.

jitl2• 3 years ago

Bluesky feels like it’s buckling under the refugee crisis load

Pxtl• 3 years ago

Mastodon.social is chugging along just fine

jtrip• 3 years ago

Meaning.. people prefer bluesky to mastodon?

bengale• 3 years ago

Is it still invite only? I don’t think I know anyone that’s got on there yet.

noarchy• 3 years ago

I've been on their waiting list for what seems like months at this point. Huge missed opportunity for them, being unable to leverage Twitter's failings.

jitl2• 3 years ago

Based on app perf today, they’re probably metering signups for good reasons.

agnosticmantis• 3 years ago

Now I’m really tempted to create a Twitter account and start reading. Maybe that’s 3D chess by design by the CTO to get people to sign up. /s

rideontime2• 3 years ago

Keep in mind that Twitter engineers are under extreme pressure, knowing that their H1B visas are at stake, to implement Elon’s whims as quickly as possible on systems they’re unfamiliar with. I doubt many of us would perform better in their situation.

xyzzyz• 3 years ago

Do you know how many current Twitter engineers are on H1B? Is this based on some data, or just speculation?

morelisp10• 3 years ago

Around 300. https://www.vice.com/en/article/n7z5px/twitter-employees-on-...

cactusplant73743• 3 years ago

You can access H1B job titles and salaries. It's public.

dcunit3d12• 3 years ago

I tried to point this out below. It's not even a DDoS, it's fake news.

https://news.ycombinator.com/item?id=36561808

it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure. the HTTPS TLS terminates at a Google VM, which gets relayed depending on the VIP used to hit it, but the traffic never gets past that Google VM. This is literally /HOW/ companies deal with DDOS.

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"

Mizoguchi• 3 years ago

This kind of stuff is unavoidable with what's going on at Twitter. Infrastructure changes, platform changes and mass layoffs all at once. I'm actually impressed they haven't experienced more and longer outages.

darkwraithcov• 3 years ago

Its a testament to how well designed and implemented the code base is, if the wheels havent started falling off yet (shockingly). I know we’re all supposed to shit on Twitter, but they had world class engineers working there before the mass layoffs and brain drain.

Solvency• 3 years ago

I can easily visualize a wooden wagon rolling along a rocky slow descent forever and all of the reasons its wheels would eventually break loose.

In the case of a platform like Twitter, aside from random acts of god and weather, aside from money, what are the sort of stuff that slowly breaks its wheels off over time such that, without staff, it inevitably stops working completely?

Sparyjerry• 3 years ago

True, the rate of improvements has been insane. I use it pretty often and haven't even noticed a second of unavailability even today and right now, not to say I use it 24/7 though. It's pretty obvious some people have a political bend to their hatred of Twitter so any second something isn't perfect they jump on it. It's really a sad state of affairs really.

arghandugh• 3 years ago

This is absolutely preposterous. Any casual user of the service can rattle off a dozen failing elements going back for months.

danShumway2• 3 years ago

a second of unavailability

Being unable to look at anyone's tweets doesn't count as unavailability to you?

EdwardDiego4• 3 years ago

Political bent?

Can't I just think Elon Musk is a twat without it being somehow political?

djur2• 3 years ago

It's been unusable all day.

DonHopkins4• 3 years ago

As if your delusional denial of reality wasn't totally political.

quote• 3 years ago

One weekend Russia attacks Russia, next weekend Twitter attacks Twitter.

throwaway202351• 3 years ago

Don't forget when Reddit attacked Reddit two weeks ago.

papito2• 3 years ago

I don't have a problem with any of the three.

stubybubs• 3 years ago

The entire world now living the "hurt itself in its confusion" meme.

spaceman_2020• 3 years ago

Everyone else is going to do the same.

They're only holding out because they still believe the Fed will cut rates and they can borrow some more cheap money to keep the gravytrain going.

sva_• 3 years ago

It is kinda funny if you consider these companies might consider their user data to be useful, especially with recent advances in LLM models. I've been thinking if you just exclude Reddit posts from training youll probably achieve much lower bullshit scores, as that seems to be what most posts on there seem to represent. I think data curation (by sources) could achieve quite a bit.

rvba2• 3 years ago

Spez trying to kill reddit for short term gain is nothing new.

I really dont understand how the "investors" who will end up holding the bag after the IPO dont see that.

kzrdude• 3 years ago

unlike what others have said, Twitter was very useful during the saturday mutiny in Russia. I follow a lot of people who supplied updates and thoughts.

paganel• 3 years ago

We're also during the France attacks France weekend, and Twitter self-immolating certainly doesn't help with staying in touch with things down there.

Method-X• 3 years ago

France is attacking France.

julienr2• 3 years ago

Haha I was looking for this one!

bitwize• 3 years ago

It hurt itself in its confusion!

brucethemoose26• 3 years ago

Meanwhile, I can see this very Mastadon post with no login just fine.

And the interface is not trying to assault me. It loads quick.

...Seems like a better product than Twitter for a public feed.

bboygravity• 3 years ago

Most of the rest of the world doesn't agree.

teawrecks• 3 years ago

Also most of the rest of the world doesn't use Twitter (I'm seeing about 240M total users?). The ones that do use Twitter do so because of a significant amount of money going into marketing Twitter's platform.

Conversely, relatively nothing goes into pushing people to use mastodon. It can only take off if it really does prove, not just useful, but more useful than a centralized version that's got money behind it.

amelius• 3 years ago

I'm sure the loads are not comparable.

paradaux• 3 years ago

Equally, neither is the budget. Load isn't an excuse.

hinata08• 3 years ago

It withstands being one of the first link on HN, and it still loads comfortably.

For a personal website, that's a great performance.

brucethemoose26• 3 years ago

Doesn't really matter if it works?

And if some massive org needs bandwidth for posts, can't they host their own public instance? I'm sure many organizations would prefer that over being at Twitter's mercy.

candiddevmike2• 3 years ago

It's not crunching a bunch of analysis or ads in the background. I bet it scales wonderfully with maybe some kind of cache/CDN if necessary.

dijit• 3 years ago

Honestly.

what the fuck is this response?

I hear always that centralising everything is great because efficiencies of scale: but then we have something that works as good or better and the response is; “ah yeah, but the load is so high!”

Why do I care? I don't honestly give a shit about how much load you have, you could be factoring Pi on every page load; it means -nothing- to me. I kindly invite you to give more of a shit about user experience.

This also goes for when “complicated” systems fail, maybe making them so complicated and centralised is not the way.

pronik• 3 years ago

Early days of Twitter were just like that. Mastodon hasn't had the time to go to shit yet.

mod50ack• 3 years ago

Well, it's a FOSS self-hostable server program. Mastodon isn't a service, so it's not susceptible to enshittification per se. A particular Mastodon host, sure. But Mastodon itself is just a codebase.

candiddevmike2• 3 years ago

How could a decentralized thing go to shit?

Macha2• 3 years ago

The point of it is that it isn't a company but a standard, so if one instance owner goes crazy and patches ads in you can move to another (or your own) and keep your network

sseagull• 3 years ago

In Twitter’s early days only one celebrity could tweet at a time

https://theoutline.com/post/4147/in-twitters-early-days-only...

HN discussion:

https://news.ycombinator.com/item?id=17147404

agluszak2• 3 years ago

Mastodon doesn't have the incentive to go to shit, as there's no company trying to earn as much money as possible behind it.

meepmorp3• 3 years ago

What? Remember the fail whale?

Twitter shit the bed all the time in the early days.

redox993• 3 years ago

On Mastodon simply discussing certain topics will get you banned, and instances that don't ban those users get defederated.

I prefer Twitter in the sense that it's more laissez faire in terms of what kinds of speech get you banned.

golergka3• 3 years ago

Can I read more about this? What topics are banned?

ceejayoz• 3 years ago

“On Mastodon” is like saying “in restaurants”. There’s a wide variety of instances to various tastes.

Saris• 3 years ago

You can hang out on an instance with like minded people, everyone else might defederate you but you'll have your space.

hinkley6• 3 years ago

When someone says, "certain topics will get you banned" that doesn't make me judge the platform, that makes me judge the person.

People are going to assume it's something you don't want to name, if you won't name it.

sleepycatgirl• 3 years ago

That is.. quite opposite from the experience I have had. I have discussed a fair amount of topics, it was civil, and things were fine.

Though, the only time when I did see that happen, was when someone was transphobic, homophobic, racist and such.

bigbillheck2• 3 years ago

What kind of topics would those be?

nkozyra• 3 years ago

I prefer Twitter in the sense that it's more laissez faire in terms of what kinds of speech get you banned.

I really don't think it is. It's still largely political, and subject to the whim of the reader.

The guy who tracked and reported on Twitter Blue subscriptions was suspended today.

You always have to kiss someone's ring.

reaperman• 3 years ago

On ~~Mastodon~~ Twitter simply discussing certain topics will get you banned.

Also true.

mastodon_acc• 3 years ago

Which instance? Sounds like you joined a niche instance of a few hundred people. I find twitter to be extremely restrictive, you can’t have open discussions, you either get banned or get piled on by abusive blue check accounts.

echelon2• 3 years ago

On Mastodon simply discussing certain topics will get you banned, and instances that don't ban those users get defederated.

This is why P2P is superior. Federation nodes can be used to strong-arm collective behavior against the will of individual users.

I don't mind being exposed to liberal and conservative thought. I want to consume almost the entire spectrum of human discourse so that I can synthesize ideas for myself and understand more effectively. As long as the signal is reasonably high.

Reddit and fediverse moderators wield absolute power over their fiefdoms. They're intellectual dictatorships. (Not to mention egotistical behaviors some of them have.)

P2P allows the end user to consume what they want, weight discussions how they want, and participate in any number of emergent clusters. It's the real path forward.

mejari• 3 years ago

I prefer Twitter in the sense that it's more laissez faire in terms of what kinds of speech get you banned.

Musk literally just said that the term 'cis' is a slur that will get you banned.

pessimizer8• 3 years ago

It's a shitty monoculture mostly filled with a particular demographic (like most new and obscure tech things), the people who program computers, and the people who they meet at parties.

This would disappear with more widespread usage. The problem is the software, not the culture. If the software is improved, or the dead ends are pruned and something else is created that learns the lessons from previous tries, the new cultures will bury the old.

If building software required experts on model trains or K-pop, the culture would suck, too. The goal is to make that a stage rather than an endpoint.

edit: I enjoy model trains, but I do not get into political or social discussions with model train guys.

papito2• 3 years ago

May I suggest 8chan? Almost no moderation at all, but make sure you have a gallon of Clorox by your desk because you will want to pour that into your eyes.

sph• 3 years ago

It's MastOdon, not MastAdon.

revskill• 3 years ago

No difference to me.

ComputerGuru• 3 years ago

Mastodon absolutely does not load quickly. I clicked the link and saw the followup tweet/post/whatever for a couple of seconds before it reflowed and showed me the actually linked tweet in a smaller body font. The had to wait longer for the whole thing to finish loading. There is a ton of unnecessary JS bloat on the linked site.

iguana_lawyer• 3 years ago

Earlier I noticed the infinite reloading happening on the logout page as well

brucethemoose26• 3 years ago

Also, taking Elon's word at face value for a second... is Twitter really worth scraping for AI training or whatever?

Its a hive of misinformation, disinformation and toxicity. Its succinct I guess, but nothing is eloquent or descriptive because of the character limit. And its full of repetitive "filler" information.

Who wants that in a foundational LLM dataset?

Maybe its OK for finding labeled images... But that still seems kidna iffy.

kitsunesoba• 3 years ago

The effectiveness of this sort of lockdown is questionable anyway, because the cat's already out of the bag and there's no getting it back in. Same for Reddit. The bulk of the data's already out there and nothing these companies can do will change that.

afterburner2• 3 years ago

Maybe someone is trying to make a disinformation bot. (half-serious)

I mean as far as uses for LLMs go that seems to me a pretty realistic one. Mass quick propaganda with little effort. Go for immediate impact, doesn't matter if people look deeper, you're just looking to get a swell of emotional reactions.

brucethemoose26• 3 years ago

Yeah, I guess its a way to make an "engagement optimization" bot using follows/likes from posts as criteria.

... That is horrifying.

Hoasi• 3 years ago

Also, taking Elon's word at face value for a second... is Twitter really worth scraping for AI training or whatever?

Maybe... if you build a LLM scrapping for the lulz?

TillE• 3 years ago

It's useful if you want your LLM to be able to generate tweet-like microblogging text. That does have some value.

Or maybe you want to get an aggregate idea of what people are currently talking about in the world, stuff that doesn't rise to the level of capital-n News. There aren't a lot of alternatives for that.

brucethemoose26• 3 years ago

Output formatting or a quick finetune/LORA can do microblogging very easily.

Yeah, lots of general chat is unfortunately stuck in Twitter (or difficult -to-scrape siloed off platforms.

ben_w3• 3 years ago

The thing that LLMs bring to the table isn't factual knowledge — we already had that, even some AI projects specifically dedicated to that — but rather the ability to correctly interact with natural language.

Twitter is great for examples of that, and the toxicity and disinformation doesn't get in the way.

Conversely, a training set doesn't need to be up to date to be useful for that.

I don't know if anyone really was trying to scrape it (examples of Musk disagreeing with his own engineers come to mind), but I assume it's possible, and given the quality of code ChatGPT spits out I can easily believe a really bad scraper has been produced by someone who thought they could do without hiring a software developer. If so, they might think they can get hot stock tips or forewarning of a pandemic from which emoji people post or something — not really what an LLM is for, but loads of people (even here!) conflate all the different kinds of AI into one thing.

epistasis7• 3 years ago

While there may be huge sections of Twitter content that are like what you describe, I haven't encountered that. Instead I see tons of hyper-focused discussion from very specialized scientists that I wouldn't see otherwise. I see lots of discussion if obscure housing policy, that I wouldn't see otherwise.

Now, this has been severely degraded by the changes that Musk has made. The spam in direct messages is off the charts now, whereas in the past I would get maybe a spam per year. And when one of my areas of interest has a post that gets popular, I have to scroll past all the insipid clout-chasing replies from blue check marks which get floated to the top of replies in an attempt to reward some of the worst people on the internet. Also the long form tweets that need to be expanded are a big deflation of user experience, as reading and replying to those are suboptimal compared to a tweet thread.

But this is also the general internet: 99% spam plus 1% quality. And the quality of the 1% of good Twitter is some of the very best of timer material out there.

And since LLMs have been trained on this same mix... they seem to be mostly good at filtering. But they do lie an awful lot.

rvba2• 3 years ago

As someone who doesnt use twitter, I dont understand how can you have any sort of a real discussion with a 140 character limit.

The best discussion platform is IMHO the older version of reddit / i.reddit with the nested comments + possibility to be indexed by google + possibility to reply to old posts. The super-nesting comments feature is great.

michaelsalim• 3 years ago

Can you share some profiles/contents like this? I've been searching for it and failing miserably

brucethemoose26• 3 years ago

Well we should train LLMs from your old feed then!

I am only half kidding. "Profiles of specialized Twitter readers" would be an excellent dataset if it could somehow be filtered down to that.

exo-pla-net• 3 years ago

Don't write Elon off. If your goal is to create a toxic misinformation bot, Twitter is indispensable.

muixoozie• 3 years ago

I once got paid $20 as an undergrad to go through hundreds of thousands of tweets and convert slang into plain english for training data. The only thing I took away from the experience, aside from finally getting good with vim macros, is the average tweet is really low effort an uninteresting. I don't recall reading a single thing that I would imagine someone retweeting (think that's what it's called). Maybe I was given only replies. Anyway, not sure if there's value there for LLMs, but I'd be skeptical.

avereveard• 3 years ago

I've a scalability problem

I'll add a dedicated microservice

I've now two scalability problems

WhereIsTheTruth• 3 years ago

That's not a scalability problem, that's a poor engineering problem, let's not start to find excuses

Talents left twitter already, I wouldn't be surprised if the ones that took over are the ones who come from the intelligence industry, as opposed to the tech industry

turtleyacht• 3 years ago

As a haiku:

  I can't scale this more
  I'll add a microservice
  I can't scale these more

oh_sigh• 3 years ago

Certainly not good, but not all GET requests are equal. If these are responded to cheaply, at the point of connection termination, then it might be the case that no one has bothered to clean it up yet.

Matthias247• 3 years ago

Not exactly sure if that's what you recommend. But connection termination is not necessarily a good thing for DDOS mitigation. The reason is that the client might just retry immediatly - and it will do that using a new TLS connection. And the handshake for that connection has a huge cost. If you plan on disconnecting clients *after* a TLS connection had been established, you will also need to implement TLS handshake rate and connection limiting. That's possible, but I've only seen a tiny amount of services every implementing it.

comboy• 3 years ago

Anybody knows if these requests were happening before the login only change? Because it would be hilarious if huge scrapping operation was a bug in their javascript.

globalise832• 3 years ago

I can say for sure that a certain flow ("back" from Profile view or similar) would trigger an infinite redirecting loop on Firefox on my Android device, with probably dozens of requests over a couple of seconds until rate limiting kicked in. Maybe there were many of these little bugs which together looked like some kind of DDOS or scraping.

andrelaszlo• 3 years ago

Perhaps the engineers that understood how to analyze the logs were laid off?

samb17292• 3 years ago

I've noticed the frontend hammering the backend quite often in the past few weeks. It would not surprise me at all to learn that the "influx of scraping" was mostly Twitter's fault.

ezoe2• 3 years ago

Are you suggesting Elon found out the true culprit the hard way?

photonerd• 3 years ago

Some of the scraping was because Twitter fucked up the API so bots moved to scraping.

Dumb but obvious consequence

sgammon• 3 years ago

wow. how would one even fix this without deliberate downtime? you'd have to deploy and hope that the frontend will make it through CDNs to reduce pressure, right?

minimaxir5• 3 years ago

At minimum, you revert the commit/deploy to prod that caused the issue. But then that would likely mean reverting the recent policies and would make Elon look weak, so he'd never support it.

whateveracct• 3 years ago

Yeah frontend retry DDoS is not a great situation to get in. I've tripped it in a test env before with a websocket app (erroneous retries caused certain clients to open the WS over and over and break the server).

bornfreddy• 3 years ago

You first remove rate limits, then implement and release exponential backoff on frontend, then apply rate limits again (on a small segment of users first, then more). No biggie, you just need to be very careful. And boss needs to chill for that time, which is unlikely to happen.

avl9993• 3 years ago

First thing I would try is seeing if the front end has a different retry strategy for a different status code (say 503). If so I'd change the status returned for throttling to be that (503).

Barring that, turning off server side throttling or atleast making it less aggressive to slow the retry storm seems the most reasonable.

lapcat• 3 years ago

Dupe: https://news.ycombinator.com/item?id=36553762

brigadier132• 3 years ago

This is why you always use exponential backoff.

fathyb• 3 years ago

And when you're at Twitter scale, sprinkle some jitter too.

oblio2• 3 years ago

What do you mean?

globalise832• 3 years ago

This is why you SHOULD always use exponential backoff. ;)

mmastrac• 3 years ago

There may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

(thanks RFC 2119)

stan_kirdey• 3 years ago

self ddos with backoff, :chef kiss:

Waterluvian5• 3 years ago

I wonder if exponential backoff should be the default behaviour for request libraries/APIs.

Their default of “just go ham on that API” feels like the same footgun of “by default this Humongous Database is wide open.”

DropInIn• 3 years ago

I doubt it ever would become the standard unless everyone was using third party libraries that forced it in some way, most likely opaque by default which would cause plenty of devs headaches, right?

The easiest path will always be the default for the majority of devs, with a simple "timer" type solution being the easiest to implement in pretty much all cases except where otherwise it's literally forced on them.

Topfi4• 3 years ago

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement.

Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof.

I can of course not say whether the people currently working at Twitter did warn that the recent measures could have such major side effects, but I would not be surprised in the slightest, considering their leadership's mode of operation.

Even as someone who very much detests what Twitter has become over the last few months and in fact did not like Twitter before the acquisition, partly due to short format making nuance impossible, but mostly for the effect Tweets easy embeddability had on reporting (3 Tweets from random people should not serve as the main basis for an article in my opinion), I must say, I feel very sorry for the people forced to work at that company under that management.

Waterluvian5• 3 years ago

At least once have I done a CC blast as a form of insurance when being asked to do something absolutely boneheaded.

7thaccount• 3 years ago

You gotta get executives in writing sometimes or they'll throw you under the bus later.

madaxe_again• 3 years ago

I have usually just produced a disclaimer and asked the relevant party to please read and sign it, indemnifying me against any consequences of the decision they have made.

That usually gets them to sit up and reconsider.

Occasionally they sign the damned thing, and then it’s popcorn “I told you so” time.

choppaface2• 3 years ago

Well "forced to execute" is somewhat subjective. If you are convinced leadership is doing the "wrong thing," then best to either leave or accept that you're just collecting your paycheck.

In the case of Twitter, the new owner has thoroughly broken the advertising business and is trying to aggressively pursue a new version of the data business Twitter once had-- E.g. Google's Caffeine, which Twitter also eventually lost https://searchengineland.com/google-search-algorithm-change-... ... The statements about "too many scrapers" are almost certainly as illegitimate as the previous pre-acquisition statements about "too many bots."

The nature of business is that there's no judiciary or referee... the purpose of a business is to make money. Tech businesses just happen to hire lots of academically-oriented engineers who developed their skills in a different environment. It's possible to build a culture of "fairness" in a business, but at the end of the day even Google dropped "don't be evil."

pavel_lishin• 3 years ago

Well "forced to execute" is somewhat subjective. If you are convinced leadership is doing the "wrong thing," then best to either leave or accept that you're just collecting your paycheck.

Я, Unless your visa is sponsored by your employer.

DonHopkins4• 3 years ago

Wouldn't it be more efficient if Twitter just served as a broker for connecting scrapers directly to bots, so they didn't have to pay the bandwidth charges?

Consultant324523• 3 years ago

I don't understand why anyone, at any job that is not C-level, has an attitude of anything other than collecting your paycheck. I care about my professional reputation and my career, but I don't care at all about the well-being of the company that has hired me to perform some task. I will advise to the best of my ability, but if they blow themselves up, why should I lose any sleep? That's their problem, not mine.

Detaching myself emotionally from my employer was one of the best things I've ever done for my mental health. When I was young I got upset when "the company" made bad decisions. Now I feel no negative emotion about it, sometimes I laugh at them.

beebmam2• 3 years ago

A union vote is the only way to save that platform. Either unionize or let the company die.

drekipus• 3 years ago

Unions are for the people, not the company

sanderjd5• 3 years ago

I'm curious about this: When you were experiencing this, why didn't you quit? I truly feel bummed for the people remaining at Twitter who are miserable but for whatever reason feel they can't just quit.

nicolas_17• 3 years ago

https://www.vice.com/en/article/n7z5px/twitter-employees-on-...

Topfi4• 3 years ago

That is a good question, but on the two specific occasions when something of this sort did not make me leave instantly, there were very specific factors at play. I cannot comment on the first situation as that would require an extensive post with an obscene amount of context on my background, politics in my country, the people involved, weighing up the public good, etc. Simply, would go beyond a HN comment and I feel I couldn't do the situation justice anyway for someone who wasn't following local news at the time these things happened.

The second instance, I feel I can comment on, though allow me to leave out details to spare the people involved public embarrassment.

There, the changes, whilst annoying and born out of a complete misunderstanding of a core part of their alleged competency (Imagine a Botanist telling you leaves are always blue because that is the color of the sky), were not going to break anything, just look silly and create unnecessary, but compensated, work.

In that case, I also viewed the specific project finally launching as vitally important to our user base and wanted the results to go public for their benefit, so the decision was made to document and execute on their requests, so we could go live.

Of course, two weeks before our go live date, they changed their requests again ("leaves are not blue, they are violet because of the wave length of light") and had a hard time understanding that changes can have a knock on effect and some things are a bit more complex than Find and Replace. If I had been forced to make those changes at that point, I'd have packed my bags.

Simply said, when my work has the potential to benefit users and I know that arguing, even though I am correct, will lead to massive delays, I'd rather just put that silly request in writing and deal with these things after the users have received what has been worked on. Try to explain when someone is wrong, but if that doesn't work, finish the project, argue later.

Of course, if after the fact responsibility isn't taken, that finished project gets mentioned in my CV for the next employer.

Frost1x• 3 years ago

There are a lot of reasons people stay in positions. Not everyone has the same degree of mobility in terms of a job. Maybe they have little time in their personal life to deal with interview prepping. Maybe they're older and deal with age discrimination. Maybe they're on a work visa and know they're in a leveraged position. Maybe they're not good at negotiation. Maybe they loathe the market. Maybe they have a pile of debt and the current situation is less risky.

As an industry we need to stop perpetuating this idea that everyone is fully mobile and software engineers are some mythical creature that isn't also shackled to the same constraints the rest of the labor force is. It's a job and a lot of people are dependent on a steady income stream. Part of that is putting up with the least worst of the options they have reasonably available to them, or sometimes taking up worse options just because it balances other aspects out. Ultimately during certain times you may have more or less leverage to put up with or reject crap practices. SWEs tend to have a lot more leeway than say a retail worker of course so we can be a bit more demanding but we can't just say crud off, I don't deal with any BS, even in the best of times. Higher salaries give you some negotiation room as well as you can lower your TC expectations in terms of negotiating better working environments (i.e. taking lower rates but demanding things like less pressured schedules, make autonomy, etc.) and as the extreme sof the markets are high enough, even taking TC hits can still let you live comfortably (this is to contrast many other professions who don't have this flexibility).

Unless you've lived modestly and invested significantly or started some side venture to become financially independent of labor based income, you're ultimately giving up some degree of agency to employers. We like to pretend this isn't the case but it is the case for most people, even highly paid labor.

praisewhitey• 3 years ago

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement.

In this case the horrible idea is being forced to push changes to production at a moments notice

tbrownaw• 3 years ago

push changes to production at a moments notice

Isn't that one of the DORA metrics?

Topfi4• 3 years ago

Yes, that is very much true. Even if, as some have commented, the people remaining at Twitter lack oversight over the entire codebase to predict that this specific change could have such a knock-on effect, not having testing and staging environments, pushing something straight to production, would give even an inexperienced person pause for thought.

As said, I don't want to speculate whether someone raised their voice on this being a likely outcome specifically, as not having testing prior to release is the much more obvious and significant issue here, driven by leadership.

martincmartin• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at least one of them was fast.

I worked in the games industry for a while, and came to understand how they could spend so much money and so much time, and yet release a game where even basic functionality was broken. It's exactly this sort of extreme schedule pressure that, ironically, makes a huge morass where changing one thing breaks 10 other things, so progress grinds to a halt.

unconed6• 3 years ago

Except, progress grinding to a halt pretty much described Twitter before Elon Musk too. The story that anyone who knew anything was let go is just that, a story, as the number of employees by year shows Twitter, and other tech companies in the same period, overhired and then went back to pre-2020 levels.

The only difference between then and now is that there is a big personality at the top who now personifies everything Twitter does, especially if things go wrong, whereas before it was mainly just a faceless bureaucracy whose Trust and Safety lead at times had more visibility than the CEO.

The subtext is that Twitter changing hands also involved trimming a lot of the dead weight, particularly hitting the softer managerial/diversity/HR side. Now a lot of people are rooting for the site to fail because it has gotten too "bro-ey", as the era of trust-and-safety and $15k backchannel bluecheck deals has made way for free-speech and monthly subscriptions.

walrus01• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left.

This is like a case study in what happens when you fire everyone except the sycophants and yes-men.

I only feel sorry for remaining non-yes-men twitter employees who might still be there because for whatever personal reasons they're in a precarious economic situation where they can't quit (H1B?) or are tied to the company for healthcare coverage (Thanks, America, for being the greatest country in the world) because they can't afford any other health insurance option.

andsoitis3• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left.

Are you saying the engineers who are now at Twitter don’t have the right skills?

humean1• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left

And never created sufficient testing.

sigzero• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left.

You have zero idea if that is true or not.

jayd163• 3 years ago

The people at Twitter who understood the system

But this is Scaling-101 stuff. It's not some super complex or unique system going wrong. At least according to the article, it's a classic case of bad retry logic leading to a death spiral.

https://en.wikipedia.org/wiki/Thundering_herd_problem

cameronfraser2• 3 years ago

So about the only thing they could do was issue multiple, parallel requests and hope that at least one of them was fast.

lol nobody would do this to solve this problem because it doesn't even remotely solve it or give the appearance of solving it, if anything it's guaranteed to make things go slower

hvis• 3 years ago

My guess is Elon said "the site's too slow!"

Seems more like a thinly disguised attempt to force people to purchase subscriptions. Blue checkmarks and all.

ineedasername11• 3 years ago

The people at Twitter who understood the system and could predict the side effects were all fired or left

Not necessarily. I’ve predicted bad outcomes for decisions in a few cases and been ignored but stuck around regardless. Mostly because I like my job and the goals of my organization even if it makes bad decisions.

Of course to remain productive and improve my influence in future decision making it is absolutely critical that when predictions come true, I do not go anywhere near an “I told you so mentality.”

Instead I do what I can to clean up the mess with a “how can I help?” attitude. And increasingly over time people take my opinions and analysis much much more seriously.

I wouldn’t say that’s the path everyone should take, especially because some work environments are just too toxic for any progress at all (I ran away, fast, from two jobs like that). And some people cherish having an entirely new type of challenge every few years instead of shepherding something through longer periods of time. All valid paths.

Topfi4• 3 years ago

It is very possible that they did not predict this specific down stream effect, for one of many reasons. I do not want to discount that, though, as I am not at Twitter and am not on a first name basis with anyone there. Maybe there are some who did see this.

What I maintain though is that most anyone still working on code at Twitter, regardless of their experience or overview of the code base, would strongly argue for testing and staging, which appear to go against current leadership's mode of operation, likely because of the time pressure you mentioned.

Not pushing such changes straight to production is a concept I feel anyone working at Twitter would subscribe to, yet has to painfully go against, lest they be led go.

ezoe2• 3 years ago

It's like stupid boss asking to turn off the air conditioner of server room to save power.

goalieca• 3 years ago

I’ll play the devils advocate here but frontend devs need to smarten up. This is basic error handling that should have been in place for years. Blocking tweets with 403 or whatever they chose shouldn’t trigger endless retries on short intervals.. ever!

jlund-molfese• 3 years ago

I’d bet (not that much, but like $20) that someone has a .ifClientError() or if responseStatus === 4xx somewhere.

If you’ve never had to handle authorization in a particular area, it might have been safe to assume that any 4xx error should have been retried when the code was originally written and someone didn’t write them all out

giovannibonetti• 3 years ago

Maybe the frontend team and QA were never made aware that 403 errors could be triggered like this.

sheepscreek2• 3 years ago

..and using exponential back-off, if not limiting the number of retries.

Though it’s hard to know for sure what really went down. Could be a number of things. Including a lack of subject matter experts (Elon recently admitted to laying off some people they shouldn’t have).

dimmke• 3 years ago

You're right about error handling, but consider that the user who posted this was logged in. The screenshot they post is of a specific tweet, but they reference the home feed being down. What kind of API call was it making where it's asking for a list of tweets to serve to a logged in user and that call is not authenticated? It makes me think that the blocking was implemented improperly.

Unless the home feed being down is simply a side effect - the service that fetches tweets being DDOS'd by other views in the app making numerous non authenticated calls.

But I was also thinking about this earlier today. These days, everybody is so quick to say "the software is easy, it's the community that's hard" - I've even said it myself a few times in the past few weeks, but I think that might be overstated.

Building good software is hard. Keeping it good is even harder. What does the codebase look like for Twitter's front-end at this point?

How many frameworks has the base functionality been ported through? How many quick pivots from product adding features, adjusting things squashed down the ability to address technical debt or even have functioning unit and regression testing?

The fact that this 1. Made it to production and 2. Was not noticed and rolled back immediately (like, in under 30 minutes) is extremely concerning (and obviously very embarrassing.) If I had private data stored on Twitter of ANY kind (like DMs that I don't want getting out - a messaging system rich, famous, and powerful people have been using like email for over a decade), at this point I would be trying to get that data removed however I could, or accept that there's a strong possibility there's going to be a huge data breach and all of the data will be leaked.

the-rc• 3 years ago

Not just frontend developers. The backend should serve a 429 or 503 error, complete with a Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re...

That would give the server side more control over the retrying logic (when the header is properly interpreted). I'm surprised Elon hasn't implemented this himself.

hakanensari• 3 years ago

Just speculating, but the server is probably throwing 429 (Too Many Requests) errors, and yes, the client should respect and back off when it encounters that. According to the post, it seems not to. Strange!

yxre• 3 years ago

I don't think its intentional. I think it's a side effect of using a listener pattern.

If you use a listener, useEffect in react, to load data, it will start the request, track it is loading with a boolean, and then store the payload. That passes unit tests and QA.

If the listener doesn't check the error before starting the api request again, you have this infinite loop happen where the loading flag goes off and the payload is still null, so it just starts it again.

It's sloppy code, but its an unintentional side effect.

cyanydeez• 3 years ago

Ok, but the only people left are either hostages or sycophants.

Doubt either care at this point.

agentgumshoe2• 3 years ago

Or they're just people trying to earn their pay. Why must everything be a political divide these days...

tomcam2• 3 years ago

How do you force someone to work at Twitter?

FranksTV2• 3 years ago

If they quit their H1-B gets revoked.

AbraKdabra• 3 years ago

I feel very sorry for the people forced to work at that company under that management.

Forced? You know slavery was abolished a long time ago no?

oddevan• 3 years ago

"Forced" could also mean holding someone's visa hostage, threatening consequences in addition to losing a job, or just in general being in a situation where, while they are not physically or legally required to keep a job, they have no other viable options.

sh34r4• 3 years ago

Unless you are in prison. You think I'm BSing then read the 13th amendment.

r00bot• 3 years ago

I'm going through this right now in my job, having built a tool to manage vulnerability tickets for my team, and having it's first use be deleting every vulnerability ticket, despite my protests. The person executing this is more concerned about looking good on paper than actually improving security.

Shorel• 3 years ago

It's just another requirement. I don't see anything terrible or even bad with the requirement itself. Rate limiting is the new normal, in a post ChatGPT world. The implementation, of course, is atrocious and should be corrected. You are putting your opinion over the requirements of the job, and basically guaranteeing a bad outcome and an atrocious implementation. Maybe you can share your stories about real bad decisions, so we can judge properly, but so far this is not a good overview.

maxlin7• 3 years ago

That's nothing. Instead hopping on the bandwagon hating on the thing we all use and care about, a situation where you would know a solution but are not allowed to act because of fear of even trying anything is so much worse. Especially when you know of it for a very long time and are not even allowed to try, while also being ostracized and being "that guy" for suggesting it repeatedly. Talking of different technical and business approaches to things where the "original" still somehow works but is slowly killing the company. Been there, done that.

But when a higher-up so clearly takes responsibility of something that is risky but deemed necessary, it provides for a lot better space to also respond to any potential problems that pop up, while not needing to suffer snarky comments in hindsight as a dev.

Of course something like this is easier to hate on ...

WeylandYutani• 3 years ago

Don't use Twitter and don't care in the slightest if it goes down.

Will admit that seeing a billionaire go mental is funny. Petty I know.

isaacremuant• 3 years ago

This is all speculation.

The push to "make the content less user friendly" and push then to our app/logged in experience is similar to reddit's push and it doesn't really present an insurmountable challenge on the technical side.

These technical hurdles are temporary. Not the end of the world. The "only in our app/only logged in" requirements are bad on a philosophical level of user choice, not on the "this is terrible technically" side.

Attacking it on that note is just perpetuating the narrative where elonmuskmanbad (which is more political than consistent with the treatment others get) above more specific truths.

thaumasiotes• 3 years ago

the effect Tweets easy embeddability had on reporting

This is something I don't get. They say that Twitter is now blocking embeds and I've seen an author respond by... including a screenshot of a tweet in their piece.

Except that's obviously better than embedding a tweet in the first place. It's better in every possible way. It's easier to write your article with an included image that you provide than to hotlink content from some other website. And when the tweet vanishes into the mists of history, your captured image of it will still be around, illustrating your article the way you were hoping the embedded tweet would do. There are so many articles out there with dead embedded tweets illustrating... something.

So, all that said... what was the effect that embeddability had on reporting? Why hasn't it been screenshots the whole time? What did easy embeddability enable that wasn't just as easy anyway?

lamontcg6• 3 years ago

I think we're about at the point where the people who predicted chaos at twitter after Elon basically fired most of the experienced engineers have been proven correct. The duct tape is all coming apart at the seams now.

It isn't quite as decisive as a submarine imploding, and ceasing to exist, but it has turned into a brightly burning tirefire.

bravoetch• 3 years ago

25 years of products being honed for shareholder value, instead of customer or user value. We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences.

OnlineGladiator2• 3 years ago

We may be at peak consumer tolerance for anti-pattern, in-app purchase, subscription-model, ad-packed, data-siphoning, dopamine driven, gated experiences.

As much as I want this to be true, I think this sentiment is really only popular on tech-savvy forums like HN. Most people don't use ad blockers, and I've had people get mad at me when I suggest that they do (directly in response to something where they are complaining about ads).

bozhark• 3 years ago

AAA video game industry sure seems to be pushing this idea with the past years of broken, unfinished, beta projects being released as complete products.

hyperpape5• 3 years ago

This is a perfect example of how "shareholder value" is a thought-terminating cliche.

Twitter was previously a public company, which was beholden to shareholders, and aimed to try and increase its stock price (as far as "shareholder value" actually means anything, this is basically it). I wouldn't praise previous management (the company wasn't profitable), but they were not a complete dumpster fire.

Then Twitter was bought out, and taken private, removing the obligation to "shareholder value." The ensuing dumpster fire is one that will be marveled at for years.

I'm not saying public corporations are better than private, or that "shareholder value" is a good slogan. I'm just saying that your comment is every bit as irrelevant as the porn spam that's clogging Twitter these days. (Thanks for fixing the spam problem, y'all!).

lr4444lr• 3 years ago

I dunno. We are several months out after the major layoffs. Maybe some very recent bad decisions were made internally that pared back too far, but I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover.

polygamous_bat• 3 years ago

Do you think if we fired every civil engineer tomorrow, the bridges and the highways they built would fall apart right afterwards, or even in a few months?

Engineering isn't like service positions where the lack of competent personnel is felt immediately; the debt keeps growing until your whole system collapses under it one day, how far the day is in the future depends on what system you're working on.

firesteelrain4• 3 years ago

Agree. Plus it’s easy to crap on Elon but it’s also the poor Twitter architecture and quality of people they have working for them that caused this despite Elon’s desire to require login to read Twitter. He didn’t write the code.

simonw3• 3 years ago

Complex systems like Twitter don't break overnight when you lay off the talented engineers.

They deteriorate piece-by-piece, potentially over the course of many months, until the compounding effects of these problems and the growing technical debt overwhelms the team that they have left.

Marazan3• 3 years ago

It is a testament to the engineers of old Twitter that their systems stayed standing for as long as they did.

anders_p• 3 years ago

I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover.

It REALLY sounds like you don't understand how any of this works.

Tech products don't stop working when you fire most of the staff.

But bugs stop being fixed and problems begin to add up, until a critical point is reached,m where the whole house of cards collapses.

Thinking that "Elon was proven right" simply because Twitter didn't implode the second he announced the layoffs, makes me think you don't understand how tech and software works.

avl9993• 3 years ago

I am not disagreeing with you but self-ddos is not entirely uncommon. When I worked at Amazon this would happen a few times a year. Not on the main amazon.com website but on supporting services often initiated by but not limited to kindle devices. Having something like this slip through the cracks of even experienced engineers isn't uncommon.

Tade0• 3 years ago

But in such cases surely there's some kind of rate limiter in place?

I mean, if I'm reading that screenshot correctly this is 700+ requests a minute.

I've tripped the rate limiter with less on other sites.

lamontcg6• 3 years ago

Yeah it happened when I worked at Amazon as well. I also more recently worked pretty closely with people at Facebook and knew something about the issues they would occasionally have (which didn't match what the headline speculations were at all). But twitter is repeatedly having these kinds of issues.

dcunit3d12• 3 years ago

I tried to point this out below. It's not even a DDoS, it's fake news.

https://news.ycombinator.com/item?id=36561808

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"

fdsa

terminatornet3• 3 years ago

right? it's a shame hacker news doesn't let you reply to old comments, there's a few threads I'd like to follow up on from people who refused to believe Elon was destroying twitter

Marazan3• 3 years ago

I think I had someone do the old sarcastic "remind me of this in one year" under one of my posts predicting technical doom for Twitter.

Definitely a good one to revisit.

hsbauauvhabzb2• 3 years ago

I think it’s intents like yours which is why they’re not allowed. Shame, because there are some genuine value in having it too..

dcunit3d12• 3 years ago

I tried to point this out below. It's not even a DDoS, it's fake news.

https://news.ycombinator.com/item?id=36561808

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life" ...

shon4• 3 years ago

I have to disagree. If you honestly take the emotion and politics out of this and evaluate on merit alone, what do you think?

Twitter wasn’t healthy before Musk bought it. It wasn’t a thriving business, it was a very old, very large startup still struggling to find market fit and loosing a lot of money.

Also, it wasn’t a thriving product. It was stagnant.

Since Twitter was purchased, the amount of features they have shipped has been impressive. They’ve shipped a lot of features and extended the platform a lot. To your point they have also done this with far less engineers than before.

Regarding any downtime, everyone has downtime. Google, Amazon, Meta… the best of the best still have it regardless of money or manpower.

Considering what that team has done with less resources, I think the achievement still pretty good. What do you think?

mooreds• 3 years ago

It wasn’t a thriving business, it was a very old, very large startup still struggling to find market fit and loosing a lot of money.

Twitter was profitable in 2018 and 2019.

https://www.netcials.com/financial-net-profit-year-quarter-u...

felipelemos• 3 years ago

[...] it was a very old, very large startup [...]

Can we start to call companies with almost 18 years old just "companies" and not startup anymore?

rvz• 3 years ago

Regarding any downtime, everyone has downtime. Google, Amazon, Meta… the best of the best still have it regardless of money or manpower.

I mean, I would expect Microsoft to do a much better job than Twitter to keep GitHub from going down every single month after acquiring it. The frequency of GitHub going down with 100M+ users using it is much worse than Twitter.

It turns out that GitHub's constant downtime for years is all fine (especially tech folks) here despite the monthly complaints anyway. The latest one here [0] But only with Twitter, the speed-bumps are exaggerated and magnified.

[0] https://news.ycombinator.com/item?id=36523843

lamontcg6• 3 years ago

https://www.npr.org/2022/11/25/1139180002/twitter-loses-50-t...

Yeah, it is going great.

I see HackerNews is counterintuitively up its own ass again.

snowwrestler3• 3 years ago

The company was not thriving but the product itself was rock solid.

It’s hilarious to think it is at all acceptable to kill public access, and drastically limit authenticated access, because of a few scrapers. There is no way Twitter prior to Musk’s acquisition would have had to do so.

What do you think?

I think you are not looking at the situation objectively.

dcunit3d12• 3 years ago

I tried to point this out below. It's not even a DDoS, it's fake news. it's so "not even wrong"

https://news.ycombinator.com/item?id=36561808

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"

shawnc2• 3 years ago

My 14yr old daughter sent me this screenshot the other day with the comment "I think that one sentence sums up twitter pretty well".

https://capture.dropbox.com/GqgTAxRimqAXzrdo

viraptor2• 3 years ago

Fyi, on mobile the image doesn't load and the download link is broken.

mkl2• 3 years ago

Loads fine for me on Firefox Android (but I don't get which sentence is supposed to sum up Twitter).

furyofantares• 3 years ago

I see a giant circle around like 10 sentences, none of which feel like the sum up twitter in any way? After some consideration I guess maybe you're referring to "What's happening?"

shawnc2• 3 years ago

Fair enough - the sentence in question was indeed 'What's Happening?'

LexiMax2• 3 years ago

"What's happening?"

That's a very good question.

tedunangst2• 3 years ago

Editor's Choice?

dcunit3d12• 3 years ago

Your 14 y/o daughter is fired.

I tried to point this out below. It's not even a DDoS, it's fake news.

https://news.ycombinator.com/item?id=36561808

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"

charcircuit3• 3 years ago

It's not a self ddos if twitter isn't going down. You can see in the video twitter is properly handling the load and is returning HTTP 429 when the client is sending too many requests. Hitting the rate limiter or requesting a post is light weight. It's not like it's spamming login requests which require a lot of resources due to key stretching.

thakoppno• 3 years ago

2 Nines is sufficient at scale?

charcircuit3• 3 years ago

For a consumer facing surface. Yes.

notquitehuman• 3 years ago

It’s Twitter. A nine is more than enough for their most critical workloads.

simonw3• 3 years ago

The theory here is that this JavaScript bug caused the huge increase in requests which is why Twitter introduced strict limits on how many tweets users could access.

So the result wasn't an outage, it was a radical reduction in functionality.

I think that still qualifies as a self-ddos.

hn927268192• 3 years ago

It's not a self ddos

Maybe not

if twitter isn't going down

I disagree. Ddos is a type of attack, not the result of an attack. If they're hitting their services way too many times in a distributed fashion, it's a ddos regardless of how it was handled.

Similarly, would you say this[0] wasn't a ddos because it was mitigated? I think not.

[0] https://blog.cloudflare.com/26m-rps-ddos/

charcircuit3• 3 years ago

Self ddos is more fuzzy because there is no intent. If I wget all pages from a site's site map is that a dos? If it was architecture to handle that kind If load the site would be unaffected and if wasn't it could potentially cause availability problems. If after starting my download script I noticed the server couldn't handle my requests then I would recognize I am dosing them.

grumple• 3 years ago

"Yeah, let's roll this out on a Friday night before the biggest national holiday weekend of the summer."

noncoml• 3 years ago

Perfect example of why you don’t just fire all your senior engineers.

You can’t debug and root cause if you do. So you end up adding hacky point fixes

wand3r2• 3 years ago

I definitely agree. I do think the major issue at Twitter is more managerial than engineering though. They likely could have run Twitter at the same output with the current team. The problem is a CEO who is learning on the job by altering basically everything at a superfast pace.

The root cause is business & feature experimentation at scale with a tight runway & no executive oversight.

pschuegr5• 3 years ago

To say he's learning on the job is speculation on your part. He's on the job, I'll give you that.

pfisch2• 3 years ago

Threads is about to launch and I have a feeling once that happens twitter will quickly become the next Truth Social.

rsynnott4• 3 years ago

Only without the main attraction. Always seemed like a weird strategy, that.

ceautery• 3 years ago

Central Services finally caught Archibald Tuttle, now everything is falling apart.

southwesterly• 3 years ago

Brilliant reference. Absolutely zapped it. Legend.

EdwardDiego4• 3 years ago

Bloody typical, they've gone back to metric without telling us.

zagrebian2• 3 years ago

The Elon Twitter movie will be the next Borat, I can’t wait.

henry20232• 3 years ago

It’s already being produced. I heard that the budget was US$44B

x86x873• 3 years ago

Let that sink in!

Right?

hinkley6• 3 years ago

Don't interrupt your opponent when they are making a mistake.

justinhj• 3 years ago

Why do people oppose Twitter? It's a social network, you can choose not to use it.

agonz253• 3 years ago

Putting the personality of the owner aside, they have been running an experiment for the last few months on how to operate a tech company, overworking your engineers and treating them like replaceable automata. Do you want them to succeed and set that as an example for others to follow?

Barrin92• 3 years ago

I've got a rule of thumb, if a communication network is so large that governments who control nuclear arms use it to communicate "just ignore it" doesn't really work

AHOHA• 3 years ago

Personally, not just twitter, all social media I’m happy to see it dies, and no, “stop using it” mindset isn’t an argument when it severely affects the majority of the people around you, and start not just influencing the populace opinions but shaping the next generation. Who ever controls that narrative in the media get to decide what’s the next laws to be considered, after all, that’s how democracy works.

vba6167• 3 years ago

Some people signed up for Twitter, years ago, specifically to avoid other people using their name. I think it was a thread on HN.

Once you are signed up, it's tempting to use it. Then you realize it's a bad idea to use it, but it takes self-control.

Now, with the current leadership, who knows if they will purge inactive accounts and/or reissue IDs.

I don't know if this scenario accounts for much of the antagonism, but it is one possibility. It is possible that some people don't like it, but can't let go of it completely, and the new management creates uncertainty and doubt that never ends.

throwawayswiss3• 3 years ago

That social network shut prominet scientists voice during the COVID that's why many people like me hate it

DecXicon-282• 3 years ago

Unlike what people imagine, the selfDDOS bug that occurs does not burden requests to the Twitter system, but becomes a loop for users requesting access.

dcunit3d12• 3 years ago

yeh, i tried to mention this, but it got flagged quick. it's unbelievable that my median salary for the past decade is $3,000 after several years of "Startup" followed by "How to Start a Reboot of My Life"

https://news.ycombinator.com/item?id=36561808

it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

DecXicon-282• 3 years ago

Not as people imagine , the selfDDOS bug that occurs does not burden requests to the Twitter system, but becomes a loop for users requesting access.

wzy• 3 years ago

So this "unforeseen" loop now eats in the users 600 tweets per whatever, so now when you land and Twitter's homepage you are pass this measly limit with 5 rotations of your scroll wheel.

danShumway2• 3 years ago

I do think this probably isn't the reason for rate-limiting tweets so harshly; I don't know for sure, but I suspect this isn't a technical issue and it is legitimately Elon snapping and just implementing a wildly bad business decision.

But that being said, a loop of users requesting access from Twitter's system is a thing that would burden Twitter's system.

Requesting access and denying access isn't free. You can definitely DDOS your own website by having all of your users repeatedly request access to it in a loop.

nwoli• 3 years ago

Seems like it might not even be a bug. Elon says they limited it to 600 viewed tweets per day which is an insane limit. Most people would go through that in 5 minutes of scrolling

samb17292• 3 years ago

I noticed the frontend hammering the backend for the past few weeks, so I suspect that these new rate limits are a response to that, even if Musk wouldn't publicly admit it.

I don't doubt that Twitter saw a massive increase in traffic recently, but I feel at least somewhat confident that it's mostly self-inflicted on Twitter's part.

impissedoff1• 3 years ago

Might be time to reflect on how much useless information we consume

johannes12343212• 3 years ago

So many things are useless, but still entertaining in one way or another.

And then Twitter is used for "relevant" information as well, from emergency or transport authorities using it for quick information instead of having their own system to news of any kind or discussions and information exchange in different peer groups.

Considering that they count all tweets being loaded towards the quota (for instance replies to a tweet, if you open the detail page) and they like to shove tweets from Musk and others in the timelines the quota can be reached quite quickly even on considerate use.

devnullbrain• 3 years ago

Entertainment is justified and good.

dyno123452• 3 years ago

<pause>

smcl3• 3 years ago

5 minutes is maybe an exaggeration. I know that when Tweetbot reported I had >500 in my feed I had enough nonsense to read for a couple of 20 minute tram rides.

aeyes2• 3 years ago

This bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active.

I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some expensive work before returning an error but that's why we have rate limits on all public endpoints.

I also guess that the webapp is the least of Twitters traffic and the native apps probably don't have this problem.

epistasis7• 3 years ago

One thing about having leadership that is known to lie about anything or everything, for any sort of imagined personal gain, is that the very concept of truth is destroyed.

I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a complete waste of my time, but it's a weird mental honeypot.

The book "Nothing is true and everything is possible" describes Putin's use of misinformation to maintain control of the populace and eliminate democratic types of politics, but it really feels like it applies here too. There will always be Musk fanbois who will parrot whatever he wants them to say, but most know it's just self-serving BS. And anybody trying to get to the root of everything gets easily sidetracked into narratives that feel right but have zero data backing them, like this bug.

Anyway, highly recommend this book if you want to see a likely path for the future of the US:

https://en.wikipedia.org/wiki/Nothing_Is_True_and_Everything...

kristianc2• 3 years ago

That’s exactly it. Musk a week ago was telling us that there were a record number of user seconds on the site. Now he’s telling us they’re all content-scrapers. The very concept of truth is eroded.

concordDance4• 3 years ago

One thing about having leadership that is known to lie about anything or everything, for any sort of imagined persona gain

I will note that the few times I investigated claims of Elon lies they were not proper lies, either being misunderstood, misleading (which IS unethical, don't get me wrong), of indeterminate truth value (he said, she said type stuff), delusional optimism or actually true.

Like journalists, Musk rarely outright knowingly makes literally false statements, but this does not mean you should take what he says at face value.

EdwardDiego4• 3 years ago

I feel like that book title is paraphrased from the maxim of the leader of the Assassins in that Alamut novel.

"Nothing is an absolute reality, all is permitted"

hgsgm2• 3 years ago

George Costanza said "it's not a lie of you believe it."

Is it a lie if you don't know or even care if it's true or what it means? That's where we are with Elon Musk.

sanderjd5• 3 years ago

I know this is getting too political so I'll deserve the downvotes, but totally ignoring anything else, this is why I fundamentally could not and cannot stand Donald Trump. This is his playbook too, just constant willful dishonesty and distortion.

Robotbeat• 3 years ago

Man, the really annoying thing is that Elon-haters are so dishonest (refusing to engage when obviously wrong claims are made, repeating things that have been long disproven, repeating hearsay as fact, etc), that it's hard to actually identify any actual lies Elon has made. Meaning, not just exaggerations or failed predictions, but straight up intentional lies.

My favorite is when things are claimed to be impossible, that Elon's lying because the thing is impossible, but then it happens anyway.

Where Elon gets in trouble is he's wildly over-optimistic on a few things, such as AI. He predicted an AGI would take over in 5 years around 2015 or so (so we're 3 years off), and I think he really believed it. That's why he's always claiming things like self-driving, and doing it without sensors or whatever. His paranoia of AGI and his over-confidence on self-driving have exactly the same root cause (believing AI will conquer all). Elon has had so many instances of overcoming status quo expert predictions (whether on solar energy, battery-electric vehicles, reusable launch vehicles, or whathaveyou), that I think it makes him increasingly unable to very effectively listen to experts.

And he's also incredibly gullible and easily taken in by all sorts of scammers, including rightists and just plain sycophants telling him what they think he wants to hear. Which is increasingly what he's left with as everyone else who is sick of his bull has left.

mrcwinn2• 3 years ago

This.

The media is flawed. Journalists are flawed. The Verge in particular has some atrocious coverage of tech.

But the problem with Elon is that in his world, we must rely on him for true information. As if the direct source is inherently true. He’s just as full of it as anyone else with interests.

evan_2• 3 years ago

I don’t think it’s necessarily saying the self-inflicted DDoS has caused a technical issue that’s forced them to shut down access. I think it’s possible that shutting down anonymous access caused the DDoS, which led to giant spikes in some metric, which led them (Elon) to conclude that there was an uptick in scraping, so they imposed the 600/tweet/day limit to punish scrapers.

Seems like either my quota reset or they changed the policy because I’m able to access the site again.

pschuegr5• 3 years ago

This. I'd bet substantial amounts of money that the evil scraper idea is the result of a) another issue + b) paranoia + c) Musk thinking he understands better than anybody else.

olalonde5• 3 years ago

According to Elon, shutting down anonymous access was itself an emergency measure to deal with the DDoS[0]. Twitter did increase the quota significantly from earlier today[1].

[0] https://twitter.com/elonmusk/status/1674865731136020505

[1] https://twitter.com/elonmusk/status/1675260424109928449

redditor98654• 3 years ago

Depends on the scale of the overall system. I have personally seen and attempted to mitigate degenerate cases where these retries overwhelmed the backend so much that the servers were falling behind in simply rejecting the requests.

Infact it got so bad because of all those retries at multiple levels from upstream callers that requests were essentially timing out at the TCP buffer/queue before they could be processed by the application.

Don’t know if the Twitter homepage backend is at similar scale.

aeyes2• 3 years ago

It is unlikely that a system with the scale of Twitter implements the API rate limiter in the backend. Usually you'd do this as early as possible together with other WAF stuff.

If IPs or IP ranges get really annoying we block them on the network level.

Big public sites like Twitter obviously need to have this technology. Due to their political content they probably also need sophisticated DDoS protection.

fwlr2• 3 years ago

Oh come on. This is console.log spam. This isn’t a “self-DDoS”.

mananaysiempre• 3 years ago

No, look closely: the animation does not show the console, it shows the network pane. Each line is a (presumably unsuccessful) request to the Twitter backend. If a lot of people have the Twitter web frontend open and running in this state, it could in fact overload the backend.

fwlr2• 3 years ago

Sure, I mis-spoke in saying it was console.log spam, that’s not specifically what’s happening. But let’s be real, if you’re getting a 429 response that is at most a cached hash table lookup per request, for a site has probably top 50 in the world in terms of “scaling to handle heavy request loads” for a decade.

There’s just no chance this specific thing is actually what’s causing issues for Twitter, it’s obviously a consequence of the heavy scraping and steps to stop the heavy scraping. It frustrates me to no end that smart technical software people, who can have intelligent discussions on not just code but a wide range of topics from nuclear powers to superconductors, suddenly lose their ability to have these discussions when a certain person is involved.

reneberlin• 3 years ago

The kind of input you have when feeding tweets into an LLM is toxicity. It looks like something you can blackmail LLM for. Look, you have all this toxic trash in your model.

Like: we have the sources and you could detox you model if you pay for it.

lmeyerov3• 3 years ago

Apparently their 5-year cloud contract failed to renew today

So, there may be a much simpler explanation for why their new rate limits on regular users to can-barely-scroll levels, and how that has all sorts of unintended consequences they weren't ready for

ikrenji• 3 years ago

google wouldn't just shut off services for a big customer like that...

lolinder6• 3 years ago

If someone completely stops paying the bills, it doesn't really matter how high of a roller they are, you can't keep serving them indefinitely.

dehrmann• 3 years ago

A medium-sized customer making a point of not paying the bills to negotiate hard? If they come back, I'd put them on a pre-paid plan.

wnevets• 3 years ago

Don't you have to pay your bill to be considered a customer?

lmeyerov3• 3 years ago

Yep, less about fully shutting off and more about negotiation. This can be Musk doing a direct cut on some of their biggest web traffic to show that they can optimize it down. Or same, but internally, telling internal engineers they only have so much budget.

jaimex22• 3 years ago

So you say, but its been working fine all day so...

Yeah, gonna keep using it.

jaimex22• 3 years ago

its still online...

sakopov• 3 years ago

What I find more ironic than this bug is is that all of these Mastodon evangelists still have Twitter accounts after their Elon meltdown. So much so that they even use it after Twitter stopped public access.

ImaCake• 3 years ago

Have you considered that a lot of people use both platforms and don’t really care about the politics of it?

Those who do care are very loud (on both platforms) but they might be a minority in practice.

raincole• 3 years ago

You're replying to a thread on HN. The thread is a link to a Mastodon post. The post has the following tags:

#TwitterDown #MastodonMigration #DDOS #TwitterFail #SelfDDOS

This is why context matters.

rsynnott4• 3 years ago

I’ve been off Twitter since naughty ol’ mr car started his war on the journalists (I did return briefly to make a silly joke when he briefly banned mention of the dread term ‘mastodon’). I’ve kept the account, because, well, I’ve had it for over 15 years (back when they had a visible auto-incrementing numeric ID it was six figures), and maybe at some point he’ll lose interest and Twitter will return to semi-competent (it was never great) management. I suspect a lot of people are in the same boat.

But also, yes, I use it to observe the latest meltdown; why shouldn’t I?

jjulius2• 3 years ago

Because maybe it'd be better if we stopped giving him attention. He thrives on it in a negative way.

summerlight• 3 years ago

If you missed the initial news of the impending showdown, Platformer reported on June 10th that Twitter had been refusing to pay Google for its cloud services ahead of their contract’s June 30th renewal date.
Twitter’s Google Cloud contract dates back to 2018.

https://www.engadget.com/twitter-has-supposedly-started-payi...

Oh... this explains all this fiasco :facepalm:

partiallypro2• 3 years ago

As the article says, and as Bloomberg, etc have also reported. Twitter finally agreed to play nice with Google and the issue was resolved. They are probably trying to get off of GCP, but this isn't because they suddenly lost access to GCP from payment refusal.

BlackjackCF• 3 years ago

Well, I'm sure that migration off of GCP will go smoothly with no problems. Won't cause any more stability issues with Twitter. /s

18pfsmt5• 3 years ago

https://www.reuters.com/technology/twitter-resumes-paying-go...

That's from a week ago.

mmcwilliams• 3 years ago

The sourcing on that story is a single unnamed person whereas both companies refused to comment. Seems like there could be more to the story than "Twitter started paying again and everything's fine."

nightshadetrie3• 3 years ago

Twitter does not use google cloud as the main backend service. All of its self hosted.

Gcloud only supports batch jobs and data analysis.

acedTrex• 3 years ago

Twitters core services are on prem dcs

fredgrott• 3 years ago

Use A marketing idiot as engineer chief, win stupid prizes

Coneylake• 3 years ago

Have they tried batching API calls? /s

mcint3• 3 years ago

Matters little, but the rate limits are opening back up. 6k/600/300 -> 8k/800/400 (~noon) -> 10k/1k/500 (~3pm)

https://twitter.com/elonmusk/status/1675214274627530754 and self-reply

kennywinker• 3 years ago

I can't read that. "Something went wrong". And I believe under the new logged-in-only rules, I wouldn't even be able to read it if the site was up.

Twitter links are now poisoned, and basically useless.

justcool393• 3 years ago

here are the tweets

To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits: - Verified accounts are limited to reading 6000 posts/day - Unverified accounts to 600 posts/day - New unverified accounts to 300/day

https://twitter.com/elonmusk/status/1675187969420828672

Rate limits increasing soon to 8000 for verified, 800 for unverified & 400 for new unverified

https://twitter.com/elonmusk/status/1675214274627530754

Now to 10k, 1k & 0.5k

https://twitter.com/elonmusk/status/1675260424109928449

mcint3• 3 years ago

I did quote a summary of the announcement contents and timestamps.

We'll see how self-sabatoging the strategy is. It's cost cutting and customer-squeezing season, and as a user, I'm as unenthused as you.

williamsmj7• 3 years ago

I regained access and got rate-limited again (presumably under these new limits) within about 5-6 minutes.

rideontime2• 3 years ago

I managed to hit the (newly raised?) rate limit simply by tapping that link. What a circus.

delecti• 3 years ago

Yeah, between the rate limit, and the algorithmic weighting toward "blue" users, there's very little incentive to interacting with high-follower accounts at the moment.

TheBlight3• 3 years ago

I don't see any unusual behavior when looking at the debugger in Chrome. People have a political issue with Elon. Which is fine. People are entitled to it.

eyelidlessness5• 3 years ago

The post suggests a race condition as the underlying cause of the behavior shown in screen captures. It’s perfectly possible the race condition does exist, and being a race condition you just didn’t happen to trigger it. It’s also perfectly possible to discover such a defect with or without any personal bias towards Musk or Twitter.

Reporting it publicly this way can also be a favor to both, as it gives them a fairly malleable narrative for reversing course, or directing responsibility, or both. Even if it’s simultaneously embarrassing to have it out there. And it’s not like either Musk or Twitter is a stranger to embarrassment, or particularly shy about courting it.

TheBlight3• 3 years ago

Clearly there is something wrong with the site. So I don't doubt there is a bug. But the OP gives no helpful information that could be used to reproduce the issue and fails to describe it in any helpful detail really. They seem content to just effectively say "Twitter is dumb." Which, again, is fine but not all that technical and interesting.

berniedurfee3• 3 years ago

Twitter told me I spend too much time on Twitter. I agree. Thanks!

tmpz22• 3 years ago

Can we please recognize that this is a holiday weekend and Elon rammed a big release forcing his engineers to come in and iterate multiple patches over the last 12 hours.

truemotive• 3 years ago

extremely recognized.

EGreg• 3 years ago

recognized even more

paxys9• 3 years ago

Twitter Engineers have had over a year to get out and find another job. At this point the terms of their employment and the expectations from their boss are all crystal clear. Hard to sympathize with those who continue to stick around for whatever reason.

buitreVirtual• 3 years ago

In this job market, it's hard. And many engineers are on H1B visas so it's even harder to jump ship.

barbazoo• 3 years ago

Without knowing much about the people I opt to approach this with a bit more empathy. Things are shitty for lots of people. Now is not the time to judge people.

nashashmi2• 3 years ago

The activist mindset is not part of professionalism. If people chose to stick with their jobs, it is their professionalism. And if people chose not to leave, it is a sacrifice they made.

Any professional worth his salt would do the same.

joe__f2• 3 years ago

You are telling people who are being abused that it's their fault that they didn't leave. This is victim blaming.

badrequest2• 3 years ago

Sounds like you live in a bubble where nobody needs immigration sponsorship.

Johnny555• 3 years ago

Someone pointed out that it looks like most of the workers that stayed at Twitter appear to be foreign born and are probably reliant on Twitter's sponsorship of their H1-B visa.

https://www.vice.com/en/article/n7z5px/twitter-employees-on-...

littlestymaar• 3 years ago

Looking for a new job requires time and energy (especially at the moment) and these are scarce resources when you've been un constant crunch for the past 9 months…

nashashmi2• 3 years ago

The arrogance of the guy is so huge, hubris does not touch him. Thoughts are with the programmers.

What I’m more surprised about is how gum and shoestring the twitter engineering is now a days. They put in no emphasis on doing deep divides into the code base and instead opt to do the simplest shortest fix. And it causes problems.

starbugs2• 3 years ago

Applying the simplest shortest fix can sometimes be a good method to synchronize the state of the code with management.

Instead of engineering trying to buffer and fix weird management decisions, this just exposes them.

bart_spoon3• 3 years ago

Often the simplest shortest fix is the result of management’s directives. “We need this thing done yesterday” leads to the shortest fix, which compounds over time. Then the same managers wonder why it’s so hard to make changes or implement things in the future, without understanding their insistence on taking the path of least resistance is the direct cause.

TheHappyOddish5• 3 years ago

What holiday is it?

voganmother42• 3 years ago

Is this really easier than consulting a calendar?

Its the weekend before the 4th of July, the independence day for USA - many employers give employees Monday and Tuesday off, creating a potential four day break during what is typically very good weather - that being said Twitter is presumably (still) an international company with people in all different holiday areas.

easton• 3 years ago

4th of July (America’s birthday). Which is actually Tuesday, but many people are taking Monday off too.

TX81Z• 3 years ago

But at least they got to deploy directly to prod.

croes3• 3 years ago

And?

These are the hardcore programmers of Twitter 2.0, aren't they?

romanovcode• 3 years ago

AFAIK most are H1B holders who are in "hostage" situation.

berniedurfee3• 3 years ago

Union workers would typically collect double or triple time in these instances. Time and a half at least.

highmastdon• 3 years ago

What a toxic thread... Both here and the post on sfba.social

tasubotadas• 3 years ago

Lots of FAANG developers here hate Musk because he has shown that 80% of them is a waste of salary.

BryantD• 3 years ago

Twitter has been severely hampered for 12+ hours, and that’s only the latest issue. I don’t think Musk has demonstrated anything conclusively quite yet.

x86x873• 3 years ago

Lol. Right on buddy.

You know what all corporations should do to maximize short term profit? Fire everyone. It's amazing how much money we spend on people /s

rs999gti2• 3 years ago

Sounds more like bad QA that forgot to test an use case

minimaxir5• 3 years ago

In that case, it's not QA's fault, it's the organization's fault since it hit production already.

throwawayswiss3• 3 years ago

At my last start-up, there were 3-4 times the number of engineers and yet they failed to do basic testing and engineers ended up do that job too. Funnily some of them were paid more than engineers

phillipcarter• 3 years ago

All Elon had to do was leave it alone and, oh, right, the 44 billion.

Hahahahaha! Man, Parag Agrawal and his team knew exactly what they were doing. A fool and his money are soon parted, indeed.

bottlepalm• 3 years ago

Playing 5 dimensional chess with that poison pill huh?

Sai_• 3 years ago

I have an alternative theory for this. Once Elon was forced by the DE court to buy Twitter, he went to autocrats and dictators with money and told them he'd burn Twitter to the ground on their behalf for 44B.

No Twitter, no Arab Spring. No live updates of disasters, no need to shutdown the internet to prevent dissenting voices from spreading.

adamcblodgett• 3 years ago

Are we doing theories? Mine is that Peter Thiel sold Musk on buying Twitter by convincing him of his ability to make it profitable with his genius, knowing full well he'd sink the thing in 18 months, and another pesky tool would be off the table for awhile.

extasia• 3 years ago

That theory is pure speculation on the basis of motive with no actual grounding in reality, no offense. There's not a shred of evidence that GlobalAutocracy™ banded together to cough up $50B to kill of Twitter...

regularjack• 3 years ago

You thought Musk was playing 4D chess? Think again. He was in fact playing 29D chess, with dictators, and autocrats.

seadan83• 3 years ago

Focusing on the technical issue at hand - can anyone tell the exact issue at play? Is this an issue of client retry logic that on a 429 without backoff? Hence, the self-ddos?

inpdx3• 3 years ago

People commenting that people who are still at Twitter because of their own volition, just stop. Certainly some are still there because they want to be, but there are absolutely people trapped there economically one way or another, working for a tyrant.

TheHappyOddish5• 3 years ago

What nonsense. Silicon Valley employees are some of the best paid in the world and have chosen to enter into a contract in a capitalist society. Even if any are "economically trapped" (they're not) and it's not of their own volition (it is), it's been a year since Musk bought the company, plenty of time to find another job.

I don't think you understand what a "tyrant" is. Companies aren't democracies.

aetimmes• 3 years ago

Anecdotally, you are wildly incorrect.

CodeWriter23• 3 years ago

So move fast and break things apparently isn’t HN’s chosen methodology any longer.

minimaxir5• 3 years ago

HN was mocking Facebook's slogan even a decade ago after their numerous outages back then.

They had to change it to "Move Fast and Build Stable Infra" for a reason.

chaxor• 3 years ago

https://youtu.be/FeAMiBKi_EM

The webdev or JavaScript mentality of if it breaks, don't worry, we can fix it live strikes again.

mrcwinn2• 3 years ago

I assume the new CEO will hold Twitter’s tech leader responsible.

0xDEF• 3 years ago

I made an account around the time ChatGPT API (GPT-3.5) access was released to follow all the LLM news. The account is now getting aggressively rate limited that I cannot even see new tweets for 2-3 minutes.

pineapple_guy• 3 years ago

On a side note, I must say that Mastadon having a side bar on mobile is a very poor design choice.

pschuegr5• 3 years ago

I want to agree with you since I generally hate this kind of thing but honestly the Mastodon web ui on mobile doesn't bother me for some reason.

throwawayswiss3• 3 years ago

It's a waste of space but in overall it looks amazing

m3kw9• 3 years ago

Worked fine with me. It’s a bit annoying seeing a mastodon user shtting on twitter, there was no self own, the post sounded good because it resonates with Elon haters.

justinator• 3 years ago

Did the Fail Whale appear, or was it too busy capsizing boats?

penjelly• 3 years ago

the irony of all the premature celebrations over the past 8 months. If you stick your head in the sand and pretend there are no problems, i guess there really arent any, until maybe there is?

runsWphotons• 3 years ago

Is there anywhere in this thread where there is actual info and not just people complaining about and taking shots at Elon Musk because they dont like him or his politics?

dcunit3d12• 3 years ago

yeh, i tried, but it got flagged quick.

https://news.ycombinator.com/item?id=36561808

it's a 429 error, so the developer who posted this is an idiot. they're not even wrong. the 429 doesn't even touch twitter's infrastructure

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

olalonde5• 3 years ago

It's an interesting theory but the DDoS predates the decision to disable anonymous access. In fact, that decision was taken in order to mitigate an ongoing DDoS[0][1]. So while it's possible that the questionnable web front end retry logic is not helping things, it's not the root cause.

[0] https://twitter.com/elonmusk/status/1674865731136020505

Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!

[1] https://twitter.com/elonmusk/status/1674942336583757825

This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.

koboll• 3 years ago

Honestly it probably has more to do with the decision to basically shut down API access without exorbitant fees than it has to do with AI startups or whatever.

Surely there are some colossal archives of all tweets before whatever date floating around that those startups are using instead.

tsimionescu4• 3 years ago

That is a post-backlash claim by Elon Musk, so it should be taken with a huge grain of salt.

olalonde5• 3 years ago

It seems more plausible than the alternative. Let's say that Twitter availability issues were really caused by disabling anonymous access and that Elon wants to back out of his decision due to public backlash, wouldn't he just reverse the decision? That would fix both of his issues at once. He could even claim to have solved the scraping issues through a clever trick that doesn't require disabling anonymous access. The fact he hasn't done so indicates that there is a genuine issue that he can't just fix by restoring anonymous access.

brigandish4• 3 years ago

There must have been an insane nest of logic that the devs had to tiptoe through to make the service workable as a private only service.

It seems that this statement can lead to different conclusions based on one's biases for or against Musk / freedom of speech:

1) The code base is a mess because the smaller, current team don't know what they're doing and/or management (i.e. Musk) is pushing them too hard and making poor decisions

2) The code base is a mess because the far greater number of engineers there before didn't know what they were doing and/or management weren't pushing them hard enough and were making poor decisions.

Both could be true but I'm going with something the more complex view that it's a little bit of everything, and that Twitter does seem to be moving in a positive direction overall.

makeitdouble• 3 years ago

This makes me realize "the code base is a mess" has been so overused it could be a standard greeting in our profession.

Everyone has their horror story and extreme examples, but for any mildly profitable company that went through a decade or more of operation, I think it's kinda of a meaningless statement.

brigandish4• 3 years ago

This makes me realize "the code base is a mess" has been so overused it could be a standard greeting in our profession.

I'm printing up t-shirts already, pre-order now and get 10% off the insane nest of logic tee while stocks still last!

smcl3• 3 years ago

Why do you say Twitter is moving in a positive direction? My experience is that I dislike their app and they killed the one I liked (Tweetbot), it's frequently down (I still get the "Something went wrong" error), and any kind of conversation is dead because blue-check replies (usually quite low-quality) are prioritised. There are other reasons to dislike today's Twitter that I needn't go into here, but these are enough to turn me away any time I decide to come back and check it out again.

brigandish4• 3 years ago

A few reasons, the first being that I can actually log in again after 2 or 3 years, as the login was broken for me for that long so I can't say I was a fan of what was going on before. I had several frustrations before but that, as you can imagine, pissed me off quite a bit.

The second being free speech, I'm a big supporter of that and there's certainly more of that now. It's not perfect but I'm not going to throw the baby out with the bathwater by arguing against perfection or pointing out some of the hypocrisies in play. It's a move in a positive direction.

They're comparisons against recent Twitter, which before Musk felt like an app with so many broken parts that were never going to get fixed (the login problem being the cherry on that cake for me). Now there's a feeling that things might get fixed, that new features definitely are coming, and that complaints might be listened to.

I also don't like the changes you've highlighted, though I do see them as inevitable at this point in internet history. We see Instagram and others moving towards paid verification, Reddit closing down its API and killing apps (and should we forget that Twitter has done this at least once before in one form or another). So, I don't think it's all rosy in the garden, but overall it's positive, and the negatives (mostly) feel inevitable.

Add to that, alternatives are now getting a bit of a look in from people that wouldn't have have bothered before. I don't think that Mastodon, to choose one example, is a direct replacement, but it's good that people are giving it a try.

andreygrehov4• 3 years ago

Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in

Well, for one, this sounds just like yet another Elon-hater's comment. In _most_ cases, Elon's businesses are fairly successful, so a sarcastic "Elon's latest genius innovation" doesn't belong here.

For second, how do you know the idea came directly from Elon? It could easily be the result of a brainstorming session led by a product team.

Elon is not writing Twitter's code. He is not a Software Engineer at Twitter. Even if the idea came from him, why is he being attacked for a bad deployment?

And lastly, why do people think that the change is bad? There are plenty of web sites that do not allow you to view their content without creating an account first. Is Twitter not allowed to experiment? Why not? I would appreciate hearing well-thought-out arguments that explain the potential negative impact on the business.

vore• 3 years ago

He did make the call to do that, did he not? He publicly stated it on Twitter.

andreygrehov4• 3 years ago

Why does it matter that he made the call to do that? I get plenty of "do this, do that" requests from my manager. And if I break things, is it my manager's fault? I'm trying to understand your logic. Please, explain.

Edit: here is my take on it + the most likely scenario of what's going to happen next. This was a bad deployment and such things are not unusual in software industry. In this specific case, there are zero reasons to blame the CTO. From what I'm observing right now, Twitter has already fixed the issue - the website loads just fine. Next, they will do a retro, learn from their mistakes and try to not repeat the same mistakes again in the future. There will be more bad deployments and that's normal. However, with time, they'll make things better, the SLA will go up and the overall stability of their services will stabilize, – one of the biggest social networks on the planet will have the least number of engineers running that same social network.

williamsmj7• 3 years ago

why is he being attacked for a bad deployment?

Because it's quite literally one of the things he's responsible for at Twitter, by his own choice and description of his role (https://twitter.com/elonmusk/status/1656748197308674048).

andreygrehov4• 3 years ago

Should a CTO perform UAT?

rsynnott4• 3 years ago

(A) He claims to be the CTO.

(B) Are you familiar with the expression “the buck stops here”?

andreygrehov4• 3 years ago

(A) How is a CTO responsible for deployments? CTO oversees the development evolving the company’s strategic technical direction so that's its beneficial for a business. Twitter is not your typical YC startup. Large companies' CTOs should not care about things like "this could DDOS us". These are the responsibilities of SDEs. (B) No, but I just googled it and I can't see how the term applies here. In my opinion, it doesn't.

alfor• 3 years ago

This is expected with a faster rate of pushing features.

Funny things is Twitter views are hiher than ever and it will be fixed quickly.

(No reason to make a big deal out of it) But I think tests and refactor of the codebase will be necessary to increase their speed of innovation.

cochne• 3 years ago

What substantial features have they been pushing? To me it looks like just a bunch of ways to limit and monetize their existing userbase.

ralfd• 3 years ago

10000 character longform tweets are now possible and 2 hour long videos.

indus• 3 years ago

Wasn’t Elon our North Star first principle thinker?

jrockway• 3 years ago

Who knew that firing all of your software engineers could make for bad software.

sh34r4• 3 years ago

I'm astounded it took this long to finally die. All those Twitter engineers who were harassed by Elmo's very online fan club, and were treated with such cruelty and disdain on their way out, clearly did an incredible job. Hats off to them.

The curse of IT Operations: if you do everything right, management grifters never think you did anything at all. Why do I need you? Everything works fine!

Most Internet platforms are put together by popsicle sticks and bubble gum. If Reddit didn't have 2000 full time employees babysitting that steaming POS, it would be offline by the end of the week. But Twitter was like a chicken running around with its head chopped off, and it didn't stop running for 9 months. It's a testament to how incredible those engineers were.

This collapse at Twitter should have happened by January. It's a real shame it didn't. How many thousands of tech workers were laid off because Musk's fellow parasitic oligarchs saw Twitter running "fine" after the lobotomy, and followed him off the cliff like a bunch of lemmings? How many billions of dollars has the economy lost because of this one despicable man?

Elon Musk is a fraud. He is not an engineer. He's a lazy bum who mooched his whole life off his groomer daddy's apartheid emerald money. He would be nothing without that disgusting, incestuous old man. Despite his enormous financial privilege, Elmo was too lazy to enter the US legally. He was in the US for many years as an illegal immigrant. He only bothered to get off his lazy ass and finish his Economics degree because the risk of his deportation was becoming a real problem for X.com. That crooked philistine doesn't know the first thing about hard work, engineering, or finance. He's a damn good conman, he's good at shitposting, and that's about it. He has no other skills. He is not an engineer. He has no STEM degree. His only real accomplishment in life is proving that you can, in fact, spend billions of dollars in one lifetime, if you simply buy a bird app, run it into the ground, and salt the earth behind it.

quenix• 3 years ago

He has no STEM degree.

Musk has a bachelor's degree in physics from the University of Pennsylvania: https://www.snopes.com/fact-check/musk-physics-degree/.

Further, while I share some criticism of the man, many very technical people from companies such as SpaceX and Tesla have come forward with public comments, praising him for "truly grasping the engineering" and "being involved in every technical design decision". Make of these what you will:

Kevin Watson, Falcon 9 avionics:

Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.

He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.

He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years.

Tom Mueller, SpaceX founding employee:

We’ll have, you know, a group of people sitting in a room, making a key decision. And everybody in that room will say, you know, basically, “We need to turn left,” and Elon will say “No, we’re gonna turn right.” You know, to put it in a metaphor. And that’s how he thinks. He’s like, “You guys are taking the easy way out; we need to take the hard way.”*

And, uh, I’ve seen that hurt us before, I’ve seen that fail, but I’ve also seen— where nobody thought it would work— it was the right decision. It was the harder way to do it, but in the end, it was the right thing.*

Garrett Reisman, engineer and former NASA astronaut:

What's really remarkable to me is the breadth of his knowledge. I mean I've met a lot of super super smart people but they're usually super super smart on one thing and he's able to have conversations with our top engineers about the software, and the most arcane aspects of that and then he'll turn to our manufacturing engineers and have discussions about some really esoteric welding process for some crazy alloy and he'll just go back and forth and his ability to do that across the different technologies that go into rockets cars and everything else he does.

Josh Boehm, former Head of Software QA at SpaceX:

Elon is both the Chief Executive Officer and Chief Technology Officer of SpaceX, so of course he does more than just ‘some very technical work’. He is integrally involved in the actual design and engineering of the rocket, and at least touches every other aspect of the business (but I would say the former takes up much more of his mental real estate). Elon is an engineer at heart, and that’s where and how he works best.

Along with many others.

jquery2• 3 years ago

Instead of judging Elon by quotes from people who might have something to gain by praising a narcissist, how about we judge Elon by the insanely dumb things he says and does on a near daily basis?

And you're right he has a degree, although he repeatedly lied about when he received it, and even lied about where he received it, so please forgive people for their confusion.

pc_edwin2• 3 years ago

Its funny how the entirety of silicon valley just forgot that Twitter was infamous for their unholy mess of a codebase which at some points looked like it had a secret mission to increase world entropy and terraform the planet.

I still remember watching podcasts, reading blogposts, hearing prominent VC's like Elad talking about etc.. Now everbody silent because Elon is the baba yaga...

pc_edwin2• 3 years ago

To go further, its crazy how much cognitive dissonance can penetrate even the smartest among us.. If you had some understanding of how modern systems at twitters scale works, you'd know such a bug is highly unlikely.

You have multiple clients (android, ios, web) and a whole cluster of microservices calling each other. The microservices calling each other will easily use up more requests.

IMO the likely culprit is rate limits being triggered by a huge increase in scraping caused by the new api pricing.

guax• 3 years ago

The difference is that Twitter was not arrogant about it. Saying they where hardcore engineers with insane working hours and other bollocks. Musk dislike comes from him being extremely dickish about other peoples work while being an absolute clusterfuck of a manager.

Twitter was not bad because of its tech. Musk made it worse.

didip• 3 years ago

Any reading materials on how messy the codebase was?

tzekid• 3 years ago

Not quite reading material, but look up George Hotz's videos about it. I think there's one Twitter Space he hosted / recorded where Musk was also in there.

Hotz is very vocal and blunt about the messy codebase situation at Twitter.

eql5• 3 years ago

Here's a guy who understands what's REALLY going on:

https://twitter.com/i/status/1675254646942896128

yafbum2• 3 years ago

"Something went wrong. Try reloading"

Man, we do need a public square.

hiepph• 3 years ago

Also 500 for me. This is almost a joke.

At least if it's blocking me from viewing the tweet's comments, returns me some decent messages. But no, just a plain "Something went wrong." like a hackathon page.

kaonashi• 3 years ago

American censorship has long relied on the manipulation model rather than the overt removal model, because the former is far more effective.

Eric_WVGG• 3 years ago

I was skeptical, but this was a really good watch. Pity that it’s a video.

For the to;dw crowd: he makes the case that Twitter is by far the most useful social network to government agencies, because the graph is mostly public (unlike Facebook) and all users are both consumers and creators (unlike YouTube). Thus, it is being used by “researchers” and state departments to build what he calls “the Death Star of AI censorship engines,” and these moves to rate limit will cripple those efforts.

jquery2• 3 years ago

When people say "Elon is a cult" this video is exactly what they mean. Elon does something incredibly dumb and you immediately have people just making up, whole cloth, fantastical reasons why it's a 5D chess master plan.

It's comical at this point.

ajuc• 3 years ago

How is Elon allowed to transport people to space?

activiation2• 3 years ago

Luckily he is not the one doing the work

PaywallBuster• 3 years ago

drama queen

429 requests handled at the load balancer are extremely light weight

rochak• 3 years ago

Elon is truly one of the CEOs of all time

activiation2• 3 years ago

There's a word missing... Starts with W

x86x873• 3 years ago

Wonderful? /s

continuational• 3 years ago

The comments in there gloating over somebody else's mistakes is exactly why I don't use Twitter - except it's apparently the same on Mastodon.

insaider• 3 years ago

My favourite part of this whole fiasco is that 'Space Karen' is now trending on twitter

timeon• 3 years ago

Less funny is that those people are still on twitter.

choas• 3 years ago

they forgot to fix the Settings page on anonymous tweets which sending constantly data and receives a HTTP 429 Too Many Requests

therouwboat• 3 years ago

I managed to get in without login few minutes ago and its still sending api request every 130ms and getting 429 Too Many Requests error, how is this possible? :)

joeythedolphin5• 3 years ago

It seems the majority of Twitter articles are about the poor quality of engineering solutions or Elon's inability to "pay his Google bill". It strikes me as propaganda when what we are truly witnessing is the rush to wall-garden the internet as AI LLM takes over.

Patrol8394• 3 years ago

We are staring to see the effects of the layoffs…it can only get worse from here…

CatDadCode• 3 years ago

I finally did it. I finally got rid of my Twitter account and migrated to Mastodon. I wish I had done it sooner because Mastodon is great. Particularly the Hachyderm.io server for the tech community <3

Twitter Is DDOSing Itself

Related Stories