Server is smoking, trying to figure out why

https://twitter.com/HNStatus/status/1401980524428627970

by diogenesjunior • 5 years ago

179 58 5 years ago

twitter.com

Hublium• 5 years ago

It's a difficult habit to quit

LOL

seryoiupfurds• 5 years ago

Something went wrong. Try reloading.

Seems like Twitter isn't doing much better.

bilalq• 5 years ago

I interpreted this much more literally. When the updated tweet about a code revert fixing it came out, it took me a second to figure out what was meant.

AceJohnny2• 5 years ago

https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing...

nomoreplease• 5 years ago

Update provided

Welp, figured out why - because I am idiot. Routine code change exploded performance. Only took me 2 hours to think of reverting it. Sorry!

https://mobile.twitter.com/HNStatus/status/14020112741043609...

bryan_w• 5 years ago

If they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.

jrockway2• 5 years ago

My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things!

This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)

MontyCarloHall• 5 years ago

Besides applying security patches, what exactly has needed to be changed in the HN codebase over the last 14 years? Featurewise, it's been essentially static for its whole existence (which is a large part of its appeal to many, including myself).

xeromal• 5 years ago

If the number of users has grown over the last 14 years, I'm sure some of the code areas have needed to be tuned over 14 years as well.

jolmg3• 5 years ago

I think I've noticed changes a few times. For example,

- the relative timestamps would go up to days, so you'd see e.g. "3121 days ago", and now they're replaced with dates at some point.

- we're now able to escape * in comments. It used to be impossible (?) to use asterisks with consecutive characters, so you'd either have to put spaces around it or substitute with something like <asterisk>. This was unfortunate for inline code.

- code blocks no longer scroll horizontally and wrap instead. This is my favorite change because many code blocks were actually indented bullet paragraphs.

- the "past" link in the navbar was added

- the subthread collapse link [-] used to use a - (ASCII 0x2d) and now it uses – (Unicode U+2013). This is my least favorite change because I used to use that for keyboard navigation. It was pretty convenient to Ctrl+F [- <Enter>... to navigate between comments then <Esc> <Enter> to collapse a thread and Ctrl+F <Enter>... to keep navigating.

These are relatively recent changes.

AnotherGoodName• 5 years ago

When you hit scale every edge case becomes a reality.

im3w1l• 5 years ago

Ajax upvotes, upvote used to be just a link iirc.

CheezeIt• 5 years ago

Automated spam/voting ring detection and ranking systems.

jensenbox• 5 years ago

Nicotine is a hell of a drug.

raverbashing• 5 years ago

If you think about it, it's impressive how such a high traffic site works out using a niche language which what I assume is not such a big ecosystem.

(On the other hand there's moderate complexity but low page sizes and nothing like modern websites with hundreds of cookies, heavy js, etc)

smegcicle• 5 years ago

something about deep comment thread folding?

1vuio0pswjnm72• 5 years ago

Yesterday I saw a strange artifact in a comment. Instead of the usual "[-]", I saw "[2 more]". Did not seem right.

extra88• 5 years ago

That's what it says after you use the [-] to collapse a comment and it has a single reply. It also counts replies to replies (and so on).

cesarb• 5 years ago

I also saw that yesterday. What's worse, the only way to expand the comments hidden behind that seems to be to temporarily unblock Javascript, reload the page, and click on that.

ballenf• 5 years ago

I really thought it was going to be someone complaining about their restaurant server having a smoke break. Thought it must have been covid related to make the front page. But don't ask me how.

bombcar• 5 years ago

I had an older Dell give up the magic smoke recently and literally - https://i.imgur.com/WHYODQS.jpeg

Amusingly enough I smelled it weeks ago and it finally shut off a few days ago.

lamontcg• 5 years ago

Yeah I've been paged before due to a crashed server failing ping and found out the datacenter tech was already on the job because it was suffering from actual smoking. I was remote so I don't know what they did, presumably pull the power and hit it with a CO2 extinguisher?

userbinator• 5 years ago

Those are buck converters, the MOSFETs in them tend to die either due to overheating or being run at the edge of design limits for long enough. It's rare that they die excitingly, however.

oxfordmale• 5 years ago

Did you try nicotine patches?

loves_mangoes• 5 years ago

...nicotine? =)

el_benhameen• 5 years ago

If it’s any consolation, I thought my wireless provider had booted me before I realized HN could be having issues. It’s generally so fast and reliable that I use it as my “is my internet broken?” sanity check page.

lapp0• 5 years ago

Ahahah, similar situation here. I thought maybe our IT department had noticed my activity here and decided to throttle my connection (which would still be an odd solution instead of just outright blocking it), but still it took me a few minutes to suspect that maybe the problem was with HN itself.

whoisburbansky• 5 years ago

Also assumed this, was only convinced otherwise when I ran into the same issue on my personal mobile phone, on a data network separate from my work network.

masswerk• 5 years ago

Same here. :-)

jtbayly• 5 years ago

It was worse for me, as I had just replaced my router!

phendrenad2• 5 years ago

"Smoking" meaning running high CPU, presumably, not literally smoldering

vlmutolo• 5 years ago

Was the last outage really in 2015

Izkata2• 5 years ago

No, but three months ago it wasn't their fault: https://news.ycombinator.com/item?id=26469116

jpeter• 5 years ago

I thought Replit started ddosing the site

shkkmo• 5 years ago

My google foo failed and I wasn't able to find the correct @hnstatus account when I was having trouble. Perhaps a ”status" link in the footer would make this easier to find?

jscipione• 5 years ago

BeOS came up with an function for this: https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_Sys... but “Smoldering doesn't count.”

sfeng• 5 years ago

I don’t mean to be rude, but is HN a refutation of the Lispy vision it was meant to demonstrate? There has been a bug where comment pages can only show a limited number of entries for years, and the wonderful mods like dang continually report it is being worked on. It’s hard for me to believe it wouldn’t be fixed much faster if it occurred in a more practical language.

Waterluvian• 5 years ago

What's the bug? Isn't there a "more..." At the bottom of those long comment pages?

tzs• 5 years ago

I wouldn't necessarily call it a bug, but the way the pagination works does not work well for active threads which is probably why they want to replace it at some point with code than can handle long threads on a single page.

A problem with pagination as implemented on HN is that the algorithm to display page N seems to be functionally (no pun intended!) equivalent to this:

  Render all the comments as one long virtual page
  Split it into smaller pages
  Render the Nth page from that set of smaller pages

Note! I'm not saying that is how it works. Just that it produces the same result as that.

So you read page 1 of an active discussion. You get to the end of the page and hit "more". But while you were reading page 1, other people were commenting, and other people were voting. Now, when it goes to render page 2 for you other comments have moved to the top, pushing the stuff you already read on page 1 down to page 2.

Or maybe what was just after page 1 when you were reading page 1 has moved up due to voting, and when you go to page 2 you end up past it.

It's like if you had a book that went through several editions, with significant changes between editions (including rearranging the order of chapters), and someone put together a printing by taking the first 10 pages from the first edition, the second 10 pages from the second edition, and so on.

This could be fixed while retaining pagination by saving a timestamp when you go to page 1, and then when you go to subsequent pages use a snapshot of the comment database from that timestamp instead of the current comments. If you want to update to the current comments you'd have to hit refresh or go back to page 1.

Another approach while retaining pagination would be keep track of what top level comments you've seen. If when rendering page N it sees that due to comment order changes one of the comments that is going to end on the page is one you've already seen it could omit it. Or maybe omit it unless the tree starting at that comment has changed since you last saw it. (If going this route, I'd also suggesting adding something to the UI to highlight what has changed in the tree and/or dehighlight what you've already seen).

That second approach could still be annoying. You could still get stuck on a single comment that is getting a lot of relies. That could be fixed, but then you are getting into probably needing to add new controls or settings.

I'm sure they have thought of both of those approaches and more, but probably want to keep the interface clean and simple and retain minimal state, hence the plan to someday make it handle very long comment threads on a single page.

Zababa• 5 years ago

On the other hand, I see spam and flamewars very rarely, and I know they have protection in place against it. I'd take that over seeing all the comments in one page.

Wowfunhappy2• 5 years ago

There has been a bug where comment pages can only show a limited number of entries for years, and the wonderful mods like dang continually report it is being worked on.

It's not a bug per se, it's a concession to conserve server resources, which will be removed once some optimization work has been completed.

You might respond "well why is HN so resource-hungry?" Answer: it's not! HN is run off a single core of a single CPU.

https://news.ycombinator.com/item?id=23187455

The software for both HN and YC was just a single Arc program (and not a large one) for the first 9 years of their existence, during which they went from nothing to massively successful to industry-changing. Written by one person, programming part-time. That is a staggering achievement. The power of using the right language for your project goes far further than most people dream. Our imagination about this is crippled by path dependence, social proof, and the conditioning that comes from only ever doing things the same few ways, like those fish in experiments (which may be urban legends?) who stick to their corner of the aquarium even after a glass barrier has been removed. The solution space of software and programming is so much larger than most of us want to imagine that it is. Sad.

bastawhiz• 5 years ago

A post from a week or two ago about APL and cellular automata (I cannot find it) perhaps illustrated OP's sentiment better: it's easy for a language to do something easily and elegantly, but quickly become fantastically difficult to accomplish with the introduction of new—and sometimes mundane—constraints. The post showed how it was trivial to implement certain cellular automata in APL, but massively difficult to do when the automata became slightly more complex (while being trivial to update a Java implementation to accommodate the new complexity).

The practical difference between running on one server with one core and five servers with four cores is nothing—to YC it's a rounding error either way. If there's a meaningful problem (e.g., comments being computationally expensive to paginate well) that the language/runtime/system/etc cannot solve easily enough for it to be remedied in O(months), it does call into question whether the choice of language is the right one. Yes, it can be patched over by a human moderator. But it still highlights a significant weakness in the technology's core function.

Perhaps ironically, most YC companies (and by consequence, YC) would fail miserably if they chose purism and principle over pragmatism when it came to technology decisions.

erik_seaberg• 5 years ago

With Arc and HN and now Bel, I think pg is still exploring what a system could look like without major compromises to accommodate the limits of today’s hardware. For example, the first cut at pagination put something like a continuation ID into a URL, which only worked until that continuation was GC’d or the one and only server process got bounced.

If you wanted a highly scalable HN and nothing more, you’d let Postgres or some graph database do the heavy lifting, add a Racket or Clojure frontend, and call it a day. But that’s pretty boring plumbing work, and doesn’t get you closer to the hundred-year language he’s aiming for.

kristianp• 5 years ago

I wonder what database or file schema they use. Many would find that interesting, given the scale that hn has reached.

NullPrefix• 5 years ago

Oh how ironic. Twitter errored out with

Something went wrong. Try reloading.

Server is smoking, trying to figure out why

Related Stories