How Facebook Keeps Messenger from Crashing on New Year's Eve

https://spectrum.ieee.org/tech-talk/computing/software/how-facebooks-software-engineers-prepare-messenger-for-new-years-eve

by victorvation • 8 years ago

275 104 8 years ago

spectrum.ieee.org

lostmsu3• 8 years ago

Mine crashes stably on launch right now on two different machines. Therefore not so good :-D

gerdesj• 8 years ago

New Years Day hasn't happened yet. Your problems indicate that you should turn it off and on again.

snazz• 8 years ago

Turn two different machines off and on again? Doesn't sound like a coincidence to me, unless some other shared hardware (network, etc) is causing the problem.

lostmsu3• 8 years ago

Well, the problem exists for quite a while now. But what's the point of the backend working, if the app does not launch. Says something about quality, when a 300MB IM app won't start.

sluggg• 8 years ago

on mobile? are you running the most up to date version?

are you on a web browser? are any of your extensions mucking things up?

lostmsu3• 8 years ago

Both machines are Windows, the app installed from the store. When it does not crash on launch, it takes forever to start.

mliker• 8 years ago

How do you know if your problem is a client app problem vs a backend issue? :P

sjroot• 8 years ago

As someone who typically works on front-end projects, this was a very interesting read. I particularly loved the discussion of “graceful degradation.” That’s the kind of collaboration across the stack that makes a service like Messenger very pleasant to use.

robertAngst• 8 years ago

Interesting how they decide to remove features like 'seen' and 'online' in an effort to prioritize the actual message.

Offtopic- I hate those features to begin with, but I know I'm the product not the customer, and those features are to keep people on the app.

oscar_wong67• 8 years ago

I also despise those features, but I agree wholeheartedly that these decisions push the money forward

bcherny2• 8 years ago

Out of curiosity, why do you not like those features?

To me, "seen" is a feedback mechanism. When I send a message to you, I don't always need a response, but I do usually want to know that you saw the message -- sort of an ACK.

AznHisoka2• 8 years ago

I haven't used FB Messenger, but I also hate features where they show if the other person is typing a response. I intentionally disable it in Slack, so the other person simply sees a response when I'm done with it.

Scoundreller3• 8 years ago

Heh, MSN messenger (back in the day... I don't know how they managed to kill it off) did this notification.

I had a 3rd party client (aMSN) that would popup a chat window as soon as someone would start typing an initial message to me. I would say "Hi" to them first and they would thing it was a big coincidence that they were trying to talk to me at the same time.

hn_throwaway_992• 8 years ago

I intentionally disable it in Slack, so the other person simply sees a response when I'm done with it.

FYI, I don't think that is possible in Slack. You can choose to hide typing indicators in your view, but AFAICT you can't prevent other people from seeing when YOU are typing: https://twitter.com/SlackHQ/status/1057288342671368194

bookofjoe2• 8 years ago

Diff'rent strokes: I really like that feature in iMessage. I find it kind of like being in a cartoon, with the little thought bubble about to produce a response.

rb808• 8 years ago

The worst is someone contacting you

Hi

(waiting) (waiting) (waiting)

< OK Hi

(typing) (pause) (typing) (pause) (typing) (pause) (typing) (pause)

FFS hurry up already

sleepinseattle• 8 years ago

I believe that started off as an accessibility feature back in the AIM days.

ngngngng• 8 years ago

I didn't realize message queues were used for this type of task. I'm assuming you would then also use autoscaling pods that respond to the number of messages in the queue. How do you scale pods fast enough for a messaging application or anything else trying for 100ms or less per operation?

jcmi• 8 years ago

Messaging queues are a core part of a lot of high scale distributed systems (source - Twitter) You want enough queue space to handle the expected volume and then some. Assuming you have that, you don't need to instantly scale instances out to match the amount of messages, you just need to catch up before the queue space runs out.

underwater• 8 years ago

Facebook runs its own hardware. How would "autoscaling" help them?

tybit• 8 years ago

Dynamically allocating hardware to different services makes sense for self managed DCs too. Breaking down cost per service of team is useful, as is seeing how much hardware is necessary for peak and trough I imagine.

I wouldn’t be surprised if on NYE services like chat take priority and get extra resources from background jobs that can wait a few hours.

gmmeyer• 8 years ago

Message queues (or similar things like Kafka, which isn't quite a proper "message queue") are used for basically everything at this scale. Messages are being passed indirectly. An event happens, it gets popped on a queue, and then the recipients do something with it.

latch3• 8 years ago

Maybe you'll consider this pedantic, or maybe even wrong, but I think you meant to say "asynchronous" as opposed to "indirectly". I think anyone googling to learn more will get better results.

latch3• 8 years ago

I think over-provisioning is way more common and sane approach that can address the bulk of spikes versus auto-scaling. Especially if you have these big known events (new years day, black friday, ...) where you can over-provision (or controlled auto-scale if you will) for a short window.

My guess is they're doing both.

Anecdote time. I worked at a company where one project was over-provisioned on dedicated hardware and another auto-scaled in the cloud. The over-provisioned project was much cheaper, had significantly better response times and was easier to manage. It was load tested to handle over an order of magnitude more traffic than the all-time-peak and even though fully over-provisioned, it was cheaper than the baseline usage (and slower, and harder to manage) cloud solution.

rhizome3• 8 years ago

The back of the napkin numbers I've seen say metal (onprem or colo) has a 100x price/performance advantage: 10x faster at 1/10th the cost.

dijit• 8 years ago

I should corroborate your anecdote with my own.

We did two games, one over-provisioned on metal, the other on auto-scaled cloud based infra.

The cost is significantly higher. We ended up with a hybrid to control for cost. But our needs are long-lived sessions which does not fit the elasticity of the cloud model well.

int0x80• 8 years ago

One way is to over-allocate in the first place. When your spare pool is draining below a watermark, you scale in. Hopefuly there is enough time for that scale event to complete before the pool drains completely.

porpoisely• 8 years ago

Essentially what everyone else does - distributed systems with load balancing, load balancing and more load balancing. And if that goes awry, triage - where they prioritize messages and simply timeout and drop the lower priority messages. Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ), aren't so lucky. Drop a few transactions or apply them out of order and it is the end of the world. Was an interesting read, though it would have been nice if there were more specifics but I guess Facebook wouldn't approve that.

linkmotif• 8 years ago

I don’t know, a missing “Happy New Years” isn’t missing dollars, but it’s definitely not cool to drop such a greeting —or any message—in my opinion. It should definitely be possible to at least store and then deliver these messages late. The baseline should be 100% deliverability and anything less than that should be subject to intense scrutiny. I mean, how big of a Kafka cluster do you need to make this happen?

arusahni• 8 years ago

Do you have any references/resources for plussing up on scaling transactions w.r.t. finance? It sounds like an intriguing set of challenges...

freddie_mercury• 8 years ago

Of course the Messenger team is lucky in that they can drop messages since your family and friends missing a "Happy New Years" message isn't the end of the world. Other systems ( such as finance ),

Messenger is (also) a finance system. You can send money via Messenger. You can purchase products directly in Messenger. It's had all that for more than two years now.

therein• 8 years ago

Then don't drop the transactions pertaining sending money. They are not necessarily treated the same way as messages in the backend.

tozeur• 8 years ago

That’s not the proper comparison. The difference is that payments sent through Messenger aren’t critical (think paying your friend for lunch), whereas for most financial transactions (eg trades of stocks, derivatives, swaps etc) it’s absolutely critical that they execute immediately.

dmix• 8 years ago

The OP mentioned a prioritization system for a reason...

radicality• 8 years ago

Actual messages with content are never dropped. Only ‘meta-messages’ like read receipts - it’s not critical if on a group chat with many participants the state of ‘who’s seen the last message” is not 100% correct on New Year’s Eve.

groestl4• 8 years ago

From one of my projects (a MMORPG) I've learned that the required accuracy in non-financial transactions is often underestimated, while, on the other hand, financial transactions are often less critical that initially assumed. After all, compensation in financial transactions is often straightforward to calculate and apply. But the damage done through dropped/failed non-financial transactions is often hard to assess, and it's also more involved to find appropriate compensation.

dominotw• 8 years ago

Cant wait for akward new yr wishes from ppl I haven't heard from since last new year.

mrdickbig2• 8 years ago

Deleted my FB 2 years ago... no regrets. You should try

secabeen• 8 years ago

Interesting that the messenger team is ~40 people, as compared to WhatsApp having 32 engineers at the time of their sale to FB.

traek• 8 years ago

The photo caption says that's just the infrastructure team. I'd imagine the product team is much larger given how many features are crammed into Messenger.

adtac• 8 years ago

I wish it was the other way around. We'd have fewer, but faster features.

pbalau• 8 years ago

Afaik the real team name is Messenger Foundation. FB has the concept of foundation teams, with the only goal of keeping a part of the service working no mater what.

/edit: been informed by some of the people in the picture that technically there are 2 teams there: Messenger Infra and Messenger Foundation

midom• 8 years ago

I can see some MySQL Infrastructure people in it :-)

mrdickbig2• 8 years ago

I read the title and instantly thought "Erlang"

ckwalsh• 8 years ago

Fun fact: FB Chat was originally implemented in Erlang

https://www.facebook.com/notes/facebook-engineering/chat-sta...

sn41• 8 years ago

I found the concept of affinity used for bunching messages to be a new concept. The other concepts are not surprising, they are the typical engineering solutions used.

One thing I am curious about: how does it compare to how the old postal systems used to handle Christmas and new year loads?

On a lighter note: perhaps you can predictively generate and cache messages at the receiver's end based on their contacts and their style of communication. When a sender actually sends a message, just send one bit across, and the local cache gets flushed and displayed :)

zimpenfish2• 8 years ago

I found the concept of affinity used for bunching messages to be a new concept.

I'm reasonably sure that UUCP + things like INN were "batching messages per destination" for efficiency a long time ago. Nowhere near the same scale, obviously, but the same kind of concept, no?

Jedd• 8 years ago

Actually, I'd expect the scale would be similar.

The raw numbers (users, bandwidth, latency) would be quite different.

ehsankia• 8 years ago

take the 16 most common messages, normalize them a bit, and encode them in 4 bit, and I would guess that would probably cover over 50% of the messages sent, maybe even close to 80%.

fhrow4484• 8 years ago

While it's a nice compression technique, I doubt you'd gain anything from this at the message level.

Even by reducing a message content to close to 0 bits, I doubt you would get that much gains. Most common messages would probably be things like "lol" or an emoji, which in utf-8 is 24 to 32 bits.

But There'll always be metadata associated with each message: message id, sender, receiver, timestamps, etc. which are unremovable. If those first 3 are typical GUIDs, this is already 3*128 bits. This will put a minimum on the entire message size , and more importantly the processing time to simply route the messages is what is costly in terms of CPU time. Then there's the different receipts mentioned in the article, adding a lot more traffic.

kaustyap• 8 years ago

It may be a good idea to provide range of fixed templates for standard messages and send just selective message Id along with recipient name over the network.

hellofunk• 8 years ago

One thing I find very manipulative about Facebook is how, when someone sends you a message, the email notification has a link to open messenger and it says that messenger is the only way you can read that message, even if you don’t have messenger installed. They are trying everything they can do to have everyone install that app. Yet, of course you can just read and respond directly on their website without any app, but they don’t link to that or mention it.

groestl4• 8 years ago

mbasic.facebook.com for anyone reading this and looking for a way to work around Facebook's dark UI pattern on mobile, where a click on the "messages" button wants you to install the app.

mario0b1• 8 years ago

Or you could just use some other service than facebook. Like signal or xmpp.

gpvos• 8 years ago

Or use "Request desktop site" in your browser, it gives you the same thing.

Jaruzel3• 8 years ago

Back in the day when SMS text message was the way to send messages to each other over the mobile phone network (in the UK), people would jump the gun by sending 'Happy NY!' messages 5 minutes before midnight, because the moment 12am hit, any messages sent then could be queued for hours as the mobile networks struggled to cope with the massive uptick in messages being sent at the same time.

Semaphor• 8 years ago

I used to use bulk sending tools (yeah, they existed as J2ME apps) to send 10-50 SMS (it's been a while, not sure how many it was) and some would only arrive on January 1st quite a while into the day. At some point, it changed and everything would arrive just a few minutes after sending. I think that was when we had iPhone and Android already and I'm not sure if it was because of messengers (were there any back then?) or because the German telcos finally upgraded their infrastructure enough.

Markoff• 8 years ago

this certainly feel like paid article to bring some good image to Facebook

John_KZ2• 8 years ago

This is a 2018 internet-connected app, not a 1985 GSM network.

1 billion 100 Byte messages sums up to an almost trivial 100GB. This might be a technical challenge for the neighborhood's web admin but not for any real company.

notacoward2• 8 years ago

What kind of tyro thinks GB/s instead of ops/s is the right measure here?

John_KZ2• 8 years ago

This is a text message. Unless you're doing it horribly wrong, ops/s should be very close to the number of messages.

How Facebook Keeps Messenger from Crashing on New Year's Eve

Related Stories