The WhatsApp Architecture Facebook Bought For $19 Billion

http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html

by bussetta • 12 years ago

276 113 12 years ago

highscalability.com

ForHackernews• 12 years ago

I wonder how Facebook will deal with integrating Erlang into their (afaik) PHP-based systems?

chollida1• 12 years ago

facebook chat was originally written in erlang.

http://www.erlang-factory.com/upload/presentations/31/Eugene...

I doubt they'll have any trouble at all integrating.

derefr2• 12 years ago

I believe that the reason they switched away from Erlang, though, was that nearly nobody in the company (besides the team that made the prototype) knew the language. In other words, they switched away because they had trouble integrating Erlang into their processes, and instead chose to rewrite it using tools everyone else was more familiar with.

This, then, is basically recreating the same problem they had before, just with 40 or so more engineers who are on the "Erlang" side of the vote.

judk• 12 years ago

Service oriented architecture, and avoiding or porting fat client libraries. Standard stuff.

zeckalpha• 12 years ago

The backend is a mix of things.

cmelbye2• 12 years ago

PHP is not the only language in use at Facebook right now...

https://github.com/facebook

They invented Thrift, I think they'll figure it out.

lttlrck• 12 years ago

Of all Facebook's accomplishments why is Thrift the one that you think makes this a non-issue?

smrtinsert• 12 years ago

they will do what comes naturally - take five years write a dual erlang php vm in D and then open source it randomly a year or so late.

nikentic• 12 years ago

"Desktop is dead and web is dying" is very pessimistic.

atwebb• 12 years ago

Is it that time again?

lnlyplnt• 12 years ago

if the web primarily became a data-layer for mostly native apps would that really be such a bad thing?

jere• 12 years ago

From a global perspective, is it?

By the end of this year, 6% of the global population will own a tablet, 20% will own PCs, and 22% will own smartphones.

http://www.businessinsider.com/smartphone-and-tablet-penetra...

grey-area• 12 years ago

Smartphones and tablets have browsers too, and they are by far the most used app on any platform.

Aaronontheweb• 12 years ago

Software is not a zero-sum game, so yes it's not just pessimistic. It's wrong.

tlack• 12 years ago

Does anyone know what service WhatsApp used for SMS delivery for phone number verification? I wish I could get text messages into all parts of the world like they do.

stickydink• 12 years ago

http://nexmo.com has pretty decent coverage

andypants• 12 years ago

Not really, there wasn't even a category for France and several first world countries did not have any available phone numbers to buy.

sjtgraham2• 12 years ago

They used multiple providers.

stanleydrew• 12 years ago

At Bolt (https://bolt.co) we use Twilio for this, and I'm pretty sure WhatsApp does too along with other providers. With Twilio we also fall back to delivering access tokens with a phone call when the SMS fails.

(Full disclosure: I am a former Twilio employee.)

username223• 12 years ago

Fuller disclosure: you're just another spammer.

willvarfar• 12 years ago

I work for a company called mblox; we put all our effort into reach.

yonasb2• 12 years ago

Authy (a two-factor auth service) uses a combination of Twilio and BulkSMS (they say that BulkSMS is particularly good for Latin America). Authy explains how they handle global SMS here: http://blog.leanstack.io/how-authy-built-a-fault-tolerant-tw...

twistedpair• 12 years ago

SNS by AWS does it too.

MichaelGG• 12 years ago

But aren't most of those messages point-to-point? Not that I'm trivializing 70M/sec! But for the most part, it's pretty straightforward right? No multi-client support (so no desktop client that requires server sync). Most messages(?) are to one recipient only, right? So send the message, send an ACK, deliver message, deliver read ACK? It seems like the perfect thing for a scale-out architecture. Even group chat can probably just be split into multiple messages, one per recipient.

Not that it's not a lot of work, but isn't most of it just hard work engineering? Add hardware, deal with bottlenecks. Versus having to come up with some novel new datastructures and inter-server communications?

SMTP sends a ton of messages, too. I don't see why an IM system couldn't take essentially the same concept, where one small cluster of services is responsible for the mailboxes for certain individuals.

31reasons• 12 years ago

Good! you got the technology, now go get those 450 million users :)

nly• 12 years ago

SMTP sends a ton of messages, too. I don't see why an IM system couldn't take essentially the same concept,

Hint's in the name: instant messaging. For the most part, nobody stops using SMTP if it takes 2 minutes to deliver an e-mail

coldtea3• 12 years ago

Hint's in the name: instant messaging. For the most part, nobody stops using SMTP if it takes 2 minutes to deliver an e-mail

Well, I don't think anybody would stop using IM too, if it took say 10 or 20 seconds to deliver an IM.

rdtsc3• 12 years ago

Phone signaling is also just making calls point to point. Very simple. Well except that making it not crash, making it scale, handling corner cases, legacy devices, regulatory stuff.

One can sort of take a high level look at a large class of system and just say "it sends messages from one part to another". A financial transaction sends one message from one account to another one. A lot of sites and businesses are "just" get data from database send to web page, then send updates back to the database. Pretty simple.

The devil is in the details and in the proof of work. If it is simple, ok, do it. Not being facetious, you might actually succeed. Just try it. Erlang is a good choice for it, start with that.

sjtgraham2• 12 years ago

"Ejabberd is a famous open source Jabber server written in Erlang… The next few years were spent re-writing and modifying quite a few parts of ejabberd, including switching from XMPP"

Anyone looked at what WhatsApp sends over the wire these days? I wonder what it is? A binary protocol?

Zash• 12 years ago

XMPP with a homegrown compression layer.

beachstartup• 12 years ago

does anyone know where they were hosted? did they colo or use a msp?

yonasb2• 12 years ago

They're hosted on SoftLayer http://leanstack.io/cloud-stacks/whatsapp. Source: http://www.ip-adress.com/whois/whatsapp.com

codegeek2• 12 years ago

I know this whole "$19B for whatsapp" topic is getting old already but I really think they solved some core problems that users had. Technically, those were probably not as challenging but functionally, this is what won me over :

- No registration required. Yes and Yes. do you hear Skype ? All you need is your phone number. And let's not get started on the privacy thing because the benefits outweigh the cost. See more below

- Easy to invite others. Just sends a simple SMS/text and wala, you are in.

- In countries like the US, SMS/Texting is very expensive (yes, i know...). Specifically, if you want to talk to someone international. Whatsapp solved this problem with ease. There were others (viber etc) but they just don't have the easy interface at least in my experience. Sure, the caveat is that everyone needs to have wifi/data plans but those are getting much more easier than convincing your phone company to stop charging bullshit rates for a simple text.

- Lets you create Groups. This sounds simple but imagine wanting to talk to your core family members in 1 text ? Whatsapp lets you do that with no pain.

- Share pictures/multimedia within groups. Again, very simple to use and comes included. I don't want to even explain what these cell phone companies charge for something they lable as MMS (not text).

Overall, when I came across Whatsapp, I was like "Finally, someone gets it and this should have been so duhhh but apparently, it wasn't". Very happy for the entire whatsapp team.

chime• 12 years ago

Lets you create Groups. This sounds simple but imagine wanting to talk to your core family members in 1 text ? Whatsapp lets you do that with no pain.

That is the #1 feature for my family. I'm in the US, my sister is in India, and our parents travel all over the world. Nearly every day, we post something to our group chat. Be it photos, personal news updates, or questions to one another, we actively use it almost daily. I have an iPhone and I think they have Androids.

dodders• 12 years ago

Siblings in England, USA and Australia and we all use the group chat.

orionsbelt• 12 years ago

Sorry, I'm normally not one of those people who correct others on spelling/grammar, but this is a pet peeve of mine: it's "voila" (french), not "wala".

maw• 12 years ago

Muphry's law strikes again!

fabrice_d• 12 years ago

It's actually "voilà" in real French... :P

guelo• 12 years ago

Oh, that's what he meant.

gahahaha• 12 years ago

It almost makes sense if he meant "wallah"

"""Just sends a simple SMS/text and I promise by God, you are in."""

http://en.wikipedia.org/wiki/Wallah_(Arabic)

He didn't mean that of course...

why-el• 12 years ago

Usually with loan words like this that are used enough, a spelling transformation is inevitable. I mean, I am sure voila itself is someone's pet peeve since its really "vois là". :)

mverwijs2• 12 years ago

Unless it's a wordplay, like Wuala(.com).

coldtea3• 12 years ago

Yep. And it's "per se" (Latin for: "in itself", "as it is") and not "per say" which doesn't mean anything at all.

tincholio2• 12 years ago

I was going to say the same... except it's "voilà"

brianmcc• 12 years ago

I'd missed that initially; I'm amused that someone's gone all this time thinking "wala" is just a noise people make to indicate "done!"

wslh• 12 years ago

And you forgot about the capability of sending other rich content like a map! ;-)

josefresco• 12 years ago

"Easy to invite others. Just sends a simple SMS/text"

and then

"In countries like the US, SMS/Texting is very expensive...Whatsapp solved this problem"

You need texting (a X dollar/month service) to use the service ... which solves the problem of... expensive texting?

I always found that to be somewhat odd, using a service you're trying to replace/compete with as the sole authenticating factor.

Edit: I realized after actually using the app, they do in fact offer an alternative authentication method (call). You have to try (and fail) at the texting method first.

Also, in response to derefr (I deleted my ignorant parent comment), I block texting as even if you don't pay for a plan, you still get random spam texts that you must page for (even if you didn't ask for them)

codegeek2• 12 years ago

What I mean is that most of us already have data plans these days and we are already paying for it. Why pay extra for texting when we can use something like whatsapp that uses the data plan. Not to mention the crazy International texting charges. Even if you have unlimited texting from say T-mobile, you will pay for international texts because the unlimited is only for domestic texts.

rand_r• 12 years ago

Big difference between a one-time invite cost and the cost of continually texting/messaging someone.

oijaf888• 12 years ago

You send a text to invite someone to download the app and then you transition all your discussion over to their network instead of the SMS network. Hence solving the problem of expensive texts.

coldtea3• 12 years ago

You need texting (a X dollar/month service) to use the service ... which solves the problem of... expensive texting?

Yes. Because texting in most countries has limits (e.g 200 sms per month, for more you pay an amount per text) while you can use WhatsApp for as many texts as you like.

And texting includes MMS (texting with images, sound etc), which is free in WhatsApp, but costs extra in most countries (can even go like $1 per text or more).

edwinnathaniel3• 12 years ago

I have a phone with no data plan so I can't use WhatsApp just to let you know... :D

In countries like Indonesia where SMS is almost free, they still use Whatsapp because of the Network effect.

Groups is the biggest one. It helps SELLING. YES, call it e-Commerce 3.0! This is how HOUSEWIVES sell stuffs. Through Whatsapp OR BBM.

rgovind• 12 years ago

Groups is the biggest one. It helps SELLING. YES, call it e-Commerce 3.0! This is how HOUSEWIVES sell stuffs. Through Whatsapp OR BBM.

Can you please elaborate on this.. How does it work? Any links?

Graham24• 12 years ago

how long until mobile phones are purely data devices and cannot make calls or send texts, but do all of that via apps?

isn't that why facebook wanted WhatsApp, as they see this is a future of mobile phones?

AlexanderDhoore• 12 years ago

The whole error handling philosophy of Erlang is extremely fascinating. The fact that it was born out of the telecommunication world makes it extremely fit for this kind of job. I've been reading "Programming Erlang" on and off for a few weeks now, and it's... eye opening. Erlang proves functional programming is for the real, real world. There isn't a language that's more down to earth, ready to "Get Shit Done" than Erlang.

Edit: When you learn Erlang you'll notice every design decision was made for performance or reliability (vs making the language "nice", think Ruby). I appreciate that.

Glide• 12 years ago

I think the golden moment for me thinking about Erlang was when I started thinking that the actor model was made to support fault tolerance and concurrency was a nice bonus.

rdtsc3• 12 years ago

Yeah Joe makes that point pretty often, and it was a nice revelation for me as well.

Actor concurrency paradigm has been copies by other languages and libraries. Not many managed to copy the fault tolerance and transparent distribution aspect. Granted there are other ways to solve that (OS processes, VMs) but still, it is interesting to learn from and think about.

derefr2• 12 years ago

Indeed, for all the free wins you get from Erlang's actor model (e.g. easy just-add-cores parallelism), you also get losses (e.g. copy-based message-passing is pretty much the least performant signalling mechanism possible.) But when you look at it all through the lens of fault-tolerance, there's really nothing you can change about Erlang's semantics without throwing away some guarantee or another.

When I hear that other languages (e.g. Go) have "actors like Erlang", I have to laugh. Unless you're willing to eat all the disadvantages and performance losses that come with Erlang's approach, you don't really have "actors like Erlang"; you just have green threads.

snake_plissken• 12 years ago

It's so awesome to see FreeBSD as the OS. Hard to learn in my opinion but worth it since it's so reliable.

_ouxp• 12 years ago

My experience was it was a lot easier to learn than various linux distros, at least several years ago. It has great documentation in the form of the Handbook, and its directory structure & the ports system are rational & predictable.

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/

edwinnathaniel3• 12 years ago

In terms of learning it's more stable than various Linux distros.

So in a way, it's easier to learn FreeBSD than Linux (Server) distros.

_nato_• 12 years ago

Something drôle I caught on Joe Armstrong's twitter feed: ``A machine learning researcher, a crypto-currency expert, and an Erlang programmer walk into a bar. Facebook buys the bar for $27 billion.''

scott_karana• 12 years ago

What are the other acquisitions/hires referenced? :-)

lostcolony2• 12 years ago

"Erlang rocks! Erlang continues to prove its capability as a versatile, reliable, high-performance platform. Though personally all the tuning and patching that was required casts some doubt on this claim."

Really? Because I would trust their claims all the more; they've been in the trenches, they had expertise in other languages (from the article, "Having built a high performance messaging bus in C++ while at Yahoo, Rick Reed is not new to the world of high scalability architectures."), and then they became -intimate- with the language, in removing bottlenecks to their specific use case...and they still make the claim Erlang is great in this domain. Whose opinion are you going to credit more than theirs?

(Also, I bet many of the patches they made that weren't specific to their use cases were submitted as patches to the BEAM)

rdtsc3• 12 years ago

Yap some patches or suggestions made it in and the greater community benefited. Some lock contention points removed, memory management stuff etc.

Also remember this is just one visible project. Erlang VM is being developed actively and many other companies and people are contributing. For an almost 30 year old language it very active. Maps are coming in R17 so are dirty schedulers, {active,N} on sockets, lots of goodies.

lostcolony2• 12 years ago

I know; R17-rc1 was tagged a month ago.

This was more my being agog at the author viewing the work that was done to the BEAM as evidence -against- Erlang being great, rather than the fact that these people with that level of experience stayed with it, and still raved about it, as evidence in Erlang's favor. And that the issues they encountered are probably fixed any way, or apply only to their use case (and so may still exist, but have a tradeoff that is worth keeping in other use cases).

wcummings2• 12 years ago

Omg dirty schedulers are included too? I need a new pair of pants :x

Dealing with NIF threads just feels dirty

signa11• 12 years ago

if someone is looking for more info on dirty-scheduler, here is slightly oldish slide: http://www.erlang-factory.com/upload/presentations/377/Ricka...

erichocean3• 12 years ago

What it means is that unless you are comfortable modifying Erlang itself (BEAM, internal data structures, hash table allocations, etc.), then your install of Erlang won't scale like WhatsApp. You'll also need a ton of monitoring because Erlang definitely isn't of the "just turn it on, it'll be fine" variety. Nothing at that scale is.

That's fine, but people do tend to oversell Erlang IMO. The types of things Rick Reed talks about in his talk are standard issue stuff. Lock contention? Okay, how about using an architecture that doesn't require locks in the first place?

With Erlang, you're forced into the Erlang way from day one, and it'll be tough to escape. The good news is that Erlang has a good design overall, so as long as you can modify the engine under the hood, you'll be fine.

I don't think anyone should be scared away from Erlang, just be aware what you're getting into—like any other tech you could deploy.

Xorlev• 12 years ago

Hard to avoid lock contention when it's happening in the OS.

austinz• 12 years ago

I read like 8 paragraphs of blather before I got to the first bit of useful technical information, a confirmation that they were using Erlang.

angelohuang• 12 years ago

This article also mentions about whatsapp in business usages. Interestingly, all those points resonate with a messenger project that we are working on. http://peer.im

izarov• 12 years ago

The performance levels they have achieved are impressive.

I think hot swapping code is a major factor in being able to quickly fix and develop a system like this.

I wonder how far one can go on another tech stack.

For example Akka is supposed to be able to handle up to 50m messages on a single box and there is some support for hot swapping in the Oracle and IBM runtime.

tuf14575• 12 years ago

As Facebook transitions into the next decade, they must focus heavily on mobile. I see a lot of people mainly using Facebook for its messenger app, so it's very smart of Facebook to buy a competing messaging service that is gaining huge traction in the international market. If WhatsApp were to expand into the States, Facebook could be in trouble. Very smart investment by Zuckerberg but I can see why people are shocked it was dealt for such a huge amount.

grogenaut• 12 years ago

First 1/3 of the article is talking about background or $16B?!?... Much of the rest is a list of hardware. Be prepared to skim a ton to find the actual subject of the article.

plainOldText• 12 years ago

Does anyone know whether their changes to the BEAM ever made into the official Erlang BEAM?

gwu78• 12 years ago

No Linux? Am I dreaming?

andyjohnson0• 12 years ago

"And using your cell number as identity and your contacts list as a social graph is diabolically simple."

Tthe linkage of phone number and identity seems to be the main reason why there are no tablet versions of the whatsapp client. I wonder how satisfied whatsapp/facebook are with this constraint, and whether they intent to address it? It is possible to imagine workarounds for devices that don't have a phone number, but they don't seem as friction-free as the current identity system.

uniclaude2• 12 years ago

You should have a look at how Line does it. You can use a login and password, but you can also use Line on another device through some QR code scanning. No password required IIRC.

jamespo• 12 years ago

I think they owe the developers of ejabberd a few drinks

retube• 12 years ago

WhatsApp = Email 2.0?

dpweb• 12 years ago

Interesting enough to check into Erlang. Even though most of us won't approach the load Whatsapp has to handle, it is very useful to understand how to pack more work into smaller hw footprints for any size business.

rehack• 12 years ago

Man, I burn with envy when read such stories of monumental technical achievements. For some reason have always been fascinated by scalability. But so far life has given me the opportunity in a very limited way.

Have been reasonably proud of scaling my own micro (in comparison) service. But right now feel very small and humbled.

edit: minor

The WhatsApp Architecture Facebook Bought For $19 Billion

Related Stories