WhatsApp scaled to 1B users with only 50 engineers

https://www.quastor.org/p/how-whatsapp-scaled-to-1-billion

by unripe_syntax • 5 years ago

636 454 5 years ago

jokoon• 5 years ago

I remember reading they worked very hard to make their app available on many, many types of phones. Which is weird because they regularly announce they drop support for certain phone OS.

nnx• 5 years ago

It made sense years ago... before the iOS/Android duopoly we have now.

toast06• 5 years ago

I was there from 2011 to 2019. We had pretty clear numerical guidelines for continuing support of a platform or OS. If the numbers are close, things like upgradability or difficulty of support make a difference: if all the devices running OS 1.1 can actually be upgraded to 1.2 and 1.2 fixes important things, then dropping support for OS 1.1 can happen with more users still there.

If builds need to be signed by the OS maker and they won't sign any more builds, user count doesn't matter (sorry Nokia S40 and iPhone 3G).

We did make some longer lasting builds for platforms we were ending support for, but there needs to be an end. Older clients don't support newer features and we didn't want to have a network where that was the status quo. I don't know what the current expirations are, but mandatory updates a couple times a year helps keep the feature support similar.

kosolam2• 5 years ago

WhatsApp is a spy app of Facebook nowadays

oblio2• 5 years ago

That was its fate anyway.

Some things just can't really make money unless they do it in scummy ways (or we haven't found a viable alternative or a way to protect them).

Public instant messengers, public forums, public file hosting, public image hosting.

shawabawa3• 5 years ago

With 50 employees their "please pay $1/year" model might have actually made enough revenue

alanfranz• 5 years ago

Whatsapp used to charge 1 eur or usd per year per user.

Assuming a 20% vat, that’s $800M in yearly revenue. If you spend $500k per engineer per year, you’re still left with $775M to handle infrastructure and all other expenses.

Whatsapp would have been sustainable without fb. That’s the regret.

jfklsdfjsdf• 5 years ago

Makes me wonder just what FB will do to Whatsapp to keep the money flowing in, as FB's core app/website declines.

octodog• 5 years ago

Whatsapp had 700m+ monthly users in early 2015 [1] before they scrapped their subscription fee in early 2016 [2]. Whatsapp was free to use in some jurisdictions for the first year, so we can't assume there was $700m in revenue, but there was certainly enough to pay for 50 engineers you would think.

[1] https://www.businessinsider.com/whatsapp-passes-700-million-...

[2] https://blog.whatsapp.com/making-whats-app-free-and-more-use...

KptMarchewa• 5 years ago

1) They won't succeed unless they were international from the beginning.

2) A lot of people would be rightly suspicious to use govt systems for their own private data.

globular-toast2• 5 years ago

Whatsapp is built on a massive, mature, global system specifically designed for sending messages. It's sad that it needs to exist and even more sad that it has to "make money" for someone. Aren't we already paying for our internet connexion/phone bill? Why do we need to be used by Whatsapp just to utilise it for its most basic purpose?

cyberpsybin• 5 years ago

And iMessage is compromised bloatware

seanhandley2• 5 years ago

I wonder if WhatsApp could be considered the most successful user-facing application with an Erlang backend.

polyomino• 5 years ago

What other candidates are there?

bradhe3• 5 years ago

Facebook Messenger, if I remember correctly, was also originally authored in Erlang.

peoplefromibiza• 5 years ago

while WhatsApp is the largest I am aware of (it's one of the leading platforms Worldwide anyway), there are quite a few, sometimes in the form of Elixir:

- Discord

- Pinterest

- Spotify

- Klarna

- Riot games (the messaging service)

- PepsiCo (for their billion making e-commerce)

- Toyota (for the connected platform)

a_humean• 5 years ago

What about its original purpose, which was telecoms infrastructure?

seanhandley2• 5 years ago

Oh, absolutely.

I framed it this way intentionally. Without a doubt the most successful Erlang apps are telecoms related.

feketegy2• 5 years ago

Probably that's why they opted for Erlang.

bjarneh6• 5 years ago

This article does very little to explain how they did that. It basically states the type of tech they used (Erlang, FreeBSD and SoftLayer), and something about not trying to over engineer stuff. The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.

willejs• 5 years ago

Theres a high scalability post from years ago that has a bit more information. http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...

vadfa5• 5 years ago

19B messages in & 40B out per day

whatsapp being a closed system I don't understand how this disparity is possible.

bjarneh6• 5 years ago

Thanks, it does seem like choosing a language that was made for doing things in parallel to begin with was a good idea. To be honest, I've always felt that programming in functional languages creates less bugs, but perhaps that's just my imagination.

aledthemathguy• 5 years ago

this is great. wondering if Telegram also have a post similar to this? (unlike whatsapp, they do store the messages)

Closi3• 5 years ago

The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.

It's something I think we have all seen a lot of times - that by the time a company is serving 1 billion users it has quickly expanded out it's engineering team hugely, and because of that additional abstraction is required and the complexity/LOC skyrockets.

bjarneh6• 5 years ago

it has quickly expanded out it's engineering team hugely

But makes little sense (as a developer/engineer myself) to think that growth in users, requires a ton of new developers/engineers. We are not tattoo artists, our job can scale indefinitely if set up properly, i.e. all you should need to scale further (to 1BN or 7BN) should be enough money to buy more hardware; which WhatsApp/Facebook clearly has.

mnsc• 5 years ago

"this team of 7 engineers is responsible for formatting the text content of a chat, including alignment and sizes of emojis and gifs"

"this team of 4 engineers is responsible for formatting the date of a message"

"this team of 7 engineers is responsible for the overall formatting of a chat"

"this team of 5 engineers is responsible for the formatting of the non-chat pars of the application, settings, profile page"

"this team of 4 ux persons is responsible for aligning the non-technical parts of formatting and the user experience over all parts of the application"

"this crossfunctional team of 5 is responsible for creating a framework to let the configuration of formatting be disconnected from the actual implementation of the formatting"

"this team of 7 QA engineers will aid in manual verification of changes and bugfixes but will also automate test cases for formatting in the entire application"

"this supporting team of 4 will develop automation tools and enable the formatting teams to collaborate in a high speed agile context"

mrweasel3• 5 years ago

The technology stack isn't even that relevant. There have been a few different articles about WhatsApp and their 1B users with only 50 engineers. The key in previous articles was that WhatsApp hired really smart people.

Having extremely talented engineers and a rather small scope is what allowed them to grow til 1B users, with a minimum staff. Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.

bjarneh6• 5 years ago

Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.

I think you are correct. It also works as some sort of advertisement. Others feel like they made the right tech choices, if they choose the same tech as WhatsApp or Spotify. Completely forgetting that they probably need to be somewhat proficient in Erlang to get the benefits from the language etc.

I guess that "hire extremely good developers" is less sexy than just choose Rust/Erlang etc...

Renaud• 5 years ago

I think the technology stack they choose is highly relevant to their success.

The choice of Erlang was crucial: it's basically built for communications and has all pesky issues like version updating, resilience to failure and scalability already taken care of by nature of its architecture and framework.

Of course, these 50 engineers were smart, most in that specialised space are, but I strongly doubt that they could have achieved the reliability and scalability of what made the success of WhatsApp with a PHP or Ruby back-end without more complexity, more ressouces and more hardware (like what FB and Twitter had to go through).

dgb232• 5 years ago

Absolutely. A relatively small, highly efficient team that invests in their knowledge, skill and processes is the cause of correct technology choices and not the other way around.

Cthulhu_3• 5 years ago

The article is honestly kinda low on details, depth or new insights, but at least it links to some sources.

LewisVerstappen• 5 years ago

Hey,

Author of the article here.

Totally agreed. Apologies for that!

The article isn't really "an article" but it's an email newsletter I send out. I'm quite surprised to see it on the frontpage of HN lol.

So what you're reading is one of my emails.

It's meant to be more of a summary with additional links to sources where people can read more if they're interested (emailing people long-form articles doesn't work well as a content format from what I've seen - even ignoring the length limit that email has).

I'll add in some more links for additional reading (like the high scalability post) to give some more detail for people interested.

Thanks a lot for the feedback.

bjarneh6• 5 years ago

OK, I guess any post on this site is dissected a bit too much - looking for insight that was sometimes never even meant to be there. It is usually a good thing, as we expect anything making it to the front page of HN to contain some profound knowledge. Sometimes a popular title referring to an interesting topic can gain a lot of up-votes as well, and then the criticism can be harsh :-)

bradhe3• 5 years ago

Some of these points are a bit suspicious, and I bet don't tell "the whole story" the way they're intended. For instance, the statement about not investing in automation unless absolutely necessary or the point about hot-swapping Erlang code.

Would be good to actually hear from one of these engineers for a discussion on the paint points of working on WhatsApp back then.

tdrdt4• 5 years ago

Automation can be good, but will also be another system to support.

I think this XKCD applies: https://xkcd.com/1205/

bryanrasmussen• 5 years ago

If you shave off 1 second I have a hard time believing that 1 second will not just be swallowed up by the entropy of coffee breaks.

bradhe3• 5 years ago

I'd like to see what their criteria for "absolutely necessary" is--I bet it doesn't fall on the right side of that table.

corobo2• 5 years ago

Is there even a correlation between scale and engineers, other than scale = VC money = hire more cause we can afford it?

Ekaros• 5 years ago

I would expect there to be certain scaling. Specially when you move to more severs and more redundancy. Cloud solves part, but still not everything. If single person can support 10k or 100k users. You might still need more than that one to move to million or hundred million.

Also some features like language support and such get more complicated when you want to widen userbase from just English speaking ones.

corobo2• 5 years ago

If single person can support 10k or 100k users

I asked if there was a correlation here, using an if statement as an answer doesn't answer it haha. You're now asking the same question I was, just in-line

tdrdt4• 5 years ago

"The simpler product makes it much easier to maintain and scale."

This is good news for the developers, the product owners and the customers.

I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped. As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later.

I have seen days go to waste because of Docker misconfigurations, problems with dependency versions, outdated dependencies not available anymore, bugs deep down in an unknown dependencies, and so on. Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems.

The (old) WhatsApp teams sounds like a great workplace.

johnfn• 5 years ago

Personally I want to build systems.

Sure. And so do I. Specifically, I don't want to rebuild systems that have already been built for me which I can use as dependencies. And I don't want to spend all my time troubleshooting bugs, either - I'll take a well-maintained repository on GitHub with thousands of users bug-testing it for me over some janky thing some guy on another team threw together a few quarters ago before he quit.

tdrdt4• 5 years ago

Having dependencies is not bad. But slapping a system together with a ton of dependencies you don't know about is bad.

Simple is better in my opinion.

choeger• 5 years ago

with thousands of users bug-testing it for me

Replace this with:

with thousands of users just like me that once picked it as a dependency and then forgot about the project

feketegy2• 5 years ago

exotic tech that is the latest and most hyped

You answered your own question. Because it's "exotic" and "hyped".

chinathrow• 5 years ago

exotic

My fast reading brain with not enough caffeine read that as 'toxic'.

nodejs_rulez_12• 5 years ago

...As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later.

But the sort of people choosing a JS stack are quite quick to move on. Make a mess, slap a new achievement on CV and dash.

paraknight• 5 years ago

I don't agree, but I also find your username ironic. Did you dash?

freeall• 5 years ago

Do we really think Javascript is exotic? Hasn't it proved itself over many years now?

That doesn't mean it can't have issues like anything else. And because of the massive scale of the language you'll see a lot of shit. It's a language and ecosystem like anything else - it's what you do with it that matters.

_Understated_• 5 years ago

I wouldn't say JavaScript is exotic, it's more to do with what is being done with it.

Traditionally it was used to enhance HTML pages with some DOM manipulation but it's now being used for a million other things too.

So historically, if your JavaScript code broke your page would still render as it was just html but now there are all sorts of build processes and long chains that use JavaScript to create the html in the first place... I'd consider that the exotic bit.

Traditional JS running in the browser manipulating DOM elements is very much a foundational aspect of the web and won't deteriorate with age (with the exception of perhaps deprecated functions in the far-off future) but all these js libraries and tools and build processes with massive dependency-chains are what's being referred to.

hasperdi• 5 years ago

These issues that you described can happen with any language, ask any programmer or system engineer with long enough experience they’ll have stories with weird problems.

Regardless which language you pick, there will always be these kinds of issues.

Cthulhu_3• 5 years ago

These issues that you described can happen with any language

While you're not wrong, I do strongly feel it depends on the language and architecture chosen. I'm a Go developer these days, and there's a big mindset to e.g. avoid dependencies, and for those dependencies to avoid including even more dependencies, keeping things fairly lightweight and with a low 'attack surface'. I mean I personally wouldn't mind a stricter type system and native enum support and some other things, but for now, I enjoy how basic it is, whilst avoiding the footguns that C/C++ brings.

trhoad• 5 years ago

A static React site, an Express backend, and a REST API is hardly exotic

capableweb• 5 years ago

Seems redundant to have a Express backend + REST API, why can't you just build your REST API on the Express backend?

And also, React is complex, hardly anyone seems to know the internals. The API interface might be easy to learn, but "simple" is not something that you should label React with.

BigJono• 5 years ago

Yeah and if that's all people used, JS would be the best language ecosystem ever.

Too bad I can't even remember the last time I jumped on a project with less than 80 dependencies.

javchz• 5 years ago

Well, WhatsApp had to support exotic cases like BlackBerry, Symbian OS (Nokia), and Windows Phone. I will say, dealing with JS stacks, seems easy in comparison to mobile fragmentation from back then.

vadfa5• 5 years ago

I don't see how this is related. Server-side you write and expose an API. Then you have 1-3 developers for each platform that write code that consumes that API. Those developers don't need to know anything about the internals of the server.

Kelteseth• 5 years ago

But this would be client side only, right? These issues would also only occur in the corresponding app repository, when building a separate native app for all platforms. The backend for all of these apps would be the same, not the frontend.

1cvmask• 5 years ago

And JAVA ME (formerly J2ME). The worst of the mobile fragmentation by far.

nobleach• 5 years ago

Having had to write a "mobile" app for Blackberry, I would have KILLED for the option of using JS.

hnlmorg• 5 years ago

Docker isn't itself the problem. It's people who insist on `latest` et al.

Docker itself can provide a stable backbone that allows you to run repeatable and predictable builds.

stackbutterflow• 5 years ago

Honest question, is specializing in a JS stack bad in terms of career prospects? I've been using express + react/vuejs professionally for a couple of years and I wonder if it'll be in demand in the next 10-15 years.

goodoldneon• 5 years ago

You’re fine — don’t listen to JS haters on Hacker News. They pine for the simpler days of sites that primarily use HTML and CSS, with maybe a little bit of JS sprinkled in. But consumers are increasingly demanding an app-like experience from the web, which requires JS frameworks.

As for Express, JS on the backend will be popular as long as JS on the frontend is popular. JS backends have their flaws, but there’s a lot of value in using the same language in the browser and server.

Cthulhu_3• 5 years ago

Yes, but you'll find that a lot of work becomes maintenance of older systems, and less new and shiny things.

I do think the 'craze' of new frameworks popping up left and right quieted down some years ago, and things have roughly settled on React, Vue, maybe Angular (which lost a lot of its shine after the Angular 2 announcement fiasco), etc.

For a new application, I picked React + Go because I'm confident that ten years down the line it will still be maintainable - although, Go moreso than the front-end, which doesn't feel nearly as solid and stable.

askonomm• 5 years ago

JS is everywhere though. You even have parasitic languages that compile to JS (ClojureScript, TypeScript, etc). You got Node and also Deno for back-end stuff with support for multithreading. Concequently, JS is among the most popular, and most used languages.

I do admit the ecosystem needs to find a way to have more stability, because NPM outdated package tree nightmare is pretty bad, but I guess that's what you get with an industry that moves so fast, so in that case it's up to you as a developer to not use too many dependencies.

Anyway this is all to say I don't think JS is going anywhere for the next 10 years.

pas3• 5 years ago

No, it's great. But keep up with the "ecosystem". Learn a bit of frontend, backend and testing too. (Then keeping it fresh won't be a problem.) And we shall see where it goes in ~5-10-... years.

halfmatthalfcat• 5 years ago

I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped.

So the same cannot be said for "newer" languages like Rust, Go, and other's whose ecosystems and paradigms aren't completely fleshed out yet?

This comment reeks of someone who is looking from the outside in when it comes to building stuff in JS. Most comments I see criticizing JS stacks are so superficial and demeaning that's its obvious the commenters have little to no experience working in JS.

shmel• 5 years ago

Why do you compare it to Rust though? JS is as old as Java, however its ecosystem seems to be much less mature (at least from outside). I haven't touched Java for 15 years, but I am fairly confident that people still use Apache Ant as build system. Every time I try to peak into JS world, it appears similar to tiktok trends in terms of how quickly people move from one thing to another.

yen223• 5 years ago

WhatsApp was built on Erlang and FreeBSD, which feels way more exotic to me

tdrdt4• 5 years ago

FreeBSD has been released in 1993 and has been a well known OS in the hosting world.

Erlang has been released in 1986 and has been a well known language for message systems.

When I think about exotic I think about some new tech that sound really nice but is still very unproven.

RNCTX2• 5 years ago

FreeBSD was a more stable/mature platform to build on when WhatsApp was designed, particularly in terms of network performance.

antihero• 5 years ago

exotic tech that is the latest and most hyped

See whilst obviously the JS ecosystem is quite volatile and can be trend-based, I think there are benefits - the language breeds innovation. Innovation doesn't always make the correct choices, and some tech-conservativism is important in many types of companies. But if our ancestors had just been lazy and just stuck with what was known and good, we'd never even had computers in the first place.

netcan4• 5 years ago

For all sorts of reasons, good and bad.

One is that the future is unknown. Your goal might be to keep the product simple and scale to 1bn users, but you don't really know if it will play out like this.

Maybe you plateau at 1m users with the initial idea and pivot the product somehow. Now the app does dating, social media or specialised communications for highway construction teams.

Tradeoffs are easier to make in hindsight. Whatsapp is a good demonstration of what you can gain with good trade offs. Do less but well. Whatsapp did trade some things off though. Their web version came late, is feature poor and a little clunky. Same for a lot of features, compared to similar apps ATT. IDK what exactly can be traced to the stack, but their decisions around user identity are similar. Phone numbers only. One device at a time. They gained simplicity, but lost options.

It's easy to see the benefits in hindsight. Real time is harder. Play that game again and it might turn out different. Most apps that set out to have 1bn never get close. At this point, scalability doesn't matter and tradeoffs made for maximum scalability don't seem as wise.

baktubi2• 5 years ago

I think the Unix philosophy should apply here. The fact that WhatsApp did one thing well and it had a good business model (was it $1 per year?) is the important bit.

When you say, “Now the app does dating…” I think the right move is to scrap the project because that’s a colossal fuckup. Unless you have Microsoft cash to have multiple colossal fuckups in a row, don’t add another dating app to your messaging app. Ergo, less is more.

WhatsApp probably started with passion and a solid vision. The Zuckerberg gave them an offer they couldn’t refuse. Anyone will take 19 billion for a basket full of Indian users.

Another thing WhatsApp did well is they targeted ALL phones. They didn’t abandon their users like the fang-bangers (sorry been watching True Blood— FAANG is an annoying acronym so they are hereby fang-bangers).

ignoramous• 5 years ago

I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech

True, but one can do simpler things with JavaScript, too.

And a word on exotic tech: Well, some of it is really good in helping small dev shops achieve scale of development, deployment, and maintenance.

teekert• 5 years ago

"Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems."

Don't we all, I guess we all crave that snap, click, build -feeling Lego gave us. But in my experience building a system means debugging all kinds of weird problems. At least, when I start something, I usually for a long time feel like I'm just hopping from weird problem to weird problem. But, TBH, those problems don't seem weird anymore as experience grows.

atirip• 5 years ago

It is called CV driven development. The cold truth is if you do not have latest SV hype in your CV, you become pariah, unhireable. It’s that simple.

onion2k2• 5 years ago

You know that JavaScript developers don't see JavaScript as weird or exotic, right?

Traster• 5 years ago

I think one thing to consider is that WhatsApp is probably by nature a scalable application. It is in the end, simply a client app that sends data to other client apps. So by nature, you scale in parallel very well, there are no 1-all communications. So unlike something like facebook where the data all needs to go to a central server and be polled by an unknown (but potentially arbitrary large) number of clients, in this case you have a huge number of point to point communications. It also means your server application can be largely stateless.

I'm not saying that it's an easy problem to solve, but it is a much easier problem to solve than a lot of other applications.

lampe3• 5 years ago

I fully agree here.

Sending messages to a known (small) number of people is a solved problem for a long time. Email for example.

With WhatsApp, it is even easier than email because WhatsApp is a closed system, and you can optimize.

They used an uncommon setup? Okay cool. How does this help me? or any other company?

I'm not sure what I should learn from this story?

The only thing is: Feature creep is bad.

crate_barre3• 5 years ago

To add to this, not only is it a simpler problem, but they stood on the shoulders of giants that already solved it. They used ejabberd, a full-featured xmpp server written in Erlang. Imagine half your job already done for you. Granted, they heavily customized it by this point, but they did a great job picking the right off the shelf solution to begin with.

Tepix• 5 years ago

Why do you assume that WhatsApp uses point-to-point connections? My assumption is that most devices cannot talk to each other directly and require central servers.

hbn2• 5 years ago

I could be wrong but I don't think he was saying "point-to-point communication" in the technical aspect of the term (i.e. peer-to-peer). Rather, just making the point that they're not storing tons of data on their server that has to be polled on demand by an unknown amount of clients forever. Even if there's a server between the users, once they receive a message and pass it onto whoever necessary, their job is done and they don't need to worry about that data any more.

namdnay• 5 years ago

It is in the end, simply a client app that sends data to other client apps

That's not the case. Consider the "one tick" messages - they must be on a server somewhere. One tick means they've left your phone but aren't yet on (all) the recipients' phones

Drew_• 5 years ago

Servers proxy the message yes but once all recipients (maybe a dozen clients in the worst cases) receive the message, the server can drop the data once and for all. No need to optimize for harder problems like search, caching, or recommender systems.

M0r13n2• 5 years ago

I'm honestly not that surprised about the size. In my experience, efficiency and the number of engineers have not necessarily correlated.

For example, I used to work for a company with about 500 employees (relatively large by German standards) and now work for a competing company in the same industry with less than 150 employees. The core engineering team only consists of a dozen engineers.

The output is noticeably larger. The number of new features and especially the stability is significantly better. It is noticeable that due to the small team, each project has one or two "heroes". They are responsible for most of the commits and know the system inside out.

These "heroes" have been working on the same or similar systems for over 30 years in some cases and have experienced a lot in their careers. As such, they can bring experience and advice that is just out of reach for others.

Given the choice, I would always prefer a small team over a large team.

altacc• 5 years ago

Having worked in a range of companies, there is definitely an inverse correlation between company size and speed, especially for individual teams. At 500 people you're well into enterprise level organisation, where a lot of processes are put in place to minimize the amount of damage a single person can do to the organisation. That also puts a speed limit on those efficient and knowledgeable contributors.

Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...

mywittyname• 5 years ago

I think this is more correlated to maturity than size. It's just that size is highly correlated to maturity.

I've been in large companies where I was the admin/owner of their AWS account. And I've worked at companies with 200 people where I have to submit tickets to get anything created in AWS. The major difference was how mature the product was.

emerongi3• 5 years ago

As long as the core system is built on solid engineering, this definitely makes sense.

From my experience in a deadline-constrained environment, things end up being slapped together until they "just work". Adding new features gets more and more difficult and if you're still constrained by deadlines, then the solution is to hire more, which is effectively a brute-force solution to meet the deadlines. Output per employee gets smaller, but total output should grow by some marginal amount.

allcentury• 5 years ago

Absolutely. This is also when bigger and more pervasive bugs enter the conversation. Decisions that kicked the can 2 years ago are now a big problem and no one team can fix it.

jawns2• 5 years ago

Heroes are a red flag for me, as a manager, trying to build a resilient team. Which is not to say that you can't have a range of experience levels. But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up? Heroes tend to be a single point of failure.

civilized• 5 years ago

You're right up to a point, but I've seen far worse red flags. Like companies that can never acquire these heroes in the first place because no one competent would work there that long.

As somebody else mentioned, these companies end up regularly throwing millions at consultant projects that always fail. In other words, they pay a hefty premium for shitty temp "heroes" that give them less than employee heroes would have given them.

You see this a lot in the traditional finance sector, where managers don't appreciate tech workers and relentlessly fuck themselves over trying to save a dime.

gen220• 5 years ago

If you have good trust and communication within a team, the scenarios you describe are surmountable.

Every strength is a weakness, and every weakness is a strength. IME, heroes are no more a red flag than pretending that good engineers are interchangeable. It depends on the context.

The expected outcomes of these teams are different, and that’s OK. If you’re a very small company and don’t have a couple heroes, you won’t build anything important. If you’re a very big company, the heroes that built it left years ago, and you need resiliency more than new heroes (unless the business is going sideways and needs saving).

M0r13n2• 5 years ago

This is definitely a potential risk. However, you have this risk with small teams (less than 10 developers) even without hero roles. The smaller the number of developers relative to the complexity of the product, the higher the probability of a single point of failure.

However, you can reduce the risk through good documentation, good engineering practices and so on. As long as you are aware that the team is partly made up of heroes, you can prepare accordingly. So as a manager, I can try to ensure that a few heroes keep productivity high while making sure that the rest of the team understands the basics of the system through things like code review and the like. In the worst case, a teammate can then at least get up to speed quickly.

thewhitetulip2• 5 years ago

The worst team is the one with a single point of failure. All hell breaks loose if they're unavailable. This makes me wonder why managers have teams like this?

vp8989• 5 years ago

Being replaceable tends to make work less satisfying. All the places I've worked at that followed your advice had the most churn and the least productivity/ROI on engineering $ spent.

jbverschoor• 5 years ago

You hire more "heroes". Pay what they're worth. Make them happy.

Or you just pay an order of magnitude on hosting, performance consults, bandwidth, etc. etc. etc. And STILL won't EVER get to the same result, because you can't always pay (I'd say almost never) your way into a stable and performant solution. Look at all the others.. Nothing works as well as whatsapp.

Good luck hiring 500 engineers who have no idea what happens throughout the whole stack.

I'd rather pay what you save on cloud providers and snowflakes to them. (not even taking into account the result of these "heroes".. the business value / dominance whatsapp has because of this)

You can hire 1000 people who can't draw, or maybe just a little bit, or you can pay an artist and get the result you'll otherwise never get.

mensetmanusman• 5 years ago

Use heroes to make jumps you can’t make with cog machine teams, then manage the prior tier platform to be resilient.

illegalmemory• 5 years ago

5 heros on vacation for 7 days, and coming back and finishing the work in next 5 days is more comfortable position than 50 clueless engineers claiming work will be over in 4 days.

killtimeatwork• 5 years ago

Many/most teams don't have heroes at all, they only have noobs :) I.e. most people stay on a team for a year or three and change jobs afterwards - there's no chance of accumulating knowledge. Nobody is sure how things work and why they are the way they are.

I believe many managers are fine with this, because the team is producing something (i.e. "velocity" is higher than zero), but are oblivious to the fact that team could be way more productive if the knowledge was actually retained in people's heads and not just lousy attempts at documentation.

throw1234651234• 5 years ago

I understand your point, but what managers fail to understand is that someone usually "carries", i.e. is actually responsible for the success of the product. I have even seen it be a manager...once.

bob1029• 5 years ago

Heroes are a red flag for me, as a manager, trying to build a resilient team

Heroes are your most powerful asset, but you have to use them responsibly. The best thing you can do when handed a 10x unicorn developer is to try to document 100% of the things they say & do, and also make it a requirement that the hero mentor others some % of the week. E.g. For 1-2 hours every Friday you force them to hold a "no stupid questions" session.

qaq• 5 years ago

My first interaction with this management philosophy was when our partner AOL replaced experienced SRE serving our project with 3 far less experienced people for the same price. What used to take an hour now literally would take weeks on the bright side for the manager he had way more reports now.

AnIdiotOnTheNet• 5 years ago

But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up?

I don't know why people insist on using examples like this instead of the more obvious one: What if they are hit by a bus?

Any of the other examples have trivial solutions in the event of an emergency: call them and maybe offer them some money. If they are very stubborn then go to their home and beat them with a wrench. But you cannot get information from a dead person no matter how much money or how big a wrench you have, and people die suddenly every single day. That's the worst case you need to be making contingencies for.

jimmont• 5 years ago

It appears you have become the hero and are maintaining that status.

SkyPuncher• 5 years ago

It depends on the company/stage/setup/etc.

Having a "hero" at a 5 person startup can mean life when death was inevitable. Having that at a 500 person org likely means another team is left cleaning up crap.

Jensson6• 5 years ago

Managers prefer a team reliably working at 0.1x speed rather than have it work at 1x speed 95% of the time. Yes, sometimes people leave and you will have less velocity for a while, but people pick it up and you go back to high velocity. To fix that you ensure the team always works at slow velocity, that way it wont get slowed down since it was already performing as if you lacked those heroes from the start.

yodsanklai• 5 years ago

Given the choice, I would always prefer a small team over a large team.

I think that's one of the conclusion of the "mythical man-month" book.

commandlinefan• 5 years ago

efficiency and the number of engineers have not necessarily correlated

Often inversely correlated...

bluedino• 5 years ago

I worked at a place that made a group of loosely-related websites. Basically different spins/themes on the same products. There were 4 of us on the development team. There were 10 websites. Some new, some in maintenance mode.

I quit after a few years, and about 3-4 years later they went on a hiring spree. Tripled the number of developers. Refreshed some products, discontinued some, and some went into maintenance mode. They still had around the same 10 websites, but 12 developers.

The products didn't get better. The products didn't get made any faster. They didn't have any less bugs. My wife actually works there now, and when she tells me about the issues they have I laugh a little bit.

It's all the same problems we had back then. Invites don't work. Teams doesn't work. Uploading profile pictures has issues. Entering data doesn't work. Registration is broken.

ratww4• 5 years ago

I had a somewhat similar experience with a large tech company. Grew from <100 to 600 in a few years, but in the meantime they weren't even able to update the homepage. Not even the content changed, in almost three years. Everything was convoluted and over-engineered. The whole thing was rewritten from scratch twice, and when I left there were plans for a third rewrite from scratch. COVID destroyed the company: there were zero customers. But the website still struggled to stay online. That's when I decided to leave.

topdancing2• 5 years ago

You can very easily build your own WhatsApp server today by simply deploying https://www.ejabberd.im/ on a box and using https://conversations.im/ on an Android phone.

southerntofu5• 5 years ago

Why is this being downvoted? Whatsapp is notoriously "just" a highly-optimized fork of ejabberd [0]. ejabberd is written in Erlang/OTP which enables quasi-horizontal scaling and amazing failure recovery, along with incredible uptime.

What's different about Whatsapp compared to a traditional XMPP experience is:

- it uses a single centralized server, without federating

- it uses phone numbers as identifiers instead of nicknames

- it has a tightly integrated (closed-source) client

So you can not only "build your own WhatsApp", but also federate with others doing the same, which is pretty cool and prevents a single actor from abusing the whole network.

[0] https://www.infoq.com/presentations/whatsapp-scalability/

AshamedCaptain2• 5 years ago

Doesn't anybody remember how literally Whatsapp's founder wrote on one of the ejabberd mailing lists with something like "I just installed the server, please help me configure authentication" or the like?

Retrospectively that has always colored my perception of how relevant technical aspects are for a successful startup (i.e. not much). Marketing (and viral, user-capturing & monopolist practices) are (sadly) much more important.

toast06• 5 years ago

http://www.erlang-factory.com/sfbay2014/rick-reed is the original link for that presentation, FWIW.

dopamean2• 5 years ago

Comments like this always remind me of that old comment on here about how dropbox could easily be recreated with fsync or something.

1f60c2• 5 years ago

Here it is, for the uninitiated: https://news.ycombinator.com/item?id=9224

paxys2• 5 years ago

And talk to yourself on it?

topdancing2• 5 years ago

Not at all, I have a bunch of friends on my private server and getting someone an XMPP client and account takes less than a minute. No phone number or email required.

Sure, everyone out there isn't going to be on my server - but for the close friends I care about - it's been flawless.

zaik• 5 years ago

I decided to delete all proprietary messaging apps and use XMPP exclusively in Janurary this year. It took some time to explain but now most of my friends and family can now be reached via XMPP.

LudwigNagasena• 5 years ago

Well, what number would one expect? Surely, the workforce doesn’t scale linearly with users.

zibzab• 5 years ago

Remember the fast-cheap-secure triangle of product development?

Now guess which one they skipped

tim333• 5 years ago

I'm not sure the development was super fast. They also kept it fairly simple.

southerntofu5• 5 years ago

The client is another story, but for the server they forked ejabberd which is a reliable, established XMPP server. So they probably had "fast-cheap-secure" on the server-side, because 6 years had been spent ironing out ejabberd bugs before the first version of Whatsapp came out.

My understanding is that vulns in Whatsapp so far were client-side, but i'm interested if i missed some on the server side.

Zash• 5 years ago

They had some homebrew encryption and authentication before, see eg https://blog.thijsalkema.de/blog/2013/10/08/piercing-through... https://blog.thijsalkema.de/blog/2013/10/08/piercing-through...

varispeed• 5 years ago

I bet these engineers got only a tiny tiny fraction of the billions of revenue.

It always fascinates me how in the engineers' mind it reconciles that their work brings corporations they work for billions and yet they have to rent a tiny apartment for a substantial portion of their salary and otherwise lead a pretty average life. Then going to lavish offices witnessing how the money they could make use of is wasted on vanity.

southerntofu5• 5 years ago

That's the founding principle of capitalism: private property.

People living someplace don't own it, and people working some field don't own it. Some "owner" owns the land and means of production and extracts value from people doing the actual work.

Yes, it's a deeply broken system. A slightly better system revolves around self-organized workers coop where everyone gets equal pay and there are no shares to hold (or everyone owns the same amount). An even better system abolishes money and private property so that people can live meaningful/useful lives without worrying about imaginary numbers ruining their entire existence.

xchip• 5 years ago

...the other 450 engineers are just for spying and tracking people

up6w6• 5 years ago

This just makes more clear that we didn't try enough creating an open-source instant messaging app that respects our privacy as it should be. It's probably much more difficult to do the same thing as WhatsApp did because of the dependency we created around it but I think we just need the right initiative to do so.

baby• 5 years ago

There are several hard problems linked to what people really mean when they say privacy.

snarf21• 5 years ago

It is hard to get the general public to care about privacy when they post pictures of every meal and share every thought and emotion they have with no filter.

jodrellblank• 5 years ago

It’s hard to get the general public to care about inferior things pushed on them by patronising people sneering at them. How is a photo of some food important to privacy enough to be the first thing you teach for? Seems more like punching down at people you see as inferior because photographing meals is “so lower class, right guys?”.

amelius3• 5 years ago

Chat is the number 1 toy problem that is explained in every Erlang book out there.

Therefore, honestly I'm not really surprised to see 50 engineers capable of building a product around it.

12ian34• 5 years ago

The many comments alluding to the technical challenge of building a chat app with 50 engineers being relatively easy are forgetting that scaling said chat app to 1 billion users is absolutely not easy and a function of more than just the engineering quality and technical challenge. There's MUCH more to a successful product than engineering...

tantalor• 5 years ago

... such as? Don't leave us hanging!

dunefox6• 5 years ago

I'd imagine getting the users in the first place. That has nothing to do with engineering quality and scaling up, though.

jimbojet• 5 years ago

User count is fairly meaningless. How much was that in daily active / weekly active users? How many of those were bot / spammer accounts being “seasoned” to then sell to abusers?

shmatt• 5 years ago

There was a joke, or maybe anecdote, going around the Israeli high tech industry when Google purchased Waze

When it came time to get an overview of the codebase being purchased. The Google Maps car navigation iOS team had over 200 engineers. The Waze iOS team had 2

lightning19• 5 years ago

off topic but I recently started working with an Israeli tech company and I'm amazed by the culture they have, what about Israel produces such high levels of entrepreneurship?

Jensson6• 5 years ago

The Google Maps car navigation iOS team had over 200 engineers.

I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many.

Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.

TacticalCoder• 5 years ago

What is amazing to me is that they did scale to 400 million monthly active users with less than 50 engineers... In 2014.

That is: on hardware from seven years ago and older. Which is an eternity in tech. Since then we've seen incredible new hardware come out (EPYC 2 in 2018 for example).

One can wonder: up to how many MAU could a service scale today, on today's hardware, with a small team? And what about tomorrow's hardware...

This is fascinating.

hnarn• 5 years ago

Seven years can be an eternity in tech, but is it an eternity counting from 2021? I mean, AWS launched in 2006, Netflix expanded to Europe in 2012, Docker had their initial release in 2013 and so on, so none of these major events are that "new" anymore. Obviously things have changed (a lot) during the past seven years, but have they changed as much as they did between 2007 and 2014?

What I'm saying is that for the general case of "scaling with few engineers", it seems to me that the tools available matter a lot more than the specifics of what hardware is available at the time. Launching an international service "from scratch" with a small team and a modest budget just wouldn't have been possible 20 years ago, but not because CPUs were too slow.

JohnJamesRambo• 5 years ago

Remember this when people talk about Tether and how it can’t be real since it has X employees.

danrocks• 5 years ago

Tether can't be real because it's a scam, not because it has X employees.

streamofdigits2• 5 years ago

Beware of survivorship bias.

Team does X, does not achieve scale, nobody knows about it

Team does X, achieves scale, X is the new sliced bread

papito4• 5 years ago

The first team does not achieve scale because they are still building microservices. The word of the day is "lean".

streamofdigits2• 5 years ago

intuitively I would agree, but the point of my comment was that intuition might be a poor guide if we want to understand what kind of team and software architectures "succeed".

if you rewind some years microservices was also the word of the day (and probably there were some iconic projects picked to justify...)

unsungNovelty4• 5 years ago

They not only used FreeBSD for their servers, their CEO at the time Jan Koum donated $1 million dollars to FreeBSD foundation too. Talk about giving back!

The only thing they failed at was making money. I really wished they had. They knew what Whatsapp meant to the world. Look at WA now.

https://freebsdfoundation.blogspot.com/2014/11/freebsd-found...

moralestapia3• 5 years ago

The only thing they failed at was making money.

Huh? They're all million/billionaires now. That part of the plan seems to have worked.

cerved• 5 years ago

presumably they meant revenue

btbuildem• 5 years ago

Because they sold to FB. I think parent meant "they failed to make money on their own"

unsungNovelty4• 5 years ago

I meant they as in Whatsapp - the company. And yeah to be more specific, I meant making revenue on their own.

paxys2• 5 years ago

WhatsApp was successful only because they had no intention of ever making money. Simple lightweight app, no tracking, no ads, no upsell. Yet they were funded by VCs who would want a return on their investment at some point.

mavelikara• 5 years ago

WhatsApp was successful only because they had no intention of ever making money.

IIRC they had 450M users when they sold to FB. Their plan was to charge $1/user/yr. A $450M ARR company run by 50 people would have done fine.

papito4• 5 years ago

Because complexity kills. Something that has yet to be learned by the new generation of developers. While disciples of FAANG create a gazillion microservices that they spend most of their time debugging in production ("observability"), smaller teams just get shit done.

Since "monolith" has become a dirty word, it would be instructive to remember that some of the major companies out there are still light on distributed systems, which we all knew even back in the olden days of the internet were an absolute killer of efficiency and productivity. Yes, "microservices" is just another word for it - believe it or not, there was life before Node and Docker.

Whatsapp, Dropbox, Instagram, StackOverflow - all existed or still exist as one easy-to-maintain application. But go ahead, "be like Google".

lumost• 5 years ago

My 2cents.

Messaging is a weird app to focus on mau/dau. There have been many messaging protocols which scale to hundreds of millions of users, and there is prior art of large not particularly valuable chat systems like aim, gchat etc.

No one is going to use a consumer messaging system which charges money or shows ads. Which brings to mind substantial questions on the business model others are using the messaging platform for e.g. data collection.

closeparen2• 5 years ago

WhatsApp at this stage had a small signup fee.

pyuser583• 5 years ago

Scaling a single, discreet, product shouldn’t take a ton of engineers.

dano• 5 years ago

I looked at it this way. The thing most engineers, hero's or not, want to work on is new projects and to build products customers enjoy. The last thing they want to work on is maintenance. In my last personally owned company, small as we were, the engineering team and I documented how things worked as part of what we did every day. We found that by having the inner workings documented so that anyone in the team could effect a repair or update a piece of code freed us to work on new products and services. Just having the system continually documented freed our minds and kept us mostly happy (everyone has a challenge from time to time) and productive.

The documentation, not that it really matters, was done in a MoinMoin wiki. It was written by engineers, for engineers. We didn't stop marketing or sales or anyone else from reading it, but the target audience was the engineering and operation staff thus no one had to over-explain the vocabulary of our architecture.

KennyFromIT• 5 years ago

How did you manage to keep the documentation from going stale or efforts being duplicated?

arthur_sav• 5 years ago

Just having the system continually documented

In my experience this is the hardest part. Can't count how many times my team started documenting and then slowly, for one or the other reason, the documentation became outdated.

The cycle is: Meetings for communication -> becomes too cumbersome and time consuming -> implement policy to document things -> everyone start documenting (for a month or two) -> docs are not up-to-date because people forget/leave etc.. -> go back to meetings

castlecrasher2• 5 years ago

In my experience it's a culture thing. When I learned to love Jira/Confluence was when the product team conducted meetings directly from Jira and during task discussion almost always asked us if tasks and documentation were done.

It could have been really annoying but the product lead was extremely empathetic and that could be felt in any task discussion, and his attitude made me want to do it instead of making it feel like busy work like it has in other companies.

albrewer• 5 years ago

This is supposed to be enforced in code review or with other team practices / processes.

deathanatos• 5 years ago

How did you get others to read the docs?

This is ~our approach, but it seemed to stop after ~150 employees or so.

(You also seem to have the time to write the docs…)

unbalancedevh• 5 years ago

In my experience, documentation is the first thing to go when the workload increases. There are always tasks that get labelled as "important, but not urgent" that get pushed out; and documentation that is already known to everyone immediately involved is easy to push out beyond the product release date. Of course, by then other priorities are moved up, and documentation never recovers.

The only way I've seen documentation get the priority it needs is if management mandates it as part of the release package on which developers' performance evaluations are based.

bcrosby95• 5 years ago

I must be weird. I love maintenance. The best bugs come out of that. And it's hard to tell if a system was well designed except in hindsight; maintenance programming is the best place to be for that.

t-writescode• 5 years ago

You’re not alone. I may not love “maintenance”; but, I do love resolving performance issues and shoring and standardizing existing stuff a lot. I think I most like a balance.

myohmy2• 5 years ago

I absolutely hate maintenance, so I love people like you. That is why its great to have a small but diverse team.

dopamean2• 5 years ago

I thought what allowed them to get to that many users was that they were on so many platforms and that they used a large team of remote engineers in Europe to do that.

EVa5I7bHFq9mnYK• 5 years ago

What allowed them to get to that many users was

1)mobile operators charged an arm and leg for text messages, especially in developing countries

2)whatsapp did not require registration with user name and password, as their competitors (Skype, ICQ and others).

bravetraveler• 5 years ago

Queue endless business majors wanting to copy them

rdsubhas• 5 years ago

My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out.

Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers.

WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long), no account creation (your phone number is enough, thanks), no hidden ad engine, no hundreds of stupid features that product people stuff down in the name of experimentation.

There are some leaders who want anything and everything: Throw 100 darts in the wall. Leave what sticks (don't touch or nurture it). Go back to throwing 100 more. The other guy does that? We want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different affiliate channels? Yes. Custom features for B2B partners? Yes. Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user profile management systems? Yes. Yes, yes, yes, every requirement is a go.

The crucial difference is: What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step – just so that more quantity of features can be seen on paper. This is how most startups are, except for those rare few which don't "talk" about focusing, but actually focus.

chpmrc• 5 years ago

no hundreds of stupid features that product people stuff down in the name of experimentation.

Amen to that!

tinus_hn• 5 years ago

Ultimately though WhatsApp ended up with a billion users and 0 revenue.

hermanradtke• 5 years ago

They charged $1 per year before Facebook removed the fee.

rglullis2• 5 years ago

Not true. They were charging $1/year from users in the richer markets. They could easily reach hundreds of millions per year in revenue if they followed this model.

bryans• 5 years ago

Except for that $19 billion exit. Companies don't need to be turning a profit anymore -- they just need to prove they can scale, and someone else will figure out how to monetize it after a buyout. Hell, you don't even need to be a startup aiming for a buyout anymore to get away with being financially incompetent. Reddit is about to file their IPO at a valuation of $10 billion, even though they've been losing money for nearly two decades and are noticeably (and unsuccessfully) scrambling to find any presentable evidence that they're not complete buffoons.

thruway516• 5 years ago

A billion users IS a goldmine. Ask Google, facebook, twitter, reddit. None of these companies had any significant revenue for the first couple years.

ryanSrich2• 5 years ago

This is why, after being both an engineer and a product manager (and VP of Product), and now a CTO of a seed stage startup, I’m fairly convinced PdMs are unnecessary. At least for the the type of company I’d like to build (WhatsApp as a primary example of that).

Off loading the founders vision to someone else is a failure of both communication and org structure (at the seed-SB stage). People may not think hiring their first PdM is offloading their vision, but it is. That person is going to come up with their own ideas, validate their own assumptions, and push for product updates that directly serve their own self interests. To think that this won’t happen is naive.

What I’ve found works the best is hiring competent engineers, giving them adequate business context, a decent design spec, and letting them figure it out.

nickthemagicman2• 5 years ago

> adequate business context, a decent design spec, and letting them figure it out.

I wish more PM's thought like you.

015a• 5 years ago

This is always a controversial opinion, at least in some circles, but: I've long said that the PM role was created solely because companies can't find enough good engineering talent.

That isn't to assert that PMs don't do fantastic, necessary work. More-so that there's a more ideal reality where the engineers should be doing that work; the outcomes, in my experience, are better in orgs where engineering is not just influential in, but responsible for, customers, business outcomes, product planning, etc.

This isn't always possible. Most orgs don't resemble Whatsapp; they resemble Google. A hundred product divisions building a thousand different things. Its already difficult to hire engineering talent to fill an org like this, let alone the additional number you'd have to hire in order to account for the fact that a double-digit percentage of an engineer's role should be PM-like stuff.

So, throw away idealism; outsource that part of the engineer role to another specialty which is easier to hire for: PM. Its reasonable.

But the issue has arisen in that industry has forgotten than this was a necessary sacrifice; its not desirable. We've now got university and bootcamp programs dedicated to training for PM roles. Many companies have entrenched the idea that every team/org needs a PM. Tons of people are now adding financial momentum to this idea that their skillset is valuable and necessary; the ship can't be turned so easily.

At the end of the day, this is a disservice to many engineers, as their people, product, and business skills fall out of practice and they're further relegated into code-monkey positions. Its actually a disservice to PMs as well; they could have been trained in engineering and unlocked far more professional mobility. In the worst cases, it leads to PMs resembling the Office Space "what is it, exactly, you do here" guy ("I bring the specs from the client, to the engineers!" "Then, why don't the engineers just talk to the clients?" "Well, they're not people people!" mhm)

twotwotwo03• 5 years ago

I can attest Product Managers were the downfall of one startup I worked at. The company had dozens of millions in funding and managed to piss it all away with ill-founded escapades driven by a handful of Product Managers. If anything, the Product Managers seemed to be managing their careers, not the product. They babied engineers until most of the engineers stopped caring about the product and just gave up working hard at all. All the brilliant ideas were trite insights you could read on two year old Gartner reports -- "release a cloud version".

CEO and CTO beware, do not let Product Managers happen to your company.

austincheney• 5 years ago

This confuses the mistakes of a mature company. Different dynamics.

For example Google had thousands of engineers back when they had only 3 products: search, auction, and AdWords. I remember Travelocity having over a thousand developers and not being able to redesign a webpage.

Its deeper than a product problem. Its an execution problem. It’s the ability to do what works with great criticality opposed to what’s expected, standard, normal, or comforting.

that_guy_iain3• 5 years ago

WhatsApp was originally a super simple product. Send and receive messages. It was fundamentally a solved problem. It was all about the fact you could send free messages in a world where sms cost 10 cents a message. It wasn't that their messages were faster, their messages were anything other than free.

They had a year fee of $1. Which I never paid yet I used their service anyways. The reason they were able to scale so large with so few devs is simple, their product was super simple. 50 tech people to build a messaging system is actually a lot.

Jensson6• 5 years ago

The reason they were able to scale so large with so few devs is simple, their product was super simple. 50 tech people to build a messaging system is actually a lot.

Tell that to Twitter. Whatsapp staying at 50 instead of growing to 5000 is impressive.

stiltzkin• 5 years ago

How things would have work around if Microsoft released MSN Messenger for Android and iOS. They had the userbase and the brand already.

daanlo• 5 years ago

In my recollection it wasn‘t about saving 10 cents in the beginning, but about saving 1-2$ per message, since one of the first use cases it enabled was free international texting. That was a HUGE benefit. All the first evangelists I knew were in long distance relationships ;)

ehPReth• 5 years ago

no iPad app either :/

anshumankmr• 5 years ago

The WABetaInfo account on Twitter said one is finally coming. And I think it seems likely considering we don't need the phone to be on while using Whatsapp Web.

amelius3• 5 years ago

Uh yeah, but it was only possible to have such a simple product because they reached mass adoption. Without network effects, you've got to differentiate and thus add complexity to gain users.

Spivak• 5 years ago

But then how did they reach mass adoption in the first place? It’s not like they were a Swiss Army knife and before and stripped features once they got big.

a_c• 5 years ago

There was an article about when do startup brings in their first "product manager". Before bringing in "product people", the founders are the product people. I see it as the inflection point when the founder start delegating the product making to others. This is when the organization started scaling.

Doing one thing good is a thing in the past. Venture capital has pushed basically all start-ups into 100-feature shops

def_true_false4• 5 years ago

That's all nice and well, but some of your examples are not things that save effort -- using phone numbers is surely better for network effects, but not in any way simpler then having regular account registration, unless you just completely ignore security.

WA apps are not better than the competition. Basic stuff like a way to disable automatic downloading of media for groups you are in is still missing. No desktop app. The web frontend is still crap. Outsourcing account registration to mobile phone companies is ridiculous (same as Signal) -- in most of the world, it makes it pretty much impossible to preserve your privacy while using WA. Endless nagging about plain text cloud backups makes WA E2E encryption completely worthless, since even if you opt-out, 99% of your contacts will have them enabled.

The impressive part, considering the size of their team, was their backend stuff and scale.

Discord, with similar team size, was able to build something that succeeded in displacing IRC. (Their Android app was laughably bad for the longest time as well.)

Zababa4• 5 years ago

Discord, with similar team size, was able to build something that succeeded in displacing IRC

I think they displaced Skype, mumble and teamspeak too.

pessimizer• 5 years ago

Whatsapp:Erlang as Discord:Elixir. The effectiveness of the teams might more be the effectiveness of OTP/BEAM for network applications.

numair4• 5 years ago

It’s comical that all of the replies to what you’ve written claim you’re wrong, because you’re actually 100% right. And this is even more true when you remember that the company had fewer than 20 people for a long time, and that there was only one real goal, that seems to be forgotten: they needed to clone BlackBerry Messenger and get it working on every platform possible. That was it. That’s all they needed to do, and that’s really all that they did.

I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.

math-dev3• 5 years ago

Nice story. What are you doing now?

dleslie4• 5 years ago

Also can't overlook that for many SMS was ludicrously overpriced, but the early basic smartphones (palm and symbian) often had wifi capability; WhatsApp provided secure, ludicrously cheap instant messaging across all meaningful handsets.

And that's all it had to do; the telecoms weren't interested in making SMS cheap or secure, RIM didn't want BBM on anything but their devices, ICQ/AIM/MSN/etc were similarly unavailable.

Scoundreller5• 5 years ago

Explains how they doubled down on hardware and lost.

BB’s market cap is currently a bit over 1/3rd of whatsapp’s acquisition price. And BB has minimal long term debt.

908B64B1972• 5 years ago

I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.

I remember selling every BlackBerry stock I had when I got my hands on their touchscreen phone (the Storm was it?).

3 years behind the original iPhone, while Apple had just launched the 3G with the App Store. Guess which phone (and stock) I purchased then?

Was there really no-one at BlackBerry following what was going on in the valley at that time?

i_have_an_idea• 5 years ago

I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.

Well, he had a point. What killed BB was the iPhone, not that their chat app was going to lose out to the competition. They simply had a product that was inferior in every single important way.

6sp• 5 years ago

Mike might have taken longer to get it than Jim but he eventually understood the existential threat they faced. He realized this at a time they could have possibly changed course but was unable to do so because the board did not have faith in such a drastic change. The plan to cannibalize device sales today vs promise of future software revenue did not sound so promising for the board, who I imagine were interested only quarterly profits and their stock price.

ryandrake• 5 years ago

I've always called this Feature Cram, and it's standard procedure at basically every company I've ever worked, small and large, B2B or B2C. Nobody is willing to say "Our software is a success and it's done. Let's fix the major bugs, put it into maintenance mode and start a new, separate, great idea!" Nope, instead it's always "OK the MVP is going well. Let's jam ten other unrelated things into it and see what takes off and what are the dogs (and be stuck with the dog features as technical debt forever). Then, when none of them reach the success of the original actually solid idea, they hire moar people and jam 40 more feature ideas in. Eventually, the original great thing that brought the company success is so hard to find and use because of the weight of the other crap, that a new upstart that Just Does One Thing Well comes in and eats their lunch.

Then, at that new upstart, someone asks "What else can we bolt on to this thing to make it really grow??"

bobthepanda• 5 years ago

This may very well be unpopular, but this is what happens when you pay someone in measures tied directly to share price, or even more directly in shares; they have every incentive to do whatever it takes to juice the share price up. VC money has even more extreme growth expectations.

Closi3• 5 years ago

Yes, although there is a balance - lest we forget that you couldn’t use WhatsApp on your computer unless your phone was turned on until a month ago, and there is still no iPad app.

I think WhatsApp is in the place where it is so ubiquitous in some social circles that it faces less competitive pressures than other apps and can afford some of these pretty large gaps.

tomnipotent2• 5 years ago

Our software is a success and it's done

Because no such state exists. Imagine if Photoshop stopped with updates twenty years ago, or even five.

onlyrealcuzzo• 5 years ago

So is SpaceX a low quality idea? Some problems are just not feasibly solved by a handful of people. Does that make them low quality ideas?

wpietri• 5 years ago

This is a great point, but I'd add that the problem isn't experimentation. Neither is it having a lot of ideas. It's that they aren't using good experiments to kill off most of the ideas.

I think of it in terms of divergence and convergence. Divergent thinking is great. If you think of an army on the march, leaders will be thinking about possible paths and sending out scouts. But when the scouts (experiments) come back with reports (results) they don't just send some soldiers along every path that looks promising. They winnow the options down, converging their thinking.

Bad leaders don't have the discipline for that. And I think a lot of them pick up that habit from sales and/or seeking investment. "What will it do? What won't it do!" That's great for manipulating the feelings of the person you're currently talking to so they'll sign the deal. But it's terrible for running an effective, focused product development organization.

I know I've seen companies die like that. And it's probably way more than we know, because as WhatsApp demonstrates, focusing hard on what's working is a great way to succeed well enough that people study and talk about you later.

toast06• 5 years ago

They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step

This is the key mistake. Having teams of 2-3 with a manager each is way too many managers. If you have a clear focus, small teams are great because they can work autonomously, with little management and good results. You can't hire your way out of coordination hell.

I was at WhatsApp between 2011 and 2019; for the most part early on, we had essentially the CEO who does a mix of product management and engineering management and engineering, an actively engineering server manager (who also did some client work), and a client engineering manager. Android team grew enough to have an engineering manager and we also ended up with one product manager. Post aquisition, staff grew and so did management.

MomoXenosaga2• 5 years ago

Post acquisition WhatsApp had to start making money.

For all the doom and gloom Facebook is still too scared of putting ads in there.

SavantIdiot• 5 years ago

Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers.

Part 1: By definition: more engineers = more ideas.

Part 2: Not so much.

On the one hand, mediocre engineers are sometimes sequestered into small teams where they can't interfere with the important projects that all the smart people and other resources are being thrown at. They just sit there and churn out crappy ideas and crappy non-vital tools. Like daily indicators.

On the other hand, I can see that as more of a tautology: ideas come from one person and are grown by a team. So be definition a quality idea comes from one person, and turns into a team.

tomc1985• 5 years ago

I really don't like working for those 100 darts people. They are a royal pain in the ass.

Much better are the folks with a solid vision who can drive folks to execute on it, successfully.

ksec• 5 years ago

WhatsApp success is not about what WhatsApp did, it's about everything they didn't do.

I wrote something similar [1] about its success not long ago on HN,

WhatApp grew because it was the only IM that worked on Smartphone. Thanks to Erlang. Yes. ICQ didn't bother. Microsoft refuse to accept Smartphone is the future of computing hence there never was MSN on Smartphone. Other contender tried but none of them managed to Scale. I still remember all my colleagues and friends used to download instant messenger on our phone to test them out, creating a group and basically DDoS each other until the system or network crash. No IM was able to handle the traffic load, not only WhatsApp did it many times better than their competitors, they also had almost all platform support, from Symbian to Blackberry.

There is another part of the story that dont get much appreciation, is how they managed 2 million connection per box on a what is now a relatively slow server. The latest M1 Max would have been a lot faster. I often wonder if they could have pushed to 10 Million connection per box with same latency under current CPU and Memory. ( Not saying it is a good idea. )

[1] https://news.ycombinator.com/item?id=25984924

papito4• 5 years ago

They also famously had no meetings, like at all.

Scoundreller5• 5 years ago

Like, 5 engineers never huddled together to figure out how to solve a problem?

pards• 5 years ago

What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step

1,000 times this.

HeavyStorm• 5 years ago

you're perfectly right

I get scared every time that senior leadership pushes a new member on the team, because I know that overall quality will drop; their view is completely related to how much things they want to done on every cycle, and they don't care about quality (until it blows up in their face)

didibus• 5 years ago

I think there is something else that I find hard to explain, but it stems from the same vein.

The best way I can call it is the impedence mismatch between what business people imagine and envision, and what the real products, their code, their systems are actually doing and are actually designed to do.

It creates a kind of cost explosion in the velocity needed and it is often solved by throwing more developers at it instead of rethinking the actual design of the products so that they can better grow to support additional use cases and scale.

You can't totally call it a hack, but it kind of is. A system was designed in a way that doesn't scale to the user demand and to the additional features or the new abstractions that it wants to grow into. Instead of rethinking holistically about how that system should be re-designed to now encompass all of this new stuff, and then put the work into changing it from its current model to that new one. What people do is they break up the current system in two, so now you have two service where you had one before. And now they hire a new team of engineers, manager, PM, etc. to start working on this new half. This repeats again where each of those two half are split in half again, hiring a new team, now there are four teams and four service, etc.

As that happens, you can no longer redesign the way the systems fundamentally work, so each team accrues more and more complicated workarounds on their individual pieces and all new feature becomes cross team projects where you now need someone to coordinate the work between all teams for each little project.

Once you start down this path, there is no way out, the only solution becomes throwing even more teams at it, breaking things down again and again into smaller pieces with a whole team behind it as each piece becomes more and more complex to all work around the inherent holistic limitations that were never reconsidered.

It's a vicious cycle, the piece cannot grow in it's current state because it is too complex, break it up, have two teams tackle each half, now they can take the complexity until they themselves add more stuff to it which makes them too complex once more, break it again, add another team, rinse and repeat.

The issue is often that complexity is accidental, due to the design in place and the tech dept accrued till now, as opposed to being inherent to the domain problem. But instead of addressing the accidental complexity, it is simply broken down in two so we can throw more people behind taming it.

jeltz• 5 years ago

As an end user I think WhatsApp is a terrible product for some the reasons you mentioned (tied to phone number, no desktop client) but I cannot deny that it was the smart thing to do from a business perspective.

demarq• 5 years ago

this

bennysomething• 5 years ago

And then I look at Dropbox and wonder why they ve made the product into a bloated mess that can't get some basic things right.

wly_cdgr2• 5 years ago

Correct

thrower123• 5 years ago

Fifty engineers almost sounds like too many, if you aren't using most of them to develop internal frameworks and reinvent wheels.

You'd have an Android team, an iOS team, a couple teams working on core APIs and infrastructure. Ideal team size is about five.

tiffanyh• 5 years ago

DuckDuckGo does around 35B searches per year with 160 employees.

https://duckduckgo.com/traffic

Not the same efficiency but it goes to show how certain domains and technology stacks can scale exceedingly well. Plus in DDG case I believe they outsource a lot of their tech to 3rd parties.

mikewarot• 5 years ago

The users are all handled through a common set of code, I don't see why its remarkable that it takes 50 engineers to keep things going. Customer support, I can see that eating manpower, but not keeping the code running on a few racks of servers.

short12• 5 years ago

That sounds about right, even slim

I've honestly never understood how faang companies can have such incredible numbers of devs when the products basically move at a snails pace for years now

Pensacola• 5 years ago

Only? Sound like plenty.

steve_adams_863• 5 years ago

It crossed my mind while considering this that a figure like this doesn’t mean the same as it would have pre-cloud infrastructure.

My team is fairly experienced but not so much in infrastructure. Even so, we build and deploy on Google cloud with fairly excellent scalability and ease. Ten years or longer ago I suspect the story would be different - the infrastructure and scale constraints could have pinned us at times. We would have needed to hire more to support that effort.

And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for.

Not to say this isn’t impressive still, I think it’s a great thing and worth aspiring to. I love small teams.

Jensson6• 5 years ago

Whatsapp used bare metal servers. It isn't hard to do, just that people no longer learn to do it.

Ten years or longer ago I suspect the story would be different

Whatsapp was built 12 years ago.

steve_adams_863• 5 years ago

I assumed they did their largest scale ups more recently, though. I could be wrong!

toast06• 5 years ago

And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for.

I was at WhatsApp from 2011 to 2019.

Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway.

Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno.

We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource).

What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly?

We self-hosted our code repository and bug tracker.

steve_adams_863• 5 years ago

I hope I didn't come across as though I was criticizing at all. I meant to point out that having 50 engineers build something today means something different than it would have a decade ago, although in retrospect my age/career are getting away from me and 15-20 years ago would have been a much better example.

I should emphasize that the accomplishment is incredible, regardless of how much we outsource things today. I know you didn't outsource crucial talent or the ability to deliver a good core product; you can't do that as far as I know.

I apologize if it seemed like I was putting down the accomplishment. I'm only meaning to reflect on how much more technical and infrastructure work happens outside of our teams these days than it has in the past. Perhaps though it isn't how much _more_ happens, but how _different_ what happens is. Building these products has changed a lot in some ways.

Edit: Also thanks for overview, that was a cool cursory look at how you operated.

dang• 5 years ago

Discussed in the past:

Why WhatsApp Only Needs 50 Engineers for Its 900M Users - https://news.ycombinator.com/item?id=10225096 - Sept 2015 (248 comments)

johnisgood• 5 years ago

I could scale to billions of users as a one-man engineer, but I would need a marketing team. :D What I am trying to say is that I could make a WhatsApp-equivalent relatively quickly, alone, yes, but that is definitely not enough to gain billions of users.

MomoXenosaga2• 5 years ago

They were at the right place at the right time. But people want to believe it is tech not luck...

There is a webshop in my country that copied Amazon in the early 2000s and today they are beating Amazon.

JoeCoo7• 5 years ago

WhatsApp has to be one of those most easily scalable projects to work on. You have the freedom to divide chatrooms across different databases/shards. Same goes for notifications, where users can be divided since they're all isolated. The hardest part is just coordinating all the moving pieces in a way that can grow well, and that mostly just involves avoiding pitfalls in your design that might bottleneck.

throwaway1777• 5 years ago

And now WhatsApp has hundreds of engineers. Take that to mean whatever you want.

seoaeu• 5 years ago

It is silly to focus on number of engineers rather than on total cost. Any company should keep hiring more engineers until the additional salary expense exceeds the decreased operating cost/increased revenue that engineer generates

28194608• 5 years ago

Dear whatsapp devs please disable the feature which shows how much time and when to EVERYONE in the world. TY.

heywherelogingo• 5 years ago

This site is getting quite repetitive. I think the balance of News to things of interest has gotten quite out of whack. This article is not news, and is also in recent memory so not an interesting re-visit. Is HN average age decreasing? The content character has changed in recent years. Something's changed/changing.

akkpxl• 5 years ago

I thought I was the only one who felt this way. Almost every day I see the same "style" of articles that isn't news or that interesting. One thing I've noticed is almost every day I see an article explaining why remote work is better and/or how tech interviews are broken, both of which are popular opinions a lot of people on this site share. Maybe it's just me though

bobberkarl• 5 years ago

Started visiting hn 4/5 years ago, i agree. The content/discussion quality decreased substantially the past 2 years.

mkl95• 5 years ago

Ultimately, how many engineers a company has is not that relevant when it comes to scaling.

A small team of product-oriented engineers with a devops mentality can easily develop, maintain, and continuously deploy a considerably complex application.

A similarly complex project in the hands of a larger sales driven company often resembles a spaghetti ball, and is held together with rotting shell scripts and unproductive manual processes.

chatman• 5 years ago

This is misleading. Likely, those 50 engineers were standing on the shoulders of the giants, i.e. those open source maintainers who build wonderful value for everyone out there.

ratww4• 5 years ago

Does this really matter? Companies with 100, 500, 1000, 5000, etc employees often stand exactly in the same position and use the same kinds of tools.

jokethrowaway2• 5 years ago

While what you says about focus is important, I've seen plenty of focused startups die.

WhatsApp had success because they executed well enough at the right time.

I hate WhatsApp with passion because of its limits and that horrible web interface that needs your phone connected in order to work.

I know plenty of people (me included) who stopped talking with some groups of people because they didn't tolerate WhatsApp.

It's mediocre software at best, but because of the network effect and first mover advantage, that was enough.

WhatsApp scaled to 1B users with only 50 engineers

Related Stories