Facebook’s code quality problem (2015)

https://www.darkcoding.net/software/facebooks-code-quality-problem/

by setra • 9 years ago

502 226 9 years ago

neals• 9 years ago

[2015]

As pointed out this is from 2015 but I'd love to hear some updated. Has any Facebook employee created newer presentations that discussed, say, their 18,000 iOS classes and the fact that in a single week 429 people contributed to the same iOS app?

I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim.

arjie• 9 years ago

I'd bet that the 18000 classes is just Thrift-generated stuff or whatever.

panic3• 9 years ago

You don't have to bet, you can see them here: http://quellish.tumblr.com/post/126712999812/how-on-earth-th...

icebraining3• 9 years ago

I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim.

Isn't this an inevitable outcome once your systems are actually reliable? I've been reading the Google SRE book and they make the same point: every new feature (read: change) is risky. Hence they delay deployments once their "error budget" starts running low.

Seems to me that the only systems that don't work more reliably when the engineers are out are the ones which require hands on deck just to keep them running.

dep_b• 9 years ago

code quality can't handle our scale

userbinator• 9 years ago

It is ironic that a codebase with so many classes is likely to occur precisely because of dogmatic adherence to those "best practices" which were originally intended to improve code quality --- aggressive refactoring/modularisation being a likely culprit. What I think they really need is not more design, not more architecture, not more of anything but KISS and YAGNI.

afarrell• 9 years ago

The thing that having an overall architecture buys you is conceptual clarity. That lets someone much more easily understand what is going on. You need to have a minimal set of ideas that are well thought out and explained.

Just saying "YAGNI" and "KISS" doesn't actually make the resulting code simpler. Why? You do have some things to build and refusing to lay out a plan for what that is going to be doesn't stop that. It just means that the complexity doesn't have a clear narrative

sidlls• 9 years ago

It seems to me your point about KISS and the author's observations are compatible.

Facebook is successful for reasons that have almost nothing to do with their code, however, and as long as that's true there isn't a reason for them to change how they develop their software. In that sense, code quality doesn't actually matter (at least, until it does).

sushobhan• 9 years ago

I don't think all the things discussed are still relevant as it's a fairly old post. Also, "best practices" are not always best depending on the scalability of the problem itself.

_pmf_2• 9 years ago

If a company hails React as a sane way to develop software, you know where it stands.

peletiah2• 9 years ago

Just trolling or could you argue that?

qaq• 9 years ago

OK so what should have they used instead?

sz4kerto• 9 years ago

It's interesting to note though that they have code quality problems but that doesn't prevent them from being massively successful.

bsaul• 9 years ago

It's a very hard to swallow pill for engineers, but many (if not most) startup or IT company success or failure is very independant of technological matters.

Once you get to "average" quality, then business and product decisions become far more important than anything else.

Gracana• 9 years ago

And it only needs to be average, or approaching average, in your niche. There's a lot of truly awful enterprise software out there, and it gets sold on promises of features and support, not quality.

I wish software engineering had the barriers and requirements of other engineering fields. I deal with industrial control systems, and we have the national electric code, UL, CE, NFPA 79, etc. With all that you can still make a machine that doesn't work well, but at least it's going to conform to certain conventions to make installation and maintenance easier, it will have a robust safety system, and it will be well-documented. Most fields of software engineering have no such minimum standards.

rumcajz3• 9 years ago

I believe "our site works when the engineers go on holiday" thing is not fair. Of course that application is less stable when it's actively modified. There's no way to make it more reliable on weekdays than on weekends except for stopping the development altogether or maybe deploying on weekends.

panic3• 9 years ago

What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"? There's always a way to improve reliability. Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions.

The problem isn't that it's impossible -- it's that it's more expensive than just hiring one more engineer to keep papering over the problems.

icebraining3• 9 years ago

What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"?

All of them. Are your roads more reliable when they're constantly being changed or when they are just being maintained? Is NASA achieving its reliability by constantly changing the designs of their ships, or by reusing the same design over and over?

Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions.

Yes. But a fixed formally verified system will still be more reliable than a formally verified system being constantly changed.

What was said wasn't that FB couldn't be more reliable. It's that they are already so reliable that only new changes introduce problems. Sure you can still work on minimizing those problems, but that's a different point.

dsr_• 9 years ago

No way at all? FB has a lot of servers and a lot of users. They have opportunities for quality practices that few other organizations get.

Here's a simple example:

Split the user population into 365* groups. Assign them to days of the year. Test new code only the group whose day has come up. Follow them until they stop having problems. Now you can deploy that to a month's worth of groups. All good? Deploy to everyone.

Yes, that means that you can't have more than 365 changes in simultaneous development. Tough.

*Yes, yes, leap years. Take a day off from deploying.

byroot• 9 years ago

They graph the number of incidents. Even if a bad release impact only 0.3% of the user base, it's still an incident.

They have to investigate it, revert or fix the bad code and start the deployment process again.

rumcajz3• 9 years ago

That's deploying on weekends, isn't it?

rb2k_• 9 years ago

That is usually how all companies at that scale release code (at least the ones that I know of).

Only that it's not 365 groups, because at the size of FB that would be several million people.

eh78ssxv2f2• 9 years ago

They might already be doing A/B testing (or in your example A1/A2/.../A365). It's not clear what's the definition of an "incident". At my workplace, even if a bad code push affects say 0.1% of users, it would be classified as an incident.

aamederen• 9 years ago

I believe, from time to time, you just need to stop developing new stuff for a period of time and dedicate the team to improve the overall quality, pay technical depts, hunt bugs, remove legacy code and refactor still existing ones.

muse900• 9 years ago

I agree with you. This is one of the main issues small-medium scale companies I've worked for have, they just have a dev team that has to keep on creating stuff constantly.

Although I've never worked on a company with the size of FB, and I believe the same logic wouldn't apply to them. They have to keep creating stuff, as fb still has a lot of potential on becoming way more profitable.

icebraining3• 9 years ago

Sounds essentially like GC. And like in GCs, it seems preferable to have pause-free processes than stop-the-world.

martinknafve• 9 years ago

Why not plan tasks like that continously and make it part of your normal iterations?

Cthulhu_• 9 years ago

Even so, with an organization the size of Facebook, it'd still cause a lot of churn. I don't know if there's any hierarchy there, like, lead developers and architects that can align architecture and maintain quality and consistency.

maxlybbert• 9 years ago

I worked at a place where non-technical management didn't like hearing things had to be refactored occasionally. So the technical managers became very good at including refactoring of old code in estimates for new (related) features.

This generally worked, except when a lot of refactoring was needed. Occasionally, we would learn after the fact that one of our designs, programs, etc. was fundamentally flawed in some way, but we couldn't fix it with a series of moderate changes. Those cases really need a dedicated refactoring effort.

onion2k• 9 years ago

And fortunately your competitors are always happy to pause and wait while you do that.

peletiah2• 9 years ago

Makes me wonder how their decision for an open-plan office is related with this...

amelius• 9 years ago

It also seems that improvements in code quality caused by React happened not because they were driven by management, but because a few "heroic" engineers thought it was an interesting side-project. At least, that's my impression from the publicly available talks they gave.

_pmf_2• 9 years ago

The article moves on, without wondering whether releases regularly breaking your app are a normal part of the software engineering process.

To be fair, that's absolutely their call to make. Nothing of value is ever lost if your platform does not provide any non-ephemeral value.

paganel2• 9 years ago

I read Richard Gabriel's "Worse Is Better" essay in my first month hired as a professional programmer, that was almost 12 years ago. Since that time the lessons of that essay have only increased in significance for me, with this FB story as additional proof.

Their back-end code might be bollocks, and I can certainly believe that judging by how sluggish their FB app feels on my phone, but the fact is that they've conquered the Internet (together with Google and a couple of other companies). It's a fact that I personally hate, but they're still winners in the end.

aaron-lebo3• 9 years ago

But it's not better, it is worse as clearly put forward by this article. ;)

We shouldn't attribute Facebook's success to poor quality code, unless we really believe that 429 people working that that slow sluggish app is productive. Do you?

Maybe Facebook would be in a better situation than they are if they had a firm foundation? What happens when a competitor comes along and knocks on their door? That mess of code becomes an anchor.

They succeeded not because "worse is better" but because ...they succeeded, maybe because they had the greatest social network in the world to expand within, or because of some other reason we don't know. But they didn't succeed because of worse is better, they succeeded in spite of it.

paganel2• 9 years ago

Do you?

It's written between the article's lines, so to speak, but it seems like the bad code quality was generated in part by FB moving at a very fast pace. I think they wouldn't have been as successful as they are had they chosen a slower path with more "correct" code and architecture.

What happens when a competitor comes along and knocks on their door?

They successfully managed to "eat" and integrate both Instagram and WahtsApp, both worthy contenders at the time, and they're now on their way to stealing Snap's lunch, so they've proved to be pretty resilient on this front. Not to forget how they successfully kept Google+ in check.

jheriko2• 9 years ago

how would that success have been different if they had used different engineering practices?

a big part of their success was delivering results... if they hadn't quickly crapped everything out of the door, how would facebook look today?

jaredklewis• 9 years ago

No one is actually arguing that worse is better. It's just a catchy title. A more accurate title would be "worse is faster than better, and being fast but mediocre is better than being slow and amazing."

Worse is better makes the observation that doing right thing slows you down. Often, an ok solution is much quicker to get finished than a great one. And it concludes that the projects that prioritize getting stuff done over code quality will suceeed.

Would facebook be as successful as it is today if it had taken longer to ship features, but maintained higher code quality? Who knows.

But the intuition in worse is better, which matches with my own experiences, is that making that trade off would have cost Facebook, as other faster moving projects closed the gap.

onewaystreet• 9 years ago

Agility trumps code quality. Stability can be addressed downstream (which Facebook does with heavy testing and rollouts of changes). Code quality problems or not, Facebook has been more stable than its competitors (MySpace, Twitter, etc.) and faster at pushing out changes.

jasode4• 9 years ago

I read Richard Gabriel's "Worse Is Better" essay

Fyi to help set the record straight... RG's "Worse Is Better" is not shorthand for "worse quality" is better than "better quality".

Richard Gabriel was trolling readers with the word "worse" for comical effect. His essay is: "simpler with less features" wins in the marketplace of ideas more than "complex". His observation isn't about "quality".

That said, there is another meme with the exact same 3 words called "worse is better" which does stand for "worse _quality_ is better than better _quality_" but that's not related to RG. It's just the more common interpretation of those 3 words -- especially for 99% of people that are not familiar with RG's original essay.

tldr: Richard Gabriel trolled people so hard that "worse is better" has taken on a life of its own divorced from his original thesis

coldtea5• 9 years ago

Richard Gabriel was trolling readers with the word "worse" for comical effect. His essay is: "simpler with less features" wins in the marketplace of ideas more than "complex". His observation isn't about "quality".

Only it IS about quality.

The simpler and with less features" examples he gave were examples of broken implementations, that will succesful in getting "out of the door" and having viral acceptance, have resulted on lots of issues, lost man-months, and piles upon piles of hacks.

That's regardless of what Gabriels himself thought he was saying.

And while "getting out of the door" pronto might be good for a product, it's not good for infrastructure software.

emodendroket9• 9 years ago

Richard Gabriel was trolling readers with the word "worse" for comical effect. His essay is: "simpler with less features" wins in the marketplace of ideas more than "complex". His observation isn't about "quality".

I have to say that reading the thing again I don't really think that's accurate, since he also talks about sacrificing correctness for the sake of ease of implementation.

ebbv3• 9 years ago

You're making an implicit assumption in this comment; that Facebook is in an optimal position and every decision they made contributed to that optimal position in a positive way and so any change is for the worse.

But what if they aren't in an optimal position? What if having a higher quality code base all along would have let them dominate social media more and faster, and perhaps be more successful in areas they have stumbled in the past?

A lot of people (especially non-technical folks and inexperienced coders) seem to think that code quality comes at a cost and that a bad code base full of technical debt is just a side effect of fast iteration and the cost of being quick to market. I say that's crap. A higher quality codebase with less technical debt actually improves speed tremendously because it lets you iterate faster with fewer rewrites down the road and it will be more stable and result in fewer distractions from having to put out fires with unforeseen bugs making it into production.

A sloppy codebase with technical debt is much more costly on your ability to go to market and add features and iterate.

emodendroket9• 9 years ago

But what if they aren't in an optimal position? What if having a higher quality code base all along would have let them dominate social media more and faster, and perhaps be more successful in areas they have stumbled in the past?

Well, I don't know. What if spending so much time on quality would have led to it never catching on? It is curious that the world is full of compromises like Unix, QWERTY, C, VHS, etc., all of which had arguably superior competitors that failed to catch on.

segmondy• 9 years ago

So?

I've become very cynical since the past year or so because there is a lot of noise/articles on the net, all knowing what's best for you or wrong with what you are doing. Yet without context or true authority to talk about it. My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.

Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case. Startup principles don't apply nor academia. Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars.

aaron-lebo3• 9 years ago

Isn't that an appeal to authority? Because someone hasn't done X they don't have the right to criticize X?

Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises.

But does it have to be this way?

evgen3• 9 years ago

Until someone provided s counter-example the assumption is that yes, this is just what happens. We have entire industries of competing methodologies, some almost cult-like in their slavish devotion to "the one true process", and none have birthed any great successes. Why is that?

watwut5• 9 years ago

You have right to criticize, but don't expect your criticism to be taken too seriously. There are two reasons for it: a.) you don't have frame of reference to compare whether facebooks result is better or worst then other companies b.) you don't know what tradeoff are necessary for making it work nor what it takes to achieve success.

emodendroket9• 9 years ago

Well, sure, they can. But don't you think someone who has been involved in a massive company would have more trenchant insights into massive companies?

feedjoelpie2• 9 years ago

Appeal to authority is not a logical fallacy. It's a valid argument unless said authority is false in some way.

thehardsphere2• 9 years ago

It's ad hominem; the author is being attacked to invalidate the argument, instead of attacking the argument itself. Which is never valid.

Appeal to authority usually takes the form of skipping making an actual argument for something and instead resting upon "X said so" and not explaining further. This is also not valid but is sometimes a shorthand for saying "X made argument Y," which can be valid if argument Y is valid (in which case the fact that X made the argument is purely incidental trivia).

ycombinete• 9 years ago

Hold forth = To talk at great length.

Have you used it as such?

yladiz• 9 years ago

Is "hold forth" an older or less common phrase? I don't know if I've ever run into it, and I wouldn't have guessed initially that is its definition.

ebbv3• 9 years ago

This is an absurd leap of logic. There's many quality programmers who have never worked at a large company and many more terrible programmers who have worked at large companies.

Anyone can share their opinion and regardless of where they have worked it may or may not be valuable. If you are judging value based on the market CAP of their LinkedIn profile then you are probably missing out on most good information out there.

randomnumber314• 9 years ago

So?

So, there's a huge lesson when an organization starts blaming every one else for scale problems; if everywhere you go there are issues, it's you.

stkni• 9 years ago

> My question these days when I read such article, "What has the author done that is equivalent?"

Well I happen to (vaguely) know the author and have worked with him in the past and he has done a lot of great work, not all of it in the public domain though.

Comparing him with an organisation that employs 17k people is a little unfair but he comes out of it favourably IMHO!

coldtea5• 9 years ago

My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.

The thing about critique is that you don't need to have made an equivalent project/effort/artwork/whatever to be able to critique it with a valid objection.

You just need to be able to spot an actual issue.

Sure, some might think they've spotted an issue when there is none. Or not understand that the issue exists for a reason (e.g. engineering tradeoff).

But those are orthogonal to the subject.

The key is that critique can come from any source, and it doesn't need someone to have done something "equivalent" or to have specific credentials to be valid.

The latter can at best serve as a rough filter to throw out some noise. But they will throw a lot of valid critique as well, which can come from everybody, including a layman.

morley• 9 years ago

You just need to be able to spot an actual issue.

My problem with the article is that it's extremely hard for humans in general, and critics in particular, to accurately model the alternative. I think that's the heart of what the commenter is talking about.

Unless you've worked at Facebook's scale, it's really difficult to tell what happens. (I don't work anywhere near that scale, so maybe take my opinion with a grain of salt.) This critic is reading signals and attributing them to a failure on Facebook's part, but who knows if an alternate scenario (a) actually makes things better, or just results in different trade-offs; or (b) is even feasible for a company of Facebook's size to do. I have no idea. There are only a handful Facebook-sized enterprises that have ever existed.

bmelton• 9 years ago

To OP's point, at what point do we consider the Chesterton's fence effect? Pointing out code quality problems is of course easy when the effects of those quality problems are obvious, but as OP mentioned, Facebook is working at a scale that isn't likely familiar to most critics. Sometimes, when things get big enough, you have to do things that nobody else is doing, because nobody else has the problems that you have.

If all you've ever built was houses, it's hard to give too much credence to your critique of a skyscraper, especially if you aren't privy to the underlying geographical features like bedrock depth, wind velocity at altitude, etc.

nradov• 9 years ago

Criticism is easy. Anyone can spot issues. So what?

Xyik• 9 years ago

The issue is that its much easier to critique and call out issues than solving them. Most of the time, the individuals involved are aware of the issues already - critique is only valuable once the criticizer has made an effort to truly understand the problem and what's already been done / being done.

matwood• 9 years ago

I agree to some extent, but think of it as more of this is how the sausage is made type of thing. As engineers we like to opine about perfect code, functional this, and monad that. It pains us to see that business success requires very little of even good code much less perfect. The reality is that the code rarely matters as long as it can be held together long enough for the next user or investor check to clear the bank.

HeroOfAges• 9 years ago

This is a reality I've had a great deal of difficulty coming to terms with. I haven't internalized it fully, but I can't help but think not acknowledging this has harmed my career.

pdpi2• 9 years ago

Facebook is a really successful company, and they blame everybody but themselves for these problems.

The obvious conclusion is: Because they're so successful, they're right and XCode, Dalvik, Git et al are wrong, and their hacks are entirely justified.

The less obvious conclusion is: despite them being so successful, they have a code quality issue, and the tools are entirely justified in breaking under the load that's being put on them.

Arguably, the big problem here is that the obvious conclusion is wrong, and the less obvious conclusion is right. And, because the right conclusion is the less obvious one, there's plenty of value in advising the kind of onlooker that's likely to want to emulate facebook's success (read: the sort of startup that's precisely HN's target audience!) that they shouldn't accept FB's excuses at face value, and should think about avoiding these problems in the future.

MaxfordAndSons• 9 years ago

Totally agree; it's a case of fallacy by results-oriented thinking. I don't actually know if that's recognized in any of the official lists of logical/rhetorical fallacies, but it ought to be. Well, I guess it's a variant of the 'Post hoc ergo propter hoc' fallacy, but seems like an important special case.

rwallace4• 9 years ago

they have a code quality issue, and the tools are entirely justified in breaking under the load that's being put on them

That's not one conclusion, it's two.

Suppose for the sake of argument we grant the first part and say Facebook has a code quality issue and they should have less code. (I'm not familiar enough with Facebook to have an opinion on the matter, but I'm willing to postulate this for the sake of argument.)

That does not in any way imply the second conclusion! One of the most important differences between a good tool and a shoddy one is precisely that a shoddy tool can only withstand the load that should be put on it, whereas a good tool can go beyond that.

TallGuyShort2• 9 years ago

I'm no fan of the "knowing-all-that's-best" trend of articles either, but here's why I don't mind this:

Facebook popularized the idea of "move fast and break things." Move fast, by all means, and don't be paralyzed by fear of all change whatsoever, but I really didn't appreciate it when my classmates thought that by being willing to break things constantly in our group project they were showing us all their potential to be the next Mark Zuckerberg or Steve Jobs. Or when some of those classmates were hired as my team mates and brought that crap into a customer-facing service. I interviewed at Facebook and was scared by the cowboy mentality I sensed in every room and with every interviewer. I'd like the glorification of that to be balanced by, "hey here's some concrete data demonstrating they have some real issues that could clearly be addressed better."

mgkimsal2• 9 years ago

I've seen that "move fast and break things" mantra used in projects, and yeah, some people think they're being the next Zuckerberg when doing so. But what I've (almost) never seen is someone who actually fixes the shit when it's broken.

Fine, go ahead and break stuff, but be willing and able to actually FIX it afterwards. That part never seems to get picked up.

Zombiefruit• 9 years ago

My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.

Literally an ad hominem.

taeric3• 9 years ago

Rephrase it to "What experience does the author have that indicates their experience would actually scale to Facebook levels?"

Is it bullet proof? Of course not. As an easy example, there are many that are more overweight than I am that know nutrition better than I do. That said, it is does seem somewhat logical to question health advice from clearly unhealthy people.

logfromblammo• 9 years ago

It is also intrinsically hypocritical.

Poster questions author's qualifications to criticize Facebook. Poster's own blog post about software quality never hit the front page of HN.

To quoque, dude. Tu quoque.~

ergl• 9 years ago

"What has the author done that is equivalent?"

Remember your words the next time you complain about the food in a restaurant.

yAnonymous• 9 years ago

"What has the author done that is equivalent?"

Being able to build something equivalent is not a requirement for criticizing things.

In fact, some of the best and most valuable criticism often comes from people who have no idea about how the thing they are criticizing works.

sanderjd• 9 years ago

Do you have some examples? I think the most valuable criticism is that which comes with some ideas for improvement, which it is difficult for people who don't know how the thing they're criticizing works to give. In my experience, the criticism of the non-experts that you're lauding, most often points out things that the experts already know about, but simply haven't been able to solve. In general, I think identifying problems is the easy part, and that good ideas for solutions is where the value is.

nstart• 9 years ago

That's a good point you make. But that's also why we have a formal system for making logical arguments. Critical essays which break things down into a premise, arguments, and a conclusion. This allows anyone to challenge anyone else without falling prey to the bias of deferring to authority.

Thus we can and should challenge writers and pundits, but probably from the perspective of first asking if they are making a logical argument or an emotional one instead of making it about their experience levels

omouse2• 9 years ago

Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars

There's something called waste. The issue is that we don't know how much better the situation can be but we can recognize that it's pretty shitty at the moment. If they're making billions of dollars and they're a big company and have all these code quality issues, what hope does a small company have when trying to make quality software? How many more billions could have been made with less churn and less maintenance costs?

pvg• 9 years ago

"What has the author done that is equivalent?

The author has picked some data points and written an article. You have said 'So?'.

BinaryIdiot3• 9 years ago

My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found.[...]Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case.

Sorry but, in my opinion, this is dangerous thinking. Saying you shouldn't say anything unless you've done equivalent / better creates the implication that anyone who hasn't worked on a Facebook scale mobile app doesn't know what they're talking about or can't provide something to the conversation that is at least informational or useful.

Do you complain about politics? If so then I would expect you to have at least held office in the past otherwise you should hold your tongue.

Have you complained about a truck driver on the road? Well, have you driven a truck? If not you should not complain.

etc...

valuearb• 9 years ago

So you are saying it's reasonable to have 18,000 objective C classes in a single iOS app?

tomcam• 9 years ago

Yeah, fuck that noise. Not a fan of Facebook or its CEO. But clearly there is no code problem – look at the stock price over extended period. QED

jheriko2• 9 years ago

one look at hhvm tells me they have poor quality development practices. :P

i get the impression that they use the struct keyword to avoid having to type public everywhere for instance...

watwut5• 9 years ago

Reality check is that most companies, including most startups, attempting to create software of such size end up tangled in their own mess way sooner and fail. Keeping it maintainable at such scale is not as easy as keeping it maintainable when there are four of you.

planetjones• 9 years ago

Well when there is a codebase of the scale of Facebook's, with so many activate contributors of course there will be a code quality problem. I see so much stuff at Google breaking or becoming inconsistent too (away from the core search features).

Even if they didn't move fast and break things, they would still have code quality issues. It's just the nature of a huge codebase like they have. And let's face facts - the development process needs to be suitable to the type of problem being solved. This is not a life and death situation and users would value new features, more than extreme reliability and consistency.

fishnchips2• 9 years ago

I spent a few months at Facebook and left because of that. Not necessarily because of poor code quality but because of all the issues it was causing, and a culture of turning a blind eye to (or worse still, being proud of!) those issues. It's obviously true that quality of their code does not affect their market position, but it sure affects who they hire and who they retain.

joeletizia2• 9 years ago

That may be true. But in the end, does it matter?

Until we see someone taking market share away, I'd argue it doesn't. Facebook still retains top tier people regardless of any perceived quality problems.

Sorry you left. I just started here, and I do agree that what I feel about code quality isn't generally shared here. But as I get older, I start realizing that in the end, as long as things work relatively well at our scale and in our problem domain (we're not medical device software, we can fail) alls well.

Sevii• 9 years ago

It matters because profits are lower. 400 devs on an iOS app is not free.

fishnchips2• 9 years ago

As I got older I realized my threshold for tolerating bullshit is getting lower and lower. At the end of the day as an engineer, I'm as much (if not more) motivated by the quality of work I'm doing as an individual, team, and company as I am by purely monetary rewards. I could trade this justified pride for a feeling that I'm building something good and important, but FB is probably a net zero on both counts.

feedjoelpie2• 9 years ago

I did some contract work for FB around when this article was written. I don't think that nobody cares about clean code. It's more like... results will vary widely based on who reviews you. And the governing philosophy is more focused on the automated checks and balances that try to keep you from doing harm to the overall system. While human code reviews were pretty lax, the machines would slap you around for all sorts of things, none of which would force you to write beautiful code.

Personally, I like for humans to be pickier than they seemed to be at Facebook, but it really opened my eyes to the value of what I think of as a "keep your garbage in your own trash can" approach to quick-and-dirty coding. As long as you didn't spill your trash into the hallway, the machines were satisfied, and your reviewer didn't find your work to be _insane_, your work shipped.

[EDIT: And I find that this can often be a wonderfully pragmatic way to work if you have the tools to keep you from doing harm.]

ptero• 9 years ago

Code quality is important, but for an organization the size of the FB good systems engineering is much more important than good software engineering. If the overall system is organized sanely, failures of subcomponents (which happen due to software bugs, hardware, humans, etc.) can be isolated, firewalled and fixed in reasonable time without bringing the system down.

chiefalchemist2• 9 years ago

Move fast...and just leave things broken. Users be damned.

chiefalchemist2• 9 years ago

But seriously (i.e., ref to my previous comment). All the tools. All the talent. All the leadership and management. All the money. And still even the mighty FB struggles.

Point being: This shit might not be rocket scientist hard, but it ain't easy either. And when you don't have the war chest of the likes of FB you're in for an ongoing and never ending (quality) battle.

raygelogic• 9 years ago

this strikes me the same way as the story of the guys who figured out where to add armor to bombers during WW2. the places on the plane that had bullet holes after a mission clearly were operational enough to return to base. the places that never had any damage were probably critical, so that's where armor was added.

if a company can make billions with a poor-quality codebase, clearly quality isn't a bottom-line concern.

what is a concern is shipping the damn product.

dawidloubser• 9 years ago

I would like to argue that, in most of these cases, it's not "code quality" that is at fault, but "design quality" (before coding) - which is often absent entirely.

The problem with design, in software, is not that most people forget to do it. It's that they never learn to do it. It always comes back to bite you.

I don't want to start a discussion on design, and how most people mess it up because of lack of skill or experience therein. But hacker culture seems to be allergic to design, and hacker culture seems to be what everybody strives for these days.

lefty23• 9 years ago

I think you hit the nail on the head. My motto for developing code is "make it simple as possible, but no simpler". Whoever, created a design that used 18,000 classes did not follow that rule.

typetypetype• 9 years ago

What's the point of an app? Generating high quality code or generating revenue? Until you hit intersections where revenue can be increased by changing code, the priority will always be on new or changed features.

baggachipz• 9 years ago

The “Hack” and “Move fast and break things” culture must make it very hard for developers to focus on quality.

Well, yeah, that's what the "break things" part means. The problem is when people/companies try to have it both ways. "Move fast and have high quality" isn't possible.

thedevil• 9 years ago

It's hard to argue with the business sense of pushing out features quickly at the cost of code quality.

But I've learned that that there are two kinds of development teams:

(A) the teams that are "moving fast and breaking things" while "creating business value"

and

(B) the teams that are "only maintaining legacy code", "not getting anything done", "costing a lot of money" and "complaining too much" about code quality that team A wrote.

As an engineer, I've learned that it's less work and more rewarding to be on the (A) teams than on the (B) teams.

nickpeterson• 9 years ago

The difference is that Team A can leave and a project is delayed. If Team B all left the company would fold.

ceronman• 9 years ago

Both are necessary. If Team A leaves, and no more business value is produced, and the devs only care about beautiful code, the company will fail too.

Both kinds of teams are needed. Or even better, teams should not be divided like this, everyone should do their shares of both breaking things, and creating value and cleaning the mess and scaling the things.

scaryclam• 9 years ago

Funnily I've actually seen a Team B leave.

There was much panic and thinking it would all go horribly wrong, and indeed it was madness for the first week. Then the people who took over started to stabilise some of the worst parts of the system and it all sort of fell into place after that.

Fast forward a couple of months, the project was pretty much hands off and we all realised that keeping a whole team on that project had pretty much been a giant waste of money. They did nothing but create fires for themselves to fix. The Team B's exodus was actually a really great thing for the company.

I suppose the thing to learn here is, that you need redundancy in your teams. If we hadn't had other devs to step in and take over, it would have been a bit different. Still, I'm not convinced that a handful of contractors couldn't have been brought in to the same result, just more expensive.

sAbakumoff2• 9 years ago

I like type (C) whose motto is "move fast and fix things" https://githubengineering.com/move-fast/

ricardobeat• 9 years ago

And yet https://github.com/blog/2106-january-28th-incident-report

collyw5• 9 years ago

I am type (D) don't move particularly fast, but produce decent code - when management isn't pushing me to be type (A).

liveoneggs• 9 years ago

That's because team "B" is actually an operations team and, believe me, you don't get into ops for "fun". :)

omouse2• 9 years ago

Yes and that's the perversity of the incentives in the industry. Automation and coding are such high value activities that even the smallest, shittiest piece of code can produce a lot of $$$ that can make up for its crappiness and its maintenance costs.

watwut5• 9 years ago

The best is to work in companies where people maintain their own code and are responsible for fixing what they broke. The moment the split is like you said, A team will produce progressively crappier and harder to maintain code. The B team will have high turnover and require higher salaries to stay in.

It is bad organizational structure.

draw_down2• 9 years ago

Yes, thank you. I've been trying to write out that very idea for so long. It comes down to, the heroes are the people making the new stuff, and the janitors are the ones stuck with all the bugs and tech debt of the new stuff. Guess who gets promoted.

When you find yourself in the janitor role, time to move on, because your employer doesn't think much of you.

temp_account01• 9 years ago

I've learned from working in one of the largest companies that there are two kinds of development teams: (1) the teams that mostly depend on the products of other teams; (2) the teams that create products that others depend on. And when the upper management starts rushing the deadlines sacrificing quality for the deliverables guess which team becomes (A) and which team becomes (B)? I currently work in a (B) team, and I try as hard as I can to "move fast and break things" but there is simply no way of doing that because every significant code change that I make requires changing the code we don't own. Since the code that we don't own is written in such scrappy way that you better don't touch it or else it breaks, our team has to resort to only minor insignificant code changes.

>As an engineer, I've learned that it's less work and more rewarding to be on the (A) teams than on the (B) teams. It is certainly more rewarding to be on the (A) team, but do we always have a choice?

mybrid• 9 years ago

It is worth noting that when something goes viral we still don't know why. Facebook represents the ultimate in viral success. Given we do not understand why things go viral then it stands to reason that the underlying software methodology is not causation. This means that the fickle finger of fate could easily be visited upon Facebook the same as with MySpace and Facebook will be in no control. This is independent of the any software methodology they use.

Claims made without evidence can be dismissed without evidence.

strken• 9 years ago

Can someone explain the example with the ramdisk? It's not clear to me how a ramdisk is relevant to the problems put forward in the paper, i.e. disk is too slow, they rely on lazy page allocation, and they have insufficient ram to hold two copies of everything simultaneously.

alex-• 9 years ago

"when Facebook employees are not actively making changes . . . the site experiences higher levels of reliability."

This seems like a great thing to me. i.e. The system is stable and the error budget is being used to facilitate change.

dwils098• 9 years ago

links to slides are all dead... in parent articles and reference articles

bobbyi_settv• 9 years ago

The second exhibit is from Facebook Research... wouldn’t you just write your disk files to a ramdisk? Surely they noticed this too

I'm not sure the author understand what a Research team is and what it does. Trying out a new solution to an existing problem is sort of their job. I'm also not sure how a research team publishing a paper discussing an alternate solution indicates anything about the company's "code quality".

Bahamut• 9 years ago

Anecdotal evidence, but I have a friend who complained that some critical parts don't have tests, and a lot of developers are loathe to write them. He got bit by it when he made a change in an area with no test coverage and broke the spam filter.

tomkin• 9 years ago

It's important to understand that Facebook is incredibly large, and the fact that any software scaled with them to the point they are presently at, is a "on the shoulders of giants" situation that seems to actually be working well enough for them to become a billion dollar enterprise.

Also, the building of infrastructure that takes on scale that's not yet been required is somewhat inefficient. But it does beg the question: If Facebook is limited by its infrastructure, what responsibilities do they have to build software that continues to scale for future organizations?

p0nce• 9 years ago

At this point it is clear that software size increase linearly with the size of your software team, whatever the problem you solve.

This affects every organization and is something that should be actively fight against. Having accidental, unplanned, unaccounted costs should not be the default path.

charleslmunger• 9 years ago

"Exhibit C: Our site works when the engineers go on holiday"

Is not all so different from "Fewer patients die when heart surgeons are on vacation". Of course your site is going to be more reliable when nobody is changing it! You should be worried if reliable isn't the steady state, and it requires constant changes to stay up!

afc• 9 years ago

I also took issue with this part of the article; it seems obvious to me that rolling out new features would directly cause reliability issues (and the only alternative, of not rolling out new features, is, obviously, a cure worse than the disease).

adrianratnapala3• 9 years ago

I am not sure that most Facebook users would actually agree about this.

But then their view is slanted towards UI changes. Performance improvements and other subtleties are features too.

joshvm• 9 years ago

My immediate thinking was this as well: when they stop touching it, it's fine, which means that whatever bugs are checked in seem to be caught quickly. What's more, the periods of high reliability are during periods when you might expect there to be heavy load - assuming people don't check Facebook at work (har har).

What counts as an incident?

Figure 1 includes data from an analysis of the timing of events severe enough to be considered an SLA (service-level agreement) violation. Each violation indicates an instance where our internal reliability goals were not met and caused an alert to be generated. Because our goals are strict most of these incidents are minor and not noticeable to users of the site.

So these are minor issues. The parent article is paraphrasing at best, and jumping to conclusions at worst. From Facebook:

We believe this is not a result of carelessness on the part of people making changes but rather evidence that our infrastructure is largely self-healing in the face of non-human causes of errors such as machine failure.

They then list a number of sane strategies to mitigate this.

http://queue.acm.org/detail.cfm?id=2839461

draw_down2• 9 years ago

Engineers should not brag about this, in fact it's the opposite of something we should brag about. "Our site doesn't go down when we're not here devising ways to break it"

ddacunha• 9 years ago

I don't see this as bragging but as an reality/consequence of introducing change. We do a lot of things to minimise occurrence and impact of change but there always is(/will be?) that (corner-)case that break things. In other context,that is often a point against upgrading old systems that have been working and doing the job for a long time.

adrianratnapala3• 9 years ago

Yes, I expect the graph in Exhibit C is perfectly normal for any company.

zo7• 9 years ago

Some of FB's ~18,000 Obj-C header files from a post linked in the article: [1]

* FBFeedAwesomizerProfileListCardViewControllerListenerAnnouncer.h

* FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h

* FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h

* FBGroupUpdateRequestToJoinSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h

* FBMemReactionAcornSportsContentSettingsSetShouldNotPushNotificationsResponsePayloadBuilder.h

* FBProfileSetEventsCalendarSubscriptionStatusInputDataContextEventActionHistory.h

* FBReactionUnitUserSettingsDisableUnitTypeMutationOptimisticPayloadFactoryProtocol-Protocol.h

oh my god

[1] http://quellish.tumblr.com/post/126712999812/how-on-earth-th...

aljones• 9 years ago

Those read like classes generated by tools. Or maybe I really don't want to think humans are capable of such names.

utexaspunk• 9 years ago

Well, the humans could be tools...

oriolid• 9 years ago

These are completely credible class names, just ask any Enterprise Java developer. I particularly enjoy FactoryProtocol-Protocol. Who would be satisfied with just a FactoryProtocol?

lefty23• 9 years ago

They could be. The effective coding standards for iOS are to make class names stupidly long.

saosebastiao• 9 years ago

Makes me wonder how much dead code makes it into that abnormally large binary.

zzzzzzzza• 9 years ago

long names are good for both searchability and documentation purposes, and not a problem with decent tooling/autocomplete

If the names were more abstract, as in some java code, then it would be somewhat dubious in value. Most of these seem very precise however.

lefty23• 9 years ago

It is a problem. The class names don't fit in the class browser window, which means you have constantly be resizing it to read the end of the class name.

collyw5• 9 years ago

The problem isn't with the names. The problem is with overcomplicated architecture that leads to names like that.

jyriand• 9 years ago

Only Kent Beck can save them. :)

whack• 9 years ago

I do think that FB, and most large orgs, have code quality problems. But the article does a pretty bad job at making its case.

"That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes."

How does the author know that 429 is too many? How does the author know that FB's goals and functionality can be best achieved with fewer people/classes. This just reads like a classic "Why is Google so big, I can do that in a weekend" comment (https://danluu.com/sounds-easy/)

"Our site works when the engineers go on holiday"

This is pretty much universally true for any dynamic site where engineers are constantly adding new features. Change always comes with risks. Changing code is always more likely to break something in the very-short-term, compared to not changing code. I have a hobby site (shameless plug, http://thecaucus.net) which has been running in auto-pilot for the past year, and almost never breaks, because there are virtually no moving pieces. The fact that the FB site breaks more often when engineers are making changes to it, is just repeating a universal law of software development.

I do think that the organizational structures at most large companies are bloated, inefficient, non-transparent, and produce sub-par code. I had high hopes when I read the article's headline, but the arguments presented simply aren't very persuasive.

combatentropy• 9 years ago

I look at programming as all design, whether deliberate or not. Martin Fowler drove the point home for me (https://www.martinfowler.com/articles/newMethodology.html).

He said that some people drew inspiration from construction, where there are designers and builders. One or two highly paid architects draw up the plans. Then you can hire a bunch of cheap labor to build it, say a bridge. This belief leads to a dichotomy in software companies, where one person is the "architect" and others are just "regular" programmers --- often outsourced to the lowest bidder.

From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker." There is little repetitive, mindless work in programming, because as most of you know, "anything that you can treat as construction can and should be automated." Therefore, "In software all the effort is design . . ."

krick• 9 years ago

I feel your sentiment, and usually oppose to the construction-driven approach you describe, but it is really easy to play devil's advocate here. I mean, let's just be realistic. There's nothing sacred in programming.

The fault in your argument is that there's no design/construction dichotomy, but rather a gradient: everything can be viewed upon as a design from close enough distance, and everything can be a construction looking from the high above. For business people all the programming is just very, very low-level construction, and there's no design to it. Damn, even hiring programmers and creating companies is just a construction stage to them.

And from the practical standpoint, the truth is that perceiving a lot of programming "just as a construction" just works. For the better or worse. It is cheaper and usually reliable enough to design the system in a way that a lot of the components would be plugins not depending one on the other that can be outsourced to anyone. Even if some fail, the system continues to work while you hire new people to rewrite some components. Sure, maybe it would be more efficient and "cool" to hire a really smart guy (or do it yourself, if you believe you can) to write it as really optimized monolith system, but if this isn't required, it is usually less risky to build something not so sophisticated, and improve it if need be. It is usually ok, that some parts of the system may fail. Fuck it, let's not lie to ourselves: there always are some parts of the system that fail, whether you think it's "acceptable" or not, let's just take it into account.

erikpukinskis• 9 years ago

There is little repetitive, mindless work in programming, because as most of you know, "anything that you can treat as construction can and should be automated."

Sure, but there is a cost to automation. We all know the programmers who will attempt to automate God if left to their own devices. cough At some point there is a tradeoff between automating something, and sending an email to an intelligent human being who maybe isn't god level at software architecture but who can fill out some organizational context with pidgen code.

I think the idea that the only code that needs to get written is type of a decorator on a subtype Haskell kind of stuff is wrong. It's the same fallacy as people who say we just need physics and all of the natural sciences will shake out of it. We need Richard Stallmans and Terry Davis's to write that stuff for sure, and maybe they are the only True Programmers out there. But there's an awful lot of automation engineering that needs to get done too, and some of that really is just putHotDogIn(basket) (unless Debbie from accounting emails you to say no) and wget http://wordpress.com/dist.zip

Let me say it another way: there are and will always be gnarled programmers who think in koans and concoct the words we code in, but spread across that will be a thick layer of humans embedded in human systems mapping the minutia of the day in code. I daresay we'll all be doing it.

I suppose I agree that work is all design. But the code isn't the part that's being designed. The code is just the log format.

norswap• 9 years ago

I was doing iOS dev three years ago, and the Facebook lib for iOS was the worst piece of shit I ever had to deal with.

Their app also used to be unbearably sluggish despite not doing anything too fancy.

babesh• 9 years ago

I thought the joke was that they had interns write that? I remember an API that would hold onto completion blocks and call them repeatedly instead of just once. They would also accidentally make non backwardly compatible changes to the API.

They didn't care since that API period propelled them to more dominance and growth for awhile. After they shut down the viral channels, the developer ecosystem died...but it had proven its usefulness.

The only issue now is developers don't trust them and they can't get enough developers to build atop them.

linkmotif• 9 years ago

While the author wrote this article FB made millions of dollars.

badpenny• 9 years ago

I recommend reading They Write the Right Stuff[1] to anybody interested in NASA's approach to writing software.

[1] https://www.fastcompany.com/28121/they-write-right-stuff

ChrisRus• 9 years ago

From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker."

Ha. That is laughable. If it were so, then we would be able to automatically graft features from one product to another. I contend this is exactly how SoC are designed and there's no reason why we shouldn't copy the methods of the digital design community (proven to scale or you wouldn't be reading this).

aqibgatoo• 9 years ago

bob martin a.k.a uncle bob always says that people consider patterns like mvvm,mvc,mvp etc as Architecture which actually are just for the ui layer.It happens commonly in mobile development.

pablo_fiesta• 9 years ago

Rub lavender oil on the servers, say "web-scale" 3 times, click you heals and everything will be OK. Oh wait, but the Russians.

Facebook’s code quality problem (2015)

Related Stories