Designing a study to see if “the 10x programmer” is a real thing

http://andrewgelman.com/2014/12/04/designing-study-see-10x-programmer-real-thing

by chrismealy • 12 years ago

26 48 12 years ago

jiggy20112• 12 years ago

What task could these programmers perform that would not be biased towards programmers who have recently solved similar tasks?

spacecowboy_lon• 12 years ago

depth of exprinace is a factor eg yesterday I wrote a feature extraction tool in 2/3 of a day (including geting up to speed with python) to strip out the usable body copy for a large uk website.

I could do this so quickly becase I had 3 years previously spent several days experimenting with new scientist's archive and allready knew what library would work.

ghshephard4• 12 years ago

Depending on the field, it could be as much as 40x. We had a firmware engineer, who in 3 months, single handedly wrote an entire FHSS MAC layer for an RF application, with a high level of quality and hundreds of functions, while simultaneously laying down the architecture for it's growth over the next seven years. He was extraordinarily capable, intelligent, and had worked on the identical project in the previous 5-6 years, and so was able to anticipate all the blind alleys to avoid.

I'm reasonably confident that a low-performing engineer with little or no experience in this space would have taken seven-ten years to accomplish the same feat at the same level of quality and completeness. (Of course, it could be argued that our engineer had taken seven years to accomplish this task, it's just that they had worked six of them at a prior company)

The challenge in these types of tests, of course is (A) who defines/measures quality, (B) How do you know which functions were essential? Programmer performance is not a function of how fast you can complete coding challenges, but how well you can architect, document, communicate, collaborate, and all the other peripheral functions not associated with cranking out algorithms.

ExpiredLink3• 12 years ago

Depending on the field, it could be as much as 40x

The best dentist I've met easily was 40x better than the worst. But what does this tell us about dentists?

awjr• 12 years ago

I'm not saying it happened but if I had a developer who had worked for 5-6 years on an identical project, I would be very careful that your codebase does not include any of that project.

I do feel you are taking somebody who had an immense understanding of the domain, who had delivered an identical project over the last 5-6 years (and probably had that codebase to refer to) and trying to compare it to an engineer with no domain experience and no identical project to refer to.

ghshephard4• 12 years ago

No codebase to refer to, everything was recreated from scratch; but the concepts (basing a hopping sequence on MAC address, agressive discovery of neighbors on power up, sending a single data/ack packet, instead of a complete POLL-ACK-DATA-ACK for small packets, tracking information for all adjacent nodes in a nodeq, keeping track of both RSSI and info success, etc...) were concepts that he had learned/developed previously (thankfully many of them in early 1990s so patent issues were probably not an issue either)

And you are correct - I am taking somebody who had an immense understanding of the domain, and testing them on a (close to) identical project and I am trying to compare it to an engineer with no domain experience - that was kind of my point. He was still an extraordinary engineer - but, moved into a new domain, maybe a 2x, possibly 3x engineer. (i.e. he could write code in a single month that would take a low level engineer three months to write to similar quality.)

But, if you are looking to build a new product - isn't the idea to hire extraordinary engineers with 7-10 years of experience in that field to get the huge boost in productivity? And, if you really could get 20x-40x performance boost in quality and performance - what would it be worth to hire those employees away from their current companies to lead your team, if only for a few years?

And, if these people really do exist - doesn't it make sense that they should be working for only one-two years at companies, where their skills are precisely matched to requirements, and get the huge boost in compensation for that time?

The only thing that is missing, is some mechanisms, system, that could match up these "world class engineers with extraordinarily specific skill" to companies that need those specific skills.

onion2k• 12 years ago

A brilliant engineer: ...had worked on the identical project in the previous 5-6 years, and so was able to anticipate all the blind alleys to avoid.

A low-performing engineer: ...with little or no experience in this space would have taken seven-ten years to accomplish the same feat

If the brilliant engineer spent 5 to 6 years working on identical apps before doing a good job for you, and the low-performing engineer would take 7 to 10 years to do a good job, doesn't that imply a good engineer is only 1.5* better, but any engineer needs 5 years experience before they're capable?

ghshephard4• 12 years ago

That's a pretty good summary.

In reality, though, what happens is that engineers with no experience first get an education. For example, in RF - getting your EE degree will quickly boost you up from a 1x to 5x engineer. Then, they go work for companies that do have experienced engineers, and they learn hundreds of skillsets, and develop deep domain knowledge in the ones that they are interested in, so my "engineer with little or no experience" in this space probably doesn't exist - a typical EE who came to work out of school for an RF company, probably learned a bunch of different concepts (PHY, MAC, Routing, DFM, RF Immunity, Antenna Design) from the 40x engineers already there with 10-20 years of relevant experience - and so, within a five years, would already be a 10x engineer in those field versus one that had no experience/education in them, and would be 20x in the projects that they had direct experience developing.

I would suggest that 40x performance increase comes from a combination of innate skill (you are already extraordinarily gifted and capable), Direct domain experience, a leadership role in which you were required to develop/invent/create and then document/x-train/and educate others on the technologies that you have mastery on, and, perhaps most importantly - passion, energy, and devotion to the field.

The caveat to all of this is that with projects of enough duration, everyone gets a chance to pick up experience, and knowledge, and so will increase their capability in a particular domain. The other caveat, is the vast amount of engineering is not simply repeating the same task over and over again - unless you are making cell phones for Samsung.

svisser• 12 years ago

How does that qualify as 40x? It's not reasonable to compare an engineer who has worked for 5-6 years on an identical project to an engineer that hasn't or has less work experience.

For this particular task he may have been 40x more effective than anyone else but only because he walked the path before.

geocar5• 12 years ago

Do you pay programmers with 5-6 years experience in a domain 40x junior programmers?

ghshephard4• 12 years ago

That was the point I was trying to make - there is more than one way to get a boost in performance. Innate Skill is one of them, but another is to do it through experience.

duncanawoods• 12 years ago

> it could be argued that our engineer had taken > seven years to accomplish this task, it's just > that they had worked six of them at a prior company

Thats a great line - I think it reveals a lot about the impact relevant experience has on productivity.

Its truly awesome when it works out but change the problem or the technology and the 40x engineer becomes a 0.2 engineer in the short/medium and sadly sometimes the long term too.

I see two main problems that prevents experience always compounding positively - technology churn whereby a master carpenter has to switch to metal every two years and the unspeakable variety and flexibility of the software medium. I see software engineering as a collection of 1000 niches rather than a homogenous whole. RT embedded vs. data center vs. client-side vs. 3d vs. mass consumer and hitting market first vs. 10 year continuous improvement vs. safety critical etc. there is only so much overlap between them.

In one niche, killer productivity might be unlocked by leveraging libraries, in another by reducing dependencies. In one niche highly defensive programming may be the winner, in another rapid iteration might be the secret sauce. In one you need to master the theory, in another you need to abandon it. We too easily learn lessons in one niche and don't realise they are not generally applicable.

This is also why comparing two engineers is pretty tough. You can test them on the same problem but this will only ever be a narrow view of their comparative strengths.

ExpiredLink3• 12 years ago

What do they want to achieve? You certainly can find one woodchopper who is 10 times as productive as one other woodchopper. But what does this say about woodchoppers?

DanBC• 12 years ago

Once you find 10x woodchoppers exist you study the differences between them and 1x woodchoppers.

Do they use different swing of the ax? Do they chop different wood? Are their axes sharper or heavier or have longer handles?

You then take those things and give them to the 1x woodchoppers and see if it makes any difference.

shurcooL• 12 years ago

The reason you might see a 10x woodchopper is that you measured their productivity for an hour, and the other guy was on his lunch break for 45 minutes.

Similarly, engineer productivity may be one level one week and a different level another week.

What if you could measure lifetime productivity? Or over 10 year period. That'd be more insightful.

vorg• 12 years ago

Programming shops are full of both 3x and -0.5x programmers. Many 3x programmers end up looking after a group of say four -0.5x programmers, and so after cleaning up after them they themselves come out looking like mere mortal 1x programmers (because 3 + 4 x -0.5 == 1 ).

yetihehe• 12 years ago

Speed will also depend on length of the task. If you use 1 hour long simple task, you will have less variance. Significant results could show only with 1-week task or longer.

jbergens2• 12 years ago

See my previous answer. I don't think it works for longer tasks either since developers learn while coding. And if it takes 10 months to build a system it still might be impossible to do it in 1 months even if you're skilled.

sklogic3• 12 years ago

I suspect that all this 10x buzz is coming from the overstretched, sleep-deprived people who accidentally managed to come to work once or twice well-rested and realised: "omg! I'm 10x compared to my usual self!".

ZoFreX• 12 years ago

I don't. I recently witnessed someone writing a 10 line function to test for a particular condition in our codebase. I could have done it in less than one-line, by re-using pre-existing code.

What took them 10 minutes would have taken me 1. My solution is also more in-line with how it's done over the rest of the codebase, if you applied my approach 100 times you would have a consistent and easy to maintain codebase, apply their approach 100 times and it would be inconsistent and harder (more time consuming, more expensive, more risky) to change.

That's where the 10x comes from.

Edit: Just in case they're reading this... I don't think this is because I'm a better engineer or you're a worse one, I blame our lack of documentation and training for this feature!

sklogic3• 12 years ago

That's an anecdote, not an evidence. Same as my own revelations after trying to work while being unusually well-rested.

The next day the same person may do something 10x times faster than you, because of the differences in your past experiences, problem solving methodology, etc.

What really matters is amortised performance, and there is no evidence that there exist a significant deviation in such among professionals of comparable experience profiles.

Of course, someone who already did something ten times will then do the same or a similar thing in no time, while a completely new problem will be faced with blank eyes by anybody otherwise smart and experienced. It's such an obvious thing that it cannot be a source of a myth on its own.

ExpiredLink3• 12 years ago

Why does the "10x programmer" myth live on? Because all supporters assume that they are one of the '10x' - or at least aspiring to be one of those. It's a self-energizing myth.

cpks2• 12 years ago

Because, for complex software, it's not a myth.

jbergens2• 12 years ago

I think it is partly a myth. It can probably only work for small parts/systems. If a large system has to be written and it takes 3 good developers 2 years to do it it will probably still won't take 3 "bad" developers 20 years to do it. They will learn things when they work. If I can write a few (like 5-10) classes to do something and it takes me an hour that still don't mean someone can write it in 6 minutes.

alkonaut3• 12 years ago

I consider myself average but have met my share of the 1/10 ones ;)

lhnz• 12 years ago

Do you want to test the productivity of programmers with FizzBuzz or writing an OS from scratch?

The distribution of productivity amongst engineers is probably going to vary depending on the complexity of the task.

Some tasks I would expect the top performers to be far better than 10x at.

And not all top performers are equally suited to each task.

Additionally this idea of measuring productivity purely by problem solving ability is nonsense as engineers are hired also based on their ability to write maintainable code and to work as a team, etc.

geocar5• 12 years ago

This made me think about how Arthur's been passing around some cs107 challenges that he found online.

Here's one of them: http://web.stanford.edu/class/cs107/assign2.html

The advice page suggests the student hours spent on this project is:

    * < 5 hours 5%
    * 5-10 hours 14%
    * 10-15 hours 19%
    * 15-20 hours 38%
    * 20-25 hours 14%
    * more than 25 9%

Pierre did a solution in 1hr 30m; I was just under an hour.

I wonder how people do here?

alexandros• 12 years ago

A developer can have negative performance, either by wasting time on the wrong thing, or by producing a solution that is flawed and needs to be rewritten (if the project survives). On the upside, a developer can invent a whole new way of doing things, and as such acheive things that an arbitrary amount of other developers wouldn't be able to. As such, 10x may be a useful metaphor, but in real terms is a gross understatement. It may make more sense in traditional enterprise settings where most things are determined from above, but in startup settings, a good developer can make or break the company.

cpks2• 12 years ago

It's hard to study. The initial studies were coding time, but the productivity difference comes down to a few things:

* Good architecture choices. These make a big difference in total development time.

* Communications. A team of 50 1x programmers has much more overhead than a team 5 10x programmers.

* What's possible. There are some things some people just can't do, even given infinite time.

* Process overhead. Bad people require much more process than good people.

Overall, the difference in productivity between small, good team of expensive people and large, bad teams of mediocre people is much more than 10x for complex systems. It's very small for large, simple systems (e.g. "business" systems which handle inventory, HR, etc., which just have a very large number of relatively independent simple bits).

That explains super-high-profit companies like Valve.

Once organizations get into the mediocre person game, it becomes almost impossible to go back. Good people won't want to work there. If they do, the amount of process will slow them down. You won't leverage them well -- you still have the communications overhead. They'll always be told what won't work. Etc.

qwerta• 12 years ago

How about making comparative test where one subject has suboptimal conditions?

- no objective given, or change objective in middle of test

- no IDE allowed, only notepad and outdated compiler

- place next to garbage bin or turbofan engine

- randomly generate some 'legacy code'. Result must contain 90% of this code...

- no internet connectivity allowed

- no external libraries allowed...

- code must be deployed within 1 week, it must past code-review before deployment, reviewer is on holiday....

- ask subject some trivial question every 10 minutes. Or thrown in 5 hours meetings randomly.

I am sure the productivity will vary... :-)

sklogic3• 12 years ago

You forgot to add a Swordfish-like hacking scene to this list.

touristtam• 12 years ago

Without mentioning the off work life challenges. I mean those have possibly an impact that often see dismissed in those stories about 10x engineers.

thorin• 12 years ago

Not sure why x10 is such a surprise. It [software] is a good field with decent money and a low barrier for entry at a lot of places. Not everyone can be Dennis Ritchie.

If songwriting had lots of well paid jobs open to virtually anyone with a hobby interest or non-related degree do you think everyone would produce like Lennon/McCartney?

You could give me a million years and I wouldn't write the Linux Kernel but I can still produce useful software which improves someone's productivity.

jiggy20112• 12 years ago

You couldn't write the 2014 Linux kernel because it is the product of thousands of people and millions of dollars , you could probably write the 1991 Linux kernel given time and motivation. Linus did a lot of things wrong in the early days.

joelthelion• 12 years ago

A productivity factor is not a good way to model programming ability.

I consider myself a decent programmer, but I'm not that much more productive in terms of features/week than other less skilled programmers.

The real difference is that I can do things that poor programmers simply cannot do: architect a system correctly, optimize performance, debug a difficult memory leak, write correct code, etc.

geocar5• 12 years ago

This is an interesting point: Experienced programmers might not need the spec.

objectobject• 12 years ago

This. Measuring productivity is useless if you are producing the wrong thing. Even ignoring all the things programmers do other than writing code, this idea still completely ignores the many external factors that interfere with productivity.

Organisations are more than the sum of their parts. It's not like you can take a bunch of 10x programmers from a successful project, put them on a failing project and expect them to just get it done. Why is 10x programmers a thing but not 10x management?

vellum• 12 years ago

Peopleware was one of the sources for the "10x" concept. However, the book only said the best:worst ratio was 10:1.

As for the rest:

Count on the best performer being about 2.5 times better than the median performer.

Count on the half better-than-median performers outdoing the other half by more than 2:1.

http://javatroopers.com/Peopleware.html

throwawayaway• 12 years ago

mythical man month was the one wot done it

pawelkomarnicki• 12 years ago

The "10x programmer" myth is a fallacy that boosts the rat-race mentality in programmers, believing you need to be able to solve problems with quick hacks in order to be considered "a good progreammer". But the problem is: productivity is not a constant. It's more like a tide, when it comes, it comes strong, but then it fades and you can't do anything about that. Unless you're a robot. Also, engineering is that fancy field, that often gets overly excited about short-term wins, loosing the track of the bigger picture — one programmer can do something in 15 minutes, using lots of poorly designed hacks, without any optimisation or deeper thought, which will come back to bite very soon; and another programmer might take few hours, making a solution that will be rock-solid even after some time. The short-term "productivity" is in favour of the 1st one, but in reality the 2nd one is more productive in the long run.

auganov• 12 years ago

The point is some people will provide an order of magnitude more business value in the long run. Most of that value will be outside of "implementation skill". The 10x programmer isn't really just a programmer. The more decision making power one has the higher the potential multiplier. Trying to find "the 10x programmer" in a large enterprise or a consulting company is like looking for a lion in Sweden. Though even with simple tasks 5x is probably not that rare.

alkonaut3• 12 years ago

Like so many before, this post seems to avoid the fact that while there is talk about a "10x variation" in productivity, that's not really the controversial claim, if by variation we mean "range" or "span". To find a 10x span just look for a 1/10 dev and you are done. There is no lower bound in productivity so the ratio of productivity between least and most productive is infinite since there are likely some that produce zero value (I'm pretty sure I've met those that deliver negative value).

So. Before one designs a test to find a unicorn one should define unicorn. Is it a certain variance in the sample? A variance such that at least one sample point is 10x more productive than the mean, that is at least well defined, but how large would the sample size have to be if you are looking for a single unicorn?

SiVal• 12 years ago

I think this is a good point. After all the claims about how the majority of self-described programmers can't write FizzBuzz, are those of us who can write FizzBuzz already 10x programmers before we even get started on anything harder?

qq66• 12 years ago

Whether "10x" exists depends entirely on the problem domain and the way you define productivity.

If the problem domain is building an HTML login page, and your measure of productivity is time to complete, then a 10x programmer probably doesn't exist. A whiz maybe writes it in 5 minutes and the average Web dude takes 20. Typing speed may be the limiting factor.

If the problem domain is building a better way to search the Web, and the way you define productivity is in long-term revenues generated, then Larry Page and Sergey Brin are billion-x programmers for building Backrub/Google.

verydapeng• 12 years ago

from my own experience ... there are negative-value-created developers ... 1/1 is 1x, 1/0.1 is 10x, 1/0.01 is 100x ... 10/0 is infinity, then 10/-1 ??? not very well defined

Designing a study to see if “the 10x programmer” is a real thing

Related Stories