Don't measure developers, measure projects

https://anaxi.com/blog/2018/11/25/do-not-measure-developers-measure-projects/

by jeanlaf • 8 years ago

164 62 8 years ago

nixpulvis• 8 years ago

Getting secure connection warnings.

daveslash• 8 years ago

Maybe the author fixed it - I see no such warning (FireFox 63), but what did catch my eye was a 3.6 MB jpeg that took 16.3 seconds to load (25 Mbps connection) -- admittedly, haven't read the actual article yet.

jeanlaf5• 8 years ago

Oh let me check this! Sorry about that.

jeanlaf5• 8 years ago

Fixed!

edejong• 8 years ago

Don't automate what you can't first do manually. Your best developers might be helping, supporting, designing and architecting your next solution.

One commit, one PR, might prove more valuable for your business than hundreds of other tickets closed.

Remove the KPIs and talk to your team members in 1-on-1s regularly. Give responsibility and trust to your team, and they'll be self-empowering.

Add KPIs and you'll take away their responsibility. Trust will be lost. Probability for innovative solutions will go down, your best developers will leave.

It is all about trust and responsibility. Hire for team spirit, conscientiousness, communicative skills and diversity.

garysahota93• 8 years ago

Agreed! I couldn't have said it better myself. It's really hard to quantify all of this without remembering that no two metrics are interchangeable / comparable.

closeparen• 8 years ago

The phase of a project with a flood of PRs and closed tickets is the easy part. The real work is done before and after (planning and debugging), in people’s heads.

zerop• 8 years ago

I saw this happening. Putting metrics puts pressure and people wont innovate and leave. You can't be innovative when there are metrics on your head.

stcredzero2• 8 years ago

A friend of mine was looking at the metrics for a project he was managing and saw that the bug rate of commits after 4pm was much higher. So then he introduced a new rule against committing after 4pm, resulting in a big drop in the bug rate. He chalked it up to less committing while fatigued and rushing to get things done to go home.

not_kurt_godel• 8 years ago

That's really interesting. Was there any drop in overall productivity/delivery pace as a result? There are definitely times when I wind up cranking out buggier-than-usual code later in the day/night to meet deadlines or get things done that otherwise wouldn't (at least, not on schedule). Sometimes when I timebox myself too much like that and it winds up dragging projects out days or weeks longer than they would have been had I not just blocked out a few hours to get the dang thing working and then cleaned it up afterwards; this is particularly true for complex projects where it can take a large portion of the day simply to build the mental context necessary to actually start executing on what needs to be done. If I cut myself off at a certain arbitrary time, the time spent building mental context earlier is totally wasted.

stcredzero2• 8 years ago

If I cut myself off at a certain arbitrary time, the time spent building mental context earlier is totally wasted.

If what you have to remember to make a certain change means you have to keep 27 things in your head at once to avoid making bugs, you might want to consider re-organizing your code so this number is smaller.

itsdrewmiller• 8 years ago

I think the title of this post is great, but the body doesn't quite deliver. The best content I've found on this subject is this book:

https://www.amazon.com/gp/product/0932633366

It talks about the different between informative and motivational metrics - the former being just for awareness while the latter being intended as explicit targets for employees. While it's easy to bleed the former into the latter, there's nothing inherently wrong with measuring game-able statistics as long as the incentive to muck with them is minimized. Easier said than done, but not impossible.

The article is pretty dismissive of story counting, which is a semi-popular approach to estimation:

https://martinfowler.com/bliki/StoryCounting.html

(Anecdotally, we had someone look at our teams' previous sprints and found that # stories / sprint was more consistent than points per sprint over the time frame they reviewed. YMMV.)

The focus on velocity-related metrics seems smart - Git Prime did some analysis on this and found that frequency of commits was a useful measure:

https://blog.gitprime.com/prolific-engineers-take-small-bite...

I don't see how these are any less game-able than klocs though.

jeanlaf5• 8 years ago

Thanks for the book recommendation! Will definitely read it :).

Concerning Gitprime, I completely agree with you. And they do say that the point is to game their metrics: https://help.gitprime.com/general/cant-you-just-game-the-met...

djcapelis6• 8 years ago

Noooooooo. All of these metrics are a mistake, don't measure any of them. Measure the things that actually matter about your product and whether or not it's getting better or worse.

Measure outcomes. Only outcomes. Try and make sure your metrics capture the real outcomes and not just what you see in a lab. This usually means you need to use field data to get that information.

zapita3• 8 years ago

In larger organizations it can be very difficult for team managers to measure business outcomes of their work directly. Talking to customers is a difficult process, if at all allowed. Information from customers percolates through many layers of support and sales staff, product managers, business stakeholders, project managers etc. By the time the information gets to you it is tainted by bias and politics. Not that it would have helped you much since your objectives are already set by your hierarchy, and again are focused on your ability to execute, not business outcomes.

That may not sound like a very productive environment, but that is the environmental in which most of the world’s software is written. For those engineering managers that have to operate in it, measuring your ability to execute is the best you can do. You might as well do it correctly (which is hard).

And if you’re one of the lucky few working in a less dystopian environment, don’t be too quick to gloat. Enjoy it while it lasts, you’re just one acquisition or reorganization away from joining the rest of us in corporate mediocrity.

djcapelis6• 8 years ago

I work for one of the world’s largest companies. If your organizational structure is a problem for connecting outcomes with engineering then it sounds like you’ve already found a great problem to solve and so go ahead and tackle that instead of spending time on bad metrics!

(Full disclaimer: I am also currently working as one of those project managers you mentioned. If your project managers aren’t communicating clearly about how engineering impacts outcomes, find new project managers.)

vlovich123• 8 years ago

The post looks reasonable to me. The point he's making is that no metric is useful for measuring developer efficacy/productivity & the underlying unstated reason is that dev works tends to be random; problem A is nothing like problem B & there's no way to objectively measure the quality of solution for problem A against another developer's solution.

He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? You can't mean the final shipping SW because that's usually way too late. You want to figure out if your velocity is going to let you hit your ship date well in advance of the actual ship date so that you have time to deploy process fixes/hire more people so that you do hit the ship date.

jeanlaf5• 8 years ago

Indeed. Plus, I would add that measuring the outcome (definitely a must-do, not arguing on this) is more about the product. The article was more focusing on metrics for the engineering organization, how it performs as a team. Could you say that your engineering organization is collaborating effectively, only based on the outcomes?

djcapelis6• 8 years ago

He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for?

None of the proposed metrics capture project health. Test coverage is the closest and that still isn’t great. (Unit tests are important but you can have 100% test coverage and a 0% working product. In fact, this is how every project begins before the first line of code is written.)

No one gives a damn how many pull requests happen or how many days apart they are.

1) Does the software work yet?

1a) How much of it works?

2) Does it do what we want?

2a) .... everytime? Less? How many 9s?

Report on the things that matter. Track the things that matter. Everything else is actively harmful and distracts your teams from doing what matters.

Also, when you ship the software you haven’t finished, you’ve begun. It is absolutely still useful to keep metrics on outcomes and track improving them over time. It is absolutely not “too late” and any planning that assumes software is done after you ship it is flawed. Features can become done, software never is.

austincheney2• 8 years ago

One thing I have noticed when working with juniors is that the wrong things are considered for story point estimation and those story points are the metric of success.

Juniors tend to be overwhelmingly concerned with basic literacy. First its a "does it work" and then it graduates to "how to write the code". This often comprises thinking about tools, frameworks, dependencies, and code style. These aren't deep problems that moves a business forward. These are competency problems that keeps a person employed. This is a distraction that consumes developers with busy work instead of thinking and planning about how to make a product better or how to release faster.

Worse still, too much emphasis on the particulars around the how of it can blow out a 30 minute task to a 2-5 day task with all the unnecessary scaffolding and extra process. This is particularly frustrating when realizing these concerns can be immediately eliminated when the proper automation is in place. It is possible to really get good at spinning on mouse wheels and cranking out those stories a little bit faster, but it still isn't moving the business forward. Nothing forces out strong developers faster than cliches like: "That is the framework way" or "You're reinventing a wheel".

Some better metrics to really measure real world success:

    1. does the change solve a business problem
    2. does it pass the project automation
    3. does it reduce total code instructions
    4. does it close out defects
    5. reduced execution time
    6. improvements to project automation (faster developers)
    7. improvements to documentation

Once the proper metrics are in place the concerns of the juniors are addressed economically. For example, if reducing instruction count is important the code gets leaner and the how of it shifts from code style to leaner code. Code style is important because developers need improved literacy from uniform code expression, but once the goal of expression is leaner code the uniformity shifts from expressive style to optimization concerns.

ascar2• 8 years ago

I'm not sure what you mean with "reducing total code instructions" and "lean code", but I'm not sold on the idea of making code less readable which often happens when you try to reduce code by being smart.

Reading code is mostly way harder than writing it, especially if you read code from someone else. Optimizing for readability is the game changer for maintainability.

* Use expressive function and variable names.

* Have clear consistent structure across your project(s).

* Use simple code where possible.

* Do not write one-liners that do multiple transformations. Each line should have one purpose.

* Document design decisions in code. Don't document "what" your code does, that should be readable by itself, but why you made the decision to do it like that.

* Documentation library style comments for complicated functions helps a ton to know what the function is supposed to do.

* Add tests that specify the expected behavior. This not only helps when changing code, but also when reading.

It's not readable code XOR documentation, it's both. Readable code is the "what", documentation is the "why". Even if the docs get outdated, the why can still be insightful.

Non of the above suggestions usually reduces code, but it will help everyone else, including future-you in 6-months, to work with your code.

codebje• 8 years ago

Less code should not mean either less readable code, or cleverer code. It should just mean ... less code.

Using abstractions, removing dead code, refactoring to share common code paths, etc, all reduce the amount of code you have, and all typically increase the readability of your code.

This doesn't seem at odds with your bullet list for maintainable code.

A useful maxim is: working software is an asset, source code is a liability. More code means more maintenance cost, so accomplishing business outcomes with less code is good, to a point.

austincheney2• 8 years ago

A simple rule is that more instructions mean more execution time and more people time. The first of those is not always true, but the second of those is generally always true.

The total instruction count is the number of instructions from the source code that are actually executed. This isn't the lines of code that people read. For example a loop may comprise 100 iterations and 3 lines of code. That loop will likely result in 300 instructions. If a loop contains a child loop than the instruction count is something like:

    (iterations * overhead) * (iterations * overhead)

I am not a fan of OOP programming because there tends to be a lot of overhead. I understand this overhead helps to bind things together at writing time and share resources at execution time, but at maintenance time years and hundreds of developers later its often a catastrophic mess. I would rather the binding be an explicit call to a reference that I can read opposed to reasoning about flow control that isn't evident from reading the source code.

I am also not a fan of pronouns, which in some languages is the this keyword. If I have to guess what a reference is contextually I am wasting time that could be better spent making a code change.

fouc• 8 years ago

Why would you think less code means the code is less readable?

Does cramming 5 lines of code into 1 line make it less code? No.

Reading code is harder, so the less code there is to read, the better.

quanticle4• 8 years ago

Of the metrics listed, the only one I will say never to use is lines of code. As a very wise manager told me, "Code is a liability. Functionality and business value are assets." Measuring developer productivity by looking at how many lines of code they've written is like measuring the health of a business by how much money it's spending.

If you can delete code and retain the same level of functionality (for example, because you found a library that will automate much of the task), then that's a huge win. Every piece of code that the team writes is one more thing that has to be unit tested, integration tested, documented, and maintained in the face of future changes. If they can offload that work to a library, or find a way to accomplish the same task by using existing code or by reconfiguring the application, that's almost always preferable to writing new code to accomplish a particular business need.

Of course, one can go too far in the other direction. It's easy to fetishize using external libraries for everything (like we see so often in the JS ecosystem). But that's where I would argue that the analogy of treating code like money shines. Sometimes it is worth spending money to have something custom made for your business, rather than buying an off-the-shelf component. Similarly, sometimes it is worth writing code to do something custom, rather than integrating a third party library, or reconfiguring an existing application. But that choice should be consciously made. Rewarding developers for lines of code written moves that choice out of the realm of conscious decision-making and into the realm of reflex. They get paid for more lines of code, so they're always going to reach for the solution that results in more lines of code, regardless of whether that code is necessary or appropriate.

jeanlaf5• 8 years ago

Completely agree! Love the quote "Code is a liability. Functionality and business value are assets."

scarejunba• 8 years ago

The Goal has a bunch of insights translatable to software engineering. This is one of them.

C1sc0cat• 8 years ago

Well some times a lot of effort is expended on "functionality" when its not a good use of time.

I worked at one company where a level 2 Engineer was bucking for promotion and spent almost a year (wall time) and ten plus man years redeveloping a system in Oracle that was functionality but added no real value.

TheCoelacanth• 8 years ago

It is reasonable to use as an indicator of scenarios where you should pay closer attention.

If someone produces an unusually high amount of code, that might indicate that they are producing a lot of functionality, but it might also indicate that they are churning out mounds of crappy boilerplate that should be abstracted better.

If someone is producing an unusually small amount of code, that might indicate that they are very unproductive, but it might also indicate that they are very efficient and are finding simpler ways to express functionality.

All of these things are things that you want to be aware of. The mistake is treating metrics like this as something that is correlated directly with performance instead of just tools to help you understand the bigger picture.

eksemplar3• 8 years ago

I agree that lines of code is a terrible measurement of productivity.

I don’t agree that short is always better. Sometimes it’s better to write the code so that it’s easier to understand than to use smart tricks to make it short, because the next person looking at the code might not be on to your tricks. You can argue both ways on this of course, but I prefer clean and easily understandable because I’ve seen what happens when you don’t do that and a junior programmer has to fix something 15 years later.

I also disagree that libraries are better. You seem wary of JS, and that’s good, but no libraries are magical and none of them can be trusted to do something for you, unless you look under the hood.

Especially not open source libraries. If you don’t know what is going on inside the libraries you use, then you’re heading for a world of pain. Of course some libraries are better than others, but using them blindly is wrong. And if you have to read up on them, then they aren’t really quick fixes.

We had a developer who used a library to handle decimals. We have our own library for this, that we’ve build because almost none of the open source libraries for decimals are very good at handling memory allocation and floats and computers aren’t 0.00000000000 they are 0.00000001436 etc. But our developer didn’t know we had our own library for this, so he used a public one. One that uses around 10 other libraries, one of which didn’t handle memory allocation the way you’d want.

So suddenly our decimals were wrong, but the computer didn’t know they were wrong, because technically the library worked it just wasn’t specific enough. And it cost us a lot of money to clean things up afterwards.

This is the extreme example, but it’s never been more relevant than it is today. Especially not in a world where less and less programmers seem to actually know how computers or even compilers work.

jgtrosh2• 8 years ago

I assume that's “especially not-open-source libraries” and not “especially not open-source libraries”

MrGilbert• 8 years ago

But our developer didn’t know we had our own library for this [...]

I guess you also addressed this problem, right?

Distributing knowledge inside the company is an issue, that's even more crucial in software engineering.

quanticle4• 8 years ago

I also disagree that libraries are better. You seem wary of JS, and that’s good, but no libraries are magical and none of them can be trusted to do something for you, unless you look under the hood.

I wouldn't go that far. You use glibc, right? Have you looked under its hood? If you're writing Java, have you really looked inside Spring or Hibernate? In the .Net world, how many people have really looked at the ASP.Net source code? How many Python developers look at the code for Requests? Even in Javascript, I can't remember the last time I had to crack open jQuery and take a look at its internals.

rejschaap• 8 years ago

As a very wise manager told me, "Code is a liability. Functionality and business value are assets."

I'm not sure he is drawing the line in the right place. Business value requires functionality which requires code. Business value would be an asset for obvious reasons. Functionality and code would be liabilities. Functionality and code that doesn't provide business value is waste. You want to minimize both functionality and code. Maybe it is just the analogy breaking down though.

quanticle4• 8 years ago

Business value requires functionality which requires code.

I don't think business value necessarily requires functionality. Business value is generated by code operating inside a specific business process. Only part of that process is software; the rest of it exists in the minds and the actions of the people interacting with the software. In some cases, it's easier to change the business process than it is to change the software.

ignoramous• 8 years ago

As a very wise manager told me, "Code is a liability. Functionality and business value are assets.

Obligatory: https://www.folklore.org/StoryView.py?story=Negative_2000_Li...

abernard15• 8 years ago

If you don’t include story points or some equivalent, this might be the most misleading metric you could use.

I'm sorry, but if you think "story points" is something useful, your organization is already doomed.

Story points is an inflationary currency. When your developers realize that somebody important in the company cares about said currency, they will make sure that their work has more of these "story points" in "sprint planning".

skybrian2• 8 years ago

Okay, how does this doom a project? It's just an arbitrary score to begin with, so what's wrong with inflation?

abernard15• 8 years ago

Because now "real" velocity goes down.

There's a negative Pavlovian reaction developers have every sprint when 1) "commitment" was not achieved, or 2) story points decreased. To avoid that pain, developers will pad points. This means that when things go according to plan, large sections of a sprint will have no work or people will try to appear busy.

The easiest way to avoid that gamesmanship is to not measure things that don't matter, which is all inputs. What matters is outcomes, which can be in the form of money, usage, satisfaction, etc.

Jtsummers3• 8 years ago

Maybe it's just because I'm more confident that I won't get fired, but I've not had any trouble telling the higher ups to stop asking about story points.

Story points are a way for your teams to improve themselves. If they don't understand that, then teach them. And never report the points (directly) to anyone above the project lead or perhaps one level above that. Any higher and they are too disconnected from both the team and the work to understand what they're seeing.

The only thing that higher ups should care about are trends up and down. Your team, after say 6 months, should be pretty steady in their performance if they're using story points correctly. Downward trends should be identified by them and their project lead early and addressed, and if it's an issue that may impact other teams they should communicate that up and out. Upward trends should be celebrated but also examined to see if some change could be duplicated elsewhere (an improvement to your build configuration resulting in faster feedback, adoption of a new development tool or technique, etc.).

abernard15• 8 years ago

Story points are a way for your teams to improve themselves.

These points have nothing to do with actual value produced to the business, or the developer's worth / skill. It's also very easy for a developer to begin to inflate the estimation of tickets they believe they will work on, creating a false sense of confidence.

Alternatively, if people work at a "shop" where tickets are so fungible that people pick up whatever the top ticket on the board is, it tells me that they are working on low-value, repeatable things where expertise is unnecessary.

It's also not clear that increasing story points (even if they were somehow related to work / hours perfectly) would be useful. I've seen teams drop down to do minimal "work" for months and produce tremendous value because they were reorienting the product to do things successfully. An organization that looked at that labor input, sans objective outcomes in the product, would be doing themselves a disservice.

Story points, tickets closed, lines of code: ultimately all of these are a belief in the labor theory of value. But that's the thing. I don't want labor, I want value, and the tragedy that is input-focused measurement is just due to it's easier to do than measure product outcomes. It's always more convenient to say "We did 24 story points this week" than "Our users hate our software, and here's how we know it."

ausjke2• 8 years ago

Is Agile solving this problem now? or has Agile gotten out of fashion?

    * release often, release early, each release should function though with less features.
    * break large goal to small steps, one step at a time.
    * sprint every 2 weeks or 4 weeks.
    * 15-min stand-up every other day
    * etc,etc

I read some agile books in the past, and now I'm experiencing real Agile in sw development for the first time, so far I found it is quite effective comparing to my past projects, assuming you have some great scrum master and project-owner that are really good at the balance of arts.

wpietri• 8 years ago

If people actually do it, it helps a lot.

In my experience, if business stakeholders are seeing real value shipped to customers every week (or 2 if you must) then they don't get very antsy. Especially so if they are using a release-early, release-often approach to the product, where they see a project as an opportunity to explore what customers really want.

The problem starts to come in when the feedback loops are long. If they ask for something in January and don't see results in June, then they've got 6 months to wig out. And one of the ways they'll try to manage their fear is by getting up in developers' business by imposing all sorts of metrics.

ausjke2• 8 years ago

in our case it's a long term internal project, it's embedded system development, we release bi-weekly internally before it reaches 1.0 for customers.

embedded product is notoriously hard to manage and I found Agile process actually helped quite a bit, via Scrum's 'story points' method.

Jtsummers3• 8 years ago

First, Agile is a collection of ideas and principles, and not a monolithic thing.

Scrum (what you mostly describe) does not address this, really. Except that it attempts to reduce metrics down to a primary metric (there are others): Story Points. These are the primary concern of a Scrum team, and are developed by the team members as an estimate of the size of the stories (tasks) as they see them. These are not given by anyone else. These are not used by anyone else. The team then evaluates themselves (if they do this correctly) by constructing a control chart [0] to see how they're performing. If they trend down, they identify the cause and share the reason with other teams so that others with a similar cause can address it more readily. If they trend up, they identify the cause and share so that others can attempt to duplicate the success.

Scrum does not address how to do personnel performance evaluations so it doesn't stop management from trying to use metrics for the purposes described in this article. For example, story points (if used as an evaluation metric for personnel) become useless as people will just inflate their numbers in a continual game of oneupmanship. At which point a large amount of the value of Scrum is lost.

The Lean school of Agile is a philosophical descendant of Deming's work in Japan post WWII and the Toyota Production System (called Lean as it spread). This school of thought does address the issue discussed in the article in that Deming's views have you reducing your focus on individual performance and increasing focus on performance of teams and projects (title of the article is related to this). Deming and others recognize that the system in which work is performed effects the outcome more than the workers themselves (NB: His focus was largely on manufacturing, but this works in service industries and to some extent in our discipline as well). Since the view in that school is on the system, it does address the problems described in the article (if you get management to understand it). So by focusing on the project level (really you should probably go higher, department or even enterprise level) you can understand where problems are occurring, where there is waste worth cutting or addressing, etc.

That said, I do not recall seeing some of those points I made (gathered from Deming and other sources) in any of the books I've read on Lean as it relates to software. It could just be that I have forgotten or missed that information (may have glossed over it as I had already internalized it) or was in sources that I haven't read yet.

[0] https://en.wikipedia.org/wiki/Control_chart

Don't measure developers, measure projects

Related Stories