geekjock

One of the main challenges when dealing with technical debt has been the lack of a way to measure it. To help overcome that problem, CISQ/OMG led the development of an Automated Technical Debt (ATD) measurement standard, which is currently being updated with a new version expected in 2023.

I'm highly skeptical about the tech debt measurement algorithm this article purports to be developing.

Google researchers recently published a paper on their attempts to measure technical debt.

They tested 117 metrics that were proposed as potential indicators.

Regressions were used to test each metric to see whether it could predict an engineer’s perceptions of technical debt.

No single metric or combination of metrics were found to be valid indicators.

"Lines of code is a much better proxy than reported productivity."

Researchers who specialize in this field largely disagree with this. Here's a round-up: https://getdx.com/blog/measuring-developer-activity/

See this previous paper: https://queue.acm.org/detail.cfm?id=3595878

"I'm a bit pessimistic that much of this research is being driven by large orgs in collaboration with researchers who aren't sufficiently independent."

I'm one of the co-authors of the study. I think your sentiment is valid but what you describe is true for most fields of research: conflicts of interest can be a problem.

I can attest to the fact that the researchers behind this study have extensive backgrounds in academic research and hold themselves to high standards. If nothing else, not doing so risks putting individual reputations on the line.

Makes sense - I agree with you. My mistake on assuming you were referring to metrics like the ones I listed.

Developer experience encompasses how developers feel about, think about, and value their work.9 In prior research, we identified more than 25 sociotechnical factors that affect DevEx. For example, interruptions, unrealistic deadlines, and friction in development tools negatively affect DevEx, while having clear tasks, well-organized code, and pain-free releases improve it.

https://queue.acm.org/detail.cfm?id=3595878

Agreed with this. We're working on it!

"Are there any plans to figure out objective ways to measure productivity"

You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies.

Here are some examples: https://newsletter.pragmaticengineer.com/p/measuring-develop...

"... and what distinguishes 'good devex' from 'bad devex'"

Yes to this. This is an ongoing effort - we have two previous journal papers that touch on this which may be of interest to you:

- https://getdx.com/research/devex-what-actually-drives-produc...

- https://getdx.com/research/conceptual-framework-for-develope...

Ever seen developers that are frustrated by technical debt and whose business leaders don't seem to care? This study is aimed to help with that.

"You could look at literally any objective measure to proxy actual productivity and be better off than this"

It's fairly well-established in research (and in practice) that there is no objective measure of developer productivity. Metrics like lines of code, number of pull requests, velocity points are incredibly poor proxies.

+1

If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.

https://queue.acm.org/detail.cfm?id=3639443

I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!

Accelerate - this book has become an excuse for managers to spend outrageous money implementing metrics like lead time and deployment frequency to measure teams, whereas the book actually advises something very different.

Do you know of specific examples of companies that offer this?

I'm a developer and am right there with you. But if you're a decades-old corporation with 10,000 engineers, you need some set of signals to help guide improvements to tools and processes, right? This benefits developers, and there should be a set of signals to enable this.

"What should we measure to improve developer productivity?" is a decades-old problem for leaders with no clear solution.

There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach.

Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity.

A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com

Shoot me an email at abinoda@getdx.com, would love to connect.

Right now we're focused on interviewing people in internal-facing roles, but we're exploring doing a series that explores interesting vendors and solutions.

I'm the host of the Engineering Enablement podcast: https://getdx.com/engineering-enablement-podcast

We interview developers and leaders who work on "platform" teams (e.g. devex, devprod, infrastructure) that focus on making developers happier and more productive. If you're interested in this kind of work, or the subject of developer productivity, I think you'd enjoy checking it out.

A couple recent episodes include an interview with a former Dropbox engineer about how they approached measuring productivity, and a conversation with a couple of engineers from LinkedIn who've built a system for gathering real-time feedback about their internal developer tools.

Direct links if helpful:

- Spotify: https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D

- Apple: https://podcasts.apple.com/us/podcast/the-engineering-enable...

I write and share summaries of papers related to developer productivity here: https://abinoda.substack.com

Two of my favorites so far are https://homepages.dcc.ufmg.br/~figueiredo/disciplinas/papers... and https://faculty.washington.edu/ajko/papers/Li2019WhatDisting...

The hard truth is that using velocity to measure performance will guarantee that developers bias their estimates to ensure they make themselves look good, thereby making your "estimates" useless.

Anyone who's built software knows that estimates are guesses and are bound to change once the work actually begins (not to mention if scope changes midway through).

Additionally, velocity does not "count" work happening outside of tickets, e.g., helping with a customer support issue, assisting another developer with legacy code, reviewing other people's work, spending time on feature planning.

"Using velocity metrics is like looking for your lost keys under the streetlamp because that's where the light is brightest."

^ 100% agree.

Tracking output metrics (including LOC, # of PRs, velocity points) doesn't help you identify or fix poor tooling, culture, or processes.

Metrics like cycle time get a little bit closer, but still fail to account for team context or surface root causes.

Check out this summary of a peer-reviewed paper titled “What Distinguishes Great Software Engineers?”

https://abinoda.substack.com/p/great-engineers

Julian reached out to me respectfully to resolve the issue. Nice guy, no hard feelings. I understand the startup grind.

Article author here — thanks for raising this. My article here was published in April 2021 and repurposed from my GitHub Universe talk given in 2019: https://www.youtube.com/watch?v=cRJZldsHS3c

Julian's article was published May 2021, one month after my Leaddev article, so hopefully it's pretty self-evident that Julian plagiarized my article pretty blatantly. I don't know Julian but I've heard from my former customers that his company has copied a lot of the work I did for my previous company, Pull Panda: https://github.blog/2019-06-17-github-acquires-pull-panda/

How do you handle PRs that are for release trains/branches?

I think Capital One Spark was shut down.

Yep, Rails

Thanks for the shout-out David! If anyone's interested here's the link to the website: https://pullreminders.com

I run http://pullreminders.com

I'm a solo founder (pullreminders.com) and have been at it for about a year and a half. There are definitely a lot of highs and lows but overall I enjoy the way my work feels very personal and autonomous.

I love this quote by Derek Sivers:

Nobody gives a novelist shit for writing alone. But an entrepreneur, programmer, or musician is expected to collaborate. I disagree, for me. I prefer the life of a novelist, whether I’m writing code, music, or systems.