Statistical Process Control: A Practitioner's Guide (2022)

https://two-wrongs.com/statistical-process-control-a-practitioners-guide.html

by jpalomaki • 2 years ago

83 46 2 years ago

roenxi6• 2 years ago

A few related observations:

Software development is not a stable process. Either a team is always building new things in which case there isn't a consistent process to measure, or there is a see-saw as people release new features, deal with the bugs in the features, then go back to building - that isn't a controlled process, it is going to oscillate in statistically weird ways.

If SPC is applied to bugs, it will be monitoring the relevant manager's habits. That is to say, if you show me a nice in-control timeseries of bug resolution, all that says is when a bug blows out horribly the manager splits it into 2x tickets or something similar. It isn't necessarily a bad outcome (small tickets are happy tickets and gently stressing managers is a good idea) - but don't expect the devs to behave differently.

It is good to have a grounding in SPC, just don't try to apply it to every timeseries that you see. Bugs are a timeseries, but they aren't expected to be a controlled process so SPC's assumptions break down and the logic doesn't work. If it does work, it is probably measuring something other than the software development aspect of the process.

jacques_chester5• 2 years ago

Software development is not a stable process.

This causes confusion in my experience. "Stable" doesn't mean what it does in casual usage. It means that you have a mean and variance that are not varying over time (and there are ways to alert if they are). A stable process can lurch around like a drunken sailor and still be considered stable for SPC purposes.

I think bugs should be stable if you normalize it by a size metric. So bugs/SLOC or bugs/story points.

roenxi6• 2 years ago

They aren't stable. Bugs, viewed as a timeseries, don't have a steady mean and standard deviation. There is a feedback loop where those measures change depending on the maintenance and development strategy of the software, and priorities are going to shift suddenly. If there is a fix-small-bugs month for example, or a fix-this-client's-bugs push, or a fix-the-bugs-in-the-new-feature campaign. The statistical properties of all these are not identical.

And once you've normalised for story points, what you're measuring is stability of the story point allocations. Which is to say, different managers will get different means and standard deviations even if the devs change nothing about their work habits. It isn't measuring the devs, software quality or even real number of bugs; it is measuring the ticket creation process.

As long as you know it isn't measuring the software, all is fine. But people do get confused.

a1369209993• 2 years ago

"Stable" [...] means that you have a mean and variance

And software development, as a process, does not have a (finite) mean or variance.

I think bugs should be stable if you normalize it by a size metric.

Bugs approximately follow a heavy-tailed distribution, such that as the number of samples increases, the empirical average work to fix (and possibly other measures of severity) does not converge to any finite value, but instead increases without bound. (I think roughly logarithmicly, but don't quote me on that.)

In particular, it doesn't satisfy the requirements of the central limit theorem, so a lot of statistical techniques work poorly or not at all on software projects that are large enough, individually, to do statistics on (as opposed to doing statistics on populations of software projects, which seems to mostly work, usually).

kqr10• 2 years ago

Things in software development that have been stable in my experience:

- Weekly deploy count

- Weekly growth in lines of code

- Fraction of candidates hired

- Weekly number of bug fixes

- Weekly number of problems with a third party collaborator

- Many internal metrics generated by the software, reflecting usage etc.

- Weekly number of consultant hours required

- Monthly growth of feature flag count

- Length of standup

- Time required to complete "small" tasks (i.e. those that don't involve novelty)

- Length of successful build in CI

- Proportion of CI builds that fail at least one test

- Growth of number of tasks in backlog

I could go on. The point is that while much of the value of product development comes from novelty and variation, there are many parts of the process that remain the same from week to week.

Figuring out what these parts are and getting variation out of them allows the developer to

(a) focus creativity on systematically solving process problems instead of doing it ad hoc, and

(b) let the process recede into the background and focus creativity on creating end-user value.

roenxi6• 2 years ago

Your heartbeat and breaths/week are also quite consistent. The issue here is that you have to have a reasonable theory of why variance in the metrics you're tracking will destroy value. And that means actual value, not whinging that standup goes too long. I like a short standup as much as anyone, but if that is a material driver of value destruction then your organisation is not ready for statistical quality control. Plus it probably goes over time consistently.

And once you start looking at the value, the lesson of software is that high variance activity is often the value add. It is the week(s) where someone implements pivot tables in Excel that creates probably billions of dollars in value over all of humanity. If that turns up as a statistical anomaly in the metrics because their line manager didn't bug them to fix bugs that week, that is a problem with the metrics not the programmer.

This isn't a Toyota production line (if you're interested in the history of this, that is no random example) where value is uniformly created with each car and optimising the daily process down to degree n creates value. This is software. The value isn't created in the same way and these tools are not powerful in driving value add decisions. Variance is untidy but by no means an enemy. It must be managed case by case in context.

jacques_chester5• 2 years ago

This is one of my favorite introductions to this topic area, especially since it gets away from the dominance of manufacturing applications for SPC.

3abiton• 2 years ago

I tried to rub shoulders with this topics, mainly for application in software observability, yet I still fail to see it's relevance compared to advanced methods.

jacques_chester5• 2 years ago

Could you elaborate? I would love to learn more.

fjkdlsjflkds8• 2 years ago

I stopped reading at this point:

Roughly half of your measurements will be above average, and the other half below it.

This is simply not true for most definitions of "average" (i.e., for all definitions of "average" except the median).

Example:

(data <- c(1:10,100))

[1] 1 2 3 4 5 6 7 8 9 10 100

sum(as.numeric(data > mean(data)))/length(data)

[1] 0.09090909

A 90/10 split is hardly "roughly half of your measurements".

kqr10• 2 years ago

Would it help you get past that point if it had said "of a stable process"?

fjkdlsjflkds8• 2 years ago

Well... what do you mean by "a stable process" in this context?

Let's try repeating the same example, but now drawing samples from a fixed distribution (in this case, a log-normal distribution):

data <- exp(rnorm(100))
sum(as.numeric(data > mean(data)))/length(data)

[1] 0.32

So, again, quite far from a 50/50 split, even though I am assuming a stable/fixed data generation process.

In general, it would help if statistical subjects are not presented in a careless way (i.e., containing things which are obviously not true). I would suggest at least adding an "assuming a symmetrical distribution" (so that at least your claim is approximately correct under the arithmetic average and for bounded variance distributions).

EDIT: If by "a stable process" you mean "a process following a stable distribution"... then, no, it doesn't help.

Here's an example with samples drawn from a Lévy distribution (which is a stable distribution):

data <- rmutil::rlevy(100)
sum(as.numeric(data > mean(data)))/length(data)

[1] 0.07

lifeisstillgood• 2 years ago

For interest: https://en.m.wikipedia.org/wiki/Stable_process

And

“Until we can define what a stable process is, we are doomed to argue forever all use of any statistical metric. For the love of a all science, please help!”

https://www.isixsigma.com/variation/what-stable-process/

shadowsun73• 2 years ago

A couple of quick notes, from someone who has actually put this to practice — and in a non-manufacturing context, to boot!

(From a brief reading of this thread, it seems like kqr, jacques_chester, and I are the only ones who have put this to practice in non-manufacturing contexts — though correct me if I'm wrong.)

The bulk of the debate in this HN thread seems to be centred around what is or isn't a 'stable process'. I think this is partially a terminology issue, which Donald Wheeler called out in the appendix of Understanding Variation. He recommends not using words like 'stable' or 'in-control', or even 'special cause variation', as the words are confusing ... and in his experience lead people to unfruitful discussions.

Instead, he suggests:

- Instead of calling this 'Statistical Process Control', call this 'Methods of Continual Improvement'

- Use the term 'routine variation' and 'exceptional variation' whenever possible. In practice, I tend to use 'special variation' in discussion, not 'exceptional variation', simply because it's easier to say.

- Use the term 'process behaviour chart' instead of 'process control chart' — we use these charts to characterising the behaviour of a process, not merely to 'control' it.

- Use 'predictable process' and 'unpredictable process' (instead of 'stable'/'in-control' vs 'unstable'/'out-of-control' processes) because these are more reflective of the process behaviours. (e.g. a predictable process should reliably show us data between two limit lines).

Using this terminology, the right question to ask is: are there processes in software development that display routine variation? And the answer is yes, absolutely. kqr has given a list in this comment: https://news.ycombinator.com/item?id=39638491

In my experience, people who haven't actually tried to apply SPC techniques outside of manufacturing do not typically have a good sense for what kinds of processes display routine variation. I would urge you to see for yourself: collect data, and then plot it on an XmR chart. It usually takes you only a couple of seconds to see if it does or does not apply — at which point you may discard the chart if you do not find it useful. But you should discover that a surprisingly large chunk of processes do display some form of routine variation. (Source: I've taught this to a handful of folk by now — in various marketing/sales and software engineering roles —and they typically find some way to use XmR charts relatively quickly within their work domains).

[Note: this 'XmR charts are surprisingly useful' is actually one of the major themes in Wheeler's Making Sense of Data — which was written specifically for usage in non-manufacturing contexts; the subtitle of the book is 'SPC for the Service Sector'. You should buy that book if you are serious about application!]

I realise that a bigger challenge with getting SPC adopted is as follows: why should I even use these techniques? What benefits might there be for me? If you don't think SPC is a powerful toolkit, you won't be bothered to look past the janky terminology or the weird statistics.

So here's my pitch: every Wednesday morning, Amazon's leaders get together to go through 400-500 metrics within one hour. This is the Amazon-style Weekly Business Review, or WBR. The WBR draws directly from SPC (early Amazon exec Colin Bryar told me that the WBR is but a 'process control tool' ... and the truth is that it stems from the same style of thinking that gives you the process behaviour chart). What is it good for? Well, the WBR helps Amazon's leaders build a shared causal model of their business, at which point they may loop on that model to turn the screws on their competition and to drive them out of business.

But in order to understand and implement the WBR, you must first understand some of the ideas of SPC.

If that whets your interest, here is a 9000 word essay I wrote to do exactly that, which stems from 1.5 years of personal research, and then practice, and then bad attempts at teaching it to other startup operator friends: https://commoncog.com/becoming-data-driven-first-principles/

I don't get into it too much, but the essay calls out various other applications of these ideas, amongst them the Toyota Production System (which was bootstrapped off a combination of ideas taught by W Edwards Deming — including the SPC theory of variation), Koch Industries's rise to powerful conglomerate, Iams pet foods, etc etc.

hef198982• 2 years ago

Very interesting to get the perspective of someone who did thisbin a non-manufacturing evironment. One interesting bit, for someone like me who knows SPC from manufacturing related processes, are the discussions around what a stable process is. Because I cannot remember a single of those discussions ever in manfucturing related fields. Intriguiging, especially since on HN sometimes discussion miss the point by turning into disputes about the exact definition of a term, something that sounds very similar to the "misunderstandings" about stuff like special-casue variation you described.

Edit: Fully agree on the Amazon style WBR, what you said is exactly what is happening at Amazon. Daily during Q4 peak for a large enough subset of metrics.

roenxi6• 2 years ago

(From a brief reading of this thread, it seems like kqr, jacques_chester, and I are the only ones who have put this to practice in non-manufacturing contexts — though correct me if I'm wrong.)

And roenxi.

So here's my pitch: every Wednesday morning, Amazon's leaders get together to go through 400-500 metrics within one hour.

Amazon's core value proposition is they maintain a large and very physical fleet of machines that they rent out. With serious standards for up-time that they can take real pride in.

They don't sell themselves as a software house. I'm sure they have tentacles everywhere and they aren't bad at it (if anything I'd expect them to be pretty good on a given project), but they've greatly benefited from using other people's software - they don't have their own DB for example, they reuse others and have a couple of PostgreSQL forks for more at-scale use cases.

I'm sure they get huge value from SPC (anything physical generally benefits from it), and I'm sure they use SPC for software out of reflex; but it doesn't follow that it is driving productive behaviour in the software branch of the business. A fleet of ~infinite servers benefits from controlling 400 metrics. Software development does not.

hef198982• 2 years ago

OP was talking about the Retail and Logistics side of Amazon, from what I can tell...

shadowsun73• 2 years ago

What would you say if I told you Bryar has lots of stories of this style of thinking applied in early Amazon? This is pre-AWS Amazon, mind you — where they were trying to figure out how to build e-commerce web software at scale, from scratch. Granted, the bulk of their process control was directed at customer-facing controllable input metrics, but the software engineers were as much a part of it as the operational folks.

I don't have his permission to tell some of these stories, but Eugene Wei has some hints of it here: https://www.eugenewei.com/blog/2017/11/13/remove-the-legend

(To be fair to you, you are adamant that SPC does not apply to software development — which I take to mean measuring the productivity or act of building software. And I think we are all in agreement there! (That said, like kqr and jacques_chester, I want to believe that this has not been sufficiently explored) But it's not true that SPC has no place in software development — one way I've used this is that because XmR charts detect changes in variation, you can use it in a customer-facing software context to see if a feature change has resulted in user behaviour change without running an A/B test. Naturally, it makes sense to have the software engineer be responsible for observing this behaviour change themselves, since XmR charts are easy enough for the layman to use, and it gives them a sense of ownership for the feature or change. Some detail (on usage vs A/B tests) here: https://commoncog.com/two-types-of-data-analysis/)

nhinck3• 2 years ago

I've applied it to adverse events in health care. Works quite well as a trigger for investigation.

ilayn• 2 years ago

Another post to drive a control theory person up the wall.

BizOpsZen2• 2 years ago

Came across this on twitter. Here's why I think SPC is related to Software Development (and Agile concepts, particular the burndown charts) more generally:

To be clear, they are related but not the same use case. IMO, both Agile and SPC leverage the same insight: variation is inevitable: what matters is not that it exists, but how you deal with it.

With SPC, you are establishing a normal variation so that you can identify abnormal activity that is warrants further investigation.

With Agile you're not really looking for outliers per se, it's more that you want to get to a place where your "normal" variation is a much smaller range. Because a smaller range leads to better quality and more output:

Variation in the software dev context is the difference between your estimates and the actual work required to deliver a feature, etc. High variation means you're constantly in a rush, need to cut corners, need to cut scope, etc.

This has a lot of downstream impacts in terms of quality but also in the actual scope of what you can deliver. In short, you need to spend more time fixing bigger problems.

Less variation means smaller problems and less time spent fixing --> more time is allocated to new feature development.

(and separate topic, but variation in Software dev has a special property where it only accrues in the "takes longer" side vs. the "take less time" direction. You never "make up time" because something is quicker than your estimate. see note below.

So the burndown chart is less about enabling you to see outliers, more providing visibility to the variation so that you can work towards making it smaller. If you're constantly loading work in the end of the sprint, you have a problem with the scoping process.

How does that track back to Agile?

One the key elements of Agile process is breaking work down into smaller batches --> and Breaking things down into smaller batches is* they key mechanism to reducing variability.

NOTE: *Software pretty much only takes longer than expected because there is high visibility into the fastest something can be done, but very little visibility into the unexpected things that can add scope to the project. So it's extremely rare for something to happen that make it take less effort than your estimates, but very common for things to add scope.

It's similar to estimating how long it will take to drive somewhere: you can get a pretty accurate sense of the fastest it will take based on distance and speed. But the things that extend the duration of the trip, like a car accident or unexpected road work, are just much more unpredictable. So if you were to plot that variation on a chart, you only see it move in one direction.

Statistical Process Control: A Practitioner's Guide (2022)

Related Stories