The Toyota Way at Codeweavers

https://www.infoq.com/news/2017/10/toyota-way-codeweavers

by fagnerbrack • 9 years ago

42 31 9 years ago

hencq2• 9 years ago

Whilst on the surface it is all about manufacturing cars if you dig a little deeper, particularly into Taiichi Ohno’s work, you soon start to see how these rules can be applied to all kinds of manufacturing and we saw many parallels in what we were reading about the Toyota Production System and the way we delivered our software.

I found this really interesting. I completely agree that the principles behind the Toyota Production System can be applied to many other disciplines than just car manufacturing. I wonder if Toyota itself actually applies its own principles when it comes to software development. When their software got audited as part of the unintended acceleration case [1] it was described as spaghetti code and having generally poor engineering practices.

I'd be interested to know why this seeming discrepancy exists between Toyota's quality in hardware vs software.

[1] http://www.safetyresearch.net/blog/articles/toyota-unintende...

maxaf• 9 years ago

The discrepancy in quality probably indicates a deeper deficit of maturity across the software engineering discipline in general.

alexanderstears• 9 years ago

My informed opinion is that Toyota's software is robust but crappy.

The software you would use to configure the parameters in the car (e.g sensitivity of headlights turning on, whether or not it'll chime when you're not wearing your seatbelt) hasn't even been ported to 64 bits yet. It seems to have been written by one of their O.E.Ms (Denso) and it seems like the cars have used many of the same design patterns since the 90s.

Look no further than the infotainment system, it's always a generation behind other automakers and we all know that most automakers don't set the high bar of consumer technology.

uncle_bob• 9 years ago

Hardware can usually be spec'ed reasonably accurately from the start. "The Toyota Way" can incrementally optimize building against a specification, from the bottom up, starting with smaller details.

Software however is all but impossible to spec from the beginning, or perhaps ever at all... so with only working with the smaller details, the larger picture is lost. At any time, any team, is instructed to stop a problem and immediately fix it, it's band-aids on top of band-aids.

Having multiple teams working on overlapping projects, with little communication leads to the spaghetti code that was mentioned.

otakucode2• 9 years ago

You can spec hardware, but when the company goes behind the engineering teams back and lies and saves a few pennies per car by not putting error-correcting RAM on the things they say they will... you're kind of left hanging on that. And that's how Toyota works or at least worked when building the Corolla (just one of the best selling cars ever).

Much of the code being dealt with by Toyota (at least that subject to the lawsuit and so made available for analysis) was embedded code which is a bit more amenable to very deliberate design, testing, and verification. NASA, for instance, has a long history of producing extremely reliable software, even proving the software correct at times. Would any private corporation in the world ever resort to following such a practice? Never. It would quadruple their software spending and, maybe worse, they would have to listen to their engineers. When the business wants to ship a project, if the engineer says 'its not ready yet', if its a bridge and they go ahead someone in a suit goes to prison. If it involves software, everyone goes home happy except the dead customers.

user5994461• 9 years ago

>> Software however is all but impossible to spec from the beginning, or perhaps ever at all... so with only working with the smaller details, the larger picture is lost. At any time, any team, is instructed to stop a problem and immediately fix it, it's band-aids on top of band-aids.

Embedded software can be spec most of the time and it should be.

blacksmith_tb• 9 years ago

That's certainly true of the software in my 2017 Prius Prime (which was mostly written by Denso for Toyota - and it does at least display a proper GPL notice, so there's that to say for it, but not much else) However Toyota does finally seem to be making some progress on the software front with the AGL rollout[1] which will also at last bring Android Auto and Carplay support (just not to all of us sad owners of older models).

1: https://www.engadget.com/2017/05/31/toyota-automotive-grade-...

mmjaa• 9 years ago

I've been in software since the 80's. I've watched all the fads and cults rise and fall .. From General Electric to Kanban and back again.

Honestly, over the decades, there are very specific patterns in all the ism's, and its the recognition of these patterns which each new generation of ignoramus decides to make their own and turn into the Hot New Cool Shit.

So I feel its really necessary to point out that while, with the Toyota Method, there is very definitely a refinement of techniques that have been well-worn, and proven, indeed in many markets around the world - there are still parts of the human psyche of productivity which remain unexplored.

One of those aspects, I feel, is the obfuscation and neglect which occurs with every wave of computer culture.

We kind of don't really work on the reasons for obfuscation - because obviously this is not productive. Productivity means, getting the shit done, in a way that increases value - for the individual producers as well as the owners of the means of production.

Toyota works. The proof is in the fact that it is one of the biggest and best brands in the world.

But what doesn't work is the occlusion of methodology that occurs, which makes the revelation here in the 21st Century of time-worn, centuries-old methodologies, relevant.

We get this way because we decide to occlude, forget, neglect, and ultimately .. reject .. the ways of the masters. How many of us read this article, gave it a cursory glance, and decided to close the tab, for whatever reason: "its too preachy", "its a culture thing", etc. And what number of us actually decided to incorporate the Ohnno way into our thinking, for the simple fact that HN gave us the clue?

Really, what I would like people to understand is that there is a desultory, occluding factor, to these different 'ways'. The moment the marketing people take over a working technology, they decide to brand-differentiate, and this means forgetting the roots of it all. "Nobody thinks of the sugar; all we see is the cola" can only happen if we actually act on the basis of ignorance.

The point being, the Toyota way seems - like so many other ways - to have the principle purpose of fighting ignorance. Ignorance of oneself, ignorance of others, of the customer/user, of the managers, and so on. Ultimately, the Zen approach is, remember that you will always be forgetting, and express your current reality in such a way to defeat that. No successful brand is ever forgotten; no successful methodology can happen in a vacuum. Yet: it requires such a vacuum before you can find your own handle on it, and ultimately apply it with success.

"Applying it with failure", on the other hand, is a sure-fire to re-invent something else, 'brand new', again and again.

Don't you feel there is something inherently intrinsic to Toyota Method? Well, there should be a list of these feelings, by now, for those of us who have been paying attention for decades ...

kitotik2• 9 years ago

Fascinating commentary.

The paradox is that the more you attempt to tie practical/tactical things like Lean, Kanban, meditation, diet, excercise, etc to a broader perspective, the more likely the average person will actively resist/ignore both.

karljtaylor• 9 years ago

it is a delight.

though I'd be remiss if I didn't point out that team jack hollis is on a path to do that yearly. if they follow through on the user-based stuff, they'll end up doing exactly what you were hoping for, @mmjaa.

https://www.youtube.com/watch?v=ULdBWNOskoY

polskibus• 9 years ago

If anyone would like to read more about the real lean techniques in software (not just kanban), I recommend Lean Software Strategies book.

https://www.amazon.com/Lean-Software-Strategies-Techniques-D...

The West has a tendency to cherry pick techniques from holistic approaches (for example quick path to mindfulness vs meditation and Buddhism). I suppose it's a cultural difference. In my opinion, it is really worth it to try to understand Lean as a whole instead of looking at kanban only.

csours3• 9 years ago

The West has a tendency to cherry pick techniques from holistic approaches

I agree that it's worth it to spend some time to understand the holistic approach... but there's also a lot of bullshit and over-reliance on form over function.

kitotik2• 9 years ago

An excellent example of the difference in approach!

The “bullshit” emphasis on form before function is a pretty standard part of a 3 stage learning method - naive and unquestioning memorization of rules/form, unique combinations of the memorized forms, and finally throwing it all out the window and inventing new forms with the newly acquired knowledge and skills.

csours3• 9 years ago

The cornerstone of Lean Manufacturing is removing waste.

One of the largest historical sources was inspection and repair: you built something and then inspected it and fixed any problems. The inspectors and repair people didn't really talk to the production people much.

In Lean Manufacturing, the main purpose of inspection is to drive process improvements back into production. (This is the simplistic explanation, but I'm only using this as an analogy)

Thinking about data management - there's a lot of dirty data out there, and a lot of data munging to fix it up. Lean Data Management would drive process improvements into data acquisition.

Of course on the other hand, it's extremely cheap to fix data, on a per-unit of data basis. It's rule management that's expensive.

keithnz• 9 years ago

the cornerstone of lean is "flow" removing of waste is to the end of achieving flow (time it takes to get sales to delivery).

Toyotas system is predicated on the idea that for manufacturing THEIR product the touch time is way less the than the buffer/quueue times ( ie, the time spent adding value is small compared to the the overall time to get an order and get it delivered. Eliminating waste is a means to an end to cut into unecessary buffers and queues and increase flow based on the constraints of the system.

In software there's a lot of applicable ideas, but it's not exactly the same and requires careful thinking around what your revenue model is and how you deliver software based on that. But the focusing idea is not the practices of lean but how to increase flow and build (reuse) practices around that.

csours3• 9 years ago

Right, as I said, that's the simplistic view, and just some of my thoughts.

Jtsummers2• 9 years ago

If you want to try Lean/Toyota Way in your sofware business, in addition to reading the suggested texts at the end, check out Mary & Tom Poppendieck's books. They offer a lot in terms of already translating lean production concepts into lean software design/development concepts. They also offer exercises that can help you move towards being lean. Because being lean isn't a state, it's a goal and a process.

tootie• 9 years ago

I have to laugh at this. I know this is in reference to their manufacturing process, but I have worked with Toyota's IT department and they don't follow anything like this. They are a typical stodgy, waterfall, design by committee, ivory tower architect kind of IT department. We pushed agile on them, but they still insisted on signing off on 100-page BRDs before work started. Maybe I just had a bad experience, but I recall constantly wondering why we had to work so hard to sell agile on the organization that practically invented it.

dragonwriter• 9 years ago

The Toyota Way is, as I understand it, much more closely related to Plan-Do-Check-Act, empirical, metrics-based approaches in Lean Manufacturing, Six Sigma, and other optimize-a-figure-of-merit methods (a lot of which are anchored in approaches championed by W. Edwards Deming and other contemporary advocates of empirical/statistical process control.)

Agile philosophy is loosely similar in that it has a “do what works without dogma” core, but has much weaker focus on actual concrete measurements and empirical validation, which is one reason why despite its fundamental (in the Manifesto) opposition to canned, one-size-fits-all process that isn't adapted to the particular context, it devolves into a lot of that—without a cultural value of empirical measurement of outcomes, social proof and “this is the way to do it because this is what other people who succeeded were using”, or “this is the way to do it because it comes from <authority figure> who makes <abstract argument> and/or gave <subjectively appealing presentation>” become the bases available to make decisions on how to implement the vague ideals of the Manifesto. And also why it can be particularly hard to sell to an organization that does believe in changing process based on a specific, testable hypothesis about improvement in a key measure.

idoh• 9 years ago

I have read the books on the Toyota Production System and it does not apply to software, except in a very loose general sense. Manufacturing is not at all similar to software because with manufacturing you are making the same physical thing again and again, while with software you are only making a thing once, and it is intangible.

There are benefits for examining the flow of deliverables and information, and trying to eliminate waste, but much of the TPS is geared around the manufacture of physical objects and it is a stretch to apply it elsewhere.

Jtsummers2• 9 years ago

It's worth looking at books like Lean Software Development to see better how the concepts of Lean can apply to software. You're correct that TPS, being meant for a manufacturing environment, doesn't directly translate. But if you examine the principles on which it's based it fits very well in software development, maintenance, deployment, and operations.

walshemj• 9 years ago

Exactly as A senior Guy at the BSI said where do you hang the defect ticket :-) I was working of a Project with the BSI that fed into BS 7570 or ISO 9000

otakucode2• 9 years ago

Is this the Toyota that put together cars that killed multiple people? The ones that were shown in court to be ASTOUNDINGLY incompetent? The ones whose developers didn't even have access to a bug tracker? The ones who followed only 4 of 90+ automotive industry standard 'required' or 'suggested' practices? The ones who didn't use even basic static analysis tools?

I'll pass, thanks. I know it's all fine for Toyota because the courts ruled essentially "Its computers and nobody knows how they work! We can't convict them of criminal negligence!" and it saves them billions by not hiring experienced people or giving engineers the tools they need or training them or anything like that... but I actually have a conscience and don't mind if I make a little bit less money if it means there won't be the dead bodies that are my responsibility like Toyotas got.

OpenDrapery• 9 years ago

I've been around long enough to see software development methodologies come and go. I worked for GE for a decade, and they tried to force Six Sigma on everyone. Devs put through training. Lean. KanBan. Single piece flow.

The thing that businesses don't want to hear is that software development is a skill game. You win by having better players. You cannot put blind process around every facet of it. I know you'd like to, so that you can then pay very little for unskilled labor. But the fact remains that you cannot, even today.

Wysiwig html editors and drag and drop UI builders for rapid application development were trending for a while in the 90's. Then everyone realized that giving these tools to monkeys resulted in a big pile of poo. So the trend reversed back to hand coding UI layouts using markup, separating business logic from display concerns, etc

I guess what I am trying to say is that if there is any hope in putting a lot of process rigor around software development, it will be enabled by tooling. Until then, skill and experience matter. And they aren't cheap.

Put another way, if the Cleveland Browns stole the New England Patriots playbook, they would still suck.

narrator• 9 years ago

One interesting thing about software methodology that most people seem to like, such as testing and version control is that it prevents bad coders from screwing up a project too badly.

I also think that languages like Java have become popular in big enterprises filled with lots of coders because the language makes it hard to write difficult to maintain code by doing away with any features that would make it easy to obfuscate the intentions of code. This concern also had a big influence on the design of Go and all the features it left out.

The upshot of these coding methodologies that stood the test of time is that they protect the project from bad coders and they prevent really smart people from writing code that is too clever such that noone else can maintain it.

lastofus• 9 years ago

Instead the really smart people write abstraction upon abstraction until they can achieve the level of cleverness they wanted in the first place. I'm not sure all that extra code is really that much better.

engi_nerd• 9 years ago

One interesting thing about software methodology that most people seem to like, such as testing and version control is that it prevents bad coders from screwing up a project too badly.

Effectively, these methodologies are a kind of bureaucracy. Besides self- propagation, the primary purpose of bureaucracy is to cover for incompetence. So testing and version control are (in part) a way to cover for incompetent coders.

davidbanham• 9 years ago

I think the core flaw here is thinking of software development as "manufacturing". It isn't at all. It's a design process.

This Toyota methodology is about taking things that have already been designed and then incrementally improving them or figuring out better ways to put them together.

Building software is more akin to the process of taking a blank sheet of paper, a knowledge of what kinds of parts already exist, then designing a new car to fit a new market segment.

52-6F-62• 9 years ago

I worked briefly in Toyota (as an operator: Toyoda Iron Works/Toyotetsu) sub-manufacturing and I'm a developer now. I agree completely.

What might be more suited is operations development. It's quite similar to DevOps, as is, with DevOps just having been adapted more explicitly to software.

hencq2• 9 years ago

Yeah, true. Though there are definitely parts of the development process that are recurring. E.g. processes for code review, testing, etc. These are the areas the article focuses on as well. Since ultimately the software is developed by this design process, improving this design process _should_ lead to better software as well (whatever way you define better).

For the actual design of individual pieces of software, I wonder if something like Toyota Product Development System is a better fit.

dnos• 9 years ago

I worked at a software company that went through a "lean transformation" where management effectively idolized Toyota's production philosophy and wanted it used in each and every department.

While it helped our customer service department quite a bit as-is. The Toyota Way is definitely not something you can directly copy/paste and use in software development. Building a Camry 5000 times is not the same as shipping 5000 different software projects, so a lot of the analogies and examples you find in text don't always apply. I always found it amusing that you never see or hear examples used from the R&D and engineering of the actual automobiles -- just once all that hard stuff has been figured out and the manufacturing starts, all the principles are magically, easily applied.

To be honest, a lot of it is just "common sense" or things that just naturally transpire if you have a good team (especially if Agile dev and ideas like continuous delivery are used anyway.)

With the negatives aside, the problem-solving culture part of it is something that can really change things for the better and is worth researching. If nothing else, I found just being able to have a framework for solving problems as they come up is huge -- surely regardless of what a company does. For instance, integrating a standard to collectively write out the "Five Why's" and clearly define the problem statement when a team is gathered to discuss solutions to a problem can save so much time. How many times have you been dragged into a meeting, go through a huge discussion, design a solution, only to realize later that it wasn't really solving what needed to be solved?

Outside the culture aspects, it mostly just felt forced, frustrating devs, and really made things feel almost like a cargo cult waiting for the goodies to drop from the sky.

The Toyota Way at Codeweavers

Related Stories