Complaint-Driven Development

http://www.codinghorror.com/blog/2014/02/complaint-driven-development.html

by dieulot • 12 years ago

243 70 12 years ago

jchung• 12 years ago

This definitely resonates. Although while feedback from discourse users naturally takes place on discourse, gathering feedback from users can often be much more difficult. We've enabled usersnap recently, which has been somewhat helpful, but hasn't created the kind of vociferous user-to-user and user-to-developer debates that you see on meta.discourse

maxerickson• 12 years ago

When I'm giving feedback, I usually don't want the developer to tell me why it is wrong. I don't mean that in the sense that my perspective is correct and my feedback is never wrong, I mean that if I am irritated enough about something to report it as a flaw, the last thing I am going to want to do is engage in a debate about it.

BTW, discourse infinite scroll could behave better when the user input is the down arrow on the scroll bar.

sergiotapia• 12 years ago

Offtopic: I find it really tacky how this blog constantly links back to other posts of his hidden in the text.

yoha4• 12 years ago

This. Websites are supposed to distinguish internal and external links. It makes it way easier to now what kind of information we will be getting when clicking. My first thought is for a blog [1] which does it pretty well: (internal links are underlined.

[1] http://www.bortzmeyer.org/ (fr)

(I could not think of an English-speaking website with a similar differentiation right away)

Edit: just look at Wikipedia https://en.wikipedia.org/wiki/Hacker_news

xentronium• 12 years ago

Wikipedia (and mediawiki in general) distinguishes between internal wikilinks and external links.

tragic• 12 years ago

Wikipedia does.

But really, the status bar hover thing in every browser since IE5 is more than enough for me to work out where a link is going to go. (Guess it's not so useful on mobile.)

bmm6o• 12 years ago

Websites are supposed to distinguish internal and external links.

Supposed to according to whom? That seems to be far from a universal custom. And if it's important, the browser can do it.

reidrac• 12 years ago

I've been writing in a blog for over 10 years and it's really hard to not refer to posts you have written in the past. A blog is a website and that what links are for.

To be honest, I should do it even more because I find very annoying when I write about something I've already written (well, it makes sense: if it was a good idea to write about that back then, that's why it's a good idea to write about it now).

kbutler2• 12 years ago

Yeah, I hate it when I'm reading a book, and it includes a list of other books in the series... So tacky.

This is a strange perspective. Any reader of the blog obviously considers the author's information or opinion on the topic useful enough to be reading the article. Isn't it likely that the reader would consider the author's writing on related topics useful?

Would you rather have an out-of-context list at the end: "If you liked this, you may also like posts X, Y, and Z"?

Yes, this can be abused in link-bait-seo-overoptimized ways, but that doesn't invalidate the utility of the practice.

mst• 12 years ago

Links are meant to provide additional information, or context.

Often, the author's views on related topics are both - perhaps they're less valuable if you've already read the archive, but ... really, sane site internal links are almost always some of the most useful ones there for long standing blogs.

yoha4• 12 years ago

Something is not clear about the character count requirement: did they set it to one or just finally found the right way to present it? If this is the former as the messages from the dialogs suggest, I think he should have highlighted the fact that the problem was not one of design (i.e. making users understand the limit), but functional (i.e. not forcing users to pad a too-short message).

EvilTrout• 12 years ago

No we left it as is, but it is configurable in the admin section. This is very important for non-English languages where one or two characters is often enough for a descriptive title.

yoha4• 12 years ago

one or two characters is often enough for a descriptive title.

I can't think of a many such titles though. Maybe "UX" for a general discussion about user experience?

Edit: oups, I had read "English speaking" rather than "non-English" ; with ideograms, it is actually quite obvious, thanks for the remark!

jorgeleo• 12 years ago

I like the fact that this blog talks about the elephant in the room

It really does not matter what methodology, or tools you will use, if at the end it does not pass the user acceptance test

So why consider it a failure if you have users telling you what they want and how off you are? Better to embrace it and make a better product

csbrooks• 12 years ago

This line toward the end is key too, though:

That's how you suss out the rare 10% of community feedback that is amazing and transformative.

So don't obsess over every bit of user feedback, but figure out what matters, and what multiple people are complaining about, not just one loud person.

codinghorror7• 12 years ago

Yes exactly -- we weight common feedback from multiple Discourse instances much more heavily than individual opinions from folks on meta.discourse, for example.

(There is also the issue of observing what people do versus what they say they do. At least early on I find the problems are severe and obvious enough that this subtlety isn't a big deal yet.)

the1• 12 years ago

why not hire good UX experts who can capture these complains in product design phase?

Contact me if you need help. I'm a good UXpert.

kbutler2• 12 years ago

Because even the best experts are not going to be able to identify requirements the users won't discover until they begin using the product.

You can approximate some of this feedback by observing user interaction with prototypes, but if you pay attention, you'll learn more when they start using it in production.

awesomerobot• 12 years ago

In my experience as a designer that changes very little - In reality it can often mean you did a whole lot of upfront work just to have it change anyway. You can put 100 UX experts in the product design phase and users will still have complaints the second you release the product.

The point of this process is to start that complaint processes as early as possible so you're not delaying the inevitable or spinning your wheels on something users don't want anyway.

npsimons2• 12 years ago

http://www.codinghorror.com/blog/2009/02/are-you-an-expert.h...

gingerlime• 12 years ago

I totally agree. The problem for us however is actually deciding what's the most complained-about request, and how to categorize those requests. I find that we each have our pet hates and loves, and even if we don't mean to, we develop selective hearing for complaints or praises.

We tried to add tags to support emails (via helpscout), but it's also hard to remember to tag things, and it's easy to use different tags for similar complaints.

I wonder about the best strategy to quantify complaints / suggestions and 'bucket' them correctly, so you can really choose the top ones.

UnoriginalGuy• 12 years ago

That's why I really liked Dell's Idea Storm. Instead of worrying about what a single user thinks, you can only worry about the current most popular issues or requests.

I don't think it has really helped Dell as to many of the things they simply said they aren't doing it (e.g. standardise power supplies in laptops) but the concept seems decent if they weren't very open to the feedback received.

ksaun• 12 years ago

We've been using User Voice for the computer game Torment: Tides of Numenera. First, we used it to get community feedback on our rewards, tiers, and approach for our Kickstarter campaign. It was very helpful in testing our ideas in advance.

Throughout development, we've been using it to get feedback from the community: on their own ideas, their thoughts on design aspects we've shared, and questions they'd like us to answer.

User Voice lets people suggest and vote on ideas. We can label them with customizable tags like "Seen," "Considering," "Not Planned," as a very time-efficient means of communicating to many at once, without the expectation of personal replies.

The system isn't perfect, and we're not using it perfectly, but it's been helpful for getting and organizing feedback.

frandroid• 12 years ago

That's when you need to look back, and not just focus on the most popular complained-about things, but also look at the qualitative feedback you're getting, and they to synthesize commonalities between different lines of feedback. Like another commenter said, sometimes feedback will help you reach a local maximum, but you need to infer /why/ complaints come in the first place, now just what people are complaining about, to know where to go next. If you have so many complaints and there's no clear direction, either you're at the nitpicking phase, or there are larger problems that you haven't identified yet.

If your current tagging system isn't helping you, flip the axis on which you're tagging to see if new patterns emerge. (area of complain on your product, versus types of tasks users are trying to achieve, for example.)

jackson1988• 12 years ago

The only thing I've ever seen work is getting down deep and dirty in the trenches with your users, communicating with them and cultivating relationships. That's how you suss out the rare 10% of community feedback that is amazing and transformative.

This is simple but practical advice I think far too many people ignore. You've inspired me with this article. Glad to see you've been successful from it!

mathattack• 12 years ago

This process takes a lot of humility. It's not easy to say, "We are smart, but we don't know, so let's just get something out there." It's antithetical to the planning mania in so many big companies, and also why innovation like this is best from small places. I wish them the best!

atmosx• 12 years ago

Shakespeare thought otherwise, apparently:

"The fool doth think he is wise, but the wise man knows himself to be a fool."- William Shakespeare, As You Like It

RyanMcGreal• 12 years ago

Or as it's known today, the Dunning-Kruger effect.

leaxdc• 12 years ago

We in our company preferred chair-driven development - it's in case of failed estimate you are lashed out by chief's office chair.

foolrush3• 12 years ago

The issue here is that it cleaves toward designing for the audience you have, as opposed to the audience you seek to appease.

Often, the design goals might exist on an alternate axis.

Random people offering random opinions amounts to random noise. However, the random noise will not appear neutral; it will appear as information.

If we consider many different things designed for particular audience members such as jet cockpits, medical tools, racing automobiles, we will see traits that exist that may seem nonsensical or otherwise when we divorce them from their designed contexts.

Bill Buxton covers this in Sketching User Interfaces when he describes Inuit coastal maps: "The Inuit have used. [...] Tactile maps of the coastline, carved out of wood. They can be carried inside your mittens, so your hands stay warm. They have infinite battery life, and can be read, even in the six months of the year that it is dark. And, if they are accidentally dropped into the water, they float. What you and I might see as a stick, for the Inuit can be an elegant design solution that is appropriate for their particular environment."[1]

Focusing on complaints of the above design in all likelihood would, given the mad rabble of audiences online, result in discarding a solid bit of design.

[1] Bill Buxton, Sketching User Experiences, pg 37

heartbreak• 12 years ago

Isn't it more likely that the tactile maps are used because they are superior to any other kind of map the Inuits could have created (non tactile drawing, or a rock that doesn't float, for example)?

foolrush3• 12 years ago

A tad of a chicken-egg dilemma there. While I can only speculate, I'd surmise that the Inuit are / were well aware of their contextual circumstance and designed accordingly, and quite likely iterated on the design.[1]

The issue I raise is a cautionary tale that when we try to use some abstract evaluation well removed from the context, the "data" can end up being non-information. Worse is when design evaluation comes with hidden cognitive bias or underlying ideology.

That is, even a valid concern / complaint of a given design my have no merit given the design context. If we take for example a vehicle, one might complain that it has a lack of cargo space or a child seat restraint. Valid complaint given a very discrete set of design contexts. To follow along the original example in the post, these otherwise valid complaints of a design are moot if the vehicle in question is an F1 automobile designed to be raced in unique circumstances with unique needs.

[1] There is also the possibility that some of the design contexts include more abstract needs such as cultural, religious, historical, aesthetic, or even the darker notions of power dynamics. To diminish the potential influence of these facets would be an ignorance in our design evaluation.

I'd encourage anyone interested in design context to read Marco Van Hout's article on the rather broken UIGarden.net site. Here's a link via the Wayback Machine: http://web.archive.org/web/20130719161851/http://uigarden.ne...

jobu• 12 years ago

Listening do user complaints doesn't always mean listening to their suggestions in fixing the problem. You shouldn't necessarily remove something that every user is asking you to remove if you feel it's a critical feature, but it definitely means you need to rethink how the feature works.

saraid216• 12 years ago

but it definitely means you need to rethink how the feature works.

Or how it's presented. It's pretty common for a feature to be the subject of complaints because no one actually realizes why you put it in.

DorintheFlora• 12 years ago

From the article:

Since that change, I haven't heard word one about our terrible, onerous, awful default body and title character limit policies. Not one. Single. Complaint.

So, they didn't do what the customers said they wanted, but they did work on the issue until people quit complaining.

“If I had asked people what they wanted, they would have said faster horses.”

― Henry Ford

Plus, IIRC, for the Edsel, they did ask customers what they wanted. And it has gone down in history as one of the worst cars ever.

sopooneo• 12 years ago

I feel like responding to user complaints can help bring you closer to the nearest local maxima. But it often won't bring you beyond that unless your users are themselves UX designers or developers.

So it's useful to make your MVC as good as possible so that your users aren't forced to complain about the results of fundamental design shortcomings. That can result in more and more complex fixes, none of which should be necessary.

RogerL• 12 years ago

While I grasp the reasoning in the article, and sometimes practice it, I also think it is often either counter productive or impossible.

For example, I have friends that released a rather ho-hum mobile app. They quickly garnered something like a 1.5 star rating, scathing reviews, and almost no conversions. The business cycle on this was a year, and they are still trying to claw back reputation and win users. It's a debacle. (the problems weren't their fault, but that is irrelevant to this point).

Then you have companies with secrecy, like Apple. I think this advice would be terrible for them (I have never worked there, and am open to correction). They can't dog food it widely due to the internal silos, and they certainly cannot test it with the public.

Then there are electronic systems - iterations on SW is easy, iterations on HW expensive and hard, even with simulations, mock ups, and what have you. I worked on an augmented reality hardware thingy several years ago; we went from foam cutouts to a couple of very expensive prototypes, and that was it.

It is awesome when we can completely sidestep a problem, and this process lets you sometimes sidestep the serious difficulty of UI design. I worry when it gets bandied about as a truism, or The One True Way (not saying Jeff is doing that, I'm remarking on the wider industry). Yes, Agile lets you sidestep the problem of scheduling and estimation - sometimes. Try that when you are making a new airliner, building a cloverleaf interchange, making a car entertainment system, and so on.

edit: the converse problem is equally as large. Someone below mentioned the 'planning mania' of companies. I don't mean to downplay that problem, just to point out the need to evaluate each situation on it's particular needs as opposed to a 'best practices' (oh, how I hate that term) unthinking approach.

foolrush3• 12 years ago

"Then you have companies with secrecy, like Apple. I think this advice would be terrible for them (I have never worked there, and am open to correction). They can't dog food it widely due to the internal silos, and they certainly cannot test it with the public"

I fear that this myth is capitalized on by Apple under the hypercapitalist Auteur theory.[1]

The value of magic tricks is that they create awe and surprise, when in reality they can often be a by-product of misdirection and street-level sleights.[2]

This is not to suggest that Apple is going out onto street corners and asking random individuals for their opinions, but rather that part of the corporate myth making is likely obscuring process.

[1] http://www.youtube.com/watch?v=p9dmcRbuTMY

[2] I believe there is a court testimony document discussing Apple's market research team via Phil Schiller. Contrast this with Mr. Schiller's other comments about avoiding market research.

codinghorror7• 12 years ago

You can't release crap, even in your first version. Not saying your friends did that, but you have to have the kernel of something decent to show people before you release. Otherwise you're blowing the launch by taking all that initial interest and squandering it.

(I'd also argue mobile apps have a severe updating problem, that web apps do not.)

dodger• 12 years ago

One time I was talking to a designer about doing it this way. The designer said "Yeah, that's a great way to find a (look of total disdain) _local_maximum_." As usual, some truth to both points of view.

wavesounds• 12 years ago

This is great for a startup but can be depressing for long term consulting.

I've had the unfortunate experience of building a product for someone else where the process was driven by a combination of Complain-Driven Development and Upper-Management Wish-lists. This alone might have been fine but at the same time anything positive like the real analytics about how successful the product was or any non-complaint communication coming from the users was hidden from me for fear, I suppose, that I might try to use that information get my company more money.

This became incredibly depressing. Everyday you show up to work putting in more and more hours into something that comes back with more and more complaints. It was hell and I did everything I could to end Complaint-Driven Development to no avail because thats how the customer liked to work. Eventually I finally just gave up and left to find something more rewarding and less soul crushing.

pointernil• 12 years ago

"Software auf Zuruf" ;)

codinghorror7• 12 years ago

It also depends what stage you are at.

This article is about early stage products that are very new. If you're looking at a mature product that has been on the market for 10+ years, it's likely (I hope!) that most of the major obvious pain / complaint points have already been addressed. I don't think CDD would be as useful in that scenario -- you want a different model.

Mz2• 12 years ago

I am still trying to work out how to get useful feedback at all. Due to a fairly unique situation, people divide up between those folks who tell I am awesome (with zero useful feedback as to why) and folks who want my head on a pike) and zero useful feedback as to why). So far, backing away from the crowds of folks who want my head on a pike is the only constructive answer I have come up with, which leaves me more and more and isolated. I remain frustrated as to how to resolve this issue. I need skeptical feedback, not hatred and not idolatry. Neither of those is productive for anyone.

So, yes, there can be big problems with this, depending in part on the problem space you are addressing, I think.

illuminate• 12 years ago

"This became incredibly depressing. Everyday you show up to work putting in more and more hours into something that comes back with more and more complaints. It was hell and I did everything I could to end Complaint-Driven Development to no avail because thats how the customer liked to work. Eventually I finally just gave up and left to find something more rewarding and less soul crushing"

My current gig is very customer-driven and we hire a number of PMs to handle and incorporate the feedback. Our customers love us because the alternative is a particularly stodgy, poorly performing and relatively unchanging product. With end-users, yes. The flood of complaints and the complaints about the fixes for the complaints and on and on can be soulcrushing.

Finding a niche who loves you and your product is hopefully possible.

igravious2• 12 years ago

Tried installing Discourse on Gentoo. Turns out to be a non-trivial project and I'm no Ruby on Rails newb. I'm no code-jock but I'd love if the install story was a tad easier. Maybe I should quit whining and figure out how to turn my pain into an ebuild :)

codinghorror7• 12 years ago

Agree that one issue in the Ruby ecosystem is that installs are too hard. Can you try our Docker install, we hope to move to this soon as our standard recommended install for that very reason, would love feedback:

https://github.com/discourse/discourse_docker

Mz2• 12 years ago

I know nothing at all about how the install works for this, so excuse me if I am putting my foot in my mouth, but let me suggest you consider a stop-gap measure of supplying a screenshot based tutorial for the install. (example: http://micheleincalifornia.blogspot.com/2013/12/jabiru.html -- I spent two days trying to get this working, with tech support and I have been told the tutorial I wrote works in 5 minutes or less)

igravious2• 12 years ago

Wow, thanks for responding!

I'll give the Docker install a spin. I'll give feedback on the Gentoo install and the Docker install when I am able to give both my full attention.

Your blog posts are frequently thought-provoking and inspirational and mercilessly bullshit-free. I'm so glad you've moved to Ruby on Rails and welcome you with open arms into the open-source world, you may not fully appreciate how much goodwill that move generated for you :)

boohoofoodoo• 12 years ago

http://laughingsquid.com/wp-content/uploads/2013/07/homer2.j...

codinghorror7• 12 years ago

Right, this is why you listen to your community, but you don't blindly do everything they tell you to do. Filtering out the best 10% of feedback -- and resisting the transformation of the car you designed into a truck because "the community demanded it" -- is the way to go.

andyl• 12 years ago

I also practice CDD. But my experience is that few people complain. Maybe 1 out of 20. The rest silently endure a bad UI, walk away from the app without comment, or badmouth the app to their trusted friends.

Complainers have to be cultivated. Complainers can be your most valuable asset.

mtrimpe• 12 years ago

I have one tip for you: grovel.

I've found that people don't respond unless they know that you really, really, really care. That's why most of my support emails include lines like "We're here to support you in any way we can so if you run into any problems whatsoever please let us know," or "We'll do everything we can to help you achieve X, so please don't hesitate to contact us if we can help in any way."

I've found that anything less strongly worded than that comes across as formality rather than a sincere offer.

npsimons2• 12 years ago

This goes even for employees of established organizations who don't interface with customers: don't wait for your yearly appraisal; be asking for feedback, always look for ways to improve, and try not to get defensive.

hngiszmo• 12 years ago

1 in 20 complains??? That is actually a great number. My assumption for my products is really really not a 1/20 thing. My estimates are rather that a crash bug will be reported by 1/100. The others will just down-vote my android-app and uninstall. My estimation is based on broken design on square displays and on Thai font issues that I was made aware of far too late.

On the other hand I give instant feedback and keep a relationship to any complainers that are my happiest customers now.

codinghorror7• 12 years ago

True. Any complaints are already the tip of an iceberg you can't see.

And the people complaining should be treated like your best friends: they cared enough to complain. Nobody else cared.

FLUX-YOU• 12 years ago

If you want to view a world where the physics of icebergs have only the tip under the water, take a look at gaming development and feedback. :)

johnny635• 12 years ago

I've seen this works extremely profitably in the past. Hard to argue with results.

xfalcox• 12 years ago

Valve does this very well using reddit. A bad bug goes for a whole year on Dota 2, and if someone complains on /r/Dota2 it gets fixed in two days.

It's great, users fells empowered, and you get the priority list for free with reddit upvote system.

As for discourse were planning to deploy it this year at our intranet, and we've got 130.000 employees.

chris_wot• 12 years ago

Perhaps this is something the Gnome guys could practice?

mrxd• 12 years ago

Terrible design. “20 to go for the reply” is comically bad UI writing. And then hiding it in a corner in grey text when the user is fixating on the field they are typing in? Not surprised at all it didn't work. And the "nuclear option" doesn't even meet basic usability guidelines of informing the user of input field requirements upfront, not just relying on error messaging.

kitd• 12 years ago

Did he consider that the way he wanted Discourse to be used may not have tallied with the way his users wanted to use it?

That's the underlying disconnection.

I am now working on my 3rd message-transformation/integration middleware product for a large corp. In each case, the issue tracker has been stock full of problems from people who took the product, read all the manuals about what they were supposed to do, then did it their way anyway. Because, well if they can, they will.

I think the basic problem is not challenging your personal assumptions about how you want your beloved system to be used.

collyw• 12 years ago

One problem with this is that users don't know what they want or what is easy / possible / hard.

I constantly get asked to add another "excel parser" to my in house Django app. These days I refuse (it is way too error prone) and build it directly into the web interface. Then they realize, that even with all the nice auto-complete and cut and paste features that excel comes with, clicking a few checkboxes and a submit button in a web interface is more efficient (and less error prone) than cutting and pasting the data into Exel in the first place.

Complaint-Driven Development

Related Stories