The $5 Guerrilla User Test

http://blog.bumblebeelabs.com/the-5-guerrilla-user-test/

by shalmanese • 16 years ago

158 48 16 years ago

blog.bumblebeelabs.com

megamark162• 16 years ago

What other types of guerrilla testing have people tried before? This sounds like a fairly witty idea for gathering user interaction information, but I wonder if it's actually functional.

jcl2• 16 years ago

I believe there's at least one website review service that is build on top of Mechanical Turk, if that helps any.

megamark162• 16 years ago

Thanks, that's good to know. I know someone once mentioned using Mechanical Turk and that those sometimes ended up being their most loyal users, but I cannot recall the context. Has anyone used these services and what results did you see?

Groxx3• 16 years ago

Anything run by a 3rd party strikes me as immediately questionable for accurate testing. They necessarily use people more geared towards using applications, as otherwise they wouldn't be there in the first place, getting paid to do something that most people wouldn't touch. They can certainly provide extensively useful insight, so running through a UI-review company is still a good idea, but can there really be a guerrilla testing company? The idea of guerrilla testing is to get totally untainted results, isn't it?

ThomPete9• 16 years ago

I call BS on this. It has no usage what so ever.

You can't test for success you can only monitor it.

forcer• 16 years ago

Why no usage what so ever? Any process of getting user's feedback is better than doing nothing. We are so skewed of using our own apps that we think all is perfect. Nothing ever is. I think its a great idea how to get more usage data. If I try it I will let you know how it went. I am thinking of using some screenrecording software like SilverBackApp to archive and analyze results later when not drunk :)

Janusz Founder of http://www.UsabilityTest.com (free usability testing for web apps)

Groxx3• 16 years ago

Interesting site... if it works like it appears to, I wouldn't be one of the survey testers, but I'm sure it'd nab a lot of people, enough to at least get some ideas. Good luck with it!

ThomPete9• 16 years ago

Because success is not measured by the usability of your product.

There are countless examples of way too complex products that are still successful and almost any successful company you can think of DIDN'T do usability testing.

Companies like Amazon do A/B testing, Apple don't do usability testing at all.

Usability starts to matter later on, when you can measure on actual customer behavior rather than user opinion.

Down vote me all you want. I challenge anyone to prove that usability tests are actually making more successful products.

ynniv2• 16 years ago

I notice that you are a design professional and have written lengthy articles on interface design. The point of this article is that buying a handful of beers will improve your product design immensely, when compared to doing nothing at all. It is an amateur move that probably looks useless from a professional standpoint, but like 24-grit sandpaper, makes a rough job look more polished.

ThomPete9• 16 years ago

Do you really think it's that simple?

It's not.

There is a whole list of issues with asking people about what they think.

Again success of a product is not secured by these kind of tests. Sure if you are doing something that is outright useless but then asking anyone (like girlfriend) is enough.

What he is suggesting is equivalent of doing code review with a first year programmer.

Wouldn't you call BS if I suggested something like that?

I am telling you as a professional, that's not any way to go about testing your product. Unless it's outright useless. But then you don't have to spend 5USD on a beer. You can ask ANYONE for free.

Opinions are not valuable feedback.

CoryMathews• 16 years ago

This is awesome. I am going to start doing this all the time. Its extremely good also that my target market is college students :) even better.

c_allison• 16 years ago

Not only a good way to test cheaply, but a good way to network! Win-win.

run4yourlives• 16 years ago

This is popular in some design circles, particularly with evaluating reactions to 2/3 designs prior to launch.

I'm not sure you'd get exactly the same quality of results with a user test, and demographics is key - after all, the blue-collar workers at the pub aren't the people to ask about your b2b conversion tool - but it's certainly the type of thing that we should be doing.

shpxnvz• 16 years ago

I'd imagine that for just about any demographic you care to target, there's a bar you can find that caters to them.

Heck, I've seen bars that have clientele heavily biased to a specific industry, and in some extreme cases, even a single employer.

jdrock3• 16 years ago

One additional note that should be added: results will only be useful if your desired customer base is represented well by people at a bar.

Spikefu• 16 years ago

FTA: "Bars are wonderful at segmenting by demographic. Match the bar you’re going to with the user population you want to target. Different bars will produce slightly different results but the variation is not huge."

jdrock3• 16 years ago

Yeah, I don't know a bar frequented by people interested in the Semantic Web or web-crawling.

alexyim• 16 years ago

Great. Now I have this to try in addition to the Parents Test

kolya3• 16 years ago

Bring a laptop that you don't care about. Drunk people spill.

rodyancy• 16 years ago

I've done the sober equivalent to this (coffee shops). Not only did it point out some usability issues, like, "oh, he thinks THAT'S a button," it also proved to be a great way to generate ideas. Most of the input we received was dead wrong, but we had to think about why it was wrong, and that got us thinking in ways we hadn't thought before, which led to good ideas. I suspect the drunken version would be more fun and just as useful, assuming you are producing a consumer product. As for B2B, if your domain is specific, this might not be the most valuable use of your time.

olegk• 16 years ago

I don't see myself doing this.

coriander• 16 years ago

So you want programmers to talk to people in real life. Interesting.

AFerenci• 16 years ago

I experienced the same effectiveness (95%-99%) of getting the "guy/gal reading the newspaper" to volunteer for the study first hand. Although, this was more focused in cafe environments.

Our team was trying to get beta users signed up for a location-based iphone app in the Boston/Cambridge area and most people reading newspapers in coffee shops or cafes are frequently willing participants. I even noticed that 7/10 times these people will engage with you in conversation regarding feedback/UX improvements etc.

In parallel with Rodyancy's experience, some of the user feedback was arbitrary (one guy started comparing our application to time/space continuum theories...etc) but nonetheless, I would advocate approaching the newspaper guy/gal as they are already on the knowledge quest and usually are more open-minded.

Oh, and avoid the book/textbook readers. These people are either really engaged in novel (imagine tapping on someone's shoulder during a movie) or are stressed-out students.

dkimball• 16 years ago

I'm kind of disappointed that Steve Krug didn't think up something like this for _Rocket Surgery Made Easy_ (http://www.sensible.com/rocketsurgery/index.html); this would be a very convenient source of manpower for his model of "hallway usability testing," and would allow even larger numbers of even more easily set-up tests than the model he describes.

It's good to see how usability testing is becoming de-mythologized these days; all of us should keep in mind, for current and future program development, that you really don't need a very sophisticated apparatus.

Groxx3• 16 years ago

Start off approaching a group of the same sex as you so that the encounter isn’t sexualized. Next, move on to a mixed sex group and then finish the night with an opposite gender interaction so you get a nice demographic spread.

I wonder if that last bit is intentionally a double-entendre. Especially since the proposed environment involves a bar.

It's an interesting technique, and I'll definitely keep the strategy in mind for future endeavors.

shalmanese9• 16 years ago

The double entendre was completely unintentional. Testing against women is important but one of the big hangups a lot of people have is the sexual component. This is one technique I've discovered to help ease out of that mindset.

As a strategy for picking up women, this is somewhere down the bottom in terms of effectiveness. As it turns out, asking someone to participate in a user study is about the least sexiest thing imaginable.

The $5 Guerrilla User Test

Related Stories