Lesson from Madlibs Signup Fad: Do Your Own Tests

http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests/

by soundsop • 16 years ago

81 30 16 years ago

patio114• 16 years ago

Crikey you guys are fast on the submit buttons. I hit the post button, try to answer one customer email before HNing it, and it is already here.

soundsop• 16 years ago

I was expecting the submission to be an upvote. Looks like my RSS feed updated while you were answering that email.

jrockway• 16 years ago

Doing your own tests is good, but I think the username/password is too simple to benefit from the madlibs style. I also thought the wording was unusual, "I'd like to use this word ...... as my password." It could be that the strange wording was what caused the drop-off.

andymism• 16 years ago

I thought the username and password was a good choice for this test. There really wasn't that much of a difference in phrasing between Patrick's form and the Huffduffer form. The most likely explanation for the drop off is the difference in the audience these sites are made for.

As far as design decisions like these go, the data doesn't say anything on an absolute scale about the merits of one form versus the other, but it does tell us which is more effective for a particular audience in a particular context.

tkiley• 16 years ago

The madlibs form has a "sign in as guest" link, while the original sign-in form does not. The article doesn't specify whether a guest sign-in is counted as a conversion or not.

patio114• 16 years ago

P.S. If you have good eyes you’ll spot the other A/B test ongoing on this page. I’m using the traditional way of mitigating cross-test interaction… ignoring the possibility of it

Both forms have a 50/50 chance of having the guest link on them, and guests don't count as trial conversions (because I have several months of stats that suggests they give me money infrequently and they cause more support issues than all trialers and paying customers put together). The thrust of that A/B test is to see whether aggregate sales go down if the guests option is taken away. If they don't, I will drop it.

nitrogen• 16 years ago

I noticed something in sprites2.png referenced in your A/Bingo page: the bottom row of pixels on the third blue background image is duplicated from the second blue background image.

arnorhs• 16 years ago

Your version of the madlibs form also looks much worse than your usual signup form. I don't know how much that affects user signup percentage, but personally I'd have an easier time trusting your standard form because if it's design, not because of the "style"

patio114• 16 years ago

Can all designers exit the room for a moment?

I am skeptical that "looks pretty" drives conversions. I think customers pretty much don't care, based on "ugly" versions winning in many, many of my tests. Remember: the customer doesn't get to see the pretty alternative laid out side by side with the ugly alternative.

Example: form submit buttons in your Internet browser of choice look fairly pretty, right? (Or at least comfortingly familiar.) I would not describe http://www.bingocardcreator.com/images/bcc.net/next-step.png as pretty -- it is what I could throw together in ten minutes. But it clocks "pretty" buttons with the same text on it.

You're welcome to try it yourself if you think pixel-perfect alignment will make a difference. Please post results.

pedalpete• 16 years ago

It isn't a matter of looking 'pretty', it is how does the brain visually understand it as simple or not. I agree that there designers go overboard with 'it doesn't have the right padding', etc. But you can't ignore HUGE test like red makes people hungry more (McDonalds) etc. That sort of design does translate, and the mad-libs form just looks confusing.

mechanical_fish2• 16 years ago

I would press that button just to make it go away. ;)

Not that it matters -- the data matter -- but I can certainly rationalize why such a button would be more effective than a standard form submit button: My peripheral vision has seen a hundred thousand form submit buttons by now. I'm not quite as blind to them as I am to standard-sized non-animated web ads, but it's close.

And it seems to me that the problem with being a designer is that it's physically impossible to design a web form in less time than the average user will spend looking at it. You will necessarily spend orders of magnitude more time looking at your forms than a visitor will: Visitors don't look at forms for fun. They click through them fast, or they click away from them even faster. If I had to stare at that button for an hour, as opposed to simply finding it and clicking it as fast as I could, I'd design it differently. Which might very well screw it up.

viraptor• 16 years ago

I'm as far from design as possible most of the time... and I simply don't feel invited by your madlibs form implementation. On the other hand Huffduffer form just looks amazing. You missed some elements which I think are crucial for what they've done...

In the Huffduffer example the keywords are exposed, the fields look clear and spacing is just the way it feels nice. There's also a rhythm - keyword, field, keyword, field... even though it's not a standard form - you just know what to do with it. LukeW's implementation is not as pleasing, but still looks good, because there's some whitespace and no clutter.

I don't think you actually compared good standard form to good madlibs form. You compared it to a hacky, badly implemented madlibs and wrote off the whole idea ("Anyhow: test ended, not touching the madlibs idea again").

ThinkWriteMute• 16 years ago

No, web browser [form] buttons are not pretty.

lmkg• 16 years ago

In general, buttons that stand out more get clicked more. Ugly things tend to stand out better. In fact, 'standing out' may be a passable proxy for measuring 'ugly.' In a similar trend, annoying TV ads were found to be more effective than non-annoying ones because people remembered them better.

ggruschow• 16 years ago

Ugly/scary drives me away from converting.

javery• 16 years ago

The design of something is the first indicator I use to evaluate how trustworthy a site is. The better the design the more I know that these people put alot of thought into it and spent money and time on it. Now, overly clever stuff is definitely annoying and bad.. but ugly is a big red flag.

rauljara• 16 years ago

Running tests is definitely good advice. But execution definitely matters in these things. Just looking at the two forms tested side by side, the original looks more well spaced out and balanced. The madlibs has the two text fields actually touching. Compare that to the original madlibs form (http://www.lukew.com/ff/entry.asp?1007) and you'll see that the original madlibs form just looks a lot better.

ramchip• 16 years ago

Isn't it a bit early to call it a fad? I only saw one website try this (the one you gave the link to). Perhaps I missed some, but still...

mechanical_fish2• 16 years ago

While everyone is piling on with the untested hypotheses, here's mine:

The madlibs form is just too wordy. Whereas the username/password field pair is not only less wordy, it's an idiom: I've seen those fields, in that arrangement and that order, thousands of times over the last decade. I barely even have to read the captions at all.

Not that this hypothesis, or any of the others, matters. The point of this article isn't to design a better form. It's to show us how to design a better form.

Batsu• 16 years ago

As an intently web savvy group, we want this simple "fix" to work for us.

You provided us with reason to believe this (albeit simple) change will not provide us with results.

Please, don't do this.

patio114• 16 years ago

If I were to play pop psychologist, I think we have some fairly simple fixes that we want to work because they're cool/sexy/etc, and some simple fixes which we don't want to work because they're "stupid" or sound marketing-y.

For example, in the original madlibs test, where seven things changed, we want to attribute the change to the madlib element (which was sexy and creative) rather than to e.g. the text on the conversion button changing. I've gotten 20% lifts out of "stupid little things" like that before, and seen good writeups of the same from other people. But e.g. designers do not want to hear that the thing they spend 98% of their time on is essentially meaningless and the elements of the page which they scrutinize least carefully -- like microcopy or button text or (in one memorable test) which way the stock photo model is facing dominate the measurable results. This is like how doctors do not want to hear that the difference in clinical outcomes due to professional experience is totally dwarfed by the difference in clinical outcomes due to following checklists. (On the plus side, designers don't kill anybody for their professional pride.)

In general, I think the point of A/B testing is that you should ruthlessly soulcrush what you want with what the data actually tells you. I respect HN folks enormously, but I'm not hearing "I tested it and it worked" -- I'm hearing "I've got a great theory on why your data does not conform to reality. Reality is, of course, what I predict it to be."

paraschopra• 16 years ago

Yeah, it fantastically proves the point: do your own A/B tests. I am guessing the implementation can create a huge difference. Maybe you can ask for help from some of the designers here on HN and see if a beautiful implementation makes a difference. Plus your two checkboxes can also go into the "malibs" form.

dpcan• 16 years ago

Well, he did it all wrong.

When you are done reading the Madlibs, it should read like a sentence. What he did was surround regular text fields with strange sentences. When you are done filling it out, you can't just read it like a regular sentence.

It should have been:

I would like to make bingo cards! Please save them for me under the name [ Your Name ], and and my email address is [ Email address] in case you need to contact me. But please [do / don't] contact me twice with hints, and [do / don't] send me a newsletter.

See, when you are done, you can read it like a sentence.

All he did was replace "Username" with longer wording. It didn't seem "fun" the way you did it, just odd and off-putting.

micrypt• 16 years ago

True. Made a few adjustments and I think it looks a bit better already: http://twitpic.com/15t9cc/full

Herring• 16 years ago

A bit, yeah. But the pixels are still really off. I don't even know why, I just see they're wrong.

maxklein• 16 years ago

Seeing as we have two totally different results from different people, my conclusion is that even though the madlibs style can improve your conversion rate, it's too easy to get it wrong and actually reduce your conversion rate.

The majority of websites have used the standard sign up form, and it's known and recongized by most users. It's risky to to go with something like this that could have a strong negative impact if you do it wrong.

ams6110• 16 years ago

Just as a personal observation, I much prefer the traditional style. All these "I want..." and "Give me..." style dialogs make me want to vomit.

Lesson from Madlibs Signup Fad: Do Your Own Tests

Related Stories