How two magical words increased signups by 28%

http://visualwebsiteoptimizer.com/split-testing-blog/ab-test-case-study-how-two-magical-words-increased-conversion-rate-by-28/

by spif • 16 years ago

84 57 16 years ago

visualwebsiteoptimizer.com

spif5• 16 years ago

Although the two words weren't revolutionary we were quite surprised by the increase. Mind you we had about 6 other variations as well, including "It's free up to 250 contacts" and an orange version of the "Signup now!" button - both we thought would have higher conversion.

jacquesm5• 16 years ago

I'm very curious about your end-to-end conversion rates, from raw unique visitors to paying customers.

spif5• 16 years ago

Right now it's about 1% of our raw unique daily visitors.

The tricky thing is to drill down and see which users these are and where are they coming from and how are they using Soocial. Any tips are welcome! ;-)

patio112• 16 years ago

One of the things on my list is to try calls to action which are not Sign Up, precisely because folks think that that involves either making a commitment or paying money. (I get at least a half dozen emails a month asking me to cancel a free trial, from folks who grew up in a generation where anyone giving you something free is going to send you a bill 30 days later.)

paraschopra13• 16 years ago

I'm curious. Do you have any suggestions for the alternatives?

patio112• 16 years ago

"Sign Up" solves a problem for the site owner. Write something which solves a problem for the customer.

I'm going to try things like, e.g., Start Making Bingo Cards.

I actually had a few dozen buttons made in a particular design to test different calls to action against each other earlier this year, but the design itself flopped -- hugely -- against my pre-existing buttons, so much so that I didn't think color or copy changes on it would recover the loss. Curses on having my testing bandwidth limited by graphical expertise...

charliepark• 16 years ago

I've found "get started now" to be quite effective. One small test showed an increase of conversions from 12/126 to 29/127, so up 140% (the loser there was "Create your free account now", the winner was "get started now").

I'm running a test right now, with "try it for free" currently at 262 out of 792 clicking through, and "get started now" at 269/778 (so up 4.5%). It's not yet conclusive, but at no point in the test has "try it for free" outpaced "get started now".

If my results ever show themselves to be "conclusive", I'll post about them at abtests.com. I'd encourage you to try "get started now", though, to see what you find.

timcederman• 16 years ago

The importance of calling out that something is free cannot be understated for conversion.

jacquesm5• 16 years ago

hey Spif,

Effectiveness of this kind of test should not just be done at the spot where you make the change but end-to-end.

It's very well possible that by placing the 'it's free!' there the signups shift to a much larger number of people that will never convert to paying members (for a freemium site).

If you only test at the point of making the change it is very well possible to actually increase free sign-ups but to decrease your conversion rate to paying members.

So, this test is about an intermediary goal, the real conversion is the rate between new visitors and paying members.

I sure hope that your overall conversion rate to paying members shows a similar increase, but make sure that really is the case.

paraschopra13• 16 years ago

Actually, spif mentions the following in the post:

>To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers

jacquesm5• 16 years ago

Ah, cool, I missed that. Thanks. It really is the crux though, any kind of partial measurement is essentially meaningless without the overall picture.

ajg1977• 16 years ago

"There is no reason why 'It’s free' should work better than 'Sign up for free'"

Actually I disagree. The phrase "sign up for free" contains some hint of a potential "but", similar to "free to join" or "download trial for free", or other carefully worded phrases that litter anything-but-free internet offers.

"it's free" is far more emphatic and open.

nreece• 16 years ago

Maybe. But "Free Sign Up" sounds/works just fine.

Having said that, a hint of a potential "but" (i.e. trial-ware) is good at times so that the users don't take the free part for granted and they are aware of an upgrade option.

siong1987• 16 years ago

I read somewhere saying that "<strike>2.99</strike> Free!" works better that just saying that "Free!" because it's hard for users to understand the real value of the latter.

pmjordan2• 16 years ago

At least in Europe, I'm fairly sure you'd run afoul of unfair competition laws with that - you're not allowed to run "infinite sales". If you show a non-sale price, you must have charged that much prior to the sale, and your sale must have an end date.

pclark• 16 years ago

how would they ever enforce/police that?

iaskwhy• 16 years ago

Just adding: I believe the end date is a pro-consumer thing and for me it's really essential because I like to make good decisions and that offers me a possible deadline days away instead of now or never.

Charuru• 16 years ago

Maybe http://visualwebsiteoptimizer.com/ should take a look at the button on their own homepage.

Edit: It doesn't work.

chaosmachine• 16 years ago

You're right, clicking "sign up" does nothing in Firefox or Safari.

That can't be good for conversions.

paraschopra13• 16 years ago

Oh, no. Mistake! Thanks, fixed it.

paraschopra13• 16 years ago

We can't advertise the product as free because it isn't free.

almost3• 16 years ago

EDIT: Just confirm, I think I was incorrect in the message below.

I'm not completely sure I'm correct (please correct me if I'm not!) here but as I understand it the article does not support the claim in the headline. The headline claims that signups increases by 28% in the changed version and that this was all attributable to the change.

It's the second bit that isn't supported, they say that the result was statistical significant but what I understand them as saying was that it was statistically significant that the new variation was better than the control. But it could be better by an amount more or less than 28%, all we know from that is it's almost certainly (95%) at least a little better. We would need to know the number of trails to be able to get a certainty for the amount of improvement.

Could someone with a slightly better understanding of statistics chip in maybe? I could use some more information in my own A/B tests, sometimes I know a change is going to be a pain to maintain so I want to know not just if it is better, but by how much.

paraschopra13• 16 years ago

Since I am the one who wrote the case study, perhaps I can explain. Loosely saying, what the article actually means is that the observed difference of 28% or more may actually be true in 95% of cases. However, there is a 5% chance that that difference is a lesser than 28%.

If you have any specific question, please feel free to ask.

almost3• 16 years ago

I must be have been wrong then, my apologies.

How are you using to arrive at that conclusion though? When I do a A/B testing I (or rather, the software I use) use the chi-squared test to give a confidence value expressed as a percentage. When I get over 95% I know I have statistical significance and I end the test. At that point I also have an "improved by" number but I would have to collect much more data for that to be statistically significant as far as I understand.

almost3• 16 years ago

HN won't let me reply to your answer below (presumably to stop me from flaming you :p). But thanks, I'll read that blog post later but it looks like it answers my questions plus a bit more.

Seems my understanding was even less than I thought :)

retube• 16 years ago

you are right - without the sample sizes (for before and after) there's no way to know whether this measured increase of 28% is statistically significant or not.

whyenot• 16 years ago

The results could really benefit from some inferential statistics. The percentages, while interesting aren't particularly meaningful on their own. I realize you may not want to share how large your sample sizes are, but something like a Pearson's chi square test would be very useful here.

bdickason2• 16 years ago

Just wanted to point out that it seems like you guys are following the feedback in this thread: http://news.ycombinator.com/item?id=1351460

Which is totally rad :)

Now if only you could modify the video on your site or post a blog post to explain... how does my site get loaded into your 'visual editor?' Does all my css stay there? etc. The video makes it look like the site has magically been loaded into the editor and everything's intact :P

paraschopra13• 16 years ago

Yes, it is really magical.

Go, try the live demo http://dev.visualwebsiteoptimizer.com/first_time_ab.php?demo...

Do you think we should blog about the fact that you don't need to do any thing special to load a site into our system?

bdickason2• 16 years ago

To me, this is the #1. I would go so far as to place that 'first_time_ab.php' box directly at the top of the page. Kind of an 'enter your webpage and get started NOW' type call to action.

After seeing this link, it's totally clear to me and i will definitely sign up when we're ready.

einarvollset• 16 years ago

I think you'll find it is three words.

joshu• 16 years ago

Can we not talk about percent changes of percentages?

kd5bjo• 16 years ago

Why not? It makes perfect sense and conveys the relevant information. You should talk about percent changes whenever the important bit is the ratio between the new and old value.

paraschopra13• 16 years ago

Not sure I got your point. Can you clarify?

pdx2• 16 years ago

I agree. When somebody uses this sort of second order percentage, it is confusing.

When a car's top speed increases from 105mph to 120mph, after you modified the engine, you say it increased 15mph ... or, perhaps you could say it increased 14.3%.

Using this paradigm, when a conversion rate increases from 14.5% to 18.6%, you would say it increased by 4.1% ... or, perhaps you could say it increased 28.28%

Woops! We have a problem. We just expressed the same increase, in the same units, using two very different numbers.

This is obviously a problem. The reader doesn't know what you're talking about.

It's perfectly valid to say the car increased by 15mph, so it must be perfectly valid to say the conversion rate increased by 4.1%. In fact, I think that when most people read something like this, this is what they assume, that the delta between the old conversion and the new conversion is that percentage. At first blush, saying the conversion rate increased by 28.28% should be OK, but it's not OK, because the whole point is to convey information, and you have failed to do so. Now you have to use context, to figure out what the hell they meant. This is unacceptable, especially when talking about mathematics, we should not need context to sort out what's going on.

What's the answer? I don't know. But there's definitely a problem here.

paraschopra13• 16 years ago

>Using this paradigm, when a conversion rate increases from 14.5% to 18.6%, you would say it increased by 4.1%

No you cannot say it increased by 4.1% and that's because you are talking about per cent. I don't know why you would ever use 4.1% in this context. It is simply wrong.

May be you can say 4.1 percentage points but I don't see a need for it when you can say (and be right) that it increased by 28%.

How two magical words increased signups by 28%

Related Stories