The Most Radical A/B Test I've Ever Done

http://www.kalzumeus.com/2010/06/04/the-most-radical-ab-test-ive-ever-done/

by patio11 • 16 years ago

127 32 16 years ago

travem• 16 years ago

My first thoughts are that removing the downloadable version will work out well by getting rid of the "Tyranny of Choice" and thus helping with conversions.

Has anyone else tried this approach?

michael_dorfman2• 16 years ago

Very interesting.

My initial thought (unencumbered by any actual knowledge) is that the folks who download the client version (instead of sign up for the web version) don't do so for any particular reason, so I don't think that removing the option should actually hurt.

Sounds like a good fit for Bayesian probability, using the old numbers as priors.

JoeAltmaier• 16 years ago

Downloaded app can be used offline.

michael_dorfman2• 16 years ago

But the question is: is that really important to the majority of the users who download it? Or are they acting blindly?

Now, it could be that there is a contingent of bingo-loving teachers who absolutely prefer to have the software locally installed, but I'm suspecting (without any good evidence, I grant) that this is not really a key decision-factor for most of Patrick's customers who choose to download.

bmelton• 16 years ago

I'm not paying to use the web version, but if it's written in HTML5 it can leverage the cache-manifest (or Gears, either way) and work offline as well.

Do you know one way or another that it isn't?

paraschopra3• 16 years ago

And I thought the post _had_ the results of this test :)

Anyway, interesting test and do keep us posted about the results.

fierarul3• 16 years ago

Isn't it odd to announce the A/B test when you've just started it ?

I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.

paraschopra3• 16 years ago

Somewhat agree with you. I recently blogged about why you shouldn't publicize that you are doing an A/B test http://visualwebsiteoptimizer.com/split-testing-blog/3-contr...

Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.

JoachimSchipper• 16 years ago

In what way would this influence things? Sure, it might draw some additional uninterested visitors, but that will just depress conversion rates all around, not exclusively for A or B. The results remain valid.

paraschopra3• 16 years ago

Yes but that changes the real difference between conversion rates of A and B. Even though trend won't get affected but actual numbers will. Not that serious but given that this problem is easily avoidable one may want to not announce the test.

fierarul3• 16 years ago

Assuming no actual possible customer reads this blog, this doesn't influences the number of new customers, but it does changes the overall numbers.

But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer.

Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.

Sukotto• 16 years ago

The chances of his userbase reading his tech blog and being impacted by something like this seems pretty small to me.

patio114• 16 years ago

If I woke up in the morning and had a shocking lack of worries, between two software products and client work I could probably find something real to worry about without having to manufacture worries out of the ether.

"Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.

fierarul3• 16 years ago

Again, I don't think blogging about this will influence your A/B test, because your (future) customers and people reading your blog don't overlap very much (or at all).

But, I do think publishing the A/B test does influence the outcome if they do overlap.

You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app.

I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.

dhimes2• 16 years ago

I'd be surprised if teachers didn't check out your blog before buying. Teachers "get" blogs, hell they LOVE blogs. They even get their students to blog (to build writing skills using peer interaction). Seems like the ones who are confident enough to go out and use your software may also be the ones who would check out the blog. Just a guess on my part, but at least keep an eye out for this.

px• 16 years ago

As a teacher, I think this is a smart decision.

Desktop applications are increasingly difficult to manage on a school computer. Many teachers return each September to a computer that has been wiped clean.

At the same time, the internet is becoming more and more accessible in schools. Teachers will be able to use the web application no matter which room they are in or which computers they have access to.

Another challenge for non-tech savvy teachers is syncing work between the school and home. I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.

maukdaddy• 16 years ago

As someone who works in information security at a very large district, I will say this is a brilliant decision! =)

The last thing I need is school personnel downloading Java apps. Not that yours is malicious - but once someone learns how to install/run something, it becomes exponentially more difficult to teach them to identify what is malicious and what is not.

+1 for syncing between work/school. Often the computers at schools are inferior to ones that teachers have a home, or at least a different platform.

patio114• 16 years ago

Many teachers return each September to a computer that has been wiped clean.

I hear that. Frequently, in September, from several dozen teachers asking me "How do I reinstall Bingo Card Creator on my computer which was just wiped clean? Also, do I have to pay?"

I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.

It totally solved that problem, where the feature I made in the downloadable version to do the same thing has been a catastrophic flop. (Twenty people have ever used it, and I still get emails asking to explain how to copy files onto a floppy so that they can be read on the machine at school. And then emails the next day saying that the computer doesn't know what a .bcf file is. And then emails the next day saying that they installed Bingo Card Creator and opened the file but it says it is the free trial which is odd because they know they purchased the program and they never had this issue at home [i.e. where they followed directions and put in their Registration Key].)

ErrantX2• 16 years ago

Teachers here in the UK have it really bad; a lot of the IT infrastructure is farmed out to BT or other completely useless corporates. Either what happens is they can't cope and so are dropped after a few months or they keep having to replace all the poorly configured laptops.

My Mum has been through about 8 new laptops in the last 14 months...

(the best one is when the laptop leads all went missing [never found out who did it] and their response was... ...to replace the laptops. rolls eyes)

And lets not even start to talk about the restrictions imposed on these laptops.... (I mean; I can understand it for the kids, but the teachers?)

markkanof• 16 years ago

"How do I copy the files to my niece’s computer? By the way it is a Mac and I use a Yahoo. Try using the web application."

This gem from the article really illustrates the added difficulty of supporting a desktop application vs a web application. For this particular request what OS she is running doesn't matter, but for many other requests it could be very important to know. Given the technical sophistication of this user, just figuring out what environment the application is running in could be an arduous process.

fdb• 16 years ago

I love this article. And his article on why he's done making desktop software is even better:

http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-a...

coolio732• 16 years ago

Enough of this uninteresting Bingo Card Creating program. How do these posts keep getting pushed up to the first page?

Grinnmarr• 16 years ago

I just wish I had enough rep to down vote your comment :P His posts and complete transparency about growing his business epitomize the type of community contribution that brings me back to this site time and time again.

coolio732• 16 years ago

printing bingo cards is a business any 12 year old with a computer could set up. Couldn't you do this on Print Shop and an Apple IIe? Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day. Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable. He should spend some of his blogging energy on developing a more interesting topic/business to blog about.

tomerico• 16 years ago

I've just searched google for:

Bingo cards Generator

Printable bingo cards

Create bingo cards

Bingo Creator

In all of them you are not one of the top 3 results. It might suggest that a better approach would be to work on your SEO than to increase your ad money.

The Most Radical A/B Test I've Ever Done

Related Stories