Ask YC: Do you test your site?

https://news.ycombinator.com/item?id=226147

by riklomas • 18 years ago

19 9 18 years ago

news.ycombinator.com

Have you used A/B testing or multi-variate testing on your site?

Did you find it useful if you did? What products did you use and did you find them easy to use? Was it effective and did you achieve the gains you expected?

If you haven't, why not?

Shadow843• 18 years ago

Hi,

I was responsible for Web Analytics in a Web Agency in Switzerland.

In my opinion A/B and multi-variate testing only has a real use if you have very high volume. If you have lower volume or are in a Beta or even a closed Phase of your site development cycle I would suggest to use direct usability testing where you watch people interact with your site and interview them about what the think about your site. This way I think you would get much better results for the same or less time and money.

Best regards

riklomas• 18 years ago

Whilst I agree that user testing is very important for a site, surely there's a difference between giving a task to someone to complete (e.g. telling someone to find a link) rather than seeing if random users can find the information that want themselves.

Plus if you're bootstrapping, it's cheaper to use something like Google Web Optimizer (as it's free) to test real traffic than paying people to interact with your site

fallentimes• 18 years ago

I don't think you necessarily need a high volume. I think you just need enough that's statistically significant for whatever you're testing. And like another commenter noted, google optimizer is free.

donal• 18 years ago

Most definitely, otherwise you are really just eye-balling it and you don't want to find out that your small sample-set wasn't representative of your actual user-base.

This is important considering that during early stages you could have early adopters and other more tech-savvy types using your site more heavily than what you will end up with.

Shadow843• 18 years ago

To reiterate what I meant about costs. For sure Web Optimizer is Free, but the software isn't the only thing you need, for multi-variate testing you need many variants of the item you want to change, you somehow develop without directions,which can be very time consuming and (maybe) expensive.

On the other hand you can also have direct usability testing for free, think family, friends (I would use my mother, because she is a user who is interested in the Internet in general but not very tech savy)

DanHulton• 18 years ago

Testing's critical. But there's multiple levels of testing, as far as websites go. At the very most basic level, there's things like Google's A/B optimizer. At the other end, there's things like Amazon's obidos - a tightly-integrated testing/recommendation engine.

It's going to come to the point where if you AREN'T live testing on some level, you're going to flounder and fall beneath competitors that are. And the better your testing, the better you'll do.

(Disclaimer - I work for a Sitebrand, company that tries to offer a middle-ground in this area. We do personalization, similar to what Amazon offers, and A/B testing of various content. My opinions are very influenced by what I've seen, which is that companies that sign on with us see significant growth because they're putting in the work, the testing, and the personalization.)

cianchette• 18 years ago

I've used A/B and multivariate testing at multiple companies. It can be incredibly helpful in optimizing a product.

I would suggest however to focus on building a solid base product first and then optimizing it.

For web the google web optimizer works ok and its free. There is a Boston startup called Sitespect who I've used in the past as well, which has a more robust platform. I can introduce you to them if you want.

craigbellot• 18 years ago

I test everything, headlines, images, buttons, etc. If you want to know what your users want, you can test your way there. Your users will tell you. Especially useful for e-commerce and lead generation sites. If you can increase conversions from 1% to 2%, you've just doubled your revenue. Google website optimizer works well for me.

Ask YC: Do you test your site?

Related Stories