ABalytics.js: Client-side A/B Testing With Google Analytics

http://danmaz74.me/2012/10/10/abalytics-js-client-side-ab-testing-with-google-analytics/

by danmaz74 • 14 years ago

60 20 14 years ago

danmaz74.me

ckluis2• 14 years ago

Very Nice! Personally, I would have loved a code example inside the post.

Thanks for going MIT.

danmaz748• 14 years ago

You're welcome!

I didn't think about the code example in the post... maybe I'll add it and resubmit the post ;)

pella• 14 years ago

more information about usage:

https://github.com/danmaz74/ABalytics

ckluis2• 14 years ago

Oh, I opened that. I just meant my favorite - check out my cool tool posts include samples of cool tool being cool. :)

jentulman• 14 years ago

looks nice and this is just an initial thought, it looks like the holding content in the HTML is replaced by either case A or B from the script, will it run with the holding text being case A and swapping out only when B is picked by the script therefore avoiding a possible content flicker some of the time? of course this may not be desirable on larger content.

danmaz748• 14 years ago

Yes, it would work - but I wouldn't advice doing so:

   * If the flicker isn't noticeable, it doesn't matter either way
   * If the flicker is noticeable and disturbing, having it only for one version could skew the results in favor of the non-flickering version

btilly• 14 years ago

My first thought upon seeing this was, "Where are the statistics so that you know if you are just ahead by chance?"

My second thought was, "Testing is so much better than not testing, this is still worthwhile."

Seriously, if you aren't testing, do it. If you are, this is something to graduate out of.

danmaz748• 14 years ago

OP here: That was the spirit of the thing. It isn't a fully fledged solution, but it's a very easy way to start testing, and in GA you can combine this data with all the other data you have there, if you want.

yahelc3• 14 years ago

I made a bookmarklet for easily calculating statistical significance, specifically for this use case.

After enabling the bookmarklet, you click the numbers you need and then it sends it to an A/B testing calculator hosted by the folks who made ABBA

Project: https://github.com/yahelc/ABBA-bookmarklet Sends to here: http://www.thumbtack.com/labs/abba/

Timmy_C• 14 years ago

Checking out the source code, this isn't going to work in IE8 and earlier without some small tweaks. IE didn't support `document.getElementsByClassName(string)` until IE9. It would be a small rewrite to make it work with IDs instead of classes (or you could use a shim).

[EDIT]: . . . But I like the idea. Might even use it next time I'm working with GA.

danmaz748• 14 years ago

Thanks for the warning! I initially developed this with jQuery, and only after seeing how little of it was needed decided to get rid of it - but I forgot to make all the due diligence. Later or this weekend I'll fix this - but if in the meantime anybody wants to send a fix or a pull request, I'll be glad to use it.

danmaz748• 14 years ago

LOL, I had forgotten: I already use a custom getElementsByClassName if there isn't one! So the it already works in IE < 9... check out the source code :)

theseanstewart• 14 years ago

Just me, or does anyone else miss Multivariate testing? It's impossible for me to use A/B testing with shopping cart software when it loads dynamic content (like the cart or checkout processes). Unless I'm doing it wrong?

siddharthdeswal• 14 years ago

Content that loads dynamically won't force you to use Multivariate Testing. In fact, most commercial services implement both A/B and Multivariate tests using client side JS.

Also, you can A/B test dynamically loading content so I think you just need to find out how to do it using your chosen software.

ohashi• 14 years ago

Great job. I came up with my own ghetto solution using event tracking, yours is definitely better.

yahelc3• 14 years ago

You can't reliably use Array#concat on the _gaq object. Once Google Analytics loads, it replaces the _gaq array with a custom object that has it's own push() method. That object doesn't have concat() on it.

So, your core coding style will only work with very particular GA implementations; anything that deviates from that could very easily break or be subject to race conditions.

You'll also get better results if you use events rather than custom variables; custom variable reporting isn't as flexible and can be very confusing (ie, visit start). It'll also mitigate the need to use concat() to cut the line, since Events can be fired at any time (whereas custom variables need to happen before the first hit of the session to be well-reported).

I also really like the idea, though. I actually pass all my Optimizely test results as Google Analytics events with this snippet:

     if (window.optimmizely && optimizely.variationNamesMap) {
		var optmap = optimizely.variationNamesMap;
		for (var expt in optmap) {
			if (optmap.hasOwnProperty(expt)) {
				_gaq.push(["_trackEvent", "Optimizely", optimizely.allExperiments[expt].name, optmap[expt], 0, true]);
			}
		}
	}

danmaz748• 14 years ago

Concat is only called before GA is loaded, so I don't think any race condition could happen.

yahelc3• 14 years ago

Only if everyone loads it the same way. But there are lots of different ways to load GA; the only reason _gaq.push() isn't scary and doesn't raise the specter of race conditions is because of the 'spoofing' GA does.

There are LOTS of different versions of the GA snippet floating around the internet; the assumptions this configuration makes would only work for some of them.

For example, some people load GA like this:

     <script async src="//google-analytics.com/ga.js">     </script>
     <script>
     var _gaq = _gaq || [];
     ...
     </script>

Folks who do this will break at minimum with older browsers that don't support async; not sure what it would do with async support, though.

It seems unnecessarily risky.

danmaz748• 14 years ago

I updated the library on Github: now it doesn't use concat anymore!

ABalytics.js: Client-side A/B Testing With Google Analytics

Related Stories