Mixpanel (YC S09) releases macro-level A/B testing tool

http://blog.mixpanel.com/releasing-mixpanel-tests-a-way-to-ab-test-you

by trefn • 16 years ago

55 15 16 years ago

blog.mixpanel.com

jfong• 16 years ago

MP is the way to go!

ALee• 16 years ago

this was holy crap awesome, presented another perspective to us at Jamlegend.

mogston• 16 years ago

I've bee tracking their progress recently - This is just the start. Good to see that Posterous are already using their solution.

vinhboy3• 16 years ago

I am gonna get neg to hell for saying this, but MP is garbage. You can't even delete a funnel. Seriously. Try it.

adriand• 16 years ago

Really? Garbage?

Other hackers: I was about to install Google Website Optimizer for multivariate testing on a startup I've been working on. Should I try Mixpanel instead? Or does Google still have the edge?

adora2• 16 years ago

I've used both. Mixpanel is much, much easier to setup and use. Plus it's more helpful in understanding where your bottlenecks are.

vinhboy3• 16 years ago

Ok let me explain myself. MP is very easy and reliable to use, but the problem is when you get into their dashboard. It's hard to analyze your data since there are no reporting tools. In Analytics you can create really detailed reports to segment your data and measure against certain metrics, in MP you just get basically a click logger. Also my gripe with the funnel setup is that you can create an unlimited amount of funnels, but you cannot delete them. You can only view your funnel by week-duration and no other time frame. There are also no built-in baseline metrics. If you want to track user's browser, OS, or whatever, you have to log those yourself. Some might consider that a feature, I didn't enjoy it.

trefn• 16 years ago

Hi vinhboy, I'm sorry you had a bad experience with us. As you can imagine, we have a lot to do as a 2-person startup and we have to prioritize things. Deleting funnels just hasn't made its way up the priority queue yet =)

I'd appreciate the chance to talk to you further and get any other feedback you have. My email is tim@mixpanel.com. Thanks!

vinhboy3• 16 years ago

Hey Tim, and anyone else reading this. I don't mean to criticize, and I have spoken with you before and I know you guys are working hard. I just wanted to give others perspective on my experience. I hope you guys keep up the good work.

adora2• 16 years ago

nice I like it

patio11• 16 years ago

I love you guys, but there is something to be said for using words like other people in the industry use the words.

The first time I noticed this was wrapping my head around funnels, which is a subject I'm fairly deeply experienced with. The three magic words in that documentation are funnel, step, and goal. In every other analytics tool I've ever encountered, the word "goal" is synonymous with conversion: somebody went through the funnel and reached a result which was positive for the business. In Mixpanel, goal does not mean goal. Goal apparently means "step name": for example, the first goal of the seven step funnel that represents the core interaction for my website is the Dashboard goal. The second goal is the new card goal. The third goal is... you get the general picture. This breaks my brain every time I directly touch the API or docs. I wrapped it in my own API and try not to think about it too much.

That is the prelude to me saying: your A/B tests are something potentially wonderful, but they're not A/B tests. You're going to suffer an impedance mismatch when explaining them to anyone who knows what they're doing in testing because when you say the words "A/B test" we're going to think of something else.

An A/B test involves:

1) A user interacting with an element which is subject to the test. An element can be almost anything: a design feature, a bit of copy, an entire checkout workflow, whatever.

2) The user being randomly assigned into one of two groups and shown the appropriate variation on the element.

3) The user does some stuff.

4) Hopefully, the user converts.

5) You compare the propensity for conversion among the populations of users in each test group to see which performed better.

The solution you have delivered is different than A/B tests in many ways:

1) It does not randomly partition users into groups. That is, presumably, something your customers have to do prior to firing an event with the property (or super-property -- I forget what you call it) testWhatever: "alternativeA".

2) A/B tests are cool not because A and B are cool letters but because they make statistical significance testing really easy, so that if alternative A has 20% more conversions than alternative B you can quickly tell whether that is solid evidence that A is indeed better or merely a possible artifact of random variation. Your testing feature does not implement significance testing.

3) There would normally be some halfhearted bow in the general direction of independence at this point (we mostly gloss over that and hope people forget stats 101 since, in practice, just wishing this requirement away tends to actually work acceptably) but the entire point of your stats feature is, as far as I can tell, taking the independence notion and beating it black or blue or purple and then seeing how that choice of color interacts with six other things.

Again: I like the feature. I think that given an hour or two I will probably even be able to do something with it that will eventually make me money. But if you try to communicate this as A/B testing you're going to cause a lot of unnecessary confusion.

Mixpanel (YC S09) releases macro-level A/B testing tool

Related Stories