Finding your One Metric That Matters using SQL

http://blog.getmetrica.com/post/48153893395/finding-your-one-metric-that-matters-using-sql

by dcraw • 13 years ago

15 2 13 years ago

blog.getmetrica.com

condiment• 13 years ago

Useful post. NTILE operations are also useful for evaluating the results of 50/50 tests against revenue or other metrics where your data fits an exponential curve, or where outliers tend to skew the performance of one group incorrectly. After normalizing your datasets so that each group has identical numbers of users, a graph comparing the distribution of your metrics on a percentile basis can make it trivially easy to visually evaluate the results of a test.

As far as using NTILE to find the one metric that matters is concerned, it's a useful tool, but the challenge isn't in structuring and executing SQL queries so much as it is deciding which metrics to focus on and attempt to correlate to user retention.

In my experience there can be hundreds of metrics to search through, and correlations between your metrics and user retention are not always obvious at face value, or worse, they might represent a tautological consequence of retention without being the cause of retention (ie, the user has lots of badges because the user is active on the site, but they are not active because of the badges). In these cases, you risk investing valuable development resources optimizing for something that will have no impact on the bottom line, when that time might have been better spent trying to identify novel metrics and getting them into the database for analysis.

dcraw• 13 years ago

Thanks for the comment. I'd love to see an example of the kind of 50/50 test analysis you do when you have time for a post.

I agree the hard part is the analysis, and I mentioned some of the challenges in the post. I hoped to use it as a way to illustrate some cool SQL techniques, rather than as a step-by-step method of finding the right metric, which I agree is much more complex task. Hopefully this analysis helps evaluate a potential metric to find out if it really is useful.

Finding your One Metric That Matters using SQL

Related Stories