Predicting Churn: When Do Veterans Quit?

http://www.gamasutra.com/view/feature/176747/predicting_churn_when_do_veterans_.php?print=1&utm_source=buffer&buffer_share=cb737

by ukdm • 14 years ago

67 32 14 years ago

gamasutra.com

praptak2• 14 years ago

This is a bit scary. Imagine your employer using machine learning on your git commit patterns (or swearwords per comment) to check when you might quit.

yummyfajitas• 14 years ago

I can imagine it. Why is it scary?

Would it also be scary if your manager observed your attitude/emotions through personal interaction and deduced you might want to quit?

dagw• 14 years ago

I'd be all for a tool like that, assuming it actually worked and it was used in a productive way. Imagine if you're getting tired of writing RoR code all day and pondering quitting and all of a sudden your boss comes up to you and says "hey how would you like stop coding Rails for a while and join a team that's about to evaluate Clojure for an upcoming project". That's just the sort of thing that could make everybody happy.

yarrel• 14 years ago

A manager who has time to log all my actions throughout the day then study them to try and guess whether he's doing such a bad job that I'm thinking of quitting has probably answered the question before they even start.

praptak2• 14 years ago

For me the difference is between expected and unexpected. I expect bosses to observe my attitude/emotions through personal interaction and even review my commits by hand (it's even mandatory where I work.) I do not expect them to compute extensive statistics on what I do in the repository and draw deep conclusions from that, it's somehow too close to what I consider private. It's obviously subjective.

benkant• 14 years ago

Google does something like this, apparently:

http://online.wsj.com/article/SB124269038041932531.html#mod=...

cadr• 14 years ago

If your boss needs machine learning to see that you aren't happy, there are much bigger issues.

westi• 14 years ago

This is really interesting.

I can think of great possibilities for using these methods to analyse the behaviour of users in any number of online services to identify the ones that might need some form of out reach to help them out and keep them as users rather than loosing them due to the problems they are having.

patio11• 14 years ago

You can do a very, very similar thing for non-game SaaS services, too. (Hubspot calls theirs CHI -- Customer Happiness Index. It attempts to predict who is going to quit based on backtested statistical models. In practice, you can get almost sickeningly good results with heuristics that take ~3 lines to program. You can also then save those accounts by something as simple as "Have the sales team email them and talk" or, in my recent experience, "Send them an automated email reminding them that the software exists.")

archon• 14 years ago

As a user, those emails have sometimes had a negative effect for the company emailing me. Something they send me reminds me that the program exists, that I'm still paying a monthly fee for it, and that I really don't use it very much at all.

But in all fairness, I doubt anything they could have said or offered would have changed my mind at that point. It was just a matter of when I cancelled, not if.

karterk• 14 years ago

The reason for emailing you could also be to find out why you aren't using the app. Even if one ends up closing the account as a result, the information gained is valuable for improving the product.

scott_w• 14 years ago

I think your last sentence is important - you were already gone, it was just a matter of time.

If the numbers work such that losing 1% of customers means keeping 3% i.e. the email prompts them to leave/stay respectively, then it's worth sending.

Getting rid of low-value users - in this case those not playing - can have long term benefits of focusing the business on getting more custom instead of getting comfortable on revenue that may disappear at any point.

rfergie• 14 years ago

Stochastic Solutions have several papers about this problem: http://www.stochasticsolutions.com/papers.html

In a subscription business who can be saved, who will leave regardless and then the tricky group of people who will leave if they get a sales call?

mfringel• 14 years ago

Alternately, noting them but not sending them an email at all.

Much like a gym membership, the people who don't use it at all but feel like they should someday (as long as they're not nagged about not using it) are one of the most profitable subscriber segments. Sure, they'll leave eventually, but they're not using your service anyway, and until then it's free money.

fooandbarify• 14 years ago

Patrick, just to be clear: Are the heuristics you're advocating here just something along the lines of "E-mail this user if they haven't touched their account in X days" or is a more in-depth approach required?

aaronjg• 14 years ago

That's a really interesting approach. Basically mining a couple of key features from the data set, and plugging it into a supervised learning algorithm. The really cool thing is the simplicity of the features used, just daily activity and daily playtime.

The write up of the methodology is very clear, but I'd love to see some more description of the results. Ninety-five percent accuracy is a pretty bold claim, and I'd love to see some ROC curves to back it up!

mturmon• 14 years ago

The high accuracy surprised me too.

To compute their accuracy, their methodology seems to require determining whether someone is a veteran user, and having a clear quit time for them (otherwise the user can't be used for training or testing). Maybe after you make these determinations, the resulting population is easier to deal with.

wccrawford• 14 years ago

My previous employer knew I was going to quit long before I told them.

But then, the fact that I was sending out resumes was probably a huge hint. I started sending them when I was a little disgruntled. Eventually, they heaped crap on me until I got serious about it and found a new job.

Not once did they attempt to do anything to stop me, and asking them to pay me what I'm worth was met with, "Can you wait a year?" This, despite the fact that every review I ever got was great. Not just good, great. The only complaints I got was that I was too quick to answer questions literally. If someone asked if something was possible, I told them. They wanted me to read their minds and ask the question they should have asked, instead of the question they did.

I eventually learned to do that, even, though. I have to admit it made things smoother. But jeez was it a pain.

zach• 14 years ago

The best response would be to offer a reason for people to come back after they leave for a week or so.

The worst response would to offer a reason for people to leave for a week or so if they weren't going to already.

If the company basically offers an incentive for not logging in, word will get around very fast. Then players have an excuse for taking a break, that they are gaming the system to earn an incentive.

Penalizing players is ineffective too because you want to welcome back your wayward customers instead of starting to burn bridges ("I'm going to throw out your stuff! Okay, I'm putting it in the trash right now!")

So unfortunately, I think all a company can do is have emails and community managers get in touch. Of course, that encourages players to stop playing for a week whenever they want to escalate a customer service issue. It's like you can't win.

I discovered at Blizzard that guild features were enormously strategic. When players get into an active guild, especially with people they already know, it becomes hard to leave. The glue that keeps players put is social expectations.

ShabbyDoo• 14 years ago

The article cites customer service interactions as the obvious (and obviously effective) remediation step once a player is identified as likely to quit in a few weeks. I'd like to know if this is really true. What if the game's fundamental problem is that players become bored over time? What can a CSR do to fix this? Probably not much. The analysis shows that changes in gameplay frequency are far from a random walk and do indeed predict future declines, but it doesn't offer much in the way of clues for how to keep customers engaged. What if it could be shown that a breakdown of a player's in-game social structure (I'm not a gamer, so I'm being vague) also predicts churn? Perhaps helping the player to find some new relationships would be effective. Is game play repetitive? Maybe decreasing frequency of novel experiences predicts churn? This would be good feedback for game designers. Predicting the future isn't of much valuable if it isn't actionable.

bloaf• 14 years ago

I bet you could apply this method to other participation-based business. One that springs immediately to mind is gym membership.

pydave• 14 years ago

There was an article with a similar goal previously posted to Gamasutra from altdevblogaday, but it's got data and software for you to play along: http://www.altdevblogaday.com/2012/08/10/business-analytics-...

Predicting Churn: When Do Veterans Quit?

Related Stories