Show HN – Apptimize: A/B testing for native Android, iOS apps

http://newton.apptimize.com/

by jorlow • 13 years ago

126 50 13 years ago

newton.apptimize.com

sync• 13 years ago

For folks interested in mobile A/B testing, also see an open-source alternative by Clutch: https://github.com/clutchio

jorlow4• 13 years ago

For what it's worth, using Clutch was one of the reasons we ended up making Apptimize. We ran into several bugs including one that was crashing the app we were working on and issues with its probability distribution. And the reporting was very simplistic.

Are you using Clutch now? If so, it'd be awesome if you could give us a try.

siglesias• 13 years ago

Additionally, incumbent mobile A/B testing service http://www.leanplum.com -- I've used them for a while now and it's a pretty robust system with good support. Worth checking out IMO.

pbharrin• 13 years ago

ClutchIO is not native, which is why something like this is needed.

jakestein• 13 years ago

Interesting, Artisan mobile is also in this space http://useartisan.com/

igorsyl• 13 years ago

Why should anyone choose to use Apptimize over a free open-source alternative?

jorlow4• 13 years ago

One of the biggest reasons is probably the reason behind using any software as a service rather than doing it yourself: you can concentrate on what you do best and not spend time managing things yourself. A couple of our users switched from using open source alternatives for that reason. We've put a lot of time into our infrastructure and decoupling things so we're very resilient against outages so you don't have to. We've also put a lot of time into making our library more robust and well tested than any alternatives we've seen to date.

chrisalvino3• 13 years ago

I'm guessing because it's better than what they could integrate and build up with an open-source alternative in a short amount of time and thus more cost effective.

philbarr2• 13 years ago

This looks interesting - although having never really considered using A/B testing on native apps before I did spend a few minutes looking around the site for examples of things to track. Maybe you could show an example project on the site showing what was changed and how that affected sales, etc.?

nancyhua5• 13 years ago

Part of apptimize team here. We want to do that! We will add more to our site; currently it looks deceptively slick bc our designer is awesome but it's really a first iteration.

Right now if you sign in, you can see an example experiment. How does it look to you?

philbarr2• 13 years ago

Yes the example gives a much better idea of what's going on. Although I did have to create an account to look at it...

Would be nice to have that accessible from the main page.

chrisalvino3• 13 years ago

Our organization often comes up with its own secondary metrics based on user data (not just CTR or usage). Is this possible when using Apptimize?

nancyhua5• 13 years ago

Part of apptimize team here! We let you track whatever goals you want, so people measure stuff like "user returned within a week" and "user used app." What goals do you want to track?

chrisalvino3• 13 years ago

That really depends on what we find to be important metrics. Flexibility is important since we need to be flexible too.

xoail• 13 years ago

I would like to see some code examples.

jorlow4• 13 years ago

If you register (takes 2 seconds) there's an example project where you can see the integration steps with the exact code you need to paste in. You're right though, we should definitely add examples to the main site.

tjbiddle• 13 years ago

I'd recommend not forcing people to register to see generic information like this - I know HN is biased towards "Connect with Facebook to login", I would assume most are also biased towards registering to see content - I know I am.

pr0filer__• 13 years ago

I really like the flat design. Nice and subtle. Also, Optimizely reference?

huhtenberg2• 13 years ago

I've got some karma points to burn, so here comes something.

A list of your clients would be nice to see. So that I'd know to never install any of their apps.

I hate apps that phone home. I hate apps that phone to some random analytics companies even more. I don't like companies that enable, entice and profit from this sort of behavior, a behavior that every single user would opt out from if they were asked upfront. So, unfortunately, the only thing I can wish you at this point is a speedy demise so that you'd be free to focus on projects that are less ethically questionable.

/rant

Rezo2• 13 years ago

How does the anonymous gathering of statistics harm you? As an app developer, this data allows me to measurably improve my apps.

I have several free (no ads either) apps in the Android market, and even in the most trivial of apps I will include Google Analytics to receive crash reports, see what screens are most popular (so I can improve my apps where it matters most), where the app is being used (for some reason one of my apps was really popular in France, I then added a French translation and the adoption really took off), what devices are most/least popular etc. If I add a new feature, I want to know if its being used or if I wasted my time, if I need to improve the discoverability of the feature etc.

Last but not least, the usage data is really motivating to me as a developer of free apps. If I release an app, and I see it being used daily, that makes me happy and motives me to keep updating it.

As far as I know, the GA data is anonymous and only shown to the developer in aggregates.

lobotryas• 13 years ago

I agree with you 100%. What it ultimately boils down to is the need for feedback.

Publishing an app is a conversation with your users instead of throwing a message in a bottle into the ocean. You need all kinds of data and metrics not only to see how well your business is doing, but also how you can help users have a better experience with your app. For better or worse, until telepathic feedback is invented, phone-home is all we have.

nileshtrivedi2• 13 years ago

If he is using your app, it implies that he somewhat trusts you. I think it's Google that he doesn't trust. Using an app shouldn't require trusting a third party with personal behavior data.

huhtenberg2• 13 years ago

As an app developer, this data allows me to measurably improve my apps.

I don't give a flying f#ck about that, sorry. This is my device and I don't like my device connecting god knows where at a whim.

As I said, if you were to tell your app users upfront that you will be tracking their usage, anonymous or not, and gave them an option to opt out, very few would not opt out. This is the exact same inherent need for privacy that is fueling current NSA debacle. It may be irrational, but it pretty damn real. People don't like to be tracked or watched.

pbsdp5• 13 years ago

The data as sent to Google (or any other analytics system) contains more than enough personally identifiable meta-data as to serve as a hugely useful database for the mining information on specific individuals.

Even just an IP address and device identifier is very useful metadata for assembling sweeping user profiles.

This creates a centralized data trove of hugely valuable data (why else do you think services collect it for free?) that is useful to both commercial and government interests.

This is why personal privacy matters, and why silently 'phoning home' without first getting explicit user permission is wrong.

gailees3• 13 years ago

Seriously? How do you feel about A/B testing on websites?

nileshtrivedi2• 13 years ago

He may have simply blocked all third party cookies and javascripts so it may not be a big deal to him.

pbsdp5• 13 years ago

Sending a request to a server is explicitly and inescapably part of requesting a web page; users explicitly grant permission for that request to be made obvious to you simply by making the request in the first place.

On the other hand, there's still an open question of what you do with that request data. Sending it to Google without permission is wrong, as it contains a slew of personally identifiable data that allows Google to build a massive profile of a specific user's behavior both across the internet, and in the real world.

Rezo2• 13 years ago

I couldn't find any pricing information even after registering. It only says its free to try, but no further clarification in the docs or the FAQ.

While it looks promising, I can't go and integrate this into my app without knowing if, when, and how much you're going to be charging for it.

nancyhua5• 13 years ago

We haven't announced our pricing yet because we're constantly adding new features our users request. Email me at nancy at apptimize dot com and we'll give you a special deal we've been giving to early adopters.

hrabago• 13 years ago

The lack of information on pricing is an instant back-button-trigger for me. I have no idea if you (eventually) plan to charge $5 per month or $200 per month. If it's in the lower end, then it can be worth evaluating. If it's a few or several hundreds per month, then I have to pass until I'm earning enough to afford it.

gailees3• 13 years ago

Do people actually do A/B testing on iOS apps? I feel like the two-week development cycle really hinders your ability to conduct valuable A/B tests with your users in time to actually make any change.

nancyhua5• 13 years ago

You're right the app store is a barrier but people still do it. That's one of the advantages of our product, that you can start and stop experiments so you can put in as many variations as you want. Currently people just suffer through the really slow time frame.

kunle• 13 years ago

Some of the more forward thinking folks (Socialcam and HauteLook for example) have rolled in house versions of these, but Apptimize is one of the first end to end solutions I've seen. But A/B testing in mobile is definitely something a lot of folks are thinking about.

michaelmior• 13 years ago

We build in DB-driven tests into our app so we can make changes at will once we find a winner. Then we just clean up the code to remove the test in the next iteration.

jasonlotito2• 13 years ago

Take a look at http://useartisan.com/

Basically, it removes the 2-week development cycle, and you can essentially make changes in real time. I've seen it demo'ed, and it's much more impressive than the website makes it out to be.

SupremumLimit• 13 years ago

It sounds interesting, but how do they get around the app store approval process?

thecodemonkey• 13 years ago

We faced the same problem, and built an A/B Testing Framework at a Hackathon this month that allows you to just drop in a line of code and run all the experiments through a web interface without app updates.

The framework is not quite ready to ship yet, but we built a fully working proof of concept, and are looking to complete it and put it out there.

You can read more and watch a video demo at the website: http://fliptest.io

Show HN – Apptimize: A/B testing for native Android, iOS apps

Related Stories