AWS Device Farm

Curious about what the built-in, no-scripting-required test suite can do. I wonder if this is what the app reviewers for the Amazon Appstore have been using.

Artemis2• 11 years ago

Amazon is coming for Google's [Cloud Test Lab](https://developers.google.com/cloud-test-lab/) and [Nativetap](https://beta.nativetap.io/).

ColinDabritz• 11 years ago

On a brief look, it appears that all of these choices (Including Amazon's new one) only support Android devices and not iOS?

I believe Xamarin Test Cloud includes iOS, although I'm not certain if it integrates outside of the Xamarin framework. http://xamarin.com/test-cloud

Seems like a gap they should all be trying to fill.

rasmuskl• 11 years ago

Xamarin Test Cloud is not limited to Xamarin apps for either platform.

Disclaimer: I work at Xamarin.

babababa• 11 years ago

Apple developer program licensing agreements make it very hard to (legally) provide iOS devices for remote testing.

yla922• 11 years ago

Pretty interesting. Now, we have more choices. The other day, I've found this project on Github called OpenSTF[1] which allows you to set up your own devices labs. It's pretty interesting and I've got to try yet. Generally, I'd prefer to set up my own device labs rather than testing with 3rd party services. It depends on your target market. For us, the apps we made are target for the local market (people and devices). And the phones sold in our market are different with the phone sold in States and I don't expect to have some China devices to be available on 3rd party services.

[1] https://openstf.github.io

shimo5037• 11 years ago

Hey that's ours :) Right now it's heavily focused on remote control rather than test automation, so it would not be a direct replacement yet. We just open sourced last week, though, so perhaps we can start looking into that in the near future after the initial batch of bugs is fixed.

jastanton• 11 years ago

Correct me if I'm wrong but does this feel pretty expensive?

Pricing is based on device minutes, which are determined by the number of devices you use and the duration of your tests. AWS Device Farm comes with a free tier of 250 device minutes. After that you are charged $0.17 per device minute. As your testing needs grow, you can opt for our unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device.

middleman90• 11 years ago

Yes, but consider that is not a virtualization, is a proper device

lubesGordi• 11 years ago

I'm guessing people will use this right before they push their first beta version, just as a quick sanity check against devices they don't already have. Most compatibility issues will probably arise from supporting pre-ICS devices and those are basically all under $250. Besides that, mostly what developers are going to be concerned with is how the app looks on different devices, having taken care of functional issues by developing on slower/lower memory devices in the first place.

Someone12342• 11 years ago

Strange, I was actually thinking the pricing was very reasonable. 250 minutes free is pretty generous and will likely be all many small developers need.

17c/minute/device is fine after that since you'll only likely need 20 minutes to test your average app against each device (and they only seem to support a handful). So $10~ to test your apps against a bunch of Amazon devices (ignoring the free 250 minutes), is fine by me.

The only reason I might look elsewhere is that other services offer a more broad range of Android devices, not just Amazon Fire stuff. So you can spin up a Galaxy S6, Nexus, Fire Tablet, etc all under one roof.

aalbertson• 11 years ago

I used to work for a mobile services company. We used technology similar to this. It is NOT cheap, nor easy to build. Some places give you a web interface that connects remotely to physical devices on the other side and some nifty robotics to control the phones/push buttons/etc... This is down right affordable!

martin_tipgain2• 11 years ago

Check us out at https://www.testmunk.com/ we do have iOS and Android support :) Don't hesitate to reach out if you have any questions.

techaddict009• 11 years ago

$ 790/mo arent you guys too costlier compared to AWS?

martin_tipgain2• 11 years ago

Amongst others, amazon doesn't provide iOS testing. With http://www.testmunk.com you will be able to run the same testcases on both iOS and Android. You will also get a QA Engineer for some hours per month that quickly assists you in getting the needed test coverage.

Someone12342• 11 years ago

That Flipboard animation/video (showing the report I guess) is super distracting/annoying. I literally deleted it with developer bar just so I could read your copy in peace.

Also calling a $800 package with 25 hours and 5x accounts the "starter" level seems a little odd. It is your lowest paid tier, but there's nothing starter-y about it. It is your normal tier. The free tier is the starter tier.

I have no idea what a "project" is. Seems like an arbitrary limitation. What do you care what code I am running with my hours? It is none of your business. Seems like you created the concept just so you could "sell us more of it" with the pro tier.

Also pro coming with only 5 users is odd. If I am paying for 100 hours/month of usage, that just seems like an annoyance. You'll force businesses into only giving out access to a handful of people rather than every single developer (and if so few people use it, the company might decide the pro tier doesn't make sense and downgrade to "starter").

Why is it so hard to find a list of devices you currently support? I actually googled it and still cannot figure out what devices your service offers. That seems like absolutely vital information for anyone even considering you.

Overall I like the service, but I like Amazon's model more. Just give me a cost/minute and get out of my way.

PS - You absolutely CAN justify charging more than Amazon. iOS support alone is a massive value add. I just like their pricing model, I am not proposing you charge 17c/minute/device.

cscharenberg• 11 years ago

Agreed on the animations. Why must everything be moving and sliding in? Just put elements like pricing section where they should be from the beginning.

jakozaur• 11 years ago

I wonder if they add IOS at some point. Maybe it's the Amazon way of doing. Start with MVP and iterate.

nickpsecurity• 11 years ago

I had an idea, more a need, for something like this many years ago. It came back to my mind when OpenBSD needed funding partly due to all the different pieces of hardware they test on. Wouldn't it be nice for portability-focused projects to be able to pool together resources on hardware to cut down on it?

Anyway, awesome to see AWS doing it in practice. As usual, it will be more interesting to see what happens when competition turns up. Cloud space has more innovation and cut-throat competition than many IT sectors. Can't wait to see what the competition costs. ;)

stickydink• 11 years ago

unlimited testing for a flat monthly fee of $250 per device

Renting remote-controlled Android devices for $250 a month, is this even remotely worth it? There aren't many devices that wouldn't pay for themselves by the end of the 2nd month...

oblio• 11 years ago

But then you have to manage the devices yourself :)

davnicwil• 11 years ago

Came here to comment this myself - was wondering if I read it right.

I don't understand the value proposition here at all. If you're subscribing for unlimited use then more likely than not you're all-in and are going to be testing for more than 2 or 3 months - probably continuously for years in fact.

At that scale doesn't the up-front cost and commitment of buying/managing the device make much more sense? What have I missed?

hexedpackets• 11 years ago

You don't pay for a specific device. You pay for unlimited runs using 1 device, but that device could change each run. So its sort of akin to leasing without an option to buy.

smackfu• 11 years ago

Generally the value proposition of these kind of services is that you can try to reproduce an issue that only shows up on a particular piece of hardware and OS. For instance, someone with iOS 7 on an iPhone 5, which you can't even buy new right now. Or a particular Samsung phone that wasn't very popular and was only sold for a few months.

RyJones• 11 years ago

The most crucial bit for me, which I don't see on the one-pager, is the ability to test devices in-market on carrier networks. Testing phones for Korean or Japanese carriers (or British, or Brazilian) on simulations or using GSM roaming is not good enough.

fbaptista• 11 years ago

ohhh :) take a look at another monkey at www.monkop.com

varelse• 11 years ago

So basically these guys?

https://en.wikipedia.org/wiki/Keynote_DeviceAnywhere

saurik• 11 years ago

I am always really curious to know how these kinds of services deal with security issues: they don't get to control the security of the devices they are running, and often will only even sort of have access to reflashing them; how do they deal with someone testing an app with a kernel exploit, installing a persistent backdoor, and then watching what everyone else later using that device is testing?

AWS Device Farm

Related Stories