Amazon’s Device Farm: First Reaction
Today Amazon launched it’s device farm service. Simply put, it’s a cloud of real devices that can run your tests, while renting it for cheap from Amazon. So, I wanted to give it a try on a simple Calculator app that I wrote quickly. Here is a screendump of an Espresso test running on AWS device farm.
Although the test took a decent amount of time to run as compared to local, the results look impressive. Without any setup, I was able to run the test on the cloud and get instant feedback. One device (LG) had an error while running the test and that was probably a device in a bad state that Amazon needs to reboot.
With the device farm, you can also run the Android monkey tool easily. It’s available under the built-in Fuzz option and the options are somewhat similar to those you can pass to monkey via command line. However, there were some functions for controlling device state (GPS, Wifi etc) that I think monkey doesn’t support.
My simple Calculator app didn’t have proper checks for invalid expressions, and that was caught by the monkey tool as shown in the screenshot here. The bad part though is that you can’t reproduce the error easily as monkey randomly tests your app. However, you might be able to figure out what went wrong by looking at the stack trace or the logs available for download.
Some bugs/issues I encountered while using the service:
- By default, Amazon let’s you run tests on 5 devices, which they call Top devices but for me this set kept changing between runs. It seemed more like currently available devices in the free quota pool instead of Top devices.
- For some tests, the performance tab didn’t show any information. Perhaps my tests were too fast or something broke.
How much does it cost? The service comes with a 250 hour free trial and after that you need to pay $0.17 /hour/device or $250 /mo/device. Although this seems to look like a large number, enterprise companies pay way more for managing the devices in terms of personnel costs and the test labs are always in fire-fighting mode after things break due to OS or library upgrades.
Amazon’s service competes with other similar services that have existed in the market and the upcoming Google Test Cloud. Amazon jumped into this competition by acquiring an existing startup AppThwack. Note that Google also acquired testing startup Appurify last year for its test cloud. Indeed these are great times for Mobile testing!
Resources:
PLUG: If you’re paying per minute to run tests, they better run fast! Google’s Espresso runs the fastest tests without sleeps in your code. To generate Espresso tests easily, checkout BARISTA!
Any thoughts? Leave me a comment?
