The State of the Art in iOS 10 Test Automation

http://bitbar.com/the-state-of-the-art-in-ios-10-test-automation-frameworks/

by carlsbaddev • 10 years ago

31 6 10 years ago

goku7• 10 years ago

iOS 10 is the buggiest iOS I have seen so far. youtube starts playing in background when on a call mixing the audio. voicemails keep getting deleted when they reach 20 second mark. the screen freezes very frequently. swiping is confusing delete lock screen notifications and switch on the camera. is it bad product design or bad testing

carlsbaddev• 10 years ago

All that, plus it is damn hard to set up everything for test automation.

joeblau2• 10 years ago

I've had the worst experience in testing with Appium while working at Amazon. The test suite was extremely slow, unreliable, and tests are written in a language that's not native to the platform. The whole Appium testing interface is a hodgepodge UI that had a lot of broken UI interactions.

Our team members who were proponents of Appium made arguments like:

- Business people can write test cases - That never happened

- You can write cross platform tests - Yes, but we end up with all of these one of cases if iOS and Android aren't written with the exact same naming and design conventions which forced multiple testing paths anyways.

KIF is also not necessary anymore since Xcode 7+ support UI Test Case.

UI Test Case(for interaction), Unit Test Case(For Unit/Performance Tests), and Snapshot Testing[1] is way better combination. You write statically compiled Obj-C/Swift, tests run a lot faster, the build on the native xcodebuild toolchain, and your tests are maintained as a first class citizen.

[1] https://github.com/facebook/ios-snapshot-test-case

mikestew2• 10 years ago

Sadly, when I clicked the "up vote" button it disappeared so that I couldn't click it again...and again.

On my current project, the client was apparently standardizing on Appium. Ooookay, don't know why you're not using the included tools, but I'll give it a whirl. Sounds like something management, who will never use the tool in real life, got sold on because "cross-platform". But let's put my cynicism aside and give it a fair shot.

It took me several tries to get it installed. The documentation even says that if installation fails, try it a few more times. No, I'm not making that up. Once installed, configuration is a poorly-documented nightmare. But after half a day I have it working on iOS and Android.

As you point out, O...M...G is it slow. Now, I understand why and I don't necessarily blames the dev for the sixteen levels of abstractions this thing has to go through. But slow it is. Tests take a long time to run as they wind their way through the Xpath to get to the UI elements.

Writing test cases is even worse. No UI recorder to speak of to at least take a first whack at it. No good accessibility viewer tool to help with control names and characteristics. And since you kind of have to just guess-compile-run, combined with the slow runtime performance, even simple tests can take an hour to write.

"Business people can write test cases"? Did I miss this outstanding feature? I can barely get a test running, and I've been writing code for decades. And you're telling me some woman with a business degree is going to come in and start knocking out UI tests?

So after assuming that I must be the idiot in this equation given that I do not see the Highest Holiness of Appium that others must see, I started asking around. Hey, you, full-time SDET: you folks are using Appium, right? How's that going for you?

Turns out they've written about 20 tests, with an average of 60 seconds to run each individual tests. 20 minutes to run 20 tests. I'd planned on writing hundreds. Meaning it is entirely possible that the UI test suite I hoped to write would not be able to run on a single platform in a single work day. So much for running them at build time.

EDIT: Oh, yeah, and cross-platform: I can barely get dev to put accessibility labels on stuff for a single platform, let alone co-ordinate between the two. And unless the UI is mostly the same, using the same classes of controls with exactly the same accessibility labels (IOW, will never happen in the real world because we have other stuff to do), cross-platform is a pipe dream.

I realized that the Emperor has no clothes, and went back to using XCTest and Espresso.

joeblau2• 10 years ago

Yeah, I didn't detail the trails and tribulations of our experience but it was terrible. Our Appium test suite took 6+ hours to run and it falsely failed 100% of the time. We would finish a feature, check our code in run the test suite over night and things would be broken and then we'd have to manually verify the fixes were correct in the A.M. Then go back in and fix Appium.

I honestly wish I still had the link to the Wiki where I put my full Appium deep dive. Appium is great if you want to work on Appium and not actually build your own iOS or Android product.

The State of the Art in iOS 10 Test Automation

Related Stories