IOS Integration Testing by Square

http://corner.squareup.com/2011/07/ios-integration-testing.html

by sahillavingia • 15 years ago

133 27 15 years ago

corner.squareup.com

SeoxyS• 15 years ago

Wow! I think I just came.

Seriously, unit testing in iOS is not great, and this kind of simulation is exactly what we need. Regular unit tests are great for model-layer stuff and data-processing code, but they will not help with most real-world problems that occur.

"Does my login flow still work?" and "Can I still use every feature of my app successfully?" is the kind of thing you need to know after committing code.

puls2• 15 years ago

Any questions? We'll be around to answer them.

ecito2• 15 years ago

How hard would it be to add gesture support? double tap here, swipe left there, etc.

Are accessibility labels the only way to reference views?

puls2• 15 years ago

Indeed, accessibility labels are the only way to reference views. Granted, your tests are running within your process, so they sky is pretty much the limit.

We're still considering the best way to implement gestures. There's some rudimentary support, but it's incomplete right now. Your input is welcome!

firestone2• 15 years ago

Also, keep in mind that many views just "do the right thing" as far as accessibility goes. Buttons for example will use their title as the default accessibility label if you don't set one explicitly.

hinathan• 15 years ago

Nice! I've been looking for some proper iOS functional testing for a weekend project. Along with WaxSim this feels like enough plumbing to roll into a Jenkins plugin.

micampe2• 15 years ago

Why didn't you use UIAutomation? Was it lacking?

(http://developer.apple.com/library/ios/#documentation/Develo...)

spindel3• 15 years ago

1. You can't automate running of the UIAutomation tests. Square really wanted this for continuous integration.

2. Writing the UIAutomation scripts is really tedious (though it's much better in iOS [redacted]).

3. The UIAutomation tests ran very slowly.

4. Having to write all the tests in JavaScript isn't very flexible.

That said, just like UIAutomation, KIF stands on the shoulders of the great accessibility features in iOS. In many ways KIF is like an Objective-C version of UIAutomation.

adolph2• 15 years ago

Does someone have to watch the integration test occur, or does it report errors to something?

firestone2• 15 years ago

It should report errors automatically. For the most part, it will detect problems automatically because a step will fail to execute. The reason is reported in the log that KIF creates. For example, if you had steps to (1) enter the user name (2) enter the password (3) click "sign in" (4) hit the refresh button

If the login failed for some reason, then step (4) would fail because it wouldn't be able to find the refresh button.

You can also add explicit verification steps, such as "wait for the view with the accessibility label 'Refresh'" if you want further validation.

dominostars• 15 years ago

Thus far, my team has likewise been unable to find a good functional testing suite for iOS. We will definitely be giving this a shot.

angerman• 15 years ago

Ok, this might be somewhat dumb; I'm just starting to get into testing. How does KIF[1] compare to Kiwi[2]?

[1] http://corner.squareup.com/2011/07/ios-integration-testing.h...

[2] http://http://www.kiwi-lib.info/

spindel3• 15 years ago

Kiwi is for unit testing, KIF is for integration testing. They don't do the same thing (nor try to). Since you said you're just starting to get into it, here's what that means:

Integration tests simulate a user using your app - tapping on controls, manipulating views, typing in text, etc (see the video in the blog post). As these steps run you have simple tests to make sure the app is behaving as you expect it to.

Unit tests exercise more discrete things like model objects, controller logic, validations. Tons of small tests like "when we add these money values together, is the sum what we expect?".

At Square we use KIF for integration tests and GHUnit for unit tests. We run both on a continuous integration system (CruiseControl.rb) that runs the tests whenever we push new code.

Klonoar• 15 years ago

The best thing I got out of this is that I discovered Square's GitHub account... with an astonishing 65 repositories. I already held Square in high regard; this just icing on the cake.

rafaelc• 15 years ago

Project followers might be a more relevant metric for impact than the number of repo's

cachemoney• 15 years ago

Indeed. I have over a hundred repos, and I'm useless ;)

senthilnayagam• 15 years ago

totally agreed :) I discovered border_patrol, rack-timezone, rack-geo

maximilianburke2• 15 years ago

Semi-related, does anyone perform automated unit test deployment, execution, and reporting on iOS devices? If so, how do you do it?

At my current company we have a rather large automated test infrastructure that builds and runs ~50 or so individual libraries and their associated unit test programs on around 20 different configurations for 10 different platforms. Most of these platforms have a way of doing an unattended deployment of our test programs (and associated data) to the device, execute the programs, and retrieve the output.

This isn't working on iOS though because of its dependency to Xcode. We've tried scripting Xcode with AppleScript but this is fragile and routinely breaks.

Has anyone tackled this problem and succeeded?

aaronbrethorst• 15 years ago

Have you tried xcodebuild? You should be able to execute unit tests from the command line with it.

ecito2• 15 years ago

scripting xcode UI to build a project? ouch, you're doing it wrong. Check out xcodebuild.

http://blog.jayway.com/2010/01/31/continuos-integration-for-...

pkaler• 15 years ago

Yup. I've setup a Jenkins server that uses a combination of xcodebuild and xcrun. The process:

  - clones the repo from git
  - builds with xcodebuild
  - signs/provisions with xcrun
  - creates a manifest file for over-the-air installs
  - copies the build to a server where users can install
  - saves off the symbol file for our crash server so it can symbolicate crashes
  - sends emails

It would run unit tests and integration tests if we had them. :)

maximilianburke2• 15 years ago

Thanks for the xcrun hint! Does that still handle the application/data deployment and execution, or does it just automate the packaging? All the information I can gather about doing OTA deployment points to still requiring the user to initiate the download, and then run it. I'd like to avoid that if possible because of the sheer number of libraries we need to test.

netpenthe• 15 years ago

how does this compare with selenium ?

falava• 15 years ago

A nice collection of iOS testing posts:

http://softnoise.wordpress.com/2010/11/14/ios-setting-up-a-t...

IOS Integration Testing by Square

Related Stories