Automated Acceptance Testing on iPhone

Recently, I spent some quality time trying to figure out what would be the cheapest yet solid testing infrastructure for iOS and possibly Mac OS X (Lion). I was intimidated how immature is the tool support from Apple when it comes to continuous integration. Probably the most obvious thing about Continuous Integration is that one must be able to run the tools from the command line. I am pretty sure Apple does have automated acceptance tests (otherwise, looking at reasonable quality of their products, I would conclude that automated acceptance testing has no value at all) they just do not say complete truth (are they lying ?) saying they give to developers the same tools they are using.  So although Xcode has its command line equivalent (xcodebuild), it is very limited – you can build, and even (after fixing some scripts) run the tests, but forget about running the production code on the simulator or the device. How can we get Continuous Integration working for iOS, if we cannot easily run their stuff from the command line ? The short answer is, we can, but it will require some creativity and maybe a bit of common sense.

I like the testing infrastructure to be simple. When working with C++ I often prefer header-only frameworks like yaffut over more sophisticated frameworks like CppUnit. Some may say I am just lazy but I simply do not like linking agains libraries unless there is clear advantage of doing that. One such situation is when you need something more than just a unit testing framework, say you want to have a mocking framework as well. When you work on C++ the choice is limited, and can be reduced to GoogleMock (please continue reading even if you strongly disagree – it should be become clear why I am saying so). But what when the framework of your choice do not even compile on the toolchain you have to use ? You have three choices: 1) you make it work, 2) you build your own framework, 3) you lower your expectations. And as sometimes getting GoogleMock to compile is just to costly, I committed to write my own mocking framework for C++, which should ideally be header-only and which I could easily modify and extended when needed. And so I did. I called the framework Tiny Mock (https://github.com/mczenko/TinyMock) - it is (of course!) header-only, simple to understand and extend, and…requires some more work to make it really useful for a wider audience. But how the above rules apply when it comes to Apple and automated acceptance testing (command line) ? How many choices do we have ? When you got infected with automated acceptance testing (as I did) it is very hard to abandon perfection just because some ass hole at Apple (sorry Apple) does not feel the same urgency as you to do proper software engineering. From the number of initiatives and determination I observe in the community, this must be the mind set of may of us (infected with automated acceptance testing virus). And so if you want to have automated acceptance testing done right you may want to check the following projects:

Frank follows a healthy pattern of adding an http server to your app so that you can communicate with it and trigger different user interface (and not only user interface) actions. iCuke tries to bring Cucumber-like testing to iPhone, but it is not maintained any more. Both projects started way before Apple came with UIAutomation instrument, with which you can talk to your app using javascript. The clear advantage of UIAutomation instrument is that your app does not have to be modified in any way for testing, which is just great. And because Jasmin, a testing framework from pivotal which allows you to write tests for javascript in a nice behavioral style, is also written in javascript, the jasmine-iphone project combines the advantages of UIAutomation with Jasmine allowing you to feed UIAutomation instrument with Jasmine-looking specs. The only problem with UIAutomation is that currently it cannot be conveniently executed from command line (dammed Apple again). Still, UIAutomation is the step in the right direction and let’s hope Apple will help us running instruments from the command line in the near future. Frank and iCuke talk to the iPhone simulator using iphonesim and WaxSim respectively. The problem I see there is that both iphonesim and WaxSim depend on the private framework iPhoneSimulatorRemoteClient which is shipped with iOS SDK. The fact that iPhoneSimulatorRemoteClient is a private framework means that we cannot officially depend on it and Apple will not fill guilty if they change it significantly one day. I spent some time trying to get iphonesim and WaxSim working with the newest  iOS SDK 5.0, but I failed miserably. This brings me to the second important rule I try to follow when building acceptance testing framework (and actually pretty much all the software I develop): no magic.

No magic means that the third party code I am using to build my testing framework ideally should: (1) use only what is considered standard, and (2) should be easy to understand. The first point for apple means: no private frameworks.In the very same way I prefer not to use HippoMock mocking framework for C++ because it is based on non-standard features of the language, I prefer not to use iPhoneSimulatorRemoteClient because it is considered private. Easy to understand for me means, I can make it working in less than one day (sometimes I even say less than one hour) and I understand how it is doing what it says it is doing. Otherwise it si probably either to complex or not well maintained. Your testing framework does not have to be perfect from the start, let it grow with your project and keep it under control.While I think it is reasonable to follow what is happening with Frank at least, I also think you should be able to build something similar yet simpler yourself in just one day and than incrementally extend it when you need more. This is especially valid taken that lots of complexity in Frank can be removed when you take advantage of UIAutomation instrument from Apple.

A few words about using Jasmine with the UIAutomation instrument. I gave jasmine-iphone a try, but it did not work with Xcode 4.2 DP 2 and 3 and SDK 5.0. UIAutomation provides import functionality (which is not present in javascript) so that you can have better modular tests. The problem is that this import functionality in UIAutomation instrument is still quite restricted. For instance it fails when it comes to more complex hierarchies, where you import a script which in turns imports another script. This is exactly what jasmine-iphone is assuming to be valid, but apparently is not. In order to make it work you have to restructure the original scripts a bit. Because I like the approach a lot and I am going to use it in my projects now I prepared an example project and some tests so that you can try it right away with the newest Xcode 4.2 and SDK 5.0. You can find it at https://github.com/mczenko/UIAutomation-jasmine-iphone. Follow the README file for more information.

And how can we solve (temporarily at least) the problem that UIAutomation works only with Instruments. As mentioned already Instruments do not work from command line and they are not even AppleScript scriptable (but Xcode is too some extent). The only way is to use Automator to run your tests and some scripting to analyze the results. I know it is not really satisfactory four some perfectionist, but if I am perfectionist and I can do that than you can too :) .

We still have to take a look at the options we have when it comes to Unit Testing on iOS, but lets postpone it to the next post.

Tags: , , , , , ,

7 Responses to “Automated Acceptance Testing on iPhone”

  1. Peter Bindels says:

    > because it is based on non-standard features of the language

    What part of this is it that irks you?

    • marcin.czenko says:

      Peter Bindels on my blog. It is a honor ! All respect for Hippo Mocks. If I remember well, Hippo Mocks play with virtual function table, something I do not consider normal programming technique. If you one feels comfortable with it, go ahead, it will simplify your life writing mocks, but I like to understand what’s happening under the hood and remain in control :) .

      • Peter Bindels says:

        To me it’s the basis of unit testing – there’s an interface and *irrelevant of the implementation, the interface says what it will do*. You don’t have to understand the implementation, you don’t have to like it, but it will do what it says.

        If you can’t accept that for a tool you use – as long as it keeps up its side of the bargain – how can you do proper unit testing? You’ll have to assume the other side of your unit tests will act as the interface specifies anyway…

        • marcin.czenko says:

          The fact that I do not trust the tool which I know is using something that may violate the standard does not contradict my general ability to practice TDD. Or does it ? I know you are intelligent enough to realize that your analogy is far from being perfect.

  2. paddy says:

    Great article,
    I’m totally torn between Frank and UIAutomation at the moment. I’ve loved working with Frank, it’s Cucumber support & CI is brill. But feel, in the long run, UIAutomation will be able to add support for these features. It’s a tricky decision…

    BTW you can run Instruments from commandline in xCode 4.0

    http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/instruments.1.html

    only prob is you can’t get it to kick off playing the script. (unless you write some appleScript as well which I haven’t got working)

    • marcin.czenko says:

      Hi,

      Sorry for not answering. My page visibility is not very good yet, so I got accustomed that nobody reads what I am writing (and I get lots of spam that’s why I have to keep the comments for approval). I am very happy you like it and I hope it helps in something. Having said that, I even more onto using UIAutomation with Jasmine and Cucumber. I will certainly post something about Cucumber/UIAutomation integration, just have to find some time.

      BTW: The new Xcode 4.2 DP allows you to do more than just running instruments from command line. It allows even to execute the script and run it directly on the device. I will post about in a view days, was a bit unsure if this is fine with Apple to write about development preview features – but this part does not seem to break any Apple’s confidential stuff.

  3. I’ve worked with Frank quite a bit and found it really useful – although it takes a while to get your head around how it works, I can get fairly good Cucumber tests up and running in CI pretty quickly.

    I’m trying to get into UIAutomation now, finding it a real pain. The limited and buggy #import are killing me at the moment… but the main thing is that you can only use the very few features Apple has made available. It feels really restricted after using Frank, where I can edit the tool to do what I want.

Leave a Reply