Recently, I spent some quality time trying to figure out what would be the cheapest yet solid testing infrastructure for iOS and possibly Mac OS X (Lion). I was intimidated how immature is the tool support from Apple when it comes to continuous integration. Probably the most obvious thing about Continuous Integration is that one must be able to run the tools from the command line. I am pretty sure Apple does have automated acceptance tests (otherwise, looking at reasonable quality of their products, I would conclude that automated acceptance testing has no value at all) they just do not say complete truth (are they lying ?) saying they give to developers the same tools they are using. So although Xcode has its command line equivalent (xcodebuild), it is very limited – you can build, and even (after fixing some scripts) run the tests, but forget about running the production code on the simulator or the device. How can we get Continuous Integration working for iOS, if we cannot easily run their stuff from the command line ? The short answer is, we can, but it will require some creativity and maybe a bit of common sense.
I like the testing infrastructure to be simple. When working with C++ I often prefer header-only frameworks like yaffut over more sophisticated frameworks like CppUnit. Some may say I am just lazy but I simply do not like linking agains libraries unless there is clear advantage of doing that. One such situation is when you need something more than just a unit testing framework, say you want to have a mocking framework as well. When you work on C++ the choice is limited, and can be reduced to GoogleMock (please continue reading even if you strongly disagree – it should be become clear why I am saying so). But what when the framework of your choice do not even compile on the toolchain you have to use ? You have three choices: 1) you make it work, 2) you build your own framework, 3) you lower your expectations. And as sometimes getting GoogleMock to compile is just to costly, I committed to write my own mocking framework for C++, which should ideally be header-only and which I could easily modify and extended when needed. And so I did. I called the framework Tiny Mock (https://github.com/mczenko/TinyMock) - it is (of course!) header-only, simple to understand and extend, and…requires some more work to make it really useful for a wider audience. But how the above rules apply when it comes to Apple and automated acceptance testing (command line) ? How many choices do we have ? When you got infected with automated acceptance testing (as I did) it is very hard to abandon perfection just because some ass hole at Apple (sorry Apple) does not feel the same urgency as you to do proper software engineering. From the number of initiatives and determination I observe in the community, this must be the mind set of may of us (infected with automated acceptance testing virus). And so if you want to have automated acceptance testing done right you may want to check the following projects:
- Frank: https://github.com/moredip/Frank.
- iCuke: https://github.com/robholland/icuke.
- iphonesim https://github.com/jhaynie/iphonesim.
- WaxSim https://github.com/robholland/WaxSim.
- jasmine-iphone https://github.com/pivotal/jasmine-iphone.
No magic means that the third party code I am using to build my testing framework ideally should: (1) use only what is considered standard, and (2) should be easy to understand. The first point for apple means: no private frameworks.In the very same way I prefer not to use HippoMock mocking framework for C++ because it is based on non-standard features of the language, I prefer not to use iPhoneSimulatorRemoteClient because it is considered private. Easy to understand for me means, I can make it working in less than one day (sometimes I even say less than one hour) and I understand how it is doing what it says it is doing. Otherwise it si probably either to complex or not well maintained. Your testing framework does not have to be perfect from the start, let it grow with your project and keep it under control.While I think it is reasonable to follow what is happening with Frank at least, I also think you should be able to build something similar yet simpler yourself in just one day and than incrementally extend it when you need more. This is especially valid taken that lots of complexity in Frank can be removed when you take advantage of UIAutomation instrument from Apple.
And how can we solve (temporarily at least) the problem that UIAutomation works only with Instruments. As mentioned already Instruments do not work from command line and they are not even AppleScript scriptable (but Xcode is too some extent). The only way is to use Automator to run your tests and some scripting to analyze the results. I know it is not really satisfactory four some perfectionist, but if I am perfectionist and I can do that than you can too .
We still have to take a look at the options we have when it comes to Unit Testing on iOS, but lets postpone it to the next post.