On issue tracking and ticketing

I often have to help clients set up a ticketing system, or wrangle an existing one into a more useful state. Here’s what I’ve learned.

The three key items of a bug report

The first and most important step is to educate the support team to ensure that any bug report has the following before it gets assigned to the development team:

  1. Steps to reproduce.
  2. Expected outcome.
  3. Observed outcome.

It is particularly important to describe the expected and observed outcomes, so as to flush out any differences in expectations that the developer might have vis-a-vis the users and testers.

As a rule, the development team should simply bounce back any tickets assigned to them that don’t include these three items, with a request for clarification. Some exceptions can be made, such as when collating reports concerning a bug that is proving hard to reproduce in the lab.

Severity ranking

I have on occasion seen ticketing systems where the severity choices were too subjective to be useful, my favourite example being “Severely affects user experience” versus “Slightly affects user experience”. A better approach is to base severity upon objective criteria.

Without further ado, here is a severity ranking based on objective criteria that has served me well.

  1. Security leak An attacker can see or modify data for which they are not authorised.
  2. Data loss Some data that the user created is irretrievably lost.
  3. Crash/Lockup The app crashes or stops responding to input and the user must kill it, but no data is lost.
  4. Showstopper Something is not fit for purpose and there is no acceptable workaround.
  5. Painful Something is not fit for purpose but there is a workaround.
  6. Not up to spec The expected behaviour per the spec doesn’t match the observed behaviour.
  7. Confusing The expected behaviour per the spec turned out to be misunderstood by real-world users.
  8. Annoyance The behaviour is as specified, but has been found to be annoying in practice by real-world users.
  9. Change request The behaviour is as specified, but we would like to modify or extend the spec for business reasons.
  10. Work unit A job that needs to be done, typically to facilitate one of the above or to pay off technical debt, but that has no user-facing impact.

Responsibility

A ticket with your name on it should be considered a responsibility: a to-do list item, a hot potato. If it is assigned to you, that means it is on your plate and you darn well need to do something about it, or follow a defined workflow to assign it to someone else.

It is vital that nobody should have a way to hide a ticket under the carpet.

It follows that it vastly degrades the utility of a ticketing system if it has dummy accounts like “Anonymous” or “Tester 1”. Don’t do that: it serves only to create a means to hide tickets under the carpet. Every account should refer to a named individual.

If somebody leaves, then their account is closed, all open tickets assigned to them are assigned back up to whoever they reported to, and it becomes that person’s responsibility to re-assign the tickets as seen fit. The “hot potato” principle.

Monitoring

Time pressure and human fallibility are facts of life. No ticketing system can help unless there is somebody actively monitoring it, prioritising tickets that need attention. Generally, that person is the project manager. If you’re a solo shop, that person is you!

Knowing when to give up

Some tickets are intractable, for a variety of reasons, and your time is limited. It’s a judgement call to know when to close a ticket as “Will not fix”, which I can only leave to you. I’ll simply say that sometimes it’s the right thing to do.

My coding guidelines

I’ve been sitting on this document for some time, and decided that it’s finally time to pull my finger out and release it.

These are my coding guidelines. They are not layout guidelines: in fact they are explicitly layout-agnostic.

In my preamble I make it clear that these are guidelines, not tram-lines. Additionally, I have given a rationale for each of the guidelines and described some permitted exceptions from them.

Scrivener is great for composing a document like this, but I haven’t yet figured out how to get it to compile to HTML with all the nice formatting that I get when I compile it to PDF.

So, it’s a PDF, I’m afraid. The permalink for the latest PDF is here.

The versioned Scrivener source is here under MIT licence. It is a living document, so comments and pull requests are welcomed!

Share and enjoy!

Migrating from SVN to Git, preserving branches and tags

Migrating from SVN to Git, preserving branches and tags

SVN was a great advance in its day, but it’s now clear that distributed version control systems are the way forward and that Git is the de facto standard. Having helped many clients migrate from SVN to Git, here are my notes for a pain-free transition that will preserve the tags and branches in your SVN repository.

While round-tripping between SVN and Git is entirely possible using git svn, I shall not discuss it here because my best advice is simply to embrace Git and leave SVN behind.

Contents

First import to a local staging repo

Examine the staging repo

Convert the SVN tags and branches to local Git tags and branches

Test pushing and cloning locally

Push your staging repo to the real host

Cleanup

First import to a local staging repo

Create a local staging directory

cd ~
mkdir staging
cd staging

Obviously you don’t need to call it ‘staging’. This directory can be wherever you like, you can always move it later. The rest of this document assumes that this is your current directory.

Initialise git svn

For a standard SVN layout

git svn init SVN_URL --stdlayout --prefix=svn/

Here SVN_URL is the fully qualified URL for the directory under which the standard, case-sensitive SVN directories trunk, branches and tags reside.

I highly recommend the option --prefix=svn/. It means that all the existing svn branches and tags will be prefixed with svn/, which helps prevent Git users from mistaking them for “native” Git branches and tags.

For a non-standard SVN layout

Suppose you had decided to depart from the standard and capitalise your “Trunk”, “Branches” and “Tags” dirs in SVN. No matter, we can use this:

git svn init SVN_URL -T Trunk -b Branches -t Tags --prefix=svn/

This syntax lets you describe any non-standard SVN layout. The -b and -t parts can be repeated as many times as necessary. See man git-svn for details.

Optional: review the config

git config --local --list

Among other output you will see a section like this:

svn-remote.svn.url=svn://svn.example.com
svn-remote.svn.fetch=some/path/trunk:refs/remotes/svn/trunk
svn-remote.svn.tags=some/path/tags/*:refs/remotes/svn/tags/*

Advanced users can tweak the config at this stage before performing the fetch.

Fetch from SVN to the staging repo

git svn fetch

Depending on the size of the repo and your bandwith, this can take anything from a minute to some hours. This initial fetch is particularly slow because git svn has to do a lot of O(N^2) work to figure out and translate the history. Rest assured that subsequent fetches will be dramatically faster, mere seconds usually.

When this command at last completes, you will have a fully-fledged Git repo that you can examine with the command line or your preferred Git GUI. But resist the temptation to make any changes to it just yet.

Back to Contents

Examine the staging repo

git status

You should see:

# On branch master
nothing to commit (working directory clean)

Now try:

git branch -a

You should see something like:

* master
  remotes/svn/tags/0.1.0
  remotes/svn/tags/0.2.0
  remotes/svn/tags/0.3.0
  remotes/svn/tags/0.4.0
  remotes/svn/trunk

Note that the SVN tags and branches (in this case there weren’t any branches) exist only as remote references. In the next section we’ll fix that up.

Back to Contents

Convert the SVN tags and branches to local Git tags and branches

SVN branches

To convert the remote SVN branches to local Git branches, use:

for branch in `git branch -r | grep "branches/" | sed 's/ branches\///'`; do
  git branch $branch refs/remotes/$branch
done

Yeah, I’m not a fan of bash script either, but it gets the job done.

SVN tags

Here it’s really a matter of taste and/or house policy whether you convert them to Git tags or Git branches, because in Git they’re much the same thing: bookmarks into Git’s directed acyclic graph of commit objects.

To convert the remote SVN tags to local Git tags, use:

for tag in `git branch -r | grep "tags/" | sed 's/ tags\///'`; do
  git tag -a -m"Converting SVN tags" $tag refs/remotes/$tag
done

To convert the remote SVN tags to local Git branches, use:

for tag in `git branch -r | grep "tags/" | sed 's/ tags\///'`; do
  git branch $tag refs/remotes/$tag
done

Back to Contents

Test pushing and cloning locally

Before pushing to the real host, it would be wise to test things by pushing to a local Git repo and then cloning the result.

Make a local bare Git repo

In Git parlance, a ‘bare’ repo is one that has no working copy.

cd ~
mkdir test
cd test
git init --bare

You now have a bare repo at ~/test.

Push to the test Git repo

cd ~/staging
git remote add test `~/test`
git push --all test
git push --tags test

We put ~/test in backquotes to expand it to an absolute path. If you’re giving an absolute path or a URL, omit the backquotes.

Despite its name, the --all option doesn’t push tags, so we have to push them separately.

Clone from the test Git repo

cd ~
mkdir aclone
cd aclone
git clone ~/test

There should now be a clone with a working copy in ~/aclone/test. Examine it and satisfy yourself that it’s all good. Assuming so, we can now go ahead and push to the real host.

Back to Contents

Push your staging repo to the real host

This assumes an admin on the real host (GitHub, Unfuddle, etc) has set up an empty Git repo for you.

I shall take Unfuddle as an example, in which case the URL will be something like git@example.unfuddle.com:example/blah.git

cd ~/staging
git remote add unfuddle REAL_HOST_URL
git push --all unfuddle
git push --tags unfuddle

In the above example I decided to name the remote ‘unfuddle’ rather than the default of ‘origin’. You can use whatever name you like, of course. Simply change it in all three lines.

Back to Contents

Cleanup

Remove the test remote

cd ~/staging
git remote rm test

Now the staging repo has forgotten all about our ‘test’ remote.

Discard the test and clone repos

cd ~
rm -rf aclone
rm -rf test

Either keep or delete the staging repo

If you envisage any need to round-trip between Git and SVN, I strongly recommend keeping the staging repo as it will save you the very time-consuming initial git svn fetch.

If on the other hand you are certain that the SVN repo is end-of-life, you can delete the staging repo:

cd ~
rm -rf staging

Back to Contents

That’s all folks! I hope that this article has been useful. Comments, questions and errata can be sent to @RichardBuckle on Twitter or App.net.

Shameless plug: I am available for contract iOS and Mac development. If you are interested in hiring me, please see http://www.sailmaker.co.uk/.

Updates:

  • 2013-05-06: Combined the git push commands into one line, from feedback by @alblue.
  • 2014-01-14: Reverted the above because git push does not allow combining the --all and --tags flags.

Don’t auto-hyphenate my code

I’ve no problem with WordPress doing auto-hyphenation of my wonderful English prose, but auto-hyphenation of any important-code-snippets cited inline within that prose is a big no-no and very confusing for the reader, who is presumably copying that code to the command line and thus wanting to be completely sure that what they’ll copy is what they’ll get.

The fact that browsers now compensate for the inserted hyphens when copying makes it even more desirable not to present hyphenated code in the first place.

After publishing my first post in this blog, it greatly bothered me when I saw that WordPress was hyphenating my inline code. As in “I won’t rest until this is sorted” bothered me.

I was able to fix this by editing style.css on my server and appending:

/* JRB: don't hyphenate code */
code {
    -webkit-hyphens: none;
    -moz-hyphens: none;
    hyphens: none;
}

I hope this is useful.

Update 2013-05-08: You may want to do this in a child theme instead.

  • Pro: Robustness against updates to the parent theme.
  • Con: Increased loading times due to the CSS @import directive for importing the parent theme. Sites such as gtmetrix.com/ flag this rather highly but I’m not sure how much it matters in real life. There is always the nuclear option of cloning the entire theme, modifying it, and assuming the burden of merging any upstream changes into your clone.

Advanced Jenkins for iOS and Mac

Jenkins is your CI butler, and any reader of P.G. Wodehouse or viewer of British period drama knows that treating your host’s butler well is the key to an enjoyable visit. Likewise, giving Jenkins a bit of extra care and attention can greatly enhance your team’s experience with CI and make the entire team more productive.

When putting the following configs together I had to wade through a great many sources, so I thought it would be useful to put everything I’ve learned together in one document. I’m going to walk you through creating a fully instrumented Jenkins setup from scratch for iOS and Mac OS X projects. It will monitor errors, static analyser results, unit tests and code coverage for you, mail you when anything goes wrong, and present the results graphically, like this:

Sample Jenkins output

GitHub archive

An archive of a sample project (based on Apple sample code) together with sample Jenkins configs is available at https://github.com/richardbuckle/AdvancedJenkins.

Contents

Installing Jenkins itself

Installing ancillary tools

Installing Jenkins plug-ins

Final Jenkins configuration

Job configuration

Conclusion

Installing Jenkins itself

I’m going to assume that you’ll be running Jenkins as a full Mac OS X user in a windowed session, on a secured box, perhaps a Mac Mini, and that said user already has Xcode 4.6.1 installed together with all the provisioning and code-signing set up for any TestFlight, HockeyApp, or other distribution that you might need.

I am not going to cover running Jenkins as a daemon, as that causes so many woes with the Xcode toolchain, code-signing etc, that I don’t recommend it.

I’m also going to recommend installing Jenkins via Homebrew, to avoid some nonsense in Jenkins’ own installer whereby it puts itself in /Users/Shared/. You really don’t want that.

Install Homebrew

Homebrew is here.

For a machine-wide installation: ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

More fine-tuned installation instructions are here.

Personally I wanted Homebrew confined entirely to my account for a bit more security, so I chose to git clone it from https://github.com/mxcl/homebrew/ to ~/homebrew and manually added $HOME/homebrew/bin to my $PATH via my ~/.bash_profile. Your preferences may vary.

Install Jenkins

This one’s easy: brew install jenkins

You can start and stop Jenkins with:

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.jenkins.plist

You might want to make shell aliases for those.

If you have installed Jenkins via other means, then the plist may be elsewhere and/or have a different name. Look in:

 ~/Library/LaunchAgents     Per-user agents provided by the user.
 /Library/LaunchAgents      Per-user agents provided by the administrator.
 /Library/LaunchDaemons     System wide daemons provided by the administrator.

Test the server

You should now be able to connect to your Jenkins sever at http://localhost:8080 on the local machine. Try the start and stop commands given above and verify that they send the server up and down as expected.

Don’t bother configuring any projects just yet: we’ll come to that.

First run

The first time you run an Xcode job in Jenkins, you will probably need to be VNC’d into the server so that you can click “Always Allow” when the process asks for confirmation of keychain access. This can only be done via the GUI, alas, but once done you should never see it again.

Secure your Jenkins installation

This requires consideration of your specific setup and a full discussion is out of scope for this document.

The level of security that you’ll need will depend upon whether the server is accessible LAN-only, VPN-only, or public Internet. This is entirely your call.

I would go with LAN-only if possible, falling back to VPN into a strictly controlled subnet of your LAN if you need to give access to outsiders. Setting up a VPN is a bore, but it is the best option: both for access control and for preventing snooping and man-in-the-middle attacks when your clients inevitably use untrusted public WiFI etc.

NEVER run your Jenkins server on a laptop that roams on public WiFi: you’re wide open to man-in-the-middle attacks if you do that!

Googling Jenkins Security will give you a lot of good advice.

Put your Jenkins jobs dir under version control

All of your Jenkins job specifications are in its jobs subdirectory. I highly recommend you put it under version control immediately and also push to a separately hosted repo that is inaccessible to anyone who can VPN in, etc.

Not only will this give you the usual benefits of tracking and diffing config changes, and making changes with the confidence that you can revert if you mess up, but it will also give you an easy migration to a new server or an additional slave machine.

Tip: if you ever change anything in the jobs dir outside the web UI, go to Dashboard > Manage Jenkins and click “Reload Configuration from Disk” to force Jenkins to see your changes.

Back to Contents

Installing ancillary tools

Phew, that’s over. Next I shall walk you through adding the ancillary stuff that will make your Jenkins installation vastly more useful.

I’m going to put these in usr/local/bin. If you prefer to put them elsewhere then modify what follows accordingly.

OCUnit2JUnit

This is a very nifty Ruby script that converts OCUnit test results into the JUnit format, so that Jenkins plug-ins can parse and display them and add some very helpful drill-down reporting.

Clone https://github.com/ciryon/OCUnit2JUnit and install it in /usr/local/bin/:

cd ~
git clone https://github.com/ciryon/OCUnit2JUnit.git
sudo cp ./OCUnit2JUnit/bin/ocunit2junit /usr/local/bin
sudo chown root:wheel /usr/local/bin/ocunit2junit
sudo chmod 755 /usr/local/bin/ocunit2junit

Optionally, cleanup the Git repo:

cd ~
rm -rf ./OCUnit2JUnit/

To use OCUnit2JUnit, pipe the output of any xcodebuild command that emits OCUnit tests results to it, e.g.:

xcodebuild -scheme "Logic Tests" test ... | /usr/local/bin/ocunit2junit

Below I’ll describe adding the post-build action “Publish JUnit test result report” to publish the test results.

gcovr

This is a Python script that converts GCC and Clang/LLVM code coverage reports into the Cobertura format, so that Jenkins plug-ins can parse and display them, again adding some very helpful drill-down reporting. It now has a home page of its own and is hosted on GitHub.

Clone https://github.com/gcovr/gcovr and install it in /usr/local/bin/:

cd ~
git clone https://github.com/gcovr/gcovr.git
sudo cp ./gcovr/scripts/gcovr /usr/local/bin
sudo chown root:wheel /usr/local/bin/gcovr
sudo chmod 755 /usr/local/bin/gcovr

Optionally, cleanup the Git repo:

cd ~
rm -rf ./gcovr/

Clang static analyzer

Surprisingly, the Xcode command-line tools don’t allow first-class reporting of Clang static analyzer errors. Until Apple fixes this, we need to install scan-build directly.

scan-build’s page is worth a read.

Go to the Clang static analyzer’s home page and grab the latest tarball. At the time of writing, it is 2.7.2. Unpack it to get a folder, say, checker-272.

cd /path/containing/it
sudo mv checker-272 /usr/local/bin/
sudo chown -R root:wheel /usr/local/bin/checker-272
sudo chmod -R 755 /usr/local/bin/checker-272

Next, create a symbolic link for it. That way you won’t need to update all your Jenkins jobs when you download a new drop of scan-build, but can simply revise the symbolic link:

sudo ln -s /usr/local/bin/checker-272 /usr/local/bin/checker-current

Now we can run an independent Clang static analyzer and report results regardless of any attempts by team-mates to turn it off in project settings, etc.

ios-sim

Until Apple get their act together and enable unit testing in the iOS Simulator from the command line, this tool plugs the gap.

To get ios-sim from Homebrew, copy it to /usr/local/bin and ensure that it has ownership root:wheel and flags -rwxr-xr-x (octal 755), we do this:

brew install ios-sim 
sudo cp $(brew list ios-sim | grep bin/ios-sim) /usr/local/bin
sudo chown -R root:wheel /usr/local/bin/ios-sim
sudo chmod -R 755 /usr/local/bin/ios-sim

Back to Contents

Installing Jenkins plug-ins

These are what I use:

Essential plug-ins

I think everyone will need these, most of which come with the default installation:

Git-related plug-ins

Most of these come with the default installation. Might as well have them installed up front and have done with it. Be sure to have git client plugin >= 1.0.5 as 1.0.4 had severe issues (promptly fixed).

Concurrency plug-ins

Social plug-ins

Static analysis plug-ins

  • Static Analysis Utilities For reporting and graphing compiler warnings etc, with drill-down.
  • Clang Scan-Build Plugin For reporting and graphing static analyzer results, with drill-down. Its input side is a bit limited, so I tend to drop to shell script for that and just use it to show the output: see below.

TDD/Code coverage plug-ins

Deployment plug-ins

Not recommended

I wish I could recommend the XCode integration plug-in, but I really can’t. It has too many known issues, particularly that it misparses quoted parameters containing white space, an issue that has been open for over a year. I’ve also found that it has trouble parsing log output, often giving spurious fatal errors such as “FATAL: Log statements out of sync“.

If any reader has the time and knowledge to fix this plug-in, I’m sure the entire community would be very grateful. Meanwhile, I’ve dropped back to shell script using the techniques shown in WWDC 2012 vid 404. You don’t need to write very much shell script and it’s vastly more flexible anyhow.

Back to Contents

Final Jenkins configuration

Next we need to do a bit of system-wide configuration of some of the plug-ins that we’ve added. From the Jenkins home page, click “Manage Jenkins”, then “Configure System”.

Locks

Find the section called “Locks”:

Locks section defaults

Click “Add” and set the name to, say, “iOS-sim”.

Locks section with iOS-sim added

Configuring Clang static analyzer

Click the button “Clang Static Analyzer installations”.

Click “Add Clang Static Analyzer” and set the name to, say, Clang-current and the “Installation directory” to /usr/local/bin/checker-current. This is the symlink we created above to avoid having to revise this config when we update scan-build.

Configuring Clang static analyzer

Any other business

You will probably want to fill out the sections titled “Jenkins Location” and “E-mail Notification” here as well.

When you’re done, click the “Save” button at the very bottom of the page.

Back to Contents

Job configuration

At this point I recommend you grab my sample configs from my GitHub repo so you can follow along. Move the two configs from the jobs directory in the repo into Jenkins’ jobs directory, go to Dashboard > Manage Jenkins and click “Reload Configuration from Disk”. Both configs should build “out of the box” but you may first want to visit the config page for each one and enter your email address under “E-mail Notification” at the very bottom.

Don’t be concerned if Jenkins doesn’t show the graphs right away: all the graphing tools need at least two successful builds to show a graph. Just kick off another build and the graphs should show up.

I’ll start by going through the Logic Tests config, as it is the simpler of the two, then point out the differences in the Application tests config and a change we need to make to the XCode project to get it to use ios-sim when we want.

The Git and Build Triggers sections are unremarkable so I will not discuss them.

Shell script walk-through

To select the version of Xcode we want, we set DEVELOPER_DIR rather than using xcode-select. This has the twin advantages of only affecting the job in hand and not requiring root privileges:

# set the desired version of Xcode
set DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
export DEVELOPER_DIR

Next, for safety, we do a default clean of the workspace. Note that ${WORKSPACE} refers to the Jenkins workspace, not the Xcode one. We will still be cleaning each target or scheme as we build it, as I’ve found that in some cases the default clean doesn’t cover everything.

# do a default clean
cd "${WORKSPACE}"
xcodebuild clean

Next we run the static analyzer on the main code base (i.e. not the unit tests, but the code that they are testing), sending the output to ./clangScanBuildReports. We could have used the Clang Scan-Build plug-in’s build step for this, but shell script gives us more flexibility:

# run the static analyzer on the main code base
/usr/local/bin/checker-current/scan-build -k -v -v --keep-empty -o ./clangScanBuildReports xcodebuild -scheme Calculator-iOS -configuration Debug ONLY_ACTIVE_ARCH=NO clean build

Note how we used the symlink /usr/local/bin/checker-current that we created above so that we won’t need to revise this config when we update our version of scan-build.

We are going to tell Xcode to put its intermediate output in "${WORKSPACE}"/tmp/objs rather than its default path of ~/Library/.../DerivedData/... so that gcovr will be able to find it. First we delete any such pre-existing directory:

# delete our custom intermediates directory
rm -rf "${WORKSPACE}"/tmp/objs

To redirect Xcode’s intermediate output, we set OBJROOT. Note that this has to be done in the parameters to xcodebuild — setting and exporting an environment variable won’t work. As mentioned above, we pipe the output to ocunit2junit so that Jenkins can display unit test results:

# build the test target using our custom intermediates directory
# and pipe the output to ocunit2junit
xcodebuild -target Calculator-iOS_LogicTests -configuration Debug -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO TEST_AFTER_BUILD=YES OBJROOT="${WORKSPACE}"/tmp/objs clean build | /usr/local/bin/ocunit2junit

Finally we call gcovr to generate the coverage report, outputting to ./coverage.xml. You may wish to tune the --exclude= parameter (note that it is one big regex, not a Unix-style glob, and needs to be in single quotes):

# generate the coverage report
cd "${WORKSPACE}"
/usr/local/bin/gcovr --root="${WORKSPACE}" --exclude='(.*./Developer/SDKs/.*)|(.*Tests\.m)' -x > ./coverage.xml

Post-build Actions

Compiler warnings

Add the “Scan for compiler warnings” post-build step and configure it to use the “Clang (LLVM based)” parser like this:

Scan for compiler warnings

It shouldn’t need any further configuration.

Update 2013-04-27:

My original post said to use the “Apple LLVM Compiler (Clang)” parser. Updates to the Warnings parser here and here shipping in Warnings parser version 4.24 have silently renamed the “Apple LLVM Compiler (Clang)” parser to “Clang (LLVM based)”, silently causing configs using “Apple LLVM Compiler (Clang)” to default to, of all things, “Acu Cobol”.

I’ve filed this ticket and encourage you to add a sensible comment if you agree.

Meanwhile, if you created any configs using the Warnings parser <= 4.23, you’ll need to update then manually when you update to Warnings parser >= 4.24.

Scan-build results

Add the “Publish Clang Scan-Build Results” post-build step:

Publish Clang Scan-Build Results

Optionally, set a threshold past which the build will be marked as unstable.

Coverage report

Add the “Publish Cobertura Coverage Report” post-build step, giving it the report pattern coverage.xml. Optionally, configure the detail to your liking:

Publish Cobertura Coverage Report

Unit test report

Add the “Publish JUnit test result report” post-build step, giving it the XML specifier test-reports/*.xml:

Publish JUnit test result report

E-mail Notification

Add the E-mail Notification post-build step if it isn’t already there and configure it to your liking. I’d advise keeping “Send separate e-mails to individuals who broke the build” on!

Differences for the Application tests config

That’s it for the Logic tests config. The application tests config differs slightly in that we need to get it to use ios-sim and we need to use a concurrency lock to stop two jobs trying to use ios-sim at the same time.

Concurrency lock

In the “Build Environment” step, check “Locks” and select the “iOS-sim” lock:

iOS-sim lock

Shell script

The only substantive difference is that we add the parameter WANT_IOS_SIM=YES to the xcodebuild command:

xcodebuild -target iOS_Calc_ApplicationTests -configuration Debug -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO TEST_AFTER_BUILD=YES GCC_TREAT_WARNINGS_AS_ERRORS=YES WANT_IOS_SIM=YES OBJROOT="${WORKSPACE}"/tmp/objs build | /usr/local/bin/ocunit2junit

This is an environment variable of our own devising which we have customised the Xcode project to notice as described next.

Other than that this Jenkins config differs only in the scheme/target that we tell it to build.

Fixing the Xcode project for app testing

While Xcode 4.6.1’s Simulator does in fact support app testing from the command line, Apple have neglected to revise Xcode’s internal scripts to reflect that. This is why we installed ios-sim above.

This code assumes that ios-sim is in /usr/local/bin. If you put it elsewhere, adjust the script accordingly (as a matter of security, I always specify absolute paths to my binary files, and don’t trust $PATH).

Open the project in Xcode, select the project then the target iOS_Calc_ApplicationTests, select the “Build Phases” tab, and expand the “Run Script” phase. You will see this:

# Run the unit tests in this test bundle.
if [ -z "${WANT_IOS_SIM}" ]
then
#   Running under Xcode
    "${DEVELOPER_TOOLS_DIR}/RunUnitTests"
else
#   Running under xcodebuild, so use ios-sim installed from Homebrew into /usr/local/bin/ios-sim
    killall -m -KILL "iPhone Simulator" || true
    our_test_bundle_path=$BUILT_PRODUCTS_DIR/$PRODUCT_NAME.$WRAPPER_EXTENSION
    our_env=("--setenv" "DYLD_INSERT_LIBRARIES=/../../Library/PrivateFrameworks/IDEBundleInjection.framework/IDEBundleInjection")
    our_env=("${our_env[@]}" "--setenv" "XCInjectBundle=${our_test_bundle_path}")
    our_env=("${our_env[@]}" "--setenv" "XCInjectBundleInto=${TEST_HOST}")
    our_app_location="$(dirname "${TEST_HOST}")"
    /usr/local/bin/ios-sim launch "${our_app_location}" "${our_env[@]}" --args -SenTest All "${our_test_bundle_path}" --exit
    killall -m -KILL "iPhone Simulator" || true
    exit 0
fi

What this means is that if the environment variable WANT_IOS_SIM is not set, we call ${DEVELOPER_TOOLS_DIR}/RunUnitTests as per usual. This means that you still get the usual behaviour when working in Xcode and in particular you will be able to run application tests on iOS devices in the usual way.

If on the other hand WANT_IOS_SIM is set, we kill any existing iPhone Simulator process, set up variables for our test bundle path etc, call ios-sim manually, and then kill the simulator when we’re done. Slimy but effective.

You may also come across advice to amend .../Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests along similar lines. While this works, I advise against it as it will be fragile against future Xcode updates. With the above fix, simply don’t set WANT_IOS_SIM and you will always get Apple’s intended behaviour.

Back to Contents

Conclusion

Wow. That has been quite a journey.

Now that you’ve got a running config, play around. Break a unit test, or introduce a static analyzer warning and see Jenkins report it. Drill down into the unit test and coverage reports to get a sense of what’s available.

I hope that this article has been useful and will help you get a lot more out of Jenkins! Comments, questions and errata can be sent to @RichardBuckle on Twitter or App.net.

Shameless plug: I am available for contract iOS and Mac development. If you are interested in hiring me, please see http://www.sailmaker.co.uk/.

Updates:

2014-09-29: Updated to reflect that gcovr is now hosted on GitHub.

2013-04-27: Added a bit more about why to install each component, from feedback by @danielctull.

2013-04-27: Added text about the change in warnings parser 4.24, with a link to my ticket.

Back to Contents