Regression hunting: From Slicer CDash error to ITK master

Community, as well as small-team, software development faces many challenges.  Kitware is constantly developing and looking for new ways to ensure that quality software processes are readily available and can be seemlessly integrated into any project, large or small.  Our efforts have lead to the development of tools such as CMake and CDash, as well as the identification of several outstanding tools and processes that have been developed by others.  Nevertheless, software development mistakes can and will happen, and this blogs explains one method for addressing them.

If, at some point in the lifetime of your project, you realize (perhaps thanks to a CDash regression test) that an error has been introduced into your project, it can sometimes be difficult to pintpoint exactly which commit caused it.  At a high level, the approach to resolving such errors is to:

1. Identify a version of the software that does not exhibit the error
2. Thoroughly analyze the subsequent software changes, until you find the faulty commit

This approach can be accomplished with standard tools, if you have very few commits to check; however, in highly active development environments and particularly for community development efforts, a massive number of commits might need to be reviewed. Luckily, git provides us with the ‘bisect’ module, allowing us to find, by binary search, the change that introduced a bug given a ‘good’ commit object name prior to the error and a later ‘bad’ commit object name that occurred after the introduction of the error.

Source: https://www.kernel.org/pub/software/scm/git/docs/git-bisect.html

Example

On July 10, 2013, within Slicer, the hash tag associated with the ITKv4 external project was updated (r22163) to point to the tip of the release branch (41abd503). The next day, the MacOSX dashboard was complaining.

Exactly 289 changes were introduced by updating the ITK external project,. Therefore, checking each change or looking at the history wasn’t an option. Indeed, we weren’t sure what to look for to try to identify the cause of the regression. We then decided to try using git bisect.

Git bisect provided us with a manual and an automatic mode. In this case, we used the automatic mode, but we will briefly describe the “manual” mode here.

Manual mode

$ git bisect start
$ git bisect bad 41abd5
$ git bisect good 1866ef4 # Last version tested that was good
Bisecting: 143 revisions left to test after this (roughly 7 steps)
[21cc50a7] Merge topic 'ImageScanlineIteartorForImageCopy' 

First, build and test the above. Then, based on the result, you can use either use

$ git bisect good

or

$ git bisect bad

until you find the faulty commit.

If, for some reason, the current commit can’t be tested, you can use

$ git bisect skip

Automatic mode

While very powerful, the manual mode can still be tedious and repetitive. To use the automatic mode, you first need to create a script that will return 0, 1, or 255 based on the outcome of a set of commands, allowing you to determine whether or not the current checkout is problematic .

In this case, we used the following script and ran the bisection using

$ git bisect start 41abd5 1866ef
$ git bisect run /Users/kitware/Dashboards/Nightly/find_itkv4_regression.sh

Et voila, it returns that the commit introducing the regression was 5bfdff361.

UPDATE: 2015/09/04 – Code for find_itkv4_regression.sh can be found here.

After some investigation to try to understand the source of the problem, we created an issue on the ITK tracker:
https://itk.icts.uiowa.edu/jira/browse/ITK-3172

We then decided to propose a patch using gerrit:
http://review.source.kitware.com/#/c/11942/

Finally, in the timespan of 3 hours, after 8 patchsets, 24 comments, and few emails, the reviewed and polished ITK commit f49a7db7 was integrated into the release and the master branch of ITK. Then, the Slicer external project was updated accordingly (r22171), and the error was resolved.

Kitware is proud to be maintainers of the ITK and Slicer open-source efforts.  We are dedicated to their success, and our tools and practices, as well as numerous other outstanding software development and debugging tools and practices being developed around the world, are invaluable to our open-source and private consulting efforts. If we can help you with developing, maintaining, or debugging your code; we welcome the opportunity to provide consulting services to your project.  For more information, email kitware@kitware.com.