3D Slicer Improves Testing for Pull Requests Using Docker and CircleCI

3D Slicer is an industry-leading, free, open-source software platform for medical image informatics, image processing, and three-dimensional (3D)/four-dimensional (4D) visualization. In the last five years, over 230,000 downloads of the software have occurred. Since its early age, a rigorous software process has been central to the development of 3D Slicer. Currently, more than 650 tests validate that 3D Slicer and its different components work as expected.

The Kitware blog post “Why should we spend time writing tests?” offers the following quote from Luis Ibáñez, Ph.D. “The safe assumption is: if it is not tested, it is broken” [1]. This quote summarizes the importance of software testing. Software testing serves as a key component of the scientific ecosystem, as it enables reproducible research. The 3D Slicer testing infrastructure also enables adherence to “Rules for tools” [2]. Dr. Ron Kikinis, who serves as the Robert Greenes distinguished director of biomedical informatics in the department of radiology at  Brigham and Women’s Hospital, first defined these informal rules, which developers should keep in mind when they work on interactive tools for translational clinical research.

Concepts, Tools, and Services

Before this article covers the most recent improvements to the 3D Slicer software process, it describes the concepts, tools, and services that support 3D Slicer development.

Continuous Integration
As the Wikipedia article for continuous integration outlines, “In software engineering, continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day” [3].

CTest
CTest is a language-agnostic testing harness that comes with CMake [4]. After CMake creates tests as part of a project build, CTest executes them [4]. Within 3D Slicer, CTest drives the execution of unit tests, which check that individual classes work as expected. CTest also drives the execution of integration tests, called “self-tests.” Users can execute self-tests after they install 3D Slicer and enable developer mode.

A given self-test downloads data and checks that the corresponding biomedical workflow, which generally involves multiple modules, functions as expected. Most of the 3D Slicer tutorials have corresponding self-tests that evaluate whether or not changes to 3D Slicer break the tutorials. Additional information on self-tests is available at https://www.slicer.org/wiki/Documentation/Nightly/Developers/Tutorials/SelfTestModule. More details on the CMake testing cycle is available at https://cmake.org.

CDash
According to its website, “CDash is an open-source, web-based software testing server” [5]. The server “aggregates, analyzes, and displays the results of software testing processes,” which CTest drives [5]. 3D Slicer developers can easily verify the state of the current 3D Slicer software and the state of associated extensions on http://slicer.cdash.org/index.php?project=Slicer4.

Pull Requests
Pull requests allow developers to propose and discuss changes before they become integrated into the main line (also known as the “trunk” of the code base or the “master” branch). Practically speaking, anyone can fork the 3D Slicer code base on GitHub and create pull requests. To date, 3D Slicer has received more than 615 pull requests. More information on pull requests is available at https://github.com/Slicer/Slicer.

On a historical note, the introduction of Git and the use of GitHub to manage the 3D Slicer code base occurred in 2010. The year 2011 marked the first pull request, which occurred a few weeks after the National Alliance for Medical Image Computing (NA-MIC) Summer Project Week hackathon. Until recently, pull requests only served as platforms for discussing and reviewing changes.

The challenge

Build servers that Kitware maintains and build scripts run every night on Linux, Mac, and Windows operating systems made it possible to consistently verify that changes in the main line did not break the build and/or cause tests to fail. While the use of nightly builds validated changes, simply relying on them delayed the development cycle. Fixes, for example, could not undergo validation until the next day. To speed up the development cycle, it was important to verify fixes as soon as they became integrated into the main line.

An initial attempt to speed up the development cycle consisted of systematically testing all the changes in the main line. Every few minutes, the build servers described above downloaded the most recent changes. These changes triggered project builds and tests, the results of which appeared on the dashboard.

This approach turned out to be ineffective for the reasons that follow:
Fixes for changes that occurred at the end of the day still had to wait until the next day for validation. Sometimes, however, developers moved on to other tasks before they attended to the regressions.

It took time to identify the source of a regression. Since the build servers did not test each change or set of changes independently, they required manual intervention to understand the problem and notify the associated developers.
The creation of another independent 3D Slicer build tree had to occur before the faster increment build could start, as the build tree associated with nightly testing had to remain untouched to allow CMake tools to build, test, and package the 3D Slicer extensions.

Improvement 1

Every night (and up until the next morning), the build servers build and test the most recent changes to 3D Slicer, build updated 3D Slicer extensions against the most recent version of 3D Slicer, and build updated extensions against the last 3D Slicer release. Accordingly, the 3D Slicer development team looked to improve the developer experience.

The Use of CircleCI and Docker to Continuously Build 3D Slicer
The team leveraged the following cloud services that come free to open-source projects.

Docker Hub
Docker Hub is a cloud-based registry that links code repositories and builds Docker images. A Docker image represents a lightweight container that can run any version of Linux. Said differently, a Docker image enables software re-execution to occur in the same context to assure that results remain consistent. More information on Docker is available at https://www.docker.com.

Dockerfile
A Dockerfile is a text document that contains all the commands that can assemble an image. Many commands can describe this document such as RUN, COPY, VOLUME, ENV, ARG, and even LABEL. These commands offer the abilities to execute processes, mount directories, set some environment variables, and describe an image each time it builds with docker build. Once an image builds, it can undergo instantiation to become a container. Information about best practices on how to write a Dockerfile is available at https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices.

CircleCI
CircleCI is a hosted continuous integration and continuous delivery service. Compared to other services such as AppVeyor or Travis, CircleCI offers built-in efficient support for Docker. CircleCI can link with GitHub and launch a script for every pull request. It refers to the circle.yml file, which has to appear in the source root folder of a project. More information on CircleCI is available at https://circleci.com.

Required Images
Systematically building 3D Slicer pull requests with CircleCI requires a 3D Slicer Docker image (or a set of images) that can complete a 3D Slicer build in under two hours. With the help of Matt McCormick, Ph.D., the 3D Slicer development team came up with the following set of images.

thewtex/centos-build
This image provides a build environment that stems from an old Linux distribution (Centos 5). The build environment depends on an older version of GlibC (2.5), and it supplies a recent toolchain (Red Hat devtoolset-2), which provides gcc-4.1.2-55. Since the build environment depends on an older version of GlibC, it enables the creation of binaries that are compatible with most of the current Linux distributions. The ABI Laboratory website illustrates the application binary interface (ABI) compatibility of the different versions of GlibC [6].

In addition to a build environment, thewtex/centos-build includes supporting tools like CMake, Git, and Apache Subversion (SVN).

slicer/slicer-base
This serves as the base image for slicer/slicer-dependencies.

slicer/slicer-dependencies
This image contains all the dependencies necessary to build 3D Slicer itself. These dependencies include the Insight Segmentation and Registration Toolkit (ITK), the Visualization Toolkit (VTK), the Common Toolkit (CTK), the Digital Imaging and Communications in Medicine Toolkit (DCMTK), and Qt.

slicer/slicer-build
This image contains both 3D Slicer and the dependencies that test a 3D Slicer extension.  An example, SlicerITKUltrasound, is available at https://github.com/KitwareMedical/SlicerITKUltrasound.

The Build of a Pull Request

The build of a pull request begins when the proposed pull request triggers CircleCI to read the configuration file (circle.yml) from the 3D Slicer code base. The file instructs CircleCI to only build the core of 3D Slicer, not its dependencies. If CircleCI detects an update to the 3D Slicer dependencies, it aborts the build, which indicates the need to manually generate a new slicer/slicer-dependencies image and publish it on DockerHub.

Each proposed pull request automatically merges with the master branch in a temporary repository. The Docker image completes the associated build in under an hour and a half. The below schema illustrates the overall improved process.

Improvement 2

To follow up on Improvement 1 and to consolidate continuous integration, the 3D Slicer development team added a testing process that follows a pull request. The tests verify that everything builds correctly for each pull request. As part of the process, CDash stores metadata about the build time. It also stores test information from the commit that triggered the testing process. More information on the CircleCI features that link with GitHub are available at https://circleci.com/docs/environment-variables.

The Use of CircleCI and Docker to Continuously Run 3D Slicer Tests

In order to make the testing process work, the 3D Slicer development team built a new Docker image, thewtex/opengl:centos, which it based on Centos 5. The Docker image provides the X11 window display that many tests require. The following section takes a closer look at this new Docker image.

OpenGL Docker Image

As previously mentioned, the team based the OpenGL Docker image on thewtex/opengl:centos, which draws inspiration from thewtex/opengl:debian. When a container that stems from thewtex/opengl:centos launches, it runs a single process [7].  The container runs supervisord as the Dockerfile command [8] to collect failed processes (process identifier reaping) and simultaneously launch multiple commands such as graphical_app [9], vnc [10], and Xdummy. These commands display windows from tests. The orchestrated X window system provides a graphical context to run the tests. Virtual Network Computing (VNC) provides an interface to the system for debugging.

The below schema summarizes the testing process.

The testing process continues with Docker.The first step to create a Docker image is to set up a Dockerfile that begins with FROM thewtex/opengl:centos (or :debian). After the container generates, the Dockerfile launches supervisord. Once supervisord launches, it deals with graphical_app, which calls the application environment variable. The Dockerfile must specify this variable using ENV APP “what_you_want_to_launch.“

Now, each time the image runs, the run.sh script from the thewtex/opengl repository configures and creates the container. (The script sets the VNC localhost, removes the container when the application is done, and displays prompts from the container.) For further information, please take a look at the README file https://github.com/thewtex/docker-opengl/blob/centos/README.rst.

Future Work

The 3D Slicer development team seeks to improve the integration of 3D Slicer with VNC and to increase testing speed by leveraging additional CircleCI containers. Please visit the Kitware blog at https://blog.kitware.com to remain updated.

For those interested in integrating 3D Slicer or its testing framework with products and processes, Kitware offers consulting and support services. To get started, please contact kitware@kitware.com.

Docker Tip

The team noticed that Docker can burn free space disk storage. During local tests on a machine, Docker saves all of the exited containers, mounted volumes, and images that it builds. To save space, the team recommends deleting useless image containers and volumes by running the following:

# remove exited containers:
docker ps --filter status=dead --filter status=exited -aq | xargs -r docker rm -v    
# remove unused images:
docker images --no-trunc | grep '<none>' | awk '{ print $3 }' | xargs -r docker rmi
# remove unused volumes:
docker volume ls -qf dangling=true | xargs -r docker volume rm

Note that the presence of the VOLUME command in a Dockerfile creates volumes.

References

Legarreta, John Haitz, and Matt McCormick. “Why should we spend time writing tests?” Kitware blog, November 21, 2016. https://blog.kitware.com/why-should-we-spend-time-writing-tests.

[1] “Documentation-Rons-Rules-For-Tools.” https://www.slicer.org/wiki/Documentation-Rons-Rules-For-Tools.

[2] Wikipedia. “Continuous integration.” https://en.wikipedia.org/wiki/Continuous_integration.

[3] Hoffman, Bill, Libby Rose, Michael Grauer, and Patrick Reynolds. “Enhancing Software Quality with CI in the Cloud.” Kitware Blog, November 22, 2016. https://blog.kitware.com/enhancing-software-quality-with-ci-in-the-cloud.

[4] Kitware, Inc. “CDash.” http://www.cdash.org.

[5] Ponomarenko, Andrey. “ABI Tracker (glibc).” Last modified December 19, 2016. https://abi-laboratory.pro/tracker/timeline/glibc.

[6] Docker Inc. “Use Supervisor with Docker.” https://docs.docker.com/engine/admin/using_supervisord.

[7] Agendaless Consulting and Contributors. “Supervisor: A Process Control System.” Last modified December 20, 2016. http://supervisord.org.

[8] GitHub, Inc. “thewtex/docker-opengl.” Last modified December 14, 2016. https://github.com/thewtex/docker-opengl/blob/centos/usr/bin/graphical-app-launcher.py.

[9] Wikipedia. “Virtual Network Computing.” Last modified October 17, 2016. https://en.wikipedia.org/wiki/Virtual_Network_Computing.

Mayeul Chassagnard is a research and development intern at Kitware. He studies computer science, mathematics, electronics, and signal processing at École Supérieure de Chimie Physique Électronique de Lyon.

Jean-Christophe Fillion-Robin is a technical expert and the lead developer of 3D Slicer at Kitware. He is also the lead developer of 3D Slicer for the NA-MIC community. As such, he developed the infrastructure for 3D Slicer extensions.