This post is geared towards VTK, ParaView, and CMB developers who are using Gitlab to contribute changes to these open source projects. The automated testing infrastructure used by these projects has several parts:

• Buildbot  is used to monitor the git repositories and schedule builds and tests when merge requests are created or the integration branches (master/release) are changed.
• CMake/CTest deals with the actual building and testing, and then posting the results on dashboards hosted by CDash.

Whenever you make a merge request for any of the buildbot supported projects listed, you can request test builds on your branch. To do that, a developer puts a @buildbot test  message as a comment on the merge request. Our buildbot instance hosted at buildbot.kitware.com monitors merge requests for such comments. As as soon as an appropriate comment posted by a developer is processed by Buildbot, it will schedule several builds on various platforms, with multiple configurations to ensure that the changes in the merge request do not introduce any regressions.

When Buildbot schedules these tests, it will post a comment back on the merge request with qualified links to CDash and Buildbot web pages that show the build and test results for the merge request, as follows.

Let's take a closer look at the Buildbot status page.

Understanding the Buildbot Status Page

Clicking on the Buildbot link will take you to the Buildbot web status page for the specfic merge request, as follows:

This is called the Grid View. The rows are builders and columns are source revisions.

Simply put, a builder is a dashboard run with a specific configuration. The name of the builder enumerates all the build options and enabled features e.g. vtk-bigmac-osx-shared-debug+clang+opengl2+python means this is a VTK build on a machine named bigmac which is an OS X machine, using shared libraries, and in debug mode using a clang compiler, with OpenGL2 rendering and Python wrapping enabled.

There are 8 builders for this project. A merge request is considered fully tested when all rows have a valid status next to it.

Sometimes there may be multiple columns, as shown below.

This happens when you amend your merge request, either by adding new commits or modifying existing commits. When a merge request is amended, buildbot will rescehule a new round of tests on the updated merge request. For any builders that haven't tested the previous state of the merge request yet, buildbot will simply skip testing the old revision(s) and schedule builds only on the newest revision. When this happens, it shows up empty cells, as seen in the image above. The column header mentions the revision that was tested by that column e.g. b063141a4f3b… in cell_data_overhaul implies revision b063141a4f3b in branch cell_data_overhaul.

To see why a builder passed (or failed), click on the link in the appropriate cell. That will take you to the builder status page.

It's a long page that provides details for the specific build. Let's look at important parts of this page.

The title mentions the name of the builder and the buildnumber. The buildnumber is a unique number for the specific builder indentifying the build. No two builds on the same builder will have the same buildnumber. 

The SourceStamp provides information about the source revision being tested. It includes the Repository, Branch, and Revision number for the source code tested.

The Steps and Logfiles item lists individual steps in this build.

Most of these steps can be ignored by users, the only ones of concern are the git step and the build-n-test step. The git step checks is the step that checks out the source for the merge request. The build-n-test step, builds and runs tests on the merge-request. When available, the stdio link takes you to the raw output generated by that step. This can be useful in determining why the step failed. The build-n-test step also has a cdash link. This link will take you to the results of this run on CDash. You can look at the details on the CDash page including build errors, test failures, image differences, etc. When there are failed tests, a link that directly takes you to the failed tests is also added to this build-n-test step.

Navigating the Buildbot Page

Besides using the link posted on the merge request, you can directly visit buildbot.kitware.com to view the Buldbot status page. The header provides links to standard pages for various projects.

Clicking on the name of the project e.g. VTK, ParaView will take you to a Grid View showing results for master branch for these projects. This can be used to check the status of various builds for the master branch. Similar to the Grid View for merge requests, this view has rows for each of the builders and columns for revisions tested on the master branch.

This page limits itself to show the master branch. To see the status for all branches, including merge requests, click on the expected link for the specific project. This page will show all branches tested/being tested with columns ordered in chronological order.

The experimental link points to the experimental builders that are still being setup, meant for developers adding new test machines, or test configurations. These will often be red i.e. failing, hence are generally not considered in determining if the merge request is ready for merge.

The superbuild link, when present, will take you to the superbuilds for the project i.e. the builders that build the project binaries.