Dynamic Google Test Discovery in CMake 3.10

If you’ve written unit tests in C++, you may have heard of Google Test. Google Test is a C++ unit testing framework that makes it easy to write and execute unit tests. Compared to writing unit tests without a framework, Google Test allows developers to write tests much faster without sacrificing quality, often resulting in tests with more useful diagnostics on failure compared to tests written without a framework, or with a lesser-quality framework.

Myths and Overview

A misconception I have encountered several times is that Google Test “replaces” CTest. This is completely untrue. CTest is a tool for managing and executing a complete suite of tests (i.e. all tests within a project), and for submitting build and test results to CDash. Google Test is a tool for writing individual C++ tests. Although Google Test does provide some overlap in that both it and CTest have notions of selecting tests or test cases to execute, the two tools are complementary. Most projects will have more than one test executable, and may have tests that are not C++ executables, and will therefore have need of CTest. Even in the case of a single text executable, CTest is still useful for submitting results to CDash.

The main purpose of CMake’s Google Test support is to help Google Test and CTest “play together”. The simplest way to register a Google Test test with CTest is to simply use add_test to add a test that runs the test executable with no arguments. This will create a single CTest test that runs all Google Test test cases in the executable. While this works, it is not very granular; if the test fails, there is very little information about the location of the failure short of inspecting the test output.

Thus, gtest_add_tests was created. This command, provided by the GoogleTest module, inspects the test sources to extract individual test cases, in order to create a separate CTest test for each Google Test test case. By doing so, the user can see from the CTest summary (or on the CDash page which lists failing tests) which specific Google Test test case or cases failed. Unfortunately, this approach comes with several drawbacks.

Problems

The first, perhaps most obvious, and perhaps worst issue is that test inspection happens during the CMake configure stage. This means that any time a test source file is change, CMake must be re-run. For some projects, this may consume a significant amount of time.

Most of the problems, however, stem from the manner in which tests are found. Since CMake is not a C++ compiler, gtest_add_tests finds tests using regular expressions. This approach is fairly simple, and is subject to a number of shortcomings by virtue of its inability to fully understand C++ code:

• It does not handle conditional compilation. If a test is disabled by preprocessor logic or C-style block comments, either because the author did not use the recommended method of disabling a test case by prepending DISABLED_ to its name, gtest_add_tests will still detect it and erroneously attempt to register it with CTest, resulting in a CTest test that cannot be executed.
• It has limited understanding of type- and value-parameterized tests. Because the mechanism used to instantiate these involves complex C++ logic, gtest_add_tests has no realistic hope of determining the individual types or values on which a test or test case is instantiated. As a result, gtest_add_tests is only able to add the “abstract” test, resulting in a CTest test that executes the test cases over all types or values. While this is an improvement over running all tests as a single CTest test, it falls short of the granularity that is possible.
• It does not understand esoteric ways of declaring a test. For example, if multiple test cases are generated via preprocessor macro expansion, such tests will not be found by gtest_add_tests. (This affected me personally and is largely why I created the dynamic discovery mechanism. More in the next section…)

History

Several years ago, I was working on a library that used Google Test. I wanted to implement some type-parameterized tests, but was unhappy with the way Google Test named the test cases using its built-in support for such tests. Instead, I ended up using a preprocessor macro to synthesize multiple test cases for the set of types to be tested. This resulted in the test cases being declared in a somewhat esoteric manner that gtest_add_tests was unable to detect.

The built-in argument handling for Google Test, however, includes a mode that lists the available tests. Thus, I wrote my own mechanism that would detect and register tests by actually running the test with this flag.

Unfortunately, the library never saw the light of day. I moved on to other projects, and did not have an opportunity to interact with Google Test again for quite some time.

Then, recently, I started working on KWIVER. At the time, KWIVER had its own unit test “framework” (if the extremely minimal set of utility macros even warrants the term), which was inherited from — and is still used by — sprokit. Initially, I attempted to use this when writing unit tests for new features I was adding, but I quickly grew to feel constrained by the limited feature set. Having used Google Test previously, I knew there was a better way.

Thus, the seed was planted to dust off the dynamic test discovery mechanism I’d written years ago. This time, I knew I didn’t want to just dump a copy into KWIVER’s repository. I wanted anyone to be able to use my discovery mechanism, which meant cleaning it up and submitting it to CMake upstream. In the process, I realized that the mechanism would be greatly simplified by being able to list more than one additional CTest include file. This led to CMake!1031, which added CTEST_INCLUDE_FILES, and CMake!1056.

Operation, Features and Usage

CMake!1056 introduces a new way of registering Google Test tests: gtest_discover_tests. Unlike gtest_add_tests, this new mechanism works by setting up a post-link step that runs the test executable after it has been built in order to discover tests. The executable’s output is parsed by a utility script (also bundled with CMake) in order to generate at build time a supplemental CTest script, which the command registers for inclusion using CTEST_INCLUDE_FILES. Although this slightly complicates the process of adding additional test properties to individual test cases, it solves the problems mentioned previously. Since discovery occurs at build time, there is no need to re-run CMake because a test source file has changed. Test discovery is also 100% accurate, and type- and value-parameterized tests are split into individual CTest tests.

Use of gtest_discover_tests is similar to gtest_add_tests, and in many cases the two commands share options. In the simplest case, the only argument that must be given to gtest_discover_tests is the target name of the test executable. Other supported arguments include passing additional arguments when executing the test, specifying properties to be set on all CTest tests registered via a gtest_discover_tests invocation, adding a prefix and/or suffix to the CTest test name, and others. (See the GoogleTest module’s documentation for details.)

If it is necessary to set properties on only certain CTest tests belonging to a test executable, this is best accomplished by writing a separate CTest script to set these properties, and adding the script to CTEST_INCLUDE_FILES (after invoking gtest_discover_tests, so that the custom script is evaluated after the script which registers the discovered tests). The list of registered tests is made available in a variable (see TEST_LIST), which can assist in determining the names of the CTest tests to be modified.

Examples

To help illustrate the difference between the old, static test case discovery, and the new, dynamic test case discovery, consider the following CMake snippet:

include(GoogleTest)

add_executable(tests tests.cpp)
target_link_libraries(tests GTest::GTest)

gtest_add_tests(TARGET tests TEST_PREFIX old:)
gtest_discover_tests(tests TEST_PREFIX new:)
add_test(NAME monolithic COMMAND tests)

In both cases, registering the tests is quite simple; the appropriate command is used, and is given the target name of the test executable. Since this example is demonstrating both styles of test registration, a prefix has been added so that the test names will not collide. For the sake of comparison, the test executable has also been registered using add_test.

The test code (not shown) used for this example exercises most of the corners of Google Test, including both type- and value-parameterized tests, explicitly disabled tests, and a test case that is guarded by a preprocessor condition.

Now, consider the output from CTest:

Test project /path/to/build
      Start  1: new:DisabledSuite.test
 1/20 Test  #1: new:DisabledSuite.test ...............***Not Run (Disabled)   0.00 sec
      Start  2: new:DisabledTest.test
 2/20 Test  #2: new:DisabledTest.test ................***Not Run (Disabled)   0.00 sec
      Start  3: new:SimpleTest.test
 3/20 Test  #3: new:SimpleTest.test ..................   Passed    0.00 sec
      Start  4: new:TypedTest/int.test1
 4/20 Test  #4: new:TypedTest/int.test1 ..............   Passed    0.00 sec
      Start  5: new:TypedTest/int.test2
 5/20 Test  #5: new:TypedTest/int.test2 ..............   Passed    0.00 sec
      Start  6: new:TypedTest/long.test1
 6/20 Test  #6: new:TypedTest/long.test1 .............   Passed    0.00 sec
      Start  7: new:TypedTest/long.test2
 7/20 Test  #7: new:TypedTest/long.test2 .............   Passed    0.00 sec
      Start  8: new:values/IntValueTest.test/1
 8/20 Test  #8: new:values/IntValueTest.test/1 .......   Passed    0.00 sec
      Start  9: new:values/IntValueTest.test/2
 9/20 Test  #9: new:values/IntValueTest.test/2 .......   Passed    0.00 sec
      Start 10: new:values/StrValueTest.test/"dog"
10/20 Test #10: new:values/StrValueTest.test/"dog" ...   Passed    0.00 sec
      Start 11: new:values/StrValueTest.test/"cat"
11/20 Test #11: new:values/StrValueTest.test/"cat" ...   Passed    0.00 sec
      Start 12: old:DisabledSuite.test
12/20 Test #12: old:DisabledSuite.test ...............***Not Run (Disabled)   0.00 sec
      Start 13: old:DisabledTest.test
13/20 Test #13: old:DisabledTest.test ................***Not Run (Disabled)   0.00 sec
      Start 14: old:ConditionalTest.test
14/20 Test #14: old:ConditionalTest.test .............   Passed    0.00 sec
      Start 15: old:SimpleTest.test
15/20 Test #15: old:SimpleTest.test ..................   Passed    0.00 sec
      Start 16: old:*/IntValueTest.test/*
16/20 Test #16: old:*/IntValueTest.test/* ............   Passed    0.00 sec
      Start 17: old:*/StrValueTest.test/*
17/20 Test #17: old:*/StrValueTest.test/* ............   Passed    0.00 sec
      Start 18: old:TypedTest/*.test1
18/20 Test #18: old:TypedTest/*.test1 ................   Passed    0.00 sec
      Start 19: old:TypedTest/*.test2
19/20 Test #19: old:TypedTest/*.test2 ................   Passed    0.00 sec
      Start 20: monolithic
20/20 Test #20: monolithic ...........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 16

The “monolithic” test can be seen at the bottom. Although a pass/fail is generated, no additional information is available without running the test verbosely. Compared to either of the other methods, there is very little information available “at a glance” if some part of the test suite fails. Both other methods thus have an immediate advantage; by arranging for each registered CTest test to execute only a subset of test cases within the test executable, CTest (and CDash) is directly communicating more information about the area of failure. This can also be of particular use if a test case is crashing, since a crash in one test case will not prevent other cases from being executed.

For simple tests, there is little difference between gtest_add_tests and gtest_discover_tests by the time CTest executes. Moreover, both handled the case of a test that has been disabled by prefixing either the test suite name or test case name with DISABLED_.

However, the less trivial cases start to show differences. Where gtest_add_tests created a single CTest test (with a wildcard in its name) per test case of a type-parameterized test, gtest_discover_tests registered a separate CTest test for each test case and type parameter combination. Similarly, for value-parameterized tests, the CTest tests that were registered by gtest_discover_tests include both the actual value, and the name of the value set (the first argument to INSTANTIATE_TEST_CASE_P) to which the value belongs. (Unnamed value sets are also supported, in which case the test name simply starts with the test suite name, as usual.)

Another case worth noting is old:ConditionalTest.test. In the source code, this test case was guarded by a preprocessor condition (which is false), and the body of the test case contained an explicit failure. This test case was, correctly, not registered by gtest_discover_tests, whereas gtest_add_tests registered a CTest test that calls the test executable with a –gtest_filter that does not match any test cases (and thus does nothing). This presents a possible cause of confusion, as the test output might lead a viewer to believe that the test was executed and passed, when in fact it was not even compiled.

Conclusion

Dynamic test discovery offers a new and exciting mechanism for integrating two great tools: CMake and Google Test. We hope we have shown how this feature is useful, and how it can be used in your own projects. Keep coding, and keep writing tests!