Cleaning up ParaView’s CDash

Ever seen this on ParaView’s CDash?

A ParaView developer has undoubtedly seen this before and surely wondered,

“What’s the deal? Why are there two rows for the same build, where first time 665 tests pass, 2 fail while second time 2 pass and none fail?”

This is a side-effect of ParaView’s testing infrastructure making a second attempt to run the failed tests to see if they pass on a second attempt. To speed up dashboards times, ParaView dashboards run multiple tests at the same time and that can often introduce issues with UI tests. This second serial attempt avoids such false positives by running one test at a time.

“That’s confusing … really confusing!”

Fret no more! Recent changes that have gone into our buildbot-based CI infrastructure merges the two rows before uploading the information to CDash, thus you’ll only see the final result i.e. if tests are flagged as failed they have failed even after a second attempt.

I am told ctest may eventually add support for the same (i.e. retry failed tests), at which point we’ll update the testing infrastructure to just use that, but until then, we are good to go!