First VTK Dashathon
On Wednesday February 27 a group of intrepid software developers gathered at Kitware HQ in Clifton Park, NY to get the VTK dashboards back on track. The major motivation here was to go from a dashboard with multiple failing tests on many platforms to a green dashboard where it would be obvious when new topics induce regression test failures and would then be held back until actually ready. This was quite a historic gathering, the first ever VTK dashathon where we managed to gather the three copyright holders in one room. Will and Bill contributed code, but we couldn’t tempt Ken back…this time.
We would like to thank Bill Lorensen for calling us all together, and coordinating this work along with the support from Berk and Will to hold the event at KHQ. Earlier today Bill summarized what motivated the dashathon in his post “Software Quality: Talking vs Walking“. We are extremely pleased to have gone from wishing to “establish a baseline of failing tests” to having a green dashboard that can be used to critically assess whether a proposed topic is ready to go in based on a simple assessment of the CDash@Home dashboard submissions. This comes after more than 2,000 topic submissions to Gerrit using code review and topic branches in VTK development. Below is the expected section of the nightly from before we started and today. If you click through, note in particular the marked difference in the Gerrit and continuous test sections.
 |
 |
| January 27 | March 8 |
In the process we also developed a better understanding of what exactly the dashboards are validating. We can now simply run scripts that query CDash and augment its presentation with detailed analysis of the platform and option set that we cover on any given day. Our testing is not just greener and more useful for committers, but also more meaningful since we are careful to use our limited testing resources to test VTK more thoroughly than we did.
There is still more to be done. We plan to:
- add testers to cover the soft spots in the option space (Windows with Java) and our software process (Mac and Windows continuous with all modules enabled to backstop the gerrit checks),
- improve the coverage of VTK’s substantial code base,
- begin migrating to external data for baseline images to further improve the clarity of topic submissions in the midst of an increasingly distributed and branchy development process.
With VTK and the dashboards restored to a green and robust state, I’m sure that all of us in the VTK development team will pay due diligence to maintaining a healthy VTK development branch (or else!!!). We would like to thank everyone who attended and contributed to VTK’s first dashathon effort, especially our external contributors Bill Lorensen, David Cole, Sean McBride and David Gobbi!