How to Improve Software Quality in Open Source Projects: Part III – Code Coverage, Documentation and Style

Index To the Series
1. Overview of Continuous Integration (CI) and Software Process in PyNWB
2. Packaging PyNWB for Multiple Operating Systems and Python Versions
3. Code Coverage, Documentation and Style

In the second post in our series, we talked about Python 2 and Python 3 support, cross-platform operating system support, source distributions and dependency versioning. In this post, we introduce code coverage metrics, code documentation and consistent style for PyNWB.

Code Coverage Metrics

Code coverage is a measurement that highlights which lines of code are reviewed (or covered) by tests, and it helps to prioritize future test development. Just because a line of code is covered by tests, however, does not mean the line is correct. As Edsger W. Dijkstra said, “Testing never proves the absence of faults, it only shows their presence.”

For PyNWB, the project’s Travis CI file runs tests on macOS. As the file currently indicates, the tests review Python versions 2.7 and 3.6. The “script” portion of the file actually runs the tests.

PyNWB’s tox file includes commands that provide coverage reports as a part of the continuous integration (CI) process. To archive, aggregate and visualize the coverage reports, we use Codecov. Codecov is also tied to the CI process. It posts coverage report updates directly into the view of a pull request on GitHub, using tags to aggregate coverage across different test suites such as unit and integration testing.

Codecov provides separate reports for each tag. In the following report, red specifies lines that were not covered by the tests, and green specifies lines that were executed during one or more tests. Lines that are not colored were ignored by the tests (e.g., comments, function definitions, etc.).

Note that while full coverage does not mean perfection, a high coverage rate is better than a low one.

Code Documentation

Good documentation is key for a successful software project, especially an open source project, as there may be new developers who need to quickly get up to speed, and there is little friction that prevents users from adopting alternate software. Documentation is also a good proxy to indicate if a software project is actually intended to be long lived and to support growth and inclusion, or if the project is intended to serve a particular purpose and may soon be abandoned. 

PyNWB documentation is automatically generated and published each time changes are integrated into the main branch. To support these capabilities, we use webhooks; Read the Docs, which is freely available for open source projects; and Sphinx.

In our previous post, we mentioned that we activate integration between GitHub and CI services with webhooks. A webhook is very simply an HTTP callback that gets triggered after an event is fired such as an update to the main branch. The callback notifies Read the Docs that the event occurred. After the update gets triggered, Sphinx builds the HTML from the function docstrings, along with the restructured text files that we added for the update. Read the Docs then gets modified to show the updated documentation.

Consistent Style

Style tools reduce the possibility of bugs, make it easier for developers to contribute code and improve the consistency of an overall package. They do so through constraints. Setting a maximum value for cyclomatic complexity, for example, requires that certain functions be refactored before they can be merged into the main repository.

An example style tool is Flake8. Flake8 provides static code analysis and code style checks to catch mistakes. Flake8 checks integrate with CI vendors to ensure that pull requests follow style requirements. If Flake8 checks fail for CI, the code will not be merged until the style issues are fixed.

Another free style tool is PEP8 Speaks, which we use for PyNWB. PEP8 Speaks runs PEP8 tests, and a bot comments on the pull request to show the specific lines that violate the style rules. An example is the image below.

Once the violations are fixed in the pull request, PEP8 Speaks posts another comment, which declares that the pull request is compliant in terms of style.

Software Process Importance

Code coverage metrics, code documentation and consistent style are important when it comes to community adoption of your project. These concepts support consistency in the code base; they make it easier for developers to start contributing, understand the codebase and comply with your project’s requirements; and they show your project’s commitment to long-term success, which welcomes contributions.

At Kitware, we are firmly committed to software process to ensure high-quality, effective code. We encourage you to also adopt such practices and hope that this tutorial series provides a useful guide. If you have questions as you proceed, please reach out to us, and let us know how we can help.