Sustainability in Action: Cleaver

In the first two blog posts addressing software sustainability, we introduced basic concepts and the software sustainability matrix [1], and then described the matrix in more detail including the sustainability scoring process [2]. In this post we review recent efforts to improve the sustainability of the Cleaver software system [3,4,5,6], a multimaterial tetrahedral meshing tool. Through this case study, we will show how our efforts to improve sustainability resulted in greater usage of the tool and an impressive increase in the size of the community.

With our partners the Scientific Computing and Imaging SCI Institute at the University of Utah [4], we significantly improved the sustainability score of the mesh generation system known as Cleaver. Cleaver, which was developed by SCI, is an algorithm for generating tetrahedral meshes that conform to physical boundaries in volumetric domains consisting of multiple materials. The method supports an arbitrary number of materials, produces high-quality tetrahedral meshes with upper and lower bounds on dihedral angles, and guarantees geometric fidelity. The algorithm is fast, and the resulting meshing process supports grading, reducing element counts in regions of material homogeneity. It can be used to create meshes used in biomedical applications and simulation, for example to simulate heart conditions [7].

The first step towards improving sustainability was to score Cleaver against the sustainability matrix. An initial review suggested that Cleaver would increase its impact by expanding the user base; energizing the community by improving testing processes and outreach (documentation, training); and  improving system architecture to support varied distribution methods. Based on this review, we developed a plan to distribute Cleaver as widely as possible by leveraging other open-source platforms, and ensuring that potential users and developers would have a satisfying experience using and contributing back to the community.

With a plan in place, the difficult question then emerged: How would we measure whether the plan was working? This can be a difficult question to answer because open source software may be accessed in multiple ways. Web site or repository downloads can be tracked, but it’s not uncommon for tools to be downloaded and then shared through other channels; for example code shared/copied directly within an organization. Or the code may be used within an application, and then the application may be distributed. Download count may also be misleading because of mirror sites, or testing processes that access distributions on a regular basis (artificially inflating download statistics). But in general, there is a belief that tracking downloads correlates with increased usage, as long as care is taken to filter out spurious accesses.

In order to distribute Cleaver as widely as possible, we leveraged other popular open-source software systems. These systems included VTK (including VTK.js), ITK, ITK.wasm, Python Wheels, Jupyter, itkwidgets, and 3D Slicer. Our idea was that if Cleaver could be readily used / accessed through these systems, this, in turn, would encourage widespread usage, resulting in a growth of the community. For example, the SlicerSegmentMesher [8] seamlessly integrates Cleaver into the popular 3D Slicer medical computing application. Reducing friction, which is a concept common to sales and marketing, was also an important part of the plan. Making software easy to find, access, interoperate, and reuse (i.e., using the FAIR principles [9]) provides a user / developer experience that makes participating in an open-source community enjoyable and productive, with the potential to increase community size and positively impact long-term software sustainability. Along similar lines, we also sought to improve developer’s lives by ensuring that the C++ code base was c++17 compatible, and removed approximately 1500 files used to parse and process command line arguments [10]. And to improve the experience of new users, we consolidated the documentation into a set of Markdown files maintained alongside the software and published automatically on ReadTheDocs. We also made sure older links would redirect to the current portal [5].

These improvements to Cleaver emerged naturally from a discussion on general modern practices to promote scientific software sustainability. Panel discussions on these topics ranged from Design Patterns for Multithreaded Algorithm Design and Implementation [11], Web Visualization [12], Scientific Software Process for the Web [13], and Best Practices for Communication of Project, Resources, Governance, and Culture [14].

The results of this effort are impressive. While downloads for this research software were approximately 100-200 per year from the Cleaver repository, the embedded versions of Cleaver (i.e., Cleaver distributed through other channels) contributed over 33,000 additional downloads over the course of a year. And this progress came after just a couple of years of focused effort to increase sustainability. At this early stage in the sustainability cycle, it’s hard to know for sure whether these downloads will continue to grow. However with more exposure, and a larger community, there is a good chance that improvements will be made and the value of the tool will increase, resulting in a virtuous cycle of sustainability.

Acknowledgements

This project was supported by the National Institute of General Medical Sciences of the National Institutes of Health under grant numbers P41 GM103545 and R24 GM136986. See also this blog post [15] which further describes the project.

References

[1] Will Schroeder, Michael Grauer, Matt McCormick, Marcus D. Hanwell and Jean-Christophe Fillion-Robin. How Sustainable is Your Software? https://www.kitware.com/how-sustainable-is-your-software.
[2] Will Schroeder, Matt McCormick and Jean-Christophe Fillion-Robin. Scoring Software Sustainability. https://www.kitware.com/scoring-software-sustainability.
[3] The Scientific Computing Institute SCI at the University of Utah, Center for Integrative Biomedical Computing. https://www.sci.utah.edu/cibc.html.
[4] SCI Institute, University of Utah. Cleaver – A MultiMaterial Tetrahedral Meshing Library and Application. https://www.sci.utah.edu/cibc-software/cleaver.html.
[5] Cleaver documentation. https://cleaver.readthedocs.io.
[6] Bronson, J.R., Levine, J.A., Whitaker, R.T. (2013). Lattice Cleaving: Conforming Tetrahedral Meshes of Multimaterial Domains with Bounded Quality. In: Jiao, X., Weill, JC. (eds) Proceedings of the 21st International Meshing Roundtable. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33573-0_12.
[7] Personalized Heart Simulation. https://www.sci.utah.edu/cibc-research/highlights/24-cibc-highlights/655-macleod-hopkins.html.
[8] Slicer Mesher Extension. https://github.com/lassoan/SlicerSegmentMesher.
[9] FAIR Principles. https://docs.nih-cfde.org/en/latest/the-fair-cookbook/content/Introduction/fair-principles.
[10] CLI11 adoption to simplify command line parsing. https://github.com/SCIInstitute/Cleaver/pull/119.
[11] Schroeder, Will, & Tsalikis, Spiros. (2023, February 17). Design Patterns for Multithreaded Algorithm Design and Implementation. Zenodo. https://doi.org/10.5281/zenodo.7939128.
[12] McCormick, Matthew, Jourdain, Sebastien, Li, Forrest, & Fillion-Robin, Jean-Christophe. (2023, March 2). Web Visualization. Zenodo. https://doi.org/10.5281/zenodo.7939136.
[13] McCormick, Matthew, & Fillion-Robin, Jean-Christophe. (2023, March 16). Scientific Software Process for the Web. Zenodo. https://doi.org/10.5281/zenodo.7939194.
[14] Fillion-Robin, Jean-Christophe, Sullivan, Bane, & McCormick, Matthew. (2023, April 27). Best Practices for Communication of Project, Resources, Governance, and Culture. Zenodo. https://doi.org/10.5281/zenodo.7939200.
[15] The Scientific Computing and Imaging (SCI) Institute and Kitware: Putting Software Sustainability into Practice. https://www.kitware.com/the-scientific-computing-and-imaging-sci-institute-and-kitware-putting-software-sustainability-into-practice.