V&V As a Way of Life

There’s nothing like a good conference to remind you of the basics.

I recently attended IEEE VisWeek and participated in the panel “Verification in Visualization: Building a Common Culture”. The panel was organized by Mike Kirby (Utah) and Claudio Silva (NYU Poly) with Robert Laramee (U. of Swansea) and myself as additional panelists. The focus of this particular panel was on the principles and implementation of validation and verification (V&V) in the computational sciences, with emphasis on the visualization process.

For those of you who don’t know, V&V in this context is about ensuring that a software system meets specifications and that it fulfills its intended purpose. Validation addresses the question “Are we building the right thing?” and verification addresses the question “Are we building it right?”

Well who’s going to disagree with a panel that advocates a culture of V&V? It’s hard to have a provocative and entertaining panel when the thesis is generally agreed on by everyone. But that’s one of the key points of the panel: we often take things for granted, so we have to build into our DNA essential, basic principles such as validation and verification to achieve the culture we desire.

My co-panelists, being the talented academics they are, provided very compelling examples of why V&V is critical to algorithm design and implementation. They even had detailed charts indicating how validation and verification should be incorporated into the computational sciences workflow.

Here’s where I think some controversy comes in, it’s how you practice V&V. Is the process a separate, discrete step as part of the creation and implementation of algorithms (with maybe an occasional V&V workshop thrown in)? Or is it a Way of Life? Meaning do we create an algorithm, make sure it’s doing what’s it’s supposed to do (validate), verify on our sample data sets, and then throw it over the fence (publish) and call it done? Or do we take the approach that validation and verification is an ongoing process that is never really done, consequently requiring us to practice it on a continual basis?

These questions remind me of three related corollaries you often hear in the open source communities we participate in:

• If it’s not reproducible, it’s not science.
• If it’s not verified, it’s not engineered.
• If it’s not tested, it’s broken.

which are all commitments to process. In the open source world, our DNA is ongoing collaboration, evolution, testing and refinement. We know that innovation never ceases, and that continual processes are necessary to stay relevant and vital. Thus V&V is implicitly bred into the bones of open source processes.

So you know where I stand on this: as a practitioner of The Way of the Source, I am a firm believer that verification and validation is a Way of Life, a continual process in which we ask “is our software doing what our users want” and “are we implementing this software correctly?” From hard experience software professionals understand that systems inherently have many dependencies ranging from hardware, to drivers, to the operating system, to system libraries, to the algorithms themselves, and that these dependencies are constantly changing, hence software needs continual testing. We also know that technology changes, and hence our use of technology, consequently a “valid” and effective software system may become invalid, and we may have to revamp or recreate a solution to meet the needs of our users.

If we are to embrace a culture of validation and verification as the panel supposes and to which we all blithely agree, then we need to do the hard work of establishing formal software testing process in combination with continual reevaluation of software effectiveness. Furthermore, because the process is by definition a public certification of the validity and veracity of our systems, open source practices are necessary to support the required transparency and essential evaluation and collaboration processes. It is only when we manifest these practices in our daily software workflow that we can truly say we have established a culture of V&V.