CoProcessing with ParaView

With the processing power of supercomputers still rapidly growing, simulation codes that can run on them generate more data at a faster rate.  While this growth enables more accurate and/or complex simulations to be run, it also makes examining the results a much more difficult task. A large part of this is due to the fact that while individual processor clock rate improvements are stagnating, more cores are being put in a processor and more processors are being strung together to form the latest supercomputers. 

As an example, the top public supercomputer in the world, a Cray XT5-HE at Oak Ridge National Laboratory, has 224,162 cores. Similarly, of the top 10 computers on this list the amount of cores range from 41,616 to 294,912 cores. With this amount of computing power available it is unrealistic to hope to efficiently analyze the results on small clusters let alone a single desktop or laptop computer. Just managing the output files from these machines can take a lengthy amount of time. ParaView’ssolution to this problem is to perform coprocessing. 

Coprocessing, or in-situ processing, is the term used for computing visualization information during a simulation run.  Some of the reasons for this are:

1. Reduce the amount of file IO needed since only the desired results are saved instead of the whole simulation output
2. Use the same amount of computing power to analyze the output as the simulation is run on

The current work flow is to set up the coprocessing pipeline with a ParaView plugin. This is output as a Python script that is used during the simulation run. The simulation code must be instrumented to use ParaView’s coprocessing library.  Most of this work involves creating an adaptor that creates a ParaView data structure (e.g., vtkUnstructuredGrid, vtkImageData, vtkGraph, etc.) from the simulation code data structures. Information on this process is available at http://paraview.org/Wiki/CoProcessing. 

We are currently testing the coprocessing library with a CFD simulation code called PHASTA (a 2009 Gordon Bell finalist). Dr. Kenneth Jansen, University of Colorado at Boulder, is the lead developer of PHASTA, with significant contributions from several other researchers.  At this point most of the testing has been done on an IBM BlueGene/L at Rensselaer Polytechnic Institute’s Computational Center for Nanotechnology Innovations. So far we have only tested between 256 and 16,384 MPI processes but plan on testing up to the full 32,768 cores available on the machine.