Community Spotlight:Visualizing Deformable Suspension Simulations in ParaView

Computational simulation of blood flow and dense suspensions of deformable particles is only possible on high performance computing resources using a highly scalable computational approach such as the hybrid lattice-Boltzmann / finite element (LB-FE) method. This method, originally developed for rigid suspensions, was extended to deformable red blood cells (RBCs), particles, capsules [1] and fibers [2, 3]. The LB-FE method resolves the fluid phase, both interior and exterior to the fluid-filled particles, with the LB method. A recent review article [4] highlights the feasibility of using the LB method for complex flows including suspension of deformable, particles, capsules and fibers.

We examine the dynamics of deformable capsules and how these dynamics affect the rheology of dense suspensions using the LB-FE method. Of particular interest are the particle pressure and a full characterization of normal stresses, which are difficult to measure experimentally. The following figure demonstrates a novel problem where the particle microstructure affects the viscosity, normal stresses, and particle pressure of a suspension. The maximum viscosity and first normal stress difference is obtained when particles are aligned in a cluster along the compressional axis.

Red blood cells (RBCs), the most numerous constituent of blood, influence continuum-level measurements by altering the suspension at microscopic scales. In moderately-large-sized arteries, the nature of blood can be described as a macroscopic non-Newtonian fluid; however, flow in smaller vessels requires treating human blood as a multiphase fluid containing 40-45% RBCs by volume. Simulations of blood at a cellular level provide a tool that allows exploration of the rheology, stress, and diffusion of individual suspended cells. In our approach, the blood plasma and hemoglobin are treated as Newtonian fluids with viscosities of 1.2 and 6 cP, respectively. The multiphase nature of the blood creates non-Newtonian flow effects, such as shear thinning, commonly observed in experiments. An image of multiple clusters forming in Hagen-Poiseuille flow is shown below.

VISUALIZATION OF DATA WITH PARAVIEW
We have shown the ability to simulate thousands of deformable particles and capture non-Newtonian characteristics that agree well with experimental observations [1]. Our approach has shown the ability to scale on as many as 65,536 computational cores of Argonne National Laboratory’s IBM Blue Gene/P [5]. When computing the dynamics of 1,000’s to 100,000’s of deformable particles suspended in fluid, datasets can easily reach 1 terabyte or greater for a single simulation. As an example, one such simulation with 100,000 deformable spheres (each with 254 triangular surfaces) requires a LB fluid domain in excess of 2 billion lattice grid points.

When fluid and solid data sets from CFD codes are O(100 GB), the use of distributed memory architectures is required for post-processing. Through the use of virtual network computing (VNC) and access to visualization clusters like Longhorn and Spur at the Texas Advanced Computing Center (TACC) via NSF’s TeraGrid HPC network, visualizations not possible on shared memory resources can be performed. Visualization clusters like those at TACC offer multiple CPU, multiple GPU nodes that are created specifically for these applications. The following is an instantaneous snapshot of a dense suspension of 2472 RBCs in a wall-bounded shear.

FUTURE WORK
Recent additions [6] have shown the ability to simulate dense suspensions of RBCs in micro-vessel sized tubes at high shear rates, in which large RBC deformations extend beyond the assumptions of linear theory. This work is based on coupling the LB method with a spectrin-link model [7] for the RBC membrane.
 

REFERENCES
[1]  R. M. MacMeccan, J. R. Clausen, G. P. Neitzel, and C. K. Aidun, “Simulating deformable particle suspensions using a coupled lattice-Boltzmann and finite-element method,” J. Fluid Mech., 618:13-39, 2009.

[2] J. Wu and C. K. Aidun, “Simulating 3D deformable particle suspensions using lattice-Boltzmann method with discrete external boundary force,” Inter. J. Num. Meth. Fluids, 62:765-783, 2009.

[3] J. Wu and C. K. Aidun, “A method for direct simulation of flexible fiber suspensions using lattice Boltzmann equation with external boundary force,” Inter. J. Multiphase Flow, 36:202-209, 2010.

[4]   C. K. Aidun and J. R. Clausen, “Lattice-Boltzmann Method for Complex Flows,” Annual Rev. Fluid Mech., 42:439-472, 2010.

[5] J. R. Clausen, D. A. Reasor, and C. K. Aidun, “Parallel Performance of a Lattice-Boltzmann/Finite Element Cellular Blood Flow Solver on the IBM Blue Gene/P Architecture,” Comp. Phys. Comm., 181:1013-1020, 2010.

[6] D. A. Reasor, J. R. Clausen, and C. K. Aidun, “Coupling the lattice-Boltzmann and spectrin-link methods for the direct numerical simulation of cellular blood flow,” submitted to Inter.l J. Num. Meth. Fluids, June 2010.

[7]  J. Li, M. Dao, C. T. Lim, and S. Suresh, “Spectrin-Level
Modeling of the Cytoskeleton and Optical Tweezers Stretching of the Erythrocyte,” Biophys. J., 88:3707-3719, 2005.

Dr. Cyrus Aidun joined the Woodruff School of Mechanical Engineering at the Georgia Institute of Technology as a Professor in 2003. He began at Tech in 1988 as an Assistant Professor at the Institute of Paper Science and Technology. Prior, he was at Battelle Research Laboratories and was Senior Research Consultant at the National Science Foundation’s Supercomputer Center at Cornell University.

Daniel Reasor received his Bachelor’s and Master’s degree in Mechanical Engineering from the University of Kentucky (2006, 2007). He is currently a Ph.D. student at the Georgia Institute of Technology performing research focused on the direct numerical simulation of cellular blood flow. He is funded by the U.S. Department of Defense through the ASEE SMART fellowship.

Dr. Jonathan Clausen received a Bachelor’s degree in Mechanical Engineering from Clemson University in 2004 and a Ph.D. from Georgia Institute of Technology in 2010. His Ph.D. research focused on the rheology and microstructure of deformable capsule suspensions. Since July 2010, Jonathan Clausen has been a Senior Member of Technical Staff at Sandia National Laboratories.

Dr. Jingshu Wu has a Master’s degree in Aerospace Engineering (2004) and a Ph.D. in Mechanical Engineering (2010) from the Georgia Institute of Technology. His Ph.D. research focused on the development of methods for multiphase flow simulation. Since June 2010, he has been an aeroacoustic engineer at Vestas Technology R&D Americas.