3D Slicer now enables you to visualize and explore 3D medical data in a fully immersive and more intuitively way, thanks to the addition of support for Virtual Reality (VR). More specifically, you can interact with a scene in 3D Slicer using a VR head-mounted device (HMD) to train for medical procedures; perform surgical planning; or identify trends, patterns, and outliers in data.

The VR support for 3D Slicer comes through the Slicer Virtual Reality extension, which relies on OpenVR modules. Every model, volume or widget that can be displayed in 3D Slicer’s 3D view can also be displayed in the VR view. This VR view is synchronized with the 3D Slicer scene. Hence, if you modify the scene in the 3D view, it will be instantly reflected in the VR view and vice versa.

With Slicer Virtual Reality, you can directly interact with the scene using the VR controllers that ship with a headset. (Compatible HMDs include Oculus Rift; HTC Vive; and Windows Mixed Reality headsets by Acer, Lenovo, and HP.) For now, you can interact with the implicit plane, the region of interest and the fiducial widgets. Interactions with other widgets will be enabled in the future.

You can also use the pose of the HMD controllers as transforms in the 3D Slicer scene. These transforms can be implemented in custom modules to reslice volumes (with the Volume Reslice Driver module in the SlicerIGT extension) or to manipulate any data in the scene. Furthermore, you can navigate the 3D scene using the touchpad of the right controller of HTC Vive and other mixed reality headsets. The orientation of the controller specifies the direction, while the position of the finger on the touchpad determines the speed.

A complete list of features include:

• View all 3D content typically displayed in the 3D view (e.g., 3D image data in the form of 2D image slices, volume renderings, and 3D models) inside the VR headset with a single click;
• Visualize 4D datasets using any rendering technique (including volume rendering) by means of the Sequences extension;
• Align the headset’s view to match the viewpoint of the selected 3D view;
• Grab and reposition objects using the trigger on the right controller on HTC Vive and other mixed reality headsets; and
• Tune advanced volume rendering performance in the VR module in order to find a good balance between image quality and refresh rate.

Now that we have provided an overview of Slicer Virtual Reality, we will describe how to install the extension and discuss a few projects actively using the extension module.

Installation

Slicer Virtual Reality can be used on any platform where OpenVR and VTK OpenVR are available and able to compile. For now, this means Windows, although there is a beta version for Linux. The are two ways to enable VR in 3D Slicer: (1) Download the extension module from the Slicer Extensions Manager, and (2) build the module from the source code.

1. Downloading from Slicer Extensions Manager

Slicer Virtual Reality is available for download using the Slicer Extensions Manager of the nightly versions of 3D Slicer, which is available on http://download.slicer.org.

Once you install the extension, the virtual reality toolbar shown below will be displayed as part of the 3D Slicer application. In this new toolbar, you can enable VR rendering by clicking the first button that shows the HMD logo (highlighted below). The settings button on the right allows you to configure the VR rendering.

2. Building from Source

To build Slicer Virtual Reality, first, complete the 3D Slicer nightly build according to the 3D Slicer Developers manual. Then, check out the VR extension source code from the Kitware Medical repository. Finally, build Slicer Virtual Reality following the instructions in 3D Slicer’s developer documentation.

Please note that 3D Slicer needs to be compiled against Qt5 and VTK 9. VTK doesn’t have to be compiled with VTK OpenVR option set to ON, however. The extension build system will take care of compiling and packaging the OpenVR module independently.

Project Showcases

SlicerVirtualReality has already been used in multiple projects. Three of these projects are a collaboration between Kitware’s medical computing team and the Department of Oral and Maxillofacial Surgery at the University of North Carolina at Chapel Hill, Children’s Hospital of Philadelphia (CHOP) and The Perk Lab at Queen’s University.

Virtual Reality Orthognathic Surgery

VR-based surgical simulators are becoming a better alternative to surgical video channels and books since they improve surgical knowledge and provide an immersive surgical experience that enhances the spatial awareness, speed, and dexterity necessary for surgery. To create a surgical simulator for Bilateral Sagittal Split Osteotomy (BSSO), we used Slicer Virtual Reality. BSSO surgery moves the jaw bone forward in order to restore the proper anatomical and functional relationship in patients with dentofacial skeletal abnormalities. 

The image below shows a user performing the bone sawing procedure with the Geomagic Touch haptic device and visualizing results with Slicer Virtual Reality and the Oculus Rift HDM.

More information about this project can be found in “High fidelity virtual reality orthognathic surgery simulator,” a paper published at the SPIE Medical Imaging 2018 conference.

Volume Rendering of Cardiac Tomographic and Echocardiographic Images in Virtual Reality

Three-dimensional cardiac imaging is increasingly utilized in clinical settings, but it is typically viewed on 2D screens. To visualize 3D data from cardiac imaging, we used Slicer Virtual Reality and HTC Vive. For this work, we imported and visualized cinematic images using the 3D Slicer Sequences extension and used Slicer Virtual Reality to display the anatomy in the HTC Vive headset. The total time from the import of the DICOM data to the interactive VR visualization in 3D Slicer was less than a minute.

The image below depicts cardiac CT, cardiac MRI and 3D echocardiographic visualizations in stereoscopic VR in 3D and 4D. 

More information about this project can be found in a recently published paper in The Journal of the American Society of Echocardiography:

Andras Lasso, Hannah H. Nam, Patrick V. Dinh, Csaba Pinter, Jean-Christophe Fillion-Robin, Steve Pieper, Sankhesh Jhaveri, Jean-Baptiste Vimort, Ken Martin, Mark Asselin, Francis X. McGowan, Ron Kikinis, Gabor Fichtinger, Matthew A. Jolley. Interaction with Volume Rendered 3D Echocardiographic Images in Virtual Reality. The Journal of the American Society of Echocardiography. Accepted June 2018, in the press. https://doi.org/10.1016/j.echo.2018.06.011

Ultrasound-guided needle insertion training using Virtual Reality

Ultrasound-guided needle insertion is an essential skill in interventions such a nerve blocks and biopsies. Such biopsies can be challenging for trainees due to simultaneous manipulation of the ultrasound device and the needle, image interpretation and lack of spatial awareness. We used Slicer Virtual Reality to improve the trainees’ hand-eye coordination and spatial awareness during ultrasound-guided needle insertion. Upon completing an insertion, trainees were able to re-watch it in VR to better understand the spatial relationship between the ultrasound device, the needle and the anatomy itself.

More information on this project is available on www.slicerigt.org.

Additional Resources

We aim to improve the feature set of Slicer Virtual Reality. Please give us know your feedback or ideas for improvement on the extension’s issue tracker. To learn more about how Kitware can tailor 3D Slicer solutions to meet your research needs, please contact kitware@kitware.com.

Acknowledgment

Research reported in this blog  was partially supported by National Institutes of Health (award 1R43DE027595-01), Department of Anesthesia and Critical Medicine at Children’s Hospital of Philadelphia,  and Collaborative Health Research Project (CHRP #127797)-a joint initiative between the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canadian Institutes of Health Research (CIHR), Southeastern Ontario Academic Medical Association (SEAMO)’s Educational Innovation and Research Fund.