VTK.js Transforms Web-based Visualization with Immersive Virtual and Augmented Reality

Are you looking to transform your web pages into immersive experiences? Kitware can help bring your medical and scientific data off the computer screen and into real-life using VTK.js, our open source rendering library for scientific visualization on the web. VTK.js now has modern web-based virtual and augmented reality visualization, thanks to the power of WebXR.

Web-Based Immersive Rendering

WebXR is a browser technology that exposes an API to enable tracking and rendering mixed reality sessions on mobile or head-mounted devices, such as your smartphone or a dedicated virtual reality headset. By integrating WebXR functionality into VTK.js, we leverage the existing rendering pipeline with WebGL (and soon, WebGPU) to bring new, immersive visualizations to customers across a broad array of hardware.

An expanding set of AR and VR examples are available as part of the VTK.js documentation. Additionally, all of the previous VTK.js VR examples have been updated for WebXR. Dedicated hardware such as a head-mounted display or a mobile device can be used to try out our VR and AR examples. Developers can also install the WebXR Emulator extension for Chrome or Firefox to simulate XR hardware in their browser.

Virtual Reality Capabilities

WebXR succeeds the deprecated WebVR and Cardboard VR APIs that were previously supported by VTK.js. All of the previous VTK.js VR examples have been updated for WebXR. Additionally, the WebXR polyfill can be dynamically included to enable backward-compatible support for these deprecated APIs. Moving forward, any native VR backend that implements the OpenXR standard, including OVR and SteamVR, can be used to drive WebXR content from VTK.js.

Virtual reality rendering is accomplished in several stages:

1. Information about the physical pose of VR hardware is acquired every frame through the WebXR API. The position and orientation of the headset relative to the user’s physical space are set as the view-to-physical transformation of the virtual camera. Additionally, the pose of interface devices such as controllers or gamepads is also collected.
2. The camera is adjusted from the physical space to the virtual world space. Various parameters about the scale and position of the user relative to the data that they will be viewing can be adjusted here. By default, we scale the scene so that the user can comfortably view and walk around their data.
3. The VTK.js rendering pipeline makes one pass for each eye, shifted slightly from the center of the camera.
4. The frame is drawn to the WebGL buffer for display. WebXR abstracts away the process of shifting the WebGL context so that the frame is always presented in the user’s head-mounted display, regardless of whether the user was previously viewing the webpage on a monitor or in their headset.

These details are all hidden in the standard VTK.js WebGL RenderWindow, which makes it easy to add VR support to existing and new projects by simply invoking the “startXR” method. Additionally, VR controller events such as trigger presses are captured with the standard VTK.js RenderWindowInteractor and can be handled with an InteractorStyle or directly in an application.

Augmented Reality Capabilities

WebXR introduces web-delivered augmented reality support into VTK.js for the first time, initially targeting mobile handheld devices. The ARCore backend drives AR experiences on Android devices and is available in the Chrome mobile browser. At the time of writing, development is still underway on WebXR’s ARKit browser compatibility on iOS devices, but iOS users can download a dedicated application such as the Mozilla WebXR Viewer to engage with WebXR content.

Much like VR rendering, AR rendering relies on pose information delivered through the WebXR API to set the virtual camera position in the virtual world for rendering with the VTK.js pipeline. WebXR hides the camera stream from the consuming application and presents it instead as a skybox with virtual objects rendered in front, with the requirement that any virtual background is first made transparent.

Projects already relying on the VTK.js WebGL RenderWindow can once again easily take advantage of the new functionality by invoking “startXR(true).” AR events can be queried directly through the WebXR Gamepad API or handled as events from RenderWindowInteractor.

To showcase the VTK.js’ AR capabilities, we have recently added VTK.js example applications for viewing volumes and geometries in AR.

Potential Applications

Initial WebXR integration with VTK.js has been funded by the NIH through multiple awards. This work is freely available for academic and commercial efforts. Web-delivered immersive visualization with VTK.js has many potential applications in a variety of fields, such as those described below:

• Guided therapy: Kitware’s Medical Computing Team has significant project experience with real-time guided therapy systems that deliver computer-aided visual feedback to experienced clinicians as well as novice users as they administer treatment to a patient. These systems typically encompass a sequence of tasks: acquiring data with a medical device, gathering insights with computer vision or other modeling systems, and then presenting information statically on a screen to inform treatment. Emerging augmented displays can enable more intuitive and more dynamic feedback directly to the user, helping them, for example, improve the positioning of a point-of-care ultrasound probe or a therapy device during data acquisition. Specific instances of this application area include helping EMS personnel triage patients suspected of traumatic brain injury or intra-abdominal bleeding or helping clinicians monitor COVID treatment response bedside.
• Shared, pervasive scientific visualization: Both virtual and augmented reality experiences offer the advantage of understanding data on a real-world scale. Web-distributed XR experiences with VTK.js make it easy to examine models and datasets with a sense of relativity to the surrounding world, which can, for example, be useful for visualizing small- and large-scale CAD designs to make adjustments before fabrication or for mixed-reality explorations of complex systems.
• Training systems: As has been demonstrated extensively by Kitware’s iMSTK platform, virtual reality has the advantage of offering a safe, repeatable, low-cost training simulation environment. WebXR offers several API extensions which allow simulations to take advantage of rich input schemes for maximizing learning through interactions, such as hand tracking or touch-sensitive controller feedback.

Partnering with Kitware

Kitware is dedicated to advancing cutting-edge scientific computing and visualization technologies. Our team is composed of computer scientists, software engineers, and scientific computing experts. They have the knowledge and expertise to support your pursuit of AR and VR technology and can help you practically incorporate these XR experiences into your work. Whether you’re a research center, university, or commercial company, partnering with Kitware will ensure you effectively leverage this technology. For more information, you can send us a message at kitware@kitware.com or visit our commercial projects page. 

Acknowledgments

Research reported in this publication was supported by the National Institute Of Biomedical Imaging And Bioengineering and the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Numbers R01EB028283, R01EB014955, and R42NS086295. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.