Holograms with VTK: Stream VTK to the Hololens 2

Following the initial proof of concept that was implemented in ActiViz, Kitware is pleased to officially announce support for rendering inside the Hololens 2 using VTK.

The recent addition of the OpenXR runtime in VTK allowed us to build a dedicated module for streaming rendered content to the Hololens 2. The new module, available in VTK 9.2.2, leverages the OpenXR Holographic Remoting extension to allow VTK users to bring advanced algorithms and visualization techniques to the Hololens 2, and interact with the scene in real time.

So far we mainly focused on building a common infrastructure for both augmented reality (AR) and virtual reality (VR). This first integration in VTK provides basic interactions for users to grab and move objects around in real time.
It also ensures that the advanced features offered by VTK, such as Volume Rendering or 3D widgets, can be effectively used and rendered in virtual scenes.

This initial support for the Hololens 2 defines a generic architecture that allows for building more complex feature in the future, including:

• Integration into Paraview and 3D Slicer
• Enriched hand interaction thanks to finger tracking
• Depth occlusion support for a better integration with the physical environment

Reminder about the approach

Some devices such as the Hololens 2 only support Universal Windows Platform (UWP) applications, and do not provide any support for OpenGL. As a result, it is a huge challenge to deploy VTK-based applications on such devices.
Holographic remoting consists in a player application running on the device, and a VTK-based remote application running on a standard Windows machine. The remote application receives camera information and rendering resources from the player. It renders the VTK scene before streaming back the resulting texture to the player application. Hence, this streaming approach allows us to keep VTK on a desktop machine. This way we avoid the need to add UWP support in VTK, and we can also keep using VTK’s OpenGL-based rendering pipeline.

Still, DirectX must be used to fill the texture to be streamed back to the Hololens. To achieve this we created a texture shared by both a DirectX and OpenGL context. This is possible thanks to the NV_DX_interop extension available on almost every recent GPU.

How to

Player application

Download the Microsoft MixedReality HolographicRemoting samples and follow the instruction to build the player application.
⚠️ The version number in the branch name must match the version of the Microsoft.Holographic.Remoting.OpenXr package used below by the remote application.

Follow the instructions to deploy the player application to the Hololens 2.
Alternatively, if you don’t have access to a device, you can use the Hololens emulator.

When the player is deployed, you should see the following message: Waiting for connection on XX.XX.XX.XX where XX.XX.XX.XX describes the IP address the remote application should connect to.

VTK Remote application

Enable the CMake option VTK_MODULE_ENABLE_VTK_RenderingOpenXRRemoting when building VTK, and set the OpenXRRemoting_BIN_DIR and OpenXRRemoting_INCLUDE_DIR to provide the path to the OpenXR Remoting headers and binary directory.
The Microsoft.Holographic.Remoting.OpenXr Nuget package that provides this dependency is available here on nuget.org.
⚠️ The version of the Microsoft.Holographic.Remoting.OpenXr package must match the branch name of the player application above.

The new RenderingOpenXRRemoting module depends on the existing RenderingOpenXR module which requires the OpenXR SDK. Download and build the latest version (1.0.25) of the OpenXR-SDK from here, in order to provide the VTK CMake configuration with the OpenXR_LIBRARY and OpenXR_INCLUDE_DIR variables.

When successfully built, use the OpenXR and OpenXRRemoting dedicated rendering stack in your application, as highlighted below.

  vtkNew<vtkOpenXRRenderer> renderer;
  vtkNew<vtkOpenXRRemotingRenderWindow> renderWindow;
  vtkNew<vtkOpenXRRenderWindowInteractor> iren;
  vtkNew<vtkOpenXRCamera> cam;
  renderWindow->AddRenderer(renderer);
  iren->SetRenderWindow(renderWindow);
  renderer->SetActiveCamera(cam);
  renderWindow->SetRemotingIPAddress("XX.XX.XX.XX");

The address of the player application to connect to must be set using vtkOpenXRRemotingRenderWindow::SetRemotingIPAddress("XX.XX.XX.XX") before starting the interactor.
See the TestOpenXRRemotingInitialization test for a complete example.

⚠️ Make sure to provide the content of the OpenXR Remoting binary directory in the system PATH or next to the executable before running the program.

Troubleshooting

Check the Troubleshooting section in the README file of the OpenXR Remoting module if you encounter any issue while trying the new module.

Acknowledgment

This work is funded by the CALM-AA European project (cofunded by the European fund for regional development), and the United States National Institute of Standards and Technology (NIST) under Kitware Contract 1333ND21PNB770285 “Modifications and Enhancements to the ParaView /VTK Software for Immersive Scientific Visualization”