Using Virtual Reality Devices with VTK
Updated September 2016 to include Oculus Rift support
Virtual reality (VR) is a technological game changer. It allows us to look at and interact with data sets, models, and point clouds to give us a glimpse of what we can accomplish with our information. As hardware and rendering performance improve, the use of VR will only become more compelling.
With this in mind, we added two new modules into the Visualization Toolkit (VTK) to make it possible to use Oculus and OpenVR-compatible devices. The new modules vtkRenderingOculus and vtkRenderingOpenVR have been tested on the Oculus Rift and HTC Vive devices respectively.
Getting Started
To use these new modules, download the master branch of VTK from the VTK respository on GitHub. Before configuring VTK you should make sure you have downloaded and installed the Simple DirectMedia Layer 2 (SDL2) as the Oculus Rift and HTV Vive modules require it. If you are targeting the HTC Vive you will also need the OpenVR SDK, for the Oculus Rift you will need the Oculus Windows SDK . At this point you can configure VTK and turn on the modules you desire and start the build process. To maximize performance, make sure you build an optimized version of VTK.
What’s Included
Our goal is to allow VTK programs to use the Oculus and OpenVR modules with few changes, if any. If you link your executable to the vtkRenderingOpenVR module, the object factory mechanism should replace the core rendering classes (e.g., vtkRenderWindow) with the OpenVR-specialized versions in VTK. vtkRenderingOpenVR contains the following classes as drop-in replacements.
vtkOpenVRRenderWindow
This class handles the bulk of interfacing to OpenVR. It currently supports one renderer that covers the entire window.
vtkOpenVRCamera
vtkOpenVRCamera gets the matrices from OpenVR to use for rendering. It contains a scale and translation that are designed to map world coordinates into the head-mounted display (HMD) space. Accordingly, you can keep world coordinates in the units that are best suited to your problem domain, and the camera will shift and scale the coordinates into the units that make sense for the HMD.
vtkOpenVRRenderer
The vtkOpenVRRenderer class computes a reasonable scale and translation. It sets these computations on OpenVRCamera. It also sets an appropriate default clipping range expansion.
vtkOpenVRRenderWindowInteractor
While VTK is designed to pick and interact based on X,Y mouse/window coordinates, OpenVR provides XYZ world coordinates and WXYZ orientations. vtkOpenVRRenderWindowInteractor catches controller events and converts them to mouse/window events. The class also stores the world coordinate positions and orientations for the styles or pickers that can use them. vtkOpenVRRenderWindowInteractor supports multiple controllers through the standard PointerIndex approach that VTK uses for MultiTouch.
Future Developments
These new Virtual Reality modules are currently still improving. Moving forward, we look to add support for the OpenVR overlay, which is great for displaying a user interface. We also aim to make the module faster and include more event interactions. Stay tuned to our blog for updates.
In the meantime, download VTK and test the modules for yourself. Let us know what you think by adding your experiences on the VTK mailing lists!
Great news! Serious VR will be a great communication tool to show scientific results.
Any chance Google Cardboard will be supported by OpenVR? From what I have read, it seems Unity supports both, but OpenVR supports only HTC Vive.
We are hoping to add support in the future for Google’s current VR efforts. It is something we are actively pursuing.
Hello! We have developed something like that for Oculus Rift SDK 2. In particular, we developed an interface between VTK, Oculus Rift and Leap Motion device. The final aim of this research project is the emulation of a virtual orthopeadic laboratory for lower limb prosthesis design (VOLab). We will publish some videos about this research work in the next months. (web site: http://dinamico2.unibg.it/vk/)
Just great!
Can you change the tracking world in vtk OpenVR Rendering to avoid using the Oculus tracking system?
I had successfully run vtkOculusTest in my environment. I hope I can do some interaction with my romote in the virtual world.Are there any examples?Now I can not even rotate the object. Thanks!
The plan is to add event interaction for Oculus once I get my hands “literally” on a pair of touch controllers. I could have added xbox controller interactions but decided to wait for the touch controller to be released. Hopefully early next year I’ll update the event code to add support for those.
Hello Ken, how can i get the vtkOculusTest example? thanks a lot.
You can choose BULID_TESTING and Mosule_vtkRenderingOculus of the latest 7.1 in Cmake.Then generate it.There are some little errors in the rendering loop.
Just a quick update. VTK master has been updated and now includes fixes for OpenVR SDK 1.0.5 as they changed a few function signatures. I haven’t had a chance to revisit Oculus lately but I am doing another pass on the OpenVR support to clean up a few issues, add a Volume Rendering example, and add a feature or two.
Some more updates. There are some new changes including the following
Add floor option to show a floor
completely rework the matrix and coordinate code to be cleaner and use standard VTK conventions (right handed)
add support for a dashboard overlay with options to subclass and support for saving/loading camera poses and adjusting motion factors and scale factors.
Improve the resetcameraclippingrange to work better for VR
change how eye position is handled so that VolumeRendering will work properly
Handle the case where people make the cubes invisible which results in a 308 error on loading the model.
Add support for the user to specify the mapping of physical coordinates into world coordinate. Allows for different view up vectors other than the old code which was hard coded to Y.
Add middle mouse support mapped to the grip button.
Add some support for a hardware clipping plane.
Hi Ken, for some reason open selecting module vtkRenderingOpenVR in cmake, the build throw several errors saying there are unresolved references. I have both SDL and openVR SDK installed and i have tried to build several times, trying with different options but still no luck.
Could someone from vtk provide files for pre-built vtk that includes vtkRenderingOpenVR module wrapped in python 3.6 ?
I would much appreciate it, as several hours of google search and running builds on Visual Studio have proved completely useless.
You might try posting on the vtkuser’s list (with the specific build errors you see), I know folks out there have it building and working and often someone pops up with some advice (or maybe they have binaries). I believe my builds are all Python 2 based which doesn’t help in this case.
Hello Ken ,
I am using VTK 8.0 for renderingin Oculus Rift DK2 , After building VTK 8.0 libraries using VS2015, Successfully tested VTK related sample programs, But am getting error with “vtkOculusRenderWindow” when i tried to render to oculus. Kindly help me with your suggestions. Am i missing some configuration setup or not ?
“Error LNK2019 unresolved external symbol main referenced in function “int __cdecl invoke_main(void)” (?invoke_main@@YAHXZ)”
Oculus changed their SDK such that old code no longer compiles and we haven’t had time to update VTK to match it. In the mean time I think you can use VTK–Oculus vis steamvr using the OpenVR module in VTK.
Hi Ken,
Does animation in work in openVR ?
I used python animation code, that works fine on flat screen, with vtk open vr module. But i only got a static image. The timer didn’t work i am guessing.
Curious to know if i am doing something wrong or its the vtk open vr implementation ?
Thanks,
Aditya
It works from the GUI, not sure about from python. Make sure you send to VR before you start the animation.
I am not sure what you mean by GUI.
I think there is a problem with Timer, it doesn’t fire the relevant function at all.
I tried the vtk animation examples of both Python and C++ in openVR (with no modification except using openVR modules for c++) and it doesn’t work. Any ideas ?
It seems like some other users are having similar issues with animation as well
Thanks 🙂
No the timer code is not supported. I do not believe OpenVR includes timers. What I do is use a while loop and adjust my animation inside the loop ala
“`
vtkNew iren;
…
iren->Initialize();
renderWindow->Render();
while (!iren->GetDone())
{
// adjust settings etc, then call
iren->DoOneEvent(renderWindow, renderer);
}
“`
Although iren->GetDone() and iren->DoOneEvent() are unavailable in python code for some reason
Putting animation within while loop and re-rendering renderWindows worked just fine!
Thanks a lot for your help!
Hi Ken,
Sorry for excessive questions 🙁 But this feature lacks much documentation so its really hard to figure out few things
Is it possible in python to turn off objectfactory replacement mechanism and explicitly call vtkOpenVRRenderer (and related modules) ?
On building vtk with open vr module but turning off objectfactory replacement mechanism in cmake
when i try: ren = vtk.vtkOpenVRRenderer()
I get: AttributeError: module ‘vtk’ has no attribute ‘vtkOpenVRRenderer’
Curious to know if explicitly calling openvr modules in python is possible or do we have to rely on automatic replacement ?
Also, one downside to auto replacement of modules is that you need two vtk packages, one for normal rendering and one for open vr rendering
So maybe some flexibility here would be great 🙂
Thanks,
Aditya
The CMakeLists file for Rendering\OpenVR excludes those classes from wrapping in Python which is why you cannot create them directly. You can try removing that exclusion, rebuilding, and it might work fine. I haven’t tried it as I do most my work in C++ but it should work. Give it a shot and let us know.
I will try that and post the results here. Thanks!
Hello! I built the latest version of VTK with Python Wrapping (also latest OpenVR version). Is there any sample/testing code out there for those of us who are begginers in OpenVR and would like to start using this toolkit?
Thanks a lot for the help!
Hi Manuel ,
vtkRenderingOpenVR function is not available for vtk python.
It is great that you have built python wrapping for OpenVR version. It would be great helpful if you share your work.
Hi ken,
vtkRenderingOpenVR for python is not available.
If it is possible add to vtkRenderingOpenVR fuctions to python it would be great and helpful.
Thanks,
Sharath