Taking ParaView into Virtual Reality
(updated September 2018)
In an earlier blog post, we talked about using the Visualization Toolkit (VTK) and the Vive to explore your data in Virtual Reality (VR). In this post, we are going to talk about the Virtual Reality capabilities in ParaView with support for the Oculus Rift, HTC Vive and Windows Enhanced Reality headsets.
Before we dig into how to get your data into VR, we really want to thank the National Institutes of Health, who sponsored the work under the grant NIHR01EB014955 “Accelerating Community-Driven Medical Innovation with VTK,” our commercial customers, and our collaborators at the Oak Ridge Leadership Computing Facility (OLCF).
Getting Some Hardware
ParaView will work with any headset supported by the OpenVR standard. This includes the Oculus Rift, HTC Vive, and Windows Enhanced Reality headsets. Currently, ParaView’s support for the HTC Vive/Vive Pro is better, so if you are undecided, have your purchasing department buy that first. You can purchase the Vive from Amazon or directly from HTC; as of this writing, it runs about $500. If you want to go the Oculus Rift route, you can buy one directly from Oculus or from some retailers such as Best Buy.
A good VR experience requires very high frame rates, typically 90 Hz per eye. This means ParaView needs to render very quickly. Having a high-end CPU/GPU is critical. Both the Rift and Vive suggest at least a GTX 970 class GPU, but the higher, the better. Both vendors sell or recommend systems that are Vive-ready PCs or Oculus-ready PCs. Likewise, both vendors have tools to check if your current system is VR ready for the Vive or Oculus.
Once you have purchased a headset, make sure that it is installed per the manufacturer’s instructions and that it works with their demo software. For all headsets you also need to download and install SteamVR as we use the OpenVR API as the interface.
VR support in ParaView
In the past we required specific VR builds for ParaView but as of mid 2018 ParaView comes with support for VR built in. It is in source tree and is included in the windows binaries that you can obtain from here. It is designed to run under a 64-bit version of Microsoft Windows such as Windows 10. The OpenVR support is a plugin in ParaView. To use it you must first load it from the Tools -> Manage Plugins menu option. Select OpenVR and load it. This will add a OpenVR specific panel to your user interface.
Before running this version of ParaView, make sure Steam VR is running. Once installed, you can then run ParaView by double clicking on the executable. ParaView will start as usual with a window on the computer screen. Using the regular mouse/keyboard, you can load or create your visualization as you normally would in ParaView. ParaView also has a full tutorial you can download.
Once you have your visualization up in ParaView, you can send it to VR by hitting the “Send to OpenVR” button in the OpenVR plugin panel (note the image below is out of date).
You can then interact with the data using the two touch controllers. Pulling the trigger will grab an object, allowing you to hold it and move it around. If you hold both grip buttons down at the same time, you can scale and translate the virtual world. Pulling your hands apart will make it larger while moving them closer together makes it smaller. Moving your hands in the same direction translates the world in that direction. Pressing down on the touch-pad or joystick will start flying in the direction the controller is pointing.
When you are done in VR, press the menu button on a controller (application menu for Vive, A button for Oculus) to return to ParaView. Now you can make more changes to your data in ParaView, and when you are done, hit the “Send to OpenVR” button to return to VR. This way, you can alternate between exploring your data on the screen and in VR.
Note that animations may not work in VR and are still in development. Many annotations will not show up. Finally, it does not support remote rendering of any kind due to the high frame rates required.
Recent additions to VTK that make this all work
Under the hood, ParaView is using the new OpenVR rendering module in VTK. This module can be found in the Rendering sub-directory and can be turned on when compiling VTK using CMake options. When VTK is built with these modules, it includes new subclasses of vtkOpenGLRenderWindow, vtkOpenGLRenderer, vtkOpenGLCamera, and vtkRenderWindowInteractor to support rendering and interacting with the OpenVR. In addition, there are some new classes in RenderingCore designed to help with picking and interaction, when you have devices that support 3D positions and orientations such as the Vive and Oculus controllers. vtkRenderWindowInteractor3D extends many of the methods in vtkRenderWindowInteractor to include a Z coordinate and allow for event coordinates to be in world coordinates in additional to window coordinates. vtkPropPicker3D is designed to perform a pick using a 3D location as opposed to the more common X, Y screen location. Finally, vtkInteractorStyle3D provides the basic interaction style that the Vive controllers use to select and move objects or translate and scale the world.
If you have built VTK with OpenVR support, then creating an immersive visualization follows the same model as existing VTK visualizations. A simple cone example is shown below.
#include "vtkActor.h"
#include "vtkConeSource.h"
#include "vtkNew.h"
#include "vtkOpenVRCamera.h"
#include "vtkOpenVRRenderer.h"
#include "vtkOpenVRRenderWindow.h"
#include "vtkOpenVRRenderWindowInteractor.h"
#include "vtkPolyDataMapper.h"
int main(int argc, char *argv[])
{
vtkNew<vtkActor> actor;
vtkNew<vtkOpenVRRenderer> renderer;
vtkNew<vtkOpenVRRenderWindow> renderWindow;
vtkNew<vtkOpenVRRenderWindowInteractor> iren;
vtkNew<vtkOpenVRCamera> cam;
renderWindow->AddRenderer(renderer.Get());
renderer->AddActor(actor.Get());
iren->SetRenderWindow(renderWindow.Get());
renderer->SetActiveCamera(cam.Get());
vtkNew<vtkConeSource> cone;
vtkNew<vtkPolyDataMapper> mapper;
mapper->SetInputConnection(cone->GetOutputPort());
actor->SetMapper(mapper.Get());
renderer->ResetCamera();
renderWindow->Render();
iren->Start();
return 0;
}
Future Directions
Since this original post support for 3D widgets in VR has been added as well as support for volume rendering and numerous interactions in VR. In the future we plan to continue developing these capabilities including collaboration. ParaView has a number of 2D annotations and rendering options that would fit well on a popup sheet in VR.
If you have an application or technology area that could benefit from visualizations in VR, feel free to contact us. There are a lot of neat opportunities in this space, and we are excited to see where it goes.
+100 well done
With the Vive can you move around in the virtual space?
Most definitely. Your head position and orientation are tracked for starters. But beyond that, using the two Vive handheld controllers you can scale and translate the world as well as fly around.
It works really well! Would be great if we can add animations
I do not have either headset, but, conceptually, would the same work with a VR Cave that has tracking? Normally, we use ParaViews VR Plugin, but this “Send to Open VR” looks like it could work with caves as well? This may allow to prepare complex scenes, while VR users look at the previous scene until you press “Send” again. Just a thought…
+1 for animations
Great idea, Axel; my University has a CAVE and we’d love to have this capability.
I believe ParaView has a CaveVR option built in (or in a plugin). The design is a little different but I believe it is in use. I’m not sure how up to date this section is but it talks about it some.
http://www.paraview.org/Wiki/ParaView/Users_Guide/CAVE_Display
Very interesting job. It opens a wide bunch of possibilities. Is there any plan to release a full version of ParaView for Windows (i.e. with Python scripting functionnalities)? And also a version with animations?
I am working on a new paraview release with improved VR support and some additional features. I’ll try to make sure I build it with python scripting included. I do hope to get the changes merged into paraview master at some point as well.
Note that If you are running Paraview with the Vive, it will only work if the SteamVR Room Setup is configured in “Room-Scale” mode. At least for me “Standing only” mode did not work.
hello! I am working on showing the processing result of openfoam to VR, and the biggest issue is exporting the 3D animation. Did the file which exported by “foamToVTK” contains all the frames from openfoam? thanks!
Hello !
Everything works good for me, thanks Kitware !
Does anyone know if this is possible to see ‘animated results’ (like moving flows) with the HTC Vive ?
Actually this is impossible, you can only see static view…
Thanks guys !
why it is impossible, can you give me advice, i am working on it.. thank you!
Animation is possible, but the logic for connecting the animation controls in ParaView to what is rendered in VR is not there. So it is technically possible once someone writes the code etc.
Hi all,
For those of you asking for animation, we have a demo with some working now, but there are a few tricks:
1. You can’t call vtkOpenVRRenderWindowInteractor::Start yet, but you can
vtkOpenVRRenderWindowInteractor* ri;
while (1) {
ri_>DoOneEvent(renderWindow, renderer);
// Now update to the next frame of your animation.
}
2. We make things more interactive by adding actors for each animation frame and then turning their visibility on/off inside the loop above.
3. You may want to disable visibility culling if rendering speed is an issue.
Hopefully we’ll be able to release a demo sometime soon.
Have you managed to get that to work? I’m very interested in getting animations working in vr!
Yes animations work … but the ParaView GUI is pretty slow for animations (even without VR) so while animation works, for some cases it will be too slow to be useful. Make sure to cache the data (an option in PV properties) and do your animation as a sequence, not real time. I have forwarded the performance issue to the core PV team to look at if they get a chance so hopefully it gets better.
Hi All,
may I ask do we have a plan to support PS VR? If possible, would you please point me to some PV documentation that I can try and test PV with PS VR? Thanks!
You might give the following a try
https://www.vrheads.com/how-play-steam-games-playstation-vr
It looks like it enables PS VR support for OpenVR. Since PV uses OpenVR that might do the trick.
Thanks Ken!
I am just curious about the Linux support. Do you have plan for it? Can i install VR enabled ParaView from source?
The OpenVR changes are now merged into the main ParaView git tree so you can build it from source. I know one person has tried it on Linux and it compiles, but doesn’t seem to run correctly yet.
That is great news! At least, there are some progress in that direction. Do you have any time frame to have working VR enabled ParaView on Linux environment. I am looking forward to test it in earth system science domain. I think that it would facilitate better understanding of complex processes such as storm evolution, air-sea interaction etc.
Some success from David Thompson on building for Linux. Here are his comments
Success on x86_64 Ubuntu 16.04.2 (with some hacks, described below for the record)!
Also, at least on my machine, the VTK OpenVR dashboard appears right-side up, but the y location of the pointer appears to be flipped (i.e., aiming at the top of the dashboard highlights the bottom row of buttons; aiming near the bottom of the dashboard highlights the top row of buttons). Maybe some transforms have changed from the release to the beta I’m running?
David
# Using the SteamVR beta with PV on Linux
1. After installing SteamVR, and opting into the beta I could not start SteamVR from the desktop (it opened a dialog claiming that a download had to complete, but it lies). However, I could start it from the command line with
~/.local/share/Steam/ubuntu12_32/steam-runtime/run.sh ~/.local/share/Steam/steamapps/common/SteamVR/tools/bin/vrmonitor.sh
2. But first (before running vrmonitor, anyway), I had to create symlinks to some vulkan and udev libraries included with SteamVR that were not in the system search path. This is a hack, but it works for now:
sudo -s # i.e., run the following as root
cd /usr/lib/x86_64-linux-gnu
ln -s ~/.local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libvulkan.so.1.0.39 .
ln -s libvulkan.so.1.0.39 libvulkan.so.1.0
ln -s libvulkan.so.1.0.39 libvulkan.so.1
ln -s libvulkan.so.1.0.39 libvulkan.so
ln -s ~/.local/share/Steam/ubuntu12_32/steam-runtime/amd64/lib/x86_64-linux-gnu/libudev.so.0 .
ldconfig
There is a new Paraview 5.3 based executable with some new features (such as a dashboard, floor) for OpenVR/Vive that can be downloaded from the ParaView 5.3 Experimental section download page here http://www.paraview.org/download/
Hello, is there any way of creating a VR menu using Paraview VR, so the user can directly select options from there and interact with the objects? Or is it any development in that area?
Animations and VR menus are both implemented and hopefully will be pushed out in a new version in the next couple weeks. We also hope to get a nightly builder of ParaView setup that includes OpenVR support.
Is there a release with Animations out yet at this moment? Very excited to try out some animations!
Hello Ken! I have tested version ParaView 5.4.1 VR, and it works great! I want to thank you and Kitware for this new release. Do you have any roadmap for the VR version of ParaView? For example, do you plan to support temporal simulation in VR, or develop interactive filter (such as dynamic volume clipping)? Thank again.
Thanks for the feedback Guillaume! I do not have a formal roadmap as most of the work is driven by customer requirements. But in my local copy there are a number of improvements to cropping, menus, and bug fixes for issues exposed by allowing ParaView and OpenVR to be running at the same time. Hopefully those fixes will get merged to master soon.
Any type of animation in PV should also work in OpenVR now, but there are some performance issues to be aware of. You definitely want to set PV to sequence mode instead of real time and make sure the animation cache settings are large enough to hold all the time steps. Even then some limits in PVs desktop animation handling can cause issues with the OpenVR performance. Something I would like to address.
I have done very little testing with volumes as my customers use surface based datasets so that code may be a bit rough right now.
Hi Ken,
First of all, also many thanks for this great work on the VR topic. We have played with versin 5.4.1-600-… and found it quiet usable. The new interaction with the two flightsticks of the HTC Vive is very nice and has way more features than what we have hacked together ourselves. I don’t have any specific wishes other than that I also miss animation performance (already in normal PV GUI, not only in VR).
Anyway, just wanted to let you know that this is very useful stuff to us. Can’t wait for some initial version merged into the regular release.
Best Regards,
Axel
Very interesting post. Thanks for sharing.
Hello everyone,
are newer VR headsets like the HTC Vive Pro or the Windows-MR-Headsets supported?
Regards,
Moritz
We have tested Windows-MR headsets and using them through steam they seem to work fine.
I have not picked up a Vive Pro so I am not sure but I would be surprised if it did not work.
Hello Ken,
which of the Windows-MR Headsets have been tested? In theory they are all identical, but I would like to rule out every possible source of error before making the investment.
Thanks in advance!
Moritz
We have tested with Acer and HP headsets, both works equally well. Current Windows MR headsets are almost identical, the main difference is how the HMD fits on your head.
Hello,
I hope you are having a good day. OpenVR is available for ParaView-5.3.0, is there a way to have it on the latest version? and I am wondering how would I be able to use it on Linux Desktops. Thank you in advance for your help.
Best regards,
Ehsan
I can no longer see experimental downloads or any VR version. Could somebody point me in the right direction.
Thanks
OpenVR support is now part of the normal windows binaries for ParaView, both the latest release and the nightlies. It is a plugin you can find under the tools menu -> plugins -> openvr
Hi I’m new to Paraview but not new to VR, I have a Oculus rift set, I can load the model in VR but the left & right commands do not work properly. Any help will be super appreciated.
Hello Paraview Team,
I just wanted to know if the animations are supported in Paraview VR? I could visualize Static Datasets on my VR Headset but while starting the animation, everything disappears.
Please Help!
Regards,
Arnav
Animation is on the agenda for the next couple months. Hopefully soon we will have it working well.
hi, have animation been supported yet? now I use 5.9.0, it seems not working
I worked for me using Meta quest 2. I was amazing to see 3 dimensional CFD post in virtual environment. Good job paraview team.