Custom Python interactor styles for CAVEInteraction plugin

This is the first in a two-part series of posts describing recent improvements to the CAVEInteraction plugin.  As it’s the first, some introduction and background about the plugin is warranted.

CAVEInteraction Plugin

The CAVEInteraction plugin has been included with ParaView for many years, although prior to ParaView 5.11.0 it was named VRPlugin.  The name was changed to better reflect the purpose of the plugin, which is to provide support for dynamic user interaction in multi-display immersive environments often called CAVEs.  The CAVEInteraction plugin is designed to allow you to easily configure as inputs the events produced by the trackers and controllers available in your CAVE, whether you have a state-of-the-art tracking system or are using repurposed hardware from an older VR setup. It supports events provided through either VRPN or VRUI event libraries.  Once configured, the plugin allows you to map those events to handlers that manipulate ParaView proxies to do, well, whatever you want!

Over the past two years, Kitware has collaborated with the National Institute of Standards and Technology (NIST) to bring a lot of improvements to the CAVEInteraction plugin.  Improved head tracking, proper stereographic rendering, better navigation capability along with support for placing scene objects in fixed and navigable coordinate systems are just a few of the fixes and features that have landed in the CAVEInteraction plugin during this time.  Two other big changes that have been completed are 1). support for custom Python interactor styles and 2). collaboration between users of CAVEs and users of the XRInterface plugin.  These features will be the focus of this blog series, starting with this post on custom Python interactor styles.

CAVE Python Interactor Styles

The CAVEInteraction plugin defines an event handling interface with essentially three functions:

HandleTracker()
HandleButton()
HandleValuator()

Supporting both VRUI and VRPN protocols for providing VR events, the plugin initially had interactor styles hard-coded in C++ that implemented the above interface and had the capability to modify a designated ParaView proxy’s properties. 

As an example, the simplest of these interactor styles was (and still is) named “Track” and implemented only the HandleTracker() method.  The Track interactor provides head tracking in the CAVE by simply applying the incoming tracker matrix to the EyeTransformMatrix property of ParaView’s active RenderView proxy.

Other interactor styles were designed a little more generically, however, and can be associated with not only the RenderView proxy, but also any source proxy in the pipeline.  Not only that, but they can also be configured to operate on any property of the selected proxy, as long as the property has the matching value type expected by the interactor style (e.g. single element, 3-element vector, or 16-element matrix).

The functionality is available in ParaView binary downloads, which included the CAVEInteration plugin, and there are several different interactor styles included.  Some allow the user to grab the scene and move it around in different ways, others support various ways of navigating through a dataset.  However, some drawbacks of this arrangement are that new interactor styles have to be committed to the ParaView repository and any changes to the functionality require the CAVEInteraction plugin to be recompiled.

Kitware’s recent collaborative work with NIST has improved the situation by supporting the development of custom Python interactor styles. Python interactor styles implement the same interface for handling tracker, button, and valuator events, but have the ability to manipulate any number of ParaView proxies and properties.  Python interactor styles have several other nice advantages over their built-in C++ counterparts as well.  The most obvious is that they support rapid prototyping, allowing the developer to do anything they might do in a ParaView Python script, without any need to recompile ParaView during the development process.  Another nice benefit is that users and CAVE installation sites can maintain a collection of useful interactor styles under source control separate from the ParaView code repository.

Before we dive in to see how this works in more detail, it might make sense to have a read through the new documentation of the CAVEInteraction plugin to learn more about the concepts involved in CAVE interaction, defining connections to VRPN and VRUI event systems, CAVE coordinate systems, etc.  Once you have a better understanding, it could be time to look at a very basic interactor style defined in Python, a head tracker:

from paraview.simple import *
from paraview.incubator.pythoninteractorbase import PythonInteractorBase

def create_interactor_style():
   return HeadTracker()

class HeadTracker(PythonInteractorBase):

   def Initialize(self, vtkSelf):
       vtkSelf.ClearAllRoles()
       vtkSelf.AddTrackerRole("HeadTracker")

   def HandleTracker(self, vtkSelf, role, sensor, matrix):
       if role == "HeadTracker":
           renderViewProxy = GetActiveView()
           renderViewProxy.SetPropertyWithName("EyeTransformMatrix", matrix)
           renderViewProxy.UpdateVTKObjects()

Although a pretty small amount of code, it still bears closer examination to understand everything that’s going on.  The first line is likely recognizable to anyone who has written any Python code to interact with ParaView. It imports ParaView’s simple Python scripting module needed for all the ParaView-specific calls such as GetActiveView().  The next line imports a relatively new module that provides a base class from which we’ll derive our tracker class.  The PythonInteractorBase class provides empty implementations of the interface methods, allowing subclasses to implement only particular methods of interest.

The create_interactor_style() method is important in that every custom Python interactor style must implement this method to return an instance of the Python object that implements the handler methods.  The ParaView application expects this method to exist, and calls it to instantiate the Python object to which it will forward all incoming tracker, button, and valuator events.

The HeadTracker class is our main class of interest in the above example.  It implements an Initialize() method as well as HandleTracker().  

The Initialize() method takes the normal self parameter you expect to see in Python class methods, along with a vtkSelf parameter, which is actually a reference to the C++ object that manages the Python object (which is itself a subclass of vtkSMVRInteractorStyleProxy).  We use this parameter to declare any named roles needed by the interactor style; see the documentation on defining interactions for more information.  In this case, we just declare that we need a single 6-DOF position tracker (hereafter referred to as just a “tracker”), and we distinguish it from any other potential trackers by associating it with the named role “HeadTracker”.  The most important part of this is that when we declare the need for a tracker and assign it the role “HeadTracker”, ParaView uses the information to show us a dialog in the user interface that allows us to choose which of our configured tracker events should be bound to the role “HeadTracker” (the dialog below is displayed when we choose the Python file shown above as the script for a Python interactor style)..

The HandleTracker() method then takes 5 parameters, the first two of which are the same as for Initialize().  The third parameter is the role, which is provided to allow us to distinguish between the various trackers we may be using in our application.  The fourth and fifth parameters are the sensor id and actual sensor values, an array of 16 elements representing the tracker matrix.  With the tracker matrix array in hand, we can simply apply it to the EyeTransformMatrix property on the active RenderView proxy, and our custom Python head tracker is complete.

Now you may be wondering how useful this is, since the CAVEInteraction plugin comes with a built-in head tracker which is just as good.  This was just a simple example, but it illustrates the basic form of Python interactor styles and you could start to build more complex interactions from these simple beginnings. 

Before we wrap up this post, we want to leave you with a performance tip: As you may discover when you begin to experiment, using a large number of interactor styles can tend to degrade overall performance if each one updates one or more ParaView proxies.  In that case, combining multiple interactor styles, e.g. flying, grabbing, and head tracking, into a single interactor style that does all three, can significantly increase the achievable frame rate. 

The new python interactor documentation includes a link to another example Python interactor style in the ParaView repository that implements navigation.  You simply point your controller in the direction you want to fly, and squeeze the trigger to control your flying speed.  If your controller doesn’t have a trigger, simply bind the flight speed role to any other valuator your controller does have.

Acknowledgements

This work was funded by the National Institute of Standards and Technology.