The Point Cloud Library (PCL)  is a well known and versatile open-source C++ library for working with point cloud data, with functionality for keypoint extraction, alignment, segmentation and much more. We have previously developed an open source PCL plugin to bring a subset of this functionality into ParaView  via support for conversion of some XYZ point clouds and the implementation of several PCL filters. Today we are pleased to announce a new and completely refactored version of the plugin that extends support beyond XYZ point clouds to all PCL point types, thereby making it possible to create arbitrary PCL-based workflows within ParaView. This update provides much greater flexibility for ParaView users while also making it trivial for developers to quickly create new filters to wrap arbitrary PCL functionality.

Workflow Example

The following short video illustrates the ease of creating a PCL workflow in ParaView in which the point clouds from two Kinect cameras are automatically aligned.

The cameras generate point cloud data that is accessed via PCL’s wrapping of the OpenNI library. Because the cameras are approximately 2 meters apart and view the scene from different angles, the resulting point clouds need to be aligned along their overlapping points. This is achieved by first downsampling each cloud with PCL’s voxel grid filter to reduce noise and calculation complexity. The surface normals are then estimated with PCL’s normal estimation filter and examined with ParaView’s built-in glyph filter to check their quality. These normals are in turn used to calculate Fast Point Feature Histograms (FPFH) descriptors for each point using yet another PCL filter. Finally, the down-sampled point clouds and feature vectors are passed to a PCL-based RANSAC filter to calculate a transformation matrix that is then used to align the input clouds directly.

As the example shows, this entire workflow can be created with just a few clicks via the ParaView user interface, even though it involves several different types of PCL data. Due to their modularity, it is just as easy to connect the plugin’s filters in other ways to create other PCL workflows. In order to provide such filters, the plugin uses a framework to facilitate their development, which we discuss next.

Framework Overview

The plugin provides a basic framework for creating new filters as outlined in the following figure.  The point cloud conversions are handled at the start and end of pure PCL code so that filter developers can focus on the PCL algorithm without worrying about how to interact with ParaView. As we will show in detail in another blog post, developing a new filter is usually as simple as copying 3 files, modifying some boilerplate code and then implementing the PCL algorithm.

These filters are chained together via the ParaView user interface to create PCL workflows, as illustrated in the figure below. The transparent conversion of PCL point clouds with preservation of all attributes effectively creates an unbroken internal PCL data pipeline that can be pieced together with considerable flexibility by ParaView users. It also allows the user to take greater advantage of the many tools provided by ParaView.

Advantages Of Working With PCL In ParaView

As we have shown, ParaView’s pipeline browser makes it possible to build entire workflows with just a few clicks. Furthermore, aside from allowing for arbitrary complexity, it also makes it trivial to quickly modify workflows to test different approaches. For example, in the alignment example above, the feature descriptors could be easily changed from FPFH to any other type of descriptor by simply swapping filters in ParaView’s pipeline browser.

The point cloud data itself is also fully accessible and can be used in multiple ways. Examples of this are shown in the video via the colorization of the point clouds using the cloud’s RGB data, the visualization of the surface normals from the cloud’s normal components, and the spreadsheet display of the histogram components of the FPFH descriptors. Providing modular filters and full data access thus gives the user greater control over their workflow: the user gains the ability to easily inspect data at any point in the pipeline and do whatever they want with it.

Thanks to ParaView’s automatic generation of Python bindings, it is also possible to fully script PCL workflows. The following code is a snippet from the ParaView Python state file generated from the workflow in the video which shows the minimalist code required to configure the pipeline. For example, the FPFH estimation filter on line 32 is created and configured with just 2 commands.

# ----------------------------------------------------------------
# setup the data processing pipelines
# ----------------------------------------------------------------

# create a new 'PCL OpenNI Source'
pCLOpenNISource1 = PCLOpenNISource()
pCLOpenNISource1.DeviceID = '1@62'
pCLOpenNISource1.WithColor = 1

# create a new 'PCL Voxel Grid Filter'
pCLVoxelGridFilter1 = PCLVoxelGridFilter(Input=pCLOpenNISource1)
pCLVoxelGridFilter1.LeafSize = [0.05, 0.05, 0.05]

# create a new 'PCL OpenNI Source'
pCLOpenNISource2 = PCLOpenNISource()
pCLOpenNISource2.DeviceID = '1@63'
pCLOpenNISource2.WithColor = 1

# create a new 'PCL Voxel Grid Filter'
pCLVoxelGridFilter2 = PCLVoxelGridFilter(Input=pCLOpenNISource2)
pCLVoxelGridFilter2.LeafSize = [0.05, 0.05, 0.05]

# create a new 'PCL Normal Estimation Filter'
pCLNormalEstimationFilter2 = PCLNormalEstimationFilter(Input=pCLVoxelGridFilter2)
pCLNormalEstimationFilter2.Radius = 0.25

# create a new 'PCL Normal Estimation Filter'
pCLNormalEstimationFilter1 = PCLNormalEstimationFilter(Input=pCLVoxelGridFilter1)
pCLNormalEstimationFilter1.Radius = 0.25

# create a new 'PCL FPFH Estimation Filter2'
pCLFPFHEstimationFilter21 = PCLFPFHEstimationFilter2(Points=pCLNormalEstimationFilter1,
    Normals=None)
pCLFPFHEstimationFilter21.Radius = 0.4

# create a new 'PCL FPFH Estimation Filter2'
pCLFPFHEstimationFilter22 = PCLFPFHEstimationFilter2(Points=pCLNormalEstimationFilter2,
    Normals=None)
pCLFPFHEstimationFilter22.Radius = 0.4

# create a new 'PCL-based Sample Consensus Model Registration Filter'
pCLbasedSampleConsensusModelRegistrationFilter1 = PCLbasedSampleConsensusModelRegistrationFilter(SourcePoints=pCLVoxelGridFilter1,
    SourceFeatures=pCLFPFHEstimationFilter21,
    TargetPoints=pCLVoxelGridFilter2,
    TargetFeatures=pCLFPFHEstimationFilter22)
pCLbasedSampleConsensusModelRegistrationFilter1.MaxIterations = 25000

# create a new 'Glyph'
glyph2 = Glyph(Input=pCLNormalEstimationFilter1,
    GlyphType='Arrow')
glyph2.OrientationArray = ['POINTS', 'Normal']
glyph2.ScaleArray = ['POINTS', 'Normal']
glyph2.ScaleFactor = 0.35
glyph2.GlyphTransform = 'Transform2'

# create a new 'Glyph'
glyph1 = Glyph(Input=pCLNormalEstimationFilter2,
    GlyphType='Arrow')
glyph1.OrientationArray = ['POINTS', 'Normal']
glyph1.ScaleArray = ['POINTS', 'Normal']
glyph1.ScaleFactor = 0.35
glyph1.GlyphTransform = 'Transform2'<br>

ParaView’s Python scripting allows the user to automate virtually anything that can be done directly via the ParaView gui, such as comparing the use of different PCL descriptors in a given pipeline or exploring the full parameter space of a PCL workflow. This is particularly useful for optimizing workflows, dealing with recurrent tasks and batch-processing large amounts of input data. Of course, Python scripting can also integrate functionality from the wider Python ecosystem for nearly endless possibilities.

Combined Filters

While the alignment example in this post uses multiple PCL filters to highlight their modularity and the previous section discusses their advantages, it is also possible to encapsulate a common sequence of steps within a single filter for convenience and/or speed. For example, all steps following the voxel grid filter in the example workflow (normal estimation, FPFH descriptor calculation and point cloud registration) can be combined into a single rigid alignment filter as shown on the right in the ParaView pipeline browser below. The previous pipeline is displayed on the left for comparison.

Internally, this filter follows the same steps as the separate filters with the same parameters, which are all exposed via its ParaView properties panel shown below. Although the voxel grid filter was omitted in this case, it could have been easily included in the combined filter as well.

Conclusion

These are just some examples of what is possible with this latest update.  While there are currently only about 15 filters (including ParaView sources, readers and writers), this PCL Plugin is already actively used for multiple applications in live Lidar point cloud processing within VeloView. Examples include robot-arm guiding, multi-sensor auto calibration and automatic object detection.

The project will continue to add filters with the goal of obtaining a one-to-one correspondence to most of the functionality in PCL. Several combined filters will likely also be added to simplify common workflows. The updated framework facilitates this and the extended point-type conversion support means that all point clouds can be passed between filters for considerable modularity and flexibility.

In the next blog post, we will provide a step-by-step guide to walk developers through the creation of new PCL filters for this plugin. Interested users and developers can find a link to the source code repository below.

Links

• Source code repository.