Mobile Application Development: VES Library Insights

According to a report published by the International Telecommunication Union, there were six billion mobile subscriptions at the end of 2011 (see Figure 1). Also, in 2011, sales of smartphone surged to 472 million units. A smartphone is a mobile phone built on a mobile computing platform, with more advanced computing ability and connectivity than a feature phone.  The new generation of smartphones offer various multimedia capabilities including interactive games and applications.  This is become possible because of the recent improvements in mobile hardware and software. These significant improvements also enabled scientific community to use mobile devices for visualization purposes.  For example, Zhou et al [3] has investigated how to achieve volume visualization on mobile devices; Tesch et al [4] developed a system that uses mobile devices for interacting and exchanging data between heterogeneous scientific visualization systems; and Roudaut et al [5] presented their work on interaction techniques for mobile devices. Though technically impressive, none of these offer a general, extensible, and open-source framework for enabling developers to build high performance visualization applications on mobile devices.

Figure 1: Unstructured Mesh for YF-17 Visualized using VTK and VES Library

Figure 2: Growth of Mobile Subscriptions

The VES framework [11] has been developed to address these limitations. It puts powerful tools at the fingertips of developers to enable the building of high performance visualization applications on mobile devices. VES is an open-source framework for accelerating mobile visualizations on current generation mobile hardware. VES consists of two core libraries: VES and Kiwi. The VES library provides rendering capabilities and infrastructure for scene management by utilizing OpenGL ES 2.0 [8] whereas the Kiwi library provides an application framework and is built on top of VES and VTK [6]. In this article, we will discuss the architecture and various components of the VES library along with code snippets to explain the API offered by the VES library.

Figure 3: Architecture of VES

Architecture & Implementation
As previously mentioned, the VES library provides scene management and rendering functionalities to the applications developed using the VES framework. It uses OpenGL ES 2.0 as the underlying rendering API, which is a subset of OpenGL for desktop and replaces the fixed function transformations and fragment pipeline of OpenGL 1.x. OpenGL does not provide a high-level, object-oriented API for the application development, which the VES library provides to reduce time and effort to build visualization applications.

Figure 4: Components of VES Library

As shown in Figure 4, the VES library is composed of multiple components, each of which offers a unique feature within the library. The next few sections will cover some of these components and the underlying technology in detail.

Scene Graph
The VES library uses scene graph data structures to manage scenes efficiently. A scene graph is a data structure that provides spatial and logical relationships between various entities of a scene. A scene graph can be implemented in many ways and some of the open source implementations of a scene graph are inspired by the design of OpenGL Performer, one of the well-known scene graph libraries from SGI [7]. The VES library is built using the same core principles and additionally provides a consistent, easy-to-use API to allow applications to take advantage of programmable pipeline functionality of OpenGL ES 2.0. Figure 5 shows the inheritance hierarchy in the VES scene graph.

Figure 5: Inheritance Hierarchy in VES Scene Graph

Rendering Pipeline
Rendering in the VES library is a three-pass algorithm as shown in Figure 6. VES renders the scene by traversing it in a depth-first manner. At the API level, the VES library separates the geometry data from the appearance of the geometry. The appearance of the geometry is captured by the notion of a material in the VES library.

Figure 6: Rendering Model in VES

This separation facilitates reuse of geometry and/or material if required by an application. Since state change in OpenGL is an expensive operation, VES library constructs and renders a scene tree in which geometry data is grouped by the material. Also, the library provides a notion of bins, where a material and associated geometries with a higher bin number are rendered after the material and associated geometries with a lower bin number.

Programmable Pipeline
OpenGL ES 2.0 replaces the fixed function pipeline of OpenGL 1.x. Figure 7 shows the programmable pipeline of ES 2.0.

Figure 7: OpenGL ES 2.0 Programmable Pipeline (source: khronos.org)

In OpenGL ES 2.0, it is necessary to provide a vertex and a fragment shader in order to render geometry primitives. Vertex shaders can be used for traditional, vertex-based operations such as transforming the position with a matrix, computing the lighting equation to generate a per-vertex color, and generating or transforming the texture coordinates. The fragment shader is a general-purpose method for interacting with fragments. A simple in-source shader to render geometry in the VES library is shown in Listing 1.

// Define vertex shader to be used
// for rendering
const std::string vertexShaderSource =
   “uniform highp mat4 modelViewMatrix;
    uniform highp mat4 projectionMatrix;
    attribute highp vec4 vertexPosition;
    attribute mediump vec4 vertexColor;
    varying mediump vec4 varColor;
    void main()
    {
      gl_Position = projectionMatrix *
        modelViewMatrix * vertexPosition;
      varColor = vertexColor;
    }”;

// Define fragment shader to be used for
// rendering
const std::string fragmentShaderSource =
   “varying mediump vec4 varColor;
    void main()
    {
      gl_FragColor = varColor;
    }”;

Listing 1: Simple Shader to Render a Geometry in the VES Library

The way vertex attributes and uniforms are passed to the pipeline are somewhat different; but due to the fact that they are both inputs to the shader program, it is important to provide a consistent API to bind uniforms and vertex attributes to a shader. In Listing 2, we have shown how to pass uniforms and vertex attributes to the rendering pipeline.

// See listing 1 for vertexShaderSource and
// fragmentShaderSource
   ….
   ….
// Create shader objects   
vesShader::Ptr m_vertexShader
   (new vesShader(vesShader::Vertex));
vesShader::Ptr m_fragmentShader
   (new vesShader(vesShader::Fragment));      
// Set shader sources (from Listing 1)
this->m_vertexShader->setShaderSource
   (vertexShaderSource);
this->m_fragmentShader->setShaderSource
   (fragmentShaderSource);

// Add shader to shader program
this->m_shaderProgram->addShader
   (this->m_vertexShader);
this->m_shaderProgram->addShader
   (this->m_fragmentShader);

// Add uniforms to the shader program
this->m_shaderProgram->addUniform
   (this->m_modelViewUniform);
this->m_shaderProgram->addUniform
   (this->m_projectionUniform);
   
// Add vertex attributes to the shader program
this->m_shaderProgram->addVertexAttribute
   (this->m_positionVertexAttribute,
    vesVertexAttributeKeys::Position);
this->m_shaderProgram->addVertexAttribute
   (this->m_normalVertexAttribute,
    vesVertexAttributeKeys::Normal);
this->m_shaderProgram->addVertexAttribute
   (this->m_colorVertexAttribute,
    vesVertexAttributeKeys::Color);
// Shader program is an attribute of
// the material.
this->m_material->addAttribute
   (this->m_shaderProgram);   

Listing 2: Uniforms and Vertex Attributes Binding  to the Pipeline

In the previous example, we used predefined vertex attributes keys such as Position, Normal, and Color. These keys are provided for convenience (internally defined as enums) and applications are allowed to add a new key if desired. This design makes it convenient and flexible to bind multiple vertex attributes to the shader, where each attribute is identified by its key. The mechanism for linking vertex attributes to the geometry data using a key is explained in the following section on geometry data.

Geometry Data
The VES library provides a very flexible data structure for defining geometry for the purpose of rendering. Some of the highlights of the VES library geometry data are:

• Support for interleaved or separated data arrays
• Any number of coordinate systems for the point data
• Support for different basic types for the point data
• Separation of point data from the cell data
• Extensible data structure

VES geometry data structure is inspired by the collada file format specifications [9] where geometry is composed of one-or-more sources. Source in the VES library defines the per vertex data to be used for rendering or computation purposes. For example, in Listing 3, we have shown a code snippet to define a source data, which stores the vertex positions of a geometry.

// Define vertex data structure
struct vesVertexDataP3f
{
   vesVector3f m_position;
};
// Define source data structure to store
   vertex positions
class vesSourceDataP3f :
  public vesGenericSourceData<vesVertexDataP3f>
{
public:
  vesTypeMacro(vesSourceDataP3f);
  vesSourceDataP3f() :
    vesGenericSourceData<vesVertexDataP3f>()
  {
    const int totalNumberOfFloats = 3;
    const int stride = sizeof(float) *
      totalNumberOfFloats;
    this->setAttributeDataType
      (vesVertexAttributeKeys::Position,
       vesDataType::Float);
    …
    …
  }
};

Listing 3: Source Data to Store Positions

It is to be noted that in the code above,vesGenericSourceData uses vesVertexData3f as the template argument. The VES library provides several such source data types for convenience. Developers can create their own source data type that can use either predefined vesVertexData* types or some other custom vertex data structure. Also in the previous section we mentioned that the vertex attribute binding to the shader uses attribute keys. In order to notify VES library to link a vertex attribute to the vertex source data it is required that they both share the same key value; for example, Position as shown in the example code.

VES library source data structure enables applications to define and use interleaved or separate arrays for each vertex attributes. Interleaved vertex arrays are useful to achieve performance in most situations. Information on how to construct cells from the source data is defined by the vesPrimitive. Just like source data, VES library geometry data can contain one or more instances of vesPrimitive. Listing 4 shows a code snippet to define triangle primitives.

vesPrimitive::Ptr triangles
  (new vesPrimitive());
vesSharedPtr< vesIndices<unsigned short> >
  indices (new vesIndices<unsigned short>());
indices->pushBackIndices(0, 3, 2);
indices->pushBackIndices(1, 0, 2);
triangles->setVesIndices(indices);
triangles->setPrimitiveType
  (vesPrimitiveRenderType::Triangles);
   triangles->setIndexCount(3);
   triangles->setIndicesValueType
  (vesPrimitiveIndicesValueType::
   UnsignedShort);

Listing 4: Creation of vesPrimitive of Type Triangles

VES library enables developers to use unsigned short or unsigned int type for the indices based on the requirement and whether or not the device and the driver supports unsigned int as the index type.

Appearance and State Management
In the VES library, appearance and rendering state of the geometry is defined by the material used by the node of the scene graph. A material is composed of one or more material attributes. For example, shader program and textures are attributes of a material. Similar to a shader program (see Listing 2), a texture can be added to a material. VES provides support for 1D and 2D textures. Below is a list of data formats and types supported by VES library texture:

Internal and pixel format:
Alpha, Luminance, LuminanceAlpha, RGB, and RGBA.

Pixel data type:
PixelDataTypeNone, UnsignedByte, UnsignedShort565,
UnsignedShort4444, UnsignedShort5551.

Control over depth buffer and blending is provided by vesDepth and vesBlend material attributes. The VES library API
makes it easy to add a new material attribute in case the functionally is not provided by the existing attributes.

OpenGL ES Extensions
The VES library provides basic support to query OpenGL ES extensions available on a hardware. For example, using this feature, an application can query for the GL_OES_element_index_uint extension and if available, can use an unsigned int type for the index to support large geometry data. However, current implementations do not support applications acquiring function pointer for the extensions.

FrameBuffer Object (FBO)
Framebuffer Objects are a mechanism for rendering to images other than the OpenGL default Framebuffer. OpenGL introduced FBO to make rendering to texture objects much more efficient and easier to perform when compared with pbuffer alternatives. The VES library enables applications to use the render to texture feature by creating an instance of vesRenderToTexture, setting appropriate parameters for the render target and setting the render target on the camera.  Currently, VES supports attachments of type  GL_COLOR_ATTACHMENT0 and GL_DEPTH_ATTACHMENT.

Interfacing with VTK
One of the strong motivations behind VES development is to enable developers to take advantage of state-of-the-art VTK algorithms on mobile devices. In order to provide an interface similar to VTK, the VES library offers vesActor and vesMapper. It is important to mention that the VES library does not depend on VTK as it is expected that an application that requires VTK support will be developed using the application level API provided by Kiwi.

Results
In Figure 7 we show the performance of VES and Kiwi against different size models.  We measured the rendering performance of the VES library using KiwiViewer [REF 10]  on a  third generation Apple iPad. We measured the interactive frames per second (FPS) by averaging the number of times rendering is triggered by the iPad in a 20 second interval. In  order to get a consistent performance measure, we didn’t include FPS value reported by the application at the start and exit of the application.

Figure 8: Performance Graph of VES Rendering

Conclusions & Future Work
The VES library, along with Kiwi, have been very successful in delivering a high performance application on iOS and Android platforms. Since the time we open sourced the VES and Kiwi libraries, we have received many contributions and much feedback from the community. We will continue to improve the current features of the VES library, and look forward to adding support for new features such as volume rendering and vector text rendering.

Figure 9: Rendering on Qt based MeeGO Platform

Special thanks to Riccardo Vianello for porting VES to the Qt-based MeeGO platform and providing screen captures for this article.

References
[1] International Telecomunicaiton Union (ICT), Facts and Figures,  http://www.itu.int/ITU-D/ict/facts/2011/material/ICTFactsFigures2011.pdf
[2]   Be Mindful of OpenGL ES State Variables: http://developer.apple.com/libraryios/#documentation/3DDrawing/
Conceptual/OpenGLES_ProgrammingGuide/OpenGLES ApplicationDesign/OpenGLESApplicationDesign.html
[3]   Hong Zhou, Huamin Qu, Yingcai Wu, Ming-Yuen Chan. 2006. “Volume Visualization on Mobile Devices”, 
Proceedings of 14th Pacific Conference on Computer Graphics and Applications (Pacific Graphics’06).
[4]  Tesch, J., Dietze, L. & Encarnação, L.M., 2003. Personal Interfaces-To-Go : Mobile devices for Data Exchange and Interaction in Heterogeneous Visualization Environments University for Applied Sciences ( FH ) Mainz. Distributed Computing, p.2-5.
[5]   Roudaut, A., 2009. Visualization and interaction techniques for mobile devices. Proceedings of the 27th international conference extended abstracts on Human factors in computing systems CHI EA 09, p.3153. Available at: http://portal.acm.org/citation.cfm?doid=1520340.1520450.
[6]    Visualization Toolkit (VTK), http://vtk.org
[7]    OpenGL Performer, http://oss.sgi.com/projects/performer
[8]    OpenGL ES, http://www.khronos.org/opengles
[9]    COLLADA, https://collada.org
[10]  KiwiViewer, http://www.kiwiviewer.org
[11]  VES, http://www.vtk.org/Wiki/VES

Aashish Chaudhary is an R&D Engineer on the Scientific Computing team at Kitware. Prior to joining Kitware, he developed  a graphics engine and open-source tools for information and geo-visualization. Some of his interests are software engineering, rendering, and visualization