From One Software to Many at Allegorithmic

Allegorithmic makes applications for texturing. Texturing is a virtual process that helps make a three-dimensional (3D) model appear realistic. With texturing, a 3D model of a wall can look like it is made of concrete. Sébastien Deguy, the CEO of Allegorithmic, describes texturing as “touching with the eyes.”

Four years ago, Allegorithmic launched the development of Substance Painter, which is shown in Figure 1. Substance Painter is an application that has helped Allegorithmic become a leader of the texturing industry. Major game studios now use the application. When Allegorithmic first created Substance Painter, the company only had a small team of software developers. They focused on a tool called Substance Designer. Substance Designer was originally developed in C++, using the Qt framework. It was built with QMake, which comes with Qt. The build process for Substance Designer was tailored over 10 years. Yet, it had a number of problems:

• Code was stored in a single, monolithic Subversion (SVN) repository that had a mix of libraries, in-house tools and application sources.
• Many handcrafted scripts were used to set up environment variables to specify items such as dependency locations, platform types and compilers.
• The build process required third-party libraries, which were manually maintained in a zipped file that each developer had to store on his or her computer. When the process incorporated a new dependency or compiler, the updated zipped file was manually deployed to all of the computers.
• Without a link between the source code and the various versions of third-party libraries, it was difficult to revert to earlier versions of the project build.
• The setup scripts were not cross-platform scripts. They required separate build and packaging processes for Windows and Mac OS X.
• Setting up new workstations—from code checkout to successful build—was a complex procedure that often took more than a day. This procedure heavily relied on detailed knowledge from developers.

Designing a Better Build Process

With the introduction of Substance Painter and the potential introduction of additional applications, Allegorithmic acknowledged that its cumbersome build process was a risk for the company. Looking toward the future, the company chose to form a new, more sustainable build process that could support new applications.

Developers at Allegorithmic wanted to easily check out source code, configure it, compile it and run it on their favorite platforms with their favorite integrated development environments (IDEs). They also wanted to maintain only one build file. Thus, the new build process needed to incorporate a fully reproducible and cross-platform build environment that could support Windows, Mac OS X and Linux without maintaining handcrafted, platform-specific scripts. The build process also needed to produce a stand-alone package that could be distributed without worrying about dependencies. The process had to be customizable, and it had to be able to generate documentation from source code, copy files at build time and use external pre-processing tools. In addition, the process had to easily handle third-party libraries. To accomplish this, the build system had to contain the correct versions of the libraries.

Since the software that Allegorithmic planned to develop would depend on Qt, QMake was initially implemented to achieve these objectives. Unfortunately, QMake did not suit the requirements due to its poor documentation and slow performance. Allegorithmic developers were unable to find a satisfactory way to use QMake to manage dependencies.

Allegorithmic next considered CMake as a candidate. Many of the third-party libraries that the company used were already compiled with CMake, and/or they provided CMake configuration files. Additionally, some of the developers at Allegorithmic had experience with CMake, and they were happy with the results. Allegorithmic found that CMake was able to fulfill all of its requirements. The company successfully implemented a new build process that used CMake.

Managing Third-party Libraries

Managing third-party libraries for a cross-platform project is a nightmare in C++. Each project uses different versions of compilers, different compilation options and different versions of libraries, which makes it complicated to maintain binary packages for each permutation. The solution is to build the libraries as part of a project to guarantee that the binaries are fully compatible with the project. These library sources are directly referenced in the source tree of the project, which ensures the availability of any given version of the project.

To link the library sources to a particular version of Substance Painter, Allegorithmic relies on the ExternalProject function of CMake. This function defines the necessary steps to download, configure, build, install and test external libraries. Allegorithmic only uses the steps to build and install the libraries. It employs Git submodules to download the appropriate versions of third-party sources.

It is not always possible to build third-party libraries as part of the project. Some libraries are proprietary, and they do not have source code available. Others cannot be integrated with the project source tree because of their respective sizes; these large libraries take too long to build. For such libraries, Allegorithmic employs a simple internal tool that allows pre-compiled binaries to be uploaded and downloaded from internal servers. This tool is used within CMake files. It downloads the appropriate version of each library. The tool enables Allegorithmic to directly keep track of binary dependencies in the project source tree as it would any other source dependencies.

As a result of its flexibility, CMake can wrap calls to make their syntax similar to that of ExternalProject. The following call, for example, downloads the appropriate version of Qt for the target platform and adds its location to the path in CMake, where it can be found when needed.

alg_add_external_archive(ALG_QT 

  WIN32 qt-5.7.1-p2 

  APPLE qt-5.7.1-p1-c++11 

  CENTOS qt-5.7.1-centos-p1 

  UBUNTU qt-5.7.1-ubuntu-p1 

  )

ExternalProject can also download pre-compiled binaries through HTTPS or common source control options. The downside is that CMake installs ExternalProject at build time and not at the time that CMake is configured. This timing prevents the use of the find_library() feature in CMake. Accordingly, all third-party management occurs in a separate CMakeLists.txt file, which must be built and configured before the actual project is built and configured.

An alternative is to provide a superbuild whose last ExternalProject step is the real project. This way, ExternalProject downloads and builds the third-party libraries before the project itself, allowing the project to properly reference the libraries.

Managing Dependencies

The target system of CMake manages dependencies. Every executable or library that is built in a CMake project is a target. Each target defines the include paths, compilation options and libraries that CMake needs for a project build. Each target also defines the instructions for any other target dependencies. CMake can treat external libraries as targets, using the IMPORTED target feature. The ability to work only with targets makes CMakeLists.txt files clean and easy to follow.

The CMake approach to dependency management is less prone to error than the previous approach that Allegorithmic used. Since the details of a particular external library are hidden inside of a target definition, developers do not need to worry if they forget a compile flag or an include path. The target system has proved important for Windows builds, and it has helped mitigate the limitations of the original build procedure.

On Windows, unlike on Mac OS X and Linux, it is not possible to specify a file location in the executable for dynamically linked libraries (DLLs). This means that a successfully compiled executable that relies on several DLLs cannot run unless those DLLs are either copied to the same directory as the executable or their locations are added to the PATH environment variable.

Allegorithmic solved this issue by writing its own introspection method, which uses properties on CMake targets. The introspection functions return the paths to the corresponding DLLs. A final post-build step was added to the executable target, which copies the DLLs to the same directory as the executable. While creating multiple copies of each DLL is not ideal, it serves as an acceptable compromise, as symlinks in Windows do not always work properly for certain levels of user permission.

Introspection is also used to generate stand-alone packages for projects with install() in CMake. The only issue with install() is that it cannot be used for IMPORTED targets. So, Allegorithmic had to create a workaround. Instead of directly calling install(), Allegorithmic created a custom command that can introspect a target as described above and generate the appropriate install() for each dependency. Thus, it takes only a single command to install a target and generate a stand-alone installation for that target. Since the install process in CMake is flexible, the custom command allows additional processes to run. Such processes fix RPATH on Mac OS X and Linux or generate symbol information for the crash reporting system.

CMake also provides a BundleUtilities module that can help generate a standalone installation of an executable. Allegorithmic has not tried to use this module yet.

Looking Back at the Move to CMake

The move from QMake to CMake initially faced resistance from some developers at Allegorithmic. In retrospect, it was the correct move. After Allegorithmic developed the framework for Substance Painter, it successfully ported Substance Designer to that framework, which allowed the development team for Substance Designer to grow. The framework also enabled the rapid startup of new products and teams that focus on software rather than on technical details of infrastructure.

Today, Allegorithmic has many software development teams, all of whom use the new framework. Developers are happy with the build system, as they no longer have to bother with the messy details of building and packaging C++ applications. The maintainable framework allows developers to continue to tackle challenges in the build environment, while they dedicate most of their time to improving their software products.

Acknowledgements

Thanks go to Allegorithmic (https://www.allegorithmic.com), which allowed this look at behind-the-scenes work. Thanks also go to colleagues, who worked on the efforts discussed in this article. Finally, thanks go to Stephane Guy and Eric Batut, who reviewed this article.

Alexandre Chassany is a senior software developer at Allegorithmic, where he has worked for seven years. He is highly versatile, and he is passionate about every aspect of software development—from software architecture to release processes. He is active on LinkedIn at https://www.linkedin.com/in/achassany and on Twitter through the handle @achassany.