Spack on Windows: A New Era in Cross-Platform Dependency Management

Ask any room full of C++ developers to identify the most aggravating aspect of their development work and you will hear an endless refrain of “dependency management”. A 2023 ISO C++ developer survey, asking about developer’s primary frustrations with the language, found that over 80% of respondents (the most of any group) identified dependency management as their number one frustration. Moreover, the problem is so pervasive that three of the top five responses following it were also related to dependency management.

This friction between software and its dependencies is perhaps most pronounced on the Windows platform, where the lack of a built-in system package manager and associated standards makes handling third party dependencies particularly challenging. On Unix-like systems we have solutions like APT, Yum, Zypper, and Homebrew establishing conventions on how and where packages should be installed, and by virtue, where packages can be found and loaded. On the other hand, Windows itself offers no consolidated system solution for package management leaving teams, organizations, and individuals in the dark to adopt their own standards and strategies for package management. While package managers do exist on the platform, adoption is splintered as teams balance solutions with added complexity and available features and supported packages. Enter Spack, a package manager robust, powerful, and feature rich enough to get your team moving smoothly.

What is Spack?

Spack was originally developed in 2013 by a team led by Todd Gamblin at Lawrence Livermore National Laboratory. Since then it has evolved into the go-to solution for HPC and scientific communities interested in handling complex packaging problems. As a completely open source package management solution, Spack offers granular control over builds, multi and cross language support, and highly flexible package expression. It has transformed the incredibly complex task of managing supercomputer software stacks from a major headache into a more manageable challenge. Given its effectiveness in such complex environments, its potential utility for Windows becomes evident.

Spack: now on Windows

We are excited to inform you that, as part of Kitware’s ongoing collaboration with LLNL and other DOE labs, over the past few years we have spearheaded the effort to port Spack to the Windows platform. With Spack, Windows users can finally experience managing complex software stacks from a single, simple, highly configurable source.

Let’s say you are building a VTK based application. You will need to install VTK, its third party dependencies, and anything that your project depends on before you can build your own application. Further, you want to test against multiple MPI implementations, with and without hardware acceleration, static and shared builds, with and without debug symbols, and you have team members on multiple platforms, each of which require specific considerations. To achieve these goals with any currently available package manager would be a titanic effort, requiring you to build and rebuild software, outline complex CI jobs, and develop custom solutions to marry all of these dependencies and ecosystems. For many package managers, this is simply impossible. For Spack however, it’s just any other day. A single parameterized Spack package recipe can be used to generate tens of thousands of combinatorial builds with varying compilers, flags, implementations of common interfaces like MPI or GL, and versions of each primary package and its dependencies.

Spack on Windows in GitLab CI

Kitware has been a collaborator in, and major contributor to, the development of Spack for over a decade. One of our primary roles in Spack maintenance has been the multifaceted task of developing and maintaining a complex series of testing workflows and pipelines coordinating GitHub, GitLab, and Amazon Web Services (AWS) for the purpose of continuous verification of Spack package recipe integrity. For more information on this effort, and its structure, check out our Spack infrastructure blog. With the introduction of Spack’s Windows support, Spack’s GitLab CI too had to grow to support this new platform, including the addition of Windows images, runners, GitLab pipelines, and most importantly, the end-to-end testing of Windows packages. Spack package recipes supported on the Windows platform now have the same assurances of stability and reproducibility as they do on other Spack supported platforms. As part of the CI workflow, Spack publishes a public binary mirror from the results of its builds where Windows binaries will soon be found.

Spack brings a number of powerful features to Windows:

• Cross-Platform Consistency: Spack ensures that the software you install behaves consistently across Windows, Linux, and macOS. No more platform-specific considerations, declare a software stack once, and build seamlessly across your teams and ecosystems.
• Performance Boost: Windows users can now benefit from Spack’s efficient re-use of installed packages. By spending less time building dependencies, you now have more time to work on the software that you actually care about.
• Easy Integration: Spack seamlessly integrates with Visual Studio and other Windows development tools, making it a breeze for developers to manage their environments effortlessly.
• Hassle-Free Dependency Management: Spack resolves dependencies using a unique approach. Employing a full SAT solver, Spack is able to optimize more efficient builds with fewer conflicts/intractable states that you may encounter with a more traditional package manager.
• Minimal Tooling: Without Spack, managing dependencies on Windows might take numerous tools, spaces, environments, and ecosystems, all of which need to be tracked, documented, and managed. Spack reduces all of this into a centralized ecosystem. Packages installed even with other package managers can be seamlessly integrated into Spack ecosystems.
• Build system agnostic: Spack doesn’t care whether a dependency you need was packaged with CMake, nmake, VS, or msbuild. With support for all Windows supported build systems, Spack is ready to build your dependencies, regardless of whether you’re familiar with CMake or not.
• ABI Compatibility: Spack provides an ABI compatibility guarantee. How do we do this? Spec Hashes! The hash of your requested package spec, the package recipe itself, and the specs of all dependencies are incorporated into a package’s spec hash. The solver is then free to pick different, compatible hashes to use on install, so you can build on top of older hashes, allowing for a software stack you can build on, without worrying about ABI compatibility.
• Combinatorics: Spack is highly combinatorial. A single package specification can be used to generate hundreds of unique builds, supporting environments where many versions and variants of a package need to be used or tested.
• Build Isolation: Compiler Wrappers intercept calls to all compilers supported by Spack, allowing for Spack to achieve partial isolation from the host system, ensuring produced binaries are reproducible and deterministic. No unwanted system interference with Spack.
• Shared Library Management: Dealing with Windows shared libraries is a unique challenge that developers on the platform are no stranger to. Spack makes this issue a consideration of the past, managing shared libraries transparently, ensuring all dependencies have access to the correct DLLs without conflict or issue.
• Ease of Installation: Installing Spack on Windows only requires a git clone or running a Windows installer.
• Customization Galore: Tailor your software stack with unparalleled granularity. Spack’s package specification and highly configurable settings make it possible for the users to customize nearly every aspect of the software being installed, almost like you were installing it from source yourself.

Summary

With Windows support, Spack has become a true cross-platform package manager capable of building complex software on Linux, Windows, Mac, and HPC systems with a single specification. If you would like to see if Spack is right for your project, see the community here, or contact Kitware.