A TriLab Success Story: GitLab CI and Runner updates

As we previously discussed, the Software Solutions team is enthusiastic about contributing to the “Tri-Lab Remote Computing Enablement (RCE) Continuous Integration (CI)” initiative. This initiative involves setting up a network of CI testing across Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories, with the supercomputer at these sites registered as a runner on each lab’s individual GitLab instance. Our primary focus for this complex undertaking is to minimize the maintenance and upkeep required to ensure the smooth operation of all runners. In the following paragraphs, we will describe how we accomplished this goal.

Runner CI token expiration

When a GitLab-Runner instance registers with a GitLab, it acquires a token that enables the runner to receive jobs and communicate with the GitLab server indefinitely. The TriLab team requested a feature to impose limits on the lifetime of each type of runner token and allow customization of the limit. Now, an administrator can use natural language to define the token’s duration, which is available for each type of runner. However, the token expiration does not require additional work for maintainers as the runners will automatically renew the token provided by GitLab. When a runner’s token is about to expire, it will request a new token with a new expiration date from the GitLab instance. This feature is available in GitLab-runner version 15.3 or newer. You can see the issue and related merge requests for this feature here.

User Maintenance

The maintenance of the users who can access these cross-site CI runners is just as important as the maintenance of the runners themselves. Although each laboratory has a security umbrella, several reasons exist to deactivate a user. That deactivation must be recognized at three sites instead, not just locally. The Software Solutions team has made several updates to GitLab to enhance the security around user accounts leaving a GitLab instance.

Variable period for user deactivation

One aspect of user maintenance involves monitoring account activity. GitLab already had a feature that deactivated accounts after 90 days of inactivity, but we submitted and merged an MR to allow administrators to set the period to any length. In our case, we set it to 30 days to align with our security plan. This work was accepted and merged into GitLab 15.4, but later updated to have a lower limit of 90 days by GitLab.To ensure compliance with the 30-day limit on each account, the Labs have implemented alternative enforcement methods. For more information on this feature, please refer to the related issue. 

Customizable messages on deactivations

When a user’s account is deactivated, they receive an email explaining the deactivation and providing steps to reactivate the account. With GitLab 15.9, administrators can now include custom text in the email to provide site-specific instructions. See the issue and related merge requests for this feature here.

Identities in the job information

The user initiating the CI pipeline or job must be authorized to execute processes on the system that will handle the job. The TriLab project has addressed this issue by incorporating additional information, such as the list of the triggering user’s external identities, into the JSON web token for each CI job. This allows the custom executor, Jacamar, to access the user’s identity information and authenticate that the user has the required permission to perform the tasks. This feature is deemed critical by the project as it is essential to guarantee that the individual initiating the pipeline has authorized access to the computing resources they are about to use. After several rounds of discussions with GitLab developers and management, our team developed a solution that adds this information to the details of each CI job to the token. This feature is now available in the GitLab 16.0 release, launched in May 2023. Refer to the issue and related merge requests for this feature here.

We want to thank TriLabs again for providing this opportunity to Kitware. At Kitware, we have expertise with a complete software development lifecycle that includes setting up and customizing GitLab CI for your code. Contact us at kitware@kitware.com if you have any questions or need help.

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.