Secure Credential Management for Artifactory with Vault

Introduction

Do you wonder how to securely manage credentials for a central artifact management system and prevent exploitation of artifacts? If so, you are in the right place.

Artifacts are files created by software development processes, such as packages, containers, configuration files, or documents. They are used in many stages in the software lifecycle. For example, artifacts can be a dependency of another artifact. Artifacts like packages or containers can be running in production. Working with artifacts can be complex because artifacts originate from many sources both inside and outside an organization. An artifact management system like Artifactory addresses complexity and reliability issues by centralizing your artifacts in a single location. You can find out more about artifact management here.

There are some challenges within the central artifact management system. For example, recently a security incident, Codecov Bash Uploader unknowingly leaked credentials stored in environment variables. In this blog, we’ll be focusing on credentials management in the central artifact management system, its consumers — CI/CD pipelines, and how to make them secure against similar attacks.

Before we dive in deeper, here are some terminologies we’ll use in this talk.

Terminology: They are basic concepts and universal across other products with different names.

• [GitLab] Merge Requests(MRs): Merge Requests are the way you check source code changes into a branch. This is analogous to Github’s Pull Request. More information: merge request official doc.
• [GitLab] Protected Branches: Permissions are fundamentally defined around the idea of having read or write permission to the w and branches. To impose further restrictions on certain branches, they can be protected. More information: protected branch official doc.
• [Artifactory] Group: Represents a role and is used with Role-Based Access Control(RBAC) rules. More information: group official doc.
• [Artifactory] Permission Target: Set the physical resources such as repositories, and select users or groups with a corresponding set of permissions that define how they can access the specified repositories. More information: permission target official doc.

Let’s dive in. we’ll go over what we used to have, good and not good, then how we have improved the system.

Starting with Manual Provisioning

As a request comes from the user, the admin creates permission targets and then assigns the user through the Artifactory Console. We create groups and assign users based on their teams. We also create service accounts so that pipelines can use its credential instead of users.

Good 👍

• No extra setup required

Not Good 👎

• Prone to human error
• Hard to reproduce in the case of disaster
• User has no visibility into what permissions are provisioned
• Extra work to audit

Progressing to Provisioning through a Pipeline

Manual provisioning quickly went out of hand. We codified all service accounts/groups/permission targets in YAML, and started applying changes using a pipeline. Users submitted MRs to request new permissions, then an admin checked and merged the MRs to provision.

Good 👍

• Two-person check at MR to ensure correctness
• Reproducible in the case of disaster
• Users can look at code to understand what permissions are provisioned
• Easy audit

Not Good 👎

• Required Knowledge: Users need to know Artifactory’s terminology such as the repository, actions and ANSI patterns. It becomes harder when repository topology becomes complicated with a multi-cluster setup.
• Credentials are static: When it requires rotation due to expiration or leakage, users need to locate and rotate all credentials.
• Credential is portable: Users can use the credential anywhere. Users can technically publish an artifact to a production repository from a local laptop. This impacts artifact integrity.

Experimenting with Ephemeral Credential — Version 1

We learned that we need an abstraction around Artifactory, and we need a solution for the previous credentials weaknesses — staticity and portability.

We initially started with Docker Registry as it was new at the time which allowed us to apply new policies around permissions. We developed Vault Plugin, and on-boarded it into an ephemeral credential suite in GitLab pipelines. In short, a pipeline with an ephemeral credential suite can request vault resources for its own pipeline, and perform token creation using a plugin on-demand during the pipeline’s runtime. The plugin accepts the top-level permission scope, creates a group and permission targets then creates a short-lived token. We predefined in a Git repository which top-level scope and alias for Docker Registry a pipeline requires. We then validate that a top-level scope can only be reserved by a single team, and create vault resources that can be consumed by pipelines.

Good 👍

• Users don’t have to deal with Artifactory’s terminology. They can work within a simplified user interface.
• Ephemeral Credential is tied to the pipeline.
• Ephemeral: No operational burden to rotate. Minimizes security posture for leakage like CodeCov attack.
• Non-portable: Only selected and pre-approved pipelines can publish artifacts to a production repository.

Not Good 👎

• The Vault Plugin wasn’t designed to be extended to other package types. It was hard to add other custom tweaks for each package type to the already Docker-customized plugin.
• Permissions are hard-coded in the plugin. It can only provide write permission. Even if a pipeline doesn’t require publishing capability, it provisions it. To workaround this, we needed to introduce some mock data.
• One pipeline can possibly publish an artifact to another pipeline’s artifact location. Let’s say that Team Tiger owns the top-level directory team-tiger. Tiger’s pipeline A publishes artifacts to team-tiger/myproject/myimage. Tiger’s pipeline B can also publish to team-tiger/myproject/myimage. Ownership of artifacts under team-tiger is a team’s responsibility
• (Plugin specific) a group and permission targets are created/updated every time a token is requested, although group and permission targets are rarely updated in Git. This causes additional API calls to the Artifactory server.
• (Plugin specific) We need to enable the vault secret engine and give a vault path for each repository. This duplicates configuration and has extra overhead.

Evolving to Ephemeral Credential — Version 2

We learned from version 1 of the ephemeral credential that we need a more generic approach to the Vault Plugin, which should be a true-interface of Artifactory with no custom processing specific to any packages. We then created another plugin, which is available open sourced under Splunk’s Github repository. This plugin exposes Artifactory’s concept of group, permission targets, and token.

Good 👍

• Pure interface to Artifactory. No custom logic in the plugin itself. We can extend this to any package types as the package type information is not tied to the plugin itself.
• Allows the setting of repositories, patterns, and actions freely. This allows granular access control to package levels.
• (Plugin specific): Creation of groups and permission targets is separated from token generation. Token generation references those pre-created groups and permission targets. This leads to fewer API calls to Artifactory, and less points of failure.
• (Plugin specific): One vault secret engine path for each Artifactory instance.

Quick Example

# Please mount a plugin, then you can enable a secret
$ vault secrets enable -path=artifactory vault-artifactory-secrets-plugin
Success! Enabled the vault-artifactory-secrets-plugin secrets engine at: artifactory/
# configure the /config backend. You must supply admin bearer token or username/password pair of an admin user.
$ vault write artifactory/config base_url=”https://artifactory.example.com/artifactory" bearer_token=$BEARER_TOKEN ttl=600 max_ttl=600
# create a role
$ cat scripts/sample_permission_targets.json
[
 {
 “repo”: {
 “include_patterns”: [“team-tiger/myimage/**”],
 “exclude_patterns”: [“”],
 “repositories”: [“docker-local”],
 “operations”: [“read”, “write”, “annotate”]
 }
 }
]
$ vault write artifactory/roles/ci-role token_ttl=600 permission_targets=@scripts/sample_permission_targets.json
# generate an ephemeral artifactory token
$ vault write artifactory/token/ci-role ttl=60
Key           Value
— —           — — -
access_token  REDACTED_BEARER_TOKEN
username      auto-vault-plugin-user.ci-role
# You can then use the token to call API or configure client
$ docker login docker-local.artifactory.example.com — username auto-vault-plugin-user.ci-role — password REDACTED_BEARER_TOKEN
$ docker push docker-local.artifactory.example.com/team-tiger/myimage:1.2.3

Bonus — version 2

We then extended the user interface from version 1 so that a user can request:

• which package repository a pipeline needs
• which package path a pipeline needs
• what capability a pipeline needs, such as read only, read and write, or delete

In the backend of this user interface, we define package repositories with:

• its corresponding readable local repositories
• writable repositories
• remote repositories
• its package type
• other properties like repositories being delete/overridable, and non/production repositories

We convert:

• simple package repository information into repositories for read, write, local, and remote
• package path to be expanded into the proper ANT-style Artifactory expects for permission target’s patterns
• capability into permission/actions that Artifactory expects

We then create an Artifactory group and accompanying permission targets for each pipeline.

Finally, we perform automatic validation of the user requests to ensure they follow security policies before provisioning resources and asking for credentials. Some examples are:

• no two pipelines can request for write capability to the same package path
• write capabilities can’t be provisioned for production repositories in an unprotected pipeline
• test repositories can’t be used in a protected pipeline

What We Learned

Simply put, this new credential management in the Artifactory system with the Vault plugin is more secure, more autonomous, and more user-friendly.

It is more secure by shifting from static credentials to ephemeral credentials. Valid persistent credentials no longer stay in the system, which reduces the risk of leakage/exposure. Even if credentials are leaked, they are short-lived. With long-lived persistent credentials, porting or leakage of credentials breaks artifact integrity. Short-lived credentials are not portable and do not allow such behavior.

The new system is more automated by moving human operation to machine operation when it comes to creating token and associated resources.

It is more user-friendly by simplifying repository-specific terminology into user-consumable style, and then converting it in the backend. It allows users to request capability, package path, and repository in code.

Some Extra Work

Outside of credentials work for the CICD pipeline, we have developed the CLI and its server to generate ephemeral credentials for local users. User access is tied to SAML, and processing is on the server-side application. It allows users to get ephemeral credentials in order to pull any artifacts from designated repositories, and also grant push permission to a user’s namespace in test repositories.

This allows us to remove all static credentials from both pipeline and users’ local machines.

Special thanks to Krishnan A. (@gotwarlost) for supervising and Nick S.(@nickshine) for collaborating in this plugin, local CLI, and server work.

You can find the vault plugin https://github.com/splunk/vault-plugin-secrets-artifactory