API Keys — What’s the Point?
When and why to authenticate with API Keys, Service Keys, Personal Access Tokens or the like, instead of plain old passwords.
TL;DR: Passwords are for humans, API keys for automated tasks or applications. Using API keys, you don’t have to worry about expiring passwords or multi-factor authentication in your automation. But make sure to keep those API keys secret.
Computers are good at doing things automatically. Take for example a CI/CD pipeline. You push some code changes to your source code repository. That triggers a build to check that your changes are good. When you merge the changes, another build runs, and the build artifacts get stored in an artifact repository. Maybe the new artifacts are deployed right away to some integration system, where an integration test suite runs additional checks, ensuring that your changes don’t break anything around them. And if you’re daring, the new artifacts get deployed to the production systems automatically as well.
That kind of automation requires many computers to work together. The CI/CD system is running somewhere, waiting for its pipelines to be triggered. The source code repository is running elsewhere, but it’s hooked up somehow to the CI/CD system. When a pipeline gets triggered, the tasks defined by that pipeline are executed on worker nodes, which are separate from the CI/CD manager nodes. The artifact repository is located yet elsewhere, as are the integration and production systems.
Let’s say you merge some code changes, and a build starts on some worker node. That worker node now has to pull the current code from the source code repository. But the source code repository will not hand out the code to just any computer that asks for it. The worker node needs to present some credentials, to prove that it is allowed to pull the code. No problem, you’ve configured those in the pipeline, along with the location of the source code repository. After the build, the worker node has to upload the results to the artifact repository. But the artifact repository will not accept files from just any computer that sends them. Again, the worker node needs to present some (other) credentials, to prove that it is allowed to upload artifacts. No problem, you’ve configured those in the pipeline as well.
So, what is the problem? It’s the type of credentials you configure.
Plain Old Passwords
In the good old days, you’d create user accounts in the various backends that need to be accessed, and configure the respective usernames and passwords in the automation. Such user accounts do not represent a person, they’re called functional or technical user accounts. And for some time, that used to be good enough.
Humans are notoriously bad at choosing good passwords and keeping them secret. To counteract this weakness, many organisations started to enforce security policies around passwords and login. These security policies apply to functional accounts as well as personal accounts. Minimum password length, range of characters, forbidden re-use of previous passwords, those are policies which don’t get in the way of automation. But there are others…
Mandatory Password Changes
One common practice is to require periodic password changes. There’s a debate or two whether that makes sense, but you might not be in a position to define the security policies. If your employer decides that passwords expire every 90 days, you’ll have to change them four to five times a year. And if those passwords are configured in automated tasks, you have to reconfigure each task, four to five times a year. With DevOps pushing for ever more automation, this turns from a minor nuisance into a major one. You can relieve the burden somewhat with a secure password store like Vault, but it remains a nuisance.
There’s another little problem with changing passwords. The moment you change a password, all the automated tasks using the old password are broken, until you get around to reconfiguring them. This can be avoided, at the cost of having two functional accounts per backend. Let’s call them green and blue, and assume that all tasks are currently configured with the green account and password. A few days before the green password expires, you reset the blue password. That gives you time to reconfigure all tasks to use the blue account and password, instead of the green one. Once you’re through with that, just let the green password expire. And a few weeks later, you repeat the procedure in the other direction.
Two Factor Authentication
Because passwords can be guessed, espied, or tricked out of users, it is good security practice to require at least a second factor for login. For example a smartcard that you have to put into a reader, or a token generator that shows a different number every minute, or a mobile phone to which a code number is sent when you log in. Or something biometric, like a retinal scan.
This extra level of security is called two-factor authentication (2FA) or multi-factor authentication (MFA). It makes the password by itself pretty much useless. If you were planning to configure the password in an automated task, you’re screwed. MFA was invented to ensure that the rightful owner of the account is present in person when the password is being used, and automation is the opposite of being present in person. There is no way for a CI/CD pipeline or other automated tasks to pass MFA. If there was, it’d be a security hole that needs to be fixed.
So, passwords are not that great for use in automation. Let’s take a step back and look at the problem on an abstract level.
Authentication and Authorization
You’ve probably read or heard the abbreviation “auth” more than once. Depending on context, it stands for Authentication, for Authorization, or both. There is a Medium story that discusses the difference. Simply put:
- Authentication (authn) establishes who or what is accessing the system.
- Authorization (authz) decides whether an access is allowed or denied.
The concepts are somewhat intertwined, and frequently mixed up. That’s because in order to decide whether an access is allowed or denied, it is often necessary to know who or what is trying to access. For example, the Basic Authentication scheme introduced with HTTP/1.0 responds to a challenge WWW-Authenticate by sending authentication information in a header called Authorization. The server then uses that header to authorize the request, so who’s to say the name is wrong?
Passwords are generally considered a weak form of authentication, because some people choose weak passwords, might be observed while typing them, or can be tricked into telling them on the telephone. Multi-factor authentication or additional challenge-response questions were invented to strengthen the authentication provided by a username and password. They’re designed to make sure that it’s the genuine user logging in, and not some crook or bot who just got hold of or guessed the user’s password.
But automated tasks are bots. Users who set up an automated task intentionally delegate some authority to that task. So they need a way for the task to authenticate automatically, while giving the system reasonable certainty that this is happening with the user’s approval. Configuring username and password is at best a rather crude way to achieve that. The task isn’t the user, and it doesn’t need the full authority of the user. And exactly there is the middle ground between security requirements and technical requirements: Instead of configuring user credentials and working around MFA, automated tasks should be configured with task credentials that have limited authority, but allow for single-factor authentication.
API Keys
API keys are task credentials. They’re created by request of a user, and they are bound to that user’s account. When creating or looking up API keys, multi-factor authentication can and should be enforced. An API key is generated by the backend system that’s going to accept it. It is a long and unwieldy string, like a very strong password. But it cannot be used for logging in as a user, nor for creating new API keys. It is only accepted as an authentication token by the APIs of the backend system that created it. Those APIs are called over a network, typically with HTTP requests. When the API key is accepted, authorization of the requested action is based on the permissions granted to the user owning the API key. The action gets executed, logged and audited as if requested by that user. Knowledge of the API key (and username, in some implementations) is the single factor that authenticates the task as acting on behalf of the user.
You don’t type an API key, ever. When you configure an API key for a CI/CD pipeline, you copy&paste it. When you have a dedicated machine for running automated tasks, maybe you store the API key in a file and upload that to the machine. The transfer of API keys can be protected by MFA, just like creation or lookup. It’s typically a manual step, performed by a human. Make sure to use encrypted connections when transferring API keys, either by file or by copy&paste.
Don’t transfer API keys into untrusted environments. API keys need to be kept secret. If you put an API key into a mobile app, and publish that in an app store, everyone can download the app, extract the API key, and call the API with it. The same problem exists with container images. Even build args which don’t get stored in the image layers are exposed through the docker history. (I fell into that trap myself. 😳)
It still makes sense to have functional user accounts. After all, you don’t want to reconfigure API keys when a team member moves on. Those functional users still have passwords, which still need to be changed periodically. But the API keys remain valid when the passwords change, so you don’t have to reconfigure anything. Whether configuration of a CI/CD pipeline requires a personal or a functional user matters neither for the pipeline nor for the APIs. I’d recommend personal user accounts, because that will leave a better audit trail.
Limited Authority
An API key grants less access than the user account it is created for. That’s a precondition for allowing single-factor authentication with the API key, when the user account itself requires multi-factor authentication. I’ve already mentioned two restrictions: an API key cannot be used for logging into a user interface, and it is only accepted by the backend system that created it. These restrictions become exceedingly apparent when you’re working in a single sign-on environment.
At my workplace, I use my intranet account and password to login to Artifactory, to GitHub Enterprise, to IBM Cloud, and several other systems. If somebody could hack into that account and access all those systems, a lot of damage might result. So there are security policies that apply to my intranet password (it’s currently 21 characters long and will have expired by the time this story gets published), and MFA is enforced wherever I use it.
As part of my work, I have to upload files to Artifactory every few weeks. I prefer to use the command-line tool for that. But that tool doesn’t implement my employer’s MFA, so I cannot use my intranet password with it. Instead, I generated an API key in the Artifactory UI, and configured that on my work laptop. Neither GitHub Enterprise, nor IBM Cloud, nor any other of the systems I can access with my intranet account, know about my Artifactory API key. And all the web-based user interfaces of these systems refer to the same login page, which also doesn’t know about the Artifactory API key. It cannot be used anywhere but with the Artifactory API.
I must keep that API key secret. If you’d learn my intranet password, nothing bad would happen. You’d know only one of multiple factors, and the others would stop you. But if you’d learn my Artifactory API key, you could use it to access unreleased build artifacts, or you could break our builds by deleting the input files I uploaded. You couldn’t access GitHub Enterprise, nor IBM Cloud, nor any of the other systems I have access to, that’s why MFA is not required. But getting API access to Artifactory would still be bad. Therefore, I’ve configured that API key only on my work laptop, and I’ll never copy&paste it anywhere else. I also make sure that this API key never gets into the docker history of a container image I build, because I just might upload that image to a test cluster, where others could inspect it. If my work laptop should ever go missing, I’ll revoke that API key right away.
Besides the two generic restrictions, some implementations of API keys provide additional ones. So let’s have a look at a few examples.
Implementation Examples
There is no standard for API keys, nor a default implementation. Every system can handle them differently. Nevertheless, there are some recurring patterns. The following examples are not a representative selection though. These are just some systems I’ve used so far. If you know of API key implementations with significant differences, please share your examples in the comments.
Single API Key
Artifactory comes with a minimalistic implementation of API keys. Every user account has a single API key, or none. The user can look up, revoke, or regenerate that API key. There’s an icon that will copy the API key directly into the clipboard, without even showing it on screen.
There are no restrictions on using the API key, besides the two generic ones explained above. If you want two API keys that provide read-write and read-only access, for example, you’ll have to create two user accounts. You also need functional users to manage access to the different repositories of an Artifactory installation.
Personal Access Tokens
Source code hosting solutions like GitHub, GitLab, or BitBucket manage Personal Access Tokens (PATs) to provide access for external services regardless of MFA. The cloud version of BitBucket calls them app passwords. Common to all three offerings, whether in the cloud or self-hosted, is that you can have many personal access tokens. You give them names to distinguish for which external service they were created. This way, you can delete a token when you stop using the external service.
The value of a PAT is shown only once, when you create or regenerate it. If you have to integrate another external service later on, create a new token. The advantage of showing the PAT value only once is that the backend doesn’t have to store the actual value, but merely a salted hash of it. That is the gold standard for keeping track of passwords, too. What isn’t stored cannot be stolen.
Another common trait of the source code hosters is that their APIs are split into multiple sections. You decide for which of these sections a PAT should be valid. That’s a case of the additional restrictions that I’ve been mentioning above. You can, for example, specify that one token should grant only read access to your repositories, while another grants read and write access.
As shown by the screenshots, the offerings differ in the number and granularity of API sections. GitHub has actually more than I could fit on the screen. There are other little differences as well. Some offerings require only the token for authentication, others expect the username with it. GitHub automatically deletes tokens that haven’t been used for a year. GitLab allows to specify an expiration date.
Cloud API Keys
Cloud providers enable a lot of user actions: managing resources for computing and storage, managing services, managing your own code that runs in the cloud, and of course managing access to the management capabilities. I’m taking this example from the IBM Cloud, because that’s the one I’m using. I assume that other cloud providers have similar capabilities, or are working to close the gap.
In the IBM Cloud, you can create as many API keys as you need. Their value is shown only once, similar to personal access tokens. But instead of selecting from a few checkboxes, you get to define fine-grained authorizations. How to do that is beyond the scope of this story though. Each API key is bound either to a user ID or to a service ID. Service IDs are the equivalent of functional user IDs. You can create as many of them as you need, without registering fake users.
Cloud Service Credentials
Cloud providers don’t just host and run your application code. They also host services that your applications can use. Need a relational database? A NoSQL database? A message broker? Just pick a suitable service and billing plan, then create an instance of the service for your own use. Behind the scenes, the cloud provider or another party is running installations of each service. Creating a service instance is similar to creating a new installation dedicated for your use, with a functional user account in it. Or else you get a new functional user account in a shared installation of the service. Either way, these functional users exist only in that service installation. They’re unknown to other services, and to the login page.
In order to use the service instance, an application or automated task needs to know the hostname and port number to connect to, maybe some service-specific pieces of data, and of course the credentials for authenticating. This collection of information is sometimes called service credentials, or service key. You can create many service credentials for a service instance, and give them different levels of access, depending on the type of service. Service credentials, including the authentication data within, can be looked up at any time by users with a sufficient level of access to the service instance.
In the IBM Cloud, the authentication data in the service credentials may be a Service API key, which uses the same infrastructure as the Cloud API keys described in the previous section. Or the authentication data may be what is now called legacy credentials: username and password for a functional user of your service instance. Do you notice how we’ve come full circle from the Plain Old Passwords above? But there’s a difference. The username and password in cloud service credentials are for authenticating applications or tasks, not for authenticating human users. They’re not subject to the password policies or multi-factor authentication requirements for users. And that’s the point of API keys vs. plain old passwords.
Summary
Both passwords and API keys are for authenticating, and subsequently authorizing, access to computer systems. Passwords are the traditional way to authenticate users. Because humans are bad at keeping passwords secret, there are additional measures to prevent abuse of passwords, like multi-factor authentication. These measures get in the way of authenticating automated tasks or applications. API keys replace passwords for that purpose.
Security Summary
API keys are more sensitive than passwords, because there is no second factor for authentication. They are less exposed than passwords, because they don’t get typed on a regular basis. Humans handle them only on rare occasions, when configuring an automated task or application. API keys are generated rather than chosen, so password-guessing attacks can be thwarted.
A compromised user account is worse than a compromised API key for that account, because API keys provide only limited access. For example, they don’t provide access to the user interface. Nevertheless, a compromised API key can be bad. If you’re hosting the API yourself, you might add network-level access control to prevent abuse of leaked API keys from outside.
API keys are long-lived, manually managed access tokens. Another way for authenticating automated tasks or applications is OAuth, which uses short-lived, automatically generated access tokens. But that is a story for another day.
