Systemd Service File for Vault
Let’s see how Vault gets along with the default init system for most of the Linux distributions, systemd, by analysing, testing and tinkering with the Vault service unit file!
When you deploy Hashicorp Vault on Linux you need to deal with the init system of your machine to set how Vault or any other service will start and run, what needs to be done before or afterwards, and some other details. This is addressed in the Vault Deployment Guide, but let’s analyse each part, tinkering a bit with the restart parameters, showing the metadata in actual commands output and covering things a bit more in depth, including the Linux capabilities and the resource limits. Ready? Let’s go!
The init system of a Linux machine… well, it doesn’t matter if you have a very pragmatic approach on the matter or you went over the five (six!) stages of grieving (systemd!): you are most likely dealing with systemctl and other systemd tools anyway. Let’s see how Vault and systemd get along then.
If you follow the aforementioned deployment guide, you will find the service unit file vault.service, going along with the .service suffix convention. A unit in systemd lingo is simply a resource that will be managed by systemd. In this particular case, the content of this file determines how systemd will start and run Vault, what it does and some other things.
Unit files can live in several different places. In the deployment guide, it is stated that the Vault service unit file lives at the /etc/systemd/system path, which is the usual place for local unit files. Systemd will give the highest priority to the files living under /etc.
The section names of the unit file (Unit, Service and Install) are well defined, and it is good to remember that they are case-sensitive! :)
[Unit]
Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.targetThe format actually brings some memories… those .ini files that I discovered as a kid with my first steps using my first MS-DOS version: 3.3 :)
Now that we have introduced our dear friend, the vault service unit file, let’s analyse each section:
The [Unit] Section of the Vault Service Unit File
This section is generally used for the definition of some information and metadata for the service unit itself. You might also find some directives regarding the relationship of the Vault service unit to other units:
[Unit]
Description="HashiCorp Vault - A tool for managing secrets" Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target ConditionFileNotEmpty=/etc/vault.d/vault.hcl StartLimitIntervalSec=60
StartLimitBurst=3The Description directive is generally used to describe the name and basic functionality, so we will simply state “HashiCorp Vault — A tool for managing secrets”.
The Documentation directive in this case is an external URL that points to the Vault project docs.
This info is returned by some systemd tools such as systemctl as we can see in the screenshot, so it is better to set this to something short, specific, and informative:
The Requires directive lists any units upon which Vault depends. If the Vault service unit is activated, the unit listed here (network-online.target) will also be activated; if any of them fail, then so will the Vault service unit.
The After directive configure an ordering dependency between units. The units listed here will be fully started before starting the Vault service unit. In this case the network-online.target unit is fully started up before the Vault service unit is started.
The ConditionFileNotEmpty directive allows us to test certain conditions prior to starting the Vault service unit, like in this case that the Vault configuration file (/etc/vault.d/vault.hcl) exists and is not empty.
The StartLimitBurst and StartLimitIntervalSec directives will set the limit on the number of times the Vault service unit can be started within an interval of time; in this case, 3 times in 60 seconds will be the limit.
If we try to restart Vault more than 3 times in a minute we will get an error: if we check the details with systemctl or journalctl, we will see the message that the Vault service failed with result “start-limit-hit” as the “start request repeated too quickly”:
The [Service] Section of the Vault Service Unit File
This section is a type-specific one for services and is used to provide configuration data applicable for the Vault service:
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinityWho is calling the shots?
In order to define who is calling the shots (running the process) we have the User and Group directives that will set the Linux user and group that the Vault processes are executed as. For the Vault service unit file we will operate with the vaultuser and the vaultgroup.
Sandboxing
There are a few sandboxing options that are an effective way to limit the exposure of the system to the unit’s processes, namely the ProtectSystem directive, which when set to full will mount the/etc directory read-only in addition to the /usr and /boot directories that will be read-only for processes invoked by the Vault service unit. Note that this directive will have no effect if the Linux kernel is built without file system namespaces.
The ProtectHome directive will affect the directories /home, /root, and /run/user that in this case will be read-only for processes invoked by this unit since the value of this directive is set to “read-only”.
Setting the directive PrivateTmp to true will configure a new file system namespace for the Vault processes and mount private directories (/tmp and /var/tmp)inside that namespace. These directories are not shared by other processes outside of the namespace, so it is useful in order to secure the access to temporary files of the Vault processes.
The PrivateDevices directive when set to yes will turn off physical device access by the Vault processes and will set up a new /dev mount for the Vault processes with access to /dev/null, /dev/zero and /dev/random, etc., but no access to physical devices, system memory or system ports.
Linux capabilities
There are also several capabilities directives that are a way to provide a subset of the root privileges to a process, so you can grant a fine-grained set of privileges to the unprivileged processes (whose effective UID is nonzero).
In this case, the SecureBits, AmbientCapabilities, Capabilities and CapabilityBoundingSet directives will model the permissions that we will grant the Vault process to allow things such as locking memory (cap_ipc_lock) or performing privileged syslog operations (cap_syslog). We can always check which capabilities have been granted to our Vault process by checking the /proc filesystem and piping the output with capsh:
Avoiding privilege elevation
Setting the directive NoNewPrivileges to yes ensures that the Vault process and all its children can never gain new privileges; so, the Vault process can never elevate its own privileges.
Service lifecycle
In order to define how the service is started, restarted, reloaded or killed we have several directives that will define the commands with their arguments that are executed when Vault is started (ExecStart):
/usr/local/bin/vault server -config=/etc/vault.d/vault.hclThe commands to trigger a Vault configuration reload (ExecReload):
/bin/kill --signal HUP $MAINPIDOr the process killing procedure configuration with the directives KillMode that will have the value processto say that only the main process itself is killed or the directive KillSignal that will be set to SIGINT in order to send an interrupt signal (2).
The directive Restart will determine whether the service shall be restarted whenever the service process exits, is killed, or a timeout is reached. As we are setting this directive to on-failure, the Vault service will be restarted in a number of failure scenarios: when the process exits with a non-zero exit code, when it is terminated by a signal, etc.
The RestartSec directive will set the number of seconds before restarting the Vault service, in this case 5 seconds.
Let's be a little patient with the TimeoutStopSec directive set to 30 to wait half a minute for a clean stop, otherwise: no kidding… as it will be terminated by SIGKILL
And finally, if Vault is started more than 3 times within a minute, it will not be allowed to start any more, which is configured setting the StartLimitIntervalSec to 60 (seconds) and the StartLimitBurst directive to 3 (attempts)
Process properties
We can set soft and hard limits on several resources for the Vault process. In this case, we will use the LimitNOFILE directive with the value 65536 to limit the number of File Descriptors and we will get rid of the limitation on the amount of memory the Vault process can use with LimitMEMLOCK set to the string infinity.
The [Install] section of the Vault service unit file
This section is optional and is is not interpreted by systemd during runtime. It is mainly used to state the behaviour of the Vault service unit if it is enabled or disabled. The section carries “installation information” for the Vault service unit.
[Install]
WantedBy=multi-user.targetThe WantedBy directive creates a weak dependency of Vault being started by the multi-user run level.
With this line systemctl will know that the Vault service unit is related to a group of unit files called multi-user.target.which is the systemd terminology for the runlevel 3 in the systemV world.
We can list the target units with systemctl list-units --type target:
And we can see the details of the multi-user.target unit with systemctl:
Many times we need to pass information to Vault such as auto-unseal parameters. In order to manage that scenario, the Environment directive within the Service section comes in handy. The following will set an environment variable for the Vault process:
[Service]
...
Environment="MY_KEY=my_favourite_value_today"
...That’s all! I hope you have enjoyed covering a bit more in depth the Vault service unit file for systemd and remember that if there is a full consensus around systemd, you can always bring up text editors, DKMS, and open source licenses to heat up those watercooler conversations ;)
