How we have applied Let’s Encrypt on hundreds of domains and managing the automatic certificate renewal and deployment
We have introduced HTTPS protocol for hundreds of domains at Livesport to increase security and partly also due to pressure from various browsers. It is directly related to certificates that need to be acquired, renewed and distributed in some way. We needed to figure out how to safely and, if possible, automatically handle hundreds of certificates.
We used RapidSSL in the past, but it caused too many problems. Certificates required manual purchases, the automation was complicated; there were problems with credibility, and, ultimately, disruption of certificate authority credibility in browsers.
The certification authority Let’s Encrypt captured our attention at the time of its creation (mainly due to the smooth automation), and, therefore, we have continuously monitored its development. After the introduction of wildcard certificate support (March 2018), the last obstacle fell, and we have started to fully deploy Let’s Encrypt certificates for our projects.
The authority issues the certificates free of charge. This fact eliminates the costs and economic agenda associated with the acquisition of certificates (invoices, payment cards, etc.). That’s why we decided to become its sponsor to support great project.
To obtain and renew certificates with Let’s Encrypt, we use the Certbot tool, which communicates with the CA by using the ACME protocol. One of the specifics of a wildcard certificate is that it can be verified in Let’s Encrypt just via DNS. We manage the DNS centrally, and for this purpose, we use NS delegation of a specific DNS record (_acme-challenge) to the server where the actual certificate acquisition and renewal takes place. The Certbot adds TXT records to the delegated zone to validate the request. The entire communication with CA uses API administered by Let’s Encrypt.
Invalidity is not a threat
Renewal of certificates in advance provides a considerable advantage. If an error occurs, we learn about it on time and can, therefore, solve any problem. Such reserve time even shields us from short-term outages of the CA or our infrastructure. These would have to last for more than 25 days to experience any problems such as certificate invalidity. In the future, we want to develop redundancy that has not yet been given a priority in the light of the previous considerations.
To deploy certificates, we use our own set of scripts communicating with a secure repository and ensuring the certificate distribution to target servers. We use HashiCorp’s Vault software as a secure certificate repository.
We learned about Vault approximately two years ago when we built a new company headquarters and needed a repository for Wi-Fi certificates and similar documents. At that time, however, we developed a repository built on easy-rsa and our API server. The solution worked but was not well suited for distributing third-party certificates and opening a repository publicly on the internet, which should be used, among other things, for development and use in the cloud.
Last year, we reencountered Vault at a conference, and after testing, we decided to apply it because it met our requirements for such type of data repositories such as security, authentication, authorisation, distribution, and high availability.
Vault has many advantages. It does not trust the backend and encrypts the data in real-time, and everything the application sends to the storage backend is, therefore, encrypted. Furthermore, Vault supports a variety of data storage options such as various SQLs and other databases, Etcd, cloud repositories, etc. We are currently using a high availability (HA) solution where we have chosen for this purpose Consul Cluster (also a HashiCorp product) as the backend.
The Vault enables us to distribute certificates securely both inside and outside our internal infrastructure, and the frequency of its use continues to increase. Cloud servers or applications download specific certificates they need to operate from the Vault.
Symbiotic use of Vault and Puppet
The Vault also guarantees a high level of security in terms of authentication and authorisation. It is possible to assign different levels of access and permissions for each authority. It is, therefore, possible to obtain a particular set of certificates with one particular identity. You can also deny list of objects in Vault to prevent attacker from exploring data in Vault.
Before the global introduction of HTTPSs, we used to have far fewer certificates and were able to upload them to all our servers by using the Puppet. Such a solution, however, increased the risks of compromising as the possible attacker would get all our certificates from a single server. Moreover, the vast majority of servers did not need any of the certificates. At the same time, we were not able to automatically transmit the certificates out of our infrastructure.
The Puppet continues to help us to distribute the certificates across our infrastructure. We distribute to all our servers the configuration specifying where to deploy the given certificate. The certificate life cycle on the server is then handled by scripts that deal with obtaining the certificate from the Vault, renewing it (if it is reloaded in the repository) and deleting it if it is no longer necessary on the server. If the server should be free of any certificates, the Puppet will also clean up the entire certificate management and deploy scripts so that it does not contain potentially sensitive information.
Distribution outside our infrastructure represents a separate issue. For example, for Kubernetes, we use the Secrets-Manager from Tuenti.
Getting a certificate from Let’s Encrypt for a new domain looks something like this:
- We manually run the script which: Setting the DNS; Obtaining the certificate from Let’s Encrypt; Upload certificate to the Hashicorp’s Vault
- We manually add the certificate to the Puppet settings (specify which certificate should be on which server)
- And finally, the renewal of the certificate is already fully automated
Our technical team ensures all activities related to certificate obtaining and renewing, and they are also responsible for the administration of on-prem servers, clusters, and related infrastructure. Mediation of certificates has become a service for us that we provide to other teams. Anyone who needs can use the up-to-date certificates stored in the Vault, and they don’t have to worry about them anymore. Besides, to be a certificate repository, we use the Vault as a certification authority for other applications because it allows issuing of certificates via API and it automates creation of certificates with a short-term validity (e.g. 30 minutes).
Vault Certificate Deployment: https://github.com/rvojcik/vault-certificate-deploy
