Infrastructure Security| SSL/TLS Certificates and Challenges
Some of us might be thinking, it is more related to application security or information security !! Well, all of it, is a part of our IT infrastructure and so be classified under the same, always.
When it comes to IT infrastructure security, the first few entities we would think about, are the applications hosted, access to the infrastructure resources, the file or data sharing and the email or relay. The information data these entities contain, is much important to us and our clients.
In the same context, one important thing we all should be worried about, is SSL/TLS certificates. There are many different roles these certificates play, when it comes to the security of our infrastructure. Some phrases readily comes to mind are:
- SSH keys.
- Certificate and keys for Web Application Servers.
- Certificate and keys for FTP servers.
- Certificate and keys for mail or relay servers.
In this article, my primary focus will be on understanding the risk associated and possible automation approaches in the certificate management process.
When a new certificate gets signed by the CA (certificate authority), a serial number(image-2 below) gets allocated to it, this serial number (a 20 byte alphanumeric code, formed by, issuer_DN + serial) is also a deemed unique identifier for a certificate over internet, then used by OCSP or equivalent protocols to keep track of a certificate status.
Most browser uses and supports OCSP, few didn’t. OCSP helps browsers to identify the authenticity of a certificate quickly, rather to download the whole CRL for the CA vendor, who signed the certificate.
Common Challenges
Let’s take a look at couple of known issues.
- How many times, we have seen the errors in browser like image-3(below) or a message, like ssl_error_bad_cert_domain or a service fails to start with invalid SSL certificate error? Why this happened?
Reason: Certificate gets applied to the incorrect web portal.
This usually is an issue, happens with organizations, that hosts, multiple web portals per web servers.
More properly, this could have happened because the sysadmin team, mistakenly, applied the certificate intended for domain A to a web portal hosting domain B. Hence, the browser, trying to check the certificate’s authenticity, gets an error on matching the certificate’s common name with the domain in the URL.
- How many times, we see an expired certificate, an image like image-4 (below) or a quoted error, like err_cert_date_invalid?
Reason: The certificate is expired
This happened, almost with every organizations, at some time. The sysadmin team, had missed the renewal deadline for the web portal and the end users, will start seeing something like image-4 shown above.
Although, if we have taken an SSL /TLS certificate from a trusted certificate vendor, we ourselves or the admin registered with the trusted certificate vendors, would receive timely emails, regarding the SSL expiry, some 30 days before the certificate actually expires and added to the CRL (Certificate Revocation list), eventually. There are, always, chances these emails are missed or overlooked.
The Overall Risk
Almost all organizations use SSL/ TLS certificates and there are many, those uses certificates a lot, say, hundreds of them for multiple domains within organizations and, then, multiple sites or portals within the individual domain. The chances of missing a certificate renewal is high, in such cases.
It doesn’t sounds that bad, not as bad as, showing, end users, an error page with a warning of a security risk, on our public facing web portal.
Now, that sounds pretty bad.
Moreover, an expired or revoked certificate, if get passed, the browser validation(which is quiet possible in few browsers, at least), exposes the whole organization to security risks, because such certificates, no longer are marked trusted over internet and with, so many malicious eyes, scanning for some vulnerability to get in.
The impact, can be more devastating when it comes to FTP or Mail/Relay servers, because more than just data would come to risk. Though, there are best practices for FTP and MAIL servers, which ensures to do a strict certificate check, but, gotchas can never be ignored. Where, with FTP, not just the credentials, its the private files and documents are at risk. With, Mail/Relay its further more, information coming in and going out comes to risk.
Such incidents can harm the public image of the organization, even, could get the complete or partial entity or portal blocked permanently.
The organizations, those, use spreadsheets for certificate management, are again, at a high risk in doing SSL management, may or may not be knowing, what kind of trouble it could bring.
Now, that we have understood the high level risk, associated with using invalid certificates, we must, take a small look on the risk associated with the responsible protocols and cryptographic hash function, too.
TLS vs SSL
When it comes to certificates, its not only the deployment that people should be worried about. Its a lot of other things, the protocols themselves. POODLE, name of a vulnerability that came recently a couple of years back and was reported in SSL v3.0 (last of SSL releases), capable of allowing an attacker to get information like passwords and cookies, despite of a secure connection. SSL v2 (deprecated 2011) and SSL v3 (deprecated 2015). So, what should we use? TLS v1.3, is in use nowadays and latest.
Similarly, SHA1 was deprecated and dropped support by almost all trusted CA already, for the same reason of exposure to vulnerabilities. So, all new certificates coming now on, are mostly SHA256/SHA2 (*while, there are many other cryptographic hash function, I chose to cover few popular ones.).
One major problems I remember, that happened while transitioning from SHA1 to SHA256 was, most tools and applications caught compatibility issues, to support this transition, few Exchange servers for an example, were one of them. Most of, such incompatible software's and applications had to be upgraded/patched, on urgent basis, to support a SHA256 signed certificate.
With, what we have so far, in this article.
The only questions, remains.
What shall we do better to avoid or minimize the risk?
Possible automation approaches
When there is always, an untold risk with manual SSL management, there are also, various paid tools in the market that provides management of SSL certificates and so are the organizations that provides the similar services. Most CA, does provide the operational services, such as service discovery, SSL push to the web servers, troubleshooting, etc. BUT.
- Most tools does not provide assurance that the SSL is correctly paired with the application.
- There are automated solutions from some vendors that can renew certificates and can discover all those are expiring, but again may or may not support multiple CA.
- There are different security risks associated with, giving a third party, access to our production network.
For some organizations, using paid tools would be a good solution, for others, with vast usage of SSL certificates, it could be too costly.
For those, looking for a self managed solution, I will try to cover some of the possible approaches to this problem.
Custom Scripts
The whole, SSL management process, can be automated by writing custom scripts, programming languages, like python and go-lang offer modules, capable of creating efficient automation framework, scripting tools like GNU shell and windows power-shell does offer similar options.
The framework that can:
- Check expiring certificates.
- Create CSR and KEY before hand, which can be picked up by sysadmin team to generate certificates and handle deployment.
I was working at an approach that can fix this problem once for all and have suggested it’s first draft to few people searching for an approach on few forums. Here’s a glimpse of that suggested approach.
Its hard to say, that there is any end-to-end automation at present, which could fit everyone’s need. But, I tried and wrote a set of scripts, that can be used to get the expiry predictions SSLexpiryPredictions. It can scan through the list of URLs or certificates stored inside a directory or external volume and create a parse-able report, which can be used for further operations.
There is an additional script that can also generate certificate requests(CSR) and keys, genSSLcsr and place them at a desirable locations. By default, certificate’s common name is used to create CSR and KEY files.
These certificate files, can be kept inside a directory named after server group name to distinguish from other web servers groups, later can be used with Ansible like configuration management tool to deploy certificates on the web server groups without having to worry about, most of the troubling issues we discussed.
For Example:
# copy certificate file
ansible -m copy -a "src=mycertificate.pem dest=/path/to/mycertificate.pem owner=apache group=apache mode=0644" webservers# copy certificate key file
ansible -m copy -a "src=mykey.pem dest=/path/to/mykey.pem owner=apache group=apache mode=0600" webservers
Using Configuration Management Tools
Some configuration management tools like, chef , puppet or ansible can also be used to create an automation framework to handle SSL certificate management throughout the infrastructure.
Ansible, on its own supports creation of CSR and KEY with its crypto modules. There are some exiting ansible roles, which could do some amount of automation for deploying SSL certificates, but may need few modifications, as per individual needs. Ansible role from jdauphant is an example of such utility.
Conclusion
There are lot of things happening these days, in terms of security best practices and a lot of open source tools coming up, offering the best we can desire. We should, at the same time, be cautious, in picking up the tool for our usage. It is always good to fork the repository and make necessary modifications, in line with the LICENSING or use a tool which is backed by a community, just in case we encounter issues, we have someone to reach out.
As I did mentioned, working on a framework myself, If any of my readers want to contribute or make use of the current utility, please check the GitHub link ssl-on-demand.
The article is open for comments and suggestions, if any.
