Solved: PM2 and a Private CA

Generating SSL certificates through a private certificate authority (CA) is both useful and common for secure communication between servers within a closed environment. Rather than paying for an SSL certificate through one of the major CAs such as Symantec or DigiCert for servers within an organization, it’s easier and more cost-effective to become a CA and issue your own SSL certificates.

Photo by Jefferson Santos on Unsplash

Private Certificate Authorities and SSL Certificates in Node

How does a Node application know when to trust the SSL certificate from a private CA? Normally it wouldn’t be trusted because Node’s SSL and TLS connections look at only the well-known root CAs. Given the broad adoption of private CAs issuing SSL certificates for internal connections within an organization, Node added the NODE_EXTRA_CA_CERTS environment variable in v7.3.0. By setting this variable to a file consisting of “one or more trusted certificates in PEM format,” a Node process will extend the well-known CAs with those in the file. This enables admins to set additional CAs without the need to edit any code. The additional certificates are only read once upon application startup. Great! It’s a “set it and forget it” type configuration. No need to involve developers or add boilerplate code to an application. As a matter of fact, a developer couldn’t set it even if they wanted to, as NODE_EXTRA_CA_CERTS cannot be set programmatically.

process.env.NODE_EXTRA_CA_CERTS=’path/to/certs/file’

The above will have no effect on the application. That’s fine, because why would an application need to do this? The answer is, “it shouldn’t.”

Process Managers and Child Processes

What if the application is spawning or forking child processes? That will work fine as long as the parent process passes down its environment variables to the child appropriately. In Node versions prior to 12, the parent needs to pass process.env as the env option to the new process. However, as of Node 12, the child’s process.env defaults to the parent’s process.env.

If there’s a process manager in charge of an application, then it should provide the environment variables correctly. PM2, a daemon process manager written in JS, does not currently do this correctly for Node versions prior to 12. When PM2 runs in cluster mode, it forks the master process and passes a stringified version of environment variables to its child processes via an environment variable called pm2_env. See the code in lib/God/ClusterMode.js.

PM2’s cluster mode forks the process with its customized environment variables

The child process then parses that pm2_env and iterates over it to set each process.env variable.

PM2’s cluster mode chid process programmatically setting its environment variables

This results in a programmatically set NODE_EXTRA_CA_CERTS variable! Therefore, the child process never loads the CA extension file and connections to servers using an SSL certificate signed by that private CA will fail.

The Solution

• There is an open PR to fix this issue in PM2, but it hasn’t gotten much love since its creation in August 2017.
• Running PM2 in a Node >= 12 environment should mitigate the issue.
• Setting the ca option of the https.globalAgent at the beginning of the application will ensure that the private CA is recognized.

const https = require(‘https’)
if (process.env.NODE_EXTRA_CA_CERTS) {
 https.globalAgent.options.ca = [
 fs.readFileSync(process.env.NODE_EXTRA_CA_CERTS),
 ]
}