Encrypted communication on Elastic server by setting up TLS and adjusting CORS permissions

On May 20, version 7.1 of Elastic Stack was released. In this version, there are no new features but some core security features are free now, including:

• TLS for encrypted communications
• File and native realm for creating and managing users
• Role-based access control for controlling user access to cluster APIs and indexes.

Which means we don’t need to set up the additional proxy on our Elastic server.

In this tutorial, we will provide step-by-step instructions on how to encrypt communication among Elasticsearch, Kibana, and Logstash. After the setup, Elasticsearch and Kibana can be accessed by web browsers via https as well.

Before we start, make sure the following conditions are satisfied or agreed:

• Elastic services are running with no issues on the server.
• The directory layout for the Elastic stack is known. We use the directory layout of RPM in this tutorial. For further information, check this website: Directory layout of RPM
• If shell prompt asks for additional options, press ENTER to choose the default one.
• Using sudo before any command if Permission denied occurs
• If any service fails to start, check the corresponding log. If the log indicates access denied for some files, make sure you have given the access permission for the files. Check this website for further information.
• Some steps like creating directories, copying files, and editing files may be omitted.
• Read the comments for the commands carefully. Make changes for the commands according to your situation.
• Since we use self-sign CA in this tutorial, the web browser will show the certificate is not trusted in the case. It is ok, later I will write a tutorial on how to use third-party CA to make sure the certificate is trusted by the web browser. Please follow me to receive the latest notification about my articles.

Step 1: Using self-sign CA generate from Elasticsearch:

cd to the binary scripts directory. In my case, cd /usr/share/elasticsearch. Make sure you type your own domain name or IP address after CN= in the following commands

bin/elasticsearch-certutil ca --ca-dn CN=your_domain_name
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -name "CN=your_domain_name,OU=Consulting Team,DC=your_domain_name,DC=com"

At this point, you have obtained two files: elastic-certificates.p12 and elastic-stack-ca.p12. Now we are going to set up passwords for built-in users. Make sure the passwords are recorded and protected in a safe place. Once the password is set up, it won’t allow being recovered. Use bin/elasticsearch-setup-passwords auto to let the script generate the passwords or use /bin/elasticsearch-setup-passwords interactive to customize the passwords for each build-in users by yourself.

Use the following command to create a directory certs under the conf directory and copy elastic-certificates.p12 to this directory.

mkdir -p /etc/elasticsearch/certs
cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/certs

Add the following lines in the Elasticsearch configuration file. In my case, the path is /etc/elasticsearch/elasticsearch.yml .

http.host: 0.0.0.0 # accept request from remote
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

Restart Elasticsearch service to make sure the above changes take effect and no error occurred at this point.

Step 2: Enable cross-origin resource sharing for Elasticsearch

Enable CORS so that a browser on another origin can send requests to Elasticsearch. Add the following lines in the Elasticsearch configuration file. In my case, the path is /etc/elasticsearch/elasticsearch.yml .

Attention, for the setting http.cors.allow-origin , it should be the URL of the website which you are giving permission to connect. In my case, I will use "https://ai.query.ai" .

http.cors.enabled: true
http.cors.allow-origin: "https://ai.query.ai"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : Authorization, X-Requested-With,X-Auth-Token,Content-Type, Content-Length

Step 3: Encrypt communication between Kibana and web browser:

cd to certs folder under the Elasticsearch conf directory. Run the following commands to obtain the key file and certificate file from elastic-certificates.p12 .

openssl pkcs12 -in /elastic-certificates.p12 -out newfile.crt.pem -clcerts -nokeys
penssl pkcs12 -in ./elastic-certificates.p12 -out newfile.key.pem -nocerts -nodes

At this point two new files are obtained: newfile.crt.pem and newfile.key.pem . Create a directory named certs under Kibana conf folder, and copy these two files to certs directory. Add the following lines to the Kibana configuration file to enable HTTP SSL for Kibana. In my case, it is located at /etc/kibana/kibana.yml

server.host: 0.0.0.0
server.ssl.enabled: true
server.ssl.key: /etc/kibana/certs/newfile.key.pem
server.ssl.certificate: /etc/kibana/certs/newfile.crt.pem

Step 4: Encrypt communication between Kibana and Elasticsearch:

See step1, we’ve obtained the password for user Kibana. Add these lines to the Kibana configuration file to enable TLS between Elasticsearch and Kibana:

elasticsearch.hosts: ["https://localhost:9200"] # https, not http
elasticsearch.username: "kibana"
elasticsearch.password: "XXXXXXXXX" # the password for user kibana
elasticsearch.ssl.verificationMode: none

Restart Kibana service, access the Kibana UI on your web browser. Make sure the Kibana UI shows up and is connected by https.

Step 5: Encrypt communication between Logstash and Elasticsearch:

There are configurations related to Elasticsaerch in Logstash’s pipeline. Since we’ve enabled https and built-in users for Elasticsearch. We should do the following changes in every pipeline config file in order to make the connection to Elasticsearch via https.

First, create a directory called certs under Logstash conf directory. In my case, it is located at /etc/logstash . Then, copy newfile.crt.pem to the certs directory.

Make changes on Elasticsearch output plugin for every Logstash pipeline

output {
elasticsearch {
hosts => "https://localhost:9200" #make sure it's https
index => "nginx-%{+YYYY.MM.dd}"
document_type => "nginx_logs"
user => logstash_system 
password => xxxxxxxxx # obtained by step1
cacert => '/etc/logstash/certs/newfile.crt.pem' # generated from step3
ssl_certificate_verification => false 
}
}

Restart Logstash in order for the changes to take effect.

Overall:

After the above changes, the communications among Elasticsearch, Kibana, and Logstash are encrypted. Also, you can access Elasticsearch and Kibana by the web browser using https. These features are performed through a tool called, X-pack, which was enabled for free in the Elastic update 7.1 at the end of May.

About me:

I am a software developer working at Query.AI ( http://query.ai ). Query.AI provides answers and insights on IT and cybersecurity data residing in log and SIEM platforms like Splunk and ELK / Elasticsearch.