Install and Configure OpenSearch for Bitbucket Data Center on Centos8 RHEL8

As git provides great flexibility to create and track versions of software applications, managing this in a crowded company requires some infrastructure like Bitbucket Data Center.

After a couple of months, I finally set up a Bitbucket Data Center in a cluster as my first task in my career. This post will be one of the prerequisites of Bitbucket Data Center in a Cluster; Shared Search Server as a single node.

There are two options for search server. One is ElasticSearch and the other is Opensearch. I chose OpenSearch since bitbucket officially will not support ElasticSearch after Bitbucket 9.0. So relying on it is not a good idea.

One more advise before we start; at first glance, documentation looks great to follow. That is true; yet it is still need further explanation for the integration process with Bitbucket.

Do not go on to the Test OpenSearch section

Because when we run the test script, it automatically give you an up and running instance of OpenSearch. However, this also creates some test ready configuration files which will be required to change. These configuration files are applicable only on the first start and it is harder to configure OpenSearch at a later time. Also, this instance is not operable in a production environment. Since it requires further TLS configuration, you would not try to serve your service with public certificates, I guess.

We will install on a remote RHEL8 and this tutorial will assist you along the way. By the way, we will need sudo privileges for the installation.

Installing OpenSearch

Bitbucket Data Center supports OpenSearch 1.3, so let start installing it on local machine,

[atlbender@shared_search_server ~]# wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.13/opensearch-1.3.13-linux-x64.tar.gz

Now, we have the tarball, transfer it to remote server reserverd for Shared Search Server,

[atlbender@shared_search_server ~]# scp /your/path/to/tarball/opensearch-1.3.13-linux-x64.tar.gz <USERNAME>@<SEARCH_SERVER>:/var/tmp

Since we transfered the tarball to server, we will go on installation proccesses on the server.

Unpacking the Tarball

After unpacking, we will copy the files under /opt. So let’s create the target folder first,

[atlbender@shared_search_server ~]# mkdir -p /opt/opensearch

Here -p option lets us create the target folder if it does not exist. If exists, do nothing. Now we can unpack the tarball and copy to target folder,

[atlbender@shared_search_server ~]# tar xzvf /your/path/to/tarball/opensearch-1.3.13-linux-x64.tar.gz

[atlbender@shared_search_server ~]# cp -rp opensearch-1.3.13/* /opt/opensearch/

This will copy the content of extracted folder and its content recursively to the target.

Configure the Linux System

Disable memory paging and swapping performance on the server,

[atlbender@shared_search_server ~]# sudo swapoff -a

Now edit the sysctl config file,

[atlbender@shared_search_server ~]# sudo vi /etc/sysctl.conf

Add following line to the file,

[atlbender@shared_search_server ~]# vm.max_map_count=262144

Reload the kernel for the changes to be applied,

[atlbender@shared_search_server ~]# sudo sysctl -p

Verify that the changes applied,

[atlbender@shared_search_server ~]# cat /proc/sys/vm/max_map_count

Setting Up the Environment

Add the following lines to the end of following file,

[atlbender@shared_search_server ~]# vi /opt/opensearch/config/opensearch.yml

#Bind OpenSearch to the correct network interface. Use 0.0.0.0
# to include all available interfaces or specify an IP address
# assigned to a specific interface.
network.host: 0.0.0.0
# Unless you have already configured a cluster, you should set
# discovery.type to single-node, or the bootstrap checks will
# fail when you try to start the service.
discovery.type: single-node

Change the jvm option, set the <n> value to the half of the system’s memory,

[atlbender@shared_search_server ~]# vi /opt/opensearch/config/jvm.options

-Xms<n>g
-Xmx<n>g

Specify Java Home

OpenSeach comes bundled Java with it. We just need to introduce it to the system,

[atlbender@shared_search_server ~]# export OPENSEARCH_JAVA_HOME="/opt/opensearch/jdk"
[atlbender@shared_search_server ~]# export PATH="$OPENSEARCH_JAVA_HOME/bin:$PATH"

• It is good idea to add these introduction to /etc/environment file to enable automatic aaplication of JAVA_HOME variable. After putting the above two lines to the file, run the following command,

echo "source /etc/environment" >> ~/.bashrc

Create the System User for OpenSearch

Create opensearch user and add it to the usergroup,

[atlbender@shared_search_server ~]# adduser --system --shell /bin/bash -U --no-create-home opensearch
[atlbender@shared_search_server ~]# usermod -aG opensearch opensearch

Change the owner of application folder,

[atlbender@shared_search_server ~]# chown -R opensearch:opensearch /opt/opensearch

Here -R operation gives, recursive ownership to child files.

Create Service File for OpenSearch

[Unit]
Description=OpenSearch
Wants=network-online.target
After=network-online.target

[Service]
Type=forking
RuntimeDirectory=data

WorkingDirectory=/opt/opensearch
ExecStart=/opt/opensearch/bin/opensearch -d

User=opensearch
Group=opensearch
StandardOutput=journal
StandardError=inherit
LimitNOFILE=65535
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143
TimeoutStartSec=75

[Install]
WantedBy=multi-user.target

Reload the kernel,

[atlbender@shared_search_server ~]# systemctl daemon-reload

Configure TLS on Server Side

This part is for self signed certificate configurations. Please refer to the documentataion for certificates signed by an authority. Any certificate related files should be under /opt/opensearch/config/ ,

[atlbender@shared_search_server ~]# cd /opt/opensearch/config

Change the “<YOUR_ROOT>” below with your Root DN. Example: opensearch.google.com.tr

Create Root CA

#Create a private key for the root certificate
openssl genrsa -out root-ca-key.pem 2048

# Use the private key to create a self-signed root certificate. Be sure to
# replace the arguments passed to -subj so they reflect your specific host.
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=TR/ST=Marmara/L=Istanbul/O=ORG/OU=UNIT/CN=<YOUR_ROOT>" -out root-ca.pem -days 730

Create Admin Cert

# Create a private key for the admin certificate.
openssl genrsa -out admin-key-temp.pem 2048

# Convert the private key to PKCS#8.
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1–3DES -out admin-key.pem

# Create the CSR. A common name (CN) of "A" is acceptable because this certificate is
# used for authenticating elevated access and is not tied to a host.
openssl req -new -key admin-key.pem -subj "/C=TR/ST=Marmara/L=Istanbul/O=ORG/OU=UNIT/CN=<YOUR_ROOT>" -out admin.csr

# Sign the admin certificate with the root certificate and private key you created earlier.
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730

Change the owner of the certificates to opensearch,

[atlbender@shared_search_server ~]# chown -R opensearch:opensearch admin.* root.*

Add configurations of certificates to opensearch.yml as below;

[atlbender@shared_search_server ~]# vi /opt/opensearch/config/opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: admin.pem
plugins.security.ssl.transport.pemkey_filepath: admin-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: admin.pem
plugins.security.ssl.http.pemkey_filepath: admin-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.authcz.admin_dn:
- 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

Note: Here there is a difference in admin_dn. The DN specified in opensearch.yml is different than we provided to the certificate authority. After creating the certificates run to get the configured admin_dn;

[atlbender@shared_search_server ~]# openssl x509 -subject -nameopt RFC2253 -noout -in admin.pem

Add the output to opensearch.yml as the last line.

Configure TLS on Client Side

NOTE: You need to complete the “Install Bitbucket” section on Install and Bitbucket Data Center paper. I did not compete that part, you can skip to the next section on this paper to complete OpenSearch installation and configurations.

Since we configured TLS connection for OpenSearch, we need to send the public key to the client side and add it to the key store of java as follows,

Take Opensearch root-ca.pem and admin.pem to node’s /var/tmp ,

[atlbender@cluster_node ~]# scp /opt/opensearch/config/root-ca.pem /opt/opensearch/config/admin.pem <SERVER_USER>@<NODE_ADDRESS>:/var/tmp

Login to the nodes’ server and go on to next step!

Change the names opensearch-ca.cerand opensearch.cer, respectively.

[atlbender@cluster_node ~]# mv /var/tmp/root-ca.pem /var/tmp/opensearch-ca.cer
[atlbender@cluster_node ~]# mv /var/tmp/admin.pem /var/tmp/opensearch.cer

Add this certificate to keystore under /opt/atlassian/bitbucket/7.21.9/jre/lib/securitydirectory otherwise it gives an error while connecting to OpenSearch server! Here is the document!

[atlbender@cluster_node ~]# cd /opt/atlassian/bitbucket/7.21.9/jre/bin
[atlbender@cluster_node ~]# keytool -import -alias bitbucket -file /var/tmp/opensearch.cer -keystore /opt/atlassian/bitbucket/7.21.9/jre/lib/security/cacerts # passwd changeit

See you in 2 years! :)

Bitbucket Data Center Related Configuration

Add the following configurations to the /opt/opensearch/config/opensearch.yml

action.auto_create_index: ".watches,.triggered_watches,.watcher-history-*"
plugins.security.restapi.roles_enabled: ["all_access"]
plugins.security.allow_default_init_securityindex: true

Replace the /securityconfig folder describes as in the documents.

OpenSearch comes with some side scripts like for generating passwords.

Make those scripts executable,

[atlbender@shared_search_server ~]# chmod +x /opt/opensearch/plugins/opensearch-security/tools/*.sh

Create a password for the bitbucket user;

[atlbender@shared_search_server ~]# /opt/opensearch/plugins/opensearch-security/tools/hash.sh

Add the hashed password to,

[atlbender@shared_search_server ~]# vi /opt/opensearch/config/securityconfig/internal_users.yml

A friendly WARNING from the document,

When using the plugins.security.allow_default_init_securityindex: true configuration, changes to the *.yml files will not have any affect after the first startup. To modify these configurations later, you can use the securityadmin.sh as below,

[atlbender@shared_search_server ~]# /opt/opensearch/plugins/opensearch-securityadmin.sh \ 
-cacert /opt/opensearch/config/root-ca.pem \
-cert /opt/opensearch/config/admin.pem \
-key /opt/opensearch/config/admin-key.pem \
-cd /opt/opensearch/plugins/opensearch-security/securityconfig/

Start the Service

Start and enable the service we have created via the following commands,

[atlbender@shared_search_server ~]# systemctl start opensearch.service
[atlbender@shared_search_server ~]# systemctl enable opensearch.service

Test the OpenSearch

Finally, you can test your OpenSearch from Bitbucket Cluster nodes via this command,

[atlbender@cluster_node ~]# curl -X GET https://<opensearch_server>:9200 -u 'bitbucket:<password>' --insecure

You will get a response like below,

{
  "name" : "hostname-here",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "efC0ANNMQlGQ5TbhNflVPg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.3.13",
    "build_type" : "tar",
    "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32",
    "build_date" : "2022-06-30T21:31:04.823801692Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

We are done! :)

Please do not hesitate to be in contact. And I am open to any comments to make this paper more readable.