Setup Elasticsearch 7.x Cluster on Raspberry Pi / ASUS Tinker Board

Published in

hepsiburadatech

3 min readSep 23, 2019

Hardware

NODES
1x Raspberry Pi 2B    - Data Node
1x Raspberry Pi 3B    - Data Node
1x Raspberry Pi 4B    - Data Node
1x Asus Tinker Board  - Master NodeSWITCH
1x 8-Port TP-LINK 10/100MbpsPOWER HUB
1x Dark Connect MasterSDCARD
4x Samsung EVO Plus 32GBPOWER CABLE
3x Codegen Micro USB - For RPi 2B, RPi 3B and Asus Tinker Board
1x Codegen Type-C    - For RPi 4BETHERNET CABLE
4x Jadaol Cat 6 Flat Ethernet Cable 1 ftCASE
1x iUniker Raspberry Pi Cluster Case

Yes, Four different single-board computer. Seems legit!

Operating System

Raspberry Pi      -> Raspbian Buster Lite
ASUS Tinker Board -> Armbian Bionic Server

Prepare all your SDCards with that images. You can use balenaEtcher for that. And don’t forget this for Raspberry Pi devices.

First Boot

Need to expand your filesystem for all your devices. And give them cool hostnames.

# Example
rpi2b-node
rpi3b-node
rpi4b-node
tinker-node

Network

I think, Static IP better for managing our cluster. Run sudo nano /etc/network/interfaces command and set your configuration to every device.

# Example
auto eth0
iface eth0 inet static
        address 192.168.1.100 # current device ip
        netmask 255.255.255.0
        gateway 192.168.1.1 # modem gateway

Update Packages and Reboot

sudo apt-get update
sudo apt-get upgrade
sudo reboot

Summary

# Example
HOSTNAME      IP              RAM   TYPE
tinker-node - 192.168.1.100 - 2GB - Master
rpi4b-node  - 192.168.1.101 - 4GB - Data
rpi3b-node  - 192.168.1.102 - 1GB - Data
rpi2b-node  - 192.168.1.103 - 1GB - Data

Prepare Java

In this article, we will setup Elasticsearch 7.3.2. So we need to install JDK11 to every device.

sudo apt-get install openjdk-11-jdk          # Raspberry Pi
sudo apt-get install openjdk-11-jre-headless # ASUS Tinker Board

And set JAVA_HOME with sudo nano /etc/environment

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-armhf

And run source /etc/environment

Setup Elasticsearch

We need to install Elasticsearch manually, because latest version do not support our distribution.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.2-no-jdk-amd64.deb
sudo dpkg -i --force-all --ignore-depends=libc6 elasticsearch-7.3.2-no-jdk-amd64.deb

After that Elasticsearch installed wrongly and APT gonna broken. So we need to fix APT before everything.

Run sudo nano /var/lib/dpkg/status and find Package: elasticsearch line.

So change Status: install ok half-configured line to Status: install ok installed and change Depends: bash (>= 4.1), lsb-base (>= 4), libc6, adduser, coreutils (>= 8.4) line to Depends: bash (>= 4.1), lsb-base (>= 4), adduser, coreutils (>= 8.4)

Basically change half-configured to installed and remove libc6 from Depends

And run sudo apt-get upgrade to check everything is okay.

After that, we need to set JAVA_HOME for Elasticsearch with sudo nano /etc/default/elasticsearch

Uncomment JAVA_HOME line and set the value /usr/lib/jvm/java-11-openjdk-armhf

run sudo chmod g+w /etc/elasticsearch command for avoid keystore permission issues.

So our Elasticsearch nodes installed. Now we can configure it before to run.

Elasticsearch Configuration

Run sudo nano /etc/elasticsearch/elasticsearch.yml

Change cluster.name (same for all nodes) and node.name (different for all nodes) to what you want.

change discovery.seed_hosts and cluster.initial_master_nodes for your cluster

Add following line to end of file

# Data Nodes Example
xpack.ml.enabled: false
node.master: false
node.data: true
discovery.seed_hosts: ["192.168.1.100", "192.168.1.101", "192.168.1.102", "192.168.1.103"] # all node ips
http.host: 192.168.1.101 # current node ip
transport.host: 192.168.1.101 # current node ip
bootstrap.system_call_filter: false
cluster.initial_master_nodes: ["tinker-node"] # master node node.name# Master Node Example
xpack.ml.enabled: false
node.master: true
node.data: false
discovery.seed_hosts: ["192.168.1.100", "192.168.1.101", "192.168.1.102", "192.168.1.103"] # all node ips
transport.host: 192.168.1.100 # current node ip
http.host: 192.168.1.100 # current node ip
bootstrap.system_call_filter: false

And maybe you need to avoid memory issues for RPi 2B and 3B. Cause that devices have low memory.

For 2B and 3B, run sudo nano /etc/elasticsearch/jvm.options command. And change

-Xms1g to -Xms512M
-Xmx1g to -Xmx512M

So our Elasticsearch cluster ready.

Start Cluster and Check It

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
curl 192.168.1.100:9200/_cat/nodes

After cluster is ready, I run our qa products ingestion process to cluster with Apache Spark (~600K document). Cluster face with heavy load but after all, our job successfully done!

In conclusion, this cluster ready for our personal projects! And its cool lab environment.

Have fun :)

Post-Install

You can monitor your cluster on Kibana with docker installed computer. (not arm)

# Example
docker run -d -p 5601:5601 \
-e "ELASTICSEARCH_HOSTS=http://192.168.1.100:9200" \
docker.elastic.co/kibana/kibana:7.3.1

Troubleshooting

# /var/log/elasticsearch permission issue solutionsudo chown -R elasticsearch:elasticsearch /var/log/elasticsearch
systemctl restart elasticsearch.service