Kubernetes: Installing MongoDB ReplicaSet on Azure using Helm
It’s time to install a MongoDB ReplicaSet on a Kubernetes cluster on Azure and try to kill it in all possible ways!
Starring:
Azure Kubernetes Cluster configuration
Let’s install a Cluster with 1 Master and 3 Nodes all running Linux using ACS Engine with following commands.
I described detailed steps on ACS Engine usage for installing a cluster in my Kubernetes Adventures on Azure — Part 3 (ACS Engine & Hybrid Cluster) article. Please refer to it for details. Here I list quickly commands to be used.
Creation of Resource Group
This is needed to group all resources for this tutorial in a single logical group, to be able to delete everything with a single command at the end.
az group create --name k8sMongoTestGroup --location westeurope
Cluster provisioning
I usually create an ssh pair for my test on acs. Please check my article here on how to do it. I will change the examples/kubernetes.json file to use previously created ssh pair, dnsPrefix and servicePrincipalProfile (following Deploy a Kubernetes Cluster suggestions from Microsoft).
my kubernetes.json file with changes in bold is:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.7"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "ivank8stest",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa yourpubkeyhere"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "yourappclientid",
"secret": "yourappsecrete"
}
}
}
Create your cluster with:
acs-engine deploy --subscription-id c8399910-5ac0-4dc4-98ec-c7774baf9a02 \
--resource-group k8sMongoTestGroup \
--location westeurope \
--api-model examples/kubernetes.json
Wait for the cluster to be up running:
INFO[0010] Starting ARM Deployment (k8sMongoGroup-2051810234). This will take some time…
INFO[0651] Finished ARM Deployment (k8sMongoGroup-2051810234).
Connect to it using kubeconfig file generated during deployment in _output folder.
export KUBECONFIG=~/acs/acs-engine/_output/ivank8stest/kubeconfig/kubeconfig.westeurope.json
Following commands can be used to determine when cluster is ready:
kubectl cluster-info
kubectl get nodesNAME STATUS AGE VERSION
k8s-agentpool1-33584487-0 Ready 46m v1.7.4
k8s-agentpool1-33584487-1 Ready 46m v1.7.4
k8s-agentpool1-33584487-2 Ready 46m v1.7.4
k8s-master-33584487-0 Ready 46m v1.7.4
Now you can open Kubernetes Dashboard if you want to use a UI to check you cluster status: kubectl proxy
and then open a browser at http://127.0.0.1:8001/ui
Helm MongoDB Charts
Helm is the Package Manager for Kubernetes. It simplifies installation and maintenance of products and services like:
- MongoDB
- Redis
- RabbitMQ
- and many other
We will use it to install and configure a MongoDB Replica Set.
Helm installation
Prerequisites, installation steps and details can be found in the Use Helm to deploy containers on a Kubernetes cluster article from Microsoft.
With all prerequisites in place, Helm installation is as simple as running command:
helm init --upgrade
Clone charts repository
Let’s clone charts repository to be able to examine and change MongoDB chart files before deploying everything on our cluster:
git clone https://github.com/kubernetes/charts.git
Now go in the /charts/stable/mongodb-replicaset
folder.
Here you will find all artifacts composing an Helm Chart. If needed you can change values.yaml
file to tailor installation based on your needs. For now let’s try a standard installation.
Run following command: helm install .
and wait for following output:
NAME: foppish-angelfish
LAST DEPLOYED: Sun Sep 10 20:42:42 2017
NAMESPACE: default
STATUS: DEPLOYEDRESOURCES:
==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
foppish-angelfish-mongodb-replicaset None <none> 27017/TCP 5s==> v1beta1/StatefulSet
NAME DESIRED CURRENT AGE
foppish-angelfish-mongodb-replicaset 3 1 5s==> v1/ConfigMap
NAME DATA AGE
foppish-angelfish-mongodb-replicaset 1 5s
foppish-angelfish-mongodb-replicaset-tests 1 5sNOTES:
...
DONE!
MongoDB Replicaset is up and running! Helm is ultra easy and powerful!
MongoDB installation test
From output of helm install pick up the NAME of your release and use it in the following command:
export RELEASE_NAME=foppish-angelfish
Here we follow a different path from Helm Chart. Let’s open an interactive shell session with remote Mongo server!
kubectl exec $RELEASE_NAME-mongodb-replicaset-0 --mongo --shellOUTPUTMongoDB shell version v3.4.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.8
type "help" for help
...
...
A LOT OF WARNING (I will check these in a future post to clean them up, if possible)
...
...
rs0:PRIMARY>
Who is the primary?
In theory Pod 0 should be the master as you can see rom rs:PRIMARY> prompt. If this is not the case tun following command to find the Master:
rs0:SECONDARY> db.isMaster().primary
Take note of the primary Pod because we are going to kill it soon and connect to it using last kubectl exec used above.
Failover testing
We need to create some data to check persistence across failures. We are already connect to the mongo shell, creating a document and leaving the session is as simple as:
rs0:PRIMARY> db.test.insert({key1: 'value1'})
rs0:PRIMARY> exit
Use following command to monitor changes in replica set:
kubectl run --attach bbox --image=mongo:3.4 --restart=Never --env="RELEASE_NAME=$RELEASE_NAME" -- sh -c 'while true; do for i in 0 1 2; do echo $RELEASE_NAME-mongodb-replicaset-$i $(mongo --host=$RELEASE_NAME-mongodb-replicaset-$i.$RELEASE_NAME-mongodb-replicaset --eval="printjson(rs.isMaster())" | grep primary); sleep 1; done; done';OUTPUT
foppish-angelfish-mongodb-replicaset-0 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",
foppish-angelfish-mongodb-replicaset-1 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",
foppish-angelfish-mongodb-replicaset-2 "primary" : "foppish-angelfish-mongodb-replicaset-0.foppish-angelfish-mongodb-replicaset.default.svc.cluster.local:27017",
...
...
...
Kill the primary!
Here it is: kubectl delete pod $RELEASE_NAME-mongob-replicaset-0
MongoDB will start an election and another Pod will become master:
foppish-angelfish-mongodb-replicaset-1 “primary” : “foppish-angelfish-mongodb-replicaset-1.foppish-angelfish-mongodb-
And in the meantime Kubernetes will immediately take corrective actions instantiating a new Pod 0.
Kill’em all
Now we have to simulate a real disaster: let’s kill all Pods and see the StatefulSet magically recreating everything with all data available.
kubectl delete po -l "app=mongodb-replicaset,release=$RELEASE_NAME"
kubectl get po --watch-only
After few minutes our MongoDB replicaset will be back online and we can test it again to see if our data are still there.
Final check and clean up
Run following command to verify that key created is still there:
kubectl exec $RELEASE_NAME-mongodb-replicaset-1 --mongo --eval=”rs.slaveOk(); db.test.find({key1:{\$exists:true}}).forEach(printjson)”
As always you can delete everything with a simple Azure CLI 2 command: az group delete --name k8sMongoTestGroup --yes --no-wait
How to expose this replicaSet externally?
This is a topic for a future post. It seems trivial as a kubectl expose
but it is not so easy.
If you expose the existing service, it will be LoadBalanced on 3 nodes behind it and this is wrong. We need 3 LoadBalancers, exposing 3 services, 1 for each Pod in the StatefulSet. Moreover we have to activate Authentication and SSL to ensure best practice from security perspective.
I will find best way to do it, while playing with Heml, Kubernetes, MongoDB and Azure!