Ceph-Ansible Deployment & Testing Using Vagrant (Part 1)
What is and why Ceph?
In short, Ceph is an open-source, software-defined storage (SDS) solution for providing highly scalable object, block & file-based storage from a single system.
Ceph is designed to be a multi-purpose, distributed software and more to be a “One Stop Solution” to overcome the ordinary enterprise's datacenter architecture following the scaling out perception.
The Ceph cluster can work combining these essential nodes and more:
- Monitors (MON) maintain a map of the cluster nodes state.
- Managers (MGR) run alongside monitors to daemons & provides external monitoring and management interface.
- Metadata Servers (MDS) store the cluster’s metadata mainly used for file-based storage.
- Object Storage Devices (OSD), the actual storage drives.
While using Red Hat’s Ceph Storage, I have investigated the many deployments available to use and deploy Ceph, mainly for testing purposes. The list of available deployments is pretty large and contains 7 different deployment options as of now.
One of the most known and popular deployments today is Ceph-Ansible. The Ceph-Ansible Deployment is best known for its versatility to update & upgrade easily the cluster used with.
In this two-part demo, I will focus on the Ceph-Ansible deployment method and demonstrate how we can easily perform various day-to-day tasks, such as Ceph’s main components configuration, using Vagrant & Virtualbox as a provider.
In this procedure, I will deploy an upstream version of Ceph-ansible (Nautilus), using the stable version 4.0 of the community project’s Github repository.
Prerequisites:
- RHEL 8
- Vagrant 2.2
- VirtualBox 6.1
- Ansible 2.9
within this procedure, the mentioned below VMs deployed with 1GB of RAM.
To start, will deploy the cluster with these nodes.
- 1 MONs
- 3 OSDs
- 2 RGWs
- 1 MGRs
Deployment
$ git clone https://github.com/ceph/ceph-ansible.git
$ cd ceph-ansible
$ git checkout stable-4.0After cloning the community repository to our host machine, make a copy of the “vagrant_variables_yml” file and change the following:
$ cp vagrant_variables.yml.sample vagrant_variables.yml
$ cp site.yml.sample site.yml
$ vim vagrant_variables.ymlAs of the beginning of the article, we want to deploy the mentioned above VM’s. We should update the vagrant variables file as follows (no need to edit any of the rest.):
mon_vms: 1
osd_vms: 3
mds_vms: 0
rgw_vms: 1
nfs_vms: 0
grafana_server_vms: 0
rbd_mirror_vms: 0
client_vms: 1
iscsi_gw_vms: 0
mgr_vms: 1Now that our Vagrant variables file is set with our demands, we can start setting our Ceph configurations by editing the “groups_vars/all.yml” file with the following:
---
ceph_origin: repository
ceph_repository: community
ceph_stable_release: nautilus
public_network: "192.168.42.0/24"
cluster_network: "192.168.43.0/24"
monitor_interface: eth1
devices:
- '/dev/sda'
- '/dev/sdb'
dashboard_enabled: FalseWhen done editing the needed files, run the next command from your ceph-ansible folder to start the cluster deployment:
[root@RHEL ceph-ansible]# vagrant up --provisionThis configuration deploys an upstream version of Ceph Nautilus with an LVM method for OSD’s daemons. As seen down below, this configuration contains 3 OSD VM’s containing 2 OSD’s per VM, making it a 6 OSD Ceph cluster. Also, take a look at other services running, such as the Monitor(MON), Manager(MGR), and the Rados Gateway (RGW).
Expending Ceph Cluster Capacity
The expansion of the cluster capacity can be done by adding a new OSD server or adding a new storage device to an existing OSD server. The expansion of the capacity can be done in two ways, automated or manual. (Make sure to add an additional storage device before trying the following)
both manually or automatically available cases performed on my testing cluster.
To add a new OSD storage daemon manually, we first have to create a Logical volume addressed to the new OSD daemon. We can do so by performing the following, resulting in a Physical Volume:
[vagrant@osd1 vagrant]$ vagrant ssh osd1
[vagrant@osd1 vagrant]$ su
[root@osd1 vagrant]# pvs
PV VG Fmt Attr PSize PFree
/dev/sdb ceph-51819e22-b1b9-4d6b-955b-22d9724569f1 lvm2 a-- <10.74g 0
/dev/sdc ceph-c3307bd4-3b8b-4e40-8756-7cddfed1d34a lvm2 a-- <10.74g 0
[root@osd1 vagrant]# parted /dev/sdd
GNU Parted 3.1
Using /dev/sdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel
New disk label type? gpt
(parted) mkpart
Partition name? []? ceph
File system type? [ext2]? xfs
Start? 2048s
End? 10.8GB
(parted) p
Model: VBOX HARDDISK (scsi)
Disk /dev/sdd: 11.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:Number Start End Size File system Name Flags
1 1049kB 10.8GB 10.8GB cephWhen a Physical Volume exists, we can start creating a Volume Group followed by a Logical Volume and add the OSD daemon.
[root@osd1 vagrant]# pvs
PV VG Fmt Attr PSize PFree
/dev/sdb ceph-51819e22-b1b9-4d6b-955b-22d9724569f1 lvm2 a-- <10.74g 0
/dev/sdc ceph-c3307bd4-3b8b-4e40-8756-7cddfed1d34a lvm2 a-- <10.74g 0
/dev/sdd1 lvm2 --- <10.06g <10.06g
[root@osd1 vagrant]# vgcreate ceph /dev/sdd1
Volume group "ceph" successfully created
[root@osd1 vagrant]# vgs
VG #PV #LV #SN Attr VSize VFree
ceph 1 0 0 wz--n- 10.05g 10.05g
ceph-51819e22-b1b9-4d6b-955b-22d9724569f1 1 1 0 wz--n- <10.74g 0
ceph-c3307bd4-3b8b-4e40-8756-7cddfed1d34a 1 1 0 wz--n- <10.74g 0
[root@osd1 vagrant]# lvcreate -n ceph -L 10GB ceph
Rounding up size to full physical extent 10.05 GiB
Logical volume "lvol0" created.
[root@osd1 vagrant]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
ceph ceph -wi-a----- 10.05g
osd-block-865339b0-91fb-47f1-8fe0-abe0c9682b68 ceph-51819e22-b1b9-4d6b-955b-22d9724569f1 -wi-ao---- <10.74g
osd-block-b69b3062-fa9f-417e-bd62-2f6cc7f28bee ceph-c3307bd4-3b8b-4e40-8756-7cddfed1d34a -wi-ao---- <10.74gNow that we have a Logical Volume configured on top of our new device, we are ready to start adding the Ceph OSD daemon to our new machine with the next commands:
[root@osd1 vagrant]# ceph-volume lvm prepare --bluestore --data ceph/ceph
[root@osd1 vagrant]# ceph-volume lvm activate --all
[vagrant@mon0 ~]$ sudo ceph -s
cluster:
id: 6bf530b3–64b7–4a09-b24e-af3202f2230a
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon0 (age 22m)
mgr: mgr0(active, since 20m)
osd: 7 osds: 7 up (since 20m), 6 in (since 20m)
rgw: 1 daemon active (rgw0.rgw0)
data:
pools: 4 pools, 128 pgs
objects: 12 objects, 1.2 KiB
usage: 6.0 GiB used, 58 GiB / 64 GiB avail
pgs: 128 active+cleanAs we can see, a new OSD is up and running successfully using the manual method.
The second and automated way is by modifying the “groups_vars/all.yml” file and adding a new line with a dash (-) followed by the storage device location path like so:
devices:
- '/dev/sda'
- '/dev/sdb'
- '/dev/sdc'After the modification is done, head over to the ceph-ansible home folder and re-run the site.yml ansible-playbook to update the cluster configuration using the vagrant command:
[razmab@RHEL ceph-ansible]$ vagrant provisionNext, ssh to MON0 VM and check the results.
[root@RHEL ceph-ansible]# vagrant ssh mon0
[vagrant@mon0 vagrant]$ su
[root@mon0 ~]$ sudo ceph -s
[vagrant@mon0 ~]$ sudo ceph -s
cluster:
id: 6bf530b3-64b7-4a09-b24e-af3202f2230a
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon0 (age 7h)
mgr: mgr0(active, since 6h)
osd: 8 osds: 8 up (since 5h), 8 in (since 5h)
rgw: 1 daemon active (rgw0.rgw0)
task status:
data:
pools: 4 pools, 128 pgs
objects: 187 objects, 1.2 KiB
usage: 9.1 GiB used, 85 GiB / 94 GiB avail
pgs: 128 active+cleanMultiple RGW’s Instances
Now let’s allow 2 Rados Gateway daemons per one RGW server. This can be done by modifying the once again the groups_vars/all.site.yml file and add the next bold line.
A multi-instance of RGW can provide us with a boost in performance in read/write workload performance. Read more here.
This configuration sets 2 instances of RGW daemons per RGW server, the default for that is 1.
---
ceph_origin: repository
ceph_repository: community
ceph_stable_release: nautilus
public_network: "192.168.42.0/24"
cluster_network: "192.168.43.0/24"
monitor_interface: eth1
devices:
- '/dev/sda'
- '/dev/sdb'
- '/dev/sdc'
dashboard_enabled: False
radosgw_num_instances: 2After the above updated, re-run the site.yml playbook again as shown earlier and see the results:
[vagrant@mon0 ~]$ sudo ceph -s
cluster:
id: 6bf530b3-64b7-4a09-b24e-af3202f2230a
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon0 (age 7h)
mgr: mgr0(active, since 6h)
osd: 8 osds: 8 up (since 5h), 8 in (since 5h)
rgw: 2 daemons active (rgw0.rgw0, rgw0.rgw1)
task status:
data:
pools: 4 pools, 128 pgs
objects: 187 objects, 1.2 KiB
usage: 9.1 GiB used, 85 GiB / 94 GiB avail
pgs: 128 active+cleanMonitor Daemons Increase
As most day-to-day operations with ceph-ansible, adding a monitor daemon is relatively easy and can be automated using a simple pre-prepared ansible-playbook. In our case, we would like to add a MON daemon to the MGR node. We will edit the vagrant ansible inventory and add the MGR node hostname (“mgr0”) underneath the “[mons]” line.
[root@RHEL ceph-ansible]# vim .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory
[mons]
mon0
mgr0[osds]
osd0
osd1
osd2[rgws]
rgw0[mgrs]
mgr0
After adding the hostname, save the changes and run the “add-mon.yml” playbook, using the node’s private-key to apply the changes.
[root@RHEL ceph-ansible]# ansible-playbook --private-key=~/.vagrant.d/insecure_private_key -u vagrant -i ~/Desktop/ceph-ansible/.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory infrastructure-playbooks/add-mon.yml -e mons
[vagrant@mon0 ~]$ sudo ceph -s
cluster:
id: 6bf530b3-64b7-4a09-b24e-af3202f2230a
health: HEALTH_OK
services:
mon: 2 daemons, quorum mon0,mgr0 (age 100s)
mgr: mgr0(active, since 79s)
osd: 9 osds: 9 up (since 20s), 9 in (since 20h)
rgw: 2 daemons active (rgw0.rgw0, rgw0.rgw1)
task status:
data:
pools: 4 pools, 128 pgs
objects: 219 objects, 1.2 KiB
usage: 9.1 GiB used, 85 GiB / 94 GiB avail
pgs: 128 active+cleanA new MON daemon has been added, making it 2 MON & 2 RGW daemons in the cluster while making it more redundant for failures.
Conclusion
This is it for the deployment & day-to-day tasks section of this first-part demonstration. The second part will discuss and cover more on Ceph object storage abilities using the rados gateway while running HAProxy load-balancer and Keepalived in the background to prevent the cluster's failover.
