Deploying Cinder with Multiple Ceph Cluster Backends

This article is about utilizing multiple Ceph clusters as cinder backends. The purpose of this use is to allow instances to attach block volumes from either or both Ceph clusters.


Cinder is tasked with creating the block volumes and providing connection parameters for those volumes to nova. The nova-compute service calls Cinder’s api to reserve and request a specific volume, and Cinder supplies a response with the following information: driver_volume_type, secret_type, secret_uuid, hosts, name, volume_id, auth_enabled, auth_username, and ports. This information likely differs between different volume types, but you can see an example response for a Ceph volume below:

Nova supplies these settings to libvirt by converting them to the appropriate xml tags and parameters.

<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='cinder'>
<secret type='ceph' uuid='secret-uuid'/>
<source protocol='rbd' name='cinder_volumes/volume-uuid'>
<host name='mon_host_ip' port='mon_host_port'/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>


Create the appropriate cinder user on your new Ceph cluster. The username must be different from the existing Ceph cluster cinder username(I am using ‘cinder2’ in this example).

Add the following section to cinder.conf

rbd_secret_uuid=<secret uuid>

Replace the <secret uuid> with a new uuid. This uuid must be different than any existing uuid’s used by cinder/libvirt.

Grab the ceph.conf file from one of your new cluster’s mon hosts save it to your cinder hosts as /etc/ceph-new.conf and add the following section:

keyring = /etc/ceph/ceph-new.client.cinder2.keyring

Grab the key for the cinder2 user from the new Ceph cluster and save it to /etc/ceph/ceph-new.client.cinder2.keyring

Store just the key for the user in a file:

printf 'key here' > cinder2.key

Note: use printf, as echo will insert a \n into your file, and you don’t want that.

Create a new nova-secret.xml file with the following contents:

<secret ephemeral="no" private="no">
<uuid>secret uuid here</uuid>
<usage type="ceph">
<name>client.cinder2 secret</name>

Once again, insert the uuid you created above.

Next, we need to create the secret and set the key in libvirt using virsh:

virsh secret-define --file nova-secret.xml
virsh secret-set-value --secret <secret uuid> --base64 $(cat cinder2.key)

Restart cinder-volumes, and you should be in business. You probably want to cleanup the cinder2.key file and the nova-secret.xml file that you created.

That’s it!

You’re done! Of course, you will need to create new cinder volume types that utilize the new backend, but that’s covered adequately elsewhere.

Of course, if you’re using OpenStack-Ansible these steps are now automated for you in the Newton release. Refer to the Ceph Client Role documentation for more information.