CEPH: Enabling one-way rbd mirror¶

Introduction¶

These notes are based on the very well written guide from RedHat and are actually a subset of it: please read the above guide for more use cases, as well as the official Ceph manual.

These notes describe setting up a one-way mirror between distinct Ceph clusters: one cluster acts as the source, and the other is the backup. In this example, mirror is enabled per-pool, but if needed you do have control over individual volumes (volume or image are used interchangeably in what follows) within the pool.

The outline of the procedure follows:

configure the pool(s) to enable mirroring and check the volumes therein are eligible to be mirrored
install the rbd-mirror package on the backup cluster
configure both clusters by adding users that mirroring will use to connect to clusters
configure mirroring by declaring the source cluster as a peer of the remote one
start rbd-mirror and enjoy

Note

Before continuing, please note that rbd-mirror only works between clusters, not intra-cluster (at least to date), and assumes the two clusters are named differently.

For this example, we imagine the source cluster to bear the default name ceph whereas the backup cluster is called cephba1.

Note

The name of the cluster is set at creation time. The only possible way to change it is scratching and recreating it.

If both clusters are named the same, you can still run rbd-mirror by using this trick with the caveat that the machine running rbd-mirror cannot be neither an OSD nor a MON, but another Ceph client in the cluster.

Mirroring mode: pool¶

Mirroring is enabled at pool level, on a pool called testfulvio, with the following command, to be repeated on both cluster:

$ rbd mirror pool enable testfulvio pool

Mirroring relies on a couple of features which need to be set for each image, namely exclusive-lock,journaling. Learn more about features at this link.

Check what features are enabled for an image at the source cluster by running (provaFG is a volume contained in pool testfulvio):

$ rbd -p testfulvio info provaFG
  rbd image 'provaFG':
      size 2048 MB in 512 objects
      order 22 (4096 kB objects)
      block_name_prefix: rbd_data.211a8a238e1f29
      format: 2
      features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
      flags:

Feature journaling is missing, add it with:

$ rbd feature enable testfulvio/provaFG journaling

For a new image you would run something like:

$ rbd create provaFG --size 2G --pool testfulvio --image-feature exclusive-lock,journaling

Once mirroring has been enabled, info on the source image will show that the volume is ready to be mirrored and is primary:

$ rbd -p testfulvio info provaFG
  rbd image 'provaFG':
      size 2048 MB in 512 objects
      order 22 (4096 kB objects)
      block_name_prefix: rbd_data.211a8a238e1f29
      format: 2
      features: exclusive-lock, journaling
      flags:
      journal: 211a8a238e1f29
      mirroring state: enabled
      mirroring global id: 4523533e-8b30-410b-a428-7a056596e68b
      mirroring primary: true

Mirroring mode: image¶

Mirroring relies on a couple of features which need to be set for each image, namely exclusive-lock,journaling. Learn more about features at this link.

Check what features are enabled for an image at the source cluster by running (provaFG is a volume contained in pool testfulvio):

$ rbd -p testfulvio info provaFG
  rbd image 'provaFG':
      size 2048 MB in 512 objects
      order 22 (4096 kB objects)
      block_name_prefix: rbd_data.211a8a238e1f29
      format: 2
      features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
      flags:

Feature journaling is missing, add it with:

$ rbd feature enable testfulvio/provaFG journaling

For a new image you would run something like:

$ rbd create provaFG --size 2G --pool testfulvio --image-feature exclusive-lock,journaling

At this point, mirroring is enabled at image level for image provaFG in pool testfulvio, with the following command:

$ rbd mirror image enable testfulvio/provaFG

Once mirroring has been enabled, info on the source image will show that the volume is ready to be mirrored and is primary:

$ rbd -p testfulvio info provaFG
  rbd image 'provaFG':
      size 2048 MB in 512 objects
      order 22 (4096 kB objects)
      block_name_prefix: rbd_data.211a8a238e1f29
      format: 2
      features: exclusive-lock, journaling
      flags:
      journal: 211a8a238e1f29
      mirroring state: enabled
      mirroring global id: 4523533e-8b30-410b-a428-7a056596e68b
      mirroring primary: true

Package installation and configuration¶

For our purposes (one-way mirroring), and with our configuration (clusters bear different names), we can use one of the MON nodes cephmon3.ba1.

Warning

Please note that mirroring should only run on a single node within the cluster.

On cephmon3.ba1 execute:

$ yum install rbd-mirror

Configure Ceph clusters¶

On each cluster, create a user with the proper permissions to access the pool testfulvio, and output its key to a file.

On the source cluster we create user mirrorct while on the backup cluster we create user mirrorba, executing:

# On source cluster, called "ceph"
ceph auth get-or-create client.mirrorct mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow pool testfulvio rwx' -o /etc/ceph/ceph.client.mirrorct.keyring --cluster ceph
# On backup cluster, called "cephba1"
ceph auth get-or-create client.mirrorba mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow pool testfulvio rwx' -o /etc/ceph/cephba1.client.mirrorba.keyring --cluster cephba1

Exchange (in our case, copy from source to backup cluster) cluster configuration file /etc/ceph/<cluster_name>.conf and mirroring user’s keyring file, and place them under /etc/ceph.

Configure and execute mirroring¶

On the backup cluster, declare the rbd-mirror service, enable and start it:

systemctl enable ceph-rbd-mirror.target
systemctl enable ceph-rbd-mirror@cephba1

On CentOS and Jewel 10.2.5, rbd-mirror will not start as you will first need to change the username associated, which apparently defaults to the cluster name. Edit file /usr/lib/systemd/system/ceph-rbd-mirror@.service, find the line with ExecStart and replace –id %i with –id mirrorba:

systemctl daemon-reload
systemctl start ceph-rbd-mirror@cephba1

Last, we need to declare, on the backup cluster, the source cluster as a peer:

rbd --id mirrorba --cluster cephba1 mirror pool peer add testfulvio client.mirrorct@ceph
rbd --id mirrorba --cluster cephba1 mirror pool info testfulvio
Mode: pool
Peers:
  UUID                                 NAME CLIENT
  1ef1ea4e-21e6-4f42-a882-09c81aec0c7f ceph client.mirrorct

If mirroring was enabled at image level, you will rather see something like:

rbd --cluster cephna2 -p cinder-ceph-ct1-cl1 mirror pool info
Mode: image
Peers:
  UUID                                 NAME CLIENT
  79e32312-a77c-430d-9c76-746870014d71 ceph client.mirrorct1

Mirroring is now running. If you create a new volume with the proper features on the source cluster you will see it pop up at the remote cluster after a while (if pool-based mirroring was chosen), otherwise (in case of image-based mirroring) you will also need to explicitly execute mirror image enable.

Image promotion and demotion¶

Command rbd info will show the mirroring status of an image, which can be either primary: true or primary: false.

To demote local image and promote remote image, execute:

rbd -p cinder-ceph-ct1-cl1 mirror image demote volume-2e1c98b2-55ad-4816-bd6b-80c3cd503294
rbd --cluster cephna2 -p cinder-ceph-ct1-cl1 mirror image promote --force volume-2e1c98b2-55ad-4816-bd6b-80c3cd503294

At this stage, if your intent was to permanently migrate image to the remote site, you can disable mirroring on both sites, executing:

rbd -p cinder-ceph-ct1-cl1 mirror image disable --force volume-2e1c98b2-55ad-4816-bd6b-80c3cd503294
rbd --cluster cephna2 -p cinder-ceph-ct1-cl1 mirror image disable volume-2e1c98b2-55ad-4816-bd6b-80c3cd503294