Ceph: restore cluster after losing all ceph-mon’s

Here we will describe how to restore a Ceph cluster after a disaster where all ceph-mon’s are lost. We obviously assume that the data on the OSD devices are preserved!

The procedure refers to a Ceph cluster created with Juju.

Suppose that you have lost (or removed by mistake) all ceph-mon’s. We start recreating them, i.e. we have three new ceph-mon units, unfortunately the new mon’s do not know anything of the OSD’s.

Stop all OSDs

N.B. change the unit IDs accordingly to your cluster:

juju run-action --wait ceph-osd/15 stop osds=all
juju run-action --wait ceph-osd/16 stop osds=all
juju run-action --wait ceph-osd/17 stop osds=all

Stop all MONs and MGRs

SSH to each MON and use systemctl to stop mon and mgr services, e.g:

systemctl stop ceph-mon@juju-c1a2b7-24-lxd-21.service
systemctl stop ceph-mgr@juju-c1a2b7-24-lxd-21.service

Rebuild OSD mon map

On the juju client machine, rebuild old mon map using existing OSDs with the following script, which will save result to ./mon-store:


rm -rf $ms
mkdir $ms

hosts=`juju status ceph-osd | grep 'ceph-osd/' | awk '{print $1}' | cut -d'*' -f1`

# collect the cluster map from stopped OSDs
for host in $hosts; do
   echo $host
   juju ssh $host sudo rm -rf /tmp/$ms
   juju scp -- -r $ms $host:/tmp
   juju ssh $host sudo bash <<EOF
     for osd in /var/lib/ceph/osd/ceph-*; do
        ceph-objectstore-tool  --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path /tmp/$ms
     chmod -R +r /tmp/$ms
   rm -rf $ms
   juju scp -- -r $host:/tmp/$ms .

keyring creation

Create a keyring file with the following data from the new mon’s:

  • admin key: copy content of /etc/ceph/ceph.client.admin.keyring from any mon;
  • mon key: copy content of /var/lib/ceph/mon/ceph-xxx/keyring for any mon;
  • mgr key, copy content of /var/lib/ceph/mgr/ceph-xxx/keyring for each mon.

the keyring file will look like the following:

        key = AQAr9TphSxvCFxAACOS8KkIROPsvgVCfcFjh1Q==
        caps mon = "allow *"
        key = AQAmBTthVM5wEhAALJS9IEVTuRKiHYRUztxgng==
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"
        key = AQAvBTth4HObLhAAlM140CBeuYrLRhxnuSwdKQ==
        caps mds = "allow *"
        caps mon = "allow profile mgr"
        caps osd = "allow *"
        key = AQAvBTthqwQ5LRAAISUJt9j4Qb3MZ5jn2B1SwQ==
        caps mds = "allow *"
        caps mon = "allow profile mgr"
        caps osd = "allow *"
        key = AQAtBTthU36NJBAAjMhYPoPcdDh5L6Coj2grqw==
        caps mds = "allow *"
        caps mon = "allow profile mgr"
        caps osd = "allow *"

(N.B.) if needed add proper permissions to mgr keys:

ceph-authtool keyring -n mgr.juju-74bc65-21-lxd-49 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph-authtool keyring -n mgr.juju-74bc65-21-lxd-50 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph-authtool keyring -n mgr.juju-74bc65-21-lxd-51 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'

Rebuild mon store

Copy local ./mon-store directory and keyring file from juju client machine to a mon unit, e.g. under /tmp:

juju scp -- -r mon-store.new keyring ceph-mon/51:/tmp

Log on that mon unit and rebuild mon store with the following command:

ceph-monstore-tool /tmp/mon-store/ rebuild – –keyring /tmp/keyring –mon-ids juju-74bc65-21-lxd-49 juju-74bc65-21-lxd-50 juju-74bc65-21-lxd-51

Important The order of –mon-ids is important: it needs to match “mon host” order in `/etc/ceph/ceph.conf~ on mon units, otherwise mon’s won’t start!

Therefore, look for the IP addresses in /etc/ceph/ceph.conf, e.g.:

mon host =

Compare the order with the IP addresses shown by “juju status ceph-mon”:

# juju status ceph-mon
  Unit          Workload  Agent  Machine    Public address  Ports  Message
  ceph-mon/49   active    idle   21/lxd/29             Unit is ready and clustered
  ceph-mon/50*  active    idle   22/lxd/21             Unit is ready and clustered
  ceph-mon/51   active    idle   24/lxd/21             Unit is ready and clustered

In this example, is the address of 24/lxd/21, is 21/lxd/29, is 22/lxd/21, so –mon-ids must be:

--mon-ids juju-c1a2b7-24-lxd-21 juju-c1a2b7-21-lxd-29 juju-c1a2b7-22-lxd-21

Copy mon-store on all mon’s

The ceph-monstore-tool command will rebuild the data in /tmp/mon-store.

Now copy the rebuilt /tmp/mon-store to all other mon’s units (e.g. again in /tmp).

On all mon’s units:

- mv /var/lib/ceph/mon/ceph-xxx/store.db /var/lib/ceph/mon/ceph-xxx/store.db.bak
- cp -r /tmp/mon-store/store.db /var/lib/ceph/mon/ceph-xxx/store.db
- chown ceph:ceph -R /var/lib/ceph/mon/ceph-xxx/store.db

Start services

Start all mon and mgr services from mon units:

systemctl start ceph-mon@juju-c1a2b7-24-lxd-21.service
systemctl start ceph-mgr@juju-c1a2b7-24-lxd-21.service

Start all OSDs

Start all osd’s from the client:

juju run-action --wait ceph-osd/17 start osds=all
juju run-action --wait ceph-osd/16 start osds=all
juju run-action --wait ceph-osd/15 start osds=all


Check the status of the cluster:

# ceph -s
id: 5d26e488-cf89-11eb-8283-00163e78c363
mons are allowing insecure global_id reclaim
67 pgs not deep-scrubbed in time
67 pgs not scrubbed in time

mon: 3 daemons, quorum juju-c1a2b7-24-lxd-21,juju-c1a2b7-21-lxd-29,juju-c1a2b7-22-lxd-21 (age 3m)
mgr: juju-c1a2b7-21-lxd-29(active, since 4m), standbys: juju-c1a2b7-22-lxd-21, juju-c1a2b7-24-lxd-21
osd: 6 osds: 6 up (since 78s), 6 in (since 3M)

pools: 19 pools, 203 pgs
objects: 5.92k objects, 22 GiB
usage: 69 GiB used, 23 TiB / 23 TiB avail
pgs: 203 active+clean

It happened to us that the ceph-osd’s were blocked with the message: “non-pristine devices detected”. This was because we tried to add new devices to ceph-osd:osd-devices while the cluster was down.

Here are some commands to check the status of the devices in the osd hosts:

# ceph-volume lvm list

====== osd.0 =======

 [block]       /dev/ceph-9eff9b57-a7d7-497d-b0bc-a502e5658e32/osd-block-9eff9b57-a7d7-497d-b0bc-a502e5658e32

   block device              /dev/ceph-9eff9b57-a7d7-497d-b0bc-a502e5658e32/osd-block-9eff9b57-a7d7-497d-b0bc-a502e5658e32
   block uuid                oCfbdk-nKLh-yH2G-R82v-WhbI-cg3Y-Wo3L5A
   cephx lockbox secret
   cluster fsid              5d26e488-cf89-11eb-8283-00163e78c363
   cluster name              ceph
   crush device class        None
   encrypted                 0
   osd fsid                  9eff9b57-a7d7-497d-b0bc-a502e5658e32
   osd id                    0
   osdspec affinity
   type                      block
   vdo                       0
   devices                   /dev/nvme1n1p1


Another command is lsblk:

# lsblk

nvme2n1                                                                                                 259:0    0 11.7T  0
├─nvme2n1p1                                                                                             259:2    0  3.9T  0 part

   └─ceph--5fd8f96e--2ccb--460f--87b8--359ff81cff8a-osd--block--5fd8f96e--2ccb--460f--87b8--359ff81cff8a 253:0    0  3.9T  0  lvm

In our case we need to zap (delete everything including partition table) device nvme3n1p1, the device that was added but not initialized, and then re-add it to ceph.

To recover this, firstly fix the keys on osd units. On a mon, get client.bootstrap-osd key and client.osd-upgrade:

ceph auth get client.bootstrap-osd
ceph auth get client.osd-upgrade

If not present, create them using the commands:

ceph auth get-or-create client.bootstrap-osd mon "allow profile bootstrap-osd"
ceph auth get-or-create client.osd-upgrade mon "allow command \"config-key\"; allow command \"osd tree\"; allow command \"config-key list\"; allow command \"config-key put\"; allow command \"config-key get\"; allow command \"config-key exists\"; allow command \"osd out\"; allow command \"osd in\"; allow command \"osd rm\"; allow command \"auth del\""

Replace the key value in following files on EACH osd unit:

/var/lib/ceph/bootstrap-osd/ceph.keyring <---- client.bootstrap-osd key
/var/lib/ceph/osd/ceph.client.osd-upgrade.keyring <----- client.osd-upgrade key

Those keys were created when new mons were installed.


  • ssh to the OSD’s and zap nvme3n1p1:

    !!! Please note this will DESTROY all data on nvme3n1p1 and CANNOT be recovered. !!!
    !!! Please double check before proceeding. !!!
    ceph-volume lvm zap /dev/nvme3n1p1 --destroy
  • recreate the partition:

    parted -a optimal /dev/nvme3n1 mkpart primary 0% 4268G
  • Now go back to juju client machine and use juju to remove it from juju’s internal db:

    juju run-action --wait ceph-osd/X zap-disk devices=/dev/nvme3n1p1 i-really-mean-it=true

Then check juju status.

juju status can be forced to update with:

juju run --unit ceph-osd/X 'hooks/update-status'

At this stage we should see the osd status is back to normal (green).

If not, let’s run the following commands:

juju run-action --wait ceph-osd/21 zap-disk devices=/dev/nvme3n1p1 i-really-mean-it=true
juju run-action ceph-osd/21 add-disk osd-devices=\"/dev/nvme3n1p1\""