How to Configure Fiber Channel Devices as OSD

Ceph, at least up to Jewel release, relies on the existence of symbolic links in the /dev/disk/by-partlabel and /dev/disk/by-partuuid directories: as discussed in https://patchwork.kernel.org/patch/9826353/ those links are missing for the case of Fiber Channel devices.

Note that Ceph Luminous release has (or will soon have) a completely different method to provide OSDs, which is more robust and should not incur this issue.

The fix is to create the file:

/etc/udev/rules.d/60-ceph-by-partuuid.rules

with the following contents:

# forward scsi device event to corresponding block device
ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", TEST=="block", ATTR{block/*/uevent}="change"

ACTION=="remove", GOTO="persistent_storage_end_two"

SUBSYSTEM!="block", GOTO="persistent_storage_end_two"

# skip rules for inappropriate block devices
KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|md*", GOTO="persistent_storage_end_two"

# ignore partitions that span the entire disk
TEST=="whole_disk", GOTO="persistent_storage_end_two"

# for partitions import parent information
ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"

# skip unpartitioned removable media devices from drivers which do not send "change" events
ENV{DEVTYPE}=="disk", KERNEL!="sd*|sr*", ATTR{removable}=="1", GOTO="persistent_storage_end_two"

# probe filesystem metadata of disks
KERNEL!="sr*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode"

# by-partlabel/by-partuuid links (partition metadata)
ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"
ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_NAME}=="?*", SYMLINK+="disk/by-partlabel/$env{ID_PART_ENTRY_NAME}"

LABEL="persistent_storage_end_two"

Then issue the following command in order to reload the udev rules:

# This one is for Ubuntu, for CentOS the option is --reload-rules
udevadm control --reload

Check that the symbolic links have been created:

$ ll /dev/disk/by-part*

If you are using Metal As A Service (MaaS) you can automate the patch by adding this procedure in the preseed section. In this way you will have your problem fixed just applying this fix and recommisioning the machine.

• On the MaaS client go to the preseed directory:

  cd /etc/maas/preseeds/
  

• Add to the curtin file a rule to write the file above, during the commissioning. The curtin file name follows this schema:

  {prefix}_{os}_{node_arch}_{node_subarch}_{release}_{node_name} or {prefix}_{node_arch}_{node_subarch}_{release}_{node_name}
  

You can find more information about custom node setup (preseed) in the MaaS documentation.

• Create or update the write_files section of the curtin file as follows:

  write_files:
  udev_rules:
   content: |
      # forward scsi device event to corresponding block device
      ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", TEST=="block", ATTR{block/*/uevent}="change"
  
      ACTION=="remove", GOTO="persistent_storage_end_two"
  
      SUBSYSTEM!="block", GOTO="persistent_storage_end_two"
  
      # skip rules for inappropriate block devices
      KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|md*", GOTO="persistent_storage_end_two"
  
      # ignore partitions that span the entire disk
      TEST=="whole_disk", GOTO="persistent_storage_end_two"
  
      # for partitions import parent information
      ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"
  
      # skip unpartitioned removable media devices from drivers which do not send "change" events
      ENV{DEVTYPE}=="disk", KERNEL!="sd*|sr*", ATTR{removable}=="1", GOTO="persistent_storage_end_two"
  
      # probe filesystem metadata of disks
      KERNEL!="sr*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode"
  
      # by-partlabel/by-partuuid links (partition metadata)
      ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"
      ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_NAME}=="?*", SYMLINK+="disk/by-partlabel/$env{ID_PART_ENTRY_NAME}"
  
      LABEL="persistent_storage_end_two"
  

• To make the curtin run only on a specific machine, add at the end of the file name the node name as shown in the example below:

  curtin_userdata_ubuntu_amd64_generic_xenial_ba1-r3-s15
  

• it is more comfortable and so suggested using a link:

  ln -sf curtin_userdata_ubuntu_amd64_generic_xenial_bypartuuidrule curtin_userdata_ubuntu_amd64_generic_xenial_ba1-r3-s15