Ceph clients can mount an RBD image using the native Linux kernel module, krbd. This module maps RBD images to Linux block devices with names such as /dev/rbd0.

Figure 6.1: Kernel environment access

The rbd device map command uses the krbd kernel module to map an image. The rbd map command is an abbreviated form of the rbd device map command. The rbd device unmap, or rbd unmap, command uses the krbd kernel module to unmap a mapped image. The following example command maps the test RBD image in the rbd pool to the /dev/rbd0 device on the host client machine:

[root@node ~]# rbd map rbd/test
/dev/rbd0

A Ceph client system can use the mapped block device, called /dev/rbd0 in the example, like any other block device. You can format it with a file system, mount it, and unmount it.

The rbd device list command, abbreviated rbd showmapped, lists the RBD images mapped in the machine.

[root@node ~]# rbd showmapped
id  pool  namespace image  snap   device
0   rbd             test   -      /dev/rbd0

The rbd device unmap command, abbreviated rbd unmap, unmaps the RBD image from the client machine.

[root@node ~]# rbd unmap /dev/rbd0

The rbd map and rbd unmap commands require root privileges.

The rbdmap service can automatically map and unmap RBD images to devices when booting and shutting down the system. This service looks for the mapped images with their credentials in the /etc/ceph/rbdmap file. The service mounts and unmounts the RBD images using their mount points as they appear in the /etc/fstab file.

The following steps configure rbdmap to persistently map and unmap an RBD image that already contains a file system:

Refer to rbdmap(8) for more information.

The librbd library provides direct access to RBD images for user space applications. It inherits the capabilities of librados to map data blocks to objects in the Ceph Object Store, and implements the ability to access RBD images and create snapshots and clones.

Cloud and virtualization solutions, such as OpenStack and libvirt, use librbd to provide RBD images as block devices to cloud instances and the virtual machines that they manage. For example, RBD images can store QEMU virtual machine images. Using the RBD clone feature, virtualized containers can boot a virtual machine without copying the boot image. The copy-on-write (COW) mechanism copies data from the parent to the clone when it writes to an unallocated object within the clone. The copy-on-read (COR) mechanism copies data from the parent to the clone when it reads from an unallocated object within the clone.

Figure 6.2: Virtual environment access

Because the user space implementation of the Ceph block device (for example, librbd) cannot take advantage of the Linux page cache, it performs its own in-memory caching, known as RBD caching. RBD caching behaves in a similar manner to the Linux page cache. When the OS implements a barrier mechanism or a flush request, Ceph writes all dirty data to the OSDs. This means that using write-back caching is just as safe as using physical hard disk caching with a VM that properly sends flushes (for example, Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) algorithm, and in write-back mode it can coalesce contiguous requests for better throughput.

If using write-back mode, then the librbd library caches and acknowledges the I/O requests when it writes the data into the local cache of the server. Consider write-through for strategic production servers to reduce the risk of data loss or file system corruption in case of a server failure. Red Hat Ceph Storage offers the following set of RBD caching parameters:

Run ceph config set client parameter value command or ceph config set global parameter value command for client or global, respectively.

All objects in an RBD image have a name that starts with the value contained in the RBD Block Name Prefix field of each RBD image and displayed using the rbd info command. After this prefix, there is a period (.), followed by the object number. The value for the object number field is a 12-character hexadecimal number.

[root@node ~]# rbd info rbdimage
rbd image 'rbdimage':
	size 10240{nbsp}MB in 2560 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 867cba5c2d68
	block_name_prefix: rbd_data.867cba5c2d68
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features:
	flags:
	create_timestamp: Thu Sep 23 18:54:35 2021
	access_timestamp: Thu Sep 23 18:54:35 2021
	modify_timestamp: Thu Sep 23 18:54:35 2021
[root@node ~]# rados -p rbd ls
rbd_object_map.d3d0d7d0b79e.0000000000000008
rbd_id.rbdimage
rbd_object_map.d42c1e0a1883
rbd_directory
rbd_children
rbd_info
rbd_header.d3d0d7d0b79e
rbd_header.d42c1e0a1883
rbd_object_map.d3d0d7d0b79e
rbd_trash

Ceph block devices allow storing data striped over multiple Object Storage Devices (OSD) in a Red Hat Ceph Storage cluster.

Figure 6.3: RBD layout

The image order is the size of the objects used for the RBD image. Image order defines a binary shift value based on the << (bitwise left shift) C operator. This operator shifts the left operand bits by the right operand value. For example, 1 << 2 = 4. Decimal 1 is 0001 in binary, so the result of the 1 << 2 = 4 operation is 0100 in binary, which is 4 in decimal. The value of the image order must be between 12 and 25, where 12 = 4 KiB and 13 = 8 KiB. for example. The default image order is 22, resulting in 4 MiB objects. You can override the default value by using the --order option of the rbd create command.

You can specify the size of the objects used with the --object-size option. This parameter must specify an object size between 4096 bytes (4 KiB) and 33,554,432 bytes (32 MiB), expressed in bytes, K or M (for example, 4096, 8 K or 4 M).

Each RBD image has three parameters associated with it:

For RBD format 2 images, you can change the value of each of those parameters. The settings must align with the following equation:

stripe_unit * stripe_count = object_size

For example:

stripe_unit = 1048576, stripe_count = 4 for default 4 MiB objects

Remember that object_size must be no less than 4096 bytes and no greater than 33,554,432 bytes. Use the --object-size option to specify this value when you create the RBD image. The default object_size is 4192304 bytes (4 MiB).