Cloud Storage with Red Hat Ceph Storage

After completing this section, you should be able to describe the purpose of cluster monitors and the quorum procedures, query the monitor map, manage the configuration database, and describe Cephx.

Ceph monitors (MONs) store and maintain the cluster map that clients use to find the MON and OSD nodes. Ceph clients must connect to a MON to retrieve the cluster map before they can read or write any data to OSDs. For this reason, the proper configuration of cluster MONs is critical.

MONs form a quorum and elect a leader by using a variation of the Paxos algorithm, to achieve consensus among a distributed set of computers.

MONs each have one of the following roles:

Synchronization always occurs when a new MON joins the cluster. Each MON periodically checks whether a neighboring monitor has a more recent version of the cluster map. If a MON does not have the most recent version of the cluster map, then it must synchronize and obtain it.

A majority of the MONs in a cluster must be running to establish a quorum. For example, if five MONs are deployed, then three must be running to establish a quorum. Deploy at least three MON nodes in your production Ceph cluster to ensure high availability. You can add or remove MONs in a running cluster.

The cluster configuration file defines the MON host IP addresses and ports for the cluster to operate. The mon_host setting can contain IP addresses or DNS names. The cephadm tool does not update the cluster configuration file. Define a strategy to keep the cluster configuration files synchronized across cluster nodes, such as with rsync.

[global]
mon_host = [v2:172.25.250.12:3300,v1:172.25.250.12:6789],[v2:172.25.250.13:3300,v1:172.25.250.13:6789],[v2:172.25.250.14:3300,v1:172.25.250.14:6789]

Verify the MON quorum status by using the ceph status or ceph mon stat commands.

[ceph: root@node /]# ceph mon stat
e4: 4 mons at {nodea=[v2:172.25.250.10:3300/0,v1:172.25.250.10:6789/0], nodeb=[v2:172.25.250.12:3300/0,v1:172.25.250.12:6789/0], nodec=[v2:172.25.250.13:3300/0,v1:172.25.250.13:6789/0]}, election epoch 66, leader 0 nodea, quorum 0,1,2 nodea,nodeb,nodec

Alternately, use the ceph quorum_status command. Add the -f json-pretty option to create a more readable output.

[ceph: root@node /]# ceph quorum_status -f json-pretty
{
    "election_epoch": 5,
    "quorum": [
        0,
        1,
        2
    ],
    "quorum_names": [
        "nodea",
        "nodeb",
        "nodec"
    ],
    "quorum_leader_name": "nodea",
    "quorum_age": 1172,
    "features": {
...output omitted...

You can also view the status of MONs in the Dashboard. In the Dashboard, click Cluster → Monitors to view the status of the Monitor nodes and quorum.

The Ceph cluster map contains the MON map, OSD map, PG map, MDS map, and CRUSH map.

The MON map contains the cluster fsid (File System ID), and the name, IP address, and network port to communicate with each MON node. The fsid is a unique, auto-generated identifier (UUID) that identifies the Ceph cluster.

The MON map also keeps map version information, such as the epoch and time of the last change. MON nodes maintain the map by synchronizing changes and agreeing on the current version.

Use the ceph mon dump command to view the current MON map.

[ceph: root@node /]# ceph mon dump
epoch 4
fsid 11839bde-156b-11ec-bb71-52540000fa0c
last_changed 2021-09-14T14:54:23.611787+0000
created 2021-09-14T14:50:37.372360+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:172.25.250.12:3300/0,v1:172.25.250.12:6789/0] mon.serverc
1: [v2:172.25.250.13:3300/0,v1:172.25.250.13:6789/0] mon.serverd
2: [v2:172.25.250.14:3300/0,v1:172.25.250.14:6789/0] mon.servere
dumped monmap epoch 4

The MON nodes store and maintain the centralized configuration database. The default location of the database on each MON node is /var/lib/ceph/$fsid/mon.$host/store.db. It is not recommended to change the location of the database.

The database might grow large over time. Run the ceph tell mon.$id compact command to compact the database to improve performance. Alternately, set the mon_compact_on_start configuration to true to compact the database on each daemon start:

[ceph: root@node /]# ceph config set mon mon_compact_on_start true

Define threshold settings that trigger a change in health status based on the database size.

Ceph uses the Cephx protocol by default for cryptographic authentication between Ceph components, using shared secret keys for authentication. Deploying the cluster with cephadm enables Cephx by default. You can disable Cephx if needed, but it is not recommended because it weakens cluster security. To enable or disable the Cephx protocol, use the ceph config set command to manage multiple settings.

[ceph: root@node /]# ceph config get mon auth_service_required
cephx
[ceph: root@node /]# ceph config get mon auth_cluster_required
cephx
[ceph: root@node /]# ceph config get mon auth_client_required
cephx

The /etc/ceph directory and daemon data directories contain the Cephx key-ring files. For MONs, the data directory is /var/lib/ceph/$fsid/mon.$host/.

Use the ceph auth command to create, view, and manage cluster keys. Use the ceph-authtool command to create key-ring files.

The following command creates a key-ring file for the MON nodes.

[ceph: root@node /]# ceph-authtool --create-keyring /tmp/ceph.mon.keyring \
--gen-key -n mon. --cap mon 'allow *'
creating /tmp/ceph.mon.keyring

The cephadm tool creates a client.admin user in the /etc/ceph directory, which allows you to run administrative commands and to create other Ceph client user accounts.