Cloud Storage with Red Hat Ceph Storage

When troubleshooting issues with Ceph, the first step is to determine which Ceph component is causing the problem. Sometimes, you can find this component in the information provided by the ceph health detail or ceph health status commands. Other times, you must investigate further to discover the issue. Verify a cluster's status to help determine if there is a single failure or an entire node failure.

The following troubleshooting checklist suggests next steps:

Red Hat Ceph Storage continually runs various health checks to monitor the health of the cluster. When a health check fails, the cluster health state changes to either HEALTH_WARN or HEALTH_ERR, depending on the severity and impact of the failed health checks. Red Hat Ceph Storage also logs the health check warnings and errors to the cluster logs.

The ceph status and ceph health commands show the cluster health status. When the cluster health status is HEALTH_WARN or HEALTH_ERR, use the ceph health detail command to view the health check message so that you can begin troubleshooting the issue.

[ceph: root@node /]# ceph health detail

Some health status messages indicate a specific issue; others provide a more general indication. For example, if the cluster health status changes to HEALTH_WARN and you see the health message HEALTH_WARN 1 osds down; Degraded data redundancy, then that is a clear indication of the problem.

Other health status messages might require further troubleshooting because they might indicate several possible root causes. For example, the following message indicates an issue that has multiple possible solutions:

[ceph: root@node /]# ceph health detail
HEALTH_WARN 1 pools have too few placement groups
[WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups
    Pool testpool has 8 placement groups, should have 32

You can resolve this issue by changing the pg_num setting on the specified pool, or by reconfiguring the pg_autoscaler mode setting from warn to on so that Ceph automatically adjusts the number of PGs.

Ceph sends health messages regarding performance when a cluster performance health check fails. For example, OSDs send heartbeat ping messages to each other to monitor OSD daemon availability. Ceph also uses the OSD ping response times to monitor network performance. A single failed OSD ping message could mean a delay from a specific OSD, indicating a potential problem with that OSD. Multiple failed OSD ping messages might indicate a failure of a network component, such as a network switch between OSD hosts.

You might want to temporarily mute some of the cluster warnings because you already know of them and do not need to fix them yet. For example, if you bring down an OSD for maintenance, then the cluster reports a HEALTH_WARN status. You can mute this warning message so that the health check does not affect the overall reported status.

Ceph specifies the health check alert by using health check codes. For example, the previous HEALTH_WARN message shows the POOL_TOO_FEW_PGS health code.

To mute a health alert message, use the ceph health command.

[ceph: root@node /]# ceph health mute health-code [duration]

The health-code is the code provided by the ceph health detail command. The optional parameter duration is the time that the health message is muted, specified in seconds, minutes, or hours. You can unmute a health message with the ceph health unmute health-code command.

When you mute a health message, Ceph automatically unmutes the alert if the health status further degrades. For example, if your cluster reports one OSD down and you mute that alert, Ceph automatically removes the mute if another OSD goes down. Any health alerts that can be measured unmute.

Configure Ceph logging by using the ceph command at runtime. If you encounter errors when starting up the cluster, then you can update the Ceph configuration database so that it logs during startup.

You can set different logging levels for each subsystem in your cluster. Debug levels are on a scale of 1 to 20, where 1 is terse and 20 is verbose.

Ceph does not send memory-based logs to the output logs except in the following circumstances:

To use different debug levels for the output log level and the memory level, use a slash (/) character. For example, debug_mon = 1/5 sets the output log level of the ceph-mon daemon to 1 and its memory log level to 5.

To activate debugging output at runtime, use the ceph tell command.

[ceph: root@node /]# ceph tell type.id config set debug_subsystem debug-level

The type and id arguments are the type of the Ceph daemon and its ID. The subsystem is the specific subsystem whose debug level you want to modify.

This example modifies the OSD 0 debug level for the messaging system between Ceph components:

[ceph: root@node /]# ceph tell osd.0 config set debug_ms 5

View the configuration settings at runtime as follows:

[ceph: root@node /]# ceph tell osd.0 config show

Configure the subsystem debug levels so that they log to the default log file at boot time. Add the debugging settings to the Ceph configuration database by using the ceph config set command.

For example, add debug levels for specific Ceph daemons by setting these parameters in your Ceph configuration database:

[ceph: root@node /]# ceph config set global debug_ms 1/5
[ceph: root@node /]# ceph config set osd debug_ms 1
[ceph: root@node /]# ceph config set osd debug_osd 1/5
[ceph: root@node /]# ceph config set mon debug_mon 20

Debug logging for Ceph components is resource-intensive and can generate a huge amount of data. If you have almost full disks, then accelerate log rotation by modifying the log rotation configuration at /etc/logrotate.d/ceph. The Cron job scheduler uses this file to schedule log rotation.

You can add a size setting after the rotation frequency, so that the log file is rotated when it reaches the specified size:

rotate 7
weekly
size size
compress
sharedscripts

Use the crontab command to add an entry to inspect the /etc/logrotate.d/ceph file.

[ceph: root@node /]# crontab -e

For example, you can instruct Cron to check /etc/logrotate.d/ceph every 30 minutes.

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1

Increase the logging level when troubleshooting a client.

On the client system, you can add the debug_ms = 1 parameter to the configuration database by using the ceph config set client debug_ms 1 command. The Ceph client stores debug messages in the /var/log/ceph/ceph-client.id.log log file.

Most of the Ceph client commands, such as rados, ceph, or rbd, also accept the --debug-ms=1 option to execute only that command with an increased logging level.

By default, Ceph clients create a UNIX domain socket on start up. You can use this socket to communicate with the client to retrieve real-time performance data or to dynamically get or set a configuration parameter.

In the /var/run/ceph/fsid directory, there is a list of admin sockets for that host. Allow one admin socket per OSD, one for each MON, and one for each MGR. Administrators can use the ceph command with the --admin-daemon socket-patch option to query the client through the socket.

[ceph: root@node /]# sudo ls -al /var/run/ceph/fsid
total 0
drwxrwx---. 2  167  167 180 Oct 19 04:43 .
drwxr-xr-x. 3 root root  60 Oct 19 03:51 ..
srwxr-xr-x. 1  167  167   0 Oct 19 03:52 ceph-client.rgw.realm.zone.serverc.agsgpq.6.93951066330432.asok
srwxr-xr-x. 1  167  167   0 Oct 19 03:51 ceph-mgr.serverc.lab.example.com.aiqepd.asok
srwxr-xr-x. 1  167  167   0 Oct 19 03:51 ceph-mon.serverc.lab.example.com.asok
srwxr-xr-x. 1  167  167   0 Oct 19 03:51 ceph-osd.0.asok
srwxr-xr-x. 1  167  167   0 Oct 19 03:51 ceph-osd.1.asok
srwxr-xr-x. 1  167  167   0 Oct 19 03:51 ceph-osd.2.asok

The following example mounts a CephFS file system with the FUSE client, gets the performance counters, and sets the debug_ms configuration parameter to 1:

[root@host ~]# ceph-fuse -n client.admin /mnt/mountpoint
2021-10-19T09:23:57.914-0400 7f1e7b914200 -1 init, newargv = 0x55d703b17a00 newargc=15
ceph-fuse[54240]: starting ceph client
ceph-fuse[54240]: starting fuse
[root@host ~]# ls /var/run/ceph/
2ae6d05a-229a-11ec-925e-52540000fa0c  ceph-client.admin.54240.94381967377904.asok
[root@host ~]# ceph --admin-daemon \
/var/run/ceph/ceph-client.admin.54240.94381967377904.asok perf dump
{
    "AsyncMessenger::Worker-0": {
        "msgr_recv_messages": 4,
        "msgr_send_messages": 3,
        "msgr_recv_bytes": 10112,
        "msgr_send_bytes": 480,
        "msgr_created_connections": 2,
        "msgr_active_connections": 1,
        "msgr_running_total_time": 0.002775454,
        "msgr_running_send_time": 0.001042138,
        "msgr_running_recv_time": 0.000868150,
...output omitted...
[root@host ~]# ceph --admin-daemon \
/var/run/ceph/ceph-client.admin.54240.94381967377904.asok config show
...output omitted...
    "debug_ms": "0/0",
...output omitted...
[root@host ~]# ceph --admin-daemon \
/var/run/ceph/ceph-client.admin.54240.94381967377904.asok config set debug_ms 5
{
    "success": ""
}
[root@host ~]# ceph --admin-daemon \
/var/run/ceph/ceph-client.admin.54240.94381967377904.asok config show
...output omitted...
    "debug_ms": "5/5",
...output omitted...

Earlier versions of Ceph clients might not benefit from features provided by the installed version of the Ceph cluster. For example, an earlier client might fail to retrieve data from an erasure-coded pool. Therefore, when upgrading a Ceph cluster, you should also update the clients. The RADOS Gateway, the FUSE client for CephFS, the librbd, or the command-line tools, such as RADOS or RBD, are examples of Ceph clients.

From a client, you can find the version of the running Ceph cluster with the ceph versions command:

[ceph: root@node /]# ceph versions
{
    "mon": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 4
    },
    "mgr": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 4
    },
    "osd": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 9
    },
    "mds": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 1
    },
    "rgw": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 2
    },
    "overall": {
        "ceph version 16.2.0-117.el8cp (0e34bb74700060ebfaa22d99b7d2cdc037b28a57) pacific (stable)": 20
    }
}

You can also list the supported level of features with the ceph features command.

If you cannot upgrade the clients, the ceph osd set-require-min-compat-client version-name command specifies the minimum client version that the Ceph cluster must support.

Using this minimum client setting, Ceph denies the use of features that are not compatible with the current client version. Historically, the main exception has been changes to CRUSH. For example, if you run the ceph osd set-require-min-compat-client jewel command, then you cannot use the ceph osd pg-upmap command. This fails because "Jewel" version clients do not support the PG upmap feature. Verify the minimum version required by your cluster with the ceph osd command:

[ceph: root@node /]# ceph osd get-require-min-compat-client
luminous

Red Hat Ceph Storage provides the Cephx protocol for cryptographic authentication. If Cephx is enabled, then Ceph looks for the key ring in the default /etc/ceph/ path.

Either enable Cephx for all components, or disable it completely. Ceph does not support a mixed setting, such as enabling Cephx for clients but disabling it for communication between the Ceph services. By default, Cephx is enabled and a client trying to access the Ceph cluster without Cephx receives an error message.

All Ceph commands authenticate as the client.admin user by default, although you can specify the user name or the user ID by using the --name and --id options.

Problems with Cephx are usually related to:

The radosgw daemon is a Ceph client that sits between the Ceph cluster and HTTP clients. It includes its own web server, Beast, which supports HTTP and HTTPS.

In case of errors, you should consult the log file in the /var/log/ceph/ folder.

To log to a file, set the log_to_file parameter to true. You can update the location of the log file and the log level by using the log_file and debug parameters, respectively. You can also enable the rgw_enable_ops_log and rgw_enable_usage_log parameters in the Ceph configuration database to log each successful RADOS Gateway operation and the usage, respectively.

[ceph: root@node /]# ceph config set client.rgw \
log_file /var/log/ceph/ceph-rgw-node.log
[ceph: root@node /]# ceph config set client.rgw log_to_file true
[ceph: root@node /]# ceph config set client.rgw debug_rgw 20
[ceph: root@node /]# ceph config set client.rgw rgw_enable_ops_log true
[ceph: root@node /]# ceph config set global rgw_enable_usage_log true

Verify the debugging logs using the radosgw-admin log list command. This command provides a list of the log objects that are available. View log file information using the radosgw-admin log show command. To retrieve the information directly from the log object, add the --object parameter with the object ID. To retrieve the information on the bucket at the timestamp, add the --bucket, --date, and --bucket-id parameters, which refer to the bucket name, the timestamp, and the bucket ID.

The most common error in RADOS Gateway is time skew between the client and the RADOS Gateway because the S3 protocol uses date and time for signing each request. To avoid this problem, use NTP on both Ceph and client nodes.

You can verify issues on RADOS Gateway request completion by looking for HTTP status lines in the RADOS Gateway log file.

The RADOS Gateway is a Ceph client that stores all of its configuration in RADOS objects. The RADOS PGs holding this configuration data must be in the active+clean state. If the state is not active+clean, then Ceph I/O requests will hang if the primary OSD becomes unable to serve data, and HTTP clients will eventually time out. Identify the inactive PGs with the ceph health detail command.