It is a common misconception that disk latency and response time are the same thing. Disk latency is a function of the device, but response time is measured as a function of the entire server.

For hard drives using spinning platters, disk latency has two components:

After the drive has positioned the heads, it can start transferring data from the platter. At that point, the sequential data transfer rate is important.

For solid-state drives (SSDs), the equivalent metric is the random access latency of the storage device, which is typically less than a millisecond. For non-volatile memory express drives (NVMes), the random access latency of the storage drive is typically in microseconds.

The number of read and write requests that the system can process per second depends on the storage device capabilities and the application. When an application issues an I/O request, the operating system transfers the request to the device and waits until the request completes. As a reference, hard drives using spinning platters achieve between 50 and 200 IOPS, SSDs are on the order of thousands to hundreds of thousands IOPS, and NVMes achieve some hundreds of thousands IOPS.

Throughput refers to the actual number of bytes per second the system can read or write. The size of the block and the data transfer rate affect the throughput. The higher the disk block size, the more you attenuate the latency factor. The higher the data transfer rate, the faster a disk can transfer data from its surface to a buffer.

As a reference value, hard drives using spinning platters have a throughput around 150 Mb/s, SSDs are around 500 Mbps, and NVMes are in the order of 2,000 Mb/s.

You can measure throughput for networks and the whole system, from a remote client to a server.

Ceph supports increasing or decreasing the number of PGs in a pool. If you do not specify a value when creating a pool, it is created with a default value of 8 PGs, which is very low.

The pg_autoscale_mode property allows Ceph to make recommendations and automatically adjust the pg_num and pgp_num parameters. This option is enabled by default when creating a new pool. The pg_num parameter defines the number of PGs for a specific pool. The pgp_num parameter defines the number of PGs that the CRUSH algorithm considers for placement.

Red Hat recommends that you make incremental increases in the number of placement groups until you reach the desired number of PGs. Increasing the number of PGs by a significant amount can cause cluster performance degradation, because the expected data relocation and rebalancing is intensive.

Use the ceph osd pool set command to manually increase or decrease the number of PGs by setting the pg_num parameter. You should only increase the number of PGs in a pool by small increments when doing it manually with the pg_autoscale_mode option disabled. Setting the total number of placement groups to a number that is a power of 2 provides better distribution of the PGs across the OSDs. Increasing the pg_num parameter automatically increases the pgp_num parameter, but at a gradual rate to minimize the impact on cluster performance.

Red Hat Ceph Storage can merge two PGs into a larger PG, reducing the total number of PGs. Merging can be useful when the number of PGs in a pool is too large and performance is degraded. Because merging is a complex process, merge only one PG at a time to minimize the impact on cluster performance.

As discussed, the PG autoscale feature allows Ceph to make recommendations and automatically adjust the number of PGs. This feature is enabled by default when creating a pool. For existing pools, configure autoscaling with this command:

[admin@node ~]$ ceph osd pool set pool-name pg_autoscale_mode mode

Set the mode parameter to off to disable it, on to enable it and allow Ceph to automatically make adjustments in the number of PGs, or warn to raise health alerts when the number of PGs must be adjusted.

View the information provided by the autoscale module:

[admin@node ~]$ ceph osd pool autoscale-status
POOL        SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO
pool1         0                 3.0        92124M  0.0000
pool2      1323                 3.0        92124M  0.0000
pool3      3702                 3.0        92124M  0.0000

TARGET RATIO  EFFECTIVE RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE
                                1.0       1              on
                                1.0      32              on
                                1.0      32          64  warn

The previous table is split in two parts to make it easier to read on this page. The first two pools have the AUTOSCALE feature set to on, with Ceph automatically adjusting the number of PGs. The third pool is configured to provide a health alert if the number of PGs needs adjusting. The PG_NUM parameter is the current number of PGs in each pool or the number of PGs that the pool is working towards. The NEW PG_NUM parameter is the number of PGs that Ceph recommends to set in the pool.

RADOS bench is a simple tool for testing the RADOS Object Store. It executes write and read tests on your cluster and provides statistics. The general syntax of the command is:

[admin@node ~]$ rados -p pool-name bench seconds write|seq|rand \
-b objsize -t concurrency

These are the common parameters for the tool:

The RBD bench measures I/O throughput and latency on an existing image, which you created for the test.

These are the default values: