The CRUSH algorithm enables Ceph clients to directly communicate with OSDs; this avoids a centralized service bottleneck. Ceph clients and OSDs use the CRUSH algorithm to efficiently compute information about object locations, instead of having to depend on a central lookup table. Ceph clients retrieve the cluster maps and use the CRUSH map to algorithmically determine how to store and retrieve data. This enables massive scalability for the Ceph cluster by avoiding a single point of failure and a performance bottleneck.

The CRUSH algorithm works to uniformly distribute the data in the object store, manage replication, and respond to system growth and hardware failures. When new OSDs are added or an existing OSD or OSD host fails, Ceph uses CRUSH to rebalance the objects in the cluster among the active OSDs.

Conceptually, a CRUSH map contains two major components:

The CRUSH hierarchy organizes OSDs into a tree of different containers, called buckets. For a large installation, you can create a specific hierarchy to describe your storage infrastructure: data centers with rows of racks, racks, hosts, and OSD devices on those hosts. By creating a CRUSH map rule, you can cause Ceph to place an object's replicas on OSDs on separate servers, on servers in different racks, or even on servers in different data centers.

To summarize, buckets are the containers or branches in the CRUSH hierarchy. Devices are OSDs, and are leaves in the CRUSH hierarchy.

Some of the most important bucket attributes are:

Figure 5.1: CRUSH map default hierarchy example

The CRUSH map contains a list of all the storage devices in the cluster. For each storage device the following information is available:

The ceph osd crush tree command shows the CRUSH map's current CRUSH hierarchy:

[ceph: root@node /]# ceph osd crush tree
  ID CLASS WEIGHT  TYPE NAME
  -1       1.52031 root default
  -3       0.48828     host serverg
   0   hdd 0.48828         osd.0 
  -5       0.48828     host serverh
   1   hdd 0.48828         osd.1 
  -7       0.48828     host serveri
   2   hdd 0.48828         osd.2 
  -9       0.01849     host serverj
   3   ssd 0.01849         osd.3 
  -11      0.01849     host serverk
   4   ssd 0.01849         osd.4 
  -13      0.01849     host serverl
   5   ssd 0.01849         osd.5 

Device classes are implemented by creating a “shadow” CRUSH hierarchy for each device class in use that contains only devices of that class. CRUSH rules can then distribute data over the shadow hierarchy. You can view the CRUSH hierarchy with shadow items with the ceph osd crush tree --show-shadow command.

Create a new device class by using the ceph osd crush class create command. Remove a device class using the ceph osd crush class rm command.

List configured device classes with the ceph osd crush class ls command.

The CRUSH map also contains the data placement rules that determine how PGs are mapped to OSDs to store object replicas or erasure coded chunks.

The ceph osd crush rule ls command lists the existing rules and the ceph osd crush rule dump rule_name command prints the details of a rule.

The decompiled CRUSH map also contains the rules and might be easier to read:

[ceph: root@node /]# ceph osd getcrushmap -o ./map.bin
[ceph: root@node /]# crushtool -d ./map.bin -o ./map.txt
[ceph: root@node /]# cat ./map.txt
...output omitted...
rule replicated_rule {  
      id 0  
      type replicated
      min_size 1   
      max_size 10  
      step take default  
      step chooseleaf firstn 0 type host  
      step emit 
}
...output omitted...

For example, you could create the following rule to select as many OSDs as needed on separate racks, but only from the DC1 data center:

rule myrackruleinDC1 {
      id 2
      type replicated
      min_size 1
      max_size 10
      step take DC1
      step chooseleaf firstn 0 type rack
      step emit
}

You can also modify the CRUSH algorithm's behavior using tunables. Tunables adjust and disable or enable features of the CRUSH algorithm. Ceph defines the tunables at the beginning of the decompiled CRUSH map, and you can get their current values by using the following command:

[ceph: root@node /]# ceph osd crush show-tunables
{
  "choose_local_tries": 0,
  "choose_local_fallback_tries": 0,
  "choose_total_tries": 50,
  "chooseleaf_descend_once": 1,
  "chooseleaf_vary_r": 1,
  "chooseleaf_stable": 1,
  "straw_calc_version": 1,
  "allowed_bucket_algs": 54,
  "profile": "jewel",
  "optimal_tunables": 1,
  "legacy_tunables": 0,
  "minimum_required_version": "jewel",
  "require_feature_tunables": 1,
  "require_feature_tunables2": 1,
  "has_v2_rules": 1,
  "require_feature_tunables3": 1,
  "has_v3_rules": 0,
  "has_v4_buckets": 1,
  "require_feature_tunables5": 1,
  "has_v5_rules": 0
}

Rather than modifying individual tunables, you can select a predefined profile with the ceph osd crush tunables profile command. Set the value of profile to optimal to enable the best (optimal) values for the current version of Red Hat Ceph Storage.

This example creates a new bucket:

[ceph: root@node /]# ceph osd crush add-bucket name type

For example, these commands create three new buckets, one of the datacenter type and two of the rack type:

[ceph: root@node /]# ceph osd crush add-bucket DC1 datacenter
added bucket DC1 type datacenter to crush map
[ceph: root@node /]# ceph osd crush add-bucket rackA1 rack
added bucket rackA1 type rack to crush map
[ceph: root@node /]# ceph osd crush add-bucket rackB1 rack
added bucket rackB1 type rack to crush map

You can then organize the new buckets in a hierarchy with the following command:

[ceph: root@node /]# ceph osd crush move name type=parent

You also use this command to reorganize the tree. For example, the following commands attach the two rack buckets from the previous example to the data center bucket, and attaches the data center bucket to the default root bucket:

[ceph: root@node /]# ceph osd crush move rackA1 datacenter=DC1
moved item id -10 name 'rackA1' to location {datacenter=DC1} in crush map
[ceph: root@node /]# ceph osd crush move rackB1 datacenter=DC1
moved item id -11 name 'rackB1' to location {datacenter=DC1} in crush map
[ceph: root@node /]# ceph osd crush move DC1 root=default
moved item id -9 name 'DC1' to location {root=default} in crush map

After you have created your custom bucket hierarchy, place the OSDs as leaves on this tree. Each OSD has a location, which is a string defining the full path to the OSD from the root of the tree. For example, the location of an OSD attached to the rackA1 bucket is:

root=default datacenter=DC1 rack=rackA1

When Ceph starts, it uses the ceph-crush-location utility to automatically verify that each OSD is in the correct CRUSH location. If the OSD is not in the expected location in the CRUSH map, it is automatically moved. By default, this is root=default host=hostname.

You can replace the ceph-crush-location utility with your own script to change where OSDs are placed in the CRUSH map. To do this, specify the crush_location_hook parameter in the /etc/ceph/ceph.conf configuration file.

...output omitted...
[osd]
crush_location_hook = /path/to/your/script
...output omitted...

Ceph executes the script with these arguments: --cluster cluster-name --id osd-id --type osd. The script must print the location as a single line on its standard output. The upstream Ceph documentation has an example of a custom script that assumes each system has an /etc/rack file containing the name of its rack:

#!/bin/sh
echo "root=default rack=$(cat /etc/rack) host=$(hostname -s)"

You can set the crush_location parameter in the /etc/ceph/ceph.conf configuration file to redefine the location for particular OSDs. For example, to set the location for osd.0 and osd.1, add the crush_location parameter inside their respective sections in that file:

[osd.0]
crush_location = root=default datacenter=DC1 rack=rackA1

[osd.1]
crush_location = root=default datacenter=DC1 rack=rackB1

This example creates a rule that Ceph can use for replicated pools:

[ceph: root@node /]# ceph osd crush rule create-replicated name root \
failure-domain-type [class]

The following example creates the new inDC2 rule to store replicas in the DC2 data center, and distributes the replicas across racks:

[ceph: root@node /]# ceph osd crush rule create-replicated inDC2 DC2 rack
[ceph: root@node /]# ceph osd crush rule ls
replicated_rule
erasure-code
inDC2

After you have defined the rule, use it when creating a replicated pool:

[ceph: root@node /]# ceph osd pool create myfirstpool 50 50 inDC2
pool 'myfirstpool' created

For erasure coding, Ceph automatically creates rules for each erasure coded pool you create. The name of the rule is the name of the new pool. Ceph uses the rule parameters you define in the erasure code profile that you specify when you create the pool.

The following example first creates the new myprofile erasure code profile, then creates the myecpool pool based on this profile:

[ceph: root@node /]# ceph osd erasure-code-profile set myprofile k=2 m=1 \
crush-root=DC2 crush-failure-domain=rack crush-device-class=ssd
[ceph: root@node /]# ceph osd pool create myecpool 50 50 erasure myprofile
pool 'myecpool' created
[ceph: root@node /]# ceph osd crush rule ls
replicated_rule
erasure-code
myecpool

You can decompile and manually edit the CRUSH map with the following commands:

For a cluster with a single pool, you can use the following formula, with 100 placement groups per OSD:

Total PGs = (OSDs * 100)/Number of replicas

Red Hat recommends the use of the Ceph Placement Groups per Pool Calculator, https://access.redhat.com/labs/cephpgc/, from the Red Hat Customer Portal Labs.

Use the ceph osd pg-upmap-items command to manually map PGs to specific OSDs. Because older Ceph clients do not support it, you must configure the ceph osd set-require-min-compat-client setting to enable the pg-upmap command.

[ceph: root@node /]# ceph osd set-require-min-compat-client luminous
set require_min_compat_client to luminous

The following example remaps the PG 3.25 from ODs 2 and 0 to 1 and 0:

[ceph: root@node /]# ceph pg map 3.25
osdmap e384 pg 3.25 (3.25) -> up [2,0] acting [2,0]
[ceph: root@node /]# ceph osd pg-upmap-items 3.25 2 1
set 3.25 pg_upmap_items mapping to [2->1]
[ceph: root@node /]# ceph pg map 3.25
osdmap e387 pg 3.25 (3.25) -> up [1,0] acting [1,0]

Remapping hundreds of PGs this way is not practical. The osdmaptool command is useful here. It takes the actual map for a pool, analyses it, and generates the ceph osd pg-upmap-items commands to run for an optimal distribution: