Cloud Storage with Red Hat Ceph Storage

After completing this section, you should be able to describe and compare replicated and erasure coded pools, and create and configure each pool type.

Pools are logical partitions for storing objects. Ceph clients write objects to pools.

Ceph clients require the cluster name (ceph by default) and a monitor address to connect to the cluster. Ceph clients usually obtain this information from a Ceph configuration file, or by being specified as command-line parameters.

A Ceph client uses the list of pools retrieved with the cluster map, to determine where to store new objects.

The Ceph client creates an input/output context to a specific pool and the Ceph cluster uses the CRUSH algorithm to map these pools to placement groups, which are then mapped to specific OSDs.

Pools provide a layer of resilience for the cluster because pools define the number of OSDs that can fail without losing data.

Ceph protects data within replicated pools by creating multiple copies of each object, called replicas. Ceph uses the CRUSH failure domain to determine the primary OSDs of the acting set to store the data. Then, the primary OSD finds the current replica size for the pool and calculates the secondary OSDs to write the objects to. After the primary OSD receives the acknowledgment of the writes and finishes writing the data, the primary OSD acknowledges a successful write operation to the Ceph client. This protects the data in the object if one or more of the OSDs fail.

Use the following command to create a replicated pool.

[ceph: root@node /]# ceph osd pool create pool-name  pg-num  pgp-num  replicated crush-rule-name

Where:

The number of placement groups in a pool can be adjusted after it is initially configured. If pg_num and pgp_num are set to the same number, then any future adjustments to pg_num automatically adjusts the value of pgp_num. The adjustment to pgp_num triggers the movement of PGs across OSDs, if needed, to implement the change. Define a new number of PGs in a pool by using the following command.

[ceph: root@node /]# ceph osd pool set my_pool pg_num 32
set pool 6 pg_num to 32

When you create a pool with the ceph osd pool create command, you do not specify the number of replicas (size). The osd_pool_default_size configuration parameter defines the number of replicas, and defaults to a value of 3.

[ceph: root@node /]# ceph config get mon osd_pool_default_size
3

Change the size of a pool with the ceph osd pool set pool-name size number-of-replicas command. Alternatively, update the default setting of the osd_pool_default_size configuration setting.

The osd_pool_default_min_size parameter sets the number of copies of an object that must be available to accept I/O requests. The default value is 2.

An erasure coded pool uses erasure coding instead of replication to protect object data.

Objects stored in an erasure coded pool are divided into a number of data chunks which are stored in separate OSDs. The number of coding chunks are calculated based on the data chunks and are stored in different OSDs. The coding chunks are used to reconstruct the object's data if an OSD fails. The primary OSD receives the write operation, then encodes the payload into K+M chunks and sends them to the secondary OSDs in erasure coded pools.

Erasure coded pools use this method to protect their objects and, unlike replicated pools, do not rely on storing multiple copies of each object.

To summarize how erasure coded pools work:

Figure 4.4: Erasure coded pools

Erasure coding uses storage capacity more efficiently than replication. Replicated pools maintain n copies of an object, whereas erasure coding maintains only k + m chunks. For example, replicated pools with 3 copies use 3 times the storage space. Erasure coded pools with k=4 and m=2 use only 1.5 times the storage space.

Erasure coded pools require less storage than replicated pools to obtain a similar level of data protection, which can reduce the cost and size of the storage cluster. However, calculating coding chunks adds CPU processing and memory overhead for erasure coded pools, reducing overall performance.

Use the following command to create an erasure coded pool.

[ceph: root@node /]# ceph osd pool create pool-name pg-num pgp-num \
erasure erasure-code-profile crush-rule-name

Where:

You can configure placement group autoscaling on a pool. Autoscaling allows the cluster to calculate the number of placement groups and to choose appropriate pg_num values automatically. Autoscaling is enabled by default in Red Hat Ceph Storage 5.

Every pool in the cluster has a pg_autoscale_mode option with a value of on, off, or warn.

This example enables the pg_autoscaler module on the Ceph MGR nodes and sets the autoscaling mode to on for a pool:

[ceph: root@node /]# ceph mgr module enable pg_autoscaler
module 'pg_autoscaler' is already enabled (always-on)
[ceph: root@node /]# ceph osd pool set pool-name pg_autoscale_mode on
set pool 7 pg_autoscale_mode to on

Erasure coded pools cannot use the Object Map feature. An object map is an index of objects that tracks where the blocks of an rbd object are allocated. Having an object map for a pool improves the performance of resize, export, flatten, and other operations.

[ceph: root@node /]# ceph osd erasure-code-profile set profile-name arguments

You can view and modify existing pools and change pool configuration settings.

Configure these example setting values to enable protection against pool reconfiguration:

A namespace is a logical group of objects in a pool. Access to a pool can be limited so that a user can only store or retrieve objects in a particular namespace. One advantage of namespaces is to restrict user access to part of a pool.

Namespaces are useful for restricting storage access by an application. They allow you to logically partition a pool and restrict applications to specific namespaces inside the pool.

You could dedicate an entire pool to each application, but having more pools means more PGs per OSD, and PGs are computationally expensive. This might degrade OSD performance as load increases. With namespaces, you can keep the number of pools the same and not dedicate an entire pool to each application.

To store an object inside a namespace, the client application must provide the pool and the namespace names. By default, each pool contains a namespace with an empty name, known as the default namespace.

Use the rados command to store and retrieve objects from a pool. Use the -N name and --namespace= name options to specify the pool and namespace to use.

The following example stores the /etc/services file as the srv object in the mytestpool pool, under the system namespace.

[ceph: root@node /]# rados -p mytestpool -N system put srv /etc/services
[ceph: root@node /]# rados -p mytestpool -N system ls
srv

List all the objects in all namespaces in a pool by using the --all option. To obtain JSON formatted output, add the --format=json-pretty option.

The following example lists the objects in the mytestpool pool. The mytest object has an empty namespace. The other objects belong to the system or the flowers namespaces.

[ceph: root@node /]# rados -p mytestpool --all ls
system  srv
flowers anemone
flowers iris
system  magic
flowers rose
        mytest
system networks
[ceph: root@node /]# rados -p mytestpool --all ls --format=json-pretty
[
    {
        "name": "srv",
        "namespace": "system"
    },
    {
        "name": "anemone",
        "namespace": "flowers"
    },
    {
        "name": "iris",
        "namespace": "flowers"
    },
    {
        "name": "magic",
        "namespace": "system"
    },
    {
        "name": "rose",
        "namespace": "flowers"
    },
    {
        "name": "mytest",
        "namespace": ""
    },
    {
        "name": "networks",
        "namespace": "system"
    }
]