Cloud Storage with Red Hat Ceph Storage

Ceph Monitors (MONs) are the daemons that maintain the cluster map. The cluster map is a collection of five maps that contain information about the state of the cluster and its configuration. Ceph must handle each cluster event, update the appropriate map, and replicate the updated map to each MON daemon.

To apply updates, the MONs must establish a consensus on the state of the cluster. A majority of the configured monitors must be available and agree on the map update. Configure your Ceph clusters with an odd number of monitors to ensure that the monitors can establish a quorum when they vote on the state of the cluster. More than half of the configured monitors must be functional for the Ceph storage cluster to be operational and accessible.

Ceph Object Storage Devices (OSDs) are the building blocks of a Ceph storage cluster. OSDs connect a storage device (such as a hard disk or other block device) to the Ceph storage cluster. An individual storage server can run multiple OSD daemons and provide multiple OSDs to the cluster. Red Hat Ceph Storage 5 supports a feature called BlueStore to store data within RADOS. BlueStore uses the local storage devices in raw mode and is designed for high performance.

One design goal for OSD operation is to bring computing power as close as possible to the physical data so that the cluster can perform at peak efficiency. Both Ceph clients and OSD daemons use the Controlled Replication Under Scalable Hashing (CRUSH) algorithm to efficiently compute information about object location, instead of depending on a central lookup table.

Ceph Managers (MGRs) provide for the collection of cluster statistics.

If no MGRs are available in a cluster, client I/O operations are not negatively affected, but attempts to query cluster statistics fail. To avoid this scenario, Red Hat recommends that you deploy at least two Ceph MGRs for each cluster, each running in a separate failure domain.

The MGR daemon centralizes access to all data that is collected from the cluster and provides a simple web dashboard to storage administrators. The MGR daemon can also export status information to an external monitoring server, such as Zabbix.

The Ceph Metadata Server (MDS) manages Ceph File System (CephFS) metadata. It provides POSIX-compliant, shared file-system metadata management, including ownership, time stamps, and mode. The MDS uses RADOS instead of local storage to store its metadata. It has no access to file contents.

The MDS enables CephFS to interact with the Ceph Object Store, mapping an inode to an object and the location where Ceph stored the data within a tree. Clients who access a CephFS file system first send a request to an MDS, which provides the needed information to get file content from the correct OSDs.

Ceph clients and OSDs require knowledge of the cluster topology. Five maps represent the cluster topology, which is collectively referred to as the cluster map. The Ceph Monitor daemon maintains the cluster map. A cluster of Ceph MONs ensures high availability if a monitor daemon fails.

The foundational library that implements the other Ceph interfaces, such as Ceph Block Device and Ceph Object Gateway, is librados. The librados library is a native C library that allows applications to work directly with RADOS to access objects stored by the Ceph cluster. Similar libraries are available for C++, Java, Python, Ruby, Erlang, and PHP.

To maximize performance, write your applications to work directly with librados. This method gives the best results to improve storage performance in a Ceph environment. For easier Ceph storage access, instead use the higher-level access methods that are provided, such as the RADOS Block Devices, Ceph Object Gateway (RADOSGW), and CephFS.

A Ceph Block Device (RADOS Block Device or RBD) provides block storage within a Ceph cluster through RBD images. Ceph scatters the individual objects that compose RBD images across different OSDs in the cluster. Because the objects that make up the RBD are on different OSDs, access to the block device is automatically parallelized.

RBD provides the following features:

Ceph Object Gateway (RADOS Gateway, RADOSGW, or RGW) is an object storage interface that is built with librados. It uses this library to communicate with the Ceph cluster and writes to OSD processes directly. It provides applications with a gateway with a RESTful API, and supports two interfaces: Amazon S3 and OpenStack Swift.

The Ceph Object Gateway offers scalability support by not limiting the number of deployed gateways and by providing support for standard HTTP load balancers. The Ceph Object Gateway includes the following use cases:

Ceph File System (CephFS) is a parallel file system that provides a scalable, single-hierarchy shared disk. Red Hat Ceph Storage  provides production environment support for CephFS, including support for snapshots.

The Ceph Metadata Server (MDS) manages the metadata that is associated with files stored in CephFS, including file access, change, and modification time stamps.

Ceph OSDs protect and constantly check the integrity of the data that is stored in the cluster. Pools are logical partitions of the Ceph storage cluster that are used to store objects under a common name tag. Ceph assigns each pool a specific number of hash buckets, called Placement Groups (PGs), to group objects for storage.

Each pool has the following adjustable properties:

Configure the number of placement groups that are assigned to each pool independently to fit the type of data and the required access for the pool.

The CRUSH algorithm determines the OSDs that host the data for a pool. Each pool has a single CRUSH rule that is assigned as its placement strategy. The CRUSH rule determines which OSDs store the data for all the pools that are assigned that rule.

A Placement Group (PG) aggregates a series of objects into a hash bucket, or group. Ceph maps each PG to a set of OSDs. An object belongs to a single PG, and all objects that belong to the same PG return the same hash result.

The CRUSH algorithm maps an object to its PG based on the hashing of the object's name. The placement strategy is also called the CRUSH placement rule. The placement rule identifies the failure domain to choose within the CRUSH topology to receive each replica or erasure code chunk.

When a client writes an object to a pool, it uses the pool's CRUSH placement rule to determine the object's placement group. The client then uses its copy of the cluster map, the placement group, and the CRUSH placement rule to calculate which OSDs to write a copy of the object to (or its erasure-coded chunks).

The layer of indirection that the placement group provides is important when new OSDs become available to the Ceph cluster. When OSDs are added to or removed from a cluster, placement groups are automatically rebalanced between operational OSDs.

A Ceph client gets the latest copy of the cluster map from a MON. The cluster map provides information to the client about all the MONs, OSDs, and MDSs in the cluster. It does not provide the client with the location of the objects; the client must use CRUSH to compute the locations of objects that it needs to access.

To calculate the Placement Group ID (PG ID) for an object, the Ceph client needs the object ID and the name of the object's storage pool. The client calculates the PG ID, which is the hash of the object ID modulo the number of PGs. It then looks up the numeric ID for the pool, based on the pool's name, and prepends the pool ID to the PG ID.

The CRUSH algorithm is then used to determine which OSDs are responsible for a placement group (the Acting Set). The OSDs in the Acting Set that are up are in the Up Set. The first OSD in the Up Set is the current primary OSD for the object's placement group, and all other OSDs in the Up Set are secondary OSDs.

The Ceph client can then directly work with the primary OSD to access the object.

Like Ceph clients, OSD daemons use the CRUSH algorithm, but the OSD daemon uses it to compute where to store the object replicas and for rebalancing storage. In a typical write scenario, a Ceph client uses the CRUSH algorithm to compute where to store the original object. The client maps the object to a pool and placement group and then uses the CRUSH map to identify the primary OSD for the mapped placement group. When creating pools, set them as either replicated or erasure coded pools. Red Hat Ceph Storage 5 supports erasure coded pools for Ceph RBD and CephFS.

Figure 1.2: Ceph pool data protection methods

For resilience, configure pools with the number of OSDs that can fail without losing data. For a replicated pool, which is the default pool type, the number determines the number of copies of an object to create and distribute across different devices. Replicated pools provide better performance than erasure coded pools in almost all cases at the cost of a lower usable-to-raw storage ratio.

Erasure coding provides a more cost-efficient way to store data but with lower performance. For an erasure coded pool, the configuration values determine the number of coding chunks and parity blocks to create.

A primary advantage of erasure coding is its ability to offer extreme resilience and durability. You can configure the number of coding chunks (parities) to use.

The following figure illustrates how data objects are stored in a Ceph cluster. Ceph maps one or more objects in a pool to a single PG, represented by the colored boxes. Each of the PGs in this figure is replicated and stored on separate OSDs within the Ceph cluster.

Figure 1.3: Objects in a Ceph pool stored in placement groups