Cloud Storage with Red Hat Ceph Storage

Operators are applications that invoke the OpenShift controller API to manage resources. Operators provide a repeatable way to package, deploy, and manage a containerized application.

The operator container image defines the requirements for deployment, such as dependent services and hardware resources. Because operators require resource access, they typically use custom security settings. Operators provide an API for resource management and service configuration, and deliver levels of automated management and upgrade strategies.

OpenShift Container Platform uses the Operator Lifecycle Manager (OLM) to manage operators. OLM orchestrates the deployment, update, resource utilization, and deletion of other operators from the operator catalog. Every operator has a Cluster Service Version (CSV) that describes the required technical information to run the operator, such as the RBAC rules that it requires and the resources that it manages or depends on. OLM is itself an operator.

Custom Resource Definition (CRD) objects define unique object types in the cluster. Custom Resource (CR) objects are created from CRDs. Only cluster administrators can create CRDs. Developers with CRD read permission can add defined CR object types into their project.

Operators use CRDs by packaging them with any required RBAC policy and other software-specific logic. Cluster administrators can add CRDs manually to a cluster independently from an Operator lifecycle, to be available to all users.

The OpenShift Container Storage operator integrates three operators as an operator bundle to initialize and manage OpenShift Data Foundation services. The three operators are OpenShift Container Storage (ocs-operator), Rook-Ceph, and Multicloud Object Gateway (NooBaa). The operator bundle includes a converged CSV and all the needed CRDs to deploy ocs-operator, Rook-Ceph, and NooBaa operators.

The operator ocs-operator initializes tasks for the OpenShift Data Foundation service and acts as a configuration gateway for Rook-Ceph and NooBaa. The operator ocs-operator depends on the configuration that is defined in the OCSInitilization and StorageCluster CRDs in the CSV bundle.

When the operator bundle is installed, the ocs-operator starts and creates an OCSInitialization resource if it does not already exist. The OCSInitialization resource performs basic setup and initializes services. It creates the openshift-storage namespace in which other bundle operators will create resources. You can edit this resource to adjust the tools that are included in the OpenShift Data Foundation operator. If the OCSInitialization resource is in a failed state, further start requests are ignored until the resource is deleted.

The StorageCluster resource manages the creation and reconciliation of CRDs for the Rook-Ceph and NooBaa operators. These CRDs are defined by known best practices and policies that Red Hat supports. You can create a StorageCluster resource with an installation wizard in OpenShift Container Platform.

Unlike the OCSInitialization resource, the StorageCluster resource creation or deletion operates outside ocs-operator control. Only one StorageCluster resource per OpenShift Container Platform cluster is supported.

Rook is a cloud-native storage orchestrator that provides the platform to abstract the complexity of Ceph layout and configuration. Rook-Ceph is the primary component for the ocs-operator. It incorporates the Ceph cluster into the operator bundle.

Rook-Ceph is responsible for the initial storage cluster bootstrap, administrative tasks, and the creation of the pods and other dependent resources in the openshift-storage namespace. Many advanced Ceph features, such as Placement Groups and CRUSH maps, are reserved for Rook management. Rook-Ceph facilitates a seamless storage consumption experience and minimizes the required cluster administration.

Monitoring is an important Rook-Ceph duty. Rook-Ceph watches the storage cluster state to ensure that it is available and healthy. Rook-Ceph monitors Ceph Placement Groups and automatically adjusts their configuration based on pool sizing, and monitors Ceph daemons. Rook-Ceph communicates with OpenShift APIs to request the necessary resources when the cluster scales.

Rook-Ceph provides two Container Storage Interface (CSI) drivers to create volumes, the RBD driver and the CephFS driver. These drivers provide the channel for OpenShift Container Platform to consume storage.

SectionFigure 13.3: Rook Architecture visualizes the Rook-Ceph operator interaction with OpenShift Container Platform.

Figure 13.3: Rook Architecture

You can update the configuration of Rook-Ceph components via CRD updates. Rook-Ceph looks for configuration changes that the service API requested and applies them to the cluster. The cluster state reflects whether the cluster is in the desired state or is approaching it. Important CRDs for the cluster configuration are CephCluster, CephObjectStore, CephFilesystem, and CephBlockPool.

NooBaa is a multicloud object storage service that delivers an S3-compatible API in the OpenShift container storage operator bundle. NooBaa provides data placement policies that enable hybrid and multicloud interconnectivity, allowing active/active reads and writes across different clouds.

The NooBaa operator creates and reconciles changes for the NooBaa service and creates the following resources:

NooBaa requires a backing store resource to save objects. A default backing store is created in an OpenShift Data Foundation deployment, but depends on the platform that OpenShift Container Platform is running on. For example, when OpenShift Container Platform or OpenShift Data Foundation is deployed on Amazon Web Services (AWS), it creates the backing store as an AWS::S3 bucket. For Microsoft Azure, the default backing store is a blob container. NooBaa can define multiple, concurrent backing stores.