Cloud Storage with Red Hat Ceph Storage

Ceph is a scalable storage solution that replicates data across commodity storage nodes. Ceph uses an object storage architecture for data storage, and provides multiple storage interfaces for object storage, block storage, and file systems.

Ceph integration with OpenStack features:

In OpenStack, the default back end for the Image service is a file store that is located with the Glance API node on the controller node. The location is configurable, with a default of /var/lib/glance. To improve scalability, the Image service implemented an image cache at the default /var/lib/glance/image-cache/ location on the controller node. When the Compute service loads images that are stored in the default QCOW2 format to convert to RAW for use on the compute nodes, then the converted image is cached.

When Red Hat OpenStack Platform is installed with the Swift Object Store, TripleO places the image service back end on Swift by default. The Swift service creates a container called glance for storing Glance images.

When Ceph storage is integrated into RHOSP, TripleO places the image service back end on Ceph RADOS Block Devices (RBD) by default. Glance images are stored in a Ceph pool called images. RHOSP works with images as immutable blobs and handles them accordingly. The pool name is configurable with the glance_pool_name property. The images pool is configured as a replicated pool by default, which means that all images are replicated across storage devices for transparent resilience.

An image pool can be configured as erasure-coded to conserve disk space with a slight increase in CPU utilization.

When using Ceph as the storage back end, it is important to disable the image cache, as it is not needed because Ceph expects Glance images to be stored in the RAW format. When using RAW images, all image interactions occur within Ceph, including image clone and snapshot creation. Disabling the image cache eliminates significant CPU and network activity on controller nodes.

When using a distributed architecture with Distributed Compute Nodes (DCN), TripleO can configure the Image service with an image pool at each remote site. You can copy images between the central (hub) site and the remote sites. The DCN Ceph cluster uses RBD technologies, such as copy-on-write and snapshot layering, for fast instance launching. The Image, Block Storage, and Compute services must all be configured to use Ceph RBD as their back-end storage.

Object storage is implemented in OpenStack by the Object Store service (Swift). The Object Store service implements both the Swift API and the Amazon S3 API. The default storage back end is file-based, and uses an XFS-formatted partition mount in subdirectories of /srv/node on the designated storage node. You can also configure the Object Store service to use an existing, external Swift cluster as a back end.

When Ceph storage is integrated into RHOSP, TripleO configures the Object Store service to use the RADOS Gateway (RGW) as the back end. Similarly, the Image service is configured for RGW because Swift would not be available as a back end.

The Ceph Object Gateway can be integrated with the Keystone identity service. This integration configures RGW to use the Identity service as the user authority. If Keystone authorizes a user to access the gateway, then the user is also initially created on the Ceph Object Gateway. Identity tokens that Keystone validates are considered valid by the Ceph Object Gateway. The Ceph Object Gateway is also configured as an object-storage endpoint in Keystone.

Block storage is implemented in OpenStack by the Block Storage service (Cinder). The Block Storage service provides persistent volumes that remain in storage and are stable when not attached to any instance. It is common to configure the Block Storage service with multiple back ends. The default storage back end is the Logical Volume Manager (LVM), which is configured to use a volume group called cinder-volumes. TripleO can create the volume group during an installation, or use an existing cinder-volumes volume group.

When Ceph storage is integrated into RHOSP, TripleO configures the Block Storage service to use RADOS Block Devices (RBD) as the back end. Block Storage volumes are stored in a Ceph pool called volumes. Volume backups are stored in a Ceph pool called backups. Ceph block device images attach to an OpenStack instance by using libvirt, which configures the QEMU interface to the librbd Ceph module. Ceph stripes block volumes across multiple OSDs within the cluster, providing increased performance for large volumes when compared to local drives.

OpenStack volumes, snapshots, and clones are implemented as block devices. OpenStack uses volumes to boot VMs, or to attach to running VMs as further application storage.

File storage is implemented in OpenStack by the Shared File Systems service (Manila). The Shared File Systems service supports multiple back ends and can provision shares from one or more back ends. Share servers export file shares by using various file system protocols such as NFS, CIFS, GlusterFS, or HDFS.

The Shared File Systems service is persistent storage and can be mounted to any number of client machines. You can detach file shares from one instance and attach them to another instance without data loss. The Shared File Systems service manages share attributes, access rules, quotas, and rate limits. Because unprivileged users are not allowed to use the mount command, the Shared File Systems service acts as a broker to mount and unmount shares that the storage operator configured.

When Ceph storage is integrated into RHOSP, TripleO configures the Shared File Systems service to use CephFS as the back end. CephFS uses the NFS protocol with the Shared File Systems service. TripleO can use the ControllerStorageNFS server role to configure an NFS Ganesha cluster as the scalable interface to the libcephfs back end.

Ephemeral storage is implemented in OpenStack by the Compute service (Nova). The Compute service uses the KVM hypervisor with libvirt to launch compute workloads as VMs. The Compute service requires two types of storage for libvirt operations:

When Ceph storage is integrated into RHOSP, TripleO configures the Compute service to use RADOS Block Devices (RBD) as the back end. With RBD, instance operating system disks can be managed either as ephemeral, to be deleted when the instance is shut down, or as a persistent volume. An ephemeral disk behaves like a normal disk, to be listed, formatted, mounted, and used as a block device. However, the disk and its data cannot be preserved or accessed beyond the instance that it is attached to.

In earlier OpenStack releases, a disk of a running VM appeared in the /var/lib/nova/instances/<uuid> directory of the hypervisor file system. Earlier Ceph versions could boot VMs only with the Block Storage services boot-from-volume function.

In recent versions, you can boot every VM inside Ceph directly without using the Block Storage service. This feature enables hypervisors to use the live-migration and evacuate operations to restore VMs in another hypervisor during a maintenance operation or on a hardware failure.