File-based storage organizes your data as a traditional file system. Data is saved as files with a name and associated metadata, such as modification time stamps, an owner, and access permissions. File-based storage uses a directory tree hierarchy to organize how files are stored.

Block-based storage provides a storage volume that operates similar to a disk device, organized into equally sized chunks. Typically, block-based storage volumes are either formatted with a file system, or applications such as databases directly access and write to them.

With object-based storage, you can store arbitrary data and metadata as a unit that is labeled with a unique identifier in a flat storage pool. Rather than accessing data as blocks or in a file-system hierarchy, you use an API to store and retrieve objects. Fundamentally, the Red Hat Ceph Storage RADOS cluster is an object store.

The Metadata Server (MDS) manages metadata for CephFS clients. This daemon provides information that CephFS clients need to access RADOS objects, such as providing file locations within the file-system tree. MDS manages the directory hierarchy and stores file metadata, such as the owner, time stamps, and permission modes, in a RADOS cluster. MDS is also responsible for access caching and managing client caches to maintain cache coherence.

MDS daemons operate in two modes: active and standby. An active MDS manages the metadata on the CephFS file system. A standby MDS serves as a backup, and switches to the active mode if the active MDS becomes unresponsive. CephFS shared file systems require an active MDS service. You should deploy at least one standby MDS in your cluster to ensure high availability.

If you do not create enough MDS pools to match the number of configured standby daemons, then the Ceph cluster displays a WARN health status. The recommended solution is create more MDS pools to provide a pool for each daemon. However, a temporary solution is to set the number of standby pools to 0, which disables the Ceph MDS standby check through the ceph fs set fs-name standby_count_wanted 0 command.

CephFS clients first contact a MON to authenticate and retrieve the cluster map. Then, the client queries an active MDS for file metadata. The client uses the metadata to access the objects that comprise the requested file or directory by communicating directly with the OSDs.

MDS features and configuration options are described in the following list:

Red Hat Ceph Storage 5 removes limitations from earlier versions.

Use ceph fs volume to directly create the CephFS volume. This command creates pools that are associated to the CephFS, creates the CephFS volume, and also starts the MDS service on the hosts.

[ceph: root@server /]# ceph fs volume create fs-name \
--placement="number-of-hosts list-of-hosts"

For more control over the deploying process, manually create the pools that are associated to the CephFS, start the MDS service on the hosts, and create the CephFS file system.

Use the Ceph Orchestrator to deploy the MDS service with the service specification. First, manually create the two required pools. Then, create a YAML file with the service details:

service_type: mds
service_id: fs-name
placements:
  hosts:
    - host-name-1
    - host-name-2
    - ...

Use the YAML service specification to deploy the MDS service with the ceph orch apply command:

[ceph: root@server /]# ceph orch apply -i file-name.yml

Finally, create the CephFS file system with the ceph fs new command.

To mount a CephFS-based file system with either client, verify the following prerequisites on the client host.

When the prerequisites are met, use the FUSE client to mount and unmount a CephFS file system:

[root@node ~]# ceph-fuse [mount-point] [options]

To provide the key ring for a specific user, use the --id option.

You must authorize the client to access the CephFS file system, by using the ceph fs authorize command:

[ceph: root@server /]# ceph fs authorize fs-name client-name path permissions

With the ceph fs authorize command, you can provide fine-grained access control for different users and folders in the CephFS file system. You can set different options for folders in a CephFS file system:

This example allows one user to read the root folder, and also provides read, write, and snapshot permissions to the /directory folder.

[ceph: root@server /]# ceph fs authorize mycephfs client.user / r /directory rws

By default, the CephFS FUSE client mounts the root directory (/) of the accessed file system. You can mount a specific directory with the ceph-fuse -r directory command.

When more than one CephFS file system is configured, the CephFS FUSE client mounts the default CephFS file system. To use a different file system, use the --client_fs option.

To persistently mount your CephFS file system by using the FUSE client, you can add the following entry to the /etc/fstab file:

host-name:_port_ mount-point fuse.ceph ceph.id=myuser,ceph.client_mountpoint=mountpoint,_netdev 0 0

Use the umount command to unmount the file system:

[root@node ~]# umount mount-point

When using the CephFS kernel client, use the following command to mount the file system:

[root@node ~]# mount -t ceph [device]:[path] [mount-point] \
-o [key-value] [other-options]

You must authorize the client to access the CephFS file system, with the ceph fs authorize command. Extract the client key with the ceph auth get command, and then copy the key to the /etc/ceph folder on the client host.

With the CephFS kernel client, you can mount a specific subdirectory from a CephFS file system. This example mounts a directory called /dir/dir2 from the root of a CephFS file system:

[root@node ~]# mount -t ceph mon1:/dir1/dir2 mount-point

You can specify a list of several comma-separated MONs to mount the device. The standard port (6789) is the default, or you can add a colon and a nonstandard port number after the name of each MON. Recommended practice is to specify more than one MON in case that some are offline when the file system is mounted.

These other options are available when using the CephFS kernel client:

To persistently mount your CephFS file system by using the kernel client, you can add the following entry to the /etc/fstab file:

mon1,mon2:/ mount_point ceph name=user1,secretfile=/root/secret,_netdev 0 0

Use the umount command to unmount the file system:

[root@node ~]# umount mount_point

You can remove a CephFS if needed. However, first back up all your data, because removing your CephFS file system destroys all the stored data on that file system.

The procedure to remove a CephFS is first to mark it as down, as follows:

[ceph: root@server /]# ceph fs set fs-name down true

Then, you can remove it with the next command:

[ceph: root@server /]# ceph fs rm fs-name --yes-i-really-mean-it

Red Hat Ceph Storage 5 provides access to Ceph storage from an NFS client, with NFS Ganesha. NFS Ganesha is a user space NFS file server that supports multiple protocols, such as NFSv3, NFSv4.0, NFSv4.1, and pNFS. NFS Ganesha uses a File System Abstraction Layer (FSAL) architecture, to support and share files from multiple file systems or lower-level storage, such as Ceph, Samba, Gluster, and Linux file systems such as XFS.

In Red Hat Ceph Storage, NFS Ganesha shares files with the NFS 4.0 or later protocol. This requirement is necessary for proper feature functioning by the CephFS client, the OpenStack Manila File Sharing service, and other Red Hat products that are configured to access the NFS Ganesha service.

The following list outlines the advantages of a user space NFS server:

You can deploy NFS Ganesha in an active-active configuration on top of an existing CephFS file system through the ingress service. The main goal of this active-active configuration is for load balancing, and scaling to many instances that handle higher loads. Thus, if one node fails, then the cluster redirects all the workload to the rest of the nodes.

System administrators can deploy the NFS Ganesha daemons via the CLI or manage them automatically if either the Cephadm or Rook orchestrators are enabled.

The following list outlines the advantages to having an ingress service on top of an existing NFS service:

You can use multiple active-active NFS Ganesha services with Pacemaker for high availability. The Pacemaker component is responsible for all cluster-related activities, such as monitoring cluster membership, managing the services and resources, and fencing cluster members.

As prerequisites, create a CephFS file system and install the nfs-ganesha, nfs-ganesha-ceph, nfs-ganesha-rados-grace, and nfs-ganesha-rados-urls packages on the Ceph MGR nodes.

After the prerequisites are satisfied, enable the Ceph MGR NFS module:

[ceph: root@server /]# ceph mgr module enable nfs

Then, create the NFS Ganesha cluster:

[ceph: root@server /]# ceph nfs cluster create cluster-name "node-list"

The node-list is a comma-separated list where the daemon containers are deployed.

Next, export the CephFS file system:

[ceph: root@server /]# ceph nfs export create cephfs fs-name \
cluster-name pseudo-path

The pseudo-path parameter is the pseudo root path.

Finally, mount the exported CephFS file system on a client node.

[root@node ~]# mount -t nfs -o port=ganesha-port node-name:_pseudo-path_ path

CephFS shared file systems require at least one active MDS service for correct operation, and at least one standby MDS to ensure high availability. The MDS autoscaler module ensures the availability of enough MDS daemons.

This module monitors the number of ranks and the number of standby daemons, and adjusts the number of MDS daemons that the orchestrator spawns.

To enable the MDS autoscaler module, use the following command:

[ceph: root@server /]# ceph mgr module enable mds_autoscaler

Red Hat Ceph Storage 5 supports CephFS multi-site configuration for geo-replication. Thus, you can replicate the CephFS file system on another Red Hat Ceph Storage cluster. With this feature, you can fail over to the secondary CephFS file system and restart the applications that use it. The CephFS file system mirroring feature requires the cephfs-mirror package.

The CephFS mirroring feature is snapshot-based. The first snapshot synchronization requires bulk transfer of the data from the source cluster to the remote cluster. Then, for the following synchronizations, the mirror daemon identifies the modified files between local snapshots and synchronizes those files in the remote cluster. This synchronization method is faster than other methods that require bulk transfer of the data to the remote cluster, because it does not need to query the remote cluster (file differences are calculated between local snapshots) and needs only to transfer the updated files to the remote cluster.

The CephFS mirroring module is disabled by default. To configure a snapshot mirror for CephFS, you must enable the mirroring module on the source and remote clusters:

[ceph: root@server /]# ceph mgr module enable mirroring

Then, you can deploy the CephFS mirroring daemon on the source cluster:

[ceph: root@source /]# ceph orch apply cephfs-mirror [node-name]

The previous command deploys the CephFS mirroring daemon on node-name and creates the Ceph user cephfs-mirror. For each CephFS peer, you must create a user on the target cluster:

[ceph: root@target /]# ceph fs authorize fs-name client_ / rwps

Then, you can enable mirroring on the source cluster. Mirroring must be enabled for a specific file system.

[ceph: root@source /]# ceph fs snapshot mirror enable fs-name

The next step is to prepare the target peer. You can create the peer bootstrap in the target node with the next command:

[ceph: root@target /]# ceph fs snapshot mirror peer_bootstrap create \
fs-name peer-name site-name

You can use the site-name string to identify the target storage cluster. When the target peer is created, you must import into the source cluster the bootstrap token from creating the peer on the target cluster:

[ceph: root@source /]# ceph fs snapshot mirror peer_bootstrap import \
fs-name bootstrap-token

Finally, configure a directory for snapshot mirroring on the source cluster with the following command:

[ceph: root@source /]# ceph fs snapshot mirror add fs-name path