Bookmark this page

Managing Shared File Storage

Objectives

After completing this section, you should be able to configure CephFS, including snapshots, replication, memory management, and client access.

CephFS Administration

Use the following commands to manage CephFS file systems:

ActionCommand
Create a file system. ceph fs new fs-name meta-pool data-pool
List existing file systems. ceph fs ls
Remove a file system. ceph fs rm fs-name [--yes-i-really-mean-it]
Force MDS to fail status. ceph mds fail gid/name/role
Declare an MDS to be repaired, triggering a failback. ceph mds repaired role

CephFS provides tools to inspect and repair MDS journals (cephfs-journal-tool) or MDS tables (cephfs-table-tool), and to inspect and rebuild metadata (cephfs-data-scan).

Mapping a File to an Object

For troubleshooting, it is useful to determine which OSDs store a file's objects. Directories or zero-length files might have any associated objects in a data pool.

This example retrieves object mapping information for a file within Ceph:

  • Retrieve the inode number for the file.

    [ceph: root@server /]# stat -c %i filepath
    1099511627776
  • Convert the inode number to hexadecimal. Use the %x formatting output of the printf command.

    [ceph: root@server /]# printf '%x\n' 1099511627776
    10000000000

    This example combines these first two steps:

    [ceph: root@server /]# printf '%x\n' $(stat -c %i filepath)
  • Search for the hexadecimal ID in the RADOS object list. A large file might return multiple objects.

    [ceph: root@server /]# rados -p cephfs_data ls | grep 10000000000
    10000000000.00000000
  • Retrieve the mapping information for the returned objects.

    [ceph: root@server /]# ceph osd map cephfs_data 10000000000.00000000
    osdmap e95 pool 'cephfs_data' (3) object '10000000000.00000000' -> pg 3.f0b56f30
     (3.30) -> up ([1,2], p1) acting ([1,2], p1)

Interpret this output as saying that the e95 map epoch of the OSD map for the cephfs_data pool (ID 3) maps the 10000000000.00000000 object to placement group 3.30, which is on OSD 1 and OSD 2, and OSD 1 is primary. If the OSDs in up and acting status are not the same, then it implies that the cluster is rebalancing or has another issue.

Controlling the RADOS Layout of Files

The RADOS layout controls how files are mapped to objects. These settings are stored in virtual extended attributes (xattrs) in CephFS. You can adjust settings to control the size of the used objects and where they are stored.

Layout attributes are initially set on the directory at the top of the CephFS file system. You can manually set layout attributes on other directories or files. When you create a file, it inherits layout attributes from its parent directory. If layout attributes are not set in its parent directory, then the closest ancestor directory with layout attributes is used.

The layout attributes for a file, such as these examples, use the ceph.file.layout prefix.

Table 10.2. File Layout Attributes

Attribute nameDescription
ceph.file.layout.pool The pool where Ceph stores the file's data objects (normally cephfs_data).
ceph.file.layout.stripe_unit The size (in bytes) of a block of data that is used in the RAID 0 distribution of a file.
ceph.file.layout.stripe_count The number of consecutive stripe units that constitute a RAID 0 "stripe" of file data.
ceph.file.layout.object_size File data is split into RADOS objects of this size in bytes (4194304 bytes, or 4 MiB, by default).
ceph.file.layout.pool_namespace The namespace that is used, if any.

The ceph.dir.layout prefix identifies the layout attributes for a directory.

Table 10.3. Directory Layout Attributes

Attribute nameDescription
ceph.dir.layout.pool This attribute specifies the pool where Ceph stores the directory's data objects (normally cephfs_data).
ceph.dir.layout.stripe_unit This attribute specifies the size (in bytes) of a block of data for the RAID 0 distribution of a directory.
ceph.dir.layout.stripe_count This attribute specifies the number of consecutive stripe units that constitute a RAID 0 "stripe" of directory data.
ceph.dir.layout.object_size Directory data is split into RADOS objects of this size (4194304 bytes, or 4 MiB, by default).
ceph.dir.layout.pool_namespace This attribute specifies the name space that is used, if any.

The getfattr command displays the layout attributes for a file or directory:

[ceph: root@server /]# getfattr -n ceph.file.layout file-path
# file: file-path
ceph.file.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304
 pool=cephfs_data"
[ceph: root@server /]# getfattr -n ceph.dir.layout directory-path
# file : directory-path
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304
 pool=cephfs_data"

The setfattr command modifies the layout attributes:

[ceph: root@server /]# setfattr -n ceph.file.layout.attribute -v value file
[ceph: root@server /]# setfattr -n ceph.dir.layout.attribute -v value directory

Important

Layout attributes are set when data is initially saved to a file. If the parent directory's layout attributes change after the file is created, then the file's layout attributes do not change. Additionally, a file's layout attributes can be changed only if it is empty.

Usage and Statistics

You can use virtual extended attributes for information about CephFS file system use. The getfattr command, when used with the ceph attribute namespace on a directory, returns a list of recursive statistics for that directory.

[ceph: root@server /]# getfattr -d -m ceph.dir.* directory-path
file: directory-path
ceph.dir.entries="1"
ceph.dir.files="0"
ceph.dir.rbytes="424617209"
ceph.dir.rctime="1341146808.804098000"
ceph.dir.rentries="39623"
ceph.dir.rfiles="37362"
ceph.dir.rsubdirs="2261"
ceph.dir.subdirs="1"

The statistics provide detailed information.

Table 10.4. CephFS Statistics

Attribute nameDescription
ceph.dir.entries Number of direct descendants
ceph.dir.files Number of regular files in the directory
ceph.dir.rbytes Total file size in the subtree (the directory and all its descendants)
ceph.dir.rctime Most recent creation time in the subtree (in seconds since the epoch, 1970-01-01 00:00:00 UTC)
ceph.dir.rentries Number of descendants in the subtree
ceph.dir.rfiles Number of regular files in the subtree
ceph.dir.rsubdirs Number of directories in the subtree
ceph.dir.subdirs Number of directories in the directory

Managing Snapshots

CephFS enables asynchronous snapshots by default when deploying Red Hat Ceph Storage 5. These snapshots are stored in a hidden directory called .snap. In earlier Red Hat Ceph Storage versions, snapshots were disabled by default, as they were an experimental feature.

Creating Snapshots

Use cephfs set to enable snapshot creation for an existing CephFS file system.

[ceph: root@server /]# ceph fs set fs-name allow_new_snaps true

To create a snapshot, first mount the CephFS file system on your client node. Use the -o fs=_fs-name option to mount a CephFS file system when you have more than one. Then, create a subdirectory inside the .snap directory. The snapshot name is the new subdirectory name. This snapshot contains a copy of all the current files in the CephFS file system.

[root@node ~]# mount.ceph server.example.com:/ /mnt/mycephfs
[root@node ~]# mkdir /mnt/mycephfs/.snap/snap-name

Authorize the client to make snapshots for the CephFS file system with the s option.

[ceph: root@target /]# ceph fs authorize fs-name client path rws

To restore a file, copy it from the snapshot directory to another normal directory.

[root@node ~]# cp -a .snap/snap-name/file-name .

To fully restore a snapshot from the .snap directory tree, replace the normal entries with copies from the chosen snapshot.

[root@node ~]# rm -rf *
[root@node ~]# cp -a .snap/snap-name/* .

To discard a snapshot, remove the corresponding directory in .snap. The rmdir command succeeds even if the snapshot directory is not empty, without needing to use a recursive rm command.

[root@node ~]# rmdir .snap/snap-name

Scheduling Snapshots

You can use CephFS to schedule snapshots. The snap_schedule module manages the scheduled snapshots. You can use this module to create and delete snapshot schedules. Snapshot schedule information is stored in the CephFS metadata pool.

To create a snapshot schedule, first enable the snap_schedule module on the MGR node.

[ceph: root@server /]# ceph mgr module enable snap_schedule

Then, add the new snapshot schedule.

[ceph: root@server /]# ceph fs snap-schedule add fs-path time-period [start-time]

If an earlier version than Python 3.7 is installed, then the start-time string must use the format %Y-%m-%dT%H:%M:%S. For Python version 3.7 or later, you can use more flexible date parsing. For example, to create a snapshot schedule to create a snapshot for the /volume folder every hour, you can use the ceph fs snap-schedule add command.

[ceph: root@server /]# ceph fs snap-schedule add /volume 1h
Schedule set for path /volume

On the client node, review the snapshots in the .snap folder on your mounted CephFS:

[root@node ~]# ls /mnt/mycephfs/.snap
scheduled-2021-10-06-08_00_00
scheduled-2021-10-06-09_00_00
scheduled-2021-10-06-10_00_00

You can list the snapshot schedules for a path with the list option:

[ceph: root@server /]# ceph fs snap-schedule list fs-path

Use the status option to verify the details for the snapshot schedules.

[ceph: root@server /]# ceph fs snap-schedule status fs-path

Remove a snapshot schedule by specifying the path.

[ceph: root@server /]# ceph fs snap-schedule remove fs-path

You can activate and deactivate snapshot schedules through the activate and deactivate options. When you add a snapshot schedule, it is activated by default if the path exists. However, if the path does not exist, then it is set as inactive, so you can activate it later when you create the path.

[ceph: root@server /]# ceph fs snap-schedule activate/deactivate

 

References

getfattr(1), and setfattr(1) man pages

For more information, refer to the Red Hat Ceph Storage 5 File System Guide at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/file_system_guide/index

For more information regarding snapshots, refer to the Ceph File System snapshots chapter in the Red Hat Ceph Storage 5 File System Guide at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/file_system_guide/index#ceph-file-system-snapshots

Revision: cl260-5.0-29d2128