After completing this section, you should be able to configure CephFS, including snapshots, replication, memory management, and client access.
Use the following commands to manage CephFS file systems:
| Action | Command |
|---|---|
| Create a file system. |
ceph fs new
|
| List existing file systems. |
ceph fs ls
|
| Remove a file system. |
ceph fs rm fs-name [--yes-i-really-mean-it]
|
| Force MDS to fail status. |
ceph mds fail
|
| Declare an MDS to be repaired, triggering a failback. |
ceph mds repaired role
|
CephFS provides tools to inspect and repair MDS journals (cephfs-journal-tool) or MDS tables (cephfs-table-tool), and to inspect and rebuild metadata (cephfs-data-scan).
For troubleshooting, it is useful to determine which OSDs store a file's objects. Directories or zero-length files might have any associated objects in a data pool.
This example retrieves object mapping information for a file within Ceph:
Retrieve the inode number for the file.
[ceph: root@server /]# stat -c %i filepath
1099511627776Convert the inode number to hexadecimal.
Use the %x formatting output of the printf command.
[ceph: root@server /]# printf '%x\n' 1099511627776
10000000000This example combines these first two steps:
[ceph: root@server /]# printf '%x\n' $(stat -c %i filepath)Search for the hexadecimal ID in the RADOS object list. A large file might return multiple objects.
[ceph: root@server /]# rados -p cephfs_data ls | grep 10000000000
10000000000.00000000Retrieve the mapping information for the returned objects.
[ceph: root@server /]# ceph osd map cephfs_data 10000000000.00000000
osdmap e95 pool 'cephfs_data' (3) object '10000000000.00000000' -> pg 3.f0b56f30
(3.30) -> up ([1,2], p1) acting ([1,2], p1)Interpret this output as saying that the e95 map epoch of the OSD map for the cephfs_data pool (ID 3) maps the 10000000000.00000000 object to placement group 3.30, which is on OSD 1 and OSD 2, and OSD 1 is primary.
If the OSDs in up and acting status are not the same, then it implies that the cluster is rebalancing or has another issue.
The RADOS layout controls how files are mapped to objects. These settings are stored in virtual extended attributes (xattrs) in CephFS. You can adjust settings to control the size of the used objects and where they are stored.
Layout attributes are initially set on the directory at the top of the CephFS file system. You can manually set layout attributes on other directories or files. When you create a file, it inherits layout attributes from its parent directory. If layout attributes are not set in its parent directory, then the closest ancestor directory with layout attributes is used.
The layout attributes for a file, such as these examples, use the ceph.file.layout prefix.
Table 10.2. File Layout Attributes
| Attribute name | Description |
|---|---|
ceph.file.layout.pool
| The pool where Ceph stores the file's data objects (normally cephfs_data). |
ceph.file.layout.stripe_unit
| The size (in bytes) of a block of data that is used in the RAID 0 distribution of a file. |
ceph.file.layout.stripe_count
| The number of consecutive stripe units that constitute a RAID 0 "stripe" of file data. |
ceph.file.layout.object_size
| File data is split into RADOS objects of this size in bytes (4194304 bytes, or 4 MiB, by default). |
ceph.file.layout.pool_namespace
| The namespace that is used, if any. |
The ceph.dir.layout prefix identifies the layout attributes for a directory.
Table 10.3. Directory Layout Attributes
| Attribute name | Description |
|---|---|
ceph.dir.layout.pool
| This attribute specifies the pool where Ceph stores the directory's data objects (normally cephfs_data). |
ceph.dir.layout.stripe_unit
| This attribute specifies the size (in bytes) of a block of data for the RAID 0 distribution of a directory. |
ceph.dir.layout.stripe_count
| This attribute specifies the number of consecutive stripe units that constitute a RAID 0 "stripe" of directory data. |
ceph.dir.layout.object_size
| Directory data is split into RADOS objects of this size (4194304 bytes, or 4 MiB, by default). |
ceph.dir.layout.pool_namespace
| This attribute specifies the name space that is used, if any. |
The getfattr command displays the layout attributes for a file or directory:
[ceph: root@server /]#getfattr -n ceph.file.layout file-path# file: file-path ceph.file.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data" [ceph: root@server /]#getfattr -n ceph.dir.layout# file : directory-path ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data"directory-path
The setfattr command modifies the layout attributes:
[ceph: root@server /]#setfattr -n ceph.file.layout.[ceph: root@server /]#attribute-v value filesetfattr -n ceph.dir.layout.attribute-v value directory
Layout attributes are set when data is initially saved to a file. If the parent directory's layout attributes change after the file is created, then the file's layout attributes do not change. Additionally, a file's layout attributes can be changed only if it is empty.
You can use virtual extended attributes for information about CephFS file system use.
The getfattr command, when used with the ceph attribute namespace on a directory, returns a list of recursive statistics for that directory.
[ceph: root@server /]# getfattr -d -m ceph.dir.* directory-path
file: directory-path
ceph.dir.entries="1"
ceph.dir.files="0"
ceph.dir.rbytes="424617209"
ceph.dir.rctime="1341146808.804098000"
ceph.dir.rentries="39623"
ceph.dir.rfiles="37362"
ceph.dir.rsubdirs="2261"
ceph.dir.subdirs="1"The statistics provide detailed information.
Table 10.4. CephFS Statistics
| Attribute name | Description |
|---|---|
ceph.dir.entries
| Number of direct descendants |
ceph.dir.files
| Number of regular files in the directory |
ceph.dir.rbytes
| Total file size in the subtree (the directory and all its descendants) |
ceph.dir.rctime
| Most recent creation time in the subtree (in seconds since the epoch, 1970-01-01 00:00:00 UTC) |
ceph.dir.rentries
| Number of descendants in the subtree |
ceph.dir.rfiles
| Number of regular files in the subtree |
ceph.dir.rsubdirs
| Number of directories in the subtree |
ceph.dir.subdirs
| Number of directories in the directory |
CephFS enables asynchronous snapshots by default when deploying Red Hat Ceph Storage 5.
These snapshots are stored in a hidden directory called .snap.
In earlier Red Hat Ceph Storage versions, snapshots were disabled by default, as they were an experimental feature.
Use cephfs set to enable snapshot creation for an existing CephFS file system.
[ceph: root@server /]# ceph fs set fs-name allow_new_snaps trueTo create a snapshot, first mount the CephFS file system on your client node.
Use the -o fs=_fs-name option to mount a CephFS file system when you have more than one.
Then, create a subdirectory inside the .snap directory.
The snapshot name is the new subdirectory name.
This snapshot contains a copy of all the current files in the CephFS file system.
[root@node ~]#mount.ceph server.example.com:/ /mnt/mycephfs[root@node ~]#mkdir /mnt/mycephfs/.snap/snap-name
Authorize the client to make snapshots for the CephFS file system with the s option.
[ceph: root@target /]# ceph fs authorize fs-name client path rwsTo restore a file, copy it from the snapshot directory to another normal directory.
[root@node ~]# cp -a .snap/snap-name/file-name .To fully restore a snapshot from the .snap directory tree, replace the normal entries with copies from the chosen snapshot.
[root@node ~]#rm -rf *[root@node ~]#cp -a .snap/snap-name/* .
To discard a snapshot, remove the corresponding directory in .snap.
The rmdir command succeeds even if the snapshot directory is not empty, without needing to use a recursive rm command.
[root@node ~]# rmdir .snap/snap-nameYou can use CephFS to schedule snapshots.
The snap_schedule module manages the scheduled snapshots.
You can use this module to create and delete snapshot schedules.
Snapshot schedule information is stored in the CephFS metadata pool.
To create a snapshot schedule, first enable the snap_schedule module on the MGR node.
[ceph: root@server /]# ceph mgr module enable snap_scheduleThen, add the new snapshot schedule.
[ceph: root@server /]# ceph fs snap-schedule add fs-path time-period [start-time]If an earlier version than Python 3.7 is installed, then the start-time string must use the format %Y-%m-%dT%H:%M:%S.
For Python version 3.7 or later, you can use more flexible date parsing.
For example, to create a snapshot schedule to create a snapshot for the /volume folder every hour, you can use the ceph fs snap-schedule add command.
[ceph: root@server /]# ceph fs snap-schedule add /volume 1h
Schedule set for path /volumeOn the client node, review the snapshots in the .snap folder on your mounted CephFS:
[root@node ~]# ls /mnt/mycephfs/.snap
scheduled-2021-10-06-08_00_00
scheduled-2021-10-06-09_00_00
scheduled-2021-10-06-10_00_00You can list the snapshot schedules for a path with the list option:
[ceph: root@server /]# ceph fs snap-schedule list fs-pathUse the status option to verify the details for the snapshot schedules.
[ceph: root@server /]# ceph fs snap-schedule status fs-pathRemove a snapshot schedule by specifying the path.
[ceph: root@server /]# ceph fs snap-schedule remove fs-pathYou can activate and deactivate snapshot schedules through the activate and deactivate options.
When you add a snapshot schedule, it is activated by default if the path exists.
However, if the path does not exist, then it is set as inactive, so you can activate it later when you create the path.
[ceph: root@server /]# ceph fs snap-schedule activate/deactivate
getfattr(1), and setfattr(1) man pages
For more information, refer to the Red Hat Ceph Storage 5 File System Guide at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/file_system_guide/index
For more information regarding snapshots, refer to the Ceph File System snapshots chapter in the Red Hat Ceph Storage 5 File System Guide at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/file_system_guide/index#ceph-file-system-snapshots