Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

OADP uses Velero to back up all the Kubernetes resources to cloud object storage. You can filter objects to back up and restore by namespace, resource type, and label.

Velero not only exports and imports resources; it also performs additional processing on the resources. The following list describes some of this additional processing:

You can add backup and export logic by using Velero plug-ins.

OAPD can back up volumes by using cloud native snapshot API or the Kubernetes Container Storage Interface (CSI) snapshot API.

Red Hat recommends the use of snapshots when possible, because this feature enables fast and consistent backup.

For volumes that are not supported by either a cloud-native snapshot API or the CSI snapshot API, OADP provides a file-system backup solution.

File System Backup (FSB), or Pod Volume Backup, enables backing up almost any kind of Kubernetes volume that does not support the snapshot capability, such as NFS and local storage.

OADP uses the open source Restic tool to back up the application volume file system.

Restic accesses the volume file system from the volume mount point on the cluster node. For that reason, the application pod must be running during the backup.

During the file system backup, the application can still alter the file system and corrupt the backup. To prevent such a scenario, the application must be quiesced to prevent any write operations while the backup is in progress.

You can use backup and restore hooks to run commands before and after the backup and restore operation. By using a pre-hook, it is possible to quiesce an application, such as a transactional database, to flush pending I/O operations to the storage back end and pause future application writing operations to avoid any data loss or corruption during the backup. When the backup is complete, a post-hook is used to resume the application operations.

If you use volume snapshots, then the hooks are executed just before and after the snapshot creation, which significantly reduces the time that the application is in read-only mode.

Some CSI drivers might store the snapshot in the same storage location as the original volume and might be vulnerable to data loss in the event of a disaster. The storage solution that is used with Red Hat OpenShift Container Platform might not guarantee snapshot durability. In such scenarios, you can enable the OADP Data Mover to copy the snapshot content to a remote backup storage location and remove the snapshot from the storage back end.

Data Mover uses the Restic component of the VolSync operator for replication. Restic secures backup content that is stored on the remote storage location with a user-defined password (also called an encryption key).

OADP uses Velero's Volume Snapshot Mover (VSM) plug-in to integrate with VolSync. The VolSync operator and the VSM Velero plug-in must be installed to enable Data Mover.

You can extend the OADP capabilities and backup logic by using the Velero plug-in system. Support for various cloud providers, storage solutions, and additional CRDs are available with integrated plug-ins.

OADP includes the following plug-ins that you can enable during the installation:

OADP can also use third-party plug-ins, such as the following examples, to extend support to additional storage solutions and custom Kubernetes resources.

The OADP operator provides the following API resources:

OADP configuration requires a user with the cluster-admin role. Backup and restore operations require administrative privileges in the OADP namespace (openshift-adp by default).

OADP requires an object storage location to store the backups. The following list is an example of object storage providers that are available for storing backups:

OpenShift Data Foundation (ODF) is the storage solution that is used in this classroom environment. ODF provides two object storage services: the MultiCloud Object Gateway (NooBaa MCG) and the Ceph RADOS Object Gateway (Ceph RGW). Both services are fully compatible with OADP by using the aws provider plug-in.

For more details about OpenShift Data Foundation, refer to the DO370: Enterprise Kubernetes Storage with Red Hat OpenShift Data Foundation training course.

Similar to a persistent volume claim, you can use an object bucket claim to request an S3-compatible bucket. ODF creates a secret and a configuration map with the same name as the object bucket claim with the credentials and information to access the bucket.

The following is an example of an object bucket claim definition:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: my-bucket-claim
  namespace: my-namespace
spec:
  storageClassName: storage-class-name 
  generateBucketName: my-bucket 

A few minutes after the object bucket claim creation, the status is set to Bound when the bucket is created and ready to use.

[user@host ~]$ oc get obc
NAME             STORAGE-CLASS                 PHASE   AGE
my-bucket-claim  openshift-storage.noobaa.io   Bound   5m

ODF creates a configuration map with the bucket information in the same namespace as the object bucket claim:

[user@host ~]$ oc get configmap/my-bucket-claim -oyaml
apiVersion: v1
data:
  BUCKET_HOST: s3.openshift-storage.svc 
  BUCKET_NAME: my-bucket-9ce46e22-2fb8-4a46-af95-f6949d87c3fd 
  BUCKET_PORT: "443" 
  BUCKET_REGION: "" 
  BUCKET_SUBREGION: ""
kind: ConfigMap
...output omitted...

If the BUCKET_HOST uses an internal service URL that ends with .svc, then the URL can be reached only from a workload that is running inside the cluster. The s3 route in the openshift-storage namespace provides the external endpoint URL.

[user@host ~]$ oc get route/s3 -n openshift-storage \
  -o jsonpath='{.spec.host}{"\n"}'
s3-openshift-storage.apps.ocp4.example.com

The S3 bucket credentials are stored in a secret in the same namespace as the object bucket claim:

[user@host ~]$ oc extract secret/my-bucket-claim --to=-
# AWS_ACCESS_KEY_ID
YEAsbMJnG3o1bGANZprt
# AWS_SECRET_ACCESS_KEY
xjaeCDhskn7lfrdA7WqzoUxpiRYuyjc9uDaWlMw3

To use volume snapshots with CSI drivers, a volume snapshot class must be created to register the CSI driver to use to create snapshots. The driver name must be the same as the provisioner attribute of the volume storage class to back up.

[user@host ~]$ oc get storageclasses
NAME                          PROVISIONER
nfs-storage (default)         k8s-sigs.io/nfs-subdir-external-provisioner
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com
ocs-storagecluster-ceph-rgw   openshift-storage.ceph.rook.io/bucket
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com
openshift-storage.noobaa.io   openshift-storage.noobaa.io/obc

[user@host ~]$ oc get volumesnapshotclasses
NAME                                        DRIVER
ocs-storagecluster-cephfsplugin-snapclass   openshift-storage.cephfs.csi.ceph.com
ocs-storagecluster-rbdplugin-snapclass      openshift-storage.rbd.csi.ceph.com

The features section describes the configuration of additional features, such as Data Mover.

  features:
    dataMover:
      enable: true 
      credentialName: restic-enc-key 
      maxConcurrentBackupVolumes: "3" 
      maxConcurrentRestoreVolumes: "3"
      volumeOptionsForStorageClasses: 
        gp2-csi-copy-1:
          destinationVolumeOptions:
            storageClassName: csi-copy-2
          sourceVolumeOptions:
            storageClassName: csi-copy-1

To encrypt and decrypt the backups, Restic uses the stored password in the secret that the credentialName attribute defines. This secret must be created in the OADP namespace by using the following YAML template:

apiVersion: v1
kind: Secret
metadata:
  name: restic-enc-key
type: Opaque
stringData:
  RESTIC_PASSWORD: my-secure-restic-password

The configuration section describes the configuration OADP components, such as Velero and Restic.

  configuration:
    restic:
      enable: false 
      podConfig: 
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 8Gi
          requests:
            cpu: 500m
            memory: 256Mi
        labels:
          mylabel: myvalue
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""
        tolerations:
        - effect: NoSchedule
          key: somekey
          operator: Exists
    velero:
      defaultPlugins: 
        - aws
        - openshift
      podConfig: 
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 8Gi
          requests:
            cpu: 500m
            memory: 256Mi

The backupLocation section describes the object storage location for backups. Multiple backup locations can be defined. The following example uses the aws provider plug-in:

  backupLocations:
    - velero:
        config: 
          profile: "default" 
          region: "us-east-1"
          s3Url: https://s3.openshift-storage.svc
          s3ForcePathStyle: "true"
          insecureSkipTLSVerify: "true"
        provider: aws 
        default: true 
        credential: 
          key: cloud
          name: cloud-credentials
        objectStorage:
          bucket: my-bucket 
          prefix: oadp 
          caCert: LLS0S0...LS0tLS0K 

With the aws provider plug-in, you can configure any S3-compatible object storage by setting the s3Url attribute with a custom S3 endpoint URL. OADP can access the S3 endpoint by using two addressing models:

If the object storage solution does not support the DNS-style method, then you can configure OADP to use the path-style model instead with the s3ForcePathStyle option.

The region attribute must be set even if the storage solution does not use a region. You can then set this attribute to the standard AWS region, such as "us-east-1".

The Velero secret contains the credentials to access the object storage. If the secret name is not specified in the configuration, then Velero uses the cloud-credentials default secret name. You can create the Velero secret with a configuration file that is specific to the provider plug-in. The following Velero configuration file is for the aws provider plug-in:

[user@host ~]$ cat credentials-velero

[myProfile] 
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

You can then create the Velero secret with the following oc create secret command:

[user@host ~]$ oc create secret generic cloud-credentials \
  --from-file cloud=credentials-velero

The snapshotLocations section describes the storage configuration to use for volume snapshots when using the cloud provider native snapshot capability. For CSI snapshots, this section is not needed, because volume snapshot classes are used instead. With Data Mover and file system backup with Restic, backups are stored in the object storage location that is defined in the backupLocations section.

You can configure multiple snapshot locations per provider. The following example defines one snapshot location by using the aws provider plug-in:

  snapshotLocations:
    - name: default
      velero:
        provider: aws
        config:
          region: us-west-2 
          profile: "default" 
        credential: 
          key: cloud
          name: cloud-credentials

If the secret name is not specified in the configuration, then Velero uses the cloud-credentials default secret name. The same secret can be shared with the snapshot location and the backup location.

See the references section for configuring OADP with other cloud providers such as Microsoft Azure and Google Cloud Platform.

CSI snapshots with OADP require additional configuration on the volume snapshot classes. To prevent deletion of snapshots if the application namespace is removed, you must set the deletion policy to Retain. You can use the following command to change the deletion policy to Retain:

[user@host ~]$ oc patch volumesnapshotclass my-snapshot-storageclass \
  --type=merge -p '{"deletionPolicy": "Retain"}'
my-snapshot-storageclass patched

[user@host ~]$ oc get volumesnapshotclass my-snapshot-storageclass
NAME                         DRIVER                DELETIONPOLICY   AGE
my-snapshot-storageclass     some.csi.driver.com   Retain           1d

For OADP to use a volume snapshot class, you must add the velero.io/csi-volumesnapshot-class: "true" label. Only one volume snapshot class per driver needs this label.

You can use the following command to add the required label:

[user@host ~]$ oc label volumesnapshotclass my-snapshot-storageclass \
  velero.io/csi-volumesnapshot-class="true"
my-snapshot-storageclass labeled