Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

You can start backing up a namespace by using the following backup definition:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: my-app-backup 
  namespace: openshift-adp
spec:
  includedNamespaces: 
  - my-app-project
  ttl: 720h0m0s  
  labelSelector:
    matchLabels: 
      app: my-app
  includedResources: 
  - deployments
  - configmaps
  - secrets
  - services
  - routes

The following example shows a stateless application, followed by the needed definition to back it up in the website namespace. The application is a static website that is built with Hugo and that uses a Source-to-Image (S2I) container image. This application uses the following resources that are created with the app=hugo label:

You can back up this application with the following backup definition:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: website
  namespace: openshift-adp
spec:
  includedNamespaces:
  - website
  labelSelector:
    matchLabels:
      app: hugo
  includedResources:
  - imagestreams
  - buildconfigs
  - deployments
  - services
  - routes

You can review the configuration and the backup status with the oc describe command:

[user@host ~]$ oc -n openshift-adp describe backup/website
Name:         website
Namespace:    openshift-adp
...output omitted...
Spec:
  Csi Snapshot Timeout:          10m0s
  Default Volumes To Fs Backup:  false
  Included Namespaces:
    website
  Included Resources:
    imagestreams
    buildconfigs
    deployments
    services
    routes
  Item Operation Timeout:  1h0m0s
  Label Selector:
    Match Labels:
      App:           hugo
  Storage Location:  oadp-config-1
  Ttl:               720h0m0s
Status:
  Completion Timestamp:  2023-12-11T14:01:42Z 
  Expiration:            2024-01-10T14:01:27Z 
  Format Version:        1.1.0
  Phase:                 Completed 
  Start Timestamp:       2023-12-11T14:01:27Z 
  Version:               1

The backup resource can have the following statuses during its lifetime:

You can initiate a restore by using the following restore definition:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore-name> 
  namespace: openshift-adp
spec:
  backupName: <backup-name> 

OADP restores only the resources that are not already in the destination namespace. The destination namespace is the same as in the backup, unless you use the namespaceMapping field to specify a different namespace.

For example, you can use the following restore definition to restore the website project from the previous example to a new website-stage project:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: website-stage
  namespace: openshift-adp
spec:
  backupName: website
  namespaceMapping:
    website: website-stage

After creating the restore resource, you can review the configuration and monitor the progress of the restore with the oc describe command:

[user@host ~]$ oc -n openshift-adp describe restore/website-stage
Name:         website-stage
Namespace:    openshift-adp
...output omitted...
Spec:
  Backup Name:  website
  Excluded Resources: 
    nodes
    events
    events.events.k8s.io
    backups.velero.io
    restores.velero.io
    resticrepositories.velero.io
    csinodes.storage.k8s.io
    volumeattachments.storage.k8s.io
    backuprepositories.velero.io
  Item Operation Timeout:  1h0m0s
  Namespace Mapping:
    Website:  website-stage
Status:
  Completion Timestamp:  2023-12-12T09:36:23Z 
  Phase:                 Completed 
  Progress: 
    Items Restored:  7
    Total Items:     7
  Start Timestamp:   2023-12-12T09:36:14Z 

The possible statuses of the restore are the same as for the backup resource. A successful restore goes through the following statuses during its lifetime:

During the restore, OADP adds two labels to the imported resources: A velero.io/restore-name label with the restore name, and a velero.io/backup-name with the backup name that the resource came from. You can use those labels to identify the resources that were restored from a specific backup or restore name.

For example, you can list the resources with the velero.io/restore-name=website-stage label that are created by the restore from the previous example:

[user@host ~]$ oc -n website-stage get all \
  -l velero.io/restore-name=website-stage
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)             AGE
service/hugo   ClusterIP   172.30.61.176   <none>       8080/TCP,8443/TCP   65m

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hugo   1/1     1            1           65m

NAME                                  TYPE     FROM   LATEST
buildconfig.build.openshift.io/hugo   Source   Git    0

NAME                            HOST/PORT                                  ...
route.route.openshift.io/hugo   hugo-website-stage.apps.ocp4.example.com   ...

OADP stores in the object storage detailed information about each backup and restore attempt, such as logs, the list of resources that are backed up or restored, and a summary of the errors or warnings that occur during the backup and restore process. This information is not stored in the OADP Kubernetes custom resources, and only limited information such as the runtime status of the backup and restore is available.

The following excerpt shows the object storage layout after the backup and restore from the previous examples:

s3
├── docker 
│   └── registry
│       └── v2
│           ├── blobs
│           └── repositories
│               └── website
│                   └── hugo
└── oadp
    ├── backups 
    │   └── website
    │       ├── website-logs.gz
    │       ├── website-results.gz
    │       └── website.tar.gz
    └── restores 
        └── website-stage
            ├── restore-website-stage-logs.gz
            └── restore-website-stage-results.gz

OADP provides the velero command-line tool that can retrieve backup and restore information from both the object storage and the OpenShift cluster.

The Velero CLI tool is available from the velero deployment in the openshift-adp namespace. You can define an alias to access the velero binary by using the following command:

[user@host ~]$ alias velero=\
  'oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'

The velero command can use the same syntax as the kubectl or oc commands, but is limited to OADP Kubernetes custom resources such as the backup, restore, and schedule resources.

The oc get command displays limited information about OADP resources. However, the velero get command provides the same runtime information from the Kubernetes resources as the oc describe command:

[user@host ~]$ oc -n openshift-adp get backup,restore
NAME                             AGE
backup.velero.io/website         24h

NAME                              AGE
restore.velero.io/website-stage   4h27m


[user@host ~]$ velero get backup
NAME     STATUS     ERRORS WARNINGS CREATED  EXPIRES STORAGE LOCATION  SELECTOR
website  Completed  0      0        ...      29d     oadp-config-1     app=hugo

[user@host ~]$ velero get restore
NAME           BACKUP  STATUS     STARTED  COMPLETED  ERRORS  WARNINGS  ...
website-stage  website Completed  ...      ...        0       0         ...

The Velero tool provides a describe command that is similar to the oc describe command, and adds a --details option that retrieves additional information about the resource from the object storage:

[user@host ~]$ velero describe backup website --details
Name:         website
Namespace:    openshift-adp
...output omitted...

Phase:  Completed 

Namespaces: 
  Included:  website
  Excluded:  <none>

Resources:
  Included:        imagestreams, buildconfigs, deployments, services, routes
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app=hugo

...output omitted...

Started:    2023-12-11 14:01:27 +0000 UTC
Completed:  2023-12-11 14:01:42 +0000 UTC

Expiration:  2024-01-10 14:01:27 +0000 UTC

Resource List: 
  apps/v1/Deployment:
    - website/hugo
  build.openshift.io/v1/BuildConfig:
    - website/hugo
  image.openshift.io/v1/ImageStream:
    - website/hugo
    - website/nginx-122
  route.openshift.io/v1/Route:
    - website/hugo
  v1/Service:
    - website/hugo

Velero-Native Snapshots: <none included>

If the backup or restore resource has errors or warnings, then the --details option adds them to the command output.

In the following example, the website project is backed up without filtering by resource type. OADP backs up all resources with the app=hugo label in the website namespace.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: website-label
  namespace: openshift-adp
spec:
  includedNamespaces:
  - website
  labelSelector:
    matchLabels:
      app: hugo

You can use the velero create restore command to restore the previous backup in a new website-dev namespace, as follows:

[user@host ~]$ velero create restore website-dev \
  --from-backup=website-label --namespace-mappings=website:website-dev
Restore request "website-dev" submitted successfully.
Run `velero restore describe website-dev` or `velero restore logs website-dev` for more details.

The restore completes with one warning about an existing resource in the target namespace:

[user@host ~]$ velero describe restore website-dev --details
Name:         website-dev
Namespace:    openshift-adp
...output omitted...

Phase:                       Completed
Total items to be restored:  11
Items restored:              11

Started:    2023-12-12 10:21:21 +0000 UTC
Completed:  2023-12-12 10:21:27 +0000 UTC

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    website-dev:  could not restore, Endpoints "hugo" already exists. Warning: the in-cluster version is different than the backed-up version.

Backup:  website-label

...output omitted...

Resource List:
  apps/v1/Deployment:
    - website-dev/hugo(created)
  build.openshift.io/v1/Build:
    - website-dev/hugo-1(skipped)
  build.openshift.io/v1/BuildConfig:
    - website-dev/hugo(created)
  discovery.k8s.io/v1/EndpointSlice:
    - website-dev/hugo-5hq2x(created)
  image.openshift.io/v1/ImageStream:
    - website-dev/hugo(skipped)
  image.openshift.io/v1/ImageStreamTag:
    - website-dev/hugo:latest(skipped)
  image.openshift.io/v1/ImageTag:
    - website-dev/hugo:latest(skipped)
  route.openshift.io/v1/Route:
    - website-dev/hugo(created)
  v1/Endpoints:
    - website-dev/hugo(failed)
  v1/Namespace:
    - website-dev(created)
  v1/Service:
    - website-dev/hugo(created)

In this example, OADP tries unsuccessfully to restore the hugo endpoint. The hugo service in the backup automatically creates this resource.

Avoid including resources that other resources manage, such as builds, endpoints, or replica sets. It is unnecessary and can cause issues during the restore. You must filter to include in the backup only those resources that your application requires for a successful deployment.

OADP creates crash-consistent backups of your application by using volume snapshots. To back up persistent volumes, you must include the persistentvolumeclaims and persistentvolumes resource types in the backup definition.

OpenShift assigns to each namespace a unique UID and GID that the application pod uses to write data to persistent volumes. If you restore an application to a new namespace, then OpenShift assigns a new set of UIDs and GIDs that prevent the application from accessing its data.

For more details about user and group ID assignments, refer to the DO180: Red Hat OpenShift Administration I: Operating a Production Cluster training course.

So that OADP can restore the UID and GID, include the namespace resource type in the backup definition. If you use a label selector in the backup definition, then you must add the corresponding label to the namespace.

Alternatively, you can use the kubernetes.io/metadata.name label that Kubernetes automatically sets on all namespaces.

To improve the consistency of a backup, you can use backup hooks to specify a list of commands to execute in the application pod before and after the backup is created. You can then use those hooks to quiesce the application and perform an application-consistent backup.

Some applications require additional steps when using volume snapshots to create a usable backup. For more details about the backup procedure, refer to the application documentation.

To configure backup hooks, you must specify the target pod (by using labels), the container name, and the commands to run on that container.

You can configure the following hook types according to your needs:

The following example is a backup definition for a MongoDB database. It uses backup hooks to flush all pending writes to the disk and to lock the database to prevent any writes during the backup.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mongodb
  namespace: openshift-adp
spec:
  includedNamespaces:
  - mongodb
  orLabelSelectors: 
  - matchLabels:
      app: mongodb
  - matchLabels:
      kubernetes.io/metadata.name: mongodb
  includedResources: 
  - deployments
  - services
  - secret
  - pvc
  - pv
  - pods
  - namespace
  hooks:
    resources:
    - name: mongodb-lock
      labelSelector: 
        matchLabels:
          app: mongodb
      pre: 
      - exec:
          container: mongodb
          command:
          - /usr/bin/mongosh
          - --eval
          - db.fsyncLock();
      post: 
      - exec:
          container: mongodb
          command:
          - /usr/bin/mongosh
          - --eval
          - db.fsyncUnlock();

The following restore resource definition restores the MongoDB backup from the previous example:

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: mongodb
  namespace: openshift-adp
spec:
  backupName: mongodb
  hooks:
    resources:
    - name: mongodb-unlock
      labelSelector:
        matchLabels:
          app: mongodb
      postHooks:
      - init: 
          initContainers:
          - name: remove-lock
            image: mongodb/mongodb-community-server:latest
            volumeMounts:
            - name: mongodb-data
              mountPath: /data/db
            command:
            - /usr/bin/rm
            - /data/db/mongod.lock

OADP Data Mover stores the volume snapshots in the object storage in the /openshift-adp/<backup_name> path.

The following excerpt shows the object storage layout after the previous backup and restore from examples:

s3
├── oadp
│   ├── backups
│   │   └── mongodb
│   │       ├── mongodb-logs.gz
│   │       └── ...
│   └── restores
│       └── mongodb
│           ├── restore-mongodb-logs.gz
│           └── ...
└── openshift-adp 
    └── mongodb
        └── snapcontent-1234-6789-pvc
            └── ...

When OADP restores a volume, OADP Data Mover imports the volume snapshot to the application namespace and then creates the application PVC from it. After the restore is complete, you can remove this volume snapshot, which is no longer needed, to free up storage space on the storage back end.

You can identify the volume snapshots and snapshot contents that belong to a restore with the velero.io/restore-name label. For example, to list the volume snapshots that are created with the mongodb restore from the previous example, use the following command:

[user@host ~]$ oc get VolumeSnapshotContent,VolumeSnapshot -A \
  -l velero.io/restore-name=mongodb
NAME
volumesnapshotcontent.snapshot.storage.k8s.io/velero-velero-mongodb-...

NAMESPACE   NAME
mongodb     volumesnapshot.snapshot.storage.k8s.io/velero-mongodb-...

You can then delete those snapshot resources with the oc delete command, as follows:

[user@host ~]$ oc delete -l velero.io/restore-name=mongodb \
  VolumeSnapshotContent,VolumeSnapshot -A
volumesnapshotcontent.snapshot.storage.k8s.io "velero-velero-mongodb-..." deleted
volumesnapshot.snapshot.storage.k8s.io "velero-mongodb-..." deleted