DO380 - ch02s06

Bookmark this page

Guided Exercise: Backup and Restore with OADP

Perform a crash-consistent and an application-consistent backup of a database, and restore each backup to a different project.

Outcomes

Back up and restore an OpenShift project.
Schedule a backup of an OpenShift project.
Remove previous backups.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available.

[student@workstation ~]$ lab start backup-restore

Instructions

Create a crash-consistent backup of the database project by using OADP and CSI snapshots, and restore it to a new database-crash project.

Schedule a weekly backup of the database project. Use hooks to quiesce the database and produce an application-consistent backup.

Trigger a backup from the schedule and restore the application-consistent backup to a new database-backup project.

Remove the backups and all related resources.

As the admin user, review the content of the database project.

[student@workstation ~]$ oc login -u admin -p redhatocp \
  https://api.ocp4.example.com:6443
Login successful.

...output omitted...

Change to the database project.

[student@workstation ~]$ oc project database
Now using project "database" on server "https://api.ocp4.example.com:6443".

List the resources with the app=mariadb label in the database project.

[student@workstation ~]$ oc get \
  pvc,svc,deployment,secret,configmap -l app=mariadb
NAME                            ...   STORAGECLASS
persistentvolumeclaim/mariadb   ...   ocs-external-storagecluster-ceph-rbd

NAME              TYPE           ...   PORT(S)
service/mariadb   LoadBalancer   ...   3306:31402/TCP

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mariadb   1/1     1            1           5m51s

NAME             TYPE     DATA   AGE
secret/mariadb   Opaque   4      5m54s

NAME                 DATA   AGE
configmap/database   1      5m53s

Review the UID and GID assignment for the database namespace. The values might be different in your environment.

[student@workstation ~]$ oc get namespace database -oyaml
...output omitted...
metadata:
  annotations:
...output omitted...
    openshift.io/sa.scc.supplemental-groups: 1000810000/10000
    openshift.io/sa.scc.uid-range: 1000810000/10000
  labels:
    kubernetes.io/metadata.name: database
...output omitted...

Connect to the MariaDB database in the mariadb deployment, and list the rows in the application_logs table from the application database.

[student@workstation ~]$ oc exec -c mariadb deploy/mariadb -- \
  bash -c \
  'mariadb -u ${MARIADB_USER} \
    -p"${MARIADB_PASSWORD}" \
    ${MARIADB_DATABASE} -te \
    "select * from application_logs order by id desc LIMIT 5;"'

+----+---------------------+-----------+-------------+
| id | time                | app       | message     |
+----+---------------------+-----------+-------------+
| 25 | 2023-12-05 10:48:12 | db-loader | ad42...785d |
| 24 | 2023-12-05 10:48:02 | db-loader | 26b3...be3e |
| 23 | 2023-12-05 10:47:52 | db-loader | ed74...6ed8 |
| 22 | 2023-12-05 10:47:42 | db-loader | f9f5...3270 |
| 21 | 2023-12-05 10:47:32 | db-loader | 2c65...b85c |
+----+---------------------+-----------+-------------+

Note

The db-loader application is deployed in the application project. This application creates a record in the application_logs table every 10 seconds to simulate database activity during the backup operations.

Create a crash-consistent backup of the database project.
1. Change to the ~/DO380/labs/backup-restore directory.
```
[student@workstation ~]$ cd ~/DO380/labs/backup-restore
[student@workstation backup-restore]$
```
2. Create a db-manual backup resource in the openshift-adp namespace to back up the database project. Use resource filtering to back up only the listed MariaDB resources in the previous step.
  Modify the partial resource definition in the ~/DO380/labs/backup-restore/backup-db-manual.yml file as follows:
```
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: db-manual
  namespace: openshift-adp
spec:
  includedNamespaces:
  - database
  orLabelSelectors:
  - matchLabels:
      app: mariadb
  - matchLabels:
      kubernetes.io/metadata.name: database
  includedResources:
  - namespace
  - deployments
  - configmaps
  - secrets
  - pvc
  - pv
  - services
```
  Important
  So that OADP can restore the UID and GID assigned to the database namespace, include the namespace resource type and the kubernetes.io/metadata.name: database label in the backup definition.
3. Apply the configuration for the backup resource.
```
[student@workstation backup-restore]$ oc apply -f backup-db-manual.yml
backup.velero.io/db-manual created
```

Review the status of the backup.

Create an alias to access the velero binary from the Velero deployment in the openshift-adp namespace.

[student@workstation backup-restore]$ alias velero='\
  oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'

Use the velero command to get the status of the db-manual backup. Monitor the output to verify that the backup status is Completed. The backup process takes several minutes. Note the backup creation time for the next step.

[student@workstation backup-restore]$ velero get backup db-manual
NAME       STATUS      ERRORS  WARNINGS  CREATED               ...
db-manual  Completed   0       0         2023-12-04 12:16:40   ...

Review the content of the application_logs table from the application database in the mariadb deployment. List the rows that were created before the backup time. You use the result to compare with the database that you restore in a later step.

[student@workstation backup-restore]$ oc exec -c mariadb deploy/mariadb -- \
  bash -c 'mariadb -u ${MARIADB_USER} -p"${MARIADB_PASSWORD}" \
    ${MARIADB_DATABASE} -te \
    "select * from application_logs where time <= \"2023-12-04 12:16:40\" \
    order by id desc LIMIT 5;"'

+----+---------------------+-----------+-------------+
| id | time                | app       | message     |
+----+---------------------+-----------+-------------+
| 36 | 2023-12-05 10:50:03 | db-loader | 2232...4341 |
| 35 | 2023-12-05 10:49:53 | db-loader | b8a2...6ddf |
| 34 | 2023-12-05 10:49:43 | db-loader | 9cb1...a6d1 |
| 33 | 2023-12-05 10:49:33 | db-loader | c0e5...5e8b |
| 32 | 2023-12-05 10:49:23 | db-loader | e3c9...1929 |
+----+---------------------+-----------+-------------+

Restore the db-manual backup to a new database-crash project.
1. Create a db-crash restore resource in the openshift-adp namespace to restore the db-manual backup. Restore the backup to the database-crash project.
  Modify the partial resource definition in the ~/DO380/labs/backup-restore/restore-db-crash.yml file as follows:
```
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: db-crash
  namespace: openshift-adp
spec:
  backupName: db-manual
  restorePVs: true
  namespaceMapping:
    database: database-crash
```
2. Apply the configuration for the restore resource.
```
[student@workstation backup-restore]$ oc apply -f restore-db-crash.yml
restore.velero.io/db-crash created
```
3. Use the velero command to get the status of the db-crash restore resource. Monitor the output to verify that the backup status is Completed. The restoration process takes several minutes.
```
[student@workstation backup-restore]$ velero get restore db-crash
NAME      BACKUP      STATUS      ...   COMPLETED   ERRORS   WARNINGS
db-crash  db-manual   Completed   ...   <nil>       0        0
```

Review the content of the database-crash project and verify the database integrity.

Change to the database-crash project.

[student@workstation backup-restore]$ oc project database-crash
Now using project "database-crash" on server "https://api.ocp4.example.com:6443".

List the resources in the database-crash project and compare them with the resources in the database project from the previous step.

[student@workstation backup-restore]$ oc get \
  pvc,svc,deployment,secret,configmap -l app=mariadb
NAME                            ...   STORAGECLASS
persistentvolumeclaim/mariadb   ...   ocs-external-storagecluster-ceph-rbd

NAME              TYPE           ...   PORT(S)
service/mariadb   LoadBalancer   ...   3306:31402/TCP

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mariadb   1/1     1            1           5m51s

NAME             TYPE     DATA   AGE
secret/mariadb   Opaque   4      5m54s

NAME                 DATA   AGE
configmap/database   1      5m53s

Review the UID and GID assignment for the database-crash namespace. Verify that the numbers are the same as for the database namespace from the previous step.

[student@workstation ~]$ oc get namespace database-crash -oyaml
...output omitted...
metadata:
  annotations:
...output omitted...
    openshift.io/sa.scc.supplemental-groups: 1000810000/10000
    openshift.io/sa.scc.uid-range: 1000810000/10000
  labels:
    kubernetes.io/metadata.name: database-crash
...output omitted...

Use the mariadb-check command to check the integrity and to repair the database tables.

[student@workstation backup-restore]$ oc exec -c mariadb deploy/mariadb -- \
  bash -c \
  'mariadb-check -u root -p"${MARIADB_ROOT_PASSWORD}" -A --auto-repair'

application.application_logs
warning  : 1 client is using or hasn't closed the table properly
warning  : Size of datafile is: 13296       Should be: 13120
error    : Wrong bytesec: 0-0-0 at linkstart: 13120
error    : Corrupt
mysql.column_stats                                 OK
...output omitted...

Repairing tables
application.application_logs
info     : Wrong bytesec:   0-  0-  0 at 13120; Skipped
status   : OK

Note

In this example, the application_logs table is corrupted and some data might be lost after the repair. The result might be different in your lab environment.

List the records in the application_logs table from the application database in the database-crash project. Compare the result with the content of the database in the database project from the previous step.

[student@workstation backup-restore]$ oc exec -c mariadb deploy/mariadb -- \
  bash -c \
  'mariadb -u ${MARIADB_USER} \
    -p"${MARIADB_PASSWORD}" \
    ${MARIADB_DATABASE} -te \
    "select * from application_logs order by id desc LIMIT 5;"'

+----+---------------------+-----------+-------------+
| id | time                | app       | message     |
+----+---------------------+-----------+-------------+
| 35 | 2023-12-05 10:49:53 | db-loader | b8a2...6ddf |
| 34 | 2023-12-05 10:49:43 | db-loader | 9cb1...a6d1 |
| 33 | 2023-12-05 10:49:33 | db-loader | c0e5...5e8b |
| 32 | 2023-12-05 10:49:23 | db-loader | e3c9...1929 |
| 31 | 2023-12-05 10:49:13 | db-loader | dc45...b7d8 |
+----+---------------------+-----------+-------------+

Note

Compare the latest row ID with the ID from the previous step. Rows can be missing. This outcome can result from the database corruption that was detected in the previous step, or because the records were not written to the disk before the backup started.

Delete the crash-consistent backup and all related resources.

Review the content of the S3 bucket by using the s3cmd command.

[student@workstation backup-restore]$ s3cmd la -r
...output omitted...
... s3://backup-.../oadp/backups/db-manual/velero-backup.json 
... s3://backup-.../oadp/backups/db-manual/....gz
...output omitted...
... s3://backup-.../oadp/restores/db-crash/restore-db-crash-logs.gz 
... s3://backup-.../oadp/restores/db-crash/....gz
...output omitted...
... s3://backup-.../openshift-adp/db-manual/snapcontent-...-pvc/config 
... s3://backup-.../openshift-adp/db-manual/snapcontent-...-pvc/data/55/551...c2b
...output omitted...

	The `/oadp/backups/<backup_name>` path contains the backup of the Kubernetes resources and the logs of the backup.
	The `/oadp/restores/<restore_name>` path contains the restore logs.
	The `/openshift-adp/<backup_name>` path contains the encrypted volume snapshots that are exported with Data Mover.

Note

The database uses a container image that is stored on an external registry and is therefore not included in the backup.

Delete the db-manual backup by using the velero command.
```
[student@workstation backup-restore]$ velero delete backup db-manual
Are you sure you want to continue (Y/N)? y
Request to delete backup "db-manual" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.
```
Note
Velero deletes all backup and restore resources that are associated with the specified backup in the OpenShift cluster. Velero also deletes the backup and restore files from the object storage.
Resources that the OADP Data Mover creates, such as snapshot backups on object storage and snapshot resources during restore, are not deleted.

Monitor the status of the db-manual backup with the velero command and wait until OADP deletes the backup.

[student@workstation backup-restore]$ velero get backup db-manual
NAME        STATUS     ERRORS   WARNINGS   ...
db-manual   Deleting   0        0          ...

Note

If the command returns the following error, then the backup is removed from both OpenShift and the object storage:

An error occurred: backups.velero.io "db-manual" not found
command terminated with exit code 1

Verify that the backup and restore resources are removed from the openshift-adp namespace. It can take a few minutes for the OADP operator to remove the resources.
```
[student@workstation backup-restore]$ oc -n openshift-adp get backup,restore
No resources found in openshift-adp namespace.
```

Review the content of the S3 bucket. The db-manual backup and restore are automatically deleted and only the snapshot backup remains.

[student@workstation backup-restore]$ s3cmd la -r
... s3://backup-.../openshift-adp/db-manual/snapcontent-...-pvc/config
... s3://backup-.../openshift-adp/db-manual/snapcontent-...-pvc/data/55/551...c2b
...output omitted...

Identify the snapshot resources that are associated with the db-crash restore resource.

[student@workstation backup-restore]$ oc get \
  -l velero.io/restore-name=db-crash \
  VolumeSnapshotContent,VolumeSnapshot -A
NAME                                          ...   VOLUMESNAPSHOTNAMESPACE
volumesnapshotcontent...-mariadb-zq96c-h2lv9  ...   database-crash

NAMESPACE        NAME                               ...
database-crash   volumesnapshot....-mariadb-zq96c   ...

Delete the snapshot resources from the previous step.

[student@workstation backup-restore]$ oc delete \
  -l velero.io/restore-name=db-crash \
  VolumeSnapshotContent,VolumeSnapshot -A
volumesnapshotcontent.snapshot.storage.k8s.io "velero-...-h2lv9" deleted
volumesnapshot.snapshot.storage.k8s.io "velero-mariadb-zq96c" deleted

Delete the db-manual snapshot from the object storage.

[student@workstation backup-restore]$ s3cmd rm -r \
  s3://backup-5f26...350d/openshift-adp/db-manual/
delete: 's3://backup-.../openshift-adp/db-manual/snapcontent-.../config'
delete: 's3://backup-.../openshift-adp/db-manual/snapcontent-.../data/...'
...output omitted...

Note

You can use the s3cmd ls command to retrieve the name of the S3 bucket.

Schedule a recurring application-consistent backup of the database project.

Use backup hooks to ensure that all database operations are flushed to the file system and that the database is locked in read-only mode during the backup.

Backups must be automatically deleted after 15 days.

Create a db-backup schedule resource in the openshift-adp namespace to back up the database project every week.

Set the schedule to 7 AM every Sunday. For this exercise, disable the schedule with the paused field to prevent unexpected backups from starting during the hands-on activity. You manually trigger a backup from this schedule in a later step.

Modify the partial resource definition in the ~/DO380/labs/backup-restore/schedule-db-backup.yml file as follows:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: db-backup
  namespace: openshift-adp
spec:
  schedule: "0 7 * * 0"
  paused: true
  template:
    ttl: 360h0m0s
    includedNamespaces:
    - database
    orLabelSelectors:
    - matchLabels:
        app: mariadb
    - matchLabels:
        kubernetes.io/metadata.name: database
    includedResources:
    - namespace
    - deployments
    - configmaps
    - secrets
    - pvc
    - pv
    - services
    - pods
    hooks:
      resources:
      - name: mariadb-readonly
        pre:
        - exec:
            container: mariadb
            command:
            - /bin/bash
            - -c
            - |
              mariadb -u "root" \
                -p"${MARIADB_ROOT_PASSWORD}" \
                ${MARIADB_DATABASE} -e \
                "set global read_only=1; \ 
                BACKUP STAGE START; \
                BACKUP STAGE BLOCK_COMMIT;"; 
              sync; 
        post:
        - exec:
            container: mariadb
            command:
            - /bin/bash
            - -c
            - |
              mariadb -u "root" \
                -p"${MARIADB_ROOT_PASSWORD}" \
                ${MARIADB_DATABASE} -e \
                "set global read_only=0;";

	The `set global read_only=1` command locks the database in read-only mode to prevent any writes during the backup.
	The `BACKUP STAGE` commands ensure that all database tables are flushed to the disk.
	The `sync` command ensures that all cached writes are flushed to the persistent storage.
	The `set global read_only=0` command removes the database lock after the backup is completed.

Warning

Because backup hooks are executed only on pods that are included in the backup, you must include the pod resource in the includedResources field.

Apply the configuration for the schedule resource.

[student@workstation backup-restore]$ oc apply -f schedule-db-backup.yml
schedule.velero.io/db-backup created

Verify the status of the db-backup schedule.

[student@workstation backup-restore]$ velero get schedule
NAME       STATUS ... SCHEDULE   BACKUP TTL  LAST BACKUP  SELECTOR     PAUSED
db-backup  New    ... 0 7 * * 0  360h0m0s    n/a          app=mariadb  true

Use the velero command to trigger a new backup from the db-backup schedule. Save the backup name for a later step.

[student@workstation backup-restore]$ velero create backup \
  --from-schedule=db-backup
...output omitted...
Creating backup from schedule, all other filters are ignored.
Backup request "db-backup-20231205114600" submitted successfully.
...output omitted...

Use the velero command to monitor the status of the backup and wait until the backup is in the Completed state. The backup process takes several minutes. Note the creation time of the backup from the CREATED field.

[student@workstation backup-restore]$ watch -n 10 oc -n openshift-adp \
  exec deployment/velero -c velero -it -- ./velero get backup \
  -l velero.io/schedule-name=db-backup
NAME                   STATUS      ERRORS  WARNINGS  CREATED             ...
db-backup-2023...00    Completed    0      0         2023-12-05 11:46:00 ...

Note

Press Ctrl+C to exit the watch command.

Identify the rows immediately before the backup time in the application_logs table from the application database in the mariadb deployment. Use the date and time from the previous step.

[student@workstation backup-restore]$ oc exec -n database \
  deploy/mariadb -c mariadb -- \
  bash -c 'mariadb -u ${MARIADB_USER} -p"${MARIADB_PASSWORD}" \
  ${MARIADB_DATABASE} -te \
  "select * from application_logs where time <= \"2023-12-05 11:46:00\" \
  order by id desc LIMIT 5;"'

+-----+---------------------+-----------+--------------+
| id  | time                | app       | message      |
+-----+---------------------+-----------+--------------+
| 371 | 2023-12-05 11:45:58 | db-loader | 325ef...f337 |
| 370 | 2023-12-05 11:45:48 | db-loader | 2505f...3215 |
| 369 | 2023-12-05 11:45:38 | db-loader | 77ba0...561a |
| 368 | 2023-12-05 11:45:27 | db-loader | 6bffb...8f5f |
| 367 | 2023-12-05 11:45:17 | db-loader | a895d...3dfd |
+-----+---------------------+-----------+--------------+

Restore the application-consistent backup to the database-backup project.

Create a db-backup restore resource in the openshift-adp namespace. Restore the backup from the previous step to the database-backup project.
Modify the partial resource definition in the ~/DO380/labs/backup-restore/restore-db-backup.yml file as follows:
```
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: db-backup
  namespace: openshift-adp
spec:
  backupName: db-backup-20231205114600
  restorePVs: true
  namespaceMapping:
    database: database-backup
```

Apply the configuration for the restore resource.

[student@workstation backup-restore]$ oc apply -f restore-db-backup.yml

Use the velero command to get the status of the db-backup restore resource. Monitor the output to verify that the status is Completed. The restoration process takes several minutes.

[student@workstation backup-restore]$ velero get restore db-backup
NAME        BACKUP                     STATUS      ...   ERRORS   WARNINGS
db-backup   db-backup-20231205114600   Completed   ...   0        0

Delete the volume snapshot that is associated with the db-backup restore to free up resources on the storage back end.

[student@workstation backup-restore]$ oc delete \
  -l velero.io/restore-name=db-backup \
  VolumeSnapshotContent,VolumeSnapshot -A
volumesnapshotcontent.snapshot.storage.k8s.io ... deleted
volumesnapshot.snapshot.storage.k8s.io ... deleted

Review the content of the database-backup project.

Change to the database-backup project.

[student@workstation backup-restore]$ oc project database-backup
Now using project "database-backup" on server "https://api.ocp4.example.com:6443".

Use the mariadb-check command to verify the integrity of the database tables.

[student@workstation backup-restore]$ oc exec -c mariadb deploy/mariadb -- \
  bash -c 'mariadb-check -u root -p"${MARIADB_ROOT_PASSWORD}" -A'

application.application_logs                       OK
mysql.column_stats                                 OK
...output omitted...
mysql.transaction_registry                         OK

Note

All tables are OK and no corruption is detected.

List the rows in the application_logs table from the application database in the database-backup project. Compare the result with the content of the database in the database project from the previous step.

[student@workstation backup-restore]$ oc exec -n database-backup \
  deploy/mariadb -c mariadb -- \
  bash -c 'mariadb -u ${MARIADB_USER} -p"${MARIADB_PASSWORD}" \
  ${MARIADB_DATABASE} -te \
  "select * from application_logs order by id desc LIMIT 5;"'

+-----+---------------------+-----------+-------------+
| id  | time                | app       | message     |
+-----+---------------------+-----------+-------------+
| 371 | 2023-12-05 11:45:58 | db-loader | 325e...f337 |
| 370 | 2023-12-05 11:45:48 | db-loader | 2505...3215 |
| 369 | 2023-12-05 11:45:38 | db-loader | 77ba...561a |
| 368 | 2023-12-05 11:45:27 | db-loader | 6bff...8f5f |
| 367 | 2023-12-05 11:45:17 | db-loader | a895...3dfd |
+-----+---------------------+-----------+-------------+

Note

Compare the latest row ID with the ID from the previous step. All rows until the start of the backup are present.

Clean up the resources.

Change to the home directory.

[student@workstation backup-restore]$ cd
[student@workstation ~]$

Delete the database-crash and database-backup projects.

[student@workstation ~]$ oc delete project database-crash database-backup
project.project.openshift.io "database-crash" deleted
project.project.openshift.io "database-backup" deleted

Delete the backup that you created in the previous step by using the velero command. Use the velero.io/schedule-name=db-backup label to delete all backups that were created from the db-backup schedule.

[student@workstation backup-restore]$ velero delete backup \
  -l velero.io/schedule-name=db-backup
Are you sure you want to continue (Y/N)? y
Request to delete backup "db-backup-20231204141509" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish backup-restore

Discuss Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Go to community

Welcome to Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise!

Syed

12 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO378.Read more about Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise here.

Revision: do380-4.14-397a507

Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Guided Exercise: Backup and Restore with OADP

Note

Important

Note

Note

Note

Note

Note

Note

Warning

Note

Note

Note