Red Hat OpenShift Administration II: Configuring a Production Cluster
Cluster administrators can use scheduled tasks to automate maintenance tasks in the cluster. Other users can create scheduled tasks for regular application maintenance.
Maintenance tasks vary in the privileges that they require. Cluster maintenance tasks require privileged pods, whereas most applications might not require elevated privileges.
You can automate tasks in OpenShift by using standard Kubernetes jobs and cron jobs. The automated tasks can be configured to run once or on a regular schedule.
- Job
Kubernetes jobs specify a task that is executed once.
- Cron Job
Kubernetes cron jobs have a schedule to execute a task regularly.
When a cron job is due for execution, Kubernetes creates a job resource. Kubernetes creates these jobs from a template in the cron job definition. Other than this relationship, Kubernetes jobs and cron jobs are workload resource types, such as deployments or daemon sets.
The job resource includes a pod template that describes the task to execute.
You can use the oc create job --dry-run=client command to get the YAML representation of the Kubernetes job resource:
[user@host ~]$ oc create job --dry-run=client -o yaml test \
--image=registry.access.redhat.com/ubi8/ubi:8.6 \
-- curl https://example.comA job contains a pod template, and this pod template must specify at least one container. You can add metadata such as labels or annotations to the job definition and pod template.
apiVersion: batch/v1 kind: Job metadata: creationTimestamp: null name: test spec:template:
metadata: creationTimestamp: null spec:
containers:
- command:
- curl - https://example.com image: registry.access.redhat.com/ubi8/ubi:8.6
name: test resources: {} restartPolicy: Never status: {}
| Job specification |
| Pod template |
| Pod specification |
| Pod containers |
| Command |
| Container image |
The cron job resource includes a job template that describes the task and a schedule.
You can use the oc create cronjob --dry-run=client command to get the YAML representation of the Kubernetes cron job resource:
[user@host ~]$ oc create cronjob --dry-run=client -o yaml test \
--image=registry.access.redhat.com/ubi8/ubi:8.6 \
--schedule='0 0 * * *' \
-- curl https://example.comIn Kubernetes, cron job resources are similar to job resources.
The jobTemplate key follows the same structure as a job.
The schedule key describes when the task runs.
apiVersion: batch/v1 kind: CronJob metadata: creationTimestamp: null name: test spec:name: test resources: {} restartPolicy: OnFailure schedule: 0 0 * * *
status: {}
| Cron job specification |
| Job template |
| Job specification |
| Pod template |
| Pod specification |
| Command |
| Container image |
| Cron job schedule specification |
The schedule specification for Kubernetes cron jobs is derived from the specification in Linux cron jobs.
The crontab file specifies the scheduled tasks for the current user.
The schedule specification has five fields to define the date and time when the job is executed.
The /etc/crontab file comments include a syntax diagram:
# Example cron job definition: # ┌─────────────────minute(0 - 59) # │ ┌──────────────hour(0 - 23) # │ │ ┌──────────day of month(1 - 31) # │ │ │ ┌──────month(1 - 12) or jan,feb,mar,apr ... # │ │ │ │ ┌──day of week(0 - 7) or sun,mon,tue,wed,thu,fri,sat # │ │ │ │ │ (Sunday is 0 or 7) # m h dom mon dow command0 */2 * * */path/to/task_executable arguments
Some examples of cron job specifications are as follows:
| Schedule specification | Description |
|---|---|
| 0 0 * * * | Run the specified task every day at midnight |
| 0 0 * * 7 | Run the specified task every Sunday at midnight |
| 0 * * * * | Run the specified task every hour |
| 0 */4 * * * | Run the specified task every four hours |
Note
Refer to the crontab(5) manual page for more information about the cron job schedule specification.
You can automate the maintenance tasks for applications that run inside the cluster, and also execute low-level commands inside privileged debug pods to apply cluster maintenance tasks.
Regular maintenance tasks might need to run for applications that run in the cluster.
For example, consider creating periodic backups for an application. This application requires the following steps to create the backup:
Activate maintenance mode.
Create a compressed database backup.
Deactivate maintenance mode.
Copy the database backup to an external location.
The following cron job definition shows a possible implementation of these steps:
apiVersion: batch/v1 kind: CronJob metadata: name: wordpress-backup spec:schedule: 0 2 * * 7registry.io/wp-maintenance/wp-cli:2.7bash--xcargs:> wp maintenance-mode activate ; wp db export | gzip > database.sql.gz ; wp maintenance-mode deactivate ; rclone copy database.sql.gzs3://bucket/backups/; rm -v database.sql.gz ;
| Schedule for every Sunday at 2 AM |
| The Kubernetes job template |
| The Kubernetes pod template |
| The Kubernetes pod specification |
| The pod container configuration |
| The container image that runs the maintenance task |
| The command to execute inside the pod |
| Maintenance commands to execute |
Note
The > symbol uses the YAML folded style, which converts all newlines to spaces when parsing.
Each command is separated with a semicolon (;), because the string in the args key is passed as a single argument to the bash -xc command.
This combination of the command and args keys has the same effect as executing the commands in a single line inside the container:
[user@host ~]$ bash -xc 'wp maintenance-mode activate ; wp db export | gzip > database.sql.gz ; wp maintenance-mode deactivate ; rclone copy database.sql.gz s3://bucket/backups/ ; rm -v database.sql.gz ;'For more information about the YAML folded style, refer to https://yaml.org/spec/1.2.2/#folded-style
Cluster maintenance might require executing complex scripts in privileged pods. You can create a shell script with the commands to execute the maintenance task, and mount the script in the pod by using a configuration map.
For example, when images are updated, clusters might accumulate unused images.
These images might occupy much space.
Executing the crictl rmi --prune command on all nodes of the cluster frees this space.
The following configuration map contains a shell script that cleans images in all cluster nodes by executing a debug pod and running the crictl command with the chroot command to access the root file system of the node:
apiVersion: v1
kind: ConfigMap
metadata:
name: maintenance
app: crictl
data:
maintenance.sh: |
#!/bin/bash
NODES=$(oc get nodes -o=name)
for NODE in ${NODES}
do
echo ${NODE}
oc debug ${NODE} -- \
chroot /host \
/bin/bash -xc 'crictl images ; crictl rmi --prune'
echo $?
done
| List the nodes in the cluster. |
| Iterate over the nodes. |
| Run a debug pod on the node. |
| Prune the images. |
This task can be scheduled regularly by using a cron job.
The quay.io/openshift/origin-cli:4.14 container provides the oc command that runs the debug pod.
The pod mounts the configuration map and executes the maintenance script.
apiVersion: batch/v1
kind: CronJob
metadata:
name: image-pruner
spec:
schedule: 0 * * * *
jobTemplate:
spec:
template:
spec:
dnsPolicy: ClusterFirst
restartPolicy: Never
containers:
- name: image-pruner
image: quay.io/openshift/origin-cli:4.14
resources: {}
command:
- /opt/scripts/maintenance.sh
volumeMounts:
- name: scripts
mountPath: /opt
volumes:
- name: scripts
configMap:
name: maintenance
defaultMode: 0555
| Path to the script |
| Mounting the configuration map as a volume |
Cluster maintenance tasks might require elevated privileges. Administrators can assign service accounts to any workload, including Kubernetes jobs and cron jobs.
You can create a service account with the required privileges, and specify the service account with the serviceAccountName key in the pod definition.
You can also use the oc set serviceaccount command to change the service account of an existing workload.
