Abstract
| Goal |
Update an OpenShift cluster and minimize disruption to deployed applications. |
| Objectives |
|
| Sections |
|
Red Hat OpenShift Container Platform 4 adds many new features by using Red Hat Enterprise Linux CoreOS. Red Hat released a new software distribution system that provides the upgrade path to update your cluster and the underlying operating system. With this new distribution system, OpenShift clusters can perform Over-the-Air updates (OTA).
This software distribution system for OTA manages the controller manifests, cluster roles, and any other resources to update a cluster to a particular version. With this feature, a cluster can run the 4.14.x version seamlessly. With OTA, a cluster can use new features as they become available, including the latest bug fixes and security patches. OTA substantially decreases downtime due to upgrades.
Red Hat hosts and manages this service at https://console.redhat.com/openshift, and hosts cluster images at https://quay.io. You use a single interface to manage the lifecycle of all your OpenShift clusters. With OTA, you can update faster by skipping intermediate versions. For example, you can update from 4.14.1 to 4.14.3, and thus bypass 4.14.2.
Starting with OpenShift 4.10, the OTA system requires a persistent connection to the internet. For more information about how to update disconnected clusters, consult the Updating a Restricted Network Cluster chapter in the references section.
![]() |
The service defines upgrade paths that correspond to cluster eligibility for certain updates. Upgrade paths belong to update channels. Consider a channel as a representation of the upgrade path. The channel controls the frequency and stability of updates. The OTA policy engine represents channels as a series of pointers to particular versions within the upgrade path.
A channel name consists of the following parts: the tier (release candidate, fast, stable, and extended update support), the major version (4), and the minor version (.12).
Example channel names include: candidate-4.14, fast-4.14, stable-4.14, and eus-4.14.
Each channel delivers patches for a given cluster version.
The candidate channel delivers updates for testing feature acceptance in the next version of OpenShift Container Platform. The release candidate versions are subject to further checks, and are promoted to the fast or stable channels when they meet the quality standards.
Red Hat does not support the updates that are listed only in the candidate channel.
The fast channel delivers updates as soon as Red Hat declares the given version as a general availability release. Red Hat supports the updates that are released in this channel, and it is best suited to development and QA environments.
Customers can help to improve OpenShift by joining the Red Hat connected customers program. If you join this program, then your cluster is registered to the fast channel.
Red Hat support and site reliability engineering (SRE) teams monitor operational clusters with the updates from the fast channel. If operational clusters pass additional testing and validation, then updates in the fast channel are enabled in the stable channel. Red Hat supports the updates that are released in this channel, and it is best suited to production environments.
If Red Hat observes operational issues from a fast channel update, then that update is skipped in the stable channel. The stable channel delay provides time to observe any unforeseen problems in OpenShift clusters that testing did not reveal.
Starting with OpenShift Container Platform 4.8, Red Hat denotes all even-numbered minor releases (for example, 4.8, 4.10, 4.12, and 4.14) as Extended Update Support (EUS) releases.
EUS releases have no difference between stable-4. and xeus-4. channels (where x denotes the even-numbered minor release) until OpenShift Container Platform moves to the EUS phase.
You can switch to the EUS channel as soon as it becomes available.x
Red Hat offers support for all released updates in the fast, stable, and eus update channels. Red Hat supports the released updates in the candidate channel only if they are also listed in the fast or stable channels.
| Update channel | Support status |
|---|---|
candidate-4.
| Supported if the update is also listed in the fast or stable channels. |
fast-4.
| Supported |
stable-4.
| Supported |
eus-4.
| Supported |
The in the channel name denotes the minor version.x
You can apply each of the upgrade channels to a Red Hat OpenShift Container Platform version 4.14 cluster in different environments. The following paragraphs describe an example scenario where the 4.14.3 version has a defect.
When using the stable-4.14 channel, you can upgrade your cluster from 4.14.0 to 4.14.1 or to 4.14.2.
If an issue is discovered in the 4.14.3 release, then you cannot upgrade to that version.
When a patch becomes available in the 4.14.4 release, you can update your cluster to that version.
This channel is suited to production environments, because the Red Hat SRE teams and support services test the releases in that channel.
The fast-4.14 channel can deliver 4.14.1 and 4.14.2 updates but not 4.14.3.
Red Hat also supports this channel, and you can apply it to development, QA, or production environments.
Administrators must specifically choose a different minor version channel, such as fast-4.14, to upgrade to a new release in a new minor version when it becomes available.
You can use the candidate-4.14 channel to install the latest features of OpenShift.
With this channel, you can upgrade to all z-stream releases, such as 4.14.1, 4.14.2, and 4.14.3.
You use this channel to access the latest features of the product as they get released. This channel is suited to development and pre-production environments.
When switching to the eus-4.14 channel, the stable-4.14 channel does not receive z-stream updates until the next EUS version becomes available.
Starting with OpenShift Container Platform 4.8, Red Hat denotes all even-numbered minor releases as Extended Update Support (EUS) releases.
The following graphic describes the update graphs for the stable and candidate channels:
Red Hat provides support for the General Availability (GA) updates that are released in the stable and fast channels. Red Hat does not support updates that are listed only in the candidate channel.
To ensure the stability of the cluster and the proper level of support, switch only from a stable channel to a fast channel. Although it is possible to switch from a stable channel or a fast channel to a candidate channel, it is not recommended. The candidate channel is best suited to testing feature acceptance and to assist in qualifying the next version of OpenShift Container Platform.
The release of updates for patch and security fixes ranges from several hours to a day. This delay provides time to assess any operational impacts to OpenShift clusters.
You can change the update channel to eus-4.14, stable-4.14, fast-4.14, or candidate-4.14 by using the web console or the OpenShift CLI client:
Navigate to the → page on the details tab, and then click the pencil icon.
![]() |
A window displays options to select an update channel.
![]() |
Execute the following command to switch to another update channel by using the oc client.
You can also switch to another update channel, such as stable-4.14, to update to the next minor version of OpenShift Container Platform.
[user@host ~]$ oc patch clusterversion version --type="merge" \
--patch '{"spec":{"channel":"fast-4.14"}}'
clusterversion.config.openshift.io/version patchedDuring the upgrade process, nodes in the cluster might become temporarily unavailable. In the case of worker nodes, the machine health check might identify such nodes as unhealthy and reboot them. To avoid rebooting such nodes, pause all the machine health check resources before updating the cluster.
The prerequisite to pause the machine health check resources is not required on single-node installations.
Run the following command to list all the available machine health check resources.
[user@host ~]$oc get machinehealthcheck -n openshift-machine-apiNAME MAXUNHEALTHY EXPECTEDMACHINES CURRENTHEALTHYmachine-api-termination-handler100%
Add the cluster.x-k8s.io/paused annotation to the machine health check resource to pause it before updating the cluster.
[user@host ~]$ oc annotate machinehealthcheck -n openshift-machine-api \
machine-api-termination-handler cluster.x-k8s.io/paused=""
machinehealthcheck.machine.openshift.io/machine-api-termination-handler annotatedRemove the annotation after the cluster is updated.
[user@host ~]$ oc annotate machinehealthcheck -n openshift-machine-api \
machine-api-termination-handler cluster.x-k8s.io/paused-
machinehealthcheck.machine.openshift.io/machine-api-termination-handler annotatedOTA follows a client-server approach. Red Hat hosts the cluster images and the update infrastructure. OTA generates all possible update paths for your cluster. OTA also gathers information about the cluster and your entitlement to determine the available upgrade paths. The web console sends a notification when a new update is available.
The following diagram describes the updates architecture: Red Hat hosts both the cluster images and a "watcher", which automatically detects new images that are pushed to Quay. The Cluster Version Operator (CVO) receives its update status from that watcher. The CVO starts by updating the cluster components via their operators, and then updates any extra components that the Operator Lifecycle Manager (OLM) manages.
With telemetry, Red Hat can determine the update path. The cluster uses a Prometheus-based Telemeter component to report on the state of each cluster operator. The data is anonymized and sent back to Red Hat servers that advise cluster administrators about potential new releases.
Red Hat values customer privacy. For a complete list of the data that Telemeter gathers, consult the Data Collection and Telemeter Sample Metrics documents in the references section.
The following components are involved in the cluster update process:
The Machine Config Operator applies the desired machine state to each of the nodes. This component also handles the rolling upgrade of nodes in the cluster, and uses CoreOS Ignition as the configuration format.
The OLM orchestrates updates to any operators that are running in the cluster.
You can update the cluster via the web console or from the command line. The → page displays an update status of Available updates when a new update is available. From this page, click , and then select the version and the cluster update option that you want to install:
![]() |
Rolling back your cluster to an earlier version is not supported. If your update is failing to complete, contact Red Hat support.
The update process also updates the underlying operating system when updates are available.
The updates use the rpm-ostree technology for managing transactional upgrades.
Updates are delivered via container images and are part of the OpenShift update process.
When the update deploys, the nodes pull the new image, extract it, write the packages to the disk, and then modify the bootloader to boot into the new version.
The machine reboots and implements a rolling update to ensure that the cluster capacity is minimally impacted.
The following steps describe the procedure for updating a cluster as a cluster administrator by using the command-line interface:
Be sure to update all operators that are installed through the OLM to the 4.14 version before updating the OpenShift cluster.
Retrieve the cluster version and review the current update channel information.
If you are running the cluster in production, then ensure that the channel reads stable.
[user@host ~]$oc get clusterversionNAME VERSION AVAILABLE PROGRESSING SINCE STATUS version4.14.0True False 2d Cluster version is4.14.0[user@host ~]$oc get clusterversion -o jsonpath='{.items[0].spec.channel}{"\n"}'stable-4.14
View the available updates and note the version number of the update to apply.
[user@host ~]$oc adm upgradeCluster version is4.14.0Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.14 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, stable-4.14) Recommended updates: VERSION IMAGE4.14.10quay.io/openshift-release-dev/ocp-release@sha256:... ...output omitted...
Apply the latest update to your cluster, or update to a specific version:
Run the following command to install the latest available update for your cluster.
[user@host ~]$ oc adm upgrade --to-latest=trueRun the following command to install a specific version.
VERSION corresponds to one of the available versions that the oc adm upgrade command returns.
[user@host ~]$ oc adm upgrade --to=VERSIONThe previous command initializes the update process. Run the following command to review the status of the Cluster Version Operator (CVO) and the installed cluster operators.
[user@host ~]$oc get clusterversionNAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.0 TrueTrue1m Working towards4.14.10... [user@host ~]$oc get clusteroperatorsNAME VERSION AVAILABLE PROGRESSING DEGRADED ... authentication 4.14.0 True False False ... baremetal4.14.10FalseTrue False ... cloud-controller-manager4.14.10True FalseTrue... ...output omitted...
Use the following command to review the cluster version history and monitor the status of the update. It might take some time for all the objects to finish updating.
The history contains a list of the most recent versions that were applied to the cluster. This list is updated when the CVO applies an update. The list is ordered by date, where the newest update is first in the list.
If the rollout completed successfully, then updates in the history have a Completed state.
Otherwise, the update has a Partial state if it failed or did not complete.
[user@host ~]$oc describe clusterversion...output omitted... History: Completion Time: 2024-02-10T04:38:12Z Image: quay.io/openshift-release-dev/ocp-release@sha256:... Started Time: 2024-02-10T03:35:05Z State:PartialVerified: true Version:4.14.10Completion Time: 2024-02-10T12:39:02Z Image: quay.io/openshift-release-dev/ocp-release@sha256:... Started Time: 2024-02-10T12:23:14Z State:CompletedVerified: false Version:4.14.10
When an update is failing to complete, the Cluster Version Operator (CVO) reports the status of any blocking components and attempts to reconcile the update.
Rolling back your cluster to a previous version is not supported. If your update is failing to complete, contact Red Hat support.
After the process completes, you can confirm that the cluster is updated to the new version.
[user@host ~]$oc get clusterversionNAME VERSION AVAILABLE PROGRESSING SINCE STATUS version4.14.10True False 30m Cluster version is4.14.10
For more information about update channels, update prerequisites, and updating clusters in disconnected environments, refer to the Updating a Restricted Network Cluster and Updating a Cluster Between Minor Versions chapters in the Red Hat OpenShift Container Platform 4.14 Updating Clusters documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/updating_clusters/index#updating-restricted-network-cluster
For more information about updating operators that are installed through the Operator Lifecycle Manager, refer to the Upgrading Installed Operators section in the Administrator Tasks chapter in the Red Hat OpenShift Container Platform 4.14 Working with Operators documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/operators/index#olm-upgrading-operators
For more information about performing an EUS-to-EUS update, refer to the Preparing to Perform an EUS-to-EUS Update chapter in the Red Hat OpenShift Container Platform 4.14 Updating Clusters documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/updating_clusters/index#updating-eus-to-eus-upgrade_eus-to-eus-upgrade
For more information about the OpenShift Container Platform upgrade paths, visit the following page in the customer portal: https://access.redhat.com/solutions/4583231
For more information about the OpenShift Container Platform update graph, visit the following page in the customer portal: https://access.redhat.com/labs/ocpupgradegraph/update_path
For more information about OpenShift Extended Update Support (EUS), visit the following page in the customer portal: https://access.redhat.com/support/policy/updates/openshift-eus
For more information about the OpenShift Container Platform lifecycle policy, visit the following page in the customer portal: https://access.redhat.com/support/policy/updates/openshift