Bookmark this page

Chapter 9.  OpenShift Updates

Abstract

Goal

Update an OpenShift cluster and minimize disruption to deployed applications.

Objectives
  • Describe the cluster update process.

  • Identify applications that use deprecated Kubernetes APIs.

  • Update OLM-managed operators by using the web console and CLI.

Sections
  • The Cluster Update Process (and Quiz)

  • Detect Deprecated Kubernetes API Usage (and Quiz)

  • Update Operators with the OLM (and Quiz)

  • OpenShift Updates (Quiz)

The Cluster Update Process

Objectives

  • Describe the cluster update process.

Introducing Cluster Updates

Red Hat OpenShift Container Platform 4 adds many new features by using Red Hat Enterprise Linux CoreOS. Red Hat released a new software distribution system that provides the upgrade path to update your cluster and the underlying operating system. With this new distribution system, OpenShift clusters can perform Over-the-Air updates (OTA).

This software distribution system for OTA manages the controller manifests, cluster roles, and any other resources to update a cluster to a particular version. With this feature, a cluster can run the 4.14.x version seamlessly. With OTA, a cluster can use new features as they become available, including the latest bug fixes and security patches. OTA substantially decreases downtime due to upgrades.

Red Hat hosts and manages this service at https://console.redhat.com/openshift, and hosts cluster images at https://quay.io. You use a single interface to manage the lifecycle of all your OpenShift clusters. With OTA, you can update faster by skipping intermediate versions. For example, you can update from 4.14.1 to 4.14.3, and thus bypass 4.14.2.

Important

Starting with OpenShift 4.10, the OTA system requires a persistent connection to the internet. For more information about how to update disconnected clusters, consult the Updating a Restricted Network Cluster chapter in the references section.

Figure 9.1: Managing clusters at cloud.redhat.com

The service defines upgrade paths that correspond to cluster eligibility for certain updates. Upgrade paths belong to update channels. Consider a channel as a representation of the upgrade path. The channel controls the frequency and stability of updates. The OTA policy engine represents channels as a series of pointers to particular versions within the upgrade path.

A channel name consists of the following parts: the tier (release candidate, fast, stable, and extended update support), the major version (4), and the minor version (.12). Example channel names include: candidate-4.14, fast-4.14, stable-4.14, and eus-4.14. Each channel delivers patches for a given cluster version.

The Candidate Channel

The candidate channel delivers updates for testing feature acceptance in the next version of OpenShift Container Platform. The release candidate versions are subject to further checks, and are promoted to the fast or stable channels when they meet the quality standards.

Important

Red Hat does not support the updates that are listed only in the candidate channel.

The Fast Channel

The fast channel delivers updates as soon as Red Hat declares the given version as a general availability release. Red Hat supports the updates that are released in this channel, and it is best suited to development and QA environments.

Note

Customers can help to improve OpenShift by joining the Red Hat connected customers program. If you join this program, then your cluster is registered to the fast channel.

The Stable Channel

Red Hat support and site reliability engineering (SRE) teams monitor operational clusters with the updates from the fast channel. If operational clusters pass additional testing and validation, then updates in the fast channel are enabled in the stable channel. Red Hat supports the updates that are released in this channel, and it is best suited to production environments.

If Red Hat observes operational issues from a fast channel update, then that update is skipped in the stable channel. The stable channel delay provides time to observe any unforeseen problems in OpenShift clusters that testing did not reveal.

The Extended Update Support Channel

Starting with OpenShift Container Platform 4.8, Red Hat denotes all even-numbered minor releases (for example, 4.8, 4.10, 4.12, and 4.14) as Extended Update Support (EUS) releases.

EUS releases have no difference between stable-4.x and eus-4.x channels (where x denotes the even-numbered minor release) until OpenShift Container Platform moves to the EUS phase. You can switch to the EUS channel as soon as it becomes available.

Support Status for Update Channels

Red Hat offers support for all released updates in the fast, stable, and eus update channels. Red Hat supports the released updates in the candidate channel only if they are also listed in the fast or stable channels.

Update channel Support status
candidate-4.x Supported if the update is also listed in the fast or stable channels.
fast-4.x Supported
stable-4.x Supported
eus-4.x Supported

Note

The x in the channel name denotes the minor version.

Upgrade Paths

You can apply each of the upgrade channels to a Red Hat OpenShift Container Platform version 4.14 cluster in different environments. The following paragraphs describe an example scenario where the 4.14.3 version has a defect.

Stable channel

When using the stable-4.14 channel, you can upgrade your cluster from 4.14.0 to 4.14.1 or to 4.14.2. If an issue is discovered in the 4.14.3 release, then you cannot upgrade to that version. When a patch becomes available in the 4.14.4 release, you can update your cluster to that version.

This channel is suited to production environments, because the Red Hat SRE teams and support services test the releases in that channel.

Fast channel

The fast-4.14 channel can deliver 4.14.1 and 4.14.2 updates but not 4.14.3. Red Hat also supports this channel, and you can apply it to development, QA, or production environments.

Administrators must specifically choose a different minor version channel, such as fast-4.14, to upgrade to a new release in a new minor version when it becomes available.

Candidate channel

You can use the candidate-4.14 channel to install the latest features of OpenShift. With this channel, you can upgrade to all z-stream releases, such as 4.14.1, 4.14.2, and 4.14.3.

You use this channel to access the latest features of the product as they get released. This channel is suited to development and pre-production environments.

EUS channel

When switching to the eus-4.14 channel, the stable-4.14 channel does not receive z-stream updates until the next EUS version becomes available.

Note

Starting with OpenShift Container Platform 4.8, Red Hat denotes all even-numbered minor releases as Extended Update Support (EUS) releases.

The following graphic describes the update graphs for the stable and candidate channels:

Figure 9.2: Update graphs for stable and candidate channels

Red Hat provides support for the General Availability (GA) updates that are released in the stable and fast channels. Red Hat does not support updates that are listed only in the candidate channel.

To ensure the stability of the cluster and the proper level of support, switch only from a stable channel to a fast channel. Although it is possible to switch from a stable channel or a fast channel to a candidate channel, it is not recommended. The candidate channel is best suited to testing feature acceptance and to assist in qualifying the next version of OpenShift Container Platform.

Note

The release of updates for patch and security fixes ranges from several hours to a day. This delay provides time to assess any operational impacts to OpenShift clusters.

Changing the Update Channel

You can change the update channel to eus-4.14, stable-4.14, fast-4.14, or candidate-4.14 by using the web console or the OpenShift CLI client:

Web console

Navigate to the AdministrationCluster Settings page on the details tab, and then click the pencil icon.

Figure 9.3: Current update channel in the web console

A window displays options to select an update channel.

Figure 9.4: Changing the update channel in the web console
Command line

Execute the following command to switch to another update channel by using the oc client. You can also switch to another update channel, such as stable-4.14, to update to the next minor version of OpenShift Container Platform.

[user@host ~]$ oc patch clusterversion version --type="merge" \
  --patch '{"spec":{"channel":"fast-4.14"}}'
clusterversion.config.openshift.io/version patched

Pausing the Machine Health Check Resource

During the upgrade process, nodes in the cluster might become temporarily unavailable. In the case of worker nodes, the machine health check might identify such nodes as unhealthy and reboot them. To avoid rebooting such nodes, pause all the machine health check resources before updating the cluster.

Note

The prerequisite to pause the machine health check resources is not required on single-node installations.

Run the following command to list all the available machine health check resources.

[user@host ~]$ oc get machinehealthcheck -n openshift-machine-api
NAME                             MAXUNHEALTHY  EXPECTEDMACHINES  CURRENTHEALTHY
machine-api-termination-handler  100%

Add the cluster.x-k8s.io/paused annotation to the machine health check resource to pause it before updating the cluster.

[user@host ~]$ oc annotate machinehealthcheck -n openshift-machine-api \
  machine-api-termination-handler cluster.x-k8s.io/paused=""
machinehealthcheck.machine.openshift.io/machine-api-termination-handler annotated

Remove the annotation after the cluster is updated.

[user@host ~]$ oc annotate machinehealthcheck -n openshift-machine-api \
  machine-api-termination-handler cluster.x-k8s.io/paused-
machinehealthcheck.machine.openshift.io/machine-api-termination-handler annotated

Over-the-air Updates

OTA follows a client-server approach. Red Hat hosts the cluster images and the update infrastructure. OTA generates all possible update paths for your cluster. OTA also gathers information about the cluster and your entitlement to determine the available upgrade paths. The web console sends a notification when a new update is available.

The following diagram describes the updates architecture: Red Hat hosts both the cluster images and a "watcher", which automatically detects new images that are pushed to Quay. The Cluster Version Operator (CVO) receives its update status from that watcher. The CVO starts by updating the cluster components via their operators, and then updates any extra components that the Operator Lifecycle Manager (OLM) manages.

Figure 9.5: OpenShift Container Platform updates architecture

With telemetry, Red Hat can determine the update path. The cluster uses a Prometheus-based Telemeter component to report on the state of each cluster operator. The data is anonymized and sent back to Red Hat servers that advise cluster administrators about potential new releases.

Note

Red Hat values customer privacy. For a complete list of the data that Telemeter gathers, consult the Data Collection and Telemeter Sample Metrics documents in the references section.

In the future, Red Hat intends to extend the list of updated operators that are included in the upgrade path to include independent software vendor (ISV) operators.

Figure 9.6: Managing cluster updates by using telemetry

The Update Process

The following components are involved in the cluster update process:

Machine Config Operator

The Machine Config Operator applies the desired machine state to each of the nodes. This component also handles the rolling upgrade of nodes in the cluster, and uses CoreOS Ignition as the configuration format.

Operator Lifecycle Manager

The OLM orchestrates updates to any operators that are running in the cluster.

Updating the Cluster

You can update the cluster via the web console or from the command line. The AdministrationCluster Settings page displays an update status of Available updates when a new update is available. From this page, click Select a version, and then select the version and the cluster update option that you want to install:

Figure 9.7: Update the cluster by using the web console

Important

Rolling back your cluster to an earlier version is not supported. If your update is failing to complete, contact Red Hat support.

The update process also updates the underlying operating system when updates are available. The updates use the rpm-ostree technology for managing transactional upgrades. Updates are delivered via container images and are part of the OpenShift update process. When the update deploys, the nodes pull the new image, extract it, write the packages to the disk, and then modify the bootloader to boot into the new version. The machine reboots and implements a rolling update to ensure that the cluster capacity is minimally impacted.

Update the Cluster by Using the Command Line

The following steps describe the procedure for updating a cluster as a cluster administrator by using the command-line interface:

  • Be sure to update all operators that are installed through the OLM to the 4.14 version before updating the OpenShift cluster.

  • Retrieve the cluster version and review the current update channel information. If you are running the cluster in production, then ensure that the channel reads stable.

    [user@host ~]$ oc get clusterversion
    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.14.0    True        False         2d      Cluster version is 4.14.0
    
    [user@host ~]$ oc get clusterversion -o jsonpath='{.items[0].spec.channel}{"\n"}'
    stable-4.14
  • View the available updates and note the version number of the update to apply.

    [user@host ~]$ oc adm upgrade
    Cluster version is 4.14.0
    
    Upstream is unset, so the cluster will use an appropriate default.
    Channel: stable-4.14 (available channels: candidate-4.14, candidate-4.15, eus-4.14, fast-4.14, stable-4.14)
    
    Recommended updates:
    
    VERSION    IMAGE
    4.14.10    quay.io/openshift-release-dev/ocp-release@sha256:...
    ...output omitted...
  • Apply the latest update to your cluster, or update to a specific version:

    • Run the following command to install the latest available update for your cluster.

      [user@host ~]$ oc adm upgrade --to-latest=true
    • Run the following command to install a specific version. VERSION corresponds to one of the available versions that the oc adm upgrade command returns.

      [user@host ~]$ oc adm upgrade --to=VERSION
  • The previous command initializes the update process. Run the following command to review the status of the Cluster Version Operator (CVO) and the installed cluster operators.

    [user@host ~]$ oc get clusterversion
    NAME     VERSION  AVAILABLE  PROGRESSING  SINCE  STATUS
    version  4.14.0   True       True         1m     Working towards 4.14.10 ...
    
    [user@host ~]$ oc get clusteroperators
    NAME                       VERSION     AVAILABLE   PROGRESSING   DEGRADED  ...
    authentication             4.14.0      True        False         False     ...
    baremetal                  4.14.10     False       True          False     ...
    cloud-controller-manager   4.14.10     True        False         True      ...
    ...output omitted...
  • Use the following command to review the cluster version history and monitor the status of the update. It might take some time for all the objects to finish updating.

    The history contains a list of the most recent versions that were applied to the cluster. This list is updated when the CVO applies an update. The list is ordered by date, where the newest update is first in the list.

    If the rollout completed successfully, then updates in the history have a Completed state. Otherwise, the update has a Partial state if it failed or did not complete.

    [user@host ~]$ oc describe clusterversion
    ...output omitted...
      History:
        Completion Time:    2024-02-10T04:38:12Z
        Image:              quay.io/openshift-release-dev/ocp-release@sha256:...
        Started Time:       2024-02-10T03:35:05Z
        State:              Partial
        Verified:           true
        Version:            4.14.10
        Completion Time:    2024-02-10T12:39:02Z
        Image:              quay.io/openshift-release-dev/ocp-release@sha256:...
        Started Time:       2024-02-10T12:23:14Z
        State:              Completed
        Verified:           false
        Version:            4.14.10

    Important

    When an update is failing to complete, the Cluster Version Operator (CVO) reports the status of any blocking components and attempts to reconcile the update.

    Rolling back your cluster to a previous version is not supported. If your update is failing to complete, contact Red Hat support.

  • After the process completes, you can confirm that the cluster is updated to the new version.

    [user@host ~]$ oc get clusterversion
    NAME     VERSION  AVAILABLE  PROGRESSING  SINCE  STATUS
    version  4.14.10  True       False        30m    Cluster version is 4.14.10

References

For more information about update channels, update prerequisites, and updating clusters in disconnected environments, refer to the Updating a Restricted Network Cluster and Updating a Cluster Between Minor Versions chapters in the Red Hat OpenShift Container Platform 4.14 Updating Clusters documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/updating_clusters/index#updating-restricted-network-cluster

For more information about updating operators that are installed through the Operator Lifecycle Manager, refer to the Upgrading Installed Operators section in the Administrator Tasks chapter in the Red Hat OpenShift Container Platform 4.14 Working with Operators documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/operators/index#olm-upgrading-operators

For more information about performing an EUS-to-EUS update, refer to the Preparing to Perform an EUS-to-EUS Update chapter in the Red Hat OpenShift Container Platform 4.14 Updating Clusters documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/updating_clusters/index#updating-eus-to-eus-upgrade_eus-to-eus-upgrade

For more information about the OpenShift Container Platform upgrade paths, visit the following page in the customer portal: https://access.redhat.com/solutions/4583231

For more information about the OpenShift Container Platform update graph, visit the following page in the customer portal: https://access.redhat.com/labs/ocpupgradegraph/update_path

For more information about OpenShift Extended Update Support (EUS), visit the following page in the customer portal: https://access.redhat.com/support/policy/updates/openshift-eus

For more information about the OpenShift Container Platform lifecycle policy, visit the following page in the customer portal: https://access.redhat.com/support/policy/updates/openshift

OpenShift 4 Data Collection

OpenShift 4 Telemeter Sample Metrics

Revision: do280-4.14-08d11e1