Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

The Red Hat Enterprise Linux CoreOS (RHCOS) operating system that runs on each node in an OpenShift cluster is not administered with the traditional methods for Red Hat Enterprise Linux. To adapt to the paradigm shift of using RHCOS as an infrastructure operating system, learning the methods for managing RHCOS is a large part of gaining the required knowledge to manage OpenShift clusters.

Although the Machine Configuration Operator (MCO) can handle many of the preferred customizations for the operating system that runs on your nodes, other special purpose operators are available for higher-level configurations, such as to focus on specialized hardware tuning, cluster organization, and advanced optimizations.

OperatorHub is a library of shared operators that provides a curated catalog for identifying and installing available operators in a cluster. The CLI and web console approaches are available for installing these special purpose operators, including those operators that are not available in OperatorHub.

Each operator runs in a designated namespace, and uses CRDs and CRs to define the implementation. An operator's function might also extend to actions across other namespaces. Red Hat recommends running each special purpose operator within its own namespace and using a service account with the appropriate permissions. Although operators typically configure the namespace and service account as part of the installation process, some operators might require manual setup.

The Node Tuning Operator is a cluster operator that manages node-level optimizations by using the TuneD service and the tuned daemon. The operator manages the containerized tuned daemon for RHOCP as a Kubernetes daemon set. Creating custom TuneD profiles can enable granular customizations to cluster nodes. This approach helps the cluster nodes provide ideal environments for the running applications.

The Node Tuning Operator is installed by default in RHOCP clusters, and runs in the openshift-cluster-node-tuning-operator namespace. You can view the default TuneD profile for the cluster by running the following command:

[user@host ~]$ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator

You can view the applied TuneD profiles for each node with the following command:

[user@host ~]$ oc get profile -n openshift-cluster-node-tuning-operator

When customizing a cluster node, create a TuneD profile and specify the optimized parameters from a TuneD plug-in. Each of the TuneD plug-ins, such as cpu, sysctl, or vm, provides various customizations for the cluster nodes. Consult the TuneD documentation from Red Hat to explore the full list of TuneD profiles.

Node Feature Discovery (NFD) is a Kubernetes add-on in OperatorHub for installation, to expose node-level information. The NFD operator detects hardware details, such as processor architecture, specialized PCI hardware such as GPUs or networking cards, and other system configurations on your cluster nodes. The NFD operator runs inside the openshift-nfd namespace for these operations.

The operator profiles each of the cluster nodes and collects information about the hardware and system attributes. This operator collects information about connected PCI devices, CPU specifications, kernel information, the operating system version, and other hardware information. The operator adds labels to the cluster nodes with the gathered information, to aid deployment placement onto nodes that provide the required specialized hardware and environments.

Consider a cluster that added high-throughput networking cards to a subset of its nodes. After the NFD operator profiles and labels these nodes, applications that require the higher network throughput can target the labeled nodes with this hardware.

The Kernel Module Management (KMM) Operator, which is available for installation in OperatorHub, manages, builds, signs, and deploys out-of-tree kernel modules and device plug-ins on OpenShift Container Platform clusters. This operator is useful for adding extra, alternative, or custom modules to the nodes in your cluster that run workloads that use those features.

Many applications, especially those applications that adopt later technologies before a full integration into modern operating systems is available, can require customizing the kernel modules for the cluster nodes. Developers who use Artificial Intelligence (AI) and Machine Learning (ML) applications often author custom kernel modules to optimize the computational operations.

You can install the KMM by searching in OperatorHub to add the operator to a cluster. The KMM uses the openshift-kmm namespace to manage kernel module deployments. A Module CRD specifies the kernel that can be added to one or more cluster nodes. A corresponding Module CR applies this CRD to install the required kernel version for each cluster node.

Many hardware vendors author special purpose operators for OpenShift to provide unique functions that correspond to a particular product that the company sells. Companies such as IBM, Nvidia, and Intel author and distribute these operators to help cluster administrators in using the specialized vendor hardware that is installed on cluster nodes. Each of these special purpose operators can vary in availability, and might or might not be available in OperatorHub. When adding hardware to cluster nodes, consider vendors who provide OpenShift operators that help with the management and function of cluster nodes.