Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

OpenShift uses Red Hat Enterprise Linux CoreOS (RHCOS) as the underlying operating system in the hosts. The entire operating system is updated as a single image, instead of on a package-by-package basis. The machine configuration operator (MCO) manages the RHCOS operating system upgrades and configuration changes.

Do not use traditional Red Hat Enterprise Linux management approaches, such as manually editing files or using system commands such as the systemctl command, for RHCOS operating system upgrades or configuration changes. Those changes can conflict with the MCO and the MCO can override them.

The MCO comprises the following pods:

The MCO also requires the machine-config-daemon systemd server that RHCOS provides. The machine configuration daemon (MCD) implements updates to machine configurations and validates each machine's state in accordance with the requested configuration.

The MCO can manage the following resources:

The MCO defines two custom resources (CRs), which are part of the machineconfiguration.openshift.io/v1 API group.

OpenShift administrators set custom node configurations by declaring the MachineConfig and MachineConfigPool CRs.

The following diagram shows the relationship between the node, MC, and MCP labels:

Figure 3.4: Relationship between the node, MC, and MCP labels

In the previous diagram, the MCP CR uses the nodeSelector parameter to select all the nodes with the worker role, and applies to them the MCs with the machineconfiguration.openshift.io/role: worker label that is selected by using the machineConfigSelector parameter.

The MCO also manages two custom resources (CRs) for modifying CRI-O container runtime settings and the Kubelet service: the ContainerRuntimeConfig and KubeletConfig CRs, respectively.

MCs declare instance configurations by using the Ignition configuration format.

Ignition files encode file contents by using the Base64 encoding scheme. Other items, such as systemd units or SSH keys, do not use the Base64 encoding scheme. In a terminal, use the base64 and base64 -d commands to encode and decode files or standard input.

You can also use Butane to create Ignition files and MCs. Butane simplifies handling Base64 encoding.

The following example shows an MC to modify the configuration file for the journald service by using the Base64 encoding scheme:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker 
  name: 60-journald 
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,VGVzdGl...E8gKItAo= 
        filesystem: root
        mode: 0644
        path: /etc/systemd/journald.conf

The following example shows an MC to change the default kernel to a real-time kernel. This configuration does not use the Base64 encoding scheme:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-kernel-realtime
spec:
  kernelType: realtime

You can use the oc explain mc command to read the parameters that you can change by using an MC.

You can list the MCs on your cluster by using the oc get machineconfig or oc get mc commands. To list the MCs for a specific node label, use the --selector argument. The following example lists the MCs for the worker role in the cluster:

[user@host ~]$ oc get machineconfig \
  --selector machineconfiguration.openshift.io/role=worker
NAME                            GENERATEDBYCONTROLLER  IGNITIONVERSION  AGE
00-worker                       52fe...bf97            3.2.0            108d
01-worker-container-runtime     52fe...bf97            3.2.0            108d
01-worker-kubelet               52fe...bf97            3.2.0            108d
99-worker-chrony-conf-override                         3.2.0            108d
99-worker-generated-registries  52fe...bf97            3.2.0            108d
99-worker-ssh                                          3.2.0            108d

The MCO reads MCs alphanumerically by the name, from 00-* to 99-*. OpenShift stores the resulting compilation of MCs in a rendered MC resource. Thus, if two or more MCs apply changes to the same file, then the MCO applies only the changes from the MC with a higher number. For two MCs with the same number, the MCO uses the last one alphabetically. For example, the 99-worker-ssh-new MC has precedence over the 99-worker-ssh-last MC.

MCPs use labels to match one or more MCs to one or more nodes. By using multiple MCs, you can split the node configuration and focus every MC on one aspect of server configuration. For example, you can configure one MC for DNS resolution and another MC to synchronize time.

You can also use the same MCs in multiple MCPs. For example, although a time synchronization MC would apply to multiple MCPs, only the MCPs for a specific node pool would include the configuration that a special hardware accelerator card requires.

MCPs specify a machineConfigSelector MC label to select one or more MCs, and a nodeSelector node label to select one or more nodes.

By default, OpenShift includes MCPs for the master and worker roles. However, you can add custom MCPs to the cluster. Red Hat recommends creating custom MCPs as a composition of worker and custom MCs that are applied to the nodes. Assigning the worker role to the custom MCP is critical so that OpenShift applies operating system updates that are labeled as worker to the machines in the pool. Thus, the machineConfigSelector match expression selects both the worker role and the custom label. The nodes that are part of the custom pool use the MCs from the worker role with the additions from the custom label.

Figure 3.5: Custom pool with the worker MCs and an additional MC

In the previous diagram, the custom MCP targets two nodes with the custom label, and the worker MCP targets three nodes with the worker label. The worker pool applies to the worker nodes the 00-worker, 01-worker-kubelet, and 99-worker-ssh MCs. The custom pool uses the MCs from the worker role, and adds the 99-custom-ntp MC to the configuration of the custom nodes.

The following MCP specification demonstrates creating a separate pool for custom nodes:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
    name: custom 
spec:
    machineConfigSelector:
        matchExpressions:
            - key: machineconfiguration.openshift.io/role
              operator: In
              values: [worker, custom] 
    nodeSelector:
        matchLabels:
            node-role.kubernetes.io/custom: "" 

You can list the MCPs in your cluster by using the oc get machineconfigpool or oc get mcp commands.

[user@host ~]$ oc get machineconfigpool
NAME    CONFIG                     UPDATED  UPDATING  DEGRADED
MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
master  rendered-master-716...267  True     False     False
3             3                  3                    3                     108d
worker  rendered-worker-573...3db  True     False     False
3             3                  3                    3                     108d

The previous command shows the following information for the MCPs:

You can add labels to your node by using the oc label command.

The following example adds the custom role to the worker03 node in the cluster.

[user@host ~]$ oc label node/worker03 node-role.kubernetes.io/custom=
node/worker03 labeled

You can verify the node roles in the cluster by using the oc get nodes command:

[user@host ~]$ oc get nodes
NAME      STATUS  ROLES                 AGE   VERSION
master01  Ready   control-plane,master  114d  v1.25.7+eab9cc9
master02  Ready   control-plane,master  114d  v1.25.7+eab9cc9
master03  Ready   control-plane,master  114d  v1.25.7+eab9cc9
worker01  Ready   worker                12d   v1.25.7+eab9cc9
worker02  Ready   worker                12d   v1.25.7+eab9cc9
worker03  Ready   custom,worker         12d   v1.25.7+eab9cc9

One important node label is the infra role. Use the infra role for nodes that host only infrastructure components, such as cluster logging, cluster monitoring, or the integrated container image registry. Adding the infra role for nodes is recommended for larger clusters to ensure the performance and stability of OpenShift cluster services, such as the router or OAuth services, or to prevent the impact of heavy infrastructure components, such as metrics and logging, on user workloads.

Infrastructure nodes do not count towards the total number of required OpenShift subscriptions to run the environment. You can create a custom MCP to apply MCs to the infrastructure nodes.

The MCD reports the state of the node updates by using node annotations. You can use these annotations to assess the state of the update.

You can list the node annotations by using the oc describe command:

[user@host ~]$ oc describe node worker01
Name:        worker01
Roles:       worker
Labels:      beta.kubernetes.io/arch=amd64
...output omitted...
             node-role.kubernetes.io/worker=
             node.openshift.io/os_id=rhcos
Annotations: machineconfiguration.openshift.io/currentConfig: rendered-worker-370...bfd 
             machineconfiguration.openshift.io/desiredConfig: rendered-worker-370...bfd 
             machineconfiguration.openshift.io/reason:
             machineconfiguration.openshift.io/state: Done 
             volumes.kubernetes.io/controller-managed-attach-detach: true
...output omitted...

When the intended configuration does not match the current configuration, the MCD applies the intended rendered MC, drains all the pods from the node, and reboots the node.

[user@host ~]$ oc describe mcp worker
...output omitted...
Status:
  Conditions:
    ...output omitted...
    Last Transition Time:  2023-10-02T10:11:37Z
    Message:               Node worker01 is reporting: "content mismatch for file \"/etc/containers/registries.conf\""
    Reason:                1 nodes are reporting degraded status on sync
    Status:                True
    Type:                  NodeDegraded
    Last Transition Time:  2023-10-02T10:11:37Z
    Message:
    Reason:
    Status:                True
    Type:                  Degraded
...output omitted...

[user@host ~]$ oc get pod -n openshift-machine-config-operator \
  --field-selector spec.nodeName=worker01
NAME                          READY   STATUS    RESTARTS   AGE
machine-config-daemon-jsrzm   2/2     Running   2          19d
[user@host ~]$ oc logs machine-config-daemon-jsrzm \
  -n openshift-machine-config-operator
...output omitted...
E1002 10:11:36.163676    2667 daemon.go:589] Preflight config drift check failed: content mismatch for file "/etc/containers/registries.conf"
E1002 10:11:36.163692    2667 writer.go:200] Marking Degraded due to: content mismatch for file "/etc/containers/registries.conf"
W1002 10:11:40.193224    2667 daemon.go:1763] current+desiredConfig is rendered-worker-d4f45006b2d83725d98af944c8296774 but state is Degraded
I1002 10:11:40.510208    2667 rpm-ostree.go:394] Running captured: rpm-ostree kargs
E1002 10:11:40.568388    2667 on_disk_validation.go:207] content mismatch for file "/etc/containers/registries.conf" (-want +got):
  []uint8(
  	"""
- 	unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
+ 	unqualified-search-registries = ["docker.io"]
  	short-name-mode = ""

  	... // 107 identical lines
  	"""
  )
E1002 10:11:40.568446    2667 daemon.go:589] Preflight config drift check failed: content mismatch for file "/etc/containers/registries.conf"
E1002 10:11:40.568467    2667 writer.go:200] Marking Degraded due to: content mismatch for file "/etc/containers/registries.conf"
E1002 10:11:46.987190    2667 daemon.go:1077] mode mismatch for file: "/etc/containers/registries.conf"; expected: -rw-r--r--/420/0644; received: -rwxrwxrwx/511/0777
E1002 10:11:46.987222    2667 writer.go:200] Marking Degraded due to: mode mismatch for file: "/etc/containers/registries.conf"; expected: -rw-r--r--/420/0644; received: -rwxrwxrwx/511/0777
...output omitted...

The MCO provides the ContainerRuntimeConfig CR to modify CRI-O container runtime settings, and the KubeletConfig CR to manage the Kubelet service. You can use these CRs to configure a subset of CRI-O and Kubelet configuration parameters. Always use valid values for the configuration parameter, because invalid values might render cluster nodes unusable.

Although you can modify the CRI-O container runtime settings and the Kubelet service by using the MachineConfig CR, using either of these two special CRs simplifies node deployment and configuration management, provides API checking, and prevents misconfigurations. Moreover, because OpenShift does not support changing all the settings of the Kubelet service and the container runtime, these CRs provide only the configuration changes that OpenShift supports.