Red Hat OpenShift Administration I: Operating a Production Cluster

Operators are important components of Red Hat OpenShift Container Platform (RHOCP). Operators automate the required tasks to maintain a healthy RHOCP cluster that would otherwise require human intervention. Operators are the preferred method of packaging, deploying, and managing services on the control plane.

Operators integrate with Kubernetes APIs and CLI tools such as kubectl and oc commands. Operators provide the means of monitoring applications, performing health checks, managing over-the-air (OTA) updates, and ensuring that applications remain in your specified state.

Because CRI-O and the Kubelet run on every node, almost every other cluster function can be managed on the control plane by using Operators. Components that are added to the control plane by using operators include critical networking and credential services.

Operators in RHOCP are managed by two different systems, depending on the purpose of the operator.

Operators use one or more pods to provide cluster services. You can find the namespaces for these pods under the relatedObjects section of the detailed output for the operator. As a user with a cluster-admin role, use the -n namespace option on the get pod command to view the pods. For example, use the following get pods command to retrieve the list of pods in the openshift-dns-operator namespace.

[user@host~]$ oc get pods -n openshift-dns-operator
NAME                            READY   STATUS    RESTARTS   AGE
dns-operator-64688bfdd4-8zklh   2/2     Running   38         38d

Use the -o yaml or -o json output formats to view or analyze more details about the pods. The resource conditions, which are found in the status for the resource, track the current state of the resource object. The following example uses the jq processor to extract the status values from the JSON output details for the dns pod.

[user@host~]$ oc get pod -n openshift-dns-operator \
dns-operator-64688bfdd4-8zklh -o json | jq .status
{
  "conditions": [
    {
      "lastProbeTime": null,
      "lastTransitionTime": "2023-02-09T21:24:50Z",
      "status": "True",
      "type": "Initialized"
    },
...output omitted...

In addition to listing the pods of a namespace, you can also use the --show-labels option of the get command to print the labels used by the pods. The following example retrieves the pods and their labels in the openshift-etcd namespace.

[user@host~]$ oc get pods -n openshift-etcd --show-labels
NAME                   READY   STATUS      RESTARTS   AGE   LABELS
etcd-master01          4/4     Running     68         35d   app=etcd,etcd=true,k8s-app=etcd,revision=3
installer-3-master01   0/1     Completed   0          35d   app=installer

Another way to gauge the health of an RHOCP cluster is to examine the compute resource usage of cluster nodes and pods. The oc adm top command provides this information. For example, to list the total memory and CPU usage of all pods in the cluster, you can use the --sum option with the command to print the sum of the resource usage.

[user@host~]$ oc adm top pods -A --sum
NAMESPACE                 NAME                      CPU(cores)   MEMORY(bytes)
metallb-system            controller-...-ddr8v      0m           57Mi
metallb-system            metallb-...-n2zsv         0m           48Mi
...output omitted...
openshift-storage         topolvm-node-9spzf        0m           68Mi
openshift-storage         vg-manager-z8g5k          0m           23Mi
                                                   ------       --------
                                                    428m         10933Mi

The -A option shows pods from all namespaces. Use the -n namespace option to filter the results to show the pods in a single namespace. Use the --containers option to display the resource usage of containers within a pod. For example, use the following command to list the resource usage of the containers in the etcd-master01 pod in the openshift-etcd namespace.

[user@host~]$ oc adm top pods etcd-master01 -n openshift-etcd --containers
POD             NAME           CPU(cores)   MEMORY(bytes)
etcd-master01   POD            0m           0Mi
etcd-master01   etcd           71m          933Mi
etcd-master01   etcd-metrics   6m           32Mi
etcd-master01   etcd-readyz    4m           66Mi
etcd-master01   etcdctl        0m           0Mi

Viewing Cluster Metrics

The OpenShift web console incorporates graphs to visualize cluster and resource analytics. Cluster administrators and users with either the view or the cluster-monitoring-view cluster role can access the Home → Overview page. The Overview page displays a collection of cluster-wide metrics, and provides a high-level view of the overall health of the cluster.

The Overview page displays the following metrics:

• Current cluster capacity based on CPU, memory, storage, and network usage

• A time-series graph of total CPU, memory, and disk usage

• The ability to display the top consumers of CPU, memory, and storage

For any of the listed resources in the Cluster Utilization section, administrators can click the link for current resource usage. The link displays a window with a breakdown of top consumers for that resource. Top consumers can be sorted by project, by pod, or by node. The list of top consumers can be useful for identifying problematic pods or nodes. For example, a pod with an unexpected memory leak might appear at the top of the list.

Viewing Project Metrics

The Project Details page displays metrics that provide an overview of the resources that are used within the scope of a specific project. The Utilization section displays usage information about resources, such as CPU and memory, along with the ability to display the top consumers for each resource.

All metrics are pulled from Prometheus. Click any graph to navigate to the Metrics page. View the executed query, and inspect the data further.

If a resource quota is created for the project, then the current project request and limits appear on the Project Details page.

Viewing Resource Metrics

When troubleshooting, it is often useful to view metrics at a smaller granularity than for the entire cluster or project. The Pod Details page displays time-series graphs of the CPU, memory, and file system usage for a specific pod. A sudden change in these critical metrics, such as a CPU spike caused by high load, is visible on this page.

Figure 2.9: Time-series graphs showing various metrics for a pod

Performing Prometheus Queries in the Web Console

The Prometheus UI is a feature-rich tool for visualizing metrics and configuring alerts. The OpenShift web console provides an interface for executing Prometheus queries directly from the web console.

To perform a query, navigate to Observe → Metrics, enter a Prometheus Query Language expression in the text field, and click Run Queries. The results of the query are displayed as a time-series graph:

Figure 2.10: Using a Prometheus query to display a time-series graph

Note

The Prometheus Query Language is not discussed in detail in this course. Refer to the references section for a link to the official documentation.

Some developers consider OpenShift logs to be too low-level, thus making troubleshooting difficult. Fortunately, RHOCP provides a high-level logging and auditing facility called events. Kubernetes generates event objects in response to state changes in cluster objects, such as nodes, pods, and containers. Events signal significant actions, such as starting a container or destroying a pod.

To read events, use the get events command. The command lists the events for the current RHOCP project (namespace). You can display the events for a different project by adding the -n namespace option to the command. To list the events for all the projects, use the -A (or --all-namespaces) option.

The following get events command prints events in the openshift-kube-controller-manager namespace.

[user@host~]$ oc get events -n openshift-kube-controller-manager
LAST SEEN  TYPE    REASON                 OBJECT                                MESSAGE
12m        Normal  CreatedSCCRanges       pod/kube-controller-manager-master01  created SCC...

You can use the describe pod pod-name command to further narrow the results to a single pod. For example, to retrieve only the events that relate to a mysql pod, you can refer to the Events field from the output of the oc describe pod mysql command:

[user@host~]$ oc describe pod mysql
...output omitted...
Events:
  FirstSeen   LastSeen    Count From         Reason          Message
  Wed, 10 ... Wed, 10 ... 1     {scheduler } scheduled       Successfully as...
...output omitted...

[user@host~]$ oc get all -n openshift-monitoring --show-kind
NAME                                              READY  STATUS   RESTARTS AGE
pod/alertmanager-main-0                           6/6    Running  85       34d
pod/cluster-monitoring-operator-56b769b58f-dtmqj  2/2    Running  34       35d
pod/kube-state-metrics-75455b796c-8q28d           3/3    Running  51       35d
...output omitted...

[user@host~]$ oc logs alertmanager-main-0 -n openshift-monitoring
ts=2023-03-16T14:21:50.479Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=rhaos-4.14-rhel-8, revision=519cbb87494d2830821a0da0a657af69d852c93b)"
ts=2023-03-16T14:21:50.479Z caller=main.go:232 level=info build_context="(go=go1.19.4, user=root@232132c11c68, date=20230105-00:26:49)"
ts=2023-03-16T14:21:50.527Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml
...output omitted...

RHOCP clusters can have several components, including at least one control plane and at least one compute node. The two components can occupy a single node. The following oc command, or the matching kubectl command, can display the overall health of all cluster nodes.

[user@host~]$ oc cluster-info

The oc cluster-info output is high-level, and can verify that the cluster nodes are running. For a more detailed view into the cluster nodes, use the get nodes command.

[user@host~]$ oc get nodes
NAME       STATUS   ROLES                         AGE   VERSION
master01   Ready    control-plane,master,worker   35d   v1.27.6+f67aeb3

The example shows a single master01 node with multiple roles. The STATUS value of Ready means that this node is healthy and can accept new pods. A STATUS value of NotReady means that a condition triggered the NotReady status and the node is not accepting new pods.

As with any other RHOCP resource, you can drill down into further details of the node resource with the describe node node-name command. For parsable output of the same information, use the -o json or the -o yaml output options with the get node node-name command. For more information about using and parsing these output formats, see the section called “ Inspect Kubernetes Resources ”.

The output of the get nodes node-name command with the -o json or -o yaml option is long. The following examples use the -jsonpath option or the jq processor to parse the get node node-name command output.

[user@host~]$ oc get node master01 -o jsonpath=\
*'{"Allocatable:\n"}{.status.allocatable}{"\n\n"}
{"Capacity:\n"}{.status.capacity}{"\n"}'
Allocatable:
{"cpu":"7500m","ephemeral-storage":"114396791822","hugepages-1Gi":"0",
"hugepages-2Mi":"0","memory":"19380692Ki","pods":"250"}

Capacity:
{"cpu":"8","ephemeral-storage":"125293548Ki","hugepages-1Gi":"0",
"hugepages-2Mi":"0","memory":"20531668Ki","pods":"250"}

The JSONPath expression in the previous command extracts the allocatable and capacity measures for the master01 node. These measures help to understand the available resources on a node.

View the status object of a node to understand the current health of the node.

[user@host~]$ oc get node master01 -o json | jq '.status.conditions'
[
  {
    "lastHeartbeatTime": "2023-03-22T16:34:57Z",
    "lastTransitionTime": "2023-02-23T20:35:15Z",
    "message": "kubelet has sufficient memory available",
    "reason": "KubeletHasSufficientMemory",
    "status": "False",
    "type": "MemoryPressure" 
  },
  {
    "lastHeartbeatTime": "2023-03-22T16:34:57Z",
    "lastTransitionTime": "2023-02-23T20:35:15Z",
    "message": "kubelet has no disk pressure",
    "reason": "KubeletHasNoDiskPressure",
    "status": "False",
    "type": "DiskPressure" 
  },
  {
    "lastHeartbeatTime": "2023-03-22T16:34:57Z",
    "lastTransitionTime": "2023-02-23T20:35:15Z",
    "message": "kubelet has sufficient PID available",
    "reason": "KubeletHasSufficientPID",
    "status": "False",
    "type": "PIDPressure" 
  },
  {
    "lastHeartbeatTime": "2023-03-22T16:34:57Z",
    "lastTransitionTime": "2023-02-23T20:35:15Z",
    "message": "kubelet is posting ready status",
    "reason": "KubeletReady",
    "status": "True",
    "type": "Ready" 
  }
]

More conditions indicate other potential problems with a node.

To gain deeper insight into a given node, you can view the logs of processes that run on the node. A cluster administrator can use the oc adm node-logs command to view node logs. Node logs might contain sensitive output, and thus are limited to privileged node administrators. Use oc adm node-logs node-name to filter the logs to a single node.

The oc adm node-logs command has other options to further filter the results.

Use oc adm node-logs --help for a complete list of command options.

For example, to retrieve the most recent log entry for the crio service on the master01 node, you can use the following command.

[user@host~]$ oc adm node-logs master01 -u crio --tail 1
-- Logs begin at Thu 2023-02-09 21:19:09 UTC, end at Fri 2023-03-17 15:11:43 UTC. --
Mar 17 06:16:09.519642 master01 crio[2987]: time="2023-03-17 06:16:09.519474755Z" level=info msg="Image status:
&ImageStatusResponse{Image:&Image{Id:6ef8...79ce,RepoTags:[],RepoDigests:
...output omitted...

When you create a pod with the CLI, the oc or kubectl command is sent to the apiserver service, which then validates the command. The scheduler service reads the YAML or JSON pod definition, and then assigns pods to compute nodes. Each compute node runs a kubelet service that converts the pod manifest to one or more containers in the CRI-O container runtime.

Each compute node must have an active kubelet service and an active crio service. To verify the health of these services, first start a debug session on the node by using the debug command.

[user@host~]$ oc debug node/node-name

Replace the node-name value with the name of your node.

Within the debug session, change to the /host root directory so that you can run binaries in the host's executable path.

sh-4.4# chroot /host

Then, use the systemctl is-active calls to confirm that the services are active.

sh-4.4# for SERVICES in kubelet crio; do echo ---- $SERVICES ---- ;
systemctl is-active $SERVICES ;  echo ""; done
---- kubelet ----
active

---- crio ----
active

For more details about the status of a service, use the systemctl status command.

sh-4.4# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─01-kubens.conf, 10-mco-default-madv.conf, 20-logging.conf, 20-nodenet.conf
   Active: active (running) since Thu 2023-03-23 14:39:11 UTC; 1h 26min ago
 Main PID: 3215 (kubelet)
    Tasks: 28 (limit: 127707)
   Memory: 391.7M
      CPU: 14min 34.568s
...output omitted...

[user@host~]$ oc logs pod-name -c container-name

[user@host~]$ oc debug

[user@host~]$ oc debug job/test --as-user=1000000

[user@host~]$ oc debug node/master01
Starting pod/master01-debug-wtn9r ...
To use host binaries, run chroot /host
Pod IP: 192.168.50.10
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-5.1#

When opening a support case, it is helpful to provide debugging information about your cluster to Red Hat Support. It is recommended that you provide the following information:

The oc adm must-gather command collects resource definitions and service logs from your cluster that are most likely needed for debugging issues. This command creates a pod in a temporary namespace on your cluster, and the pod then gathers and downloads debugging information. By default, the oc adm must-gather command uses the default plug-in image, and writes into the ./must-gather.local. directory on your local system. To write to a specific local directory, you can also use the --dest-dir option, such as in the following example:

[user@host~]$ oc adm must-gather --dest-dir /home/student/must-gather

Then, create a compressed archive file from the must-gather directory. For example, on a Linux-based system, you can run the following command:

[user@host~]$ tar cvaf mustgather.tar must-gather/

Replace must-gather/ with the actual directory path.

Then, attach the compressed archive file to your support case in the Red Hat Customer Portal.

Similar to the oc adm must-gather command, the oc adm inspect command gathers information on a specified resource. For example, the following command collects debugging data for the openshift-apiserver and kube-apiserver cluster operators.

[user@host~]$ oc adm inspect clusteroperator/openshift-apiserver \
clusteroperator/kube-apiserver

The oc adm inspect command can also use the --dest-dir option to specify a local directory to write the gathered information. The command shows all logs by default. Use the --since option to filter the results to logs that are later than a relative duration, such as 5s, 2m, or 3h.

[user@host~]$ oc adm inspect clusteroperator/openshift-apiserver --since 10m