Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Instructions

Your company requires you to send OpenShift logs to the company's centralized syslog server for long-term storage.

You must forward the following log types:

Install the OpenShift Logging operator and configure log forwarding to the syslog server. Deploy the OpenShift Event Router to capture Kubernetes events and forward them to the syslog server.

Verify that all required log types are forwarded and available on the syslog server. Forwarded OpenShift logs are stored in the /var/log/openshift path on the utility machine.

1. As the admin user, install the OpenShift Logging operator.

   1. Open a web browser and navigate to https://console-openshift-console.apps.ocp4.example.com

   2. Click Red Hat Identity Management and log in as the admin user with redhatocp as the password.

   3. Click Operators → OperatorHub. In the Filter by keyword field, type openshift logging to locate the OpenShift Logging operator, and then click Red Hat OpenShift Logging.

   4. The web console displays information about the Red Hat OpenShift Logging operator. Click Install to proceed to the Install Operator page.

   5. Click Install to install the operator with the default options.

   6. Wait until the installation is complete and the web console displays the ready for use message.

2. Configure OpenShift Logging to enable the Vector log collector.

   1. From the terminal, log in to your OpenShift cluster as the admin user.

      [student@workstation ~]$ oc login -u admin -p redhatocp \
        https://api.ocp4.example.com:6443
      Login successful.
      
      ...output omitted...

   2. Change to the openshift-logging project.

      [student@workstation ~]$ oc project openshift-logging
      Now using project "openshift-logging" on server "https://api.ocp4.example.com:6443".

   3. Change to the ~/DO380/labs/logging-forward directory.

      [student@workstation ~]$ cd ~/DO380/labs/logging-forward
      [student@workstation logging-forward]$

   4. Edit and complete the ClusterLogging resource definition in the ~/DO380/labs/logging-forward/clusterlogging.yml file.

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        managementState: Managed
        collection:
          type: vector

      Note

      You can use the solution in the ~/DO380/solutions/logging-forward/clusterlogging.yml file.

   5. Apply the cluster logging configuration.

      [student@workstation logging-forward]$ oc apply -f clusterlogging.yml
      clusterlogging.logging.openshift.io/instance created

3. Configure OpenShift Logging to forward logs to the syslog server.

   The syslog server is configured to filter OpenShift audit, infrastructure, and application logs into separate files in the /var/log/openshift path by using the msgID syslog attribute with the audit, infra, and apps values, respectively.

   Configure the cluster log forwarder and define three log pipelines to send each log type to the syslog server by using the matching msgID value.

   Ensure that only application logs with the logging: critical label are forwarded.

   The syslog server DNS name is utility.lab.example.com and the service listens to the TCP port 514.

   1. Edit and complete the ClusterLogForwarder resource definition in the ~/DO380/labs/logging-forward/clusterlogforwarder.yml file.

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
          name: instance
          namespace: openshift-logging
      spec:
          inputs:
          - name: critical-apps
            application:
              selector: 
                matchLabels:
                  logging: critical
      
          outputs:
          - name: audit-syslog 
            type: syslog
            url: tcp://utility.lab.example.com:514
            syslog:
              msgID: audit
      
      ...output omitted...
      
          - name: apps-syslog 
            type: syslog
            url: tcp://utility.lab.example.com:514
            syslog:
              msgID: apps
      
      ...output omitted...
      
          - name: infra-syslog 
            type: syslog
            url: tcp://utility.lab.example.com:514
            syslog:
              msgID: infra
      
      ...output omitted...
      
          pipelines: 
          - name: critical-apps-syslog
            inputRefs:
              - critical-apps
            outputRefs:
              - apps-syslog
      
          - name: infra-syslog
            inputRefs:
              - infrastructure
            outputRefs:
              - infra-syslog
      
          - name: audit-syslog
            inputRefs:
              - audit
            outputRefs:
              - audit-syslog

      	Label selector to include application pod logs with the logging: critical label only
      	Syslog parameters for the audit log type
      	Syslog parameters for the application log type
      	Syslog parameters for the infrastructure log type
      	Log pipelines for each log type

      Note

      You can use the solution in the ~/DO380/solutions/logging-forward/clusterlogforwarder.yml file.

   2. Apply the configuration for the cluster log forwarder.

      [student@workstation logging-forward]$ oc apply -f clusterlogforwarder.yml
      clusterlogforwarder.logging.openshift.io/instance created

   3. Verify that the OpenShift Logging operator deploys the collector pod on each node.

      [student@workstation logging-forward]$ oc get daemonset -l component=collector
      NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   ...
      collector   6         6         6       6            6           ...

4. Deploy the Event Router component to capture Kubernetes events.

   1. Use the OpenShift template from the /home/student/DO380/labs/logging-forward/eventrouter.yml file to deploy the Event Router component.

      [student@workstation logging-forward]$ oc process -f eventrouter.yml \
        | oc apply -f -
      serviceaccount/eventrouter created
      clusterrole.rbac.authorization.k8s.io/event-reader created
      clusterrolebinding.rbac.authorization.k8s.io/event-reader-binding created
      configmap/eventrouter created
      deployment.apps/eventrouter created

   2. Verify that the Event Router is running.

      [student@workstation logging-forward]$ oc get pod -l component=eventrouter
      NAME                           READY   STATUS    RESTARTS   AGE
      eventrouter-59dfb998cc-9v7gb   1/1     Running   0          8m17s

5. Verify that the syslog service received the logs on the utility machine.

   1. Open a new terminal and connect to the utility machine with SSH as the root user.

      [student@workstation logging-forward]$ ssh root@utility

      Note

      The remainder of this activity refers to this new terminal as the utility terminal, and the first terminal as the workstation terminal.

   2. Check that the OpenShift log files are created in the /var/log/openshift path.

      [root@utility ~]# ls -l /var/log/openshift/
      total 298564
      -rw-------.  1 root root    863869 Jan 11 11:38 apps.log
      -rw-------.  1 root root  70381151 Jan 11 11:38 audit.log
      -rw-------.  1 root root 204538257 Jan 11 11:38 infra.log

6. Verify that the journal logs from the cluster nodes are forwarded to the syslog server.

   1. On the utility terminal, change to the /var/log/openshift directory.

      [root@utility ~]# cd /var/log/openshift
      [root@utility openshift]#

   2. Monitor the infrastructure log and filter the output with the jq command to list events from the sshd daemon on the cluster nodes.

      [root@utility openshift]# tail -f infra.log | egrep -o "\{.*}$" \
        | jq '. | select(.systemd.u.SYSLOG_IDENTIFIER=="sshd")
          | .hostname + " " + .message'

      Note

      The select() function of the jq command filters the log messages that match the provided expression. In this specific case, only messages with the .systemd.u.SYSLOG_IDENTIFIER field with the value sshd are listed.

   3. On the workstation terminal, connect to the master01 cluster node with the ssh command as the core user.

      [student@workstation logging-forward]$ ssh core@master01.ocp4.example.com
      [core@master01 ~]$

   4. On the utility terminal, verify that the SSH connection is logged. Press Ctrl+C to exit the tail command.

      "master01 main: sshd: ssh-rsa algorithm is disabled"
      "master01 Accepted publickey for core from 172.25.250.9 port 51158 ssh2: RSA SHA256:M8i...BT0"
      "master01 pam_unix(sshd:session): session opened for user core(uid=1000) by (uid=0)"
      ^C
      [root@utility openshift]#

   5. On the workstation terminal, close the SSH connection with the exit command or Ctrl+D.

      [core@master01 ~]$ exit
      logout
      Connection to master01.ocp4.example.com closed.
      [student@workstation logging-forward]$

7. Verify that the audit logs from the cluster nodes are forwarded to the syslog server.

   1. On the utility terminal, review the audit log. Use the jq command to list the USER_LOGIN Linux audit events that are generated from the previous SSH connection to the master01 node.

      [root@utility openshift]# egrep -ho "\{.*}$" audit.log* \
        | jq '. | select(."audit.linux".type == "USER_LOGIN")
          | ."@timestamp" + " " + .hostname + " " + .message'
      ...output omitted...
      "2024-01-12T11:08:25.618+00:00 master01 type=USER_LOGIN msg=audit(1705057705.618:937): pid=106902 uid=0 auid=1000 ses=11 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe=\"/usr/sbin/sshd\" hostname=? addr=172.25.250.9 terminal=/dev/pts/0 res=success'\u001dUID=\"root\" AUID=\"core\" ID=\"core\""

      Note

      To use a JSON key that contains special characters, such as ., @ or - with the jq command, you must surround the key name with double quotes.

8. Verify that the infrastructure container logs are forwarded to the syslog server.

   1. On the utility terminal, monitor the infrastructure log. Use the jq command to display container logs from the openshift-storage namespace. Press Ctrl+C to exit the tail command.

      [root@utility openshift]# tail -f infra.log | egrep -o "\{.*}$" \
        | jq '. | select(.kubernetes.namespace_name == "openshift-storage")
          | .kubernetes.pod_name + " " + .message'
      ...output omitted...
      "noobaa-core-0 ... time=\"2024-01-12T11:45:54Z\" level=info msg=\"..."
      "noobaa-operator-... time=\"2024-01-12T11:45:54Z\" level=info msg=\"..."
      ^C
      [root@utility openshift]#

9. Verify that the Kubernetes audit logs are forwarded to the syslog server.

   1. On the utility terminal, monitor the audit log. Use the jq command to display audit events about the developer user.

      [root@utility openshift]# tail -f audit.log | egrep -o "\{.*}$" \
        | jq -c '.|select(.user.username == "developer")
          | [.user.username, .annotations, .verb, .objectRef]'

   2. On the workstation terminal, log in to the OpenShift cluster as the developer user with developer as the password.

      [student@workstation logging-forward]$ oc login -u developer -p developer \
        https://api.ocp4.example.com:6443
      Login successful.
      
      ...output omitted...

   3. Create the logger-app project.

      [student@workstation logging-forward]$ oc new-project logger-app
      Now using project "logger-app" on server "https://api.ocp4.example.com:6443".
      
      ...output omitted...

   4. On the utility terminal, verify that the Kubernetes audit events from the developer user are logged. Press Ctrl+C to exit the tail command.

      ["developer",{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"basic-users\" of ClusterRole \"basic-user\" to Group \"system:authenticated\""},"get",{"apiGroup":"user.openshift.io","apiVersion":"v1","name":"~","resource":"users"}]
      
      ...output omitted...
      
      ["developer",{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"self-provisioners\" of ClusterRole \"self-provisioner\" to Group \"system:authenticated:oauth\""},"create",{"apiGroup":"project.openshift.io","apiVersion":"v1","name":"logger-app","resource":"projectrequests"}]
      ^C
      [root@utility openshift]#

10. Verify that the Kubernetes events are forwarded to the syslog server.

    1. On the utility terminal, monitor the infrastructure log. Use the jq command to display the Kubernetes events on the logger-app namespace.

       [root@utility openshift]# tail -f infra.log | egrep -o "\{.*}$" \
         | jq -c '.
           | select(.kubernetes.event.involvedObject.namespace == "logger-app")
           | [.kubernetes.event.involvedObject.kind,
           .kubernetes.event.involvedObject.name, .message]'

    2. On the workstation terminal, deploy the logger application by using the definition in the ~/DO380/labs/logging-forward/logger.yml file.

       [student@workstation logging-forward]$ oc apply -f logger.yml
       deployment.apps/logger-app created

    3. On the utility terminal, verify that the Kubernetes events from the logger-app namespace are logged. Press Ctrl+C to exit the tail command.

       ...output omitted...
       ["Pod","logger-app-...","Add eth0 [10.11.0.18/23] from ovn-kubernetes"]
       ["Pod","logger-app-...","Container image \"registry.ocp4.example.com:8443/redhattraining/access-logger:v0.1\" already present on machine"]
       ["Pod","logger-app-...","Created container access-logger"]
       ["Pod","logger-app-...","Started container access-logger"]
       ^C
       [root@utility openshift]#

11. Verify that only for critical applications, such as the shop-web application in the shop project, the container logs are forwarded to the syslog server.

    1. On the utility terminal, review the application log. Verify that container logs from the shop-web application are forwarded.

       [root@utility openshift]# grep -m1 shop-web apps.log | egrep -o "\{.*}" | jq
       {
         "@timestamp": "2024-01-11T16:23:14.051244304Z",
       
       ...output omitted...
       
           "container_name": "shop-web",
           "labels": {
             "app": "shop-web",
             "logging": "critical",
             "pod-template-hash": "5cf65ffc4"
           },
           "namespace_name": "shop",
       
       ...output omitted...
       
         "log_type": "application",
         "message": "2024/01/11 16:23:14 \"GET /products?id=371 HTTP/1.1\" 403 \"Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/33.0\"",
       }

    2. Verify that container logs from the logger-app application that was deployed in the previous step are not forwarded.

       [root@utility openshift]# grep logger-app apps.log
       no output expected

    3. Close the utility terminal and return to the workstation terminal.

12. Clean up the resources.

    1. Change to the home directory.

       [student@workstation logging-forward]$ cd
       [student@workstation ~]$

    2. Log in to your OpenShift cluster as the admin user.

       [student@workstation ~]$ oc login -u admin -p redhatocp \
         https://api.ocp4.example.com:6443
       Login successful.
       
       ...output omitted...

    3. Delete the logger-app project.

       [student@workstation ~]$ oc delete project logger-app
       project.project.openshift.io "logger-app" deleted

    4. Delete the ClusterLogging and ClusterLogForwarder resources.

       [student@workstation ~]$ oc -n openshift-logging \
         delete clusterlogging,clusterlogforwarder --all
       clusterlogging.logging.openshift.io "instance" deleted
       clusterlogforwarder.logging.openshift.io "instance" deleted

    5. Remove the Event Router deployment.

       [student@workstation ~]$ oc -n openshift-logging delete deployment eventrouter
       deployment.apps "eventrouter" deleted