Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Instructions

The security policy of your company requires you to send OpenShift audit logs to a dedicated syslog server for long-term storage.

The developer team asks you to provide them access to the production application logs and to the CI job logs that are running in the build-ci namespace. To limit the footprint of the log storage, the developers agree to keep only seven days of logs for the build-ci namespace.

Configure OpenShift Logging to forward only audit logs to the syslog server. The syslog server DNS name is utility.lab.example.com and the service listens on the TCP port 514. Forwarded audit logs are stored in the /var/log/openshift/audit.log file on the utility machine.

Configure Loki as the internal log store for both infrastructure logs for the cluster administrators and application logs for the developers.

An S3 bucket is available for you, in the lab environment, to configure as log storage for Loki. The bucket information and credentials are available in the ~/DO380/labs/logging-review/s3bucket.env file on the workstation machine.

Ensure that only application logs with the environment: production label or in the build-ci namespace are collected. Ensure that audit logs are not stored in the internal log store.

Configure the global log retention in Loki to 30 days. Configure the log retention for pods that are running in the build-ci namespace to seven days.

Provide access to the application logs to the ocpdevs group. You can use the developer user, which is in the ocpdevs group, to verify that the permissions are correct.

Use the following applications in the cluster to verify that the logging configuration is working as expected.

Applications that are running in the following projects have the environment: production label:

Applications that are running in the following projects have the environment: staging label:

Applications that are running in the following projects have the environment: development label:

1. Create a secret for Loki with the object storage credentials from the ~/DO380/labs/logging-review/s3bucket.env environment file.

   1. Connect to the OpenShift cluster as the admin user with redhatocp as the password.

      [student@workstation ~]$ oc login -u admin -p redhatocp \
        https://api.ocp4.example.com:6443
      Login successful.
      ...output omitted...

   2. Change to the ~/DO380/labs/logging-review directory.

      [student@workstation ~]$ cd ~/DO380/labs/logging-review

   3. Change to the openshift-logging project.

      [student@workstation logging-review]$ oc project openshift-logging
      Now using project "openshift-logging" on server "https://api.ocp4.example.com:6443".

   4. Use the ~/DO380/labs/logging-review/s3bucket.env environment file to create the logging-loki-odf secret in the openshift-logging namespace.

      [student@workstation logging-review]$ oc create secret generic logging-loki-odf \
        -n openshift-logging --from-env-file=s3bucket.env
      secret/logging-loki-odf created

   Hide Solution

2. Create and configure a logging-loki LokiStack instance to use the S3 bucket, and set the log retention for the build-ci namespace.

   You can use the partial resource definition in the ~/DO380/labs/logging-review/lokistack.yml file.

   1. Modify the partial resource definition in the ~/DO380/labs/logging-review/lokistack.yml file as follows:

      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: logging-loki
        namespace: openshift-logging
      spec:
        limits:
          global:
            retention:
              days: 30
              streams:
                - selector: '{kubernetes_namespace_name="build-ci"}'
                  priority: 1
                  days: 7
        size: 1x.demo
        storage:
          tls:
            caName: openshift-service-ca.crt
          secret:
            name: logging-loki-odf
            type: s3
        storageClassName: ocs-external-storagecluster-ceph-rbd
        tenants:
          mode: openshift-logging

   2. Create the LokiStack resource.

      [student@workstation logging-review]$ oc create -f lokistack.yml
      lokistack.loki.grafana.com/logging-loki created

   3. Verify that the LokiStack pods are up and running.

      [student@workstation logging-review]$ oc get deployment,statefulset \
        -l app.kubernetes.io/name=lokistack
      NAME                                          READY  UP-TO-DATE  AVAILABLE  AGE
      deployment.apps/logging-loki-distributor      1/1    1           1          86s
      deployment.apps/logging-loki-gateway          2/2    2           2          85s
      deployment.apps/logging-loki-querier          1/1    1           1          85s
      deployment.apps/logging-loki-query-frontend   1/1    1           1          85s
      
      NAME                                          READY  AGE
      statefulset.apps/logging-loki-compactor       1/1    85s
      statefulset.apps/logging-loki-index-gateway   1/1    85s
      statefulset.apps/logging-loki-ingester        1/1    86s

   Hide Solution

3. Configure OpenShift Logging to forward audit logs to the syslog server, and to forward the infrastructure and application logs to the internal log store.

   Ensure that only application logs with the environment: production label or in the build-ci namespace are collected. Ensure that audit logs are not stored in the Loki instance.

   You can use the partial resource definition in the ~/DO380/labs/logging-review/clusterlogforwarder.yml file.

   1. Modify the partial resource definition in the ~/DO380/labs/logging-review/clusterlogforwarder.yml file as follows:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        inputs:
          - name: production-apps
            application:
              selector:
                matchLabels:
                  environment: production
          - name: ci
            application:
              namespaces:
              - build-ci
        outputs:
          - name: audit-syslog
            type: syslog
            url: tcp://utility.lab.example.com:514
            syslog:
              msgID: audit
              appName: ocp-lab
              facility: user
              procID: vector
              rfc: RFC5424
              severity: informational
        pipelines:
          - name: to-syslog
            inputRefs:
              - audit
            outputRefs:
              - audit-syslog
          - name: to-loki
            inputRefs:
              - infrastructure
              - ci
              - production-apps
            outputRefs:
              - default

   2. Apply the configuration for the cluster log forwarder.

      [student@workstation logging-review]$ oc apply -f clusterlogforwarder.yml
      clusterlogforwarder.logging.openshift.io/instance created

   Hide Solution

4. Configure OpenShift Logging to deploy Vector and the web console plug-in, and use the Loki instance that you configure in a previous step as the log store.

   You can use the partial resource definition in the ~/DO380/labs/logging-review/clusterlogging.yml file.

   1. Modify the partial resource definition in the ~/DO380/labs/logging-review/clusterlogging.yml file as follows:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        managementState: Managed
        logStore:
          type: lokistack
          lokistack:
            name: logging-loki
        collection:
          type: vector
        visualization:
          type: ocp-console

   2. Apply the cluster logging configuration.

      [student@workstation logging-review]$ oc apply -f clusterlogging.yml
      clusterlogging.logging.openshift.io/instance created

   3. Verify that the OpenShift Logging operator deploys the collector pod on each node.

      [student@workstation logging-review]$ oc get daemonset -l component=collector
      NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   ...
      collector   6         6         6       6            6           ...

   Hide Solution

5. Connect to the utility server and verify that the syslog service receives the audit logs. Audit logs are stored in the /var/log/openshift/audit.log file on the utility machine.

   1. Connect to the utility machine with SSH as the root user.

      [student@workstation logging-review]$ ssh root@utility

   2. Check that the OpenShift audit log file exists in the /var/log/openshift path.

      [root@utility ~]# ls -l /var/log/openshift/
      total 978112
      -rw-------.  1 root root 462107806 Jan 30 10:10 audit.log

   3. Review the content of the audit log.

      [root@utility ~]# tail -1 /var/log/openshift/audit.log
      "2024-01-30T10:10:25.618+00:00 ...output omitted..."

   4. Disconnect from the utility machine.

      [root@utility ~]# exit
      logout
      Connection to utility closed.
      [student@workstation logging-review]$

   Hide Solution

6. Enable the web console plug-in for the OpenShift Logging operator and verify that infrastructure and application logs are available.

   1. Open a web browser and navigate to https://console-openshift-console.apps.ocp4.example.com. Click Red Hat Identity Management and log in as the admin user with redhatocp as the password.

   2. Click Operators → Installed Operators, and select All Projects from the drop-down menu.

   3. Click Red Hat OpenShift Logging, click Console plugin, select Enable, and click Save.

   4. Reload the web console, and navigate to Observe → Logs. If the Observe → Logs menu is not available, then wait until the web console shows the Web console update is available message and reload the web console.

      Verify that the application logs are available.

   5. From the drop-down menu, select infrastructure and verify that the infrastructure logs are available.

      

   6. From the drop-down menu, select audit and confirm that no audit logs are stored in the log store.

   Hide Solution

7. Grant view access to the application logs to the ocpdevs group.

   1. Add the cluster-logging-application-view role to the ocpdevs group.

      [student@workstation logging-review]$ oc adm policy add-cluster-role-to-group \
        cluster-logging-application-view ocpdevs
      clusterrole.rbac.authorization.k8s.io/cluster-logging-application-view added: "ocpdevs"

   Hide Solution

8. Verify that the developer user can access the application logs from the web console.

   Verify that only application logs with the environment: production label or in the build-ci namespace are collected.

   1. Open a new private browser window, and navigate to https://console-openshift-console.apps.ocp4.example.com

   2. Click Red Hat Identity Management and log in as the developer user with developer as the password. Click Skip tour.

   3. Navigate to Observe and select build-ci from the project drop-down menu. Change to the Logs tab. Verify that the application logs are available.

   4. Change to the shop-prod project and verify that the application logs are available.

   5. Change to the shop-dev project, and verify that no application logs are available, because the application does not have the required label.

   6. Close both the web browser windows and change to the /home/student directory in the terminal window.

      [student@workstation logging-review]$ cd

   Hide Solution