Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Instructions

Your company provides you a Bash script to verify the health of an OpenShift cluster. Because this script requires authentication to the cluster, you must generate an SA token to authenticate it. Use the TokenRequest API to generate the SA token. Do not use the automatically generated token secret. The username for the SA must be health-robot. The SA token must be valid for one week. The script verifies the health status for all the pods in the OpenShift cluster, and creates the /tmp/cluster.log file if a pod is in a pending, failed, or unknown state. Thus, because you need to read the pod information for the whole cluster, you must assign the cluster-reader cluster role to the SA. You can find the Bash script in the ~/DO380/labs/auth-tls/ocp_health.sh file. Create a RHEL cron job to run the script every minute.

Your company also needs a backdoor administrator account to use if the IdP that provides the administrator account fails. As a solution, you decide to create a client certificate. The username for the administrator certificate must be admin-backdoor. The admin-backdoor account must be part of the backdoor-administrators group, and you must assign the cluster-admin cluster role to that group to have administrator permissions for the cluster. The expiration for the administrator certificate must be one year.

1. Create the health-robot SA in the auth-tls OpenShift project and assign it the cluster-reader role.

   1. Connect to the OpenShift cluster as the admin user with redhatocp as the password.

      [student@workstation ~]$ oc login -u admin -p redhatocp \
        https://api.ocp4.example.com:6443
      Login successful.
      
      You have access to 70 projects, the list has been suppressed. You can list all
      projects with 'oc projects'
      
      Using project "default".
      Welcome! See 'oc help' to get started.

   2. Change to the auth-tls project.

      [student@workstation ~]$ oc project auth-tls
      Now using project "auth-tls" on server "https://api.ocp4.example.com:6443".

   3. Create the health-robot SA in the auth-tls project.

      [student@workstation ~]$ oc create sa health-robot
      serviceaccount/health-robot created

   4. Assign the cluster-reader cluster role to the health-robot SA so the SA can retrieve information from most of the objects in the cluster.

      [student@workstation ~]$ oc adm policy add-cluster-role-to-user \
        cluster-reader system:serviceaccount:auth-tls:health-robot
      clusterrole.rbac.authorization.k8s.io/cluster-reader added: "system:serviceaccount:auth-tls:health-robot"

2. Create the health-robot SA token and store it in the HEALTHROBOT_TOKEN variable. Create a kubeconfig file with the health-robot user credentials. Store the kubeconfig file in the ~/DO380/labs/auth-tls/robot-cert/health-robot.config file.

   1. Generate the health-robot SA account token and store it in the HEALTHROBOT_TOKEN variable. The SA token must be valid for one week, which is 604800 seconds.

      [student@workstation ~]$ HEALTHROBOT_TOKEN=$(oc create token -n auth-tls \
        health-robot --duration 604800s)

   2. Change to the ~/DO380/labs/auth-tls directory.

      [student@workstation ~]$ cd DO380/labs/auth-tls

   3. Add the health-robot user credentials to the kubeconfig file.

      [student@workstation auth-tls]$ oc config set-credentials health-robot \
        --token $HEALTHROBOT_TOKEN --kubeconfig robot-cert/health-robot.config
      User "health-robot" set.

   4. Set the cluster options in the kubeconfig file.

      [student@workstation auth-tls]$ oc config set-cluster \
        $(oc config view -o jsonpath='{.clusters[0].name}') \
        --server https://api.ocp4.example.com:6443 \
        --kubeconfig robot-cert/health-robot.config
      Cluster "api-ocp4-example-com:6443" set.

   5. Set the context for the health-robot user.

      [student@workstation auth-tls]$ oc config set-context health-robot \
        --cluster $(oc config view -o jsonpath='{.clusters[0].name}') \
        --namespace auth-tls --user health-robot \
        --kubeconfig robot-cert/health-robot.config
      Context "health-robot" created.

   6. Use the context for the health-robot user.

      [student@workstation auth-tls]$ oc config use-context health-robot \
        --kubeconfig robot-cert/health-robot.config
      Switched to context "health-robot".

3. Verify the permissions for the health-robot SA. Use the health-robot SA token through the kubeconfig file and user impersonation.

   1. Verify the identity for the health-robot user.

      [student@workstation auth-tls]$ oc whoami \
        --kubeconfig robot-cert/health-robot.config
      system:serviceaccount:auth-tls:health-robot

   2. Verify that the health-robot SA can retrieve information from the users in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i get users -A \
        --as system:serviceaccount:auth-tls:health-robot
      yes

   3. Verify that the health-robot SA cannot create projects in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i create project -A \
        --as system:serviceaccount:auth-tls:health-robot
      no

   4. Verify that the health-robot SA can retrieve information from the pods in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i get pods -A \
        --as system:serviceaccount:auth-tls:health-robot
      yes

   5. Run the cluster-health.sh script to verify the status of the pods in the OpenShift cluster. The script uses the credentials that are stored in the ~/DO380/labs/auth-tls/robot-cert/health-robot.config file.

      [student@workstation auth-tls]$ sh cluster-health.sh
      Connected to the cluster as the 'system:serviceaccount:auth-tls:health-robot' user
      ✔ OpenShift is reacheable and up, at version: '4.14.3'
      ✔ All pods are either running or succeeded.

4. Create a local RHEL cron job to execute the cluster-health.sh script every minute. Create a failing pod, and verify that the script creates a log file with the projects that contain pods in a pending, failed, or unknown state.

   1. Open the crontab file for the student with the default text editor.

      [student@workstation auth-tls]$ crontab -e

   2. Insert the following line.

      * * * * * /home/student/DO380/labs/auth-tls/cluster-health.sh

   3. Press Esc and type :wq to save the changes and exit the editor. When the editor exits, you should see the following output:

      crontab: installing new crontab
      [student@workstation auth-tls]$

   4. Create a failing pod in the auth-tls project.

      [student@workstation auth-tls]$ oc run failing-pod \
        --image registry.ocp4.example.com:8443/failing-pod
      ...output omitted...
      pod/failing-pod created

   5. Wait for a few minutes and verify the log file. The log file contains the date and the projects with pods in a pending, failed, or unknown state.

      [student@workstation auth-tls]$ cat /tmp/cluster.log
      ...output omitted...
      Tue Jul 11 10:29:04 EDT 2023
      ✘ Namespaces with pending pods:
      ✘ auth-tls

5. Create a client certificate for a backdoor administrator account to use if the IdP that provides the administrator account fails. The username for the administrator certificate must be admin-backdoor, and the user must be part of the backdoor-administrators group with the cluster-admin cluster role. Set the expiration time for the certificate to one year.

   1. Create the backdoor-administrators group in the OpenShift cluster.

      [student@workstation auth-tls]$ oc adm groups new backdoor-administrators
      group.user.openshift.io/backdoor-administrators created

   2. Assign the cluster-admin cluster role to the backdoor-administrators group so users in that group can act as cluster administrators.

      [student@workstation auth-tls]$ oc adm policy add-cluster-role-to-group \
        cluster-admin backdoor-administrators
      clusterrole.rbac.authorization.k8s.io/cluster-admin added: "backdoor-administrators"

   3. Create the admin-cert directory.

      [student@workstation auth-tls]$ mkdir admin-cert

   4. Create an OpenSSL certificate signing request (CSR) for the admin-backdoor username. The user must be part of the backdoor-administrators group.

      [student@workstation auth-tls]$ openssl req -newkey rsa:4096 -nodes \
        -keyout tls.key -subj "/O=backdoor-administrators/CN=admin-backdoor" \
        -out admin-cert/admin-backdoor-auth.csr
      Generating a RSA private key
      .................++++
      ...............................................++++
      writing new private key to 'tls.key'
      -----

   5. Define a signing request resource in a YAML file called admin-backdoor-csr.yaml. Set the expiration time for the certificate to one year. The name for the CSR must be admin-backdoor-access.

      [student@workstation auth-tls]$ cat << EOF >> admin-cert/admin-backdoor-csr.yaml
      apiVersion: certificates.k8s.io/v1
      kind: CertificateSigningRequest
      metadata:
        name: admin-backdoor-access
      spec:
        signerName: kubernetes.io/kube-apiserver-client
        expirationSeconds: 31536000  # one year
        request: $(base64 -w0 admin-cert/admin-backdoor-auth.csr)
        usages:
        - client auth
      EOF

   6. Create the CSR for the admin-backdoor user in the OpenShift cluster.

      [student@workstation auth-tls]$ oc create -f admin-cert/admin-backdoor-csr.yaml
      certificatesigningrequest.certificates.k8s.io/admin-backdoor-access created

   7. Verify the CSR status for the admin-backdoor user in the OpenShift cluster.

      [student@workstation auth-tls]$ oc get csr admin-backdoor-access
      NAME                  AGE   SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
      admin-backdoor-access 3m28s kuberne... admin     30d               Pending

   8. Approve the CSR in the OpenShift cluster.

      [student@workstation auth-tls]$ oc adm certificate approve admin-backdoor-access
      certificatesigningrequest.certificates.k8s.io/admin-backdoor-access approved

   9. Verify again the status for the CSR for the admin-backdoor user in the OpenShift cluster.

      [student@workstation auth-tls]$ oc get csr admin-backdoor-access
      NAME                  AGE   SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
      admin-backdoor-access 4m23s kuberne... admin     30d               Approved,Issued

6. Create a kubeconfig file with the admin-backdoor user credentials. Store the kubeconfig file in the ~/DO380/labs/auth-tls/admin-cert/admin-backdoor.config file.

   1. Extract the signed client certificate for the admin-backdoor user. Store the signed client certificate in the ~/DO380/labs/auth-tls/admin-cert/admin-backdoor-access.crt file.

      [student@workstation auth-tls]$ oc get csr admin-backdoor-access \
        -o jsonpath='{.status.certificate}' \
        | base64 -d > admin-cert/admin-backdoor-access.crt

   2. Add the admin-backdoor user credentials to the kubeconfig file.

      [student@workstation auth-tls]$ oc config set-credentials admin-backdoor \
        --client-certificate admin-cert/admin-backdoor-access.crt --client-key tls.key \
        --embed-certs --kubeconfig admin-cert/admin-backdoor.config
      User "admin-backdoor" set.

   3. Get the public certificate from the API server.

      [student@workstation auth-tls]$ openssl s_client -showcerts \
        -connect api.ocp4.example.com:6443 </dev/null 2>/dev/null|openssl x509 \
        -outform PEM > ocp-apiserver-cert.crt

   4. Set the cluster options in the kubeconfig file.

      [student@workstation auth-tls]$ oc config set-cluster \
        $(oc config view -o jsonpath='{.clusters[0].name}') \
        --certificate-authority ocp-apiserver-cert.crt --embed-certs=true \
        --server https://api.ocp4.example.com:6443 \
        --kubeconfig admin-cert/admin-backdoor.config
      Cluster "api-ocp4-example-com:6443" set.

   5. Set the context for the admin-backdoor user.

      [student@workstation auth-tls]$ oc config set-context admin-backdoor \
        --cluster $(oc config view -o jsonpath='{.clusters[0].name}') \
        --namespace default --user admin-backdoor \
        --kubeconfig admin-cert/admin-backdoor.config
      Context "admin-backdoor" created.

   6. Use the context for the admin-backdoor user.

      [student@workstation auth-tls]$ oc config use-context admin-backdoor \
        --kubeconfig admin-cert/admin-backdoor.config
      Switched to context "admin-backdoor".

7. Verify the permissions for the admin-backdoor user. Use the admin-backdoor user client certificate through the kubeconfig file and user impersonation.

   1. Verify the identity for the admin-backdoor user.

      [student@workstation auth-tls]$ oc whoami \
        --kubeconfig admin-cert/admin-backdoor.config
      admin-backdoor

   2. Verify that the admin-backdoor user can retrieve information from the users in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i get users -A \
        --as admin-backdoor --as-group backdoor-administrators
      yes

   3. Verify that the admin-backdoor user can create users in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i create users -A \
        --as admin-backdoor --as-group backdoor-administrators
      yes

   4. Verify that the admin-backdoor user can create projects in the OpenShift cluster.

      [student@workstation auth-tls]$ oc auth can-i create project -A \
        --as admin-backdoor --as-group backdoor-administrators
      yes

   5. Change to the /home/student directory.

      [student@workstation auth-tls]$ cd