DO120 - ch02s12

Bookmark this page

Guided Exercise: Configure Log Forwarding

Deploy OpenShift Logging to forward logs to Amazon CloudWatch.

Outcomes

Prepare AWS Identity and Access Management (IAM) resources to forward OpenShift log messages to Amazon CloudWatch.
Deploy logging for Red Hat OpenShift.
Configure the log collector to send log messages to CloudWatch.
Review log messages from CloudWatch.

To perform this exercise, ensure that you have completed the section called “Guided Exercise: Configure Developer Self-service for a ROSA Cluster ”.

Procedure 2.6. Instructions

Verify that you are logged in to your ROSA cluster from the OpenShift CLI.
1. Open a command-line terminal on your system, and then run the oc whoami command to verify your connection to the ROSA cluster. If the command succeeds, then skip to the next step.
```
$ oc whoami
wlombardogh
```
  The username is different in your command output.
2. If the command returns an error, then reconnect to your ROSA cluster. Run the rosa describe cluster command to retrieve the URL of the OpenShift web console.
```
$ rosa describe cluster --cluster do120-cluster
...output omitted...
Console URL:     https://console-openshift-console.apps.do120-cluster.jf96.p1.openshiftapps.com
...output omitted...
```
  The URL in the preceding output is different on your system.
3. Open a web browser, and then navigate to the OpenShift web console URL. Click github-do120. If you are not already logged in to GitHub, then provide your GitHub credentials.
4. Click your name in the upper right corner of the web console, and then click Copy login command. If the login page is displayed, then click github-do120 and use your GitHub credentials for authentication.
5. Click Display Token, and then copy the oc login --token command to the clipboard.
6. Paste the command into the command-line terminal, and then run the command.
```
$ oc login --token=sha256~1NofZkVCi3qCBcBJGc6XiOJTK5SDXF2ZYwhAARx5yJg
  --server=https://api.do120-cluster.jf96.p1.openshiftapps.com:6443
Logged into "https://api.do120-cluster.jf96.p1.openshiftapps.com:6443" as "wlombardogh" using the token provided.
...output omitted...
```
  In the preceding command, the token and the URL are different on your system.
Create the RosaCloudWatch IAM policy that gives access to Amazon CloudWatch operations.
1. Download the policy.json file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/policy.json.
2. Review the policy.json file. You do not have to change its contents.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:PutRetentionPolicy"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}
```
3. Use the aws iam create-policy command to create the RosaCloudWatch policy. Note the Amazon Resource Name (ARN) that the command returns. You use it in a following step.
  On a Microsoft Windows system, replace the line continuation character (\) in the following long command with the backtick (`) character, which is the line continuation character in PowerShell.
```
$ aws iam create-policy --policy-name RosaCloudWatch \
  --policy-document file://policy.json --query Policy.Arn --output text
arn:aws:iam::452954386616:policy/RosaCloudWatch
```
  The ARN in the preceding output is different on your system.
Prepare the trust relationship between the OpenShift service account and the IAM role.
1. List the OpenID Connect identity provider that the ROSA creation process created during the cluster installation. Note the ARN of the provider. You use it in a following step.
```
$ aws iam list-open-id-connect-providers
{
    "OpenIDConnectProviderList": [
        {
            "Arn": "arn:aws:iam::452954386616:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/235a3shus1umik6dfaln9gd11d894aun"
        }
    ]
}
```
  The provider in the preceding output is different on your system.
  If the command lists several providers, then run the rosa describe cluster -c do120-cluster command to help you select the correct one. The output displays the OIDC endpoint URL at the end. Select the ARN which last part matches the last part of this URL.
2. Download the trust-policy.json file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/trust-policy.json.
3. Edit the trust-policy.json file. You use this file in a following step. Set the Federated attribute to the provider ARN that you retrieved in a previous step. Compose the key in the StringEquals section from the URL of the provider. This URL is the part of the provider ARN after the first forward slash.
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::452954386616:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/235a3shus1umik6dfaln9gd11d894aun"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "rh-oidc.s3.us-east-1.amazonaws.com/235a3shus1umik6dfaln9gd11d894aun:sub": "system:serviceaccount:openshift-logging:logcollector"
                }
            }
        }
    ]
}
```
Create the do120-cluster-RosaCloudWatch IAM role that enables Vector to use the IAM policy to log messages with CloudWatch.
1. Use the aws iam create-role command to create the do120-cluster-RosaCloudWatch IAM role.
```
$ aws iam create-role --role-name do120-cluster-RosaCloudWatch \
  --assume-role-policy-document file://trust-policy.json
...output omitted...
```
2. Attach the IAM policy to the IAM role. In the following command, replace the policy ARN with the ARN that you retrieved in a previous step.
```
$ aws iam attach-role-policy --role-name do120-cluster-RosaCloudWatch \
  --policy-arn arn:aws:iam::452954386616:policy/RosaCloudWatch
```

Install the Red Hat OpenShift Logging operator.

Download the subscription.yaml resource file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/subscription.yaml.

Review the subscription.yaml file. You do not have to change its contents.

---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging
spec:
  channel: "stable"
  installPlanApproval: Automatic
  name: cluster-logging
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Use the oc apply command to deploy the operator.

$ oc apply -f subscription.yaml
subscription.operators.coreos.com/cluster-logging created

Wait for the operator to complete its installation. It takes up to five minutes for the process to complete. Rerun the oc get csv command regularly until it reports success in the PHASE column.

$ oc get csv -n openshift-logging
NAME                                     DISPLAY                   ...  PHASE
cluster-logging.v5.6.5                   Red Hat OpenShift Logging ...  Succeeded
observability-operator.v0.0.20           Observability Operator    ...  Succeeded
route-monitor-operator.v0.1.500-6152b76  Route Monitor Operator    ...  Succeeded

The ROSA cluster deploys the two other operators during cluster installation. They are not related to the logging subsystem.

Configure Vector to forward logs to CloudWatch by creating a ClusterLogForwarder resource named instance.

Retrieve the ARN of the do120-cluster-RosaCloudWatch IAM role that you created in a previous step. You use that ARN to create an OpenShift secret in the following step.
```
$ aws iam get-role --role-name do120-cluster-RosaCloudWatch --query Role.Arn \
  --output text
arn:aws:iam::452954386616:role/do120-cluster-RosaCloudWatch
```
The ARN in the preceding output is different on your system.

Create the cloudwatch-credentials secret that stores the role ARN. For the --from-literal option, use the role ARN that you retrieved in the preceding step.

$ oc create secret generic cloudwatch-credentials -n openshift-logging \
  --from-literal \
  "role_arn=arn:aws:iam::452954386616:role/do120-cluster-RosaCloudWatch"
secret/cloudwatch-credentials created

Download the vector-conf.yaml resource file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/vector-conf.yaml.

Edit the vector-conf.yaml file. Set the region attribute to your AWS Region. You can retrieve your AWS Region by running the rosa whoami command.

---
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: cw
      type: cloudwatch
      cloudwatch:
        groupBy: namespaceName
        groupPrefix: rosa-do120-cluster
        region: us-east-1
      secret:
        name: cloudwatch-credentials
  pipelines:
    - name: to-cloudwatch
      inputRefs:
        - infrastructure
        - audit
        - application
      outputRefs:
        - cw

The logging subsystem uses the groupPrefix parameter as a prefix to the name of the CloudWatch groups.

Use the oc apply command to create the resource.

$ oc apply -f vector-conf.yaml
clusterlogforwarder.logging.openshift.io/instance created

Configure the logging subsystem to use Vector as the log collector.
1. Download the logging-conf.yaml resource file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/logging-conf.yaml.
2. Review the logging-conf.yaml file. You do not have to change its contents.
```
---
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
  managementState: Managed
```
3. Use the oc apply command to create the resource.
```
$ oc apply -f logging-conf.yaml
clusterlogging.logging.openshift.io/instance created
```
Verify that the logging subsystem forwards some log messages to CloudWatch.
1. Use the aws logs describe-log-groups command to list the CloudWatch groups. Because OpenShift should already have sent some messages, the audit and infrastructure groups are displayed. You might have to wait a minute for the command to list the two groups.
```
$ aws logs describe-log-groups
{
    "logGroups": [
        {
            "logGroupName": "rosa-do120-cluster.audit",
            "creationTime": 1682333951009,
...output omitted...
        },
        {
            "logGroupName": "rosa-do120-cluster.infrastructure",
            "creationTime": 1682333949512,
...output omitted...
        }
    ]
}
```
  The rosa-do120-cluster prefix comes from the groupPrefix parameter that you defined in the ClusterLogForwarder resource.
2. Log in to the AWS Management Console at https://console.aws.amazon.com/ by using a web browser.
3. Select the AWS Region for your cluster.
  If you do not remember the name of your region, then use the rosa describe cluster command:
```
$ rosa describe cluster -c do120-cluster
...output omitted...
Region:                     us-east-1
...output omitted...
```
  The region in the preceding output might be different on your system.
4. Navigate to Services → Management & Governance → CloudWatch.
5. Select Logs → Log groups. The page displays the two groups: rosa-do120-cluster.audit and rosa-do120-cluster.infrastructure.
6. Click rosa-do120-cluster.infrastructure, and then click one of the log stream to access the log messages.
7. Expand one of the log entries to access its details.
8. Do not log out from the AWS Management Console, and do not close the web browser.

To verify that the logging subsystem also forwards application logs to CloudWatch, create the configure-logs project, and then deploy the application from the access-logger.yaml resource file.

From the terminal, use the oc new-project command to create the configure-logs project.

$ oc new-project configure-logs
Now using project "configure-logs" on server "https://api.do120-cluster.jf96.p1.openshiftapps.com:6443".
...output omitted...

Deploy the application from the access-logger.yaml resource file at https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/access-logger.yaml.
In the following command the URL is long: you must type it on a single line without any space.
```
$ oc apply -f https://raw.githubusercontent.com/RedHatTraining/DO12X-apps/main/ROSA/configure-logs/access-logger.yaml
deployment.apps/access-logger created
```
Wait for the pod to start. You might have to rerun the command several times for the pod to be ready and to report a Running status.
```
$ oc get pods
NAME                           READY   STATUS    RESTARTS   AGE
access-logger-dc8979c7-bvhxc   1/1     Running   0          16s
```
The pod name in the preceding output is different on your system.

Verify that the pod generates some log messages:

$ oc logs access-logger-dc8979c7-bvhxc
2023/04/26 10:34:29 "PUT /cart HTTP/1.1" 500 "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0"
2023/04/26 10:34:30 "GET /products?id=14 HTTP/1.1" 200 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
2023/04/26 10:34:32 "GET /products?id=371 HTTP/1.1" 403 "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/33.0"
2023/04/26 10:34:33 "GET /static/css HTTP/1.1" 404 "Mozilla/5.0 (Windows NT 6.0; rv:34.0)"
...output omitted...

Verify that the logging subsystem created a CloudWatch group for the configure-logs project.

$ aws logs describe-log-groups
{
    "logGroups": [
        {
            "logGroupName": "rosa-do120-cluster.audit",
...output omitted...
        },
        {
            "logGroupName": "rosa-do120-cluster.configure-logs",
            "creationTime": 1682335142555,
...output omitted...
        },
        {
            "logGroupName": "rosa-do120-cluster.infrastructure",
...output omitted...
        }
    ]
}

Review the log messages from the AWS Management Console.
1. Switch back to the AWS Management Console. Navigate to Services → Management & Governance → CloudWatch.
2. Select Logs → Log groups. The page displays the three groups: rosa-do120-cluster.audit, rosa-do120-cluster.configure-logs, and rosa-do120-cluster.infrastructure.
3. Click rosa-do120-cluster.configure-logs, and then click the kubernetes.var.log.pods.configure-logs_access-logger log stream to access the log messages for the access-logger pod.
4. Expand one of the log entries to access its details. The message attribute displays the message that the pod logged.

Because the logging subsystem forwards the log messages to CloudWatch, only the log collector component is running on OpenShift.

List the pods in the openshift-logging project to verify that only the pods for the operator and for the log collector are running.

$ oc get pods -n openshift-logging
NAME                                        READY   STATUS    RESTARTS   AGE
cluster-logging-operator-6fd6dbc575-m2nkj   1/1     Running   0          113m
collector-g2zkt                             2/2     Running   0          68m
collector-s8n4t                             2/2     Running   0          68m
collector-vdg28                             2/2     Running   0          68m
collector-wvxr2                             2/2     Running   0          68m
collector-zfhpq                             2/2     Running   0          68m

The pod names in the preceding output are different on your system.

Clean up your work by deleting the configure-logs project. Delete the ClusterLogging and ClusterLogForwarder resources to stop forwarding logs, and then uninstall the Red Hat OpenShift Logging operator. Optionally, delete your do120-cluster ROSA cluster if you do not plan to run other exercises.

Use the oc delete project command to delete the project.

$ oc delete project configure-logs
project.project.openshift.io "configure-logs" deleted

Stop forwarding logs to Amazon CloudWatch by deleting the ClusterLogging and ClusterLogForwarder resources named instance.

$ oc delete ClusterLogging instance -n openshift-logging
clusterlogging.logging.openshift.io "instance" deleted
$ oc delete ClusterLogForwarder instance -n openshift-logging
clusterlogforwarder.logging.openshift.io "instance" deleted

Uninstall the Red Hat OpenShift Logging operator by deleting the subscription and the clusterserviceversion resources.

$ oc delete subscription cluster-logging -n openshift-logging
subscription.operators.coreos.com "cluster-logging" deleted
$ oc get csv -n openshift-logging
NAME                                      DISPLAY                   ...
cluster-logging.v5.6.5                    Red Hat OpenShift Logging ...
$ oc delete csv cluster-logging.v5.6.5 -n openshift-logging
clusterserviceversion.operators.coreos.com "cluster-logging.v5.6.5" deleted

Optional. Delete the do120-cluster ROSA cluster.

$ rosa delete cluster --cluster do120-cluster
? Are you sure you want to delete cluster do120-cluster? Yes
I: Cluster 'do120-cluster' will start uninstalling now
...output omitted...

Revision: do120-4.12-b978842

Introduction to Red Hat OpenShift Service on AWS (ROSA)

Guided Exercise: Configure Log Forwarding