Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

OpenShift Logging does not automatically deploy Loki. You must deploy Loki by installing the Loki operator from OperatorHub or by using the CLI. See the references section for instructions.

The Loki operator creates and manages the components of the log store.

Storing logs for your cluster might require significant disk space. Because the required disk space depends on the size of your cluster and application workloads, you might need to plan your storage solution with application developers.

Similar to a persistent volume claim, you can use an object bucket claim to request an S3-compatible bucket. ODF creates a secret and a configuration map with the same name as the object bucket claim; between them, they provide the credentials and information to access the bucket.

The following excerpt is an example of an object bucket claim definition in the openshift-logging namespace:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: loki-bucket-odf
  namespace: openshift-logging
spec:
  generateBucketName: loki-bucket-odf
  storageClassName: openshift-storage.noobaa.io

ODF creates a configuration map with the bucket information in the openshift-logging namespace. You can retrieve one part of the credentials for the bucket from the configuration map as follows:

[user@host ~]$ oc get configmap/loki-bucket-odf -o yaml
apiVersion: v1
data:
  BUCKET_HOST: s3.openshift-storage.svc
  BUCKET_NAME: loki-bucket-odf-69d9...ab91
  BUCKET_PORT: "443"
  BUCKET_REGION: ""
  BUCKET_SUBREGION: ""
kind: ConfigMap
...output omitted...

You can retrieve the other part of the bucket credentials from the secret as follows:

[user@host ~]$ oc extract --to=- secret/loki-bucket-odf -n openshift-logging
# AWS_ACCESS_KEY_ID
o0sOY8oaZtPfP18DRjSa
# AWS_SECRET_ACCESS_KEY
z1QtDH37lUvLnkjpE3E4aS8yQI56CLPozGJOt31e

After you retrieve the credentials, create a secret that contains those credentials in the openshift-logging namespace for Loki to use the credentials:

[user@host ~]$ oc create secret generic logging-loki-odf \
  -n openshift-logging \
  --from-literal=access_key_id=${ACCESS_KEY_ID} \
  --from-literal=access_key_secret=${SECRET_ACCESS_KEY} \
  --from-literal=bucketnames=${BUCKET_NAME} \
  --from-literal=endpoint=https://${BUCKET_HOST}:${BUCKET_PORT}

After installing the Loki operator and creating the object bucket, you create and configure the LokiStack instance with the bucket credentials.

The following example is a LokiStack resource that uses the logging-loki-odf secret that contains the bucket credentials:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  limits: 
    global:
      retention: 
        days: 20
        stream: 
        - days: 4
          priority: 1
          selector: '{kubernetes_namespace_name=~"test.+"}'
  size: 1x.demo 
  storage:
    secret:
      name: logging-loki-odf 
      type: s3 
    tls:
      caName: openshift-service-ca.crt 
  storageClassName: ocs-external-storagecluster-ceph-rbd 
  tenants:
    mode: openshift-logging

After you create the LokiStack instance, the Loki operator starts scheduling pods for components. You can verify the installation by listing the pods in the openshift-logging project and verifying that the Loki pods are running.

The OpenShift Logging ClusterLogging CR configures and manages deploying the OpenShift Logging components.

To configure OpenShift Logging to use Loki as the log store, you must create a cluster logging resource in the openshift-logging namespace and set lokistack as the log store type. Then, OpenShift Logging uses Loki to store the infrastructure and application logs.

The following excerpt shows a minimum configuration for OpenShift Logging to enable Loki as the log store:

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  managementState: Managed
  logStore:
    type: lokistack 
    lokistack:
      name: logging-loki 
  collection:
    type: vector

After you create the ClusterLogging instance, the OpenShift Logging operator starts scheduling pods for components. You can verify the installation by listing the pods in the openshift-logging project and verifying that the collector pods are running.

Loki organizes logs into streams that are called chunks. A chunk is a sequence of log entries that use the same set of labels. Labels are key-value pairs that help identify and filter log entries. Labels can include information such as the application name, environment, severity, or any other relevant metadata. Loki uses LogQL as the query language for filtering and searching logs with labels and other criteria.

By using LogQL in the OpenShift web console, you can extract and filter specific information from the logs.

A LogQL query consists of the following parameters:

When using LogQL in OpenShift Logging, you must always use at least the log_type log stream selector for the log type.

For example, the following LogQL query shows the infrastructure logs:

log_type="infrastructure" | json

Adding the | json string to your query extracts all JSON properties as labels for ease of viewing. After you type the LogQL query, click Run Query to apply it to your logs.

By using LogQL you can filter your logs with more advanced expressions. For example, the following LogQL query filters the audit logs only from audit.log files in any directory:

log_type="audit" | json | file=~".*audit.log"

You can use multiple log stream selectors and filter expressions to increase filter granularity. The following example shows a LogQL query for the audit logs that contain the error string but not the warning string, and that return a 403 or 503 status.

log_type="audit" | json |= "error" != "warning" |~ "status [45]03"

Vector adds OpenShift metadata to pod logs to provide fields for querying and filtering. You can use these fields in your LogQL queries for more precise filtering.

Some fields that you can use in your queries include the following strings: