Bookmark this page

Chapter 5.  OpenShift GitOps

Abstract

Goal

Deploy OpenShift GitOps for managing clusters and applications.

Sections
  • GitOps for Kubernetes (and Quiz)

  • GitOps for Cluster Administration (and Guided Exercise)

  • GitOps for Application Management (and Guided Exercise)

Lab
  • OpenShift GitOps

GitOps for Kubernetes

Objectives

  • Define the fundamentals of GitOps and its use with Kubernetes clusters and applications.

Introduction to GitOps

Red Hat OpenShift GitOps is an operator that helps implement GitOps practices to manage OpenShift clusters.

Kubernetes can import and export resource definitions as text files. By working with resource definitions in text files, administrators describe their workloads instead of using a sequence of operations to create them. This approach is called declarative resource management.

By storing resource definitions in text files, Kubernetes administrators can use any tool or process for text files. For example, by keeping cluster resource definitions in a version control system, administrators can track changes to resources.

GitOps or infrastructure as code are terms to describe practices that relate to these concepts.

By enabling cluster users to manage Kubernetes resources by using a Git repository, administrators can further restrict direct access to the cluster. If users can modify Kubernetes resources only through a Git repository, then administrators can use Git features to enforce controls such as reviews and approvals. For example, many organizations use pull request workflows to update repositories.

Additionally, cluster management presents significant challenges when multiple processes can update cluster resources. Cluster administrators can face problems if they cannot clearly visualize the intended state of the cluster. When multiple users and processes update the state of the cluster, the intended state for the cluster can become unclear. The term drift describes the difference between the intended and actual states of a cluster.

With GitOps, a single process changes the cluster based on a definition of the intended cluster state. Administrators can then reference a single cluster definition to track what is happening with the cluster. Also, administrators who invest resources in documenting their changes can have reliable information about each change to the cluster.

Additionally, with a single cluster definition, administrators can reproduce the deployments of a cluster on separate clusters. For example, a cluster administrator can create a test cluster based on their production cluster definition. Then, the administrator can test cluster updates and other changes, some of which might be risky or hard to revert in a production environment, with reasonable guarantees that results from the test cluster apply to the production cluster. Production clusters often have different loads and interact with external services, so test clusters cannot reproduce the exact conditions. However, tests can provide the level of confidence that your organization requires for such changes.

GitOps and Kubernetes

Although the Kubernetes resource model is suited to GitOps, some Kubernetes idiosyncrasies require extra care.

Kubernetes resources contain information about both the definition and the status of the resource.

Consider the following excerpt from a Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: "2023-12-14T17:02:18Z"
  generation: 1
  name: hello-node
  namespace: test
...output omitted...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-node
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hello-node
    spec:
      containers:
      - command:
        - /agnhost
        - serve-hostname
        image: registry.k8s.io/e2e-test-images/agnhost:2.43
        name: agnhost
      restartPolicy: Always
      schedulerName: default-scheduler
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-12-14T17:02:22Z"
    lastUpdateTime: "2023-12-14T17:02:22Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
...output omitted...
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

The status key contains information about the number of available replicas, which can change during the lifetime of the deployment. Although the content of the metadata and spec keys is mostly the definition of the deployment, Kubernetes adds some fields, such as the creationTimestamp field, which are not necessary to re-create the deployment. Additionally, you might choose not to include in your resource definition some fields in the spec key with default values, such as the strategy key.

The kubectl command has subcommands to manipulate resource definitions, such as the apply, edit, and patch subcommands. These commands handle some complexities with updating resources.

Other Kubernetes features, such as mutating admission webhooks, can modify resources. These features can increase the complexity of updating resources.

Finally, although you can usually create resources in any order, with resources ultimately reaching the intended state, in some cases creating resources can bring more complexity. For example, you must create a namespace before creating the resources in the namespace. You can create custom resources only after the installation of the operator deploys the custom resource definitions.

Due to these factors, implementing GitOps involves additional sophistication. Simple GitOps implementations that are based on built-in features, such as the kubectl apply command, can present problems when resources increase in complexity.

System administrators can use other tools, such as Terraform and Ansible, to implement GitOps practices. Tools might differ in their approach and scope. Argo CD focuses on synchronizing Kubernetes resources from Git repositories to the cluster.

Introduction to OpenShift GitOps

OpenShift GitOps is an operator that manages Argo CD in OpenShift clusters. Argo CD is a tool that can synchronize Kubernetes resources to definitions that are stored in Git repositories.

Cluster administrators can use OpenShift GitOps to create Argo CD instances with specific access rights. Administrators can create an Argo CD instance that can manage specific namespaces and grant specific users access to the instance. With these instances, teams can be autonomous in deploying applications to specific namespaces.

OpenShift GitOps also includes a default administrative Argo CD instance that can manage Kubernetes resources that require administrative access, such as non-namespaced resources or authentication resources. OpenShift administrators can use Argo CD instances with these configurations to manage the cluster itself.

Argo CD uses the concept of applications. An application is an Argo CD resource that references a Git repository with Kubernetes resource definitions. Users can create and manage Argo CD applications from the web console or with the Kubernetes API.

Applications define a synchronization policy. Argo CD can detect and synchronize changes automatically, or users can trigger synchronizations on demand.

The Argo CD web console displays the status of application resources graphically. This representation includes both the resources that are defined in the application, and resources that are related to the application. For example, for an application that contains an operator subscription, the Argo CD web console also shows the install plans, operator workloads, and other related resources.

In this representation, you can view whether the resources are synchronized with the application definition, and inspect their status.

With OpenShift GitOps, cluster administrators can restrict permissions so that only Argo CD instances can update cluster resources, and Git repositories fully describe the intended cluster state. Even if other processes can modify the cluster state, Argo CD can detect and address issues with drift. Additionally, cluster administrators might use Argo CD to manage multiple clusters from the same cluster definition. Besides testing, having a cluster definition can help with other tasks, such as sharing resource definitions between clusters.

Continuous Integration, Continuous Delivery, and Continuous Deployment

The Argo CD name refers to the continuous delivery and continuous deployment practices. The CI/CD abbreviation refers to these practices and to continuous integration.

Continuous integration

Originally, continuous integration referred to integrating changes often to reduce conflicts from developing changes in isolation for long periods. Nowadays, continuous integration often refers to running tests automatically, and to practices such as requiring tests to pass before merging changes.

Continuous delivery

Continuous delivery is the practice of making each change a potential release of the project. With continuous delivery, the team can deploy the project at any time.

Continuous deployment

Continuous deployment is a variant of continuous delivery where every change triggers an automated deployment. Continuous deployment maximizes the release frequency of the project and shortens feedback loops.

CI/CD practices aim to reduce the lead time to deliver a new product or feature. When production issues are found, you can roll back small increments to known stable states.

OpenShift GitOps helps implement continuous delivery and deployment, but does not cover continuous integration. Red Hat provides other products that can help with implementing CI/CD:

Red Hat OpenShift Pipelines

The OpenShift Pipelines product is based on the open source Tekton project, and can run processes when changes occur to a Git repository. OpenShift Pipelines can implement the modern interpretation of CI, by validating changes. OpenShift Pipelines can also update container images with changes to source repositories.

With third-party applications, the application provider often provides updated images that you can deploy with Argo CD. However, with in-house applications, you can create updated images with OpenShift Pipelines.

Jenkins

Jenkins is a generic automation tool, to implement any type of CI/CD process.

For OpenShift automation, Red Hat recommends OpenShift Pipelines and OpenShift GitOps instead of Jenkins. The functions of the latter products require extra effort to implement with Jenkins. The OpenShift documentation covers migrating from Jenkins to OpenShift Pipelines.

By combining these products to implement CI/CD practices, organizations can create highly automated workflows that speed up development and that follow the necessary quality and security guidelines.

GitOps Workflows

The following steps describe an example of such a workflow:

  • A developer updates the source code of an application.

  • The developer pushes the change to a Git repository branch that contains the source code of the application, and creates a pull request.

  • OpenShift Pipelines detects the new pull request and runs automated checks:

    • The application source code is checked for mechanical errors and incorrect formatting. OpenShift Pipelines posts an update to the Git repository, to indicate whether the pull request passed the checks or conversely contains errors.

    • OpenShift Pipelines triggers an Argo CD job that deploys the application with the specified change to a testing environment. Any team members can access the deployment to validate the change. Argo CD can mark the change as failed if the application fails to deploy.

  • After the changes are validated, other team members can review the pull request. If validation failed, then a team member can fix the issues and resubmit. With this process, reviewers review only changes that passed automated validation.

  • For any issues that reviewers find in the change, they can request changes from the original developer to address issues.

  • Approved changes are merged.

  • OpenShift Pipelines detects the merged change and builds an updated container image from the source code of the application, and triggers an Argo CD synchronization of the production environment.

  • Argo CD deploys the updated image.

To ensure adequate velocity, the organization should ensure that the automated processes can complete quickly, and limit manual approvals to what is strictly required. For example, minor changes might not require a review; typical changes might require a single approver; and significant changes might require approvals from multiple specific stakeholders. These processes aim to automate and speed up the delivery of changes, and introduce only necessary validation. Organizations can choose their tradeoffs between speed and quality, according to their needs.

Warning

Following GitOps procedures can reduce security risks by introducing validation steps. However, version control systems such as Git can also introduce security risks.

For example, if you push sensitive data to a repository, then although your process might prevent pushing the data to production, unintended people might access the sensitive data. Furthermore, permanently removing information from a Git repository can require significant effort. Invalidating the data (for example, by creating an authentication token and disabling the leaked token) can be more efficient than permanently removing the sensitive data from the repository.

You can use elements, such as Git hooks, to improve the security of version control systems. Git hooks can prevent users from pushing sensitive information from their private workstation environment to a more accessible environment.

The following list describes some improvements to GitOps workflows that Argo CD can implement:

  • Propagate changes through multiple static environments, such as testing, pre-production, and production environments.

  • Limit synchronizations to specific days of the week or periods during the day.

  • Add hooks for further processes, such as for sending notifications, pausing monitoring, or triggering backups before an Argo CD synchronization.

References

For more information, refer to the Understanding OpenShift GitOps chapter in the Red Hat OpenShift GitOps 1.10 Understanding OpenShift GitOps documentation at https://access.redhat.com/documentation/en-us/red_hat_openshift_gitops/1.10/html-single/understanding_openshift_gitops/index#what-is-gitops

For more information about Red Hat OpenShift Pipelines, refer to the About Red Hat OpenShift Pipelines chapter in the Red Hat OpenShift Pipelines 1.13 About OpenShift Pipelines documentation at https://access.redhat.com/documentation/en-us/red_hat_openshift_pipelines/1.13/html-single/about_openshift_pipelines/index#understanding-openshift-pipelines

Revision: do380-4.14-397a507