Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Red Hat OpenShift GitOps is an operator in OperatorHub, from the Red Hat operator catalog. OperatorHub provides a latest channel, with the most recent stable version of the operator. OperatorHub also provides gitops-version channels, such as the gitops-1.10 channel, to install specific versions of the operator.

You can install the operator in a similar way to any other operator, for example by using the installation wizard from the web console. You can use the default operator installation options, although most deployments require customization after the operator installation.

The operator creates its workload in the openshift-gitops-operator namespace. This workload includes a deployment that monitors Argo CD custom resources and manages their workloads.

The OpenShift GitOps operator installs the ArgoCD custom resource definition. Cluster administrators can use this custom resource definition to declaratively create multiple Argo CD instances in the cluster. The operator manages the workloads to run the Argo CD instances.

With this capability, administrators can introduce instances with different permissions. Because Argo CD applications can create and modify any type of OpenShift resource, limiting the permissions of Argo CD instances can reduce risks. With the OpenShift GitOps operator, you can create Argo CD instances by creating a custom resource that describes an instance (including RBAC configuration), and the operator manages the Argo CD instance. This process helps administrators apply the principle of least privilege, by creating Argo CD instances with granular permissions.

OpenShift GitOps includes further resources. For example, Argo CD uses the Application custom resource to manage applications. When you create an application with the Argo CD web interface, Argo CD creates an Application custom resource. This course does not cover working with those resources.

By default, the OpenShift GitOps operator creates a default Argo CD instance with the openshift-gitops name for both the instance and the namespace. This default instance has a set of permissions to support cluster administration. Red Hat recommends using this instance only for administering the cluster or for creating other instances for regular application deployment with minimal privileges.

The default instance has a limited configuration that might require further configuration to meet organizational requirements.

With the openshift-gitops-server route in the openshift-gitops namespace, the Argo CD web console is accessible outside the OpenShift cluster. By default, the web console is at the openshift-gitops-server-openshift-gitops.apps.cluster_domain URL.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  creationTimestamp: "2024-01-02T11:27:35Z"
  finalizers:
  - argoproj.io/finalizer
  generation: 1
  name: openshift-gitops
  namespace: openshift-gitops
  ownerReferences:
  - apiVersion: pipelines.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: GitopsService
    name: cluster
    uid: a7a0a073-0328-449e-8ffc-038854feb1c9
  resourceVersion: "262080"
  uid: 1d530e04-8068-4f6f-a066-c3ae36ba34f4
spec:
  applicationSet:
...output omitted...

Users can interact with Argo CD through the Kubernetes API, or through the Argo CD API, web interface, or command-line tool. Each Argo CD instance has a role-based access control system to manage permissions in the Argo CD interfaces.

Argo CD can use both local users that are defined in Argo CD and users from external authentication systems. The OpenShift GitOps operator includes features to automate integration with the OpenShift cluster authentication.

The default Argo CD instance includes a local admin user. This user has full permissions in the Argo CD interfaces. Argo CD updates the cluster by using a Kubernetes service account. Even unrestricted users in Argo CD are limited in how they can update the cluster, according to the permissions of the Argo CD service account.

The openshift-gitops-cluster secret in the openshift-gitops namespace contains the password for this user. Although this user can be suitable for small environments or experiments in test clusters, administrators usually integrate Argo CD with other authentication sources, such as the OpenShift cluster authentication.

The following excerpt from the default Argo CD instance shows the default authentication configuration:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
...output omitted...
spec:
...output omitted...
  rbac:
    defaultPolicy: ""
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
    scopes: '[groups]'
...output omitted...
  sso:
    dex:
      openShiftOAuth: true
...output omitted...

The rbac key contains the Argo CD role-based access control configuration. This configuration grants the admin role to the cluster-admins group.

You can create a cluster-admins group in the cluster and add users to have administrator privileges for the Argo CD instance. Alternatively, you can edit the policy to grant administrator privileges to existing groups.

The sso key contains the authentication configuration. In the default Argo CD instance, the openShiftOAuth key is set to the true value. With this configuration, OpenShift users can log in to the Argo CD web console by clicking LOG IN VIA OPENSHIFT. By default, OpenShift users have limited privileges.

Argo CD validates HTTPS certificates when accessing Git repositories, and can access only trusted repositories.

The repo server component of Argo CD handles communicating with Git repositories. To change the trusted certificate authorities, modify the certificate bundle in the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem path of the repo server.

The Argo CD custom resource contains fields to customize the repo server, similar to the volumes and volumeMounts keys of a pod template. The following example shows how to replace the certificate bundle with a bundle that is stored in a configuration map:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
...output omitted...
spec:
...output omitted...
  repo:
    volumeMounts:
    - mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      name: volume_name
      subPath: bundle_name
    volumes:
    - configMap:
        name: configuration_map_name
      name: volume_name
...output omitted...

The default Argo CD instance has its own self-signed HTTPS certificate. The Argo CD resource has a termination key to change the route termination. Use the reencrypt termination so that Argo CD uses the default router certificate.

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
...output omitted...
spec:
...output omitted...
  server:
...output omitted...
    route:
      enabled: true
      tls:
        termination: reencrypt
...output omitted...

By default, the web console for an Argo CD instance is at the argo_cd_instance-server-namespace.apps.cluster_domain URL. Log in to Argo CD by accessing this URL.

Figure 5.1: The Argo CD login page

If the Argo CD instance is configured with the openShiftOAuth key, then click LOG IN VIA OPENSHIFT to log in by using the cluster authentication.

Alternatively, you can use the Username and Password fields, and the SIGN IN button to use Argo CD local users.

When you log in, the web console displays the Applications page. The web console contains a navigation panel:

Figure 5.2: The Argo CD navigation panel

You can view and create applications in the Applications page. You can use the User Info page to view details about the logged-in user. These details can be useful to troubleshoot authentication issues.

Figure 5.3: The Argo CD applications page

Figure 5.4: Application details

The application details page displays the Kubernetes resources that are associated with an Argo CD application. For each resource, you can view the synchronization status and resource details.

Creating Applications

You can create Argo CD applications by using the Kubernetes API, the argocd command-line application, or the web console.

By default, the Argo CD web console shows a form to customize the application. The console also provides a YAML resource editor, which is similar to the OpenShift web console.

Figure 5.5: Application creation form

Use the following fields to create your application:

Application Name

The application resource name.

Project Name

Argo CD uses projects to group applications and to apply further access controls. Projects are not covered in this course. You can create applications in the default project that Argo CD provides.

Sync Policy

The Automatic policy synchronizes periodically. With the Manual policy, administrators must trigger synchronizations.

Retry

As described later in this lecture, you can use these fields to add retries to the synchronization process. You might need to configure retries when working with complex applications.

Repository URL

The URL of the repository with the application definition. If you use the HTTPS protocol to access the repository, then the repo server must trust the HTTPS certificate.

Path

The path in the Git repository that contains the resource definitions. You can define multiple applications from different paths in the same repository. A repository might also contain other content.

Cluster URL

The URL of the Kubernetes API for the cluster to deploy the application to. Use the https://kubernetes.default.svc URL to deploy to the same cluster where Argo CD is deployed.

When you create an application, Argo CD retrieves the contents of the repository. In the exercises of this course, applications are YAML files that describe application resources. Argo CD supports other ways to define applications, including the following technologies:

• Kustomize

• Helm charts

• Jsonnet

• Other configuration management tools that additional plug-ins support

Although you can usually create resources in any order and ultimately resources reach the intended state, in some cases creating resources can require further work. For example, you must create a namespace before creating the resources in the namespace. With custom resources, you usually need to wait until their operator installs the custom resource definitions.

You can use the argocd.argoproj.io/sync-wave annotation on resources to help with dependencies. The annotation value must be a positive or negative integer or zero. Resources without the annotation default to 0.

A group of resources with the same annotation is a sync wave.

Argo CD synchronizes sync waves in order, starting with sync waves with lower numbers. Argo CD synchronizes a sync wave only when all lower sync waves are healthy.

Argo CD performs a dry run before synchronizations. When deploying custom resources, the synchronization continues only if the corresponding custom resource definitions exist when Argo CD performs the dry run. To prevent this problem, add the argocd.argoproj.io/sync-options annotation to these resources by adding the SkipDryRunOnMissingResource=true option.

For example, the following manifest defines an operator subscription and a custom resource:

---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: example-operator
...output omitted...
---
kind: ExampleCustomResource
metadata:
  name: example
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: "1"
...output omitted...

Because the subscription does not have an explicit sync wave, Argo CD considers the subscription to be part of the 0 sync wave. Because the custom resource belongs to the 1 sync wave, Argo CD creates the custom resource only when the subscription is healthy. The sync options also exclude the custom resource from the dry run.

However, the previous example might still fail. The next section describes additional issues with deploying complex applications.

Although organizations often use Argo CD to deploy applications, cluster administrators can also use the default Argo CD instance to administer a cluster. By using Argo CD, administrators can customize authentication or deploy operators with all the benefits of GitOps.

When administering a cluster with Argo CD, you might need to patch existing resources, which is an uncommon scenario when deploying applications.

For example, cluster authentication uses the cluster resource of the OAuth type. You must edit this resource instead of creating other resources.

Argo CD can use the server-side apply process to edit existing resources. The server-side apply process is an alternative to the client-side apply process that kubectl apply and other processes use by default.

With client-side apply, commands such as kubectl apply submit the full definition of a resource to the API server. The API server overwrites the entire resource with the submitted definition.

With server-side apply, API clients submit partial resource definitions, and the API server updates only the submitted parts. Then, your application definitions can include only your changes and exclude parts of the resource that you do not want to manage.

For patching, use the ServerSideApply=true sync option. Also, add the Validate=false sync option to disable validation, because the partial resource definition might not be a valid complete resource definition.

The following manifest shows how to patch a resource:

apiVersion: operator.openshift.io/v1
kind: Console
metadata:
  name: cluster
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true,Validate=false
spec:
  customization:
    customProductName: Production

This specification does not contain all the fields that a full console specification requires. The Validate=false sync option ensures that even if the partial definition lacks mandatory fields, Argo CD submits the change.

This change is less likely to interfere with other changes to the cluster such as cluster updates.