Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

By default, the Red Hat OpenShift GitOps operator creates an Argo CD instance in the openshift-gitops namespace. This Argo CD instance has permissions to manage cluster-wide resources. Thus, Red Hat recommends using this instance only for administration purposes, and to create other, less privileged instances for regular application deployment.

To deploy regular applications, you can deploy new Argo CD instances. By default, the new Argo CD instances have permissions to manage resources only in the namespace where they are deployed.

To create an Argo CD instance, you can create a YAML file as follows:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: argocd-name
  namespace: argocd-ns
spec:
...output omitted...

You can allow an Argo CD instance to manage resources in other namespaces than where it is deployed by adding the argocd.argoproj.io/managed-by label to the namespace. The following example allows the Argo CD instance in the argocd-ns namespace to manage the resources in the managed-ns namespace.

[user@host ~]$ oc label namespace managed-ns \
  argocd.argoproj.io/managed-by=argocd-ns

The following diagram shows an example of using an Argo CD instance to manage resources in other namespaces.

In this diagram, a cluster administrator creates an Argo CD instance in the argocd-ns namespace. Thus, the Argo CD instance has permissions to manage resources only in that namespace.

The cluster administrator includes the argocd.argoproj.io/managed-by label in the managed-ns namespace. Thus, the Argo CD instance can manage resources in that namespace.

Developer users need only read permission in the managed-ns namespace for troubleshooting, according to the principle of least privilege, because they would be managing application resources in their projects by using Git.

Argo CD can both use local users and integrate with other identity providers.

The OpenShift GitOps operator configures the default Argo CD instance to use Dex. Dex is an identity service that uses the OpenID connect protocol to connect Argo CD with the identity provider.

The default Argo CD instance in the openshift-gitops namespace has permission to manage cluster-wide resources. You can log in with write permission to this instance only by using the local admin user with the credentials in the openshift-gitops-cluster secret in the openshift-gitops namespace, or by using an OpenShift user from the cluster-admins group in OpenShift.

You can also configure the Argo CD instances to use Keycloak instead of Dex. For example, use the groups in Keycloak to determine the privileges in Argo CD. Keycloak acts as an identity broker between the Argo CD instance and OpenShift.

...output omitted...
spec:
  sso:
    provider: dex
    dex:
      openShiftOAuth: true
...output omitted...

By default, only the local Argo CD admin user and the OpenShift users in the cluster-admins group can log in to the default Argo CD instance with write permission. Other OpenShift users have only read permission. However, you can change the user and group level access by configuring the RBAC section in the Argo CD custom resource.

Argo CD comes with the following predefined roles:

For example, the default Argo CD instance includes the following RBAC configuration:

...output omitted...
spec:
  ...output omitted...
  rbac:
    defaultPolicy: ""
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
    scopes: '[groups]'
...output omitted...

Thus, for OpenShift users in the system:cluster-admins and cluster-admins groups, the predefined role:admin role is assigned by default.

To configure the Argo CD instance to assign the predefined role:readonly role to the users in the cluster-readers group, you can use the following RBAC configuration:

...output omitted...
spec:
  ...output omitted...
  rbac:
    defaultPolicy: ""
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
      g, cluster-readers, role:readonly
    scopes: '[groups]'
...output omitted...

You can use the defaultPolicy field to define a policy for other users that do not conform to an existing policy.

p, role/user/group, resource, action, target

...output omitted...
spec:
  ...output omitted...
  rbac:
    defaultPolicy: ''
    policy: |
      g, project-devs, role:project-devs
      p, role:project-devs, applications, get, */*, allow
      p, role:project-devs, projects, get, *, allow
      p, role:project-devs, clusters, get, *, allow
      g, project-admins, role:project-admins
      p, role:project-admins, applications, *, */*, allow
      p, role:project-admins, projects, get, *, allow
      p, role:project-admins, clusters, get, *, allow
    scopes: '[groups]'
...output omitted...

If you give a user access to a Git repository, then the user has access to all the files in the repository. You cannot allow users to read or write only in certain paths within a repository. However, some products that build on top of Git, such as GitHub or GitLab, add features to improve protecting specific paths within a repository.

When setting up Git workflows for GitOps, you can either create only one repository for all your files, which is called a monorepo environment, or create repositories for different concerns, which is called a polyrepo environment.

In a monorepo environment, all the application sources, container files, Kubernetes manifests, and policy manifests are in a single Git repository.

A monorepo environment offers a centralized location for all the configuration changes. Thus, tracking, testing, and releasing processes are limited to a single Git repository. Moreover, having all your code in a single repository simplifies managing dependencies between your applications. However, users with access to the Git repository have access to all the code in the projects, which can lead to security issues.

From an Argo CD perspective, you can either treat the entire monorepo as a single application, to deploy changes across various services or components concurrently, or use paths to create several applications from a single repository. With the second option, you can use different paths for the application files and for the application configurations, such as network policies or quotas. This separation prevents application configuration changes from triggering unnecessary application rebuilds.

Another option when setting up Git workflows for GitOps is to separate the concerns by repository. A polyrepo environment enables different designs depending on your company needs.

For example, a polyrepo environment has several Git repositories for applications and configurations. Red Hat recommends using different repositories for application files and for application configurations, to prevent unnecessary application rebuilds. Independent lifecycles for application code and application configuration, along with specific approval processes, ensure smoother continuous integration. However, coordinating dependencies across repositories requires extra effort.

From an Argo CD perspective, you can create applications from the various repositories in a polyrepo environment. However, a polyrepo environment requires more complex configuration than a monorepo environment to manage the dependencies between repositories.

To prevent security issues, you must ensure encryption or other protection of sensitive data or secrets. Sensitive data and secrets include passwords, APIs, encryption keys, SSH keys, tokens, or other digital credentials that allow users and applications access to other sensitive data, systems, and services.

Because Git users with access to the repository have access to all the files in the repository, they have access to any sensitive data that is stored in the repository. Thus, storing your sensitive data in Git repositories can introduce security concerns.

You can manually store your sensitive data in secret objects, outside a Git repository. However, if you do not correctly configure RBAC rules on your cluster, then anyone with API or etcd access can retrieve or modify a secret. Moreover, a user with permission to create a pod in a namespace can read any secret in that namespace. By default, cluster administrators can see the secrets of all users.

A mitigation is to use third-party secret management tools. Red Hat recommends using these secret management tools to mitigate security issues with features such as credential encryption, credential rotation, and short-lived tokens. Both commercial and open source solutions exist for this purpose.

For example, you can use the Secrets Store CSI Driver operator to manage multiple secrets, keys, and certificates. This driver enables Kubernetes pods to mount secrets that are stored in external secret management systems such as Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault as volumes. This driver enhances security by centralizing secret management outside the cluster and allowing applications to access these secrets seamlessly through the Kubernetes API. The operator simplifies integrating and managing external secrets within Kubernetes deployments, to handle sensitive information more securely and efficiently.

When you deploy an application to production, even with thorough testing you cannot predict or prevent all potential issues. Also, deploying an application requires actions before, during, and after application synchronization, and if that process fails.

For any production issues, you can roll back to known stable states the small increments of your application that CI/CD practices release. Increasing the release size also results in larger rollbacks, and implies more risks.

Argo CD provides some of these features for deploying your application.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: schema-migrate-database
  annotations:
    argocd.argoproj.io/hook: PreSync
...output omitted...

apiVersion: batch/v1
kind: Job
metadata:
  generateName: integration-test-app
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
...output omitted...