Openshift Gitops Access Control — Argo CD Gitops controller

12 min readSep 10, 2023

Kubernetes with extensions built over itself, is now facilitating end to end application development and delivery . With rapid growth of Kubernetes ecosystem, we can feel overwhelmed by range of product and its configuration.

Declarative configuration is one of the biggest selling point of Kubernetes, declaring configuration and letting Kubernetes take care of reconciliation process. YAML (data serialization language) file which is used to define and configure Kubernetes object - its management can very quickly go out of hand.

Gitops paradigm has brought much structure to the management of Kubernetes object. Gitops principle align git as the single source of truth. The Kubernetes manifest can then be versioned and changes to manifest tracked. Also providing simple pathways to rollback.

GitOps Principles v1.0.0 <— https://opengitops.dev/
- Declarative
- Versioned and Immutable
- Pulled Automatically
- Continuously Reconciled

There are different Gitops controller available in the system. These controller implement the core principles mentioned above. Argo CD is one of the prominent Gitops controller.

In this article we will see how we can use Openshift Gitops (Red Hat build of ArgoCD) and manifests available on Github to create Openshift(Red Hat build of Kubernetes) objects. Then we will see the resources involved both in Openshift & Openshift Gitops and how we can configure access control on Gitops controller and User(in next article).

Index:

- Quickstart with Openshift Gitops
  - Installing Openshift Gitops
  - Deploying application manifest using Argo CD

- Resources involved
  - Openshift
  - Openshift Gitops
 
- Actions Involved

- Understanding Access Control on Gitops Controller
  - Restricting control using Role and ClusterRole
  - Restricting access at Namespace level
    - Example Restriction at Namespace level
    - Access over different namespace

- Conclusion

Quickstart with Openshift Gitops

We will start with installing Openshift Gitops operator and deploying a sample application. Once it is successful we can dig deeper and try to understand the various components involved in access management.

Installing Openshift Gitops

We can deploy Openshift Gitops using Operator hub and deploying it over Openshift is pretty straight forward.

Go to OperatorHub and search for Openshift Gitops

2. Install Openshift Gitops Operator in openshift-operators namespace

3. Once installation is successful, an instance of Argo CD is created within openshift-gitops namespace with cluster config privileges

rishabh@gitops:> oc get csv openshift-gitops-operator.v1.7.2 -o yaml | grep -A1 ARGOCD_CLUSTER_CONFIG_NAMESPACES
                - name: ARGOCD_CLUSTER_CONFIG_NAMESPACES
                  value: openshift-gitops
rishabh@gitops:>

ARGOCD_CLUSTER_CONFIG_NAMESPACES is used to configure namespaces with cluster config privileges. These privileges are not exhaustive but limited and additional roles can be added on need basis.

4. Check if all the pods are running in openshift-gitops namespace

Deploying application manifest using Argo CD

We can now deploy a sample application using installed Argo CD instance. First we need to extract the admin password from openshift-gitops-cluster secret.

rishabh@gitops:> oc get secret  openshift-gitops-cluster -o yaml -n openshift-gitops | grep admin.password | awk '{print $2}'  | base64 -d ; echo ""
<password..truncated>

Access the ArgoCD instance using the console link or from the Networking -> Route section

Login using the admin credential, extracted in previous step
Create a pricelist namespace

rishabh@gitops:> oc new-project pricelist
Already on project "pricelist" on server "https://api.xx.xx.xx.xx:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

rishabh@gitops:>

Label the namespace argocd.argoproj.io/managed-by=openshift-gitops. This will allow ArgoCD deployed in openshift-gitops namespace to manage application in pricelist namespace.

rishabh@gitops:> oc label namespace pricelist argocd.argoproj.io/managed-by=openshift-gitops
namespace/pricelist labeled
rishabh@gitops:>

Create a new ArgoCD Application with following configuration :

We will use this github repository to deploy a pricelist application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-application
spec:
  destination:
    name: ''
    namespace: pricelist
    server: 'https://kubernetes.default.svc'
  source:
    path: pricelist-bare
    repoURL: 'https://github.com/rishabhsvats/gitops-examples'
    targetRevision: HEAD
  project: default

rishabh@gitops:> oc get application -n openshift-gitops
No resources found in openshift-gitops namespace.
rishabh@gitops:> oc apply -f pricelist.yaml -n openshift-gitops
application.argoproj.io/sample-application created
rishabh@gitops:> oc get application -n openshift-gitops
NAME                 SYNC STATUS   HEALTH STATUS
sample-application   OutOfSync     Missing
rishabh@gitops:>

Sync the ArgoCD Application

rishabh@gitops:> argocd app sync openshift-gitops/sample-application
TIMESTAMP                  GROUP                     KIND              NAMESPACE                  NAME    STATUS    HEALTH            HOOK  MESSAGE
2023-09-01T14:11:04+05:30   apps               Deployment              pricelist             pricelist  OutOfSync  Missing                  
2023-09-01T14:11:04+05:30  batch                      Job              pricelist  pricelist-postdeploy  OutOfSync  Missing                  
2023-09-01T14:11:04+05:30  route.openshift.io       Route              pricelist       pricelist-route  OutOfSync  Missing                  
2023-09-01T14:11:04+05:30                      PersistentVolumeClaim   pricelist      pricelist-db-pvc    Synced   Healthy                  
2023-09-01T14:11:04+05:30                         Service              pricelist                 mysql  OutOfSync  Missing                  
2023-09-01T14:11:04+05:30                         Service              pricelist             pricelist  OutOfSync  Missing                  
2023-09-01T14:11:04+05:30   apps               Deployment              pricelist                 mysql    Synced   Progressing              
......... Trunctated

Name:               openshift-gitops/sample-application
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          pricelist
URL:                https://openshift-gitops-server-openshift-gitops.apps.xx.xx.xxxx.xx/applications/openshift-gitops/sample-application
Repo:               https://github.com/rishabhsvats/gitops-examples
Target:             HEAD
Path:               pricelist-bare
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from HEAD (261055d)
Health Status:      Healthy

Operation:          Sync
Sync Revision:      261055d9083bf8cd25edd4d0e63fd94e6ecc699f
Phase:              Succeeded
Start:              2023-09-01 13:59:08 +0530 IST
Finished:           2023-09-01 13:59:33 +0530 IST
Duration:           25s
Message:            successfully synced (all tasks run)

GROUP               KIND                   NAMESPACE  NAME                  STATUS     HEALTH   HOOK  MESSAGE
                    PersistentVolumeClaim  pricelist  pricelist-db-pvc      Synced     Healthy        persistentvolumeclaim/pricelist-db-pvc created
apps                Deployment             pricelist  mysql                 Synced     Healthy        deployment.apps/mysql created
                    Service                pricelist  mysql                 Synced     Healthy        service/mysql created
apps                Deployment             pricelist  pricelist             Synced     Healthy        deployment.apps/pricelist created
                    Service                pricelist  pricelist             Synced     Healthy        service/pricelist created
batch               Job                    pricelist  pricelist-postdeploy  Synced     Healthy        job.batch/pricelist-postdeploy created
route.openshift.io  Route                  pricelist  pricelist-route       OutOfSync  Healthy        route.route.openshift.io/pricelist-route created
rishabh@gitops:>

Review all the resources created in pricelist namespace.

rishabh@gitops:> oc get all -n pricelist
NAME                                READY   STATUS      RESTARTS   AGE
pod/mysql-6bfc5d869c-l7h98          1/1     Running     0          6m36s
pod/pricelist-547c64fc8d-jtclj      1/1     Running     0          6m23s
pod/pricelist-postdeploy--1-ckznz   0/1     Completed   0          6m17s

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql       ClusterIP   172.30.166.221   <none>        3306/TCP   6m25s
service/pricelist   ClusterIP   172.30.23.136    <none>        8080/TCP   6m19s

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql       1/1     1            1           6m37s
deployment.apps/pricelist   1/1     1            1           6m24s

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-6bfc5d869c       1         1         1       6m37s
replicaset.apps/pricelist-547c64fc8d   1         1         1       6m24s

NAME                             COMPLETIONS   DURATION   AGE
job.batch/pricelist-postdeploy   1/1           5s         6m19s

NAME                                       HOST/PORT                                            PATH   SERVICES    PORT   TERMINATION   WILDCARD
route.route.openshift.io/pricelist-route   pricelist-route-pricelist.apps.xx.xx.xxxx.xx          pricelist   8080                 None
rishabh@gitops:>

Access the application

Resources involved

In the previous section we saw how we can deploy our Kubernetes manifests using Openshift Gitops. The user experience of Openshift Gitops is really amazing and it brings in the features of git.

Now if you want to geek about access control and get deeper understanding, then following section is for you. There are different resources that are involved behind scene. Lets first look into those at Openshift and Openshift Gitops level.

Openshift

Openshift resources are Openshift and Kubernetes objects — like Deployment, Services, Route, Namespaces etc.

We can view the list of all resources in Openshift cluster using oc api-resources

Openshift Gitops

At Openshift Gitops level following are few commonly used resources:

clusters, projects, applications, applicationsets, repositories

These resources are the Argo CD primitves which help us to deploy our application manifest ( available in specific repositories ) to our Openshift clusters .

Access control on Openshift Gitops (Red Hat build of Argo CD) will be discussed in upcoming article.

Actions involved

We as an Openshift or Openshift Gitops user can perform different actions on above mentioned resources. These actions are the entry point to limit user/controller access. Broadly both Openshift and Openshift Gitops has similar action like: get, create, update, delete etc.

In further section we will see how Openshift Gitops operator is designed to limit access of Gitops controller on Kubernetes and Openshift object.

Understanding Access Control on Gitops Controller

Now that we know different kind of resources and actions available on them, we can think about how Openshift Gitops manage access of Gitops Controller.

Gitops controller is the reconciliation agent of Argo CD , which reconciles the current application state in Openshift with application created within Argo CD. It is crucial to think deeply about the access control of Gitops controller, as extended access would allow unwanted control of Argo CD over Openshift cluster. And broad access over Argo CD can expose backdoor access to bad actors.

So how does Openshift Gitops operator controls access of Gitops Controller. This can be categorized into 2 section :

Restricting control using Role and ClusterRole
Restricting access at Namespace level

Restricting control using Role and ClusterRole

Openshift Gitops operator control the access of Argo CD controller using the ClusterRole , Role and its binding with the Argo CD controller’s service account.

All the cluster level access is controlled using the ClusterRole and access within a specific namespace is controlled by Role .

kind: Role is created by Openshift Gitops operator, when Argo CD instance is created with name as {instance-name}-argocd-application-controller.


rishabh@gitops:> oc get role -n openshift-gitops | grep argocd-application-controller
openshift-gitops-argocd-application-controller   2023-09-01T06:27:17Z
rishabh@gitops:> 

# ClusterRole for cluster level access
rishabh@gitops:> oc get clusterrole | grep argocd-application-controller
openshift-gitops-openshift-gitops-argocd-application-controller             2023-09-01T06:27:17Z
rishabh@gitops:>

So this openshift-gitops-argocd-application-controller role has access to verbs for specific apiGroups — like following section provides access to apps apiGroups.

- apiGroups:
  - apps
  resources:
  - daemonsets
  - deployments
  - deployments/rollback
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  - statefulsets/scale
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update

Ultimately the rolebindings and clusterrolebindings map the role to service account of gitops controller.

rishabh@gitops:> oc get rolebindings -n openshift-gitops | grep argocd-application-controller
openshift-gitops-argocd-application-controller   Role/openshift-gitops-argocd-application-controller   6d3h
rishabh@gitops:> 
rishabh@gitops:> oc get clusterrolebindings | grep argocd-application-controller
openshift-gitops-openshift-gitops-argocd-application-controller             ClusterRole/openshift-gitops-openshift-gitops-argocd-application-controller             6d3h
rishabh@gitops:>

The default role has access to certain apiGroups andResourceQuotas apiGroup is not one of them. In following section I will show you an example to provide access on ResourceQuotas apiGroups.

Let us first try to deploy pricelist application manifest like earlier in this article, but this time with addtional ResourceQuotas object. The Application object will be like following:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pricelist-with-resourcequota
spec:
  destination:
    name: ''
    namespace: pricelist
    server: 'https://kubernetes.default.svc'
  source:
    path: pricelist
    repoURL: 'https://github.com/rishabhsvats/gitops-examples'
    targetRevision: HEAD
  project: default

When we apply above Application object we will get following error:

Permission denied on ResourceQuota object

Permission for resourcequotas is denied to openshift-gitops-argocd-application-controller service account. It is because the service account does not has the role to create a resourcequota object. To provide access over resourcequota object we can perform following steps:

rishabh@gitops:> oc get rolebindings | grep argocd-application-controller
openshift-gitops-argocd-application-controller   Role/openshift-gitops-argocd-application-controller   8s

We can create a Role with complete access on resourcequotas like below:


#Creating role with complete access over resourcequotas and limitranges
rishabh@gitops:> cat quota-limit-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: quota-limit-role
rules:
- apiGroups: [""] 
  resources: ["resourcequotas", "limitranges"]
  verbs: ["*"]
rishabh@gitops:>
rishabh@gitops:> oc apply -f quota-limit-role.yaml -n pricelist
role.rbac.authorization.k8s.io/quota-limit-role created

We can then map the role created in previous step to service account of gitops controller like below using RoleBinding :

#Applying role binding with gitops controller installed in openshift-gitops namespace
# This role binding allows Service Account to create resource quotas and limit ranges
rishabh@gitops:> cat quota-limit-rolebinding.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: quota-limit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: quota-limit-role
subjects:
- kind: ServiceAccount
  name: openshift-gitops-argocd-application-controller
  namespace: openshift-gitops
rishabh@gitops:> 
rishabh@gitops:> 
rishabh@gitops:> 
rishabh@gitops:> oc apply -f quota-limit-rolebinding.yaml -n pricelist
rolebinding.rbac.authorization.k8s.io/quota-limit created
rishabh@gitops:>

Once the service account has required access, the resourcequota is synced successfully.

Restricting access at Namespace level

Openshift Gitops operator restrict the installed Argo CD instance to its own specific namespace, that is the namespace where it is installed. Using the Openshift gitops operator we can create Argo CD instance for each namespace. That way Argo CD instance will have access to its own namespace only. But then we can have a common requirement to install a single Argo CD instance and manage multiple namespaces with same instance.

Example Restriction at Namespace level

For this example I have created a sample namespace namespace-test-project . And now we will create a new instance of Argo CD within the same namespace through : Installed Operators → Red Hat OpenShift GitOps → Argo CD → Create Argo CD → Create with default configuration.

When we create Argo CD instance in a specific namespace, Openshift Gitops will only create namespace level role. The Role provide the list of access to specific apiGroups. In case additional access over specific apiGroups object is required, we can add the Role and RoleBinding as mentioned in previous section.

rishabh@gitops:> oc get role | grep application-controller
namespace-test-project-gitops-argocd-application-controller   2023-09-08T09:08:13Z
rishabh@gitops:>

Access the Argo CD server using the ${argocd-instance-name}-server route within namespace-test-project namespace. The admin password can be extracted like before.

rishabh@gitops:> oc get secret  ${argocd-instance-name}-cluster -o yaml -n namespace-test-project | grep admin.password | awk '{print $2}'  | base64 -d ; echo ""
<password truncated>
rishabh@gitops:>

In cluster config we can see that the Argo CD instance is restricted to namespace-test-project namespace. Using this instance the Openshift resources can only be created in namespace-test-project namespace.

Access over different namespace

In case we want to manage any other namespace (lets say we call it controlled-by-namespace-test-project)using Argo CD instance installed in namespace-test-project , we can use argocd.argoproj.io/managed-by label to give namespace-test-project access over controlled-by-namespace-test-project .

The label will create the Role and RoleBinding object to provide the gitops controller of namespace-test-project access over controlled-by-namespace-test-project .


#Namespace object with argocd.argoproj.io/managed-by label
rishabh@gitops:> cat controlled-by-namespace-test-project.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: controlled-by-namespace-test-project
  labels:
    argocd.argoproj.io/managed-by: namespace-test-project
rishabh@gitops:> 

#Creating the namespace
rishabh@gitops:> oc apply -f controlled-by-namespace-test-project.yaml 
namespace/controlled-by-namespace-test-project created
rishabh@gitops:> 

# Role object created within controlled-by-namespace-test-project namespace
rishabh@gitops:> oc get role -n controlled-by-namespace-test-project
NAME                                                          CREATED AT
namespace-test-project-gitops-argocd-application-controller   2023-09-08T09:44:40Z
namespace-test-project-gitops-argocd-grafana                  2023-09-08T09:44:40Z
namespace-test-project-gitops-argocd-redis                    2023-09-08T09:44:40Z
namespace-test-project-gitops-argocd-server                   2023-09-08T09:44:40Z
rishabh@gitops:>

# RoleBinding object created within controlled-by-namespace-test-project namespace
rishabh@gitops:> oc get rolebinding -n controlled-by-namespace-test-project | grep argocd-application-controller
namespace-test-project-gitops-argocd-application-controller   Role/namespace-test-project-gitops-argocd-application-controller   3m28s
rishabh@gitops:>

Also if we review cluster config within Argo CD , we can see the controlled-by-namespace-test-project is now part of namespaces configuration.

Conclusion

In this article we saw how Openshift Gitops Operator manage access control on Openshift resources. Even though Openshift Gitops Operator performs most of the access control for us, this behind the scene understanding could be helpful to manage your Argo CD instances much more securely.

To followup with this article I will be writing about how we can control access of Argo CD users over Argo CD(Openshift Gitops) resources.