How we moved from Rundeck to Argo-Workflows

Published in

Israeli Tech Radar

6 min readJan 17, 2022

How does our story begin?

Our task was to find a better alternative to Rundeck.
After a daily discussion with the customer, we chose Argo-workflows as the best alternative for Rundeck

This article will show you how to install Argo-Workflows with integrated Dex OIDC using SAML Provider to connect the Okta app, and how to create a pipeline, Step by Step.

What is Argo-Workflows?
Argo Workflows is an open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD.

So we started researching, understanding, testing what Argo-workflows are and what their strengths and weaknesses are.
We started installing on AWS’s EKS Kubernetes environment and… it Works!

The commands we run the installation Argo-Workflows is:

Start with creating Namespace in name argo.

kubectl create namespace argo

Add Repo for installation with Namespace.

helm repo add https://argoproj.github.io/argo-helm -n argo

Run the Helm-install.

helm install argo-workflows argo/argo-workflows -n argo

Creating a secret in k8s is easy! Here’s how to do it:

kubectl create secret --from-literral=ClientId=<"realclientid"> --from-literral=ClientSecret=<"realClientSecret">

To optimize the Argo-Workflows we needed to connect with our login which is Okta for user management. To create the connection we added Dex Provider with SAML (Security Assertion Markup Language) support.

Then add the Values needed for creating an ingress ALB and SSO for Okta.
After that, add the following Args:

controller:
      containerRuntimeExecutor: k8sapi
server:
      serviceAccount:
        create: true
        name: "argo-server"
        annotations: {
          workflows.argoproj.io/rbac-rule: "'<name-group>' in groups",
          workflows.argoproj.io/rbac-rule-precedence: "1"
        }
      extraArgs: [--auth-mode=sso, --auth-mode=server]
      ingress:
        enabled: true
        annotations: {
          alb.ingress.kubernetes.io/auth-idp-oidc: '{"issuer":"https://account.okta.com","authorizationEndpoint":"https://account.okta.com/oauth2/v1/authorize","tokenEndpoint":"https://account.okta.com/oauth2/v1/token","userInfoEndpoint":"https://account.okta.com/oauth2/v1/userinfo","secretName":"mysecret"}',
          alb.ingress.kubernetes.io/auth-scope: 'openid profile',
          alb.ingress.kubernetes.io/auth-type: oidc,
          alb.ingress.kubernetes.io/group.name: <name-group>,
          alb.ingress.kubernetes.io/healthcheck-path: /<example>,
          alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]',
          alb.ingress.kubernetes.io/load-balancer-name:<load-balancer-name>,
          alb.ingress.kubernetes.io/scheme: "internet-facing",
          alb.ingress.kubernetes.io/ssl-policy: "ELBSecurityPolicy-TLS-1-2-2017-01",
          alb.ingress.kubernetes.io/target-type: ip,
          }hosts: 
          - argo-workflows.<YourDomain>.com
        paths:
          - /
        pathType: Prefix
        tls: 
          - secretName: mysecret
            hosts:
              - argo-workflows.<YourDomain>.com
    
      sso:
        issuer: https://account.okta.com
        sessionExpiry: 240h
        clientId:
          name: myscecte
          key: ClientId
        clientSecret:
          name: myscecte
          key: ClientSecret
        redirectUrl: https://argo-argo-workflows.<YourDomain>.com/dex/callback
        scopes:
          - groups
          - email
          - name
        customGroupClaimName: <name-group>
        rbac:
          enabled: true

Finally, run the command:

helm upgrade argo-workflows argo/argo-workflows -f values.yaml -n argo

Check the Helm is passed in this command: helm list -n argo

Our story evolves into our Workflow … how we work nowadays is PR’s …
pr merge to master — triggers our Argo-Workflow
We have Argo-workflows in Production.

After a lot of effort, paid investment, and Argo-workflows in Production,
We started working on writing our first workflow
Hello-World, we wanted to get smart and look and add alerts to Slack.
(We’ll get back later…)

Dex OIDC? How to use it?

What is Dex OIDC?
Dex is an identity service that uses OpenID Connect to drive authentication for other apps.
Dex acts as a portal to other identity providers through “connectors.” This lets Dex defer authentication to LDAP servers, SAML providers, or established identity providers like GitHub, Google, and Active Directory. Clients write their authentication logic once to talk to Dex, then Dex handles the protocols for a given backend

After research, we realized that in order for us to link the Okta to our Argo-Workflows we needed a provider and the perfect fit for all connectivity is SAML version 2. *

before installing Dex Oidc, do create the Okta app for SAML Provider in the Okta panel.

The guide: https://saml-doc.okta.com/SAML_Docs/Configure-SAML-2.0-for-Org2Org.html

The commands we run the installation Dex is:

Start with creating Namespace in name dex.

kubectl create namespace dex

Add Repo for installation with Namespace.

helm repo add https://charts.dexidp.io -n dex

Run the Helm-install.

helm install dex dex/dex -n dex

After running the commands, and Dex started to run, we have started working on setting up the SAML inside the Dex.

For SAML work with we needs at issuer Okta, we create a CSR file and paste the ca.pem file, which you get after creating SAML app in the Okta panel. The file looks like this:

After that add the values to the values.yaml file like this:

config:
      issuer: http://www.okta.com/<id-saml>
      storage:
        type: memory
      enablePasswordDB: trueconnectors:
    - type: saml
      id: okta
      name: Okta
      config:
        ssoURL:  https://account .okta.com/app/<name-app>/<id-saml>/sso/saml
        ca: path/to/ca.pem
        redirectURI: https://argo-argo-workflows.<YourDomain>.com/dex/callback
        usernameAttr: email
        emailAttr: email
        groupsAttr: group

Finally, run the command:

helm upgrade dex dex/dex -f values.yaml -n dex

Check the Helm is passed in this command: helm list -n dex

Where did the challenges arise?

We started to create a Helm-Chart for the installation.
Then suddenly we noticed that the installation didn’t really work and other files did not materialize properly.

So, we went back to writing our workflows, and an interesting challenge arose: our workflows didn’t work with Workflow Executors Docker. We did some trials and tests with other Workflow Executors like Kubelet, K8sapi, Pns.

At first, we selected Kubelet: we checked and discovered many errors and finally, we found it was a matter of permissions. We tried again and again without success, through the creation of Role and RoleBinding. After several tests, we found that Kubelet only works with ClusterRole and ClusterRoleBinding but it’s not suitable for us in terms of security, so we moved to another Workflow Executors.

Another attempt we tried, without success, was with Pns.

We finally choose K8sapi, because of the level of security and running
of Workflow and also with the beginning of Role and RoleBinding.
We added the Value in Values-file.yaml of Argo-workflows:

controller:
      containerRuntimeExecutor: k8sapi

Evolution of Workflows in our work

First, we created a Workflows generic.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: hello-hello
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: hello        
        templateRef:
          name: workflow-template-whalesay-template
        template:
          name: whalesay-template
        arguments:
          parameters:
          - name: message
            value: "hello-My-World"

After, we created a WorkrflowsTemplate with parameters.
That way, if we want to run many workflows we can run them using one template.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-whalesay-template
spec:
  entrypoint: whalesay-template
  templates:
  - name: whalesay-template
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]

For Notifications, we create a ClusterWorkrflowsTemplate.
Thus, if workflows are running on other namespaces, the alerts system will receive the events and send the required output to Slack.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: cluster-template-exit-handler-slack
spec:
  templates:
  - name: exit-handler
    container:
      image: curlimages/curl
      command: [sh, -c]
      args: [
        "curl -X POST --data-urlencode 'payload={
          \"text\": \{{workflow.name}} finished\",
          \"blocks\": [
            {
              \"type\": \"section\",
              \"text\": {
                \"type\": \"mrkdwn\",
                \"text\": \"Workflow {{"{{workflow.name}} {{workflow.status}}\",
              }
            }
          ]
        }'
        https://hooks.slack.com/services/<webhook>"
      ]

Workflows with steps and notifications now look like this:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: hello-hello
spec: 
  onExit: exit-handler
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: hello        
        templateRef:
          name: workflow-template-whalesay-template
        template:
          name: whalesay-template
        arguments:
          parameters:
          - name: message
            value: "hello-My-World"- name: exit-handler
    steps:
    - - name: exit-handler
        templateRef:
          name: cluster-template-exit-handler-slack
          template: exit-handler
          when: ["{{workflow.status}} != Succeeded"]
          clusterScope: true

If we want to schedule the Workflow, the Workflow will look like that:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
 name: hello-hello
spec:
  schedule: "* * * * *"
  timezone: "Etc/UTC"
  startingDeadlineSeconds: 0        
  concurrencyPolicy: "Replace"       
  successfulJobsHistoryLimit: 4      
  failedJobsHistoryLimit: 4        
  suspend: false                   
  workflowSpec:
  onExit: exit-handler
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: hello        
        templateRef:
          name: workflow-template-whalesay-template
        template:
          name: whalesay-template
        arguments:
          parameters:
          - name: message
            value: "hello-My-World"
  - name: exit-handler
    steps:
    - - name: exit-handler
        templateRef:
          name: cluster-template-exit-handler-slack
          template: exit-handler
          when: ["{{workflow.status}} != Succeeded"]
          clusterScope: true

Conclusions:

Working with Argo-Workflows is interesting and challenging and there have been quite a few challenges and failures. But with a lot of research, we were able to configure the product we wanted.