Open in app

Sign in

Write

Sign in

Design Light Weight CI/CD on kubernetes using Argo Workflows

Akshay Bobade
8 min readAug 2, 2022

--

What is Argo Workflows?

Agro workflow is an Open source project and has open source container-native workflow engine that enables CI/CD pipeline management and the orchestration of parallel jobs on Kubernetes.

Argo Workflow

Argo Workflows is implemented as a Kubernetes Custom Resource Definition(CRD) which allow us to-

  1. Define a workflow where each step from workflow is separate container.
  2. Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a graph (DAG).
  3. Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.
  4. easily Run CI/CD pipelines natively on Kubernetes without configuring complex software development products/tools.

Installation on Kubernetes cluster:

Prerequisite:

Kubernetes cluster (version v.1.18.0 and above)

  1. Connect to k8s cluster and create a argo namespace in k8s cluster.
devopsguy@devopsguyvm:~$ k create ns argo
namespace/argo created

2. get the latest stable released manifest file from official documentation

devopsguy@devopsguyvm:~/argo-workflow$ wget https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/install.yaml
--2022-08-02 12:35:54--  https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/install.yaml
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/100982449/5fc2d9b4-3b43-4744-9eea-2379e860ff58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220802T070554Z&X-Amz-Expires=300&X-Amz-Signature=fd214a455df46ded262b1096949f12094b81c863d08a44e469ef3c78c5e879d6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=100982449&response-content-disposition=attachment%3B%20filename%3Dinstall.yaml&response-content-type=application%2Foctet-stream [following]
--2022-08-02 12:35:54--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/100982449/5fc2d9b4-3b43-4744-9eea-2379e860ff58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220802T070554Z&X-Amz-Expires=300&X-Amz-Signature=fd214a455df46ded262b1096949f12094b81c863d08a44e469ef3c78c5e879d6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=100982449&response-content-disposition=attachment%3B%20filename%3Dinstall.yaml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29105 (28K) [application/octet-stream]
Saving to: ‘install.yaml’install.yaml                               100%[========================================================================================>]  28.42K  --.-KB/s    in 0.07s2022-08-02 12:35:55 (428 KB/s) - ‘install.yaml’ saved [29105/29105]

3. Apply manifest file directly, it will create some custom resource defination(CRD),cluster role, cluster rolebinding, role, rolebinding, Service accounts and deployments.Once deployed check if all Pods are Up and Running and not in argo namespace.

devopsguy@devopsguyvm:~/argo-workflow$ k apply -f install.yaml
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io configured
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role configured
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role configured
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
configmap/workflow-controller-configmap created
service/argo-server created
service/workflow-controller-metrics created
priorityclass.scheduling.k8s.io/workflow-controller unchanged
deployment.apps/argo-server created
deployment.apps/workflow-controller created
-----------------------------------------------------------------devopsguy@devopsguyvm:~/argo-workflow$ k get pod -n argo
NAME                                   READY   STATUS    RESTARTS   AGE
workflow-controller-75bddfc66f-6fmnv   1/1     Running   0          2m49s
argo-server-6bd44d75f7-wlfzr           1/1     Running   0          2m49s

4. Create ingress,LB service or Node port service to access UI. Here I have used Node Port service for it.

apiVersion: v1
kind: Service
metadata:
  name: argo-server
spec:
  type: NodePort
  selector:
    app: argo-server
  ports:
    - port: 2746
      targetPort: 2746
      nodePort: 30008

Once Successful You can access UI using https://<kubernetes-any-node-ip>:30008/ (Please note: in my case I am using Oracle VM Virtual Box and hence I have port-forward Port 30008 to 8089 on my local machine)

How it is working?

This diagram illustrates the core concepts in Argo Workflows. Let’s discuss each concept in details and working.

We have workflow controller and Argo Server this two deployments in argo namespace. Argo Workflow Controller is the controller component for the Argo Workflows engine, which is meant to orchestrate Kubernetes jobs in parallel. Argo Sever is the main Workflow engine.

We can run CI/CD Workflow in any user namespace. There are two ways to apply and run workflows

  1. Open argo workflow UI -> click on workflow ->click on SUBMIT NEW WORKFLOW -> Select any workflow template you have Or edit and Paste below template -> click on submit.
  2. directly apply below yaml file using kubectl on user namespace.
apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world   # name of the workflow spec
spec:
  entrypoint: whalesay          # invoke the whalesay template
  templates:
  — name: whalesay              # name of the template
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]
Argo workflow componets

Once done it will create pod with the name which we mentioned in metadata section of the template and execute all the parts which we defined inside specs section. you can check the pod and logs in user namespace.

devopsguy@devopsguyvm:~/argocd$ k get pod |grep -i hello-worldhello-world                                     0/2     Completed   0          101s
devopsguy@devopsguyvm:~/argocd$ k logs -f hello-world
 _____________
< hello world >
 -------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Manage authentication settings:

There are two ways using which you can authenticate with argo workflows.

1. You can use single sign-on with argo workflow.

This link is explaning how to enabled single sign-on — > https://argoproj.github.io/argo-workflows/argo-server-sso/

2. Using Token based Client authentication

There are multiple ways using which you can generate authentication token

i) Install argo cli on local machine using below commands.

# Download the binary
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/argo-linux-amd64.gz
# Unzip
gunzip argo-linux-amd64.gz
# Make binary executable
chmod +x argo-linux-amd64
# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo
# Test installation
argo version

Once Installation done,run below command by passing cluster kubeconfig file(if not provided it will take default kubeconfig file). it will return token that you can use for authentication.

argo auth token --kubeconfig .kube/kubeconfig

ii)exec into argo-server pod and run argo auth token command from it.

devopsguy@devopsguyvm:~/argocd/argo-workflows-demo/workflows$ k exec -it argo-server-6bd44d75f7-wlfzr argo auth token -n argo
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImZFdkRSQjBOc1hYQ2JyVVZTM0ZIc1kySnN5R2V2==============================........

iii) Above two procedure will create expiry token which will expire after 24 hours. if you want non-expiry token then create sa,role and rolebinding as shown below.

#kubectl create role argo-role --verb=list,update --resource=workflows.argoproj.io
#kubectl create sa argo-sa
#kubectl create rolebinding argo --role=argo-role --serviceaccount=argo:argo-sa

You now need to get a token using below commands.

SECRET=$(kubectl get sa jenkins -o=jsonpath='{.secrets[0].name}')
ARGO_TOKEN="Bearer $(kubectl get secret $SECRET -o=jsonpath='{.data.token}' | base64 --decode)"
echo $ARGO_TOKEN
Bearer ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltS...

Workflow templates:

WorkflowTemplates are definitions of Workflows that live in your cluster. This allows you to create a library of frequently-used templates and reuse them either by submitting them directly (v2.7 and after) or by referencing them from your Workflows.

WorkflowTemplate vs template:

you can create a workflow template which can be stored in your k8s cluster as a CRD. you can use this template to run workflows.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  entrypoint: whalesay-template     # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
  arguments:
    parameters:
      - name: message
        value: hello world
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["Hello World!"]

In workflow yaml inside sepcs -> workflowTemplateRef: define the name of the workflow template you want to use.


apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  workflowTemplateRef:
    name: workflow-template-submittable

Archiving Argo Workflows:

Argo workflow knows how to archive the results of its workflows to a SQL database either we can use mysql or postgresql.Argo Workflows archives workflow state into persistent storage using a Postgres database and store past workflow runs provides you with an accurate record of your past workflow states. You can use them to better understand how your jobs run and where you might be able to improve.

The archive stores only the previous workflow states, not their detailed instance logs. we can store audit logs locally by MinIO,Google cloud GCS and AWS S3 service.

For enabling archive with workflow just create below two configmap and secret and restart the pod.it will start storing workflow state into postgresql database.

apiVersion: v1
kind: Secret
metadata:
  labels:
    app: postgres
  name: argo-postgres-config
stringData:
  password: password
  username: postgres
type: Opaque
-------------------------------------------------------apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 0s
    nodeStatusOffLoad: true
    archive: true
    archiveTTL: 7d
    postgresql:
      host: postgres
      port: 5432
      database: postgres
      tableName: argo_workflows
      userNameSecret:
        name: argo-postgres-config
        key: username
      passwordSecret:
        name: argo-postgres-config
        key: password
  retentionPolicy: |
    completed: 10
    failed: 3
    errored: 3

End to End CI/CD Example:

Step1: Git Checkout

first thing is to checkout git repository to build you docker file and deploy it using ansible script which Placed inside your repository. Argo can access your repository using sshPrivateKeySecret annotation which placed git: section and key is used to access private repository.

- name: checkout-git
    inputs:
      artifacts:
      - name: git-repo
        path: /src/git
        git:
          repo: "{{workflow.parameters.repo}}"
          revision: "{{workflow.parameters.revision}}"
          sshPrivateKeySecret:
            name: git-creds
            key: id_rsa
    metadata:
      labels:
        app: argo
    container:
      image: alpine/git
      command: [sh, -c]
      args: ["cd /src && git rev-parse --short HEAD > /tmp/git-commit"]
    outputs:
      artifacts:
      - name: source
        path: /src
      parameters:
      - name: tag
        valueFrom:
          path: /tmp/git-commit

Step2: Building Docker Images

next step is to build docker file so that you can test inside docker container. You can define sidecars container in your Argo workflow, which will run a Docker daemon so that you can build docker file inside your docker container.in order to push docker image to your private docker repository is done by using the credentials file .dockerconfigjson mounting at root directory.

inputs:
      artifacts:
      - name: git-repo
        path: /src
      parameters:
      - name: image-tag
    metadata:
      labels:
        app: argo
    container:
      image: docker:18.09
      workingDir: /src
      command: [sh, -c]
      args: ["until docker ps; do sleep 1; done; cd /src \ 
        && docker build . -t {{workflow.parameters.image-name-backend}}:{{inputs.parameters.image-tag}} && docker push {{workflow.parameters.image-name-backend}}:{{inputs.parameters.image-tag}}"]
      env:
      - name: DOCKER_HOST
        value: 127.0.0.1
      volumeMounts:
      - name: docker-config
        mountPath: /root/.docker/config.json
        subPath: .dockerconfigjson
    sidecars:
    - name: docker-in-docker
      image: docker:18.09-dind
      securityContext:
        privileged: true
      mirrorVolumeMounts: true

Step 3: Run tests

This part will use ansible to deploy application in kubernetes cluster on which we can run some test cases.

- name: run-tests
    inputs:
      artifacts:
      - name: git-repo
        path: /src
      parameters:
      - name: image-tag
    metadata:
      labels:
        app: argo
    container:
      image: bouwe/ansible-kubectl-credstash:0.0.1
      workingDir: /src
      command: [sh, -c]
      args: ["cd /src \
          && ansible-playbook run-backend-test-on-k8s.yml -i environments/backend-test/backend-k8s -e docker_image_tag={{inputs.parameters.image-tag}}"]

Step 4: Deploy Application to k8s Cluster

This is the CD part in which we will deploy our application to k8s cluster using ansible Playbook.you can pass details to the input tag which git repo you want to use and image tag etc.

- name: deploy-kubernetes
    inputs:
      artifacts:
      - name: git-repo
        path: /src
      parameters:
      - name: image-tag
    metadata:
      labels:
        app: argo
    container:
      image: bouwe/ansible-kubectl-credstash:0.0.1
      workingDir: /src
      command: [sh, -c]
      args: ["cd /src/ansible \
        && ansible-playbook deploy-backend-to-k8s.yml -i environments/backend/backend-k8s -e docker_image_tag={{inputs.parameters.image-tag}}"]

References:

https://argoproj.github.io/argo-workflows/
https://argoproj.github.io/workflows/

Thanks!

--

--

Akshay Bobade

Written by Akshay Bobade

38 Followers

I have total 3 Plus years of experience as a Devops engineer and currently dealing with Cloud, Containers, Kubernates and Bigdata technologies.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams