Design Light Weight CI/CD on kubernetes using Argo Workflows

Argo Workflow
  1. Define a workflow where each step from workflow is separate container.
  2. Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a graph (DAG).
  3. Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.
  4. easily Run CI/CD pipelines natively on Kubernetes without configuring complex software development products/tools.
  1. Connect to k8s cluster and create a argo namespace in k8s cluster.
devopsguy@devopsguyvm:~$ k create ns argo
namespace/argo created
devopsguy@devopsguyvm:~/argo-workflow$ wget https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/install.yaml
--2022-08-02 12:35:54-- https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/install.yaml
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/100982449/5fc2d9b4-3b43-4744-9eea-2379e860ff58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220802T070554Z&X-Amz-Expires=300&X-Amz-Signature=fd214a455df46ded262b1096949f12094b81c863d08a44e469ef3c78c5e879d6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=100982449&response-content-disposition=attachment%3B%20filename%3Dinstall.yaml&response-content-type=application%2Foctet-stream [following]
--2022-08-02 12:35:54-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/100982449/5fc2d9b4-3b43-4744-9eea-2379e860ff58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220802T070554Z&X-Amz-Expires=300&X-Amz-Signature=fd214a455df46ded262b1096949f12094b81c863d08a44e469ef3c78c5e879d6&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=100982449&response-content-disposition=attachment%3B%20filename%3Dinstall.yaml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29105 (28K) [application/octet-stream]
Saving to: ‘install.yaml’
install.yaml 100%[========================================================================================>] 28.42K --.-KB/s in 0.07s2022-08-02 12:35:55 (428 KB/s) - ‘install.yaml’ saved [29105/29105]
devopsguy@devopsguyvm:~/argo-workflow$ k apply -f install.yaml
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtaskresults.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io configured
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role configured
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role configured
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
configmap/workflow-controller-configmap created
service/argo-server created
service/workflow-controller-metrics created
priorityclass.scheduling.k8s.io/workflow-controller unchanged
deployment.apps/argo-server created
deployment.apps/workflow-controller created
-----------------------------------------------------------------devopsguy@devopsguyvm:~/argo-workflow$ k get pod -n argo
NAME READY STATUS RESTARTS AGE
workflow-controller-75bddfc66f-6fmnv 1/1 Running 0 2m49s
argo-server-6bd44d75f7-wlfzr 1/1 Running 0 2m49s
apiVersion: v1
kind: Service
metadata:
name: argo-server
spec:
type: NodePort
selector:
app: argo-server
ports:
- port: 2746
targetPort: 2746
nodePort: 30008
  1. Open argo workflow UI -> click on workflow ->click on SUBMIT NEW WORKFLOW -> Select any workflow template you have Or edit and Paste below template -> click on submit.
  2. directly apply below yaml file using kubectl on user namespace.
apiVersion: argoproj.io/v1alpha1
kind: Workflow # new type of k8s spec
metadata:
generateName: hello-world # name of the workflow spec
spec:
entrypoint: whalesay # invoke the whalesay template
templates:
— name: whalesay # name of the template
container:
image: docker/whalesay
command: [cowsay]
args: ["hello world"]
Argo workflow componets
devopsguy@devopsguyvm:~/argocd$ k get pod |grep -i hello-worldhello-world                                     0/2     Completed   0          101s
devopsguy@devopsguyvm:~/argocd$ k logs -f hello-world
_____________
< hello world >
-------------
\
\
\
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
# Download the binary
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.3.8/argo-linux-amd64.gz
# Unzip
gunzip argo-linux-amd64.gz
# Make binary executable
chmod +x argo-linux-amd64
# Move binary to path
mv ./argo-linux-amd64 /usr/local/bin/argo
# Test installation
argo version
argo auth token --kubeconfig .kube/kubeconfig
devopsguy@devopsguyvm:~/argocd/argo-workflows-demo/workflows$ k exec -it argo-server-6bd44d75f7-wlfzr argo auth token -n argo
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImZFdkRSQjBOc1hYQ2JyVVZTM0ZIc1kySnN5R2V2==============================........
#kubectl create role argo-role --verb=list,update --resource=workflows.argoproj.io
#kubectl create sa argo-sa
#kubectl create rolebinding argo --role=argo-role --serviceaccount=argo:argo-sa
SECRET=$(kubectl get sa jenkins -o=jsonpath='{.secrets[0].name}')
ARGO_TOKEN="Bearer $(kubectl get secret $SECRET -o=jsonpath='{.data.token}' | base64 --decode)"
echo $ARGO_TOKEN
Bearer ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltS...
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-submittable
spec:
entrypoint: whalesay-template # Fields other than "arguments" and "templates" not supported in v2.4 - v2.6
arguments:
parameters:
- name: message
value: hello world
templates:
- name: whalesay-template
inputs:
parameters:
- name: message
container:
image: docker/whalesay
command: [cowsay]
args: ["Hello World!"]

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: workflow-template-hello-world-
spec:
workflowTemplateRef:
name: workflow-template-submittable
apiVersion: v1
kind: Secret
metadata:
labels:
app: postgres
name: argo-postgres-config
stringData:
password: password
username: postgres
type: Opaque
-------------------------------------------------------
apiVersion: v1
kind: ConfigMap
metadata:
name: workflow-controller-configmap
data:
persistence: |
connectionPool:
maxIdleConns: 100
maxOpenConns: 0
connMaxLifetime: 0s
nodeStatusOffLoad: true
archive: true
archiveTTL: 7d
postgresql:
host: postgres
port: 5432
database: postgres
tableName: argo_workflows
userNameSecret:
name: argo-postgres-config
key: username
passwordSecret:
name: argo-postgres-config
key: password
retentionPolicy: |
completed: 10
failed: 3
errored: 3
- name: checkout-git
inputs:
artifacts:
- name: git-repo
path: /src/git
git:
repo: "{{workflow.parameters.repo}}"
revision: "{{workflow.parameters.revision}}"
sshPrivateKeySecret:
name: git-creds
key: id_rsa
metadata:
labels:
app: argo
container:
image: alpine/git
command: [sh, -c]
args: ["cd /src && git rev-parse --short HEAD > /tmp/git-commit"]
outputs:
artifacts:
- name: source
path: /src
parameters:
- name: tag
valueFrom:
path: /tmp/git-commit
inputs:
artifacts:
- name: git-repo
path: /src
parameters:
- name: image-tag
metadata:
labels:
app: argo
container:
image: docker:18.09
workingDir: /src
command: [sh, -c]
args: ["until docker ps; do sleep 1; done; cd /src \
&& docker build . -t {{workflow.parameters.image-name-backend}}:{{inputs.parameters.image-tag}} && docker push {{workflow.parameters.image-name-backend}}:{{inputs.parameters.image-tag}}"]
env:
- name: DOCKER_HOST
value: 127.0.0.1
volumeMounts:
- name: docker-config
mountPath: /root/.docker/config.json
subPath: .dockerconfigjson
sidecars:
- name: docker-in-docker
image: docker:18.09-dind
securityContext:
privileged: true
mirrorVolumeMounts: true
- name: run-tests
inputs:
artifacts:
- name: git-repo
path: /src
parameters:
- name: image-tag
metadata:
labels:
app: argo
container:
image: bouwe/ansible-kubectl-credstash:0.0.1
workingDir: /src
command: [sh, -c]
args: ["cd /src \
&& ansible-playbook run-backend-test-on-k8s.yml -i environments/backend-test/backend-k8s -e docker_image_tag={{inputs.parameters.image-tag}}"]
- name: deploy-kubernetes
inputs:
artifacts:
- name: git-repo
path: /src
parameters:
- name: image-tag
metadata:
labels:
app: argo
container:
image: bouwe/ansible-kubectl-credstash:0.0.1
workingDir: /src
command: [sh, -c]
args: ["cd /src/ansible \
&& ansible-playbook deploy-backend-to-k8s.yml -i environments/backend/backend-k8s -e docker_image_tag={{inputs.parameters.image-tag}}"]

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Akshay Bobade

Akshay Bobade

I have total 3 Plus years of experience as a Devops engineer and currently dealing with Cloud, Containers, Kubernates and Bigdata technologies.