How to apply policies in Kubernetes using Open Policy Agent (OPA) and Gatekeeper

Why Admission Control Policies?

Using RBAC (role-based access control) use can create roles. Roles has set of Permission defined. Then you Assign this role to users using Rolebinding. When user Performing any action(update,delete,get,list etc) on kubernetes resource, they can do so if they have appropriate role.

  1. Deny a pod from running on the cluster if it has a label “nonprod”.
  2. Deny a Pod running in the cluster has CPU limit set higher than “1G”.
  3. Create a service only it is of type clusterIP.
  1. Working K8s cluster.
root@devopsguyvm:~# wget https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.3/deploy/gatekeeper.yaml
--2022-03-15 14:03:03-- https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.3/deploy/gatekeeper.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23434 (23K) [text/plain]
Saving to: ‘gatekeeper.yaml.1’
gatekeeper.yaml.1 100%[========================================================================================>] 22.88K 90.1KB/s in 0.3s2022-03-15 14:03:08 (90.1 KB/s) - ‘gatekeeper.yaml.1’ saved [23434/23434]root@devopsguyvm:~# kubectl apply -f gatekeeper.yaml
namespace/gatekeeper-system created
serviceaccount/gatekeeper-admin created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/gatekeeper-admin created
role.rbac.authorization.k8s.io/gatekeeper-manager-role created
clusterrole.rbac.authorization.k8s.io/gatekeeper-manager-role created
rolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
secret/gatekeeper-webhook-server-cert created
service/gatekeeper-webhook-service created
Warning: spec.template.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/manager]: deprecated since v1.19, non-functional in v1.25+; use the "seccompProfile" field instead
deployment.apps/gatekeeper-audit created
deployment.apps/gatekeeper-controller-manager created
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
root@devopsguyvm:~#
root@devopsguyvm:~# git clone https://github.com/open-policy-agent/gatekeeper-library.git
Cloning into 'gatekeeper-library'...
remote: Enumerating objects: 2008, done.
remote: Counting objects: 100% (731/731), done.
remote: Compressing objects: 100% (347/347), done.
remote: Total 2008 (delta 467), reused 463 (delta 383), pack-reused 1277
Receiving objects: 100% (2008/2008), 388.13 KiB | 834.00 KiB/s, done.
Resolving deltas: 100% (1105/1105), done.
root@devopsguyvm:~# cd gatekeeper-library
root@devopsguyvm:~/gatekeeper-library# ls
build library LICENSE Makefile mutation NOTICE README.md scripts src test test.shroot@devopsguyvm:~/gatekeeper-library# cd library/
root@devopsguyvm:~/gatekeeper-library/library# ls
general kustomization.yaml pod-security-policy
root@devopsguyvm:~/gatekeeper-library/library# kustomize build | kubectl apply --filename -
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sblockendpointeditdefaultrole created
constrainttemplate.templates.gatekeeper.sh/k8sblocknodeport created
constrainttemplate.templates.gatekeeper.sh/k8scontainerlimits created
constrainttemplate.templates.gatekeeper.sh/k8scontainerratios created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedtags created
constrainttemplate.templates.gatekeeper.sh/k8sexternalips created
constrainttemplate.templates.gatekeeper.sh/k8shttpsonly created
constrainttemplate.templates.gatekeeper.sh/k8simagedigests created
constrainttemplate.templates.gatekeeper.sh/k8spspallowedusers created
constrainttemplate.templates.gatekeeper.sh/k8spspallowprivilegeescalationcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspapparmor created
constrainttemplate.templates.gatekeeper.sh/k8spspcapabilities created
constrainttemplate.templates.gatekeeper.sh/k8spspflexvolumes created
constrainttemplate.templates.gatekeeper.sh/k8spspforbiddensysctls created
constrainttemplate.templates.gatekeeper.sh/k8spspfsgroup created
constrainttemplate.templates.gatekeeper.sh/k8spsphostfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnamespace created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnetworkingports created
constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspprocmount created
constrainttemplate.templates.gatekeeper.sh/k8spspreadonlyrootfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spspseccomp created
constrainttemplate.templates.gatekeeper.sh/k8spspselinuxv2 created
constrainttemplate.templates.gatekeeper.sh/k8spspvolumetypes created
constrainttemplate.templates.gatekeeper.sh/k8sreplicalimits created
constrainttemplate.templates.gatekeeper.sh/k8srequiredannotations created
constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
constrainttemplate.templates.gatekeeper.sh/k8srequiredprobes created
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
constrainttemplate.templates.gatekeeper.sh/k8suniqueserviceselector created
root@devopsguyvm:~# kubectl get pods -n gatekeeper-system
NAME READY STATUS RESTARTS AGE
gatekeeper-controller-manager-765cd6fbf5-9dxkc 1/1 Running 0 4h6m
gatekeeper-audit-68db8cdbc7-wc8ct 1/1 Running 0 4h7m
gatekeeper-controller-manager-765cd6fbf5-9pq9k 1/1 Running 0 4h7m
gatekeeper-controller-manager-765cd6fbf5-sghtg 1/1 Running 0 4h6m
root@devopsguyvm:~# kubectl get crd -n gatekeeper-system
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2022-03-15T08:30:17Z
bgppeers.crd.projectcalico.org 2022-03-15T08:30:17Z
blockaffinities.crd.projectcalico.org 2022-03-15T08:30:18Z
clusterinformations.crd.projectcalico.org 2022-03-15T08:30:18Z
felixconfigurations.crd.projectcalico.org 2022-03-15T08:30:18Z
globalnetworkpolicies.crd.projectcalico.org 2022-03-15T08:30:18Z
globalnetworksets.crd.projectcalico.org 2022-03-15T08:30:19Z

Real time use cases:

1.All the service which should be created in your cluster should not be of type NodePort due to security reasons.

root@devopsguyvm:~/opa-gatekeeper-demo/opa# cat block-node-port.yaml
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockNodePort
metadata:
name: block-node-port
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Service"]
root@devopsguyvm:~/opa-gatekeeper-demo/opa# k apply -f block-node-port.yaml
k8sblocknodeport.constraints.gatekeeper.sh/block-node-port created
root@devopsguyvm:~# kubectl get crd |grep -i k8sblocknodeport.constraints.gatekeeper.sh
k8sblocknodeport.constraints.gatekeeper.sh 2022-03-15T09:11:46Z
root@devopsguyvm:~/opa-gatekeeper-demo/app# cat app.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
selector:
matchLabels:
app: devops-toolkit
template:
metadata:
labels:
app: devops-toolkit
spec:
containers:
- name: devops-toolkit
image: vfarcic/devops-toolkit-series:2.7.0
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
readinessProbe:
httpGet:
path: /
port: 80
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
---apiVersion: v1
kind: Service
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: devops-toolkit
---apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-toolkit
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: devops-toolkit
port:
number: 80
host: acme.com
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl apply -f orig.yaml
deployment.apps/devops-toolkit created
ingress.networking.k8s.io/devops-toolkit created
Error from server ([block-node-port] User is not allowed to create service of type NodePort): error when creating "orig.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [block-node-port] User is not allowed to create service of type NodePort
root@devopsguyvm:~/opa-gatekeeper-demo/opa# cat container-must-have-limits.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
name: container-must-have-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
cpu: "1000m"
memory: "1Gi"
root@devopsguyvm:~/opa-gatekeeper-demo/opa# kubectl apply -f container-must-have-limits.yaml
k8scontainerlimits.constraints.gatekeeper.sh/container-must-have-limits created
root@devopsguyvm:~/opa-gatekeeper-demo/opa# kubectl get crd | grep k8scontainerlimits.constraints.gatekeeper.sh
k8scontainerlimits.constraints.gatekeeper.sh 2022-03-15T09:11:46Z
root@devopsguyvm:~/opa-gatekeeper-demo/app# cat orig.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
selector:
matchLabels:
app: devops-toolkit
template:
metadata:
labels:
app: devops-toolkit
spec:
containers:
- name: devops-toolkit
image: vfarcic/devops-toolkit-series:latest
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
readinessProbe:
httpGet:
path: /
port: 80
---apiVersion: v1
kind: Service
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: devops-toolkit
---apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-toolkit
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: devops-toolkit
port:
number: 80
host: acme.com
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl apply -f orig.yaml
deployment.apps/devops-toolkit created
service/devops-toolkit unchanged
ingress.networking.k8s.io/devops-toolkit unchanged
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl get pod
No resources found in default namespace.
root@devopsguyvm:~/opa-gatekeeper-demo/app# k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 84s deployment-controller Scaled up replica set devops-toolkit-c964cc947 to 1
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl get rs
NAME DESIRED CURRENT READY AGE
devops-toolkit-c964cc947 1 0 0 104s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 27s (x15 over 109s) replicaset-controller Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [container-must-have-limits] container <devops-toolkit> has no resource limits

References:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Akshay Bobade

Akshay Bobade

I have total 3 Plus years of experience as a Devops engineer and currently dealing with Cloud, Containers, Kubernates and Bigdata technologies.