How to apply policies in Kubernetes using Open Policy Agent (OPA) and Gatekeeper
What is OPA?
The Open Policy Agent Gatekeeper project can be leveraged to help enforce policies and strengthen governance in your Kubernetes environment. OPA and Gatekeeper used for Admission Control policies. for example if we want to enforce Policies like all the pod Manifest files has label defined, CPU and Memory limit set etc. Instead of using multiple policies for each component of your stack, you can easily use OPA and make sure that you have one single policy framework that works for most of your use cases.
Why Admission Control Policies?
Using RBAC (role-based access control) use can create roles. Roles has set of Permission defined. Then you Assign this role to users using Rolebinding. When user Performing any action(update,delete,get,list etc) on kubernetes resource, they can do so if they have appropriate role.
Now consider below situation:
- Deny a pod from running on the cluster if it has a label “nonprod”.
- Deny a Pod running in the cluster has CPU limit set higher than “1G”.
- Create a service only it is of type clusterIP.
RBAC is not useful in those situations, as RBAC are limited at the resource level. We cannot say it is limitation of RBAC but this is how things are supposed to work and has this Purpose which it build for. We will achieved this situation using admission control.
Now a days OPA is becoming a standard practice for writing admission controls. Gatekeeper make it simple writing the controls in kubernetes using a native approach.
How Gatekeeper works?
Policies means set of constraints. Consider the following constraints.
User group A can use CPU LIMIT 1G. (say, constraint 1 )
User group B can use CPU LIMIT 2G. (say, constraint 2)
What if you would like to customize this Policies time-to-time so that You need to build these constraints as kubernetes native objects.As per kubernetes Approach we would get it if we do something like “kubectl get constraint 1”.Which means we need to build a CRD that will allows users to create constraints as k8s resources. This is callled as “template” in Gatekeeper.
Prerequisite :
- Working K8s cluster.
Gatekeeper uses OPA(Open Policy Agent) under the hood. Lets install Gatekeeper and some sample Policies using below command.
root@devopsguyvm:~# wget https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.3/deploy/gatekeeper.yaml
--2022-03-15 14:03:03-- https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.3/deploy/gatekeeper.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23434 (23K) [text/plain]
Saving to: ‘gatekeeper.yaml.1’gatekeeper.yaml.1 100%[========================================================================================>] 22.88K 90.1KB/s in 0.3s2022-03-15 14:03:08 (90.1 KB/s) - ‘gatekeeper.yaml.1’ saved [23434/23434]root@devopsguyvm:~# kubectl apply -f gatekeeper.yaml
namespace/gatekeeper-system created
serviceaccount/gatekeeper-admin created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/gatekeeper-admin created
role.rbac.authorization.k8s.io/gatekeeper-manager-role created
clusterrole.rbac.authorization.k8s.io/gatekeeper-manager-role created
rolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
secret/gatekeeper-webhook-server-cert created
service/gatekeeper-webhook-service created
Warning: spec.template.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/manager]: deprecated since v1.19, non-functional in v1.25+; use the "seccompProfile" field instead
deployment.apps/gatekeeper-audit created
deployment.apps/gatekeeper-controller-manager created
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "gatekeeper.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
root@devopsguyvm:~#root@devopsguyvm:~# git clone https://github.com/open-policy-agent/gatekeeper-library.git
Cloning into 'gatekeeper-library'...
remote: Enumerating objects: 2008, done.
remote: Counting objects: 100% (731/731), done.
remote: Compressing objects: 100% (347/347), done.
remote: Total 2008 (delta 467), reused 463 (delta 383), pack-reused 1277
Receiving objects: 100% (2008/2008), 388.13 KiB | 834.00 KiB/s, done.
Resolving deltas: 100% (1105/1105), done.
root@devopsguyvm:~# cd gatekeeper-library
root@devopsguyvm:~/gatekeeper-library# ls
build library LICENSE Makefile mutation NOTICE README.md scripts src test test.shroot@devopsguyvm:~/gatekeeper-library# cd library/
root@devopsguyvm:~/gatekeeper-library/library# ls
general kustomization.yaml pod-security-policy
root@devopsguyvm:~/gatekeeper-library/library# kustomize build | kubectl apply --filename -
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sblockendpointeditdefaultrole created
constrainttemplate.templates.gatekeeper.sh/k8sblocknodeport created
constrainttemplate.templates.gatekeeper.sh/k8scontainerlimits created
constrainttemplate.templates.gatekeeper.sh/k8scontainerratios created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedtags created
constrainttemplate.templates.gatekeeper.sh/k8sexternalips created
constrainttemplate.templates.gatekeeper.sh/k8shttpsonly created
constrainttemplate.templates.gatekeeper.sh/k8simagedigests created
constrainttemplate.templates.gatekeeper.sh/k8spspallowedusers created
constrainttemplate.templates.gatekeeper.sh/k8spspallowprivilegeescalationcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspapparmor created
constrainttemplate.templates.gatekeeper.sh/k8spspcapabilities created
constrainttemplate.templates.gatekeeper.sh/k8spspflexvolumes created
constrainttemplate.templates.gatekeeper.sh/k8spspforbiddensysctls created
constrainttemplate.templates.gatekeeper.sh/k8spspfsgroup created
constrainttemplate.templates.gatekeeper.sh/k8spsphostfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnamespace created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnetworkingports created
constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspprocmount created
constrainttemplate.templates.gatekeeper.sh/k8spspreadonlyrootfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spspseccomp created
constrainttemplate.templates.gatekeeper.sh/k8spspselinuxv2 created
constrainttemplate.templates.gatekeeper.sh/k8spspvolumetypes created
constrainttemplate.templates.gatekeeper.sh/k8sreplicalimits created
constrainttemplate.templates.gatekeeper.sh/k8srequiredannotations created
constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
constrainttemplate.templates.gatekeeper.sh/k8srequiredprobes created
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
constrainttemplate.templates.gatekeeper.sh/k8suniqueserviceselector created
After successful installation,check if Pods are up and Running in gatekeeper-system namespace.
root@devopsguyvm:~# kubectl get pods -n gatekeeper-system
NAME READY STATUS RESTARTS AGE
gatekeeper-controller-manager-765cd6fbf5-9dxkc 1/1 Running 0 4h6m
gatekeeper-audit-68db8cdbc7-wc8ct 1/1 Running 0 4h7m
gatekeeper-controller-manager-765cd6fbf5-9pq9k 1/1 Running 0 4h7m
gatekeeper-controller-manager-765cd6fbf5-sghtg 1/1 Running 0 4h6mYou can see some many CRDs created at gatekeeper-system namespace.
root@devopsguyvm:~# kubectl get crd -n gatekeeper-system
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2022-03-15T08:30:17Z
bgppeers.crd.projectcalico.org 2022-03-15T08:30:17Z
blockaffinities.crd.projectcalico.org 2022-03-15T08:30:18Z
clusterinformations.crd.projectcalico.org 2022-03-15T08:30:18Z
felixconfigurations.crd.projectcalico.org 2022-03-15T08:30:18Z
globalnetworkpolicies.crd.projectcalico.org 2022-03-15T08:30:18Z
globalnetworksets.crd.projectcalico.org 2022-03-15T08:30:19ZReal time use cases:
1.All the service which should be created in your cluster should not be of type NodePort due to security reasons.
Lets Create a template for the policy.When You apply template, gatekeeper creates CRD.When you Apply the constraints,the CRD object is created.Now Admission control will enforce given policies.
root@devopsguyvm:~/opa-gatekeeper-demo/opa# cat block-node-port.yaml
---apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockNodePort
metadata:
name: block-node-port
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Service"]
Lets apply this template using below commands.
root@devopsguyvm:~/opa-gatekeeper-demo/opa# k apply -f block-node-port.yaml
k8sblocknodeport.constraints.gatekeeper.sh/block-node-port createdCheck if below CRD got created or not using below command.
root@devopsguyvm:~# kubectl get crd |grep -i k8sblocknodeport.constraints.gatekeeper.sh
k8sblocknodeport.constraints.gatekeeper.sh 2022-03-15T09:11:46ZNow we have enforced K8sBlockNodePort Policy in our cluster. Now Lets try to apply below manifest file which has one deployment,Ingress and Service of type NodePort.
root@devopsguyvm:~/opa-gatekeeper-demo/app# cat app.yaml
---apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
selector:
matchLabels:
app: devops-toolkit
template:
metadata:
labels:
app: devops-toolkit
spec:
containers:
- name: devops-toolkit
image: vfarcic/devops-toolkit-series:2.7.0
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
readinessProbe:
httpGet:
path: /
port: 80
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi---apiVersion: v1
kind: Service
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: devops-toolkit---apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-toolkit
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: devops-toolkit
port:
number: 80
host: acme.com
Now Lets try to apply above Manifest file. We have k8sblocknodeport Policy already in placed so it should restrict user to create Service of type NodePort.
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl apply -f orig.yaml
deployment.apps/devops-toolkit created
ingress.networking.k8s.io/devops-toolkit created
Error from server ([block-node-port] User is not allowed to create service of type NodePort): error when creating "orig.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [block-node-port] User is not allowed to create service of type NodePortAs expected gatekeeper restrict user to create service of type NodePort. Now you can change Service type from your Manifest file to clusterIP or LB and then Once You apply the change then Service will be created.
2. All the Pod should have CPU and Memory Limit set and Limit should not be more than “1000m” for CPU and “1G” for memory.
Lets apply below Policy which is related to Container Limits. It will create one CRD related container Limits. check if it got created or now.
root@devopsguyvm:~/opa-gatekeeper-demo/opa# cat container-must-have-limits.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
name: container-must-have-limits
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
parameters:
cpu: "1000m"
memory: "1Gi"
root@devopsguyvm:~/opa-gatekeeper-demo/opa# kubectl apply -f container-must-have-limits.yaml
k8scontainerlimits.constraints.gatekeeper.sh/container-must-have-limits created
root@devopsguyvm:~/opa-gatekeeper-demo/opa# kubectl get crd | grep k8scontainerlimits.constraints.gatekeeper.sh
k8scontainerlimits.constraints.gatekeeper.sh 2022-03-15T09:11:46ZNow Lets Apply below Manifest file which has one deployment, Service of type ClusterIP and Ingress.
root@devopsguyvm:~/opa-gatekeeper-demo/app# cat orig.yaml
---apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
selector:
matchLabels:
app: devops-toolkit
template:
metadata:
labels:
app: devops-toolkit
spec:
containers:
- name: devops-toolkit
image: vfarcic/devops-toolkit-series:latest
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
readinessProbe:
httpGet:
path: /
port: 80---apiVersion: v1
kind: Service
metadata:
name: devops-toolkit
labels:
app: devops-toolkit
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: devops-toolkit---apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-toolkit
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: devops-toolkit
port:
number: 80
host: acme.comroot@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl apply -f orig.yaml
deployment.apps/devops-toolkit created
service/devops-toolkit unchanged
ingress.networking.k8s.io/devops-toolkit unchanged
Check if Pod is Up and Running or not using below command. we see that Pod is no Pod is Up and Running so there should be issue with deployment. Lets debug it now.
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl get pod
No resources found in default namespace.Lets check the deployment events. It is saying there is some issue with replica set and hence Pod is not able to scale Up. lets debug Replica set now.
root@devopsguyvm:~/opa-gatekeeper-demo/app# k get deploy
NAME READY UP-TO-DATE AVAILABLE AGEEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 84s deployment-controller Scaled up replica set devops-toolkit-c964cc947 to 1
Lets describe Replicaset and check the events of it. we can see now Container has no resource limits error in it.It is due to k8scontainerlimits Policy which we Applied. You can now add limits to above Manifest file and then Your deployment will be successful.
root@devopsguyvm:~/opa-gatekeeper-demo/app# kubectl get rs
NAME DESIRED CURRENT READY AGE
devops-toolkit-c964cc947 1 0 0 104sEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 27s (x15 over 109s) replicaset-controller Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [container-must-have-limits] container <devops-toolkit> has no resource limits
OPA uses rego language for writing Policies and good thing is you can write a lot in just few lines of code.
Cheers!!!
