How To Store Kubernetes Secrets In Git Repositories securely using Bitnami Sealed Secrets
Nowadays in the GitOps era, all of our manifests are stored in an SCM tool like GitHub, Gitlab, BitBucket, etc. But where are your Kubernetes secrets stored? Do you store them directly in any Git? If yes, then probably it will cause trouble for you because Kubernetes secrets are base64 encoded. Anyone can easily decode your secrets easily. So where do you want to store your secrets now? May be store all the manifests in GitHub and store the secrets elsewhere? And then write a wrapper for that and pull the files from multiple places? Or maybe manually create all the secrets? What if you forgot a value for one of the secrets? What if you lost your machine and all the secrets are exposed? To avoid all this here comes the life savior SealedSecrets. The kubeseal is a utility that would basically convert the secrets to SealedSecrets which means that the secret sealed by the kubeseal utility can only be decrypted by the controller in the Kubernetes cluster from which the SealedSecret resource has been created.
- Running Kubernetes Cluster
Step 1: Installing the kubeseal Client
kubeseal CLI takes a Kubernetes
Secret manifest as an input, encrypts it and outputs a
SealedSecret manifest.This utility allows you to seal Kubernetes
Secrets using the asymmetric crypto algorithm. The
SealedSecrets are Kubernetes resources that contain encrypted
Secrets that only the controller Which is Running in kubernetes Cluster can decrypt.
root@devopsguyvm:~# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64 -O kubeseal
--2021-10-20 10:32:11-- https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64
Resolving github.com (github.com)... 126.96.36.199root@devopsguyvm:~# sudo install -m 755 kubeseal /usr/local/bin/kubeseal
root@devopsguyvm:~# kubeseal --help
Usage of kubeseal:
--add_dir_header If true, adds the file directory to the header
--allow-empty-data Allow empty data in the secret object
Step2: Installing the Custom Controller and CRD for SealedSecret
Below controller.yaml file will create Cluster-role,Cluster-role binding,deployments,Services and Custom Resource inside kube-system namespace whose main task is to created k8s native Secret object after decrypting the secret data from the SealedSecret resource.
root@devopsguyvm:~# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
--2021-10-20 10:24:30-- https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
Resolving github.com (github.com)... 188.8.131.52root@devopsguyvm:~# kubectl apply -f controller.yaml
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
Step3: Verify if the sealed-secret controller pod is running or not
Check if sealed-secrets-controller Pod is Running or not.
root@devopsguyvm:~# kubectl get pods -n kube-system | grep sealed-secrets-controller
sealed-secrets-controller-5b54cbfb5f-8zmz5 1/1 Running 0 28s
If you check the logs of the sealed-secrets-controller pod, you can see the secret that this controller has created for its own functionality which contains the public/private key pair which it will use for encrypting/decrypting the secrets.
Step4: Use the kubeseal command to create SealedSecret resource manifest file.
Create a Secret manifest file with name secrets.yaml. We want to secure the key and value data.Let us now use the kubeseal command to create SealedSecret resource manifest file from the secrets.yaml file.
root@devopsguyvm:~# kubectl create secret generic my-secret --from-literal=key=value --dry-run=client -o json |kubeseal | tee secret.yaml
kubeseal gets the public key from the K8s cluster and uses that to encrypt the data.Now you can Securely Upload encrypted Secret file to Git or to any SCM tool.
Step5: Let’s create resource in k8s using the Sealed Secret manifest file.
controller which is Running in kubernetes cluster on creation of the sealed secret intercepted the request and created k8s native Secret object after decrypting the secret data from the SealedSecret resource.
root@devopsguyvm:~# kubectl apply -f secret.yaml
You can see kubernetes native secret object is Created Successfully in the Cluster.
No one apart from the running controller can decrypt the
SealedSecret, not even the author of the
Secret.It’s a general best practice to disallow users to have direct access to read secrets. You can create RBAC rules to forbid low-privilege users from reading
Secrets. You can also restrict users to only be able to read
Secrets from their namespace.
SealedSecret resources provide multiple ways to prevent such misuse. They are namespace-aware by default. Once you generate a
kubeseal for a particular namespace, you can’t use the
SealedSecret in another namespace.
What if you have to define a
SealedSecrets that you want to move access the namespaces?
These possibilities can be achieved using scopes.
There are three scopes you can create your
strict(default): In this case, you need to seal your
Secretconsidering the name and the namespace. You can’t change the name and the namespaces of your
SealedSecretonce you've created it. If you try to do that, you get a decryption error.
namespace-wide: This scope allows you to freely rename the
SealedSecretwithin the namespace for which you’ve sealed the
cluster-wide: This scope allows you to freely move the
Secretto any namespace and give it any name you wish.
Apart from the name and namespace, you can rename the secret keys without losing any decryption capabilities.
You can select the scope with the
--scope flag while using
$ kubeseal --scope cluster-wide --format yaml <secret.yaml >sealed-secret.yaml
You can also use annotations within your
Secret to apply scopes before you pass the configuration to
If you don’t specify any annotations, then
kubeseal assumes a
strict scope. If you set both annotations,
cluster-wide takes precedence.
How it is working?
Sealed Secrets are a “one-way” encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, post to twitter, etc. Once the SealedSecret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.
The SealedSecrets implementation consists of two components:
- A controller that runs in-cluster, and implements a new SealedSecret Kubernetes API object via the “third party resource” mechanism.
kubesealcommand line tool that encrypts a regular Kubernetes Secret object (as YAML or JSON) into a SealedSecret.
Once decrypted by the controller, the enclosed Secret can be used exactly like a regular K8s Secret (it is a regular K8s Secret at this point!). If the SealedSecret object is deleted, the controller will garbage collect the generated Secret.