How To Store Kubernetes Secrets In Git Repositories securely using Bitnami Sealed Secrets

5 min readOct 20, 2021

Nowadays in the GitOps era, all of our manifests are stored in an SCM tool like GitHub, Gitlab, BitBucket, etc. But where are your Kubernetes secrets stored? Do you store them directly in any Git? If yes, then probably it will cause trouble for you because Kubernetes secrets are base64 encoded. Anyone can easily decode your secrets easily. So where do you want to store your secrets now? May be store all the manifests in GitHub and store the secrets elsewhere? And then write a wrapper for that and pull the files from multiple places? Or maybe manually create all the secrets? What if you forgot a value for one of the secrets? What if you lost your machine and all the secrets are exposed? To avoid all this here comes the life savior SealedSecrets. The kubeseal is a utility that would basically convert the secrets to SealedSecrets which means that the secret sealed by the kubeseal utility can only be decrypted by the controller in the Kubernetes cluster from which the SealedSecret resource has been created.

Pre-requisites:

Running Kubernetes Cluster

Installation:

Step 1: Installing the kubeseal Client

The kubeseal CLI takes a Kubernetes Secret manifest as an input, encrypts it and outputs a SealedSecret manifest.This utility allows you to seal Kubernetes Secrets using the asymmetric crypto algorithm. The SealedSecrets are Kubernetes resources that contain encrypted Secrets that only the controller Which is Running in kubernetes Cluster can decrypt.

root@devopsguyvm:~# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64 -O kubeseal
--2021-10-20 10:32:11--  https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64
Resolving github.com (github.com)... 13.234.210.38root@devopsguyvm:~# sudo install -m 755 kubeseal /usr/local/bin/kubeseal
root@devopsguyvm:~# kubeseal --help
Usage of kubeseal:
      --add_dir_header                   If true, adds the file directory to the header
      --allow-empty-data                 Allow empty data in the secret object

Step2: Installing the Custom Controller and CRD for SealedSecret

Below controller.yaml file will create Cluster-role,Cluster-role binding,deployments,Services and Custom Resource inside kube-system namespace whose main task is to created k8s native Secret object after decrypting the secret data from the SealedSecret resource.

root@devopsguyvm:~# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
--2021-10-20 10:24:30--  https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
Resolving github.com (github.com)... 13.234.176.102root@devopsguyvm:~# kubectl apply -f controller.yaml
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
service/sealed-secrets-controller created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
deployment.apps/sealed-secrets-controller created

Step3: Verify if the sealed-secret controller pod is running or not

Check if sealed-secrets-controller Pod is Running or not.

root@devopsguyvm:~# kubectl get pods -n kube-system | grep sealed-secrets-controller
sealed-secrets-controller-5b54cbfb5f-8zmz5   1/1     Running   0          28s

If you check the logs of the sealed-secrets-controller pod, you can see the secret that this controller has created for its own functionality which contains the public/private key pair which it will use for encrypting/decrypting the secrets.

Step4: Use the kubeseal command to create SealedSecret resource manifest file.

Create a Secret manifest file with name secrets.yaml. We want to secure the key and value data.Let us now use the kubeseal command to create SealedSecret resource manifest file from the secrets.yaml file.

root@devopsguyvm:~# kubectl create secret generic my-secret --from-literal=key=value --dry-run=client -o json |kubeseal | tee secret.yaml
{
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",
  "metadata": {
    "name": "my-secret",
    "namespace": "default",
    "creationTimestamp": null
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "my-secret",
        "namespace": "default",
        "creationTimestamp": null
      },
      "data": null
    },
    "encryptedData": {
      "key": "AgAmQWAVgI0sZHlR0YbKRt54FRI8Lk9o9Gk9whJxM3NhqmyRVE3TMXhm5oPJe9iHnCUvYeKr8aaN8RWrPyCTLUg2pQsnGJLEiXUZUNAMXXUoIMZF+XquwYXMFJVPuXWCC7cuW6t4NeEeZ7Eklf4T9z4TnTFME4/d6JkH02u7ALC7fjoso9jfxusgEPzIJlJuB4YWIT0OXi8VqCq3WdWc3tWgyEfrva7ZglS8bX/pXVrTWeQ9hnal5KcncvjeFTya4PWJrtxAGgLGn1BPga4JzWQKVlkkDwQ9NMBkEI4aKA6Ecs1LpzCilA6cFVZ2KVEI2nLBojYvs7oQT5XEHCpc/8j9M6b9HVi8rSaNy8AVf9ZMKEZ0jDo97YExXkjloD68p6u+fCvBBbOwmauE7sLC2eZCCPNs93LTQpWgTy165D/S80Zkjd3CPOB5ER/egHbDKahkvoqJBOTstqzjT6XYIb0PMHz3108kmHSjW0do9zUkaI6rzwmhPj9tDEB3q3PC90j2h9mso14uaBpMsqV9qmViR4OTJwb90lIqi6+Agvr/rxA8MiMvAF1XkPovXi7SF2jOUzC3Kt0CVCsl1enf8JtiuRB1Hjtzlv8eRUunMu5wvm9ymabbGhDbrExQVdJNXCuSERb3RIgKfiMh1XTxxKW0tL2KDQ5vE/f2pyXeEPs/lxvStuunRnhA1DB11GZhpVVSNhFEig=="
    }
  }
}

kubeseal gets the public key from the K8s cluster and uses that to encrypt the data.Now you can Securely Upload encrypted Secret file to Git or to any SCM tool.

Step5: Let’s create resource in k8s using the Sealed Secret manifest file.

controller which is Running in kubernetes cluster on creation of the sealed secret intercepted the request and created k8s native Secret object after decrypting the secret data from the SealedSecret resource.

root@devopsguyvm:~# kubectl apply -f secret.yaml
sealedsecret.bitnami.com/my-secret created

You can see kubernetes native secret object is Created Successfully in the Cluster.

SealedSecret Scopes

No one apart from the running controller can decrypt the SealedSecret, not even the author of the Secret.It’s a general best practice to disallow users to have direct access to read secrets. You can create RBAC rules to forbid low-privilege users from reading Secrets. You can also restrict users to only be able to read Secrets from their namespace.

SealedSecret resources provide multiple ways to prevent such misuse. They are namespace-aware by default. Once you generate a SealedSecret using kubeseal for a particular namespace, you can’t use the SealedSecret in another namespace.

What if you have to define a SealedSecrets that you want to move access the namespaces?

These possibilities can be achieved using scopes.

There are three scopes you can create your SealedSecrets with:

strict (default): In this case, you need to seal your Secret considering the name and the namespace. You can’t change the name and the namespaces of your SealedSecret once you've created it. If you try to do that, you get a decryption error.
namespace-wide: This scope allows you to freely rename the SealedSecret within the namespace for which you’ve sealed the Secret.
cluster-wide: This scope allows you to freely move the Secret to any namespace and give it any name you wish.

Apart from the name and namespace, you can rename the secret keys without losing any decryption capabilities.

You can select the scope with the --scope flag while using kubeseal:

$ kubeseal --scope cluster-wide --format yaml <secret.yaml >sealed-secret.yaml

You can also use annotations within your Secret to apply scopes before you pass the configuration to kubeseal:

sealedsecrets.bitnami.com/namespace-wide: "true" for namespace-wide
sealedsecrets.bitnami.com/cluster-wide: "true" for cluster-wide

If you don’t specify any annotations, then kubeseal assumes a strict scope. If you set both annotations, cluster-wide takes precedence.

How it is working?

Sealed Secrets are a “one-way” encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, post to twitter, etc. Once the SealedSecret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.

The SealedSecrets implementation consists of two components:

A controller that runs in-cluster, and implements a new SealedSecret Kubernetes API object via the “third party resource” mechanism.
A kubeseal command line tool that encrypts a regular Kubernetes Secret object (as YAML or JSON) into a SealedSecret.

Once decrypted by the controller, the enclosed Secret can be used exactly like a regular K8s Secret (it is a regular K8s Secret at this point!). If the SealedSecret object is deleted, the controller will garbage collect the generated Secret.

Cheers!!!

How To Store Kubernetes Secrets In Git Repositories securely using Bitnami Sealed Secrets

Pre-requisites:

Installation:

SealedSecret Scopes

How it is working?

References:

Written by Akshay Bobade

No responses yet