Sealed Secrets for Kubernetes

Introduction

In Kubernetes, we use secrets to store sensitive data within the Kubernetes cluster. But where do you store them? If we want to keep them in SCM (Source Code Management) tool, it becomes a security threat.

Secret in Kubernetes is used to store sensitive information like passwords, ssh keys, certificates, tokens etc. Secrets are encoded in base64 not encrypted and automatically decoded when they are attached to the pod.

These data are “only” encoded so if a user has access to your secrets, he can simply execute a base64 decode command to see your sensitive data (kubectl get secret my-secret -o jsonpath=”{.data.password}” | base64 - - decode).

And as it is not encrypted it can be insecure to commit them to your Source Code Management (SCM). If we want to store these secrets in SCM security problem appears. To avoid all these issues we can use SealedSecrets.

Sealed Secrets

Sealed Secrets allow for “one-way” encryption of your Kubernetes Secrets and can only be decrypted by the Sealed Secrets controller running in your target cluster. This mechanism is based on public-key encryption, a form of cryptography consisting of a public key and a private key pair. One can be used for encryption, and only the other key can be used to decrypt what was encrypted. The controller will generate the key pair, publish the public key certificate to the logs and expose it over an HTTP API request.

These encrypted Secrets can also be deployed to a Kubernetes cluster using normal workflows with tools such as kubectl.

Prerequisites:

Kubernetes Cluster is up and running.
A namespace is created where you can deploy your secret. For this tutorial, we are going to use the “demo” namespace.

Installation: -

We assume you have all the prerequisites and now you are ready to use SealedSecret on your Kubernetes cluster.

Installation of Sealed Secret contains 2 steps: -

Installing Kubeseal CLI on the local machine.
Installing the Custom Controller for SealedSecret.

Installing Kubeseal CLI on the local machine.

We use the Kubeseal CLI tool to generate a public key certificate to encrypt our secret in a sealed secret. Kubeseal can communicate with the controller through the Kubernetes API server and retrieve the public key needed for encrypting a Secret at runtime. The public key may also be downloaded from the controller and saved locally to be used offline.

The client tool can be installed into /usr/local/bin by following the below-mentioned steps: -

First, we need to download the kubeseal tar file by running the following command:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-amd64.tar.gz

Once the tar file is downloaded extract that tar file using the below-mentioned command:

tar -xfz kubeseal-0.18.0-linux-amd64.tar.gz

Now we can install kubeseal on location /usr/local/bin/kubeseal, Use the below-mentioned command:

sudo install -m 755 kubeseal /usr/local/bin/kubeseal

To check kubeseal is installed or not run this command:

kubeseal --version

Installing the Custom Controller and CRD for SealedSecret

Now we will install the controller and the SealedSecret custom resource definition in the kube-system namespace. The sealed secrets controller will enable the lifecycle operations of Sealed Secrets in your cluster.

To deploy the sealed-secret controller follow the steps mentioned below:-

First, we need to download the controller, use the below-mentioned command to download the controller manifest.

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

Once the controller is downloaded, deploy it on the Kubernetes cluster using the command mentioned below

kubectl apply -f controller.yaml

You can check the status of your deployment by using the following command

kubectl get pods -n kube-system | grep sealed-secrets-controller

You can view the contents of the Secret which contains the public/private key pair in YAML format, by running this command: -

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

Output: -

Now your controller is deployed.

Sealing the secrets

Let’s create a secret file named secret.yaml. You can use the below-mentioned example or you can create your own secret.yaml file.

apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: my-secret
namespace: demo
data:
user: YWRtaW4=
password: TXlTZWNyZXREZW1vRmlsZQ==

There are two ways to create sealed secrets: -

Without the certificate file (Online).
With the certificate file (Offline).

Online (Without the certificate file).

Now, let’s use this secret.yaml file to create SealedSecret YAML manifests with kubeseal.

Make sure you have access to the target Kubernetes cluster and that you are able to run the kubectl command against the target Kubernetes cluster, before executing the following command.

kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

After running this command successfully you will get a file named sealed-secret.yaml

2. Offline (With the certificate file).

An alternative approach is to fetch the public key from the controller and use it offline to seal your Secrets.

Access to the Kubernetes cluster is only required to fetch the pem file. Once done you can use that pem file to seal your secret for the target cluster from any machine which has kubeseal installed on it.

This command will fetch the cert file and save it in your local machine with the name public-key-cert.pem

kubeseal --fetch-cert > public-key-cert.pem

You can now use this command to create the sealed-secret.yaml

kubeseal --cert=public-key-cert.pem --format=yaml < secret.yaml > sealed-secret.yaml

This command will also create a sealed-secret.yaml file.

You can check both files (secret.yaml and sealed-secret.yaml )

Secret.yaml file

cat secret.yaml

Output: -

apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: my-secret
namespace: demo
data:
user: YWRtaW4=
password: TXlTZWNyZXREZW1vRmlsZQ==
apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: my-secret
namespace: demo
data:
user: YWRtaW4=
password: TXlTZWNyZXREZW1vRmlsZQ==

sealed-secret.yaml

cat sealed-secret.yaml

Output: -

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: my-secret
namespace: demo
spec:
encryptedData:
  password: AgB/XlNUiGvhKT25K(…)bOtCIIDAid
  user: AgAI9gJiQt0NHULDqdWw4(…)SSlw==
 template:
 data: null
 metadata:
   creationTimestamp: null
   name: my-secret
   namespace: demo

Note that the keys in the original Secret, namely, user and password, are not encrypted in the SealedSecret, only their values are encrypted.

Now we can deploy the sealed secret in our Kubernetes cluster.

kubectl apply -f sealed-secret.yaml

If you check the logs of the sealed controller running in the kube-system namespace, you will see the SealedSecret custom resource that was just deployed.

kubectl logs -n kube-system sealed-secrets-controller-7bdbc75d47–5wxvf

Output:

2022/08/30 07:54:49 Updating demo/my-secret
2022/08/30 07:54:49 Event(v1.ObjectReference{Kind:”SealedSecret”, Namespace:”demo”, Name:”my-secret”, UID:”330bcf2a-9433–4696–8784-a00190758b86", APIVersion:”bitnami.com/v1alpha1", ResourceVersion:”221612", FieldPath:””}): type: ‘Normal’ reason: ‘Unsealed’ SealedSecret unsealed successfully

Testing

We are going to deploy an application and in that deployment, we will pass our newly created secret as ENV variables. Then we will check if we are able to access those secrets from inside the pod or not.

For testing, you can deploy your own application or you can use the below-mentioned manifest for testing.

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo
namespace: demo
spec:
replicas: 1
selector:
  matchLabels:
    app: demo
template:
  metadata:
    labels:
      app: demo
  spec:
    containers:
    - name: demo
      image: sagar27/testing:latest
      ports:
      - containerPort: 80
      env:
      - name: KUBE_USER
        valueFrom:
          secretKeyRef:
            name: my-secret
            key: user
      - name: KUBE_PASS
        valueFrom:
          secretKeyRef:
            name: my-secret
            key: password

As you can see in the above manifest we have created 2 variables KUBE_USER and KUBE_PASS we will map the values stored in secret with this env variable. Save this file with the name deploy.yaml.

Deploy this manifest using the command,

Kubectl apply -f deploy.yaml

Wait for a few minutes and then run the below-mentioned command.

kubectl get po -n demo

You will get output like this.

Copy the pod name and get the shell to this pod by using the below-mentioned command.

kubectl exec -it demo-d8745c657-l6d42 bash -n demo

Now, you can check the values of the variables by running the below-mentioned command.

root@demo-d8745c657-l6d42:/# echo $KUBE_USER
admin
root@demo-d8745c657-l6d42:/# echo $KUBE_PASS
MySecretDemoFile

The YAML file sealed-secret.yaml, that pertains to the SealedSecret is safe to be stored in a Git repository along with YAML manifests pertaining to other Kubernetes resources such as DaemonSets, Deployments, ConfigMaps etc. deployed in the cluster.