In most enterprise systems where software release cycles consist of separate environments like dev, stage, live, having multiple environments that can be dynamically configured is common. An application may have three different sets of database credentials for authentication. Each set of credentials would be respective to an instance for a particular environment. This approach essentially allows software developers to interact with a developer-friendly database when carrying out their day-to-day coding.

Kubernetes secrets is used to store very sensitive data in your Kubernetes cluster. Secrets are native Kubernetes resources saved in the cluster data store (i.e., etcd database) and can be made available to your containers at runtime. Managing these secrets is a challenging process so in this guide we will explore how to use External Secrets to simplify that.

Kubernetes External Secrets allows you to use external secret management systems like GCP Secrets manager, AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.

An ExternalSecret declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets. The conversion is completely transparent to Pods that can access Secrets normally. By default Secrets are not encrypted at rest and are open to attack, either via the etcd server or via backups of etcd data. To mitigate this risk, use an external secret management system with a KMS plugin to encrypt Secrets stored in etcd.

This is the workflow of how the External Secrets work:

1. ExternalSecrets are added in the cluster (e.g., kubectl apply -f external-secret-example.yml)
2. Controller fetches ExternalSecrets using the Kubernetes API
3. Controller uses ExternalSecrets to fetch secret data from external providers (e.g, GCP Secrets Manager)
4. Controller upserts Secrets
5. Pods can access Secrets normally

Checkout related content:

• How to create and manage Secrets in GCP Secret Manager using Terraform
• How to Create a Service Account for Terraform in GCP
• How to use Terraform to create a vpc network and a Cloud SQL in GCP
• How to use Terraform to create a vpc network and a GKE in GCP

Installing External Secrets

External secrets is available officially as a helm chart in this repo. To add the repo and install it as external-secrets in our cluster, use this:

1
2

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
helm install external-secrets external-secrets/kubernetes-external-secrets

You can also install it with kubectl if you don’t want to use helm in your cluster by generating the manifests then applying:

1

helm template --include-crds --output-dir ./output_dir external-secrets/kubernetes-external-secrets

The generated kubernetes manifests will be in ./output_dir and can be applied to deploy kubernetes-external-secrets to the cluster.

Ensure you are logged in to GCP

External secrets supports fetching secrets from GCP Secret Manager. The external secrets will poll for changes of the secret according to the value set for POLLER_INTERVAL_MILLISECONDS in env. Depending on the time interval this is set to you may incur additional charges as Google Secret Manager charges per a set number of API calls. A service account is required to grant the controller access to pull secrets.

To do this, first ensure you are logged in to gcp

1
2

gcloud auth login
gcloud auth application login

If you have multiple project ensure you are in the right one

1

gcloud config set project citizix-prj

Then enable GCP secret manager:

1

gcloud services enable secretmanager.googleapis.com

Creating a Service account

A service account is required to grant the controller access to pull secrets. To add a service account, use this command:

1
2
3

gcloud iam service-accounts create citizix-sa \
    --description="Citizix External Secrets SA" \
    --display-name="External Secrets"

Create service account keys:

1
2

gcloud iam service-accounts keys create sm-key.json \
  --iam-account=citizix-sa@citizix-prj.iam.gserviceaccount.com

Make sure to grant permissions to the created sa:

1
2
3
4
5

gcloud projects add-iam-policy-binding citizix-prj \
  --member="serviceAccount:citizix-sa@citizix-prj.iam.gserviceaccount.com" \
  --role="roles/secretmanager.viewer" \
  --role="roles/secretmanager.secretAccessor" \
  --condition="None"

Keys will be generated as sm-key.json and saved in the current directory.

Create a kubernetes secret with the service account credentials to be used to fetch the secrets:

1
2

kubectl create secret generic gcp-sm-sa-secret \
  --from-file=secret-access-credentials=sm-key.json

Adding a secret to GCP Secret manager

Let us add some secret data to the secret manager backend using GCP SDK. You can do this in the Cloud Console as well, it will work just fine.

We are adding a database connection.

1

echo -ne 'postgres://db_user:p4ssw8@10.2.11.4/db?sslmode=disable' | gcloud secrets create sm-citizix-db-secret --data-file=-

You should see an output similar to this:

1

Created version [1] of the secret [sm-citizix-db-secret].

If the secret needs to be updated, you can do so using this command:

1

echo -n 'postgres://new_db_user:p4ssw8@10.2.11.4/new_db?sslmode=disable' | gcloud secrets versions add sm-citizix-db-secret --data-file=-

You can also create the secret as json. A json can hold multiple key value pairs. This is how you could create a json secret:

1

echo -ne '{"host":"10.2.11.4","username":"db_user","password":"p4ssw8","db":"db"}' | gcloud secrets create sm-json-citizix-db-secret --data-file=-

You should get the same output as shown above. If for any reason you want to confirm that the secret has been set successfully, you can retrieve the latest version using this command:

1
2

gcloud secrets versions access latest --secret="sm-citizix-db-secret"
gcloud secrets versions access latest --secret="sm-json-citizix-db-secret"

Deploy Secret store

A Secret Store is a namespaced resource that determines how your external Secret will be accessed from an authentication perspective. It contains references to Secrets that have the credentials to access the external API. External Secrets also has a ClusterSecretStore which is a global or cluster-wide SecretStore that can be referenced from all namespaces to provide a central gateway to your secrets manager. In our case we will use a SecretStore.

Let us create a Secret store called google-secret-manager that will use the earlier defined secret gcp-sm-sa-secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: google-secret-manager
spec:
  provider:
      gcpsm:
        projectID: citizix-prj
        auth:
          secretRef:
            secretAccessKeySecretRef:
              name: gcp-sm-sa-secret
              key: secret-access-credentials
EOF

We can confirm the secret store creation with this command:

1
2
3
4

$ kubectl get secretstore

NAME                    AGE   STATUS
google-secret-manager   9s    Valid

Creating External Secrets

An ExternalSecret is a resource that declares the data you want to fetch from the external secrets manager. It will reference the SecretStore to know how to access sensitive data.

Let us create an external secret for our db secret. Here we are accessing the secret sm-citizix-db-secret and creating a target citizix-db-secret secret in kubernetes with a key db-url:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-citizix-db-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: google-secret-manager
    kind: SecretStore
  target:
    name: citizix-db-secret
    creationPolicy: Owner
  data:
  - secretKey: db-url
    remoteRef:
      key: sm-citizix-db-secret
EOF

We can confirm that the defined external secret is working as expected:

1
2
3
4

$ kubectl get externalsecret

NAME                   STORE                   REFRESH INTERVAL   STATUS
es-citizix-db-secret   google-secret-manager   1h                 SecretSynced

We can also check that it has created the respective kubernetes secret:

1
2
3
4

$ kubectl get secret

NAME                  TYPE                                  DATA   AGE
citizix-db-secret     Opaque                                1      62s

To confirm the content of the created secret, describe the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ k describe secret citizix-db-secret

Name:         citizix-db-secret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: 9ca2636815e1610df9f9458d03e3814d

Type:  Opaque

Data
====
db-url:  54 bytes

Extracting json values

We can extract json key values as secret data. External secret provides the dataFrom attribute that does just that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-json-citizix-db-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: google-secret-manager
    kind: SecretStore
  target:
    name: json-citizix-db-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: sm-json-citizix-db-secret
EOF

We can confirm that the defined external secret is working as expected:

1
2
3
4

$ kubectl get es

NAME                        STORE                   REFRESH INTERVAL   STATUS
es-json-citizix-db-secret   google-secret-manager   1h                 SecretSynced

We can also check that it has created the respective kubernetes secret:

1
2
3
4

$ kubectl get secret

NAME                     TYPE                                  DATA   AGE
json-citizix-db-secret   Opaque                                4      14s

To confirm the content of the created secret, describe the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ kubectl describe secret json-citizix-db-secret

Name:         json-citizix-db-secret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: 9c7f95c6c59808128f2ab2630e980dee

Type:  Opaque

Data
====
db:        2 bytes
host:      9 bytes
password:  6 bytes
username:  7 bytes

Conclusion

In this guide, we learnt what External Secrets are and how we can leverage them to populate our kubernetes secrets from GCP secret manager.