In most enterprise systems where software release cycles consist of separate environments like dev, stage, live, having multiple environments that can be dynamically configured is common. An application may have three different sets of database credentials for authentication. Each set of credentials would be respective to an instance for a particular environment. This approach essentially allows software developers to interact with a developer-friendly database when carrying out their day-to-day coding.
Kubernetes secrets is used to store very sensitive data in your Kubernetes cluster. Secrets are native Kubernetes resources saved in the cluster data store (i.e., etcd database) and can be made available to your containers at runtime. Managing these secrets is a challenging process so in this guide we will explore how to use External Secrets to simplify that.
Kubernetes External Secrets allows you to use external secret management systems like GCP Secrets manager, AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.
An ExternalSecret
declares how to fetch the secret data, while the controller converts all ExternalSecrets
to Secrets
. The conversion is completely transparent to Pods
that can access Secrets
normally. By default Secrets
are not encrypted at rest and are open to attack, either via the etcd
server or via backups of etcd
data. To mitigate this risk, use an external secret management system with a KMS plugin to encrypt Secrets
stored in etcd
.
This is the workflow of how the External Secrets work:
ExternalSecrets
are added in the cluster (e.g.,kubectl apply -f external-secret-example.yml
)- Controller fetches
ExternalSecrets
using the Kubernetes API - Controller uses
ExternalSecrets
to fetch secret data from external providers (e.g, AWS Secrets Manager) - Controller upserts
Secrets
Pods
can accessSecrets
normally
Checkout related content:
- How to create and manage Secrets in GCP Secret Manager using Terraform
- How to Create a Service Account for Terraform in GCP
- How to use Terraform to create a vpc network and a Cloud SQL in GCP
- How to use Terraform to create a vpc network and a GKE in GCP
Step 1 – Installing External Secrets
External secrets is available officially as a helm chart in this repo. To add the repo and install it as external-secrets
in our cluster, use this:
$ helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
$ helm install external-secrets external-secrets/kubernetes-external-secrets
You can also install it with kubectl
if you don’t want to use helm in your cluster by generating the manifests then applying:
helm template --include-crds --output-dir ./output_dir external-secrets/kubernetes-external-secrets
The generated kubernetes manifests will be in ./output_dir
and can be applied to deploy kubernetes-external-secrets
to the cluster.
Step 2 – Ensure you are logged in to GCP
External secrets supports fetching secrets from GCP Secret Manager. The external secrets will poll for changes of the secret according to the value set for POLLER_INTERVAL_MILLISECONDS
in env. Depending on the time interval this is set to you may incur additional charges as Google Secret Manager charges per a set number of API calls. A service account is required to grant the controller access to pull secrets.
To do this, first ensure you are logged in to gcp
gcloud auth login
gcloud auth application login
If you have multiple project ensure you are in the right one
gcloud config set project citizix-prj
Then enable GCP secret manager:
gcloud services enable secretmanager.googleapis.com
Step 3 – Creating a Service account
A service account is required to grant the controller access to pull secrets. To add a service account, use this command:
gcloud iam service-accounts create citizix-sa \
--description="Citizix External Secrets SA" \
--display-name="External Secrets"
Create service account keys:
gcloud iam service-accounts keys create sm-key.json --iam-account=citizix-sa@citizix-prj.iam.gserviceaccount.com
Make sure to grant permissions to the created sa:
gcloud projects add-iam-policy-binding citizix-prj \
--member="serviceAccount:citizix-sa@citizix-prj.iam.gserviceaccount.com" \
--role="roles/secretmanager.viewer" \
--role="roles/secretmanager.secretAccessor" \
--condition="None"
Keys will be generated as <em>sm-key.json</em>
and saved in the current directory.
Create a kubernetes secret with the service account credentials to be used to fetch the secrets:
kubectl create secret generic gcp-sm-sa-secret --from-file=secret-access-credentials=sm-key.json
Step 4 – Adding a secret to GCP Secret manager
Let us add some secret data to the secret manager backend using GCP SDK. You can do this in the Cloud Console as well, it will work just fine.
We are adding a database connection.
echo -ne 'postgres://db_user:p4ssw8@10.2.11.4/db?sslmode=disable' | gcloud secrets create sm-citizix-db-secret --data-file=-
You should see an output similar to this:
Created version [1] of the secret [sm-citizix-db-secret].
If the secret needs to be updated, you can do so using this command:
echo -n 'postgres://new_db_user:p4ssw8@10.2.11.4/new_db?sslmode=disable' | gcloud secrets versions add sm-citizix-db-secret --data-file=-
You can also create the secret as json. A json can hold multiple key value pairs. This is how you could create a json secret:
echo -ne '{"host":"10.2.11.4","username":"db_user","password":"p4ssw8","db":"db"}' | gcloud secrets create sm-json-citizix-db-secret --data-file=-
You should get the same output as shown above. If for any reason you want to confirm that the secret has been set successfully, you can retrieve the latest version using this command:
$ gcloud secrets versions access latest --secret="sm-citizix-db-secret"
$ gcloud secrets versions access latest --secret="sm-json-citizix-db-secret"
Step 5 – Deploy Secret store
A Secret Store is a namespaced resource that determines how your external Secret will be accessed from an authentication perspective. It contains references to Secrets that have the credentials to access the external API. External Secrets also has a ClusterSecretStore which is a global or cluster-wide SecretStore that can be referenced from all namespaces to provide a central gateway to your secrets manager. In our case we will use a SecretStore.
Let us create a Secret store called google-secret-manager
that will use the earlier defined secret gcp-sm-sa-secret
.
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: google-secret-manager
spec:
provider:
gcpsm:
projectID: citizix-prj
auth:
secretRef:
secretAccessKeySecretRef:
name: gcp-sm-sa-secret
key: secret-access-credentials
EOF
We can confirm the secret store creation with this command:
$ kubectl get secretstore
NAME AGE STATUS
google-secret-manager 9s Valid
Step 6 – Creating External Secrets
An ExternalSecret is a resource that declares the data you want to fetch from the external secrets manager. It will reference the SecretStore to know how to access sensitive data.
Let us create an external secret for our db secret. Here we are accessing the secret sm-citizix-db-secret
and creating a target citizix-db-secret
secret in kubernetes with a key db-url
:
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: es-citizix-db-secret
spec:
refreshInterval: 1h
secretStoreRef:
name: google-secret-manager
kind: SecretStore
target:
name: citizix-db-secret
creationPolicy: Owner
data:
- secretKey: db-url
remoteRef:
key: sm-citizix-db-secret
EOF
We can confirm that the defined external secret is working as expected:
$ kubectl get externalsecret
NAME STORE REFRESH INTERVAL STATUS
es-citizix-db-secret google-secret-manager 1h SecretSynced
We can also check that it has created the respective kubernetes secret:
$ kubectl get secret
NAME TYPE DATA AGE
citizix-db-secret Opaque 1 62s
To confirm the content of the created secret, describe the secret:
$ k describe secret citizix-db-secret
Name: citizix-db-secret
Namespace: default
Labels: <none>
Annotations: reconcile.external-secrets.io/data-hash: 9ca2636815e1610df9f9458d03e3814d
Type: Opaque
Data
====
db-url: 54 bytes
Extracting json values
We can extract json key values as secret data. External secret provides the dataFrom attribute that does just that:
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: es-json-citizix-db-secret
spec:
refreshInterval: 1h
secretStoreRef:
name: google-secret-manager
kind: SecretStore
target:
name: json-citizix-db-secret
creationPolicy: Owner
dataFrom:
- extract:
key: sm-json-citizix-db-secret
EOF
We can confirm that the defined external secret is working as expected:
$ kubectl get es
NAME STORE REFRESH INTERVAL STATUS
es-json-citizix-db-secret google-secret-manager 1h SecretSynced
We can also check that it has created the respective kubernetes secret:
$ kubectl get secret
NAME TYPE DATA AGE
json-citizix-db-secret Opaque 4 14s
To confirm the content of the created secret, describe the secret:
$ kubectl describe secret json-citizix-db-secret
Name: json-citizix-db-secret
Namespace: default
Labels: <none>
Annotations: reconcile.external-secrets.io/data-hash: 9c7f95c6c59808128f2ab2630e980dee
Type: Opaque
Data
====
db: 2 bytes
host: 9 bytes
password: 6 bytes
username: 7 bytes
Conclusion
In this guide, we learnt what External Secrets are and how we can leverage them to populate our kubernetes secrets from GCP secret manager.