How to use External Secrets with AWS Secrets manager

In most enterprise systems where software release cycles consist of separate environments like dev, stage, live, having multiple environments that can be dynamically configured is common. An application may have three different sets of database credentials for authentication. Each set of credentials would be respective to an instance for a particular environment. This approach essentially allows software developers to interact with a developer-friendly database when carrying out their day-to-day coding.

Kubernetes secrets is used to store very sensitive data in your Kubernetes cluster. Secrets are native Kubernetes resources saved in the cluster data store (i.e., etcd database) and can be made available to your containers at runtime. Managing these secrets is a challenging process so in this guide we will explore how to use External Secrets to simplify that.

Kubernetes External Secrets allows you to use external secret management systems like GCP Secrets manager, AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.

An ExternalSecret declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets. The conversion is completely transparent to Pods that can access Secrets normally. By default Secrets are not encrypted at rest and are open to attack, either via the etcd server or via backups of etcd data. To mitigate this risk, use an external secret management system with a KMS plugin to encrypt Secrets stored in etcd.

This is the workflow of how the External Secrets work:

ExternalSecrets are added in the cluster (e.g. kubectl apply -f external-secret-example.yml)
Controller fetches ExternalSecrets using the Kubernetes API
Controller uses ExternalSecrets to fetch secret data from external providers (e.g, AWS Secrets Manager)
Controller upserts Secrets
Pods can access Secrets normally

Checkout related content:

Configuring AWS Access

We first starting by creating a policy named secrets-reader:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
POLICY_ARN=$(aws iam create-policy --policy-name secrets-reader --policy-document '{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Effect": "Allow",
            "Action": &#91;
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": &#91;
                "*"
            ]
        }
    ]
}' | jq -r .Policy.Arn)

We now create a group that will use this policy:

1
2
aws iam create-group --group-name secret-readers
aws iam attach-group-policy --policy-arn $POLICY_ARN --group-name secret-readers

Next, we create an username and attach it to the recently created-group:

1
2
aws iam create-user --user-name external-secrets
aws iam add-user-to-group --group-name secret-readers --user-name external-secrets

Finally, we create a set of credentials for that user, and add it as a secret in kubernetes:

1
2
3
4
5
aws iam create-access-key --user-name external-secrets > creds.json
ACCESS_KEY=$(cat creds.json | jq -r .AccessKey.AccessKeyId)
SECRET_KEY=$(cat creds.json | jq -r .AccessKey.SecretAccessKey)

kubectl create secret generic aws-secret --from-literal=access-key=$ACCESS_KEY --from-literal=secret=$SECRET_KEY

Now, let’s add some secrets in our secret Store!

1
2
3
4
5
6
7
8
9
aws secretsmanager --region=eu-west-1 create-secret --name live/citizix-db-secret \
    --description "DB Secret for live citizix" \
    --secret-string '{
        "password": "SuperS3cure",
        "host": "citizix-db.cvytwhgsiext.eu-west-1.rds.amazonaws.com",
        "port": "5432",
        "db": "citizix",
        "user": "citizix"
    }'

Installing External Secrets

External secrets is available officially as a helm chart in this repo. To add the repo and install it as external-secrets in our cluster, use this:

1
2
helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
helm install external-secrets external-secrets/kubernetes-external-secrets

You can also install it with kubectl if you don’t want to use helm in your cluster by generating the manifests then applying:

1
helm template --include-crds --output-dir ./output_dir external-secrets/kubernetes-external-secrets

The generated kubernetes manifests will be in ./output_dir and can be applied to deploy kubernetes-external-secrets to the cluster.

Deploy a Secret Store

The first thing we need to do is define a SecretStore . A Secret Store is a namespaced resource that determines how your external Secret will be accessed from an authentication perspective. This resource is where all backend-related configuration is going to be stored, to allow external-secrets to reach out to Secrets-Manager.

It contains references to Secrets that have the credentials to access the external API. External Secrets also has a ClusterSecretStore which is a global or cluster-wide SecretStore that can be referenced from all namespaces to provide a central gateway to your secrets manager. In our case we will use a SecretStore.

Let us create a Secret store called aws-secret-manager that will use the earlier defined secret aws-secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secret-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-secret
            key: access-key
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret-access-key
EOF

NOTE: In case of a ClusterSecretStore, Be sure to provide namespace in accessKeyIDSecretRef and secretAccessKeySecretRef with the namespaces where the secrets reside.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secret-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-secret
            key: access-key
            namespace: external-secrets
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret-access-key
            namespace: external-secrets
EOF

We can confirm the secret store creation with this command:

1
2
3
4
$ kubectl get secretstore

NAME                    AGE   STATUS
aws-secret-manager      9s    Valid

Creating External Secrets

An ExternalSecret is a resource that declares the data you want to fetch from the external secrets manager. It will reference the SecretStore to know how to access sensitive data.

Let us create an external secret for our db secret. Here we are accessing the secret sm-citizix-db-secret and creating a target citizix-db-secret secret in kubernetes with a key db-url:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-citizix-db-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secret-manager
    kind: SecretStore
  target:
    name: citizix-db-secret
    creationPolicy: Owner
  data:
  - secretKey: db-url
    remoteRef:
      key: sm-citizix-db-secret
EOF

We can confirm that the defined external secret is working as expected:

1
2
3
4
$ kubectl get externalsecret

NAME                   STORE                   REFRESH INTERVAL   STATUS
es-citizix-db-secret   aws-secret-manager      1h                 SecretSynced

We can also check that it has created the respective kubernetes secret:

1
2
3
4
$ kubectl get secret

NAME                  TYPE                                  DATA   AGE
citizix-db-secret     Opaque                                1      62s

To confirm the content of the created secret, describe the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ k describe secret citizix-db-secret

Name:         citizix-db-secret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: 9ca2636815e1610df9f9458d03e3814d

Type:  Opaque

Data
====
db-url:  54 bytes

Extracting json values

We can extract json key values as secret data. External secret provides the dataFrom attribute that does just that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-json-citizix-db-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secret-manager
    kind: SecretStore
  target:
    name: json-citizix-db-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: sm-json-citizix-db-secret
EOF

We can confirm that the defined external secret is working as expected:

1
2
3
4
$ kubectl get es

NAME                        STORE                   REFRESH INTERVAL   STATUS
es-json-citizix-db-secret   aws-secret-manager      1h                 SecretSynced

We can also check that it has created the respective kubernetes secret:

1
2
3
4
$ kubectl get secret

NAME                     TYPE                                  DATA   AGE
json-citizix-db-secret   Opaque                                4      14s

To confirm the content of the created secret, describe the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$ kubectl describe secret json-citizix-db-secret

Name:         json-citizix-db-secret
Namespace:    default
Labels:       <none>
Annotations:  reconcile.external-secrets.io/data-hash: 9c7f95c6c59808128f2ab2630e980dee

Type:  Opaque

Data
====
db:        2 bytes
host:      9 bytes
password:  6 bytes
username:  7 bytes

Conclusion

In this guide, we learnt what External Secrets are and how we can leverage them to populate our kubernetes secrets from AWS secret manager.