Zitadel is an open-source identity and access management platform — a self-hosted OIDC/OAuth2 provider you can run instead of Keycloak, Auth0 or Okta. This guide walks through deploying Zitadel on Kubernetes with the official Helm chart, an external PostgreSQL database, Traefik ingress with TLS, secrets pulled from HashiCorp Vault, and a NetworkPolicy that does not quietly break the whole thing.

I wrote this after standing Zitadel up on a k3s cluster to replace Keycloak. The happy path is short. The traps are not obvious, and two of them cost me days — so they get their own sections here rather than a footnote.

In this guide you’ll: create the DNS record and database role Zitadel needs; store the master key and passwords in Vault and sync them with External Secrets; write the Helm values (including the one annotation that makes the Console work); apply a NetworkPolicy that allows DNS, the Kubernetes API and Postgres; deploy with helmfile; and verify OIDC discovery and the Console.

Prerequisites:

• A Kubernetes cluster (this was built on k3s, but any cluster works) with kubectl and Helm 3
• Traefik as the ingress controller — k3s bundles it in kube-system
• cert-manager with a working ClusterIssuer for Let’s Encrypt
• A reachable PostgreSQL 14+ server (external or in-cluster)
• Optional but recommended: HashiCorp Vault + External Secrets Operator, so no password ever lands in git
• Optional: helmfile, for declarative releases

Table of contents

• What Zitadel is, and why self-host it
• Architecture overview
• 1. Create the DNS record
• 2. Create the PostgreSQL role
• 3. Generate the master key and store the secrets
• 4. Sync the secret with External Secrets
• 5. Create the namespace
• 6. Write the NetworkPolicy
• 7. Write the Helm values
• 8. Deploy with helmfile
• 9. Verify the deployment
• Troubleshooting
• What to build next in the Console
• Frequently Asked Questions (FAQ)
• Related content
• Conclusion

What Zitadel is, and why self-host it

Zitadel is a cloud-native identity platform written in Go. It speaks OIDC, OAuth 2.0, SAML 2.0 and LDAP, supports multi-tenancy through organizations, and ships passwordless/WebAuthn, TOTP and per-application role management out of the box.

Compared with the alternative most people reach for first:

The event-sourced model is the real differentiator: every change is an immutable event, so you get an audit log for free rather than as an add-on. The cost is that Zitadel is PostgreSQL-only — there is no other supported backend.

Two properties of Zitadel drive almost every decision below, so it is worth stating them up front:

1. The Console talks gRPC-web over HTTP/2. Your proxy must forward HTTP/2, not downgrade to HTTP/1.1.
2. The master key encrypts everything at rest and must be exactly 32 bytes. Lose it and the database is unreadable.

Architecture overview

Here is what we are building:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

                    Internet
                       |
                   [ Traefik ]  <- terminates TLS (cert-manager / Let's Encrypt)
                       |
        +--------------+---------------+
        |  h2c (HTTP/2 cleartext)      |  HTTP/1.1
        v                              v
  [ zitadel :8080 ]              [ zitadel-login :3000 ]
   API + Console                  Login UI (v2), served at /ui/v2/login
        |
        | TCP 5432
        v
  [ PostgreSQL ]   external server, or in-cluster

  Secrets: Vault (kv/infra/zitadel-secrets)
             -> External Secrets Operator
               -> Secret "zitadel-secrets"  (masterkey + config-yaml)

Three things are worth noticing in that sketch.

Zitadel v4 splits the login UI into its own Deployment. The chart deploys zitadel (API + Console, port 8080) and zitadel-login (port 3000). They share a hostname: the login UI is served under /ui/v2/login. That path needs its own Ingress — the main Ingress only routes the API. Miss this and every login attempt 404s.

TLS stops at Traefik. Zitadel runs with TLS.Enabled: false inside the pod and speaks h2c — HTTP/2 cleartext — on :8080. It does not serve HTTP/1.1 on that port at all.

Nothing sensitive is in the values file. The Helm values point at an existing Kubernetes Secret; External Secrets Operator renders that Secret from Vault.

1. Create the DNS record

Zitadel bakes its external domain into issued tokens (ExternalDomain), so pick the hostname before you deploy and do not change it casually — changing it later invalidates issuer URLs your clients have already discovered.

Point an A record at your ingress load balancer or node IP:

1
2
3
4
5

# whatever your DNS provider is — Route 53, Cloudflare, a zone file
# zitadel.example.com.  A  203.0.113.5

dig +short zitadel.example.com
# -> 203.0.113.5

Do not continue until dig returns the right address. cert-manager’s HTTP-01 challenge will fail otherwise, and you will end up debugging TLS when the problem is DNS.

2. Create the PostgreSQL role

Zitadel needs two database identities: an admin connection (runs migrations, creates the database) and a user connection (the running application). They can be the same role, which keeps things simple and keeps superuser credentials out of the cluster entirely.

Create a login role with CREATEDB — Zitadel’s init Job creates the zitadel database itself on first run:

1

CREATE ROLE zitadel WITH LOGIN CREATEDB PASSWORD 'a-strong-password-here';

Confirm the cluster can actually reach the server. Use a pod that lives long enough to test with:

1
2

kubectl run netcheck --image=nicolaka/netshoot --restart=Never -it --rm -- \
  nc -zv 203.0.113.10 5432

Security note. If your Postgres server sits outside the cluster and the traffic crosses a public network, enable TLS on the server and set the SSL mode to require in the values (shown in step 7). An identity provider’s database is the last place to accept plaintext replication of every credential change. If the path is a private network or a VPN, disable is defensible — but decide it deliberately rather than inheriting the default.

3. Generate the master key and store the secrets

The master key encrypts secrets Zitadel stores in the database. It must be exactly 32 bytes, and this is where most first attempts die:

1
2

openssl rand -hex 16     # 32 hex characters = 32 bytes  ✅
openssl rand -base64 16  # 24 characters                 ❌

openssl rand -base64 16 produces 24 characters, and the setup Job fails with masterkey must be 32 bytes, but is 24. Use -hex 16.

Generate the key and store all three secrets in Vault:

1
2
3
4
5
6
7
8
9

MASTERKEY=$(openssl rand -hex 16)
echo -n "$MASTERKEY" | wc -c    # must print 32

vault kv put kv/infra/zitadel-secrets \
  masterkey="$MASTERKEY" \
  db-password='a-strong-password-here' \
  admin-password='a-strong-bootstrap-admin-password'

vault kv get kv/infra/zitadel-secrets

Back the master key up somewhere outside the cluster. Without it, an existing Zitadel database is unrecoverable — there is no reset flow.

4. Sync the secret with External Secrets

The Helm chart reads two keys from one Kubernetes Secret:

• masterkey — the 32-byte key
• config-yaml — a YAML fragment deep-merged on top of the ConfigMap config, which is where the passwords go

External Secrets Operator can template the whole config-yaml document, injecting only the sensitive fields from Vault. Save this as external-secret.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: zitadel-secrets
  namespace: zitadel
spec:
  refreshInterval: 5m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend
  target:
    name: zitadel-secrets
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        masterkey: "{{ .masterkey }}"
        config-yaml: |
          Database:
            Postgres:
              User:
                Password: "{{ .dbPassword }}"
              Admin:
                Password: "{{ .dbPassword }}"
          FirstInstance:
            Org:
              Name: ExampleOrg
              Human:
                UserName: admin
                FirstName: Example
                LastName: Admin
                Email:
                  Address: [email protected]
                  Verified: true
                Password: "{{ .adminPassword }}"
                PasswordChangeRequired: true
  data:
    - secretKey: masterkey
      remoteRef:
        key: infra/zitadel-secrets
        property: masterkey
    - secretKey: dbPassword
      remoteRef:
        key: infra/zitadel-secrets
        property: db-password
    - secretKey: adminPassword
      remoteRef:
        key: infra/zitadel-secrets
        property: admin-password

The FirstInstance block bootstraps the initial organization and human admin. PasswordChangeRequired: true forces a password change at first login, which is what you want for a password that briefly existed in your shell history.

If you are not running Vault

The chart does not care where the Secret comes from — it just reads it. Without ESO, create the same Secret by hand:

1
2
3

kubectl -n zitadel create secret generic zitadel-secrets \
  --from-literal=masterkey="$MASTERKEY" \
  --from-file=config-yaml=./config.yaml

…where config.yaml is the same YAML document as the template above with real values substituted. Just keep that file out of git. If you want the Vault-backed version, see HashiCorp Vault on Kubernetes: A Comprehensive Production Guide and How to Use External Secrets With AWS Secrets Manager for the operator setup.

5. Create the namespace

Create it explicitly rather than letting Helm do it, so you control the labels NetworkPolicy selectors depend on. Save as namespace.yml:

1
2
3
4
5
6

apiVersion: v1
kind: Namespace
metadata:
  name: zitadel
  labels:
    name: zitadel

6. Write the NetworkPolicy

This section is the one that cost me the most time, so read it before you write your own policy.

NetworkPolicy is enforced on k3s. k3s bundles a kube-router policy controller alongside flannel. Because there is no Calico or Cilium pod to see in the cluster, it is very easy to conclude that policy is inert. It is not. And there are three specific traps.

Trap 1: rules are programmed 15–30 seconds after a pod starts. A short-lived debug pod frequently finishes its work before its rules land, so it looks completely unrestricted while a real workload is firewalled. Never test a policy with a throwaway kubectl run ... --rm one-liner. Use a pod that lives for minutes.

Trap 2: select namespaces by kubernetes.io/metadata.name. The API server sets that label on every namespace automatically. Many namespaces also carry a bare name: <ns> label — but kube-system does not. A selector like namespaceSelector: {matchLabels: {name: kube-system}} therefore matches nothing, and the rule is silently dead. That single mistake blocked DNS egress for every pod the policy selected. The symptom was zitadel-login’s wait-for-zitadel init container CrashLooping for days with lookup zitadel: i/o timeout — it could not resolve anything at all.

Trap 3: the Kubernetes API server is not a pod. It is reached at a ClusterIP (10.43.0.1:443 on k3s) that DNATs to a node on :6443. No namespaceSelector or podSelector can ever match it — it must be an ipBlock, and you should allow both ports, since the policy may be evaluated before or after DNAT. The chart’s setup Job can run kubectl sidecars; without this rule they hang until activeDeadlineSeconds and fail the entire release.

Here is the working policy. Save as network-policy.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: zitadel-network-policy
  namespace: zitadel
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: zitadel
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Traefik. On k3s it runs in kube-system, not a `traefik` namespace.
    # API :8080 (h2c) and the v2 login UI :3000.
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 3000
    # Intra-namespace (zitadel <-> login). Kubelet probes come from the node
    # and are not subject to NetworkPolicy, so they need no rule.
    - from:
        - podSelector: {}
  egress:
    # DNS to CoreDNS. Without this, pods resolve NOTHING.
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Intra-namespace: login -> zitadel API :8080, and back.
    - to:
        - podSelector: {}
    # Kubernetes API server — must be an ipBlock, both ports.
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 6443
    # PostgreSQL. External host, hence ipBlock.
    - to:
        - ipBlock:
            cidr: 203.0.113.10/32
      ports:
        - protocol: TCP
          port: 5432

A debugging tip that saves hours: the kubelet’s liveness and readiness probes originate from the node and bypass NetworkPolicy entirely. So does kubectl port-forward. A pod can sit there 1/1 Ready, answer your port-forward perfectly, and still be unreachable from every other pod in the cluster. If probes pass but pod-to-pod traffic fails, suspect NetworkPolicy first.

7. Write the Helm values

Now the chart configuration. Save as values.yml:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111

replicaCount: 2

zitadel:
  # 32-byte master key. The chart reads key "masterkey" from this Secret.
  masterkeySecretName: zitadel-secrets

  # Sensitive config (DB passwords + the FirstInstance admin) lives in the
  # SAME Secret under key "config-yaml". Zitadel deep-merges it on top of
  # configmapConfig below.
  configSecretName: zitadel-secrets
  configSecretKey: config-yaml

  configmapConfig:
    ExternalDomain: zitadel.example.com
    ExternalSecure: true   # behind a TLS-terminating proxy
    ExternalPort: 443
    TLS:
      Enabled: false       # no in-pod TLS; Traefik speaks h2c to :8080

    # Bootstrap only a human admin. See the note below on the default
    # machine user.
    FirstInstance:
      Org:
        Machine: null

    Database:
      Postgres:
        Host: "203.0.113.10"
        Port: 5432
        Database: zitadel
        User:
          Username: zitadel
          SSL:
            Mode: disable      # use "require" if the path is not private
        Admin:
          Username: zitadel
          SSL:
            Mode: disable

# THE annotation. Traefik must speak HTTP/2 cleartext to the backend or the
# Console's gRPC-web calls fail.
service:
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: h2c

ingress:
  enabled: true
  className: traefik
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod-issuer
  hosts:
    - host: zitadel.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: zitadel-tls
      hosts:
        - zitadel.example.com

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 512Mi

pdb:
  enabled: true
  minAvailable: 1

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: zitadel
          topologyKey: kubernetes.io/hostname

# Zitadel v4 ships a SEPARATE login UI (its own Deployment/Service on :3000),
# served under the same domain at /ui/v2/login. It needs its own Ingress —
# the main ingress above only routes the API — or login/Console will 404.
login:
  replicaCount: 2
  pdb:
    enabled: true
    minAvailable: 1
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 256Mi
  ingress:
    enabled: true
    className: traefik
    hosts:
      - host: zitadel.example.com
        paths:
          - path: /ui/v2/login
            pathType: Prefix
    # Reuse the cert issued by the main ingress. No cluster-issuer annotation
    # here, so cert-manager does not create a second Certificate for the host.
    tls:
      - secretName: zitadel-tls
        hosts:
          - zitadel.example.com

Four decisions in there are worth explaining.

h2c is the critical detail

1
2
3

service:
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: h2c

The Zitadel Console is a single-page app that talks to the API over gRPC-web, which requires HTTP/2. With TLS terminated at Traefik, Traefik must speak HTTP/2 cleartext to the pod. Without this annotation you get the classic symptom: the login page renders fine, and the Console is blank or throws errors on every gRPC call. If you use NGINX Ingress instead, the equivalent is nginx.ingress.kubernetes.io/backend-protocol: "GRPC".

FirstInstance runs exactly once

FirstInstance is applied only on the first setup of a fresh instance. Every subsequent run ignores the whole block. So editing the admin password, the org name or the machine user in your values has no effect on a live instance — those changes have to be made in the Console or through the API. Worth knowing before you spend an afternoon wondering why a password change in git did nothing.

Disabling the default machine user

FirstInstance.Org.Machine: null disables the chart’s default iam-admin machine user. That user is not harmful, but it makes the setup Job spawn two “machinekey/PAT writer” kubectl sidecars that run in-cluster to export the generated key into a Secret. Those sidecars are exactly what hangs if your NetworkPolicy blocks API-server egress. If you build your configuration in the Console rather than through an API client, you do not need it. If you do want to automate against the API later, re-enable it — the policy in step 6 already permits it.

Two replicas, PDB, anti-affinity

An identity provider is a hard dependency for everything behind it. Two replicas spread across nodes plus a PodDisruptionBudget means a node drain does not log everyone out.

8. Deploy with helmfile

You can install directly with Helm:

1
2
3
4
5
6
7

helm repo add zitadel https://charts.zitadel.com
helm repo update
helm upgrade --install zitadel zitadel/zitadel \
  --version 10.0.4 \
  --namespace zitadel \
  --values values.yml \
  --timeout 10m

I prefer helmfile, because it keeps the chart version, the values and the supporting manifests in one reviewable file. Save as helmfile.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

helmDefaults:
  createNamespace: true
  timeout: 600
  wait: true
  # NOTE: intentionally NOT atomic. This chart's setup/cleanup run as Helm hook
  # Jobs; on a hook failure `atomic` uninstalls the whole release, which fires
  # the post-delete cleanup hook Job — compounding one failure into a messy
  # teardown and leaving nothing to inspect. Without atomic, a failed run stays
  # put (release marked failed) so it can be diagnosed and re-applied.

repositories:
  - name: zitadel
    url: https://charts.zitadel.com

releases:
  - name: zitadel
    namespace: zitadel
    chart: zitadel/zitadel
    # Chart 10.0.4 -> Zitadel appVersion v4.15.3. Pin explicitly; bump deliberately.
    version: "10.0.4"
    values:
      - ./values.yml
    hooks:
      - events: ["presync"]
        command: "kubectl"
        args: ["apply", "-f", "./namespace.yml"]
      - events: ["presync"]
        command: "kubectl"
        args: ["apply", "-f", "./external-secret.yml"]
      - events: ["presync"]
        command: "kubectl"
        args: ["apply", "-f", "./network-policy.yml"]

That atomic comment is not decoration. With atomic: true, a failing setup hook uninstalls the release, which triggers the post-delete cleanup Job, which deletes the evidence you need to work out why the setup failed. Leave it off until the release installs cleanly.

The recommended flow materializes the Secret before the chart’s init and setup Jobs run, since those Jobs mount it. This removes a race with External Secrets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# 1) namespace + ExternalSecret, then wait for the real Secret to appear
kubectl apply -f namespace.yml -f external-secret.yml
kubectl -n zitadel wait --for=condition=Ready \
  externalsecret/zitadel-secrets --timeout=120s

# sanity-check the templated config merged correctly
kubectl -n zitadel get secret zitadel-secrets \
  -o jsonpath='{.data.config-yaml}' | base64 -d

# 2) preview, then deploy
helmfile diff
helmfile apply

The presync hooks mean a bare helmfile apply will usually work on its own. Doing it explicitly just removes the timing race on a first install.

9. Verify the deployment

Work through these in order. Each one isolates a different layer.

Pods running, Jobs completed:

1

kubectl -n zitadel get pods,jobs

You want zitadel and zitadel-login pods Running, and the init and setup Jobs Completed. If setup is still running after a few minutes, go straight to its logs:

1

kubectl -n zitadel logs job/zitadel-setup --all-containers --tail=100

Master key length — the single most common install failure:

1
2
3

kubectl -n zitadel get secret zitadel-secrets \
  -o jsonpath='{.data.masterkey}' | base64 -d | wc -c
# must print exactly 32

OIDC discovery, which proves ingress, TLS and the API are all working:

1
2

curl -s https://zitadel.example.com/.well-known/openid-configuration | jq .issuer
# "https://zitadel.example.com"

The Console — and load it in a real browser, not curl. This is the h2c test:

1

open https://zitadel.example.com/ui/console

Log in as admin with the bootstrap password from Vault. You will be forced to change it immediately. If the login page works but the Console is blank or errors, the h2c annotation is missing or Traefik is not honouring it.

Troubleshooting

The Console is blank, the login page is fine. h2c. Confirm the annotation landed on the Service, not just in your values file:

1

kubectl -n zitadel get svc zitadel -o jsonpath='{.metadata.annotations}' | jq

masterkey must be 32 bytes, but is 24. You used openssl rand -base64 16. Regenerate with openssl rand -hex 16. On a fresh install you can update the Vault entry and re-apply; on an instance that has already stored encrypted data, you cannot change the master key.

zitadel-login CrashLoops with lookup zitadel: i/o timeout. DNS egress is blocked. Your NetworkPolicy’s DNS rule almost certainly selects kube-system by the wrong label. Use kubernetes.io/metadata.name, not a bare name label.

The setup Job hangs and dies at 300 seconds. Its kubectl sidecars cannot reach the API server. Add the ipBlock egress rule for ports 443 and 6443, or disable the default machine user with FirstInstance.Org.Machine: null.

Port 8080 looks closed when you probe it by hand. Zitadel speaks h2c only, so plain HTTP/1.1 gets you nothing. Use prior-knowledge HTTP/2:

1
2
3

kubectl -n zitadel port-forward svc/zitadel 8080:8080 &
curl --http2-prior-knowledge -s \
  http://localhost:8080/.well-known/openid-configuration | jq .issuer

Remember that port-forward bypasses NetworkPolicy — a successful port-forward proves the app is healthy, not that the network is.

You changed the admin password / org name in values and nothing happened. FirstInstance runs once, on the first setup of a fresh instance. Make the change in the Console.

A Helm release is stuck in a failed state. Do not reach for --atomic to clean it up. Inspect the hook Job logs, fix the cause, and re-run helmfile apply.

What to build next in the Console

Standing the server up is Phase 1. To make it useful as an SSO provider you then build the model — a project, roles, and one OIDC application per client. That part is done in the Console.

The piece most people hit immediately is the shape of the roles claim. oauth2-proxy (oidc_groups_claim = "groups"), ArgoCD’s policy.csv and Grafana’s RBAC all expect a flat array:

1

"groups": ["sysAdmins"]

Zitadel does not emit that. It emits a nested object keyed by role name:

1
2
3

"urn:zitadel:iam:org:project:roles": {
  "sysAdmins": { "<orgId>": "example.zitadel.example.com" }
}

None of those consumers can read it. Zitadel’s answer is Actions — JavaScript that runs during token issuance. This one flattens roles into a groups claim:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

function flattenGroups(ctx, api) {
  if (ctx.v1.user.grants == undefined || ctx.v1.user.grants.count == 0) {
    return;
  }

  let groups = [];
  ctx.v1.user.grants.grants.forEach((grant) => {
    grant.roles.forEach((role) => {
      groups.push(role);
    });
  });

  api.v1.claims.setClaim("groups", groups);
}

Four gotchas, each of which fails silently:

1. The action name in the Console must be exactly flattenGroups. Zitadel v1 Actions invoke the function whose name matches the action name. A mismatch produces no error — the claim simply never appears.
2. Bind it to both triggers of the Complement Token flow — “Pre access token creation” and “Pre userinfo creation”. Binding only the first is the classic “the claim is in the token but not in /userinfo” bug, and oauth2-proxy reads /userinfo.
3. api.v1.claims.setClaim() will not overwrite an existing claim. It only sets the key if it is absent.
4. The client must request the audience scope urn:zitadel:iam:org:project:id:<projectId>:aud. Without it, ctx.v1.user.grants is empty, the function returns early, and you get no claim and no error.

Verify the claim before you cut any application over to Zitadel. Getting a real token and inspecting it takes two minutes and saves an outage.

Frequently Asked Questions (FAQ)

Can I run Zitadel with an in-cluster PostgreSQL?

Yes. The chart can deploy a Postgres dependency for you, and pointing at an in-cluster Postgres you manage separately works fine too. For production, use a database with real backups and a restore procedure you have actually tested — Zitadel’s entire state lives there.

Does Zitadel work behind NGINX Ingress instead of Traefik?

Yes. The requirement is HTTP/2 to the backend, not Traefik specifically. On NGINX Ingress use nginx.ingress.kubernetes.io/backend-protocol: "GRPC" instead of the serversscheme: h2c annotation. Everything else in this guide is unchanged.

What happens if I lose the master key?

The database becomes unreadable. There is no recovery path and no reset flow. Store it in a secret manager, back it up outside the cluster, and treat it with the same care as a root CA private key.

Can I change ExternalDomain after deployment?

Technically yes, but it changes the OIDC issuer, which invalidates configuration in every client that has already done discovery, and can break existing sessions. Pick the hostname before the first deploy.

How do I upgrade the chart safely?

Pin the chart version in your values or helmfile and bump deliberately. Read the Zitadel release notes for database migrations, run helmfile diff first, and remember the setup Job runs migrations on upgrade. Snapshot the database before a major version bump.

Is one replica enough?

For a lab, yes. For anything real, no — everything behind your IdP fails when it does. Two replicas with a PodDisruptionBudget and pod anti-affinity is a low-cost floor.

Related content

• HashiCorp Vault on Kubernetes: A Comprehensive Production Guide
• How to Use External Secrets With AWS Secrets Manager
• How to Set Up OAuth2 Proxy for Central Authentication in Kubernetes with GitLab
• How to Setup a Kubernetes Cluster with K3S in Rocky Linux 8/9
• How to Set Up Coraza WAF with Traefik on k3s Kubernetes
• How to Configure Ingress TLS/SSL Certificates in Kubernetes
• How to Manage Postgres Users and Database Permissions
• How to Create and Work With Helm Charts in Kubernetes

Conclusion

You now have Zitadel running on Kubernetes: two replicas behind Traefik with Let’s Encrypt TLS, an external PostgreSQL backend, secrets sourced from Vault so nothing sensitive sits in git, and a NetworkPolicy that restricts traffic without severing DNS.

If you take three things away from this guide, make them these:

• h2c on the backend, or the Console never loads.
• openssl rand -hex 16 for the master key, and back it up outside the cluster.
• NetworkPolicy is enforced on k3s — select namespaces with kubernetes.io/metadata.name, use an ipBlock for the API server, and never trust a throwaway pod to tell you a policy is working.

From here, build the project, roles and applications in the Console, verify the groups claim, then start moving clients across one at a time.