DevOps Kubernetes Infrastructure Configuration Management

Setting Up Salt Master on Kubernetes: Complete Production Guide

Comprehensive guide to deploying SaltStack Master on Kubernetes with GitFS, NodePort/LoadBalancer configuration, firewall rules, and minion setup. Production-ready configuration management at scale.

Running SaltStack Master in Kubernetes provides a modern, scalable approach to infrastructure management. This comprehensive guide walks you through deploying Salt Master on Kubernetes, from basics to production-ready configuration, including service exposure, firewall setup, and minion configuration.

Table of Contents

  1. Introduction
  2. Why Run Salt Master on Kubernetes?
  3. Prerequisites
  4. Understanding Salt Architecture
  5. Kubernetes Deployment Overview
  6. Step-by-Step Deployment
  7. Exposing Salt Master: NodePort vs LoadBalancer
  8. Firewall Configuration
  9. Setting Up Salt Minions
  10. GitFS Integration
  11. Common Pitfalls and Things to Watch Out For
  12. Troubleshooting
  13. Production Considerations

Introduction

SaltStack (now part of Broadcom’s portfolio) is a powerful configuration management and remote execution system. While traditionally deployed on VMs or bare metal, running Salt Master in Kubernetes offers several advantages including easier management, automatic recovery, and consistent deployment patterns.

According to recent data, while most implementations use Salt to deploy Kubernetes clusters, running Salt Master as a containerized workload is gaining traction for modern infrastructure management patterns.

Why Run Salt Master on Kubernetes?

Running Salt Master on Kubernetes provides several benefits:

  • High Availability: Kubernetes automatically restarts failed pods
  • Consistent Deployment: Infrastructure as code using YAML manifests
  • Resource Management: CPU and memory limits/requests for predictable performance
  • Persistent Storage: PersistentVolumes ensure minion keys and configurations survive pod restarts
  • GitOps Integration: Configuration managed through version control
  • Scalability: Easy to scale and manage alongside other workloads
  • Portability: Run on any Kubernetes cluster (on-prem, cloud, or hybrid)

Prerequisites

Before starting, ensure you have:

  1. Kubernetes Cluster (v1.19 or later)

    • Running cluster with kubectl access
    • Admin privileges to create namespaces and resources

  2. Storage Provisioner

    • Dynamic PersistentVolume provisioner or ability to create PVs manually
    • At least 20GB of available storage

  3. External Access Method

    • LoadBalancer support (MetalLB, cloud provider LB) OR
    • Ability to use NodePort with firewall rules

  4. Network Access

    • Ports 4505 and 4506 accessible from minions to master
    • If using NodePort: Ports in 30000-32767 range accessible

  5. Optional: Git Repository

    • For storing Salt states and formulas (GitFS)
    • SSH key or access token for private repositories

Understanding Salt Architecture

Salt uses a master-minion architecture with these key components:

Communication Ports

  • Port 4505 (Publish Port): Master publishes commands to minions via ZeroMQ
  • Port 4506 (Return Port): Minions return results and establish bi-directional communication
  • Port 8000 (API Port): Optional Salt API for REST-based management

Communication Flow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

┌──────────────────────────────────────────────────┐
│              Salt Master (Kubernetes)            │
│  ┌────────────────────────────────────────────┐  │
│  │  Port 4505: Publish (Master → Minions)    │  │
│  │  Port 4506: Return (Minions ↔ Master)     │  │
│  │  Port 8000: REST API (Optional)           │  │
│  └────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────┘
            ↓                    ↓
    ┌───────────┐         ┌───────────┐
    │  Minion 1 │         │  Minion 2 │
    └───────────┘         └───────────┘

Important: Communication is always initiated by minions. The master never initiates connections to minions, meaning you only need to allow inbound connections to the master on ports 4505 and 4506. This is a critical security consideration that simplifies firewall configuration.

Kubernetes Deployment Overview

Our Salt Master deployment consists of these Kubernetes resources:

1
2
3
4
5
6

├── Namespace (salt-master)
├── ConfigMap (master configuration + GitFS settings)
├── Secret (API credentials, SSH keys)
├── PersistentVolumeClaims (data, keys, logs, config)
├── Deployment (Salt Master pod)
└── Service (LoadBalancer or NodePort)

The deployment uses the cdalvaro/docker-salt-master Docker image, which includes:

  • Salt Master with all dependencies
  • PyGit2 for GitFS support
  • Salt API (CherryPy)
  • Proper init system and logging

Step-by-Step Deployment

Step 1: Create Namespace

First, create a dedicated namespace for Salt Master:

1
2
3
4
5
6
7
8

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: salt-master
  labels:
    name: salt-master
    app: salt-master

Apply the namespace:

1

kubectl apply -f namespace.yaml

Why a dedicated namespace? Isolating Salt Master in its own namespace provides better resource management, security boundaries, and easier cleanup if needed.

Step 2: Configure Persistent Storage

Salt Master requires persistent storage for:

  • Keys: Minion authentication keys (critical - losing these requires re-keying all minions)
  • Data: States, formulas, and pillar data
  • Logs: Master logs for troubleshooting
  • Config: Additional configuration files
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: salt-master-data
  namespace: salt-master
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  # Optionally specify storage class
  # storageClassName: standard
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: salt-master-keys
  namespace: salt-master
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: salt-master-logs
  namespace: salt-master
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: salt-master-config
  namespace: salt-master
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Apply the PVCs:

1

kubectl apply -f pvc.yaml

Storage Considerations:

  • Use your cluster’s default storage class or specify one explicitly
  • For production, use storage with backup capabilities
  • Keys volume is most critical - consider using storage with replication
  • Size data volume based on expected number of states and formulas

Step 3: Create ConfigMap for Salt Master Configuration

The ConfigMap contains the main Salt Master configuration, including GitFS settings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: salt-master-config
  namespace: salt-master
  labels:
    app: salt-master
data:
  master: |
    # Salt Master Configuration

    # Interface to bind to (0.0.0.0 = all interfaces)
    interface: 0.0.0.0

    # Publish and request server ports
    publish_port: 4505
    ret_port: 4506

    # User to run salt as (set by Docker image)
    user: salt

    # Auto-accept minion keys
    # CRITICAL: Set to False in production!
    # When False, you must manually accept each minion key
    auto_accept: False

    # Logging configuration
    log_level: info
    log_level_logfile: info

    # File roots - where Salt looks for state files
    file_roots:
      base:
        - /home/salt/data/srv/salt
        - /home/salt/data/srv/formulas

    # Pillar roots - where Salt looks for pillar data
    pillar_roots:
      base:
        - /home/salt/data/srv/pillar

    # Fileserver backend configuration
    # Order matters: roots is checked first, then git
    fileserver_backend:
      - roots
      - git

    # GitFS provider (pygit2 is recommended and included in the image)
    gitfs_provider: pygit2

    # GitFS remotes - Git repositories containing Salt states
    gitfs_remotes:
      # Example with SSH authentication
      - [email protected]:yourusername/salt-states.git:
        - base: main
        - root: salt  # Subdirectory in repo containing states
        - privkey: /etc/salt/ssh/deploy_key
        - pubkey: /etc/salt/ssh/deploy_key.pub

      # Example with HTTPS (for public repos)
      # - https://github.com/yourusername/salt-states.git:
      #   - base: main
      #   - root: salt

    # GitFS update interval (seconds)
    # How often to check Git for changes
    gitfs_update_interval: 300

    # GitFS branches to make available
    gitfs_branch_whitelist:
      - main
      - production
      - staging

    # GitFS SSL verification
    gitfs_ssl_verify: True

    # Worker threads - adjust based on number of minions
    # Rule of thumb: 1 worker per 20-50 minions
    worker_threads: 5

    # Timeout settings (seconds)
    timeout: 30
    gather_job_timeout: 30

    # Keep job results for 24 hours
    keep_jobs: 24

    # Enable job cache
    job_cache: True

    # Enable mine for service discovery
    mine_enabled: True

    # Optional: Reactor system for event-driven automation
    # reactor:
    #   - 'salt/minion/*/start':
    #     - /home/salt/data/config/reactor/minion_start.sls

Configuration Explained:

  • auto_accept: False: Critical security setting. In production, always manually accept minion keys to prevent unauthorized systems from connecting
  • file_roots and pillar_roots: Define where Salt looks for states and pillar data. Multiple paths allow organizing code from different sources
  • fileserver_backend: Order defines priority. “roots” checks local filesystem first, then “git” checks GitFS repositories
  • worker_threads: Each worker handles minion connections. More workers = more concurrent minion management
  • timeout settings: Adjust based on network latency and expected command execution times

Apply the ConfigMap:

1

kubectl apply -f configmap.yaml

Step 4: Create Secrets

Create a secret for Salt API credentials:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: salt-api-credentials
  namespace: salt-master
type: Opaque
stringData:
  username: "salt_api"
  password: "CHANGE_THIS_SECURE_PASSWORD"

IMPORTANT: Never commit secrets to Git! For production:

1
2
3
4
5

# Create secret imperatively
kubectl create secret generic salt-api-credentials \
  --from-literal=username=salt_api \
  --from-literal=password=$(openssl rand -base64 32) \
  -n salt-master

If using private Git repositories with SSH, create SSH key secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Generate SSH key for GitFS (if you don't have one)
ssh-keygen -t ed25519 -C "salt-gitfs" -f ./salt_deploy_key -N ""

# Add public key to your Git repository (GitHub/GitLab deploy keys)

# Create Kubernetes secret
kubectl create secret generic salt-ssh-keys \
  --from-file=salt_deploy_key=./salt_deploy_key \
  --from-file=salt_deploy_key.pub=./salt_deploy_key.pub \
  -n salt-master

# Securely delete local keys after creation
shred -u ./salt_deploy_key ./salt_deploy_key.pub

Step 5: Create the Deployment

The Deployment defines the Salt Master pod specification:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: salt-master
  namespace: salt-master
  labels:
    app: salt-master
spec:
  replicas: 1
  # Recreate strategy ensures only one master runs at a time
  # RollingUpdate can cause split-brain with multiple masters
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: salt-master
  template:
    metadata:
      labels:
        app: salt-master
    spec:
      # Init container to set up SSH keys with correct permissions
      initContainers:
        - name: setup-ssh-keys
          image: busybox:latest
          command:
            - sh
            - -c
            - |
              # Copy keys from read-only secret mount
              cp /tmp/ssh-keys-ro/* /etc/salt/ssh/

              # Set correct permissions (SSH is strict about this)
              chmod 600 /etc/salt/ssh/salt_deploy_key
              chmod 644 /etc/salt/ssh/salt_deploy_key.pub

              # Set ownership to salt user (UID 1000)
              chown -R 1000:1000 /etc/salt/ssh
          volumeMounts:
            - name: ssh-keys-ro
              mountPath: /tmp/ssh-keys-ro
              readOnly: true
            - name: ssh-keys
              mountPath: /etc/salt/ssh

      containers:
        - name: salt-master
          image: ghcr.io/cdalvaro/docker-salt-master:latest
          imagePullPolicy: Always

          ports:
            - name: publish
              containerPort: 4505
              protocol: TCP
            - name: request
              containerPort: 4506
              protocol: TCP
            - name: api
              containerPort: 8000
              protocol: TCP

          env:
            # Log level (debug, info, warning, error)
            - name: SALT_LOG_LEVEL
              value: "info"

            # Enable Salt API for REST access
            - name: SALT_API_ENABLED
              value: "True"

            # Salt API credentials from secret
            - name: SALT_API_USER
              valueFrom:
                secretKeyRef:
                  name: salt-api-credentials
                  key: username

            - name: SALT_API_USER_PASS
              valueFrom:
                secretKeyRef:
                  name: salt-api-credentials
                  key: password

            # Auto-restart master when config changes
            - name: SALT_RESTART_MASTER_ON_CONFIG_CHANGE
              value: "True"

          volumeMounts:
            # Master configuration from ConfigMap
            - name: master-config
              mountPath: /home/salt/data/config/gitfs.conf
              subPath: master

            # Persistent volumes
            - name: salt-data
              mountPath: /home/salt/data/srv

            - name: salt-keys
              mountPath: /home/salt/data/keys

            - name: salt-logs
              mountPath: /home/salt/data/logs

            - name: salt-config
              mountPath: /home/salt/data/config

            # SSH keys for GitFS
            - name: ssh-keys
              mountPath: /etc/salt/ssh

          # Resource limits prevent one container from consuming all node resources
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "1000m"

          # Liveness probe - restarts container if failing
          livenessProbe:
            tcpSocket:
              port: 4505
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3

          # Readiness probe - removes from service if not ready
          readinessProbe:
            tcpSocket:
              port: 4506
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3

      volumes:
        - name: master-config
          configMap:
            name: salt-master-config

        - name: salt-data
          persistentVolumeClaim:
            claimName: salt-master-data

        - name: salt-keys
          persistentVolumeClaim:
            claimName: salt-master-keys

        - name: salt-logs
          persistentVolumeClaim:
            claimName: salt-master-logs

        - name: salt-config
          persistentVolumeClaim:
            claimName: salt-master-config

        - name: ssh-keys-ro
          secret:
            secretName: salt-ssh-keys
            defaultMode: 0400

        - name: ssh-keys
          emptyDir: {}

      # Security context - run as salt user (UID 1000)
      securityContext:
        fsGroup: 1000

Deployment Configuration Explained:

  • strategy: Recreate: Ensures only one Salt Master runs at a time. RollingUpdate could cause issues with minion connections
  • initContainer: Sets up SSH keys with correct permissions before main container starts
  • Resources: Adjust based on minion count. Start conservative and monitor actual usage
  • Probes: Ensure pod is healthy and ready to accept connections
  • fsGroup: 1000: Ensures persistent volumes are writable by salt user

Apply the deployment:

1

kubectl apply -f deployment.yaml

Step 6: Verify Pod is Running

Check the pod status:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Check pod status
kubectl get pods -n salt-master

# Expected output:
# NAME                           READY   STATUS    RESTARTS   AGE
# salt-master-xxxxxxxxxx-xxxxx   1/1     Running   0          2m

# View logs
kubectl logs -n salt-master -l app=salt-master -f

# Check Salt Master is ready
kubectl exec -n salt-master deployment/salt-master -- salt-run manage.status

Common startup issues:

  • Pod in Pending: Check PVC status with kubectl get pvc -n salt-master
  • Pod in CrashLoopBackOff: Check logs for configuration errors
  • SSH key errors: Verify secret was created correctly

Exposing Salt Master: NodePort vs LoadBalancer

Salt minions need to reach the master on ports 4505 and 4506. Kubernetes provides multiple ways to expose services externally.

Understanding Service Types

According to Kubernetes best practices, there are several ways to expose services:

  • ClusterIP: Internal only (default) - not suitable for external minions
  • NodePort: Opens static ports (30000-32767) on all nodes
  • LoadBalancer: Provisions external load balancer (cloud provider specific)
  • Ingress: HTTP/HTTPS only - not suitable for Salt (requires TCP/UDP)

LoadBalancer is the standard way to expose services to the internet, providing a single IP address for all traffic.

Advantages:

  • Simple configuration for minions (single IP)
  • Automatic load balancing if running multiple masters
  • Health checks handled by load balancer
  • Standard ports (4505/4506) work without modification

Disadvantages:

  • Requires cloud provider or MetalLB
  • Cost (one load balancer per service)
  • May not be available in all environments
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# service.yaml (LoadBalancer)
apiVersion: v1
kind: Service
metadata:
  name: salt-master
  namespace: salt-master
  labels:
    app: salt-master
spec:
  type: LoadBalancer
  selector:
    app: salt-master
  ports:
    - name: publish
      port: 4505
      targetPort: 4505
      protocol: TCP
    - name: request
      port: 4506
      targetPort: 4506
      protocol: TCP

  # Optional: Request specific external IP (MetalLB)
  # loadBalancerIP: 192.168.1.100

  # Optional: Cloud provider specific annotations
  # For AWS ELB
  # annotations:
  #   service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

  # For GCP
  # annotations:
  #   cloud.google.com/load-balancer-type: "External"
---
# Optional: Internal API service (cluster-only access)
apiVersion: v1
kind: Service
metadata:
  name: salt-api
  namespace: salt-master
spec:
  type: ClusterIP
  selector:
    app: salt-master
  ports:
    - name: api
      port: 8000
      targetPort: 8000
      protocol: TCP

Apply the service:

1
2
3
4
5
6
7
8

kubectl apply -f service.yaml

# Get the external IP (may take a minute)
kubectl get svc salt-master -n salt-master -w

# Example output:
# NAME          TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)
# salt-master   LoadBalancer   10.96.100.50    52.49.97.175     4505:31505/TCP,4506:31506/TCP

Use this external IP in minion configuration as the master address.

Option 2: NodePort (For Environments Without LoadBalancer)

NodePort is suitable for small to medium deployments, development, or when LoadBalancer isn’t available.

Advantages:

  • Works in any Kubernetes environment
  • No external dependencies
  • No additional cost
  • Simple setup

Disadvantages:

  • Must track which nodes have open ports
  • Requires firewall rules on each node
  • Port range limitation (30000-32767)
  • Minions must use non-standard ports
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: salt-master
  namespace: salt-master
  labels:
    app: salt-master
spec:
  type: NodePort
  selector:
    app: salt-master
  ports:
    - name: publish
      port: 4505
      targetPort: 4505
      protocol: TCP
      nodePort: 30505 # Optional: specify exact port
    - name: request
      port: 4506
      targetPort: 4506
      protocol: TCP
      nodePort: 30506 # Optional: specify exact port

Apply the NodePort service:

1
2
3
4
5
6
7
8
9

kubectl apply -f service-nodeport.yaml

# Get node IPs and ports
kubectl get nodes -o wide
kubectl get svc salt-master -n salt-master

# Example output:
# NAME          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)
# salt-master   NodePort   10.96.100.50    <none>        4505:30505/TCP,4506:30506/TCP

With NodePort, minions connect to:

  • Master address: Any node’s external IP
  • Publish port: 30505 (instead of 4505)
  • Return port: 30506 (instead of 4506)

Example minion configuration with NodePort:

1
2
3
4

# /etc/salt/minion
master: node-external-ip
publish_port: 30505
master_port: 30506

Hybrid Approach: External IP with NodePort

Some clusters support externalIPs, allowing you to use standard ports with NodePort:

1
2
3
4
5
6
7
8

spec:
  type: NodePort
  externalIPs:
    - 192.168.1.100 # Assign specific external IP
  ports:
    - name: publish
      port: 4505
      targetPort: 4505

Firewall Configuration

Proper firewall configuration is critical for Salt Master connectivity. Salt uses a master-initiated communication model where minions always initiate connections.

Understanding Salt’s Communication Model

According to Salt documentation, communication is always initiated by minions, never by the master. This means:

  • Minions establish outbound connections to master ports 4505 and 4506
  • Master never initiates inbound connections to minions
  • Firewall rules only needed on the master (inbound on 4505/4506)
  • No firewall rules needed on minions (outbound connections)

Salt uses AES-encrypted ZeroMQ connections over TCP:

  • Port 4505: Publish port (master → minions broadcast)
  • Port 4506: Return port (minions ↔ master bi-directional)

Kubernetes Node Firewall Rules

If using LoadBalancer: The load balancer typically handles firewall rules automatically.

If using NodePort: You must configure firewall rules on Kubernetes nodes.

For RHEL/CentOS/Rocky/AlmaLinux with firewalld

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Allow NodePort range (if using NodePort service)
sudo firewall-cmd --permanent --add-port=30505/tcp
sudo firewall-cmd --permanent --add-port=30506/tcp

# Or allow entire NodePort range
sudo firewall-cmd --permanent --add-port=30000-32767/tcp

# Reload firewall
sudo firewall-cmd --reload

# Verify rules
sudo firewall-cmd --list-ports

For Ubuntu/Debian with UFW

1
2
3
4
5
6
7
8
9

# Allow specific NodePorts
sudo ufw allow 30505/tcp
sudo ufw allow 30506/tcp

# Or allow entire NodePort range
sudo ufw allow 30000:32767/tcp

# Check status
sudo ufw status

For iptables

1
2
3
4
5

# Allow NodePorts
sudo iptables -A INPUT -p tcp --dport 30505:30506 -j ACCEPT

# Save rules
sudo iptables-save > /etc/iptables/rules.v4

Cloud Provider Firewall Rules

AWS Security Groups

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# For LoadBalancer
aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxx \
  --protocol tcp \
  --port 4505 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxx \
  --protocol tcp \
  --port 4506 \
  --cidr 0.0.0.0/0

# For NodePort
aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxx \
  --protocol tcp \
  --port 30505 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxx \
  --protocol tcp \
  --port 30506 \
  --cidr 0.0.0.0/0

GCP Firewall Rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# For LoadBalancer
gcloud compute firewall-rules create salt-master \
  --allow tcp:4505,tcp:4506 \
  --source-ranges 0.0.0.0/0 \
  --target-tags kubernetes-node

# For NodePort
gcloud compute firewall-rules create salt-master-nodeport \
  --allow tcp:30505,tcp:30506 \
  --source-ranges 0.0.0.0/0 \
  --target-tags kubernetes-node

Azure Network Security Group

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# For LoadBalancer
az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name myNSG \
  --name salt-master-publish \
  --protocol tcp \
  --priority 1000 \
  --destination-port-range 4505

az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name myNSG \
  --name salt-master-request \
  --protocol tcp \
  --priority 1001 \
  --destination-port-range 4506

Testing Connectivity

From a potential minion, test connectivity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Test port 4505 (LoadBalancer)
telnet 52.49.97.175 4505
# or
nc -zv 52.49.97.175 4505

# Test port 4506
telnet 52.49.97.175 4506

# For NodePort
telnet node-ip 30505
telnet node-ip 30506

Successful connection shows port is open. If connection fails, check:

  1. Service has external IP assigned
  2. Firewall rules are in place
  3. Network path allows traffic
  4. No intermediate firewalls blocking ports

Setting Up Salt Minions

Once Salt Master is running and accessible, you can add minions (managed nodes).

Minion Installation

RHEL/CentOS/Rocky/AlmaLinux

1
2
3
4
5
6

# Add Salt repository
curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.repo \
  | sudo tee /etc/yum.repos.d/salt.repo

# Install Salt Minion
sudo dnf install salt-minion -y

Ubuntu/Debian

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Add Salt repository GPG key
curl -fsSL https://packages.broadcom.com/artifactory/api/gpg/key/public \
  | sudo gpg --dearmor -o /usr/share/keyrings/salt-archive-keyring.gpg

# Add repository
echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.gpg] https://packages.broadcom.com/artifactory/saltproject-deb stable main" \
  | sudo tee /etc/apt/sources.list.d/salt.list

# Update and install
sudo apt update
sudo apt install salt-minion -y

Minion Configuration

Basic Configuration

Edit /etc/salt/minion:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Salt Master address
master: 52.49.97.175 # Use your LoadBalancer IP or node IP

# Unique minion ID (defaults to hostname)
id: web-server-01

# Logging
log_level: info

# Enable mine for service discovery
mine_enabled: True
mine_functions:
  network.ip_addrs: []
  grains.items: []

# Auto-restart minion on config changes
autorestart_wait: 10

With NodePort (Non-Standard Ports)

If using NodePort, specify the custom ports:

1
2
3
4
5
6
7

master: node-external-ip

# Use NodePort port numbers
publish_port: 30505
master_port: 30506

id: web-server-01

Advanced Configuration with Grains

Grains are static data about minions used for targeting:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

master: 52.49.97.175
id: prod-web-01

# Custom grains for targeting
grains:
  environment: production
  roles:
    - webserver
    - nginx
  datacenter: aws-us-east-1
  team: platform

# Backup mode
backup_mode: minion

# Optional: Schedule automatic state application
schedule:
  highstate:
    function: state.apply
    minutes: 30
    maxrunning: 1

Grains Usage: Target minions with salt -G 'environment:production' test.ping

Start and Enable Minion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Enable on boot
sudo systemctl enable salt-minion

# Start service
sudo systemctl start salt-minion

# Check status
sudo systemctl status salt-minion

# View logs (watch for connection to master)
sudo journalctl -fu salt-minion

Expected log output:

1
2
3

salt-minion[1234]: [INFO] Setting up the Salt Minion "web-server-01"
salt-minion[1234]: [INFO] Connecting to master 52.49.97.175:4506
salt-minion[1234]: [INFO] Authentication credentials requested

Accept Minion Keys on Master

Minions generate a key pair and send their public key to the master for authentication.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# List all keys (from your workstation with kubectl access)
kubectl exec -n salt-master deployment/salt-master -- salt-key -L

# Output shows:
# Accepted Keys:
# Denied Keys:
# Unaccepted Keys:
# web-server-01
# Rejected Keys:

# Accept a specific minion
kubectl exec -n salt-master deployment/salt-master -- salt-key -a web-server-01

# Accept all pending keys (use carefully!)
kubectl exec -n salt-master deployment/salt-master -- salt-key -A -y

# Verify acceptance
kubectl exec -n salt-master deployment/salt-master -- salt-key -L

Security Note: Only accept keys from known minions. Verify the minion ID matches the expected system.

Test Minion Connectivity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Ping all minions
kubectl exec -n salt-master deployment/salt-master -- salt '*' test.ping

# Expected output:
# web-server-01:
#     True

# Get minion information
kubectl exec -n salt-master deployment/salt-master -- salt 'web-server-01' grains.items

# Run a command
kubectl exec -n salt-master deployment/salt-master -- salt 'web-server-01' cmd.run 'uptime'

Common Minion Setup Issues

  1. Minion can’t connect to master

    • Verify master IP: ping 52.49.97.175
    • Test ports: telnet 52.49.97.175 4505
    • Check firewall rules on minion (outbound should be allowed)
    • Check master logs: kubectl logs -n salt-master -l app=salt-master | grep web-server-01

  2. Key not appearing on master

    • Check minion logs: sudo journalctl -fu salt-minion
    • Verify minion can reach master
    • Restart minion: sudo systemctl restart salt-minion

  3. Minion accepted but not responding

    • Check minion service: sudo systemctl status salt-minion
    • Restart minion service
    • Check for key mismatch: Delete keys on both sides and regenerate

GitFS Integration

GitFS allows Salt Master to pull states and formulas directly from Git repositories, enabling Infrastructure as Code workflows.

Benefits of GitFS

  • Version Control: All state changes tracked in Git
  • Code Review: Pull requests for infrastructure changes
  • Branching: Test changes in branches before production
  • Rollback: Easy revert to previous versions
  • Collaboration: Multiple team members can contribute
  • Consistency: Single source of truth for configurations

Git Repository Structure

Create a repository with this structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

salt-states/
├── salt/                    # Salt states directory
│   ├── top.sls             # Top file (maps states to minions)
│   ├── common/
│   │   └── init.sls        # Common state for all minions
│   ├── webserver/
│   │   ├── init.sls        # Webserver installation
│   │   ├── nginx.sls       # Nginx configuration
│   │   └── files/
│   │       └── nginx.conf  # Template files
│   ├── database/
│   │   └── init.sls        # Database setup
│   └── firewall/
│       └── init.sls        # Firewall configuration
└── pillar/                  # Pillar data directory (optional)
    ├── top.sls
    └── common.sls

Example State Files

salt/top.sls (maps states to minions):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

base:
  "*":
    - common
    - firewall

  "roles:webserver":
    - match: grain
    - webserver
    - webserver.nginx

  "roles:database":
    - match: grain
    - database

salt/common/init.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Common packages for all servers
common_packages:
  pkg.installed:
    - pkgs:
        - vim
        - git
        - htop
        - curl
        - wget

# Ensure timezone is set
timezone_config:
  timezone.system:
    - name: UTC
    - utc: True

# Create admin group
admin_group:
  group.present:
    - name: admins
    - gid: 5000

salt/webserver/init.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Install web server
nginx:
  pkg.installed:
    - name: nginx

  service.running:
    - name: nginx
    - enable: True
    - require:
        - pkg: nginx
    - watch:
        - file: /etc/nginx/nginx.conf

# Deploy nginx configuration
/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://webserver/files/nginx.conf
    - user: root
    - group: root
    - mode: 644
    - require:
        - pkg: nginx

salt/firewall/init.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# Install firewalld
firewalld:
  pkg.installed:
    - name: firewalld

  service.running:
    - name: firewalld
    - enable: True
    - require:
      - pkg: firewalld

# Allow SSH
allow_ssh:
  firewalld.present:
    - name: public
    - services:
      - ssh
    - require:
      - service: firewalld

# Allow HTTP/HTTPS for web servers
{% if 'webserver' in grains['roles'] %}
allow_web:
  firewalld.present:
    - name: public
    - services:
      - http
      - https
    - require:
      - service: firewalld
{% endif %}

Configuring GitFS in Salt Master

Update your ConfigMap with GitFS settings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# In configmap.yaml
gitfs_remotes:
  # Public repository (HTTPS)
  - https://github.com/yourusername/salt-states.git:
      - base: main
      - root: salt # Subdirectory containing states

  # Private repository (SSH with deploy key)
  - [email protected]:yourusername/private-salt-states.git:
      - base: main
      - root: salt
      - privkey: /etc/salt/ssh/deploy_key
      - pubkey: /etc/salt/ssh/deploy_key.pub

  # Multiple branches
  - https://github.com/yourusername/salt-states.git:
      - base: production
      - root: salt

  - https://github.com/yourusername/salt-states.git:
      - base: staging
      - root: salt

# Specify which branches to make available
gitfs_branch_whitelist:
  - main
  - production
  - staging

# Update interval (seconds)
gitfs_update_interval: 300

Apply updated configuration:

1
2
3
4

kubectl apply -f configmap.yaml

# Restart deployment to load new config
kubectl rollout restart deployment/salt-master -n salt-master

Managing GitFS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Update fileserver (pull latest from Git)
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.update

# List all available files
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.file_list

# View specific file contents
kubectl exec -n salt-master deployment/salt-master -- \
  salt-run fileserver.read salt://webserver/init.sls

# Check GitFS backends
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.backends

Testing States

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Test state application (dry run)
kubectl exec -n salt-master deployment/salt-master -- \
  salt 'web-server-01' state.apply webserver test=True

# Apply state
kubectl exec -n salt-master deployment/salt-master -- \
  salt 'web-server-01' state.apply webserver

# Apply all states from top.sls
kubectl exec -n salt-master deployment/salt-master -- \
  salt '*' state.highstate

# Apply to specific grain
kubectl exec -n salt-master deployment/salt-master -- \
  salt -G 'roles:webserver' state.apply webserver

Common Pitfalls and Things to Watch Out For

1. SSH Key Permissions for GitFS

Problem: GitFS can’t authenticate with Git repository

Solution: SSH keys must have correct permissions (600 for private key). The init container handles this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

initContainers:
  - name: setup-ssh-keys
    command:
      - sh
      - -c
      - |
        cp /tmp/ssh-keys-ro/* /etc/salt/ssh/
        chmod 600 /etc/salt/ssh/salt_deploy_key
        chmod 644 /etc/salt/ssh/salt_deploy_key.pub
        chown -R 1000:1000 /etc/salt/ssh

2. Auto-Accept in Production

Problem: auto_accept: True allows any system to connect

Solution: Always set auto_accept: False in production. Manually review and accept each minion key:

1
2

# In ConfigMap
auto_accept: False

3. Losing Minion Keys

Problem: If the keys PVC is deleted, all minions lose authentication

Solution:

  • Ensure keys PVC has appropriate backup strategy
  • Consider using storage with replication
  • Document key recovery process

4. Pod Restart Causes Minion Disconnections

Problem: When Salt Master pod restarts, minions temporarily lose connection

Solution:

  • Minions automatically reconnect (default retry)
  • Use liveness/readiness probes to ensure pod is ready
  • Consider StatefulSet for more stable network identity (advanced)

5. NodePort Port Conflicts

Problem: NodePort might conflict with other services

Solution:

  • Explicitly specify nodePort values in service
  • Document which NodePorts are allocated
  • Use LoadBalancer in production if possible

6. Resource Limits Too Low

Problem: Salt Master OOMKilled or CPU throttled with many minions

Solution: Scale resources based on minion count:

  • Start: 512Mi memory, 250m CPU
  • Medium (50-100 minions): 2Gi memory, 1000m CPU
  • Large (500+ minions): 4Gi+ memory, 2000m+ CPU
  • Monitor actual usage and adjust

7. GitFS Not Updating

Problem: Changes pushed to Git don’t appear on master

Solution:

1
2
3
4
5
6
7
8
9

# Manually trigger update
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.update

# Check update interval in config
gitfs_update_interval: 300  # 5 minutes

# Verify Git authentication
kubectl exec -n salt-master deployment/salt-master -- \
  ssh -T [email protected]

8. Timeout Issues with Slow Minions

Problem: Commands timeout on slow networks or busy minions

Solution: Increase timeout values in ConfigMap:

1
2

timeout: 30 # Increase to 60 or more
gather_job_timeout: 30 # Increase proportionally

9. Wrong GitFS Root Path

Problem: States not found even though they’re in repository

Solution: Verify root path matches repository structure:

1
2
3
4

gitfs_remotes:
  - [email protected]:user/repo.git:
      - base: main
      - root: salt # Must match directory in repo

If states are at repository root, use root: . or omit root parameter.

10. Firewall Rules Not Applied

Problem: Minions can’t connect despite service being exposed

Solution:

  • Check cloud provider firewall/security groups
  • Verify node-level firewall (firewalld/ufw)
  • Test connectivity with telnet/nc from minion
  • Check for intermediate firewalls

11. ConfigMap Changes Not Applied

Problem: Updated ConfigMap doesn’t take effect

Solution:

1
2
3
4
5

# ConfigMaps mounted as volumes don't trigger pod restart
# Must manually restart deployment
kubectl rollout restart deployment/salt-master -n salt-master

# Or set SALT_RESTART_MASTER_ON_CONFIG_CHANGE=True in deployment

12. Storage Class Issues

Problem: PVCs stuck in Pending state

Solution:

1
2
3
4
5
6
7
8
9

# Check PVC status
kubectl get pvc -n salt-master

# Check events
kubectl describe pvc salt-master-keys -n salt-master

# Specify storage class explicitly
spec:
  storageClassName: standard  # or your cluster's default

Troubleshooting

Debugging Pod Issues

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Check pod status
kubectl get pods -n salt-master

# Describe pod (shows events)
kubectl describe pod -n salt-master -l app=salt-master

# View logs
kubectl logs -n salt-master -l app=salt-master -f

# Get shell in container
kubectl exec -n salt-master -it deployment/salt-master -- /bin/bash

# Inside container:
ps aux | grep salt-master
cat /etc/salt/master
salt-key -L
salt '*' test.ping

Debugging Minion Connection Issues

From master:

1
2
3
4
5
6
7
8

# Check if minion key arrived
kubectl exec -n salt-master deployment/salt-master -- salt-key -L

# Check master logs for minion authentication
kubectl logs -n salt-master -l app=salt-master | grep minion-id

# Try pinging specific minion
kubectl exec -n salt-master deployment/salt-master -- salt 'minion-id' test.ping -v

From minion:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Check minion service
sudo systemctl status salt-minion

# View minion logs
sudo journalctl -fu salt-minion

# Test master connectivity
telnet master-ip 4505
telnet master-ip 4506

# Run minion in debug mode
sudo systemctl stop salt-minion
sudo salt-minion -l debug

Debugging GitFS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Check GitFS backends
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.backends

# Update fileserver
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.update

# List files in fileserver
kubectl exec -n salt-master deployment/salt-master -- salt-run fileserver.file_list

# Check master logs for Git errors
kubectl logs -n salt-master -l app=salt-master | grep -i git

# Test SSH key from container
kubectl exec -n salt-master -it deployment/salt-master -- ssh -T [email protected]

Debugging Service Exposure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Check service status
kubectl get svc -n salt-master

# Describe service
kubectl describe svc salt-master -n salt-master

# For LoadBalancer, check external IP assignment
kubectl get svc salt-master -n salt-master -w

# Test from outside cluster
telnet external-ip 4505
nc -zv external-ip 4505

Common Error Messages

“Authentication failed” on minion:

  • Minion key not accepted on master
  • Key mismatch (delete both sides and regenerate)
  • Master IP incorrect in minion config

“Connection refused” on minion:

  • Firewall blocking ports
  • Service not exposed correctly
  • Master not running

“gitfs: Failed to authenticate with remote”:

  • SSH key permissions incorrect
  • SSH key not mounted correctly
  • Git repository URL incorrect
  • Deploy key not added to repository

“Permission denied” in pod:

  • fsGroup not set correctly
  • PVC permissions issue
  • Check securityContext in deployment

Production Considerations

High Availability

For production, consider running multiple Salt Masters:

1
2
3
4
5
6

spec:
  replicas: 2 # Run 2 masters


  # Use StatefulSet instead of Deployment
  # Provides stable network identity

Configure minions with multiple masters:

1
2
3
4
5
6

# /etc/salt/minion
master:
  - master1-ip
  - master2-ip

master_type: failover # Try next master if first fails

Monitoring and Alerts

Integrate with monitoring systems:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Prometheus ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: salt-master
  namespace: salt-master
spec:
  selector:
    matchLabels:
      app: salt-master
  endpoints:
    - port: api
      path: /metrics

Monitor these metrics:

  • Pod CPU and memory usage
  • PVC usage
  • Minion connection count
  • Job execution times
  • Failed state applications

Security Hardening

  1. Network Policies: Restrict access to Salt Master
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: salt-master-policy
  namespace: salt-master
spec:
  podSelector:
    matchLabels:
      app: salt-master
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.0.0.0/8 # Internal network only
      ports:
        - protocol: TCP
          port: 4505
        - protocol: TCP
          port: 4506

  1. Pod Security Standards: Run with restricted permissions

  2. Secrets Management: Use external secrets management (Vault, Sealed Secrets)

  3. Audit Logging: Enable Salt audit logging

1
2
3
4

# In ConfigMap
publish_session: 86400
keep_jobs: 24
job_cache: True

Backup Strategy

Automate backups of critical data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash
# backup-salt-master.sh

NAMESPACE=salt-master
BACKUP_DIR=/backups/salt-master/$(date +%Y%m%d-%H%M%S)
mkdir -p $BACKUP_DIR

# Backup minion keys
POD=$(kubectl get pod -n $NAMESPACE -l app=salt-master -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n $NAMESPACE $POD -- \
  tar czf /tmp/keys-backup.tar.gz -C /home/salt/data/keys .

kubectl cp $NAMESPACE/$POD:/tmp/keys-backup.tar.gz \
  $BACKUP_DIR/keys-backup.tar.gz

# Backup configurations
kubectl get configmap salt-master-config -n $NAMESPACE -o yaml \
  > $BACKUP_DIR/configmap.yaml

kubectl get secret salt-api-credentials -n $NAMESPACE -o yaml \
  > $BACKUP_DIR/secret.yaml

echo "Backup completed: $BACKUP_DIR"

Run via CronJob:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: batch/v1
kind: CronJob
metadata:
  name: salt-master-backup
  namespace: salt-master
spec:
  schedule: "0 2 * * *" # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: bitnami/kubectl:latest
              command: ["/scripts/backup-salt-master.sh"]

Version Pinning

Never use latest tag in production:

1

image: ghcr.io/cdalvaro/docker-salt-master:3007.1 # Specific version

Resource Scaling Guidelines

MinionsMemoryCPUWorkers
1-50512Mi-1Gi250m-500m5
50-2001Gi-2Gi500m-1000m10
200-5002Gi-4Gi1000m-2000m20
500+4Gi+2000m+30+

Monitor actual usage and adjust accordingly.

Conclusion

Running Salt Master on Kubernetes provides a modern, scalable approach to configuration management. Key takeaways:

  1. Service Exposure: Use LoadBalancer for production, NodePort for development
  2. Firewall Rules: Only master needs inbound rules on 4505/4506
  3. Persistent Storage: Critical for minion keys - ensure backups
  4. GitFS: Enables Infrastructure as Code workflows
  5. Security: Never auto-accept keys in production
  6. Monitoring: Track resource usage and scale accordingly
  7. Testing: Always test state changes with test=True first

With this setup, you have a production-ready Salt Master running on Kubernetes, capable of managing infrastructure at scale while maintaining the flexibility and reliability of modern cloud-native applications.

Resources and References

This guide was created for citizix.com based on practical experience deploying Salt Master on Kubernetes. For questions or improvements, please reach out.

comments powered by Disqus
© 2020 - 2026 Citizix, Ltd.
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy