Production-Grade SaltStack: Multi-Environment Setup with GitOps on AlmaLinux 10

Introduction

In this comprehensive guide, we’ll build a real-world, production-ready SaltStack configuration that manages multiple environments (development, staging, production) on AlmaLinux 10. This isn’t a toy example—we’ll tackle real challenges including git-backed configurations, SSL certificates, security hardening, and the troubleshooting lessons learned from actual deployment.

By the end of this guide, you’ll have:

✅ Git-backed Salt configuration with automatic deployment
✅ Multi-environment setup on a single server
✅ NGINX with Let’s Encrypt SSL certificates
✅ Security hardening (SSH, firewall, SELinux)
✅ Self-healing infrastructure
✅ Production-ready monitoring and troubleshooting

Time Required: 2-3 hours Difficulty: Advanced Prerequisites: Basic Linux administration, Git knowledge, understanding of Salt basics

Architecture Overview
Infrastructure Setup
Project Structure
Salt Master Configuration
Git-Backed States and Pillar
- Security Hardening
Multi-Environment Configuration
NGINX Multi-Site Setup
SSL Automation with Let’s Encrypt
Salt Minion Configuration
Applying States
Troubleshooting Real Issues
Production Best Practices
Monitoring and Maintenance
Scaling Up
Performance Tuning
Conclusion

Architecture Overview

The Challenge

We need to manage three environments (dev, staging, production) with:

Separate domains for each environment
Individual SSL certificates
Cost-effective single-server deployment
Git-based configuration management
Automatic deployment on code changes

Solution Architecture

                    ┌─────────────────────┐
                    │   Git Repository    │
                    │  (GitHub/GitLab)    │
                    └──────────┬──────────┘
                               │
                               │ Auto-pull every 5 min
                               │
                    ┌──────────▼──────────┐
                    │   Salt Master       │
                    │  10.2.10.10       │
                    │  - GitFS            │
                    │  - Git Pillar       │
                    └──────────┬──────────┘
                               │
                               │ ZeroMQ (4505/4506)
                               │
                    ┌──────────▼──────────┐
                    │   Salt Minion       │
                    │  10.2.10.11       │
                    ├─────────────────────┤
                    │  NGINX Virtual Hosts│
                    ├─────────────────────┤
                    │ dev.citizix.com     │ ← /var/www/dev     (Purple)
                    │ stage.citizix.com   │ ← /var/www/stage   (Pink)
                    │ prod.citizix.com    │ ← /var/www/prod    (Blue)
                    └─────────────────────┘
                    Each with own SSL cert

Why This Architecture?

Benefits:

Cost-Effective: One server for all environments (great for small-medium traffic)
Git-Backed: Infrastructure as Code, full version control
Automated: Push to git, auto-deploys in 5 minutes
Scalable: Easy to migrate environments to separate servers later
Secure: Hardened SSH, firewall, SSL/TLS

Trade-offs:

Shared resources (CPU, RAM, disk)
Not suitable for high-traffic production or compliance requirements
Single point of failure (mitigated by cloud provider reliability)

Infrastructure Setup

Servers Required

Salt Master:

OS: AlmaLinux 10
vCPU: 2
RAM: 2GB
Disk: 20GB
IP: 10.2.10.10 (example)
Hostname: salt-master

Salt Minion:

OS: AlmaLinux 10
vCPU: 2
RAM: 4GB
Disk: 40GB
IP: 10.2.10.11 (example)
Hostname: learn-stage-eutychus-dev-node2

DNS Configuration

Before starting, configure DNS A records:

dev.citizix.com      → 10.2.10.11
stage.citizix.com    → 10.2.10.11
prod.citizix.com     → 10.2.10.11

Verify DNS:

dig dev.citizix.com +short
dig stage.citizix.com +short
dig prod.citizix.com +short
# All should return 10.2.10.11

Initial Server Setup

On both servers:

# Update system
dnf update -y

# Install essential packages
dnf install -y vim wget curl git htop nc

# Set timezone
timedatectl set-timezone UTC

# Configure hostname
hostnamectl set-hostname salt-master  # or appropriate hostname

Project Structure

Create this directory structure on your local machine:

salt-infrastructure/
├── salt-repo/                    # Push to Git
│   ├── salt/                     # States
│   │   ├── top.sls
│   │   ├── hardening/
│   │   │   ├── init.sls
│   │   │   └── sshd_config.j2
│   │   ├── firewall/
│   │   │   └── init.sls
│   │   ├── nginx/
│   │   │   ├── init.sls
│   │   │   └── files/
│   │   │       ├── nginx.conf.j2
│   │   │       ├── site.conf.j2
│   │   │       ├── index.html.j2
│   │   │       └── security-headers.conf
│   │   └── letsencrypt/
│   │       └── init.sls
│   └── pillar/                   # Configuration data
│       ├── top.sls
│       ├── common.sls
│       └── multi-env.sls
├── salt-master-configs/          # Master configuration
│   ├── gitfs.conf
│   ├── pillar_git.conf
│   └── security.conf
└── salt-minion-configs/          # Minion configuration
    ├── minion-master.conf
    └── minion-webserver.conf

Let’s build this step by step.

Salt Master Configuration

Step 1: Install Salt Master

ssh [email protected]

# Add the official Salt Project repository FIRST — salt-master is not in
# AlmaLinux's own repos, and there is no Salt package in EPEL 10
curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.repo \
  | sudo tee /etc/yum.repos.d/salt.repo
sudo dnf clean expire-cache

# Install Salt
dnf install -y salt-master salt-minion

# Install patchelf (required by pygit2 installation)
sudo dnf install -y patchelf

# Install pygit2 for GitFS support
# CRITICAL: Salt uses bundled Python at /opt/saltstack/salt/
/opt/saltstack/salt/bin/pip3 install pygit2

# Verify pygit2 installation (use python3, not a hardcoded minor version —
# the bundled interpreter moves between Salt releases)
/opt/saltstack/salt/bin/python3 -c "import pygit2; print(pygit2.__version__)"

The repository step is not optional. Earlier versions of this guide ran dnf install -y salt-master and then dnf install -y epel-release. That cannot work on AlmaLinux 10: EPEL carries the classic (non-onedir) Salt packages only up to EPEL 9, and EPEL 10 has no Salt package at all — so the install fails before EPEL is even added. The salt.repo file above is the supported path from the Salt install guide, and it is also what puts Salt’s bundled Python at /opt/saltstack/salt/ — the pip3 and python3 calls above depend on that onedir layout existing.

Step 2: Enable master.d/ Configuration

CRITICAL STEP: Salt master ignores files in /etc/salt/master.d/ unless you enable it.

vim /etc/salt/master

# Find and UNCOMMENT this line (around line 16):
default_include: master.d/*.conf

# Save and exit

Verify:

grep "default_include" /etc/salt/master
# Should show: default_include: master.d/*.conf

Step 3: Configure SSH Deploy Keys

Salt will use SSH keys to access your private Git repository.

# Create SSH directory
mkdir -p /etc/salt/ssh

# Generate SSH key WITHOUT passphrase
ssh-keygen -t ed25519 -f /etc/salt/ssh/salt_deploy_key -N ""

# CRITICAL: Set correct ownership
# Salt master runs as user 'salt', not 'root'
chown salt:salt /etc/salt/ssh/salt_deploy_key
chown salt:salt /etc/salt/ssh/salt_deploy_key.pub
chmod 600 /etc/salt/ssh/salt_deploy_key
chmod 644 /etc/salt/ssh/salt_deploy_key.pub

# Set directory permissions
chown root:salt /etc/salt
chown root:salt /etc/salt/ssh
chmod 750 /etc/salt
chmod 750 /etc/salt/ssh

# Verify salt user can read the key
sudo -u salt ls /etc/salt/ssh/salt_deploy_key.pub
# Should succeed without "Permission denied"

# Display public key to add to GitHub
cat /etc/salt/ssh/salt_deploy_key.pub

Add deploy key to GitHub:

Go to your repository settings
Navigate to: Settings → Deploy keys
Click “Add deploy key”
Paste the public key
Title: “Salt Master Deploy Key”
Do NOT check “Allow write access”
Click “Add key”

Test SSH connection:

ssh -i /etc/salt/ssh/salt_deploy_key -T [email protected]
# Should show: "Hi USERNAME! You've successfully authenticated..."
# Should NOT ask for passphrase

Step 4: Configure GitFS (Git-Backed States)

Create /etc/salt/master.d/gitfs.conf:

# Git-backed fileserver configuration

# Enable git backend
fileserver_backend:
  - git

# Use pygit2 provider (required for SSH keys)
gitfs_provider: pygit2

# Git repository for states
gitfs_remotes:
  - [email protected]:USERNAME/REPO.git:
      - base: main # or your branch name
      - root: salt-repo/salt # path to states in repo
      - privkey: /etc/salt/ssh/salt_deploy_key
      - pubkey: /etc/salt/ssh/salt_deploy_key.pub

# Git update interval (seconds)
gitfs_update_interval: 300

Key Points:

base: specifies the git branch
root: is the path WITHIN the git repo to your states
Update interval: 300 seconds = 5 minutes auto-refresh

Step 5: Configure Git Pillar

Create /etc/salt/master.d/pillar_git.conf:

# Git-backed pillar configuration

# External pillar - Git repository
ext_pillar:
  - git:
      - main [email protected]:USERNAME/REPO.git:
          - env: base # CRITICAL: maps branch to base environment
          - root: salt-repo/pillar # path to pillar in repo
          - privkey: /etc/salt/ssh/salt_deploy_key
          - pubkey: /etc/salt/ssh/salt_deploy_key.pub

# Pillar update interval (seconds)
pillar_gitfs_update_interval: 300

CRITICAL: The env: base line is essential. Without it, Salt won’t find your pillar data!

Step 6: Configure Master Security

Create /etc/salt/master.d/security.conf:

# Salt master security configuration

# Auto accept minion keys (DISABLE in production!)
# auto_accept: False

# Timeout for minion responses
timeout: 60

# Keep job results for 24 hours
keep_jobs: 24

# Log level
log_level: warning

# Enable backup mode (creates .bak files)
backup_mode: minion

Step 7: Start Salt Master

# Start and enable
systemctl enable --now salt-master

# Check status
systemctl status salt-master

# Check logs for configuration loading
journalctl -u salt-master -n 50 | grep "Including configuration"
# Should show:
# Including configuration from '/etc/salt/master.d/gitfs.conf'
# Including configuration from '/etc/salt/master.d/pillar_git.conf'
# Including configuration from '/etc/salt/master.d/security.conf'

If configs aren’t loading, you forgot Step 2 (enable default_include)!

Step 8: Configure Firewall

# Install firewalld if not installed
dnf install -y firewalld
systemctl enable --now firewalld

# Allow SSH (critical - don't lock yourself out!)
firewall-cmd --permanent --add-service=ssh

# Allow Salt master ports from specific IPs
# Replace with your minion's IP
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.2.10.11" port port="4505" protocol="tcp" accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.2.10.11" port port="4506" protocol="tcp" accept'

# Reload firewall
firewall-cmd --reload

# Verify
firewall-cmd --list-all

Git-Backed States and Pillar

Create Git Repository

# On your local machine
mkdir -p ~/salt-infrastructure/salt-repo/{salt,pillar}
cd ~/salt-infrastructure/salt-repo

git init

Create States

1. Top File (salt/top.sls)

base:
  "*":
    - hardening
    - firewall

  "roles:web":
    - match: grain
    - nginx

  "multi_env:True":
    - match: grain
    - letsencrypt

This tells Salt:

Apply hardening and firewall to ALL minions
Apply nginx only to minions with roles:web grain
Apply letsencrypt to multi-environment minions

2. Hardening State (salt/hardening/init.sls)

# Base OS hardening for AlmaLinux

# Keep system patched
system-updates:
  pkg.uptodate:
    - refresh: True

# Ensure time sync
chrony:
  pkg.installed:
    - name: chrony

chrony-service:
  service.running:
    - name: chronyd
    - enable: True
    - require:
      - pkg: chrony

# Harden SSH configuration
sshd_config:
  file.managed:
    - name: /etc/ssh/sshd_config
    - source: salt://hardening/sshd_config.j2
    - template: jinja
    - mode: '0600'
    - user: root
    - group: root
    - backup: minion

sshd:
  service.running:
    - name: sshd
    - enable: True
    - watch:
      - file: sshd_config

# SELinux: Set to permissive mode
{% if salt['grains.get']('selinux:enabled', False) %}
selinux-permissive:
  cmd.run:
    - name: setenforce 0
    - unless: getenforce | grep -i permissive

selinux-config-permissive:
  file.replace:
    - name: /etc/selinux/config
    - pattern: '^SELINUX=enforcing'
    - repl: 'SELINUX=permissive'
    - backup: '.bak'
{% endif %}

SSH Config Template (salt/hardening/sshd_config.j2)

# Managed by Salt - DO NOT EDIT MANUALLY

Port 22
Protocol 2

# Authentication
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Security
X11Forwarding no
PrintMotd no
AcceptEnv LANG LC_*

# Subsystem
Subsystem sftp /usr/libexec/openssh/sftp-server

# Allow specific users (if needed)
# AllowUsers user1 user2

3. Firewall State (salt/firewall/init.sls)

# Firewall configuration using firewalld

firewalld:
  pkg.installed:
    - name: firewalld
    - reload_modules: True

firewalld-service:
  service.running:
    - name: firewalld
    - enable: True
    - require:
      - pkg: firewalld

# Allow SSH (required for management)
allow-ssh:
  firewalld.present:
    - name: public
    - services:
      - ssh
    - require:
      - service: firewalld-service

# Web servers: allow HTTP and HTTPS
{% if 'web' in grains.get('roles', []) %}
allow-http-https:
  firewalld.present:
    - name: public
    - services:
      - ssh
      - http
      - https
    - require:
      - service: firewalld-service
{% endif %}

# Master: restrict Salt ports to whitelisted IPs
{% if 'master' in grains.get('roles', []) %}
{% set allowed_ips = pillar.get('salt_master', {}).get('allowed_ips', []) %}

{% for ip in allowed_ips %}
salt-master-4505-{{ ip | replace('.', '-') }}:
  cmd.run:
    - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ ip }}" port port="4505" protocol="tcp" accept'
    - unless: firewall-cmd --list-rich-rules | grep -q "4505.*{{ ip }}"
    - require:
      - service: firewalld-service

salt-master-4506-{{ ip | replace('.', '-') }}:
  cmd.run:
    - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ ip }}" port port="4506" protocol="tcp" accept'
    - unless: firewall-cmd --list-rich-rules | grep -q "4506.*{{ ip }}"
    - require:
      - service: firewalld-service
{% endfor %}

firewall-reload-after-salt-rules:
  cmd.run:
    - name: firewall-cmd --reload
    - onchanges:
{% for ip in allowed_ips %}
      - cmd: salt-master-4505-{{ ip | replace('.', '-') }}
      - cmd: salt-master-4506-{{ ip | replace('.', '-') }}
{% endfor %}
{% endif %}

This firewall state:

Installs firewalld
Allows SSH on all servers
Allows HTTP/HTTPS on web servers
Restricts Salt ports to whitelisted IPs (security!)

Multi-Environment Configuration

Pillar Structure

1. Pillar Top File (pillar/top.sls)

base:
  "*":
    - common

  "multi_env:True":
    - match: grain
    - multi-env

2. Common Pillar (pillar/common.sls)

# Common pillar data shared across all environments

# Salt master configuration
salt_master:
  # Whitelist of IPs allowed to connect to Salt master ports
  allowed_ips:
    - 10.2.10.10 # salt-master (self)
    - 10.2.10.11 # learn-stage-eutychus-dev-node2
    # Add more minion IPs here:
    # - 1.2.3.4       # future-node-1

# Timezone
timezone: "UTC"

# Common packages
common_packages:
  - vim
  - wget
  - curl
  - git
  - htop

3. Multi-Environment Pillar (pillar/multi-env.sls)

# Multi-environment configuration for single server

# NGINX configuration with multiple virtual hosts
nginx:
  sites:
    - server_name: dev.citizix.com
      root: /var/www/dev
      env: dev

    - server_name: stage.citizix.com
      root: /var/www/stage
      env: stage

    - server_name: prod.citizix.com
      root: /var/www/prod
      env: prod

# Let's Encrypt configuration
letsencrypt:
  email: [email protected]
  domains:
    - dev.citizix.com
    - stage.citizix.com
    - prod.citizix.com

Update the email address with your actual email!

NGINX Multi-Site Setup

NGINX State (salt/nginx/init.sls)

# NGINX web server with hardened configuration
# Supports multiple virtual hosts for multi-environment setup

nginx:
  pkg.installed:
    - name: nginx

nginx-service:
  service.running:
    - name: nginx
    - enable: True
    - require:
      - pkg: nginx

# Main nginx.conf
/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://nginx/files/nginx.conf.j2
    - template: jinja
    - mode: '0644'
    - user: root
    - group: root
    - require:
      - pkg: nginx

# Create snippets directory for includes
/etc/nginx/snippets:
  file.directory:
    - mode: '0755'
    - user: root
    - group: root
    - require:
      - pkg: nginx

# Security headers include
/etc/nginx/snippets/security-headers.conf:
  file.managed:
    - source: salt://nginx/files/security-headers.conf
    - mode: '0644'
    - user: root
    - group: root
    - require:
      - file: /etc/nginx/snippets

# Remove default site config
/etc/nginx/conf.d/default.conf:
  file.absent

# Create site configurations for each environment
{% for site in pillar.get('nginx', {}).get('sites', []) %}
{% set server_name = site.server_name %}
{% set env = site.get('env', 'default') %}
{% set root = site.get('root', '/var/www/' ~ env) %}

# Create web root for {{ env }}
{{ root }}:
  file.directory:
    - mode: '0755'
    - user: nginx
    - group: nginx
    - makedirs: True

# Create index file for {{ env }}
{{ root }}/index.html:
  file.managed:
    - source: salt://nginx/files/index.html.j2
    - template: jinja
    - mode: '0644'
    - user: nginx
    - group: nginx
    - context:
        env: {{ env }}
        server_name: {{ server_name }}
    - require:
      - file: {{ root }}

# Site configuration for {{ server_name }}
/etc/nginx/conf.d/{{ server_name }}.conf:
  file.managed:
    - source: salt://nginx/files/site.conf.j2
    - template: jinja
    - mode: '0644'
    - user: root
    - group: root
    - context:
        server_name: {{ server_name }}
        root: {{ root }}
        env: {{ env }}
    - require:
      - pkg: nginx
{% endfor %}

# Reload nginx on config changes
nginx-reload:
  cmd.run:
    - name: nginx -t && systemctl reload nginx
    - onchanges:
      - file: /etc/nginx/nginx.conf
      - file: /etc/nginx/snippets/security-headers.conf
{% for site in pillar.get('nginx', {}).get('sites', []) %}
      - file: /etc/nginx/conf.d/{{ site.server_name }}.conf
{% endfor %}

# Configure SELinux for nginx (only if SELinux is enabled)
{% if salt['grains.get']('selinux:enabled', False) %}
nginx-selinux-httpd-can-network-connect:
  cmd.run:
    - name: setsebool -P httpd_can_network_connect on
    - unless: getsebool httpd_can_network_connect | grep -q "on$"
    - require:
      - pkg: nginx

{% for site in pillar.get('nginx', {}).get('sites', []) %}
{% set root = site.get('root', '/var/www/' ~ site.get('env', 'default')) %}
nginx-selinux-context-{{ site.get('env', 'default') }}:
  cmd.run:
    - name: restorecon -Rv {{ root }}
    - onchanges:
      - file: {{ root }}
    - require:
      - pkg: nginx
{% endfor %}
{% endif %}

Security Headers (salt/nginx/files/security-headers.conf)

# Security headers

add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;

# HSTS (uncomment after SSL is working)
# add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Site Config Template (salt/nginx/files/site.conf.j2)

server {
    listen 80;
    server_name {{ server_name }};
    root {{ root }};

    index index.html;

    # Include security headers
    include /etc/nginx/snippets/security-headers.conf;

    location / {
        try_files $uri $uri/ =404;
    }

    # SSL configuration (will be added by letsencrypt state)
    # listen 443 ssl http2;
    # ssl_certificate /etc/letsencrypt/live/{{ server_name }}/fullchain.pem;
    # ssl_certificate_key /etc/letsencrypt/live/{{ server_name }}/privkey.pem;
}

Index Page Template (salt/nginx/files/index.html.j2)

<!DOCTYPE html>
<html lang="en">
  <head>
    
    
    <title>{{ env|upper }} - Salt-Managed Environment</title>
    <style>
      * { margin: 0; padding: 0; box-sizing: border-box; }
      body {
          font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
          min-height: 100vh;
          display: flex;
          align-items: center;
          justify-content: center;
          {% if env == 'dev' %}
          background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
          {% elif env == 'stage' %}
          background: linear-gradient(135deg, #f093fb 0%, #f5576c 100%);
          {% elif env == 'prod' %}
          background: linear-gradient(135deg, #4facfe 0%, #00f2fe 100%);
          {% else %}
          background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
          {% endif %}
          color: white;
      }
      .container {
          text-align: center;
          padding: 40px;
          background: rgba(255, 255, 255, 0.1);
          backdrop-filter: blur(10px);
          border-radius: 20px;
          box-shadow: 0 8px 32px 0 rgba(31, 38, 135, 0.37);
      }
      h1 {
          font-size: 3em;
          margin-bottom: 20px;
          text-shadow: 2px 2px 4px rgba(0,0,0,0.3);
      }
      .env-badge {
          display: inline-block;
          padding: 10px 30px;
          background: rgba(255, 255, 255, 0.2);
          border-radius: 50px;
          font-size: 1.5em;
          margin: 20px 0;
          text-transform: uppercase;
          letter-spacing: 2px;
      }
      .info {
          margin-top: 30px;
          font-size: 1.1em;
          line-height: 1.8;
      }
      .info-item {
          margin: 10px 0;
      }
    </style>
  </head>
  <body>
    <div class="container">
      <h1>🚀 Salt-Managed Environment</h1>
      <div class="env-badge">{{ env }}</div>
      <div class="info">
        <div class="info-item"><strong>Domain:</strong> {{ server_name }}</div>
        <div class="info-item"><strong>Server:</strong> {{ grains['id'] }}</div>
        <div class="info-item">
          <strong>OS:</strong> {{ grains['os'] }} {{ grains['osrelease'] }}
        </div>
        <div class="info-item"><strong>Managed by:</strong> SaltStack</div>
      </div>
    </div>
  </body>
</html>

Each environment gets a different color:

Dev: Purple
Stage: Pink
Prod: Blue

SSL Automation with Let’s Encrypt

Let’s Encrypt State (salt/letsencrypt/init.sls)

# Let's Encrypt SSL certificate automation

# Install certbot
certbot:
  pkg.installed:
    - pkgs:
      - certbot
      - python3-certbot-nginx

# Get certificates for all domains
{% set email = pillar.get('letsencrypt', {}).get('email', '[email protected]') %}
{% set domains = pillar.get('letsencrypt', {}).get('domains', []) %}

{% for domain in domains %}
letsencrypt-cert-{{ domain }}:
  cmd.run:
    - name: certbot certonly --nginx -d {{ domain }} --non-interactive --agree-tos --email {{ email }}
    - unless: test -d /etc/letsencrypt/live/{{ domain }}
    - require:
      - pkg: certbot
{% endfor %}

# Set up auto-renewal
certbot-renewal-cron:
  cron.present:
    - name: certbot renew --quiet --post-hook "systemctl reload nginx"
    - user: root
    - minute: 0
    - hour: '3,15'
    - require:
      - pkg: certbot

# Enable certbot timer (systemd)
certbot-renewal-timer:
  service.running:
    - name: certbot-renew.timer
    - enable: True
    - require:
      - pkg: certbot

This state:

Installs certbot
Obtains SSL certificates for each domain
Sets up auto-renewal via cron and systemd timer
Reloads nginx after renewal

Push to Git

cd ~/salt-infrastructure/salt-repo

# Add all files
git add .

# Commit
git commit -m "Initial Salt configuration with multi-environment setup"

# Create repository on GitHub/GitLab, then push
git remote add origin [email protected]:USERNAME/REPO.git
git push -u origin main

Salt Minion Configuration

Configure Web Server Minion

SSH to the minion (10.2.10.11):

ssh [email protected]

# Install Salt minion
dnf install -y salt-minion

Create minion configuration:

vim /etc/salt/minion.d/custom.conf

Add:

# Salt minion configuration
# This server hosts ALL environments (dev, staging, prod)

# Master IP
master: 10.2.10.10

# Minion ID
id: learn-stage-eutychus-dev-node2

# Set grains for targeting
grains:
  roles:
    - web
  multi_env: True
  environments:
    - dev
    - stage
    - prod

# Backup files before editing
backup_mode: minion

# Enable minion scheduler for automatic state application
schedule:
  highstate:
    function: state.apply
    minutes: 5
    maxrunning: 1
    return_job: False
    kwargs:
      saltenv: base

# Log level
log_level: warning

Start and enable:

systemctl enable --now salt-minion
systemctl status salt-minion

Accept Minion Key on Master

Back on the Salt Master:

# List keys
salt-key -L

# Accept the minion
salt-key -a learn-stage-eutychus-dev-node2

# Test connectivity
salt '*' test.ping
# Output: learn-stage-eutychus-dev-node2: True

Verify Git Configuration

# Force update from git
salt-run fileserver.update

# List files from git
salt-run fileserver.file_list saltenv=base
# Should show all your state files

# Check pillar
salt '*' pillar.items
# Should show common pillar and multi-env configuration

# Specifically check nginx pillar
salt '*' pillar.get nginx
# Should show the 3 sites configuration

Applying States

Phase 1: Hardening (CAREFUL!)

Before applying hardening:

✅ Ensure SSH key authentication works
✅ Test SSH key access
✅ Keep existing SSH session open

# Test in dry-run mode first
salt '*' state.apply hardening test=True

# Apply hardening
salt '*' state.apply hardening

# Immediately test SSH (from another terminal)
ssh -i ~/.ssh/your_key [email protected]
# Should work!

# If locked out, use cloud provider console

Phase 2: Firewall

# Apply firewall state
salt '*' state.apply firewall

# Verify firewall rules
salt '*' cmd.run 'firewall-cmd --list-all'

Phase 3: NGINX

# Apply nginx state
salt '*' state.apply nginx

# Verify directories created
salt '*' cmd.run 'ls -la /var/www/'
# Should show: dev/, stage/, prod/

# Verify nginx configs
salt '*' cmd.run 'ls -la /etc/nginx/conf.d/'
# Should show configs for each domain

# Test sites (HTTP)
curl http://dev.citizix.com
curl http://stage.citizix.com
curl http://prod.citizix.com
# Each should show different colored page

Phase 4: SSL Certificates

# Apply letsencrypt state
salt '*' state.apply letsencrypt

# Verify certificates
salt '*' cmd.run 'ls -la /etc/letsencrypt/live/'
# Should show directories for each domain

# Test HTTPS
curl -I https://dev.citizix.com
curl -I https://stage.citizix.com
curl -I https://prod.citizix.com
# Should show: HTTP/2 200

Troubleshooting Real Issues

During actual deployment, we encountered several critical issues. Here’s how to identify and fix them.

Issue 1: master.d/ Configs Not Loading

Symptom:

journalctl -u salt-master | grep "Including configuration"
# Shows nothing - configs not loading

Cause: default_include commented out in /etc/salt/master

Fix:

vim /etc/salt/master
# Uncomment: default_include: master.d/*.conf
systemctl restart salt-master

Issue 2: Pillar Data Not Loading

Symptom:

salt '*' pillar.items
# Returns empty: {}

salt-run pillar.show_top
# Returns nothing

Cause: Missing env: base in pillar_git.conf

Fix:

ext_pillar:
  - git:
      - main [email protected]:USERNAME/REPO.git:
          - env: base # ADD THIS LINE
          - root: salt-repo/pillar

Issue 3: SSH Key Permission Denied

Symptom:

# Master logs show:
[CRITICAL] SSH pubkey (/etc/salt/ssh/salt_deploy_key.pub) could not be found

Cause: Salt master runs as user salt, not root

Fix:

chown salt:salt /etc/salt/ssh/salt_deploy_key*
chmod 600 /etc/salt/ssh/salt_deploy_key
chmod 644 /etc/salt/ssh/salt_deploy_key.pub
chown root:salt /etc/salt /etc/salt/ssh
chmod 750 /etc/salt /etc/salt/ssh

# Verify
sudo -u salt ls /etc/salt/ssh/salt_deploy_key.pub

Issue 4: Firewall Module Not Available

Symptom:

'firewalld' __virtual__ returned False: firewall-cmd is not available

Cause: Salt modules not reloaded after installing firewalld

Fix:

firewalld:
  pkg.installed:
    - name: firewalld
    - reload_modules: True # ADD THIS

Issue 5: pygit2 Not Found

Symptom:

[CRITICAL] No suitable gitfs provider module is installed

Cause: pygit2 installed to system Python, not Salt’s bundled Python

Fix:

# Install to Salt's Python
/opt/saltstack/salt/bin/pip3 install pygit2

# Verify
/opt/saltstack/salt/bin/python3.10 -c "import pygit2; print(pygit2.__version__)"

Issue 6: Minion Can’t Connect After Firewall

Symptom:

salt '*' test.ping
# learn-stage-eutychus-dev-node2: [Not connected]

Cause: Firewall blocking minion’s IP

Fix:

# On master, allow minion IP immediately
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.2.10.11" port port="4505" protocol="tcp" accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.2.10.11" port port="4506" protocol="tcp" accept'
firewall-cmd --reload

Issue 7: SELinux Blocking NGINX

Symptom:

403 Forbidden when accessing site

Fix:

salt '*' cmd.run 'setsebool -P httpd_unified on'
salt '*' cmd.run 'restorecon -Rv /var/www/'

Production Best Practices

1. Always Use Test Mode First

salt '*' state.apply test=True

2. Version Control Everything

# Commit frequently
git add .
git commit -m "Add firewall rules for new minion"
git push

# Wait 5 minutes for auto-apply, or force:
salt-run fileserver.update

3. Use Specific Targeting

# Bad - affects all minions
salt '*' state.apply

# Good - specific minion
salt 'web-server' state.apply

# Better - grain-based
salt -G 'roles:web' state.apply nginx

4. Monitor State Runs

# View recent state runs
salt '*' state.show_top

# Check last highstate
salt '*' state.show_highstate

# View schedule
salt '*' schedule.list

5. Keep Pillar Secure

# Never commit passwords to git
# Use pillar with encrypted values

mysql_password: {{ pillar['secrets']['mysql_password'] }}

6. Document Changes

git commit -m "Add HTTPS redirect to nginx

- Updated site.conf.j2 template
- Redirects HTTP to HTTPS
- Preserves query strings
- HSTS header enabled"

7. Backup Before Major Changes

# Backup current config
salt '*' cmd.run 'tar czf /root/backup-$(date +%Y%m%d).tar.gz /etc/nginx /var/www'

# Apply changes
salt '*' state.apply

# Rollback if needed
salt '*' cmd.run 'tar xzf /root/backup-20240127.tar.gz -C /'

8. Use Requisites Properly

nginx-config:
  file.managed:
    - name: /etc/nginx/nginx.conf
    - require:
        - pkg: nginx # Install package first
    - watch_in:
        - service: nginx # Reload service if config changes

9. Implement Change Control

# Development workflow:
# 1. Make changes in git
# 2. Test on dev environment first
# 3. Verify in stage
# 4. Deploy to production

# Tag releases
git tag -a v1.0.0 -m "Production release 1.0.0"
git push --tags

10. Monitor Logs

# Master logs
journalctl -u salt-master -f

# Minion logs
salt '*' cmd.run 'journalctl -u salt-minion -n 50'

# NGINX logs
salt '*' cmd.run 'tail -f /var/log/nginx/access.log'

Monitoring and Maintenance

Health Checks

Create a health check script:

#!/bin/bash
# /usr/local/bin/salt-health-check.sh

echo "=== Salt Master Health Check ==="
echo

echo "Master Status:"
systemctl status salt-master | grep Active

echo
echo "Minion Connectivity:"
salt '*' test.ping

echo
echo "Git Sync Status:"
salt-run fileserver.update
salt-run fileserver.envs

echo
echo "Pillar Status:"
salt '*' pillar.get _errors

echo
echo "Service Status:"
salt '*' cmd.run 'systemctl is-active nginx'

echo
echo "SSL Certificates:"
salt '*' cmd.run 'certbot certificates'

echo
echo "Disk Usage:"
salt '*' disk.usage

echo
echo "Memory Usage:"
salt '*' status.meminfo

Automated Alerts

# Add to states for monitoring
/usr/local/bin/check-and-alert.sh:
  file.managed:
    - source: salt://monitoring/check-and-alert.sh
    - mode: "0755"

monitoring-cron:
  cron.present:
    - name: /usr/local/bin/check-and-alert.sh
    - user: root
    - minute: "*/15"

Backup Strategy

# Automated backups
backup-cron:
  cron.present:
    - name: tar czf /backup/salt-$(date +\%Y\%m\%d).tar.gz /srv/salt /srv/pillar /etc/salt
    - user: root
    - hour: 2
    - minute: 0

Scaling Up

Adding New Environments

Update pillar:

nginx:
  sites:
    - server_name: qa.citizix.com
      root: /var/www/qa
      env: qa

Update DNS
Push to git
Wait 5 minutes for auto-apply

Adding New Minions

Install Salt minion on new server
Configure minion:

master: 10.2.10.10
id: new-server
grains:
  roles:
    - web

Accept key on master:

salt-key -a new-server

Add to firewall whitelist:

# pillar/common.sls
salt_master:
  allowed_ips:
    - 10.2.10.10
    - 10.2.10.11
    - NEW_SERVER_IP

Migrating to Separate Servers

When traffic grows:

Provision new server
Install and configure minion
Update DNS for specific environment
Apply states to new server
Verify and cut over

Performance Tuning

Salt Master

# /etc/salt/master.d/performance.conf
worker_threads: 5
timeout: 60
keep_jobs: 24
max_event_size: 1048576

NGINX

# Increase worker processes
worker_processes auto;

# Tune buffer sizes
client_body_buffer_size 10K;
client_header_buffer_size 1k;
client_max_body_size 8m;
large_client_header_buffers 2 1k;

# Enable caching
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m;

Conclusion

You now have a production-grade SaltStack setup with:

✅ Git-backed configuration management ✅ Multi-environment support on single server ✅ Automated SSL certificates ✅ Security hardening ✅ Self-healing infrastructure (auto-apply every 5 minutes) ✅ Scalable architecture ✅ Comprehensive troubleshooting knowledge

Key Takeaways

Always enable default_include - Most common mistake
Salt runs as salt user - Set permissions accordingly
Use env: base in git_pillar - Essential for pillar to load
Test with test=True - Avoid breaking production
Keep SSH access secure - Test keys before hardening
Monitor logs actively - Catch issues early
Version control everything - Git is your safety net

Next Steps

Set up monitoring (Prometheus/Grafana)
Implement log aggregation (ELK stack)
Add automated testing (Salt’s test suite)
Explore Salt Reactor for event-driven automation
Implement blue-green deployments
Add application deployment states

Resources

Salt Documentation: https://docs.saltproject.io
Troubleshooting guide: See TROUBLESHOOTING.md in repo
Community: https://saltproject.io/community/

Feedback

Found this guide helpful? Have questions or suggestions? Reach out on Twitter or leave a comment below!

Published on Citizix.com - Deep dives into DevOps, Infrastructure as Code, and Production Systems

Author: Experienced with real-world Salt deployments managing hundreds of servers

Last Updated: January 2025

Introduction

Table of Contents

Architecture Overview

The Challenge

Solution Architecture

Why This Architecture?

Infrastructure Setup

Servers Required

DNS Configuration

Initial Server Setup

Project Structure

Salt Master Configuration

Step 1: Install Salt Master

Step 2: Enable master.d/ Configuration

Step 3: Configure SSH Deploy Keys

Step 4: Configure GitFS (Git-Backed States)

Step 5: Configure Git Pillar

Step 6: Configure Master Security

Step 7: Start Salt Master

Step 8: Configure Firewall

Git-Backed States and Pillar

Create Git Repository

Create States

1. Top File (salt/top.sls)

2. Hardening State (salt/hardening/init.sls)

SSH Config Template (salt/hardening/sshd_config.j2)

3. Firewall State (salt/firewall/init.sls)

Multi-Environment Configuration

Pillar Structure

1. Pillar Top File (pillar/top.sls)

2. Common Pillar (pillar/common.sls)

3. Multi-Environment Pillar (pillar/multi-env.sls)

NGINX Multi-Site Setup

NGINX State (salt/nginx/init.sls)

Security Headers (salt/nginx/files/security-headers.conf)

Site Config Template (salt/nginx/files/site.conf.j2)

Index Page Template (salt/nginx/files/index.html.j2)

SSL Automation with Let’s Encrypt

Let’s Encrypt State (salt/letsencrypt/init.sls)

Push to Git

Salt Minion Configuration

Configure Web Server Minion

Accept Minion Key on Master

Verify Git Configuration

Applying States

Phase 1: Hardening (CAREFUL!)

Phase 2: Firewall

Phase 3: NGINX

Phase 4: SSL Certificates

Troubleshooting Real Issues

Issue 1: master.d/ Configs Not Loading

Issue 2: Pillar Data Not Loading

Issue 3: SSH Key Permission Denied

Issue 4: Firewall Module Not Available

Issue 5: pygit2 Not Found

Issue 6: Minion Can’t Connect After Firewall

Issue 7: SELinux Blocking NGINX

Production Best Practices

1. Always Use Test Mode First

2. Version Control Everything

3. Use Specific Targeting

4. Monitor State Runs

5. Keep Pillar Secure

6. Document Changes

7. Backup Before Major Changes

8. Use Requisites Properly

9. Implement Change Control

10. Monitor Logs

Monitoring and Maintenance

Health Checks

Automated Alerts

Backup Strategy

Scaling Up

Adding New Environments

Adding New Minions

Migrating to Separate Servers

Performance Tuning

Salt Master

NGINX

Conclusion