Secure Kubernetes With Firewalld and SaltStack on AlmaLinux 10

Tue, 28 Jul 2026 17:46:54 +0000

When running Kubernetes clusters in production, securing your infrastructure goes beyond pod security policies and RBAC. One critical—yet often overlooked—aspect is proper firewall configuration. Many tutorials and quick-start guides leave cluster ports wide open to the internet, creating serious security vulnerabilities that attackers can exploit.

In this comprehensive guide, I’ll show you how I implemented automated, auditable firewall rules for production Kubernetes clusters using SaltStack and firewalld on AlmaLinux 10. This approach works equally well with k3s, standard Kubernetes, or any CNI-based cluster, and requires zero manual intervention to maintain.

The Security Problem Nobody Talks About

After following a typical k3s or Kubernetes installation guide, you might run firewall-cmd --list-all and see something like this:

$ firewall-cmd --list-all
public (active)
 ports: 6443/tcp 8472/udp 10250/tcp 30000-32767/tcp 30000-32767/udp
 rich rules:

This output should make any security-conscious engineer nervous. Here’s what’s exposed:

Port 10250 (kubelet API): Full cluster management capabilities accessible from anywhere
Port 8472 (flannel VXLAN): Pod networking protocol exposed to the internet
Ports 30000-32767 (NodePort range): Every NodePort service publicly accessible
No rich rules: No source IP restrictions whatsoever

This is a production incident waiting to happen.

Real-world impact:

Attackers can enumerate cluster internals via kubelet API
Pod-to-pod traffic can be intercepted or disrupted
Services intended for internal use are exposed publicly
Compliance frameworks (PCI-DSS, SOC 2) would fail this configuration

The Solution: Automated, Auditable Firewall Management

We’ll build a Salt-based infrastructure-as-code solution that:

✅ Restricts cluster ports to whitelisted IPs using firewalld rich rules ✅ Automatically removes orphaned rules when IPs change ✅ Configures CNI networking properly to prevent 504 Gateway Timeouts ✅ Version controls everything for full auditability ✅ Maintains itself with zero manual intervention ✅ Scales effortlessly to clusters of any size

Real-World Cluster Architecture

The layout below mirrors a three-node cluster I run in production. The addresses are from 203.0.113.0/24, the RFC 5737 documentation range — substitute your own. Do not publish your real node addresses in a blog post, a README, or a public Salt repo: a public master IP plus a list of the ports you opened on it is most of the reconnaissance work done for an attacker.

Master Node: k8s-master-01 (203.0.113.10)
Worker Node 1: k8s-node-1 (203.0.113.21)
Worker Node 2: k8s-node-2 (203.0.113.22)

Access Policy:

Public (Internet-facing):

API Server (6443) - Required for kubectl and CI/CD
HTTP/HTTPS (80/443) - Ingress controller traffic

Cluster-Internal (Restricted):

Kubelet (10250) - Master ↔ Workers communication
Flannel VXLAN (8472/UDP) - Pod networking overlay
Metrics (9100, 9104) - Monitoring endpoints

Prerequisites

Before diving in, ensure you have:

AlmaLinux 10 (or Rocky Linux 9/10, RHEL 9/10, CentOS Stream 9/10) on all cluster nodes
firewalld 1.0 or newer — AlmaLinux 10 ships firewalld 2.x on the nftables backend. Everything here works on the 0.9 series too, with one caveat noted in Part 4
Salt Master 3007 LTS (3006 works) installed and configured
Salt Minions running on all cluster nodes with accepted keys
k3s or Kubernetes already installed and operational
Git repository for Salt states (GitFS recommended)
SSH access with sudo privileges to all nodes
Console access (IPMI/hosting panel) as a safety net

New to SaltStack?

SaltStack is an infrastructure automation and configuration management tool. If you’re new to Salt, start with Getting Started with SaltStack on AlmaLinux 10, then read Production-Grade SaltStack: Multi-Environment Setup with GitOps for the GitFS layout this guide assumes. The official documentation covers installation in depth.

Do not have a cluster yet? Set up a k3s cluster first.

Part 1: Understanding the Salt Architecture

Our Salt configuration follows a clean, maintainable structure:

salt/
├── pillar/ # Configuration data (IPs, ports)
│ ├── top.sls # Pillar assignment
│ └── common.sls # Cluster configuration
├── salt-states/ # Desired system state
│ ├── top.sls # State assignment
│ ├── grains/ # Auto role assignment
│ │ └── init.sls
│ └── firewall/ # Firewall rules
│ └── init.sls
└── docs/ # Documentation
 └── K8S-FIREWALL-SETUP.md

Key Concepts:

Pillar: Stores configuration data like IP addresses and ports. Think of it as your cluster’s inventory.
States: Defines the desired configuration. Salt ensures reality matches the state.
Grains: Metadata about nodes (OS, roles). We use this for automatic role assignment.

Part 2: Automatic Role Assignment with Grains

Instead of manually configuring roles for each node, we’ll use Salt grains to automatically assign roles based on hostname patterns. This is infrastructure-as-code at its finest.

File: salt/salt-states/grains/init.sls

# Auto-assign roles based on minion ID patterns
# No manual configuration required!

{% set minion_id = grains['id'] %}

# Kubernetes master server
# Matches: k8s-master-01, k8s-master-prod-02, kube-master-01, etc.
{% if minion_id.startswith('k8s-master-') or minion_id.startswith('kube-master-') %}

set_k8s_master_role:
 grains.present:
 - name: roles
 - value:
 - k8s-master
 - force: True

sync_k8s_master_grains:
 module.run:
 - name: saltutil.sync_grains
 - require:
 - grains: set_k8s_master_role

# Kubernetes worker nodes
# Matches: k8s-node-*, kube-node-*, etc.
{% elif minion_id.startswith('k8s-node-') or minion_id.startswith('kube-node-') %}

set_k8s_node_role:
 grains.present:
 - name: roles
 - value:
 - k8s-node
 - force: True

sync_k8s_node_grains:
 module.run:
 - name: saltutil.sync_grains
 - require:
 - grains: set_k8s_node_role

{% endif %}

What this achieves:

Servers matching k8s-master-* automatically get role: k8s-master
Servers matching k8s-node-* automatically get role: k8s-node
Roles determine which firewall rules are applied
Add a new node? Just name it correctly and it’s auto-configured

Resist the temptation to special-case one hostname here. The moment the state carries an exact minion ID, “no manual configuration required” stops being true and the next node you build silently gets no role at all.

Naming Convention

Establish a clear hostname naming convention for your infrastructure. Our pattern:

Masters: k8s-master-[environment]-[number]
Workers: k8s-node-[environment]-[number]
Example: k8s-master-prod-01, k8s-node-prod-02

Part 3: Cluster Configuration in Pillar

Define your cluster topology once, in one place:

File: salt/pillar/common.sls

# Kubernetes (k3s) cluster configuration
# This is your single source of truth for cluster topology
kubernetes:
 # Master node IP with /32 notation
 master_ip: 203.0.113.10/32

 # Worker node IPs
 # Add/remove nodes here - firewall rules update automatically
 node_ips:
 - 203.0.113.21/32 # k8s-node-1 (worker-01.prod.example.com)
 - 203.0.113.22/32 # k8s-node-2 (worker-02.prod.example.com)

 # k3s default CNI network ranges
 # Change if you customized during installation
 pod_cidr: 10.42.0.0/16 # Pod network (Flannel default)
 service_cidr: 10.43.0.0/16 # Service network (k3s default)

 # Master node ports
 master_ports:
 api_server: 6443 # Kubernetes API (public access)
 kubelet: 10250 # Kubelet API (cluster-only)
 flannel_vxlan: 8472 # Flannel VXLAN (cluster-only, UDP)
 node_exporter: 9100 # Prometheus node exporter
 metrics_server: 9104 # Metrics API

 # Worker node ports
 node_ports:
 kubelet: 10250 # Kubelet API (cluster-only)
 flannel_vxlan: 8472 # Flannel VXLAN (cluster-only, UDP)
 node_exporter: 9100 # Node exporter (cluster-only)
 metrics_server: 9104 # Metrics API (cluster-only)

 # Public services exposed on master
 # Ingress controller listens here
 public_services:
 - http # Port 80
 - https # Port 443

File: salt/pillar/top.sls

base:
 # All minions get common configuration
 "*":
 - common

CIDR Notation

Always use /32 notation for single IP addresses in pillar. This prevents ambiguity:

✅ Correct: 203.0.113.10/32
❌ Wrong: 203.0.113.10 (ambiguous, might be interpreted as /24)

Part 4: Master Node Firewall Configuration

Now for the critical component: firewall rules with automatic orphan cleanup.

File: salt/salt-states/firewall/init.sls (Master portion)

# Ensure firewalld is running
firewalld-service:
 service.running:
 - name: firewalld
 - enable: True

# Kubernetes Master firewall rules
{% if 'k8s-master' in grains.get('roles', []) %}
{% set k8s_config = pillar.get('kubernetes', {}) %}
{% set node_ips = k8s_config.get('node_ips', []) %}
{% set api_port = k8s_config.get('master_ports', {}).get('api_server', 6443) %}
{% set kubelet_port = k8s_config.get('master_ports', {}).get('kubelet', 10250) %}
{% set flannel_port = k8s_config.get('master_ports', {}).get('flannel_vxlan', 8472) %}
{% set node_exporter = k8s_config.get('master_ports', {}).get('node_exporter', 9100) %}
{% set metrics_port = k8s_config.get('master_ports', {}).get('metrics_server', 9104) %}

# ============================================================================
# SECURITY CLEANUP: Remove insecure direct port openings
# ============================================================================
# k3s/k8s installers often add ports directly to the firewall, making them
# accessible from the internet. We remove these and replace with IP-restricted
# rich rules below.

cleanup-k8s-master-direct-kubelet-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ kubelet_port }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ kubelet_port }}/tcp"
 - require:
 - service: firewalld-service

cleanup-k8s-master-direct-flannel-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ flannel_port }}/udp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ flannel_port }}/udp"
 - require:
 - service: firewalld-service

cleanup-k8s-master-direct-node-exporter-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ node_exporter }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ node_exporter }}/tcp"
 - require:
 - service: firewalld-service

cleanup-k8s-master-direct-metrics-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ metrics_port }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ metrics_port }}/tcp"
 - require:
 - service: firewalld-service

# ============================================================================
# PUBLIC ACCESS: API Server and Ingress
# ============================================================================

# Kubernetes API server (6443) - Required for kubectl, CI/CD, etc.
# Note every check below queries the PERMANENT config, matching what we change.
# Mixing `--permanent --add-*` with a runtime `--list-*` check is the most common
# way these states end up reporting "changed" on every single run.
k8s-master-api-server:
 cmd.run:
 - name: firewall-cmd --permanent --add-port={{ api_port }}/tcp
 - unless: firewall-cmd --permanent --query-port={{ api_port }}/tcp
 - require:
 - service: firewalld-service

# HTTP/HTTPS for ingress controller on master
{% for service in k8s_config.get('public_services', []) %}
k8s-master-public-{{ service }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-service={{ service }}
 - unless: firewall-cmd --permanent --query-service={{ service }}
 - require:
 - service: firewalld-service
{% endfor %}

# ============================================================================
# CNI NETWORKING: Critical for pod-to-pod communication
# ============================================================================
# Without these settings, pods cannot communicate across nodes and you'll see
# 504 Gateway Timeout errors for cross-node traffic.

# Enable masquerading for pod network NAT
k8s-master-masquerade:
 cmd.run:
 - name: firewall-cmd --permanent --add-masquerade
 - unless: firewall-cmd --permanent --query-masquerade
 - require:
 - service: firewalld-service

# Add CNI interfaces to trusted zone
# This allows bidirectional pod traffic without interference
k8s-master-trust-cni0:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-interface=cni0
 - unless: firewall-cmd --permanent --zone=trusted --query-interface=cni0
 - require:
 - service: firewalld-service

k8s-master-trust-flannel:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-interface=flannel.1
 - unless: firewall-cmd --permanent --zone=trusted --query-interface=flannel.1
 - require:
 - service: firewalld-service

# Trust pod and service network CIDRs
{% set pod_cidr = k8s_config.get('pod_cidr', '10.42.0.0/16') %}
{% set service_cidr = k8s_config.get('service_cidr', '10.43.0.0/16') %}

k8s-master-trust-pod-cidr:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-source={{ pod_cidr }}
 - unless: firewall-cmd --permanent --zone=trusted --query-source={{ pod_cidr }}
 - require:
 - service: firewalld-service

k8s-master-trust-service-cidr:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-source={{ service_cidr }}
 - unless: firewall-cmd --permanent --zone=trusted --query-source={{ service_cidr }}
 - require:
 - service: firewalld-service

# Allow traffic forwarded *within* the trusted zone (firewalld 0.9+).
# This is what actually carries pod-to-pod traffic across cni0/flannel.1 on
# modern firewalld, and it is the supported replacement for most --direct rules.
k8s-master-trusted-forward:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-forward
 - unless: firewall-cmd --permanent --zone=trusted --query-forward
 - require:
 - service: firewalld-service

# ============================================================================
# FORWARD CHAIN RULES: belt-and-braces for cross-node pod communication
# ============================================================================
# The trusted-zone configuration above is what does the real work on firewalld
# 1.0+. These explicit FORWARD rules are a compatibility net for older firewalld
# (RHEL/Rocky 8, firewalld 0.9) and for hosts where something else has narrowed
# the forward path. Drop this whole section if you only target AlmaLinux 10.

k8s-master-forward-cni0-in:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-master-forward-cni0-out:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -o cni0 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -o cni0 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-master-forward-flannel-in:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i flannel.1 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -i flannel.1 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-master-forward-flannel-out:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -o flannel.1 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -o flannel.1 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-master-forward-pod-cidr-src:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -s {{ pod_cidr }} -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 1 -s {{ pod_cidr }} -j ACCEPT
 - require:
 - service: firewalld-service

k8s-master-forward-pod-cidr-dst:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -d {{ pod_cidr }} -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 1 -d {{ pod_cidr }} -j ACCEPT
 - require:
 - service: firewalld-service

# ============================================================================
# CLUSTER-INTERNAL PORTS: IP-restricted access
# ============================================================================
# These ports are restricted to worker node IPs only using firewalld rich rules.
# The cleanup mechanism ensures only current pillar IPs are allowed.

{% if node_ips %}
# AUTOMATIC ORPHAN CLEANUP
# Remove ALL existing rich rules for these ports before adding new ones
# This ensures when you update node IPs in pillar, old IPs are automatically removed
#
# Note the `while IFS= read -r` loop: a rich rule contains spaces, so iterating
# over `$(...)` would word-split each rule into fragments and every removal would
# fail silently. Read one whole rule per line instead. Matching on the full
# `port port="N" protocol="P"` fragment also stops port 9100 rules from colliding
# with unrelated rules that merely contain the digits 9100.

cleanup-k8s-master-kubelet-rules:
 cmd.run:
 - name: |
 firewall-cmd --permanent --list-rich-rules \
 | grep 'port port="{{ kubelet_port }}" protocol="tcp"' \
 | while IFS= read -r rule; do
 firewall-cmd --permanent --remove-rich-rule="$rule"
 done
 - onlyif: firewall-cmd --permanent --list-rich-rules | grep -q 'port port="{{ kubelet_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service

cleanup-k8s-master-flannel-rules:
 cmd.run:
 - name: |
 firewall-cmd --permanent --list-rich-rules \
 | grep 'port port="{{ flannel_port }}" protocol="udp"' \
 | while IFS= read -r rule; do
 firewall-cmd --permanent --remove-rich-rule="$rule"
 done
 - onlyif: firewall-cmd --permanent --list-rich-rules | grep -q 'port port="{{ flannel_port }}" protocol="udp"'
 - require:
 - service: firewalld-service

cleanup-k8s-master-node-exporter-rules:
 cmd.run:
 - name: |
 firewall-cmd --permanent --list-rich-rules \
 | grep 'port port="{{ node_exporter }}" protocol="tcp"' \
 | while IFS= read -r rule; do
 firewall-cmd --permanent --remove-rich-rule="$rule"
 done
 - onlyif: firewall-cmd --permanent --list-rich-rules | grep -q 'port port="{{ node_exporter }}" protocol="tcp"'
 - require:
 - service: firewalld-service

cleanup-k8s-master-metrics-rules:
 cmd.run:
 - name: |
 firewall-cmd --permanent --list-rich-rules \
 | grep 'port port="{{ metrics_port }}" protocol="tcp"' \
 | while IFS= read -r rule; do
 firewall-cmd --permanent --remove-rich-rule="$rule"
 done
 - onlyif: firewall-cmd --permanent --list-rich-rules | grep -q 'port port="{{ metrics_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service

# Now add ONLY the current worker node IPs from pillar
#
# The `unless` checks use `grep -qF` against the exact rule fragment. An earlier
# version of this state matched `'<port>.*<ip>'`, which never matched, because
# firewalld prints the source address BEFORE the port. The states re-ran on every
# highstate and reported changes that had not happened.
{% for node_ip in node_ips %}
# Allow kubelet (10250) from worker node {{ node_ip }}
k8s-master-kubelet-{{ node_ip | replace('.', '-') | replace('/', '-') }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ node_ip }}" port port="{{ kubelet_port }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ node_ip }}" port port="{{ kubelet_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service
 - cmd: cleanup-k8s-master-kubelet-rules

# Allow flannel VXLAN (8472 UDP) from worker node {{ node_ip }}
k8s-master-flannel-{{ node_ip | replace('.', '-') | replace('/', '-') }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ node_ip }}" port port="{{ flannel_port }}" protocol="udp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ node_ip }}" port port="{{ flannel_port }}" protocol="udp"'
 - require:
 - service: firewalld-service
 - cmd: cleanup-k8s-master-flannel-rules

# Allow node exporter (9100) from worker node {{ node_ip }}
k8s-master-node-exporter-{{ node_ip | replace('.', '-') | replace('/', '-') }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ node_ip }}" port port="{{ node_exporter }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ node_ip }}" port port="{{ node_exporter }}" protocol="tcp"'
 - require:
 - service: firewalld-service
 - cmd: cleanup-k8s-master-node-exporter-rules

# Allow metrics server (9104) from worker node {{ node_ip }}
k8s-master-metrics-{{ node_ip | replace('.', '-') | replace('/', '-') }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ node_ip }}" port port="{{ metrics_port }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ node_ip }}" port port="{{ metrics_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service
 - cmd: cleanup-k8s-master-metrics-rules
{% endfor %}
{% else %}
# Warning if no worker nodes configured
k8s-master-no-nodes-configured:
 test.configurable_test_state:
 - name: WARNING - No k8s worker nodes configured
 - changes: False
 - result: True
 - comment: |
 WARNING: No worker node IPs configured in pillar.kubernetes.node_ips
 Cluster internal ports will NOT be opened in firewall.
 Configure worker node IPs in pillar/common.sls to enable cluster communication.
{% endif %}

# ============================================================================
# RELOAD FIREWALL: Apply all changes
# ============================================================================
# This state sits OUTSIDE the node_ips conditional on purpose. Every change
# above is `--permanent`, so nothing takes effect until a reload. Nesting the
# reload inside the conditional meant a cluster with no workers configured yet
# would apply masquerading, trusted interfaces and FORWARD rules and then never
# load them into the running firewall.
firewall-reload-after-k8s-master:
 cmd.run:
 - name: firewall-cmd --reload
 - onchanges:
 - cmd: cleanup-k8s-master-direct-kubelet-port
 - cmd: cleanup-k8s-master-direct-flannel-port
 - cmd: cleanup-k8s-master-direct-node-exporter-port
 - cmd: cleanup-k8s-master-direct-metrics-port
 - cmd: k8s-master-api-server
{% for service in k8s_config.get('public_services', []) %}
 - cmd: k8s-master-public-{{ service }}
{% endfor %}
 - cmd: k8s-master-masquerade
 - cmd: k8s-master-trust-cni0
 - cmd: k8s-master-trust-flannel
 - cmd: k8s-master-trust-pod-cidr
 - cmd: k8s-master-trust-service-cidr
 - cmd: k8s-master-trusted-forward
 - cmd: k8s-master-forward-cni0-in
 - cmd: k8s-master-forward-cni0-out
 - cmd: k8s-master-forward-flannel-in
 - cmd: k8s-master-forward-flannel-out
 - cmd: k8s-master-forward-pod-cidr-src
 - cmd: k8s-master-forward-pod-cidr-dst
{% if node_ips %}
 - cmd: cleanup-k8s-master-kubelet-rules
 - cmd: cleanup-k8s-master-flannel-rules
 - cmd: cleanup-k8s-master-node-exporter-rules
 - cmd: cleanup-k8s-master-metrics-rules
{% for node_ip in node_ips %}
 - cmd: k8s-master-kubelet-{{ node_ip | replace('.', '-') | replace('/', '-') }}
 - cmd: k8s-master-flannel-{{ node_ip | replace('.', '-') | replace('/', '-') }}
 - cmd: k8s-master-node-exporter-{{ node_ip | replace('.', '-') | replace('/', '-') }}
 - cmd: k8s-master-metrics-{{ node_ip | replace('.', '-') | replace('/', '-') }}
{% endfor %}
{% endif %}
{% endif %}

This is a production-grade configuration. Let me break down the key innovations:

Automatic Orphan Cleanup

The cleanup mechanism is the secret sauce. Before adding new rich rules, we remove ALL existing rules for that port. This means when you update node IPs in pillar, old IPs are automatically removed—no orphans, no manual cleanup!

Old way: Update IP in config, manually remove old firewall rule, apply new rule Our way: Update IP in pillar, apply Salt state—done!

Three details in that state are worth calling out, because they are the ones that bite in practice.

Every check queries the permanent config. firewall-cmd --permanent --add-port writes to disk; firewall-cmd --list-ports reads the running firewall. Guarding a permanent change with a runtime check means the guard never matches, the state runs on every highstate, and your Salt output is full of changes that did not happen. All the unless/onlyif clauses above use --permanent --query-*, which is both exact and cheap.

Rich rules contain spaces. The cleanup loop reads one rule per line with while IFS= read -r. Iterating over $(...) instead would split each rule on whitespace into fragments like family="ipv4", and every --remove-rich-rule would be handed a fragment and fail — leaving exactly the orphans the state exists to remove.

Match the whole port fragment, not the digits. Grepping for 9100 matches any rule containing those digits anywhere. Grepping for port port="9100" protocol="tcp" matches the port and nothing else.

--direct is deprecated

The firewall-cmd --direct interface has been deprecated since firewalld 1.0. It still works on AlmaLinux 10’s firewalld 2.x, but it injects raw rules ahead of firewalld’s own chains and it is the first thing to break on a major upgrade.

On firewalld 1.0+ the trusted zone already has target ACCEPT, and --zone=trusted --add-forward permits forwarding between interfaces and sources in that zone. Between them, the k8s-*-trust-* and k8s-*-trusted-forward states above cover what the --direct FORWARD rules were doing.

Keep the --direct block only if you also run firewalld 0.9 hosts (RHEL/Rocky 8), where --add-forward does not exist. On an AlmaLinux 10-only cluster, delete it and verify pod-to-pod traffic still works before you call it done.

Part 5: Worker Node Firewall Configuration

Worker nodes have similar but simplified rules since they don’t serve public traffic:

This block belongs in the same init.sls, appended below the master section — it is not a second file. Every state here requires firewalld-service, which is declared once at the top of the master block, outside the role conditional. Paste this on its own and Salt will fail rendering with an unresolved requisite.

File: salt/salt-states/firewall/init.sls (Worker portion - append after the master section)

# Kubernetes Worker Node firewall rules
{% if 'k8s-node' in grains.get('roles', []) %}
{% set k8s_config = pillar.get('kubernetes', {}) %}
{% set master_ip = k8s_config.get('master_ip', '') %}
{% set node_ips = k8s_config.get('node_ips', []) %}
{% set kubelet_port = k8s_config.get('node_ports', {}).get('kubelet', 10250) %}
{% set flannel_port = k8s_config.get('node_ports', {}).get('flannel_vxlan', 8472) %}
{% set node_exporter = k8s_config.get('node_ports', {}).get('node_exporter', 9100) %}
{% set metrics_port = k8s_config.get('node_ports', {}).get('metrics_server', 9104) %}
{% set my_ips = grains.get('ipv4', []) %}

# ============================================================================
# SECURITY CLEANUP: Remove insecure direct port openings
# ============================================================================

cleanup-k8s-node-direct-kubelet-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ kubelet_port }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ kubelet_port }}/tcp"
 - require:
 - service: firewalld-service

cleanup-k8s-node-direct-flannel-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ flannel_port }}/udp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ flannel_port }}/udp"
 - require:
 - service: firewalld-service

cleanup-k8s-node-direct-node-exporter-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ node_exporter }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ node_exporter }}/tcp"
 - require:
 - service: firewalld-service

cleanup-k8s-node-direct-metrics-port:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port={{ metrics_port }}/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "{{ metrics_port }}/tcp"
 - require:
 - service: firewalld-service

# Remove NodePort range (workers should NOT expose NodePorts to internet)
cleanup-k8s-node-nodeport-tcp:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port=30000-32767/tcp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "30000-32767/tcp"
 - require:
 - service: firewalld-service

cleanup-k8s-node-nodeport-udp:
 cmd.run:
 - name: firewall-cmd --permanent --remove-port=30000-32767/udp 2>/dev/null || true
 - onlyif: firewall-cmd --permanent --list-ports | grep -q "30000-32767/udp"
 - require:
 - service: firewalld-service

# Remove HTTP/HTTPS from worker nodes (only master serves ingress)
cleanup-k8s-node-http:
 cmd.run:
 - name: firewall-cmd --permanent --remove-service=http
 - onlyif: firewall-cmd --permanent --query-service=http
 - require:
 - service: firewalld-service

cleanup-k8s-node-https:
 cmd.run:
 - name: firewall-cmd --permanent --remove-service=https
 - onlyif: firewall-cmd --permanent --query-service=https
 - require:
 - service: firewalld-service

# ============================================================================
# CNI NETWORKING: Same as master node
# ============================================================================

k8s-node-masquerade:
 cmd.run:
 - name: firewall-cmd --permanent --add-masquerade
 - unless: firewall-cmd --permanent --query-masquerade
 - require:
 - service: firewalld-service

k8s-node-trust-cni0:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-interface=cni0
 - unless: firewall-cmd --permanent --zone=trusted --query-interface=cni0
 - require:
 - service: firewalld-service

k8s-node-trust-flannel:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-interface=flannel.1
 - unless: firewall-cmd --permanent --zone=trusted --query-interface=flannel.1
 - require:
 - service: firewalld-service

{% set pod_cidr = k8s_config.get('pod_cidr', '10.42.0.0/16') %}
{% set service_cidr = k8s_config.get('service_cidr', '10.43.0.0/16') %}

k8s-node-trust-pod-cidr:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-source={{ pod_cidr }}
 - unless: firewall-cmd --permanent --zone=trusted --query-source={{ pod_cidr }}
 - require:
 - service: firewalld-service

k8s-node-trust-service-cidr:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-source={{ service_cidr }}
 - unless: firewall-cmd --permanent --zone=trusted --query-source={{ service_cidr }}
 - require:
 - service: firewalld-service

k8s-node-trusted-forward:
 cmd.run:
 - name: firewall-cmd --permanent --zone=trusted --add-forward
 - unless: firewall-cmd --permanent --zone=trusted --query-forward
 - require:
 - service: firewalld-service

# FORWARD chain rules (compatibility net for firewalld < 1.0 — see master notes)
k8s-node-forward-cni0-in:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-node-forward-cni0-out:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -o cni0 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -o cni0 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-node-forward-flannel-in:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i flannel.1 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -i flannel.1 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-node-forward-flannel-out:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -o flannel.1 -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 0 -o flannel.1 -j ACCEPT
 - require:
 - service: firewalld-service

k8s-node-forward-pod-cidr-src:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -s {{ pod_cidr }} -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 1 -s {{ pod_cidr }} -j ACCEPT
 - require:
 - service: firewalld-service

k8s-node-forward-pod-cidr-dst:
 cmd.run:
 - name: firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 1 -d {{ pod_cidr }} -j ACCEPT
 - unless: firewall-cmd --permanent --direct --query-rule ipv4 filter FORWARD 1 -d {{ pod_cidr }} -j ACCEPT
 - require:
 - service: firewalld-service

# ============================================================================
# CLUSTER ACCESS: Allow master and peer workers
# ============================================================================

{% if master_ip %}
# Allow kubelet (10250) from master only
k8s-node-kubelet-from-master:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ master_ip }}" port port="{{ kubelet_port }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ master_ip }}" port port="{{ kubelet_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service

# Allow flannel VXLAN (8472 UDP) from master
k8s-node-flannel-from-master:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ master_ip }}" port port="{{ flannel_port }}" protocol="udp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ master_ip }}" port port="{{ flannel_port }}" protocol="udp"'
 - require:
 - service: firewalld-service

# Allow node exporter from master
k8s-node-exporter-from-master:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ master_ip }}" port port="{{ node_exporter }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ master_ip }}" port port="{{ node_exporter }}" protocol="tcp"'
 - require:
 - service: firewalld-service

# Allow metrics server from master
k8s-node-metrics-from-master:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ master_ip }}" port port="{{ metrics_port }}" protocol="tcp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ master_ip }}" port port="{{ metrics_port }}" protocol="tcp"'
 - require:
 - service: firewalld-service

# Allow peer-to-peer flannel VXLAN from the OTHER worker nodes.
# Skip our own IP - a node does not need a rich rule to talk to itself, and the
# stray self-rule makes rule counts disagree with the node count during audits.
{% for peer_ip in node_ips if peer_ip.split('/')[0] not in my_ips %}
k8s-node-flannel-peer-{{ peer_ip | replace('.', '-') | replace('/', '-') }}:
 cmd.run:
 - name: firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{{ peer_ip }}" port port="{{ flannel_port }}" protocol="udp" accept'
 - unless: firewall-cmd --permanent --list-rich-rules | grep -qF 'source address="{{ peer_ip }}" port port="{{ flannel_port }}" protocol="udp"'
 - require:
 - service: firewalld-service
{% endfor %}
{% else %}
# Error if no master IP configured
k8s-node-no-master-configured:
 test.configurable_test_state:
 - name: ERROR - No k8s master IP configured
 - changes: False
 - result: False
 - comment: |
 ERROR: No master IP configured in pillar.kubernetes.master_ip
 Worker node firewall rules cannot be configured.
 Configure master IP in pillar/common.sls before applying to worker nodes.
{% endif %}

# Reload firewall after all rules configured (outside the master_ip conditional,
# for the same reason as on the master: permanent changes need a reload)
firewall-reload-after-k8s-node:
 cmd.run:
 - name: firewall-cmd --reload
 - onchanges:
 - cmd: cleanup-k8s-node-direct-kubelet-port
 - cmd: cleanup-k8s-node-direct-flannel-port
 - cmd: cleanup-k8s-node-direct-node-exporter-port
 - cmd: cleanup-k8s-node-direct-metrics-port
 - cmd: cleanup-k8s-node-nodeport-tcp
 - cmd: cleanup-k8s-node-nodeport-udp
 - cmd: cleanup-k8s-node-http
 - cmd: cleanup-k8s-node-https
 - cmd: k8s-node-masquerade
 - cmd: k8s-node-trust-cni0
 - cmd: k8s-node-trust-flannel
 - cmd: k8s-node-trust-pod-cidr
 - cmd: k8s-node-trust-service-cidr
 - cmd: k8s-node-trusted-forward
 - cmd: k8s-node-forward-cni0-in
 - cmd: k8s-node-forward-cni0-out
 - cmd: k8s-node-forward-flannel-in
 - cmd: k8s-node-forward-flannel-out
 - cmd: k8s-node-forward-pod-cidr-src
 - cmd: k8s-node-forward-pod-cidr-dst
{% if master_ip %}
 - cmd: k8s-node-kubelet-from-master
 - cmd: k8s-node-flannel-from-master
 - cmd: k8s-node-exporter-from-master
 - cmd: k8s-node-metrics-from-master
{% for peer_ip in node_ips if peer_ip.split('/')[0] not in my_ips %}
 - cmd: k8s-node-flannel-peer-{{ peer_ip | replace('.', '-') | replace('/', '-') }}
{% endfor %}
{% endif %}
{% endif %}

Part 6: State Assignment

Tell Salt which states to apply:

File: salt/salt-states/top.sls

base:
 # All Kubernetes nodes get both states
 "k8s-*":
 - grains
 - firewall

Part 7: Deployment Process

Time to deploy this to your production cluster. Follow these steps carefully:

Step 1: Commit Configuration to Git

cd /path/to/your/salt/repo

# Add all configuration files
git add salt/pillar/common.sls
git add salt/pillar/top.sls
git add salt/salt-states/grains/init.sls
git add salt/salt-states/firewall/init.sls
git add salt/salt-states/top.sls

# Commit with descriptive message
git commit -m "Add Kubernetes firewall configuration with automatic orphan cleanup

- Implement role-based firewall rules for k8s-master and k8s-node
- Add automatic orphan cleanup for IP changes
- Configure CNI networking (masquerading, trusted interfaces, FORWARD rules)
- Restrict cluster ports (kubelet, flannel) to whitelisted IPs only
- Open API server (6443) and ingress (80/443) on master
- Remove insecure direct port openings
"

# Push to remote repository
git push origin main

Step 2: Update Salt Master from Git

# Update Salt fileserver from Git
sudo salt-run fileserver.update

# Refresh pillar data on all minions
sudo salt '*' saltutil.refresh_pillar

# Verify pillar data is correct
sudo salt 'k8s-*' pillar.get kubernetes
# Should show your cluster configuration

Step 3: Apply Grains State (Role Assignment)

# Apply grains state to auto-assign roles
sudo salt 'k8s-*' state.apply grains

# Verify master node has correct role
sudo salt 'k8s-master-01' grains.get roles
# Expected output: - k8s-master

# Verify worker nodes have correct role
sudo salt 'k8s-node-*' grains.get roles
# Expected output: - k8s-node

Wrong Role?

If a node gets the wrong role, check:

Hostname matches patterns in grains/init.sls
Grains state applied successfully without errors
Run sudo salt 'node-name' saltutil.sync_grains to force sync

Step 4: Apply Firewall State (Test Mode First)

# IMPORTANT: Always test first!
sudo salt 'k8s-*' state.apply firewall test=True

# Review the output carefully
# Look for:
# - Cleanup commands removing insecure direct ports
# - Rich rules being added for each worker node
# - CNI networking configuration
# - FORWARD chain rules

Review the test output. You should see actions like:

cleanup-k8s-master-direct-kubelet-port: Will remove port 10250/tcp
k8s-master-kubelet-203-0-113-21-32: Will add rich rule for worker node 1
k8s-master-trust-cni0: Will add cni0 to trusted zone

If everything looks good, apply for real:

# Apply firewall configuration
sudo salt 'k8s-*' state.apply firewall

# Watch the output for errors
# This will take 30-60 seconds per node

Emergency Access

Before applying firewall changes, ensure you have:

Console/IPMI access to all nodes
A tested rollback plan
Your SSH private key handy

If something goes wrong and you lose SSH access, use console access to:

sudo systemctl stop firewalld

Part 8: Comprehensive Verification

After deployment, thoroughly verify the configuration:

Verify Firewall Rules on Master

sudo salt 'k8s-master-01' cmd.run 'firewall-cmd --list-all'

Expected output:

k8s-master-01:
 public (default, active)
 interfaces: eth0
 ports: 22/tcp 6443/tcp 80/tcp 443/tcp
 masquerade: yes
 rich rules:
 rule family="ipv4" source address="203.0.113.21/32" port port="10250" protocol="tcp" accept
 rule family="ipv4" source address="203.0.113.22/32" port port="10250" protocol="tcp" accept
 rule family="ipv4" source address="203.0.113.21/32" port port="8472" protocol="udp" accept
 rule family="ipv4" source address="203.0.113.22/32" port port="8472" protocol="udp" accept
 rule family="ipv4" source address="203.0.113.21/32" port port="9100" protocol="tcp" accept
 rule family="ipv4" source address="203.0.113.22/32" port port="9100" protocol="tcp" accept

✅ Verify:

Direct ports: Only 22 (SSH), 6443 (API), 80/443 (Ingress)
No direct 10250, 8472, 9100 in ports list
Rich rules: One for each worker node IP
Rich rules: Separate rules for TCP and UDP ports

Verify Firewall Rules on Worker Nodes

sudo salt 'k8s-node-*' cmd.run 'firewall-cmd --list-all'

Expected output:

k8s-node-1:
 public (default, active)
 interfaces: eth0
 ports: 22/tcp
 masquerade: yes
 rich rules:
 rule family="ipv4" source address="203.0.113.10/32" port port="10250" protocol="tcp" accept
 rule family="ipv4" source address="203.0.113.10/32" port port="8472" protocol="udp" accept
 rule family="ipv4" source address="203.0.113.22/32" port port="8472" protocol="udp" accept
 rule family="ipv4" source address="203.0.113.10/32" port port="9100" protocol="tcp" accept

✅ Verify:

Only SSH port (22) in direct ports
No HTTP/HTTPS, no NodePort range
Rich rules allowing master IP for kubelet
Rich rules allowing the master plus the other workers for flannel. k8s-node-1 is 203.0.113.21, so 203.0.113.21 is absent from its own rule set — that is the self-skip in the peer loop working, not a missing rule

Verify CNI Networking Configuration

sudo salt 'k8s-*' cmd.run 'firewall-cmd --zone=trusted --list-all'

Expected output:

k8s-master-01:
 trusted
 interfaces: cni0 flannel.1
 sources: 10.42.0.0/16 10.43.0.0/16
 masquerade: no
 forward: yes

✅ Verify:

Both cni0 and flannel.1 in trusted zone
Pod CIDR (10.42.0.0/16) and Service CIDR (10.43.0.0/16) as sources
forward: yes — on firewalld 1.0+ this is what actually carries pod-to-pod traffic between the two CNI interfaces

Verify FORWARD Chain Rules

Only relevant if you kept the --direct compatibility block:

sudo salt 'k8s-*' cmd.run 'firewall-cmd --direct --get-all-rules'

Expected output:

k8s-master-01:
 ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
 ipv4 filter FORWARD 0 -o cni0 -j ACCEPT
 ipv4 filter FORWARD 0 -i flannel.1 -j ACCEPT
 ipv4 filter FORWARD 0 -o flannel.1 -j ACCEPT
 ipv4 filter FORWARD 1 -s 10.42.0.0/16 -j ACCEPT
 ipv4 filter FORWARD 1 -d 10.42.0.0/16 -j ACCEPT

✅ Verify:

FORWARD rules for both cni0 and flannel.1
Both input (-i) and output (-o) directions
FORWARD rules for pod CIDR

Empty output on an AlmaLinux 10 cluster where you deleted the --direct block is the expected result, not a failure — check forward: yes on the trusted zone instead.

Test Cluster Functionality

Deploy test workloads to verify everything works:

# Deploy nginx pods on different nodes
kubectl run test-master --image=nginx --restart=Never \
 --overrides='{"spec":{"nodeName":"k8s-master-01"}}'

kubectl run test-node1 --image=nginx --restart=Never \
 --overrides='{"spec":{"nodeName":"k8s-node-1"}}'

kubectl run test-node2 --image=nginx --restart=Never \
 --overrides='{"spec":{"nodeName":"k8s-node-2"}}'

# Wait for pods to be running
kubectl get pods -o wide

# Test cross-node communication
kubectl exec test-master -- curl -m 5 http://$(kubectl get pod test-node1 -o jsonpath='{.status.podIP}')
kubectl exec test-node1 -- curl -m 5 http://$(kubectl get pod test-node2 -o jsonpath='{.status.podIP}')

# Should succeed without timeouts

✅ All tests should pass without 504 Gateway Timeout errors!

Test External Access

# Test API server access from your workstation
kubectl --kubeconfig=/path/to/kubeconfig get nodes

# Should succeed and show all nodes Ready

# Test ingress controller (if deployed)
curl -v http://203.0.113.10
# Should reach ingress controller or default backend

# Test that cluster ports are NOT accessible externally
nc -zv 203.0.113.10 10250
# Should timeout or refuse connection (good!)

nc -zvu 203.0.113.10 8472
# Should timeout or refuse connection (good!)

Part 9: Understanding the Magic

Automatic Orphan Cleanup Explained

This is the game-changer. Here’s how it works:

Scenario: You need to replace a worker node

Old Configuration (k8s-node-2 being replaced):

node_ips:
 - 203.0.113.21/32 # k8s-node-1
 - 203.0.113.22/32 # k8s-node-2 (old)

Update Pillar (add new node, remove old):

node_ips:
 - 203.0.113.21/32 # k8s-node-1
 - 203.0.113.23/32 # k8s-node-3 (new)

Apply Salt State:

sudo salt-run fileserver.update
sudo salt 'k8s-master-01' saltutil.refresh_pillar
sudo salt 'k8s-master-01' state.apply firewall

What Happens (automatically):
- Cleanup state removes ALL rich rules for port 10250
- Cleanup state removes ALL rich rules for port 8472
- New rich rules added ONLY for IPs in current pillar:
  - 203.0.113.21/32 (k8s-node-1)
  - 203.0.113.23/32 (k8s-node-3)
- Old IP 203.0.113.22 is gone—no orphan!

Traditional Approach (manual):

# You'd have to remember to do this manually:
firewall-cmd --permanent --remove-rich-rule='rule family="ipv4" source address="203.0.113.22/32" port port="10250" protocol="tcp" accept'
firewall-cmd --permanent --remove-rich-rule='rule family="ipv4" source address="203.0.113.22/32" port port="8472" protocol="udp" accept'
# ... repeat for every port
# Easy to miss one and create an orphan!

Our Approach (automated):

# Salt does it all automatically!
sudo salt 'k8s-master-01' state.apply firewall
# Zero orphans guaranteed

CNI Networking Deep Dive

The CNI configuration prevents 504 Gateway Timeout errors. Let’s understand why each component is critical:

1. Masquerading (NAT)

firewall-cmd --permanent --add-masquerade

Without this:

Pods can’t reach services outside the cluster
External traffic can’t find its way back to pods
DNS resolution fails

2. Trusted CNI Interfaces

firewall-cmd --permanent --zone=trusted --add-interface=cni0
firewall-cmd --permanent --zone=trusted --add-interface=flannel.1

Without this:

Pod-to-pod traffic gets filtered by firewalld
Cross-node traffic drops
Services fail with connection timeouts

3. FORWARD Chain Rules

firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i cni0 -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -o cni0 -j ACCEPT

Without this:

Traffic forwarding between CNI interfaces blocked
Cross-node pod communication fails
504 Gateway Timeout errors

All three components must be present for Kubernetes networking to function properly.

Rich Rules vs Direct Port Openings

Understanding this difference is crucial for security:

Direct Port Opening (Insecure):

firewall-cmd --permanent --add-port=10250/tcp

Effect:

Port 10250 accessible from ANY IP address on the internet
No source IP verification
Maximum attack surface

Rich Rule (Secure):

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.21/32" port port="10250" protocol="tcp" accept'

Effect:

Port 10250 accessible ONLY from 203.0.113.21
Source IP verified before allowing traffic
Minimal attack surface

Our Salt configuration automatically removes any direct port openings and replaces them with source-restricted rich rules.

Part 10: Day-2 Operations

Adding a New Worker Node

This is where the automation really shines:

Step 1: Install k3s on new node and join to cluster

Step 2: Update pillar with new node IP:

# salt/pillar/common.sls
kubernetes:
 node_ips:
 - 203.0.113.21/32 # k8s-node-1
 - 203.0.113.22/32 # k8s-node-2
 - 203.0.113.23/32 # NEW: k8s-node-3

Step 3: Commit and apply:

git add salt/pillar/common.sls
git commit -m "Add k8s-node-3 (203.0.113.23) to production cluster"
git push

sudo salt-run fileserver.update
sudo salt 'k8s-*' saltutil.refresh_pillar
sudo salt 'k8s-*' state.apply firewall

Result:

Master: New rich rules added for 203.0.113.23
All nodes: Flannel VXLAN rules updated to include new peer
New node: Gets all firewall rules via grains role assignment
Total manual firewall commands: ZERO

Removing a Worker Node

Step 1: Drain and delete node from Kubernetes:

kubectl drain k8s-node-2 --ignore-daemonsets --delete-emptydir-data
kubectl delete node k8s-node-2

Step 2: Remove IP from pillar:

# salt/pillar/common.sls
kubernetes:
 node_ips:
 - 203.0.113.21/32 # k8s-node-1
 # REMOVED: - 203.0.113.22/32 # k8s-node-2 (decommissioned)
 - 203.0.113.23/32 # k8s-node-3

Step 3: Commit and apply:

git add salt/pillar/common.sls
git commit -m "Remove k8s-node-2 (203.0.113.22) from cluster - decommissioned"
git push

sudo salt-run fileserver.update
sudo salt 'k8s-*' saltutil.refresh_pillar
sudo salt 'k8s-*' state.apply firewall

Result:

Old IP 203.0.113.22 automatically removed from ALL firewall rules
No orphaned rich rules
No manual cleanup required
Full audit trail in Git history

Updating Port Numbers

Rare but sometimes necessary:

Scenario: Change kubelet port from 10250 to 10255

Step 1: Update k3s configuration on all nodes to use new port

Step 2: Update pillar:

# salt/pillar/common.sls
kubernetes:
 master_ports:
 kubelet: 10255 # Changed from 10250
 node_ports:
 kubelet: 10255 # Changed from 10250

Step 3: Apply:

git add salt/pillar/common.sls
git commit -m "Change kubelet port from 10250 to 10255"
git push

sudo salt-run fileserver.update
sudo salt 'k8s-*' saltutil.refresh_pillar
sudo salt 'k8s-*' state.apply firewall

Result:

Old port 10250 rules automatically removed
New port 10255 rules automatically added
Cleanup happens atomically

Part 11: Troubleshooting Guide

Problem: 504 Gateway Timeout for Services

Symptom: Pods can communicate on same node but timeout cross-node

Diagnosis:

# Check if FORWARD rules exist
sudo salt 'k8s-*' cmd.run 'firewall-cmd --direct --get-all-rules'

# Should show FORWARD rules for cni0 and flannel.1
# If missing, that's your problem

Solution:

# Reapply firewall state
sudo salt 'k8s-*' state.apply firewall

# Verify FORWARD rules now exist
sudo salt 'k8s-*' cmd.run 'firewall-cmd --direct --get-all-rules | grep FORWARD'

Root Cause: FORWARD chain rules not applied or removed by other tools

Problem: Nodes Show NotReady

Symptom: kubectl get nodes shows nodes as NotReady

Diagnosis:

# Check if kubelet port has firewall rules
sudo salt 'k8s-node-1' cmd.run "firewall-cmd --list-rich-rules | grep 10250"

# Should show rule allowing master IP
# If missing, kubelet can't communicate

# Test kubelet connectivity from master
nc -zv 203.0.113.21 10250
# Should succeed

Solution:

# Verify master IP in pillar is correct
sudo salt 'k8s-node-1' pillar.get kubernetes.master_ip

# Reapply firewall state
sudo salt 'k8s-node-*' state.apply firewall

# Restart kubelet if needed
sudo salt 'k8s-node-1' cmd.run 'systemctl restart k3s-agent'

Root Cause: Kubelet port (10250) blocked by firewall

Problem: Port Still Open to Internet

Symptom: Port 10250 or 8472 appears in direct ports list

Diagnosis:

# Check direct ports
sudo salt 'k8s-master-01' cmd.run 'firewall-cmd --list-ports'

# If 10250/tcp or 8472/udp appears here, that's insecure

Solution:

# Manually remove the direct port
sudo salt 'k8s-*' cmd.run 'firewall-cmd --permanent --remove-port=10250/tcp'
sudo salt 'k8s-*' cmd.run 'firewall-cmd --permanent --remove-port=8472/udp'
sudo salt 'k8s-*' cmd.run 'firewall-cmd --reload'

# Reapply firewall state to ensure rich rules exist
sudo salt 'k8s-*' state.apply firewall

Root Cause: k3s installer or manual commands added direct ports

Problem: Orphaned Firewall Rules

Symptom: Firewall rules for old/removed nodes still present

Diagnosis:

# List all rich rules for kubelet port
sudo salt 'k8s-master-01' cmd.run "firewall-cmd --list-rich-rules | grep 10250"

# Count rules
sudo salt 'k8s-master-01' cmd.run "firewall-cmd --list-rich-rules | grep 10250 | wc -l"

# Compare with number of worker nodes in pillar
sudo salt 'k8s-master-01' pillar.get kubernetes.node_ips

Solution:

# Cleanup happens automatically, just reapply state
sudo salt 'k8s-master-01' state.apply firewall

# Verify rule count matches pillar
sudo salt 'k8s-master-01' cmd.run "firewall-cmd --list-rich-rules | grep 10250 | wc -l"
# Should equal number of worker nodes

Root Cause: State not applied after removing nodes from pillar

Problem: CNI Interfaces Not in Trusted Zone

Symptom: Pods randomly can’t communicate

Diagnosis:

# Check trusted zone
sudo salt 'k8s-*' cmd.run 'firewall-cmd --zone=trusted --list-interfaces'

# Should show: cni0 flannel.1
# If empty or missing one, that's the issue

Solution:

# Reapply firewall state
sudo salt 'k8s-*' state.apply firewall

# Force add if still missing (temporary workaround)
sudo salt 'k8s-*' cmd.run 'firewall-cmd --permanent --zone=trusted --add-interface=cni0'
sudo salt 'k8s-*' cmd.run 'firewall-cmd --permanent --zone=trusted --add-interface=flannel.1'
sudo salt 'k8s-*' cmd.run 'firewall-cmd --reload'

Root Cause: CNI interfaces created after firewall state applied

Part 12: Production Best Practices

1. Version Control Everything

Every change should go through Git:

# Good commit message
git commit -m "Add k8s-node-4 to production cluster

- New worker node: k8s-node-4 (10.0.1.45)
- Added to kubernetes.node_ips in pillar
- Firewall rules will auto-update for master and all workers
- Node provisioned via Terraform
"

# Bad commit message
git commit -m "update config"

Why: Git history becomes your audit log and rollback mechanism.

2. Test in Staging First

Maintain separate pillar for staging:

# salt/pillar/staging/k8s.sls
kubernetes:
 master_ip: 10.10.10.10/32
 node_ips:
 - 10.10.10.20/32
 - 10.10.10.21/32

Process:

Apply changes to staging cluster
Verify functionality for 24 hours
Apply to production with confidence

3. Regular Security Audits

Monthly audit checklist:

# 1. Verify no direct port openings for cluster ports
sudo salt 'k8s-*' cmd.run 'firewall-cmd --list-ports'
# Should only see: SSH, API server (master), HTTP/HTTPS (master)

# 2. Count rich rules vs pillar IPs
sudo salt 'k8s-master-01' cmd.run "firewall-cmd --list-rich-rules | grep 10250 | wc -l"
# Should equal number of worker nodes

# 3. Verify CNI configuration
sudo salt 'k8s-*' cmd.run 'firewall-cmd --zone=trusted --list-interfaces'
# Should show: cni0 flannel.1

# 4. Check for orphaned rules
sudo salt 'k8s-*' cmd.run 'firewall-cmd --list-rich-rules' > /tmp/current-rules.txt
# Review for any IPs not in pillar

4. Monitor Firewall Changes

Set up alerting for unexpected firewall changes:

# Create audit script
cat > /usr/local/bin/firewall-audit.sh <<'EOF'
#!/bin/bash
set -uo pipefail

EXPECTED_RULES=$(salt-call --local pillar.get kubernetes.node_ips --out=json | jq -r '.local[]' | wc -l)
# Match the full port fragment, not the bare digits, or an unrelated rule that
# happens to contain "10250" will quietly inflate the count.
ACTUAL_RULES=$(firewall-cmd --list-rich-rules | grep -c 'port port="10250" protocol="tcp"')

if [ "$EXPECTED_RULES" != "$ACTUAL_RULES" ]; then
 echo "ALERT: Firewall rules mismatch! Expected: $EXPECTED_RULES, Actual: $ACTUAL_RULES"
 exit 1
fi
EOF

chmod +x /usr/local/bin/firewall-audit.sh

# Run via cron
echo "0 */4 * * * /usr/local/bin/firewall-audit.sh" | crontab -

5. Document Your Topology

Keep pillar/common.sls well-documented:

kubernetes:
 # Last Updated: 2026-07-28 by ops-team
 # Production Cluster: 3 nodes (1 master, 2 workers)
 # k3s channel: stable (record the exact version you are running)
 # CNI: Flannel VXLAN
 # Ingress: Traefik (deployed via Helm)

 master_ip: 203.0.113.10/32 # prod-k8s-master-01.example.com

 node_ips:
 - 203.0.113.21/32 # prod-k8s-worker-01.example.com (16 vCPU, 32GB RAM)
 - 203.0.113.22/32 # prod-k8s-worker-02.example.com (16 vCPU, 32GB RAM)

 # Default k3s CNI ranges - DO NOT change without rebuilding cluster
 pod_cidr: 10.42.0.0/16
 service_cidr: 10.43.0.0/16

6. Maintain Emergency Runbooks

Document recovery procedures:

# Emergency: Lost Firewall Access

## Symptoms

- Cannot SSH to nodes
- kubectl commands timeout

## Immediate Actions

1. Access master via console (provider dashboard)
2. Stop firewall: `systemctl stop firewalld`
3. SSH should now work
4. Review firewall rules: `firewall-cmd --list-all`
5. Reapply Salt state: `salt-call state.apply firewall`
6. If successful, start firewall: `systemctl start firewalld`

## Prevention

- Always keep console access credentials handy
- Test firewall changes on one node first
- Maintain staging environment for testing

7. Backup Current Configuration

Before major changes, backup current firewall state:

# Backup script
sudo salt 'k8s-*' cmd.run 'firewall-cmd --permanent --list-all > /root/firewall-backup-$(date +%Y%m%d-%H%M%S).txt'

# Verify backups created
sudo salt 'k8s-*' cmd.run 'ls -lht /root/firewall-backup-* | head -5'

Conclusion

You now have a production-grade, automated Kubernetes firewall configuration that:

✅ Eliminates security holes by restricting cluster ports to authorized IPs ✅ Requires zero manual maintenance with automatic orphan cleanup ✅ Prevents 504 Gateway Timeout with proper CNI networking ✅ Scales effortlessly to clusters of any size ✅ Provides full auditability through Git version control ✅ Maintains itself with infrastructure-as-code principles

This approach has been battle-tested in production environments and saves countless hours of manual firewall management while dramatically improving security posture.

Next Steps

Implement this in your cluster following the deployment steps above
Layer network policies on top — firewalld guards the node, NetworkPolicy guards pod-to-pod traffic inside the cluster. You want both
Set up monitoring for firewall rule changes and violations
Create runbooks for common operations (add/remove nodes)
Automate testing of firewall rules in CI/CD pipeline

Getting Started with SaltStack on AlmaLinux 10 — pillar, states and grains from scratch
Production-Grade SaltStack: Multi-Environment Setup with GitOps — the GitFS and multi-environment layout this guide assumes
How to Setup a Kubernetes Cluster with K3S in Rocky Linux 8/9 — build the cluster you are about to lock down
Setting Up Salt Master on Kubernetes — running the Salt master itself in-cluster
Automate WireGuard VPN on AlmaLinux 10 With SaltStack — the same pillar-driven pattern applied to a VPN mesh
How to Setup Prometheus Monitoring on a Kubernetes Cluster — what is listening on the 9100 and 9104 rules above

Further Resources

Kubernetes Network Policies — in-cluster segmentation that complements node firewalling
SaltStack Best Practices — official Salt documentation
Firewalld Rich Language — rich rule syntax reference
k3s Networking — the ports k3s expects and why
CNI Specification — how CNI plugins work

Questions, corrections, or a failure mode I missed? Leave a comment below.

Written for AlmaLinux 10 (firewalld 2.x on the nftables backend), k3s with the default Flannel VXLAN CNI, and Salt 3007 LTS. Record the exact versions you run in pillar/common.sls as shown above.

Nftables on Citizix, The Best blog for Linux, Opensource, Devops, Cloud and all things awesome tech.

Secure Kubernetes With Firewalld and SaltStack on AlmaLinux 10

The Security Problem Nobody Talks About

The Solution: Automated, Auditable Firewall Management

Real-World Cluster Architecture

Prerequisites

Part 1: Understanding the Salt Architecture

Part 2: Automatic Role Assignment with Grains

Part 3: Cluster Configuration in Pillar

Part 4: Master Node Firewall Configuration

Part 5: Worker Node Firewall Configuration

Part 6: State Assignment

Part 7: Deployment Process

Step 1: Commit Configuration to Git

Step 2: Update Salt Master from Git

Step 3: Apply Grains State (Role Assignment)

Step 4: Apply Firewall State (Test Mode First)

Part 8: Comprehensive Verification

Verify Firewall Rules on Master

Verify Firewall Rules on Worker Nodes

Verify CNI Networking Configuration

Verify FORWARD Chain Rules

Test Cluster Functionality

Test External Access

Part 9: Understanding the Magic

Automatic Orphan Cleanup Explained

CNI Networking Deep Dive

Rich Rules vs Direct Port Openings

Part 10: Day-2 Operations

Adding a New Worker Node

Removing a Worker Node

Updating Port Numbers

Part 11: Troubleshooting Guide

Problem: 504 Gateway Timeout for Services

Problem: Nodes Show NotReady

Problem: Port Still Open to Internet

Problem: Orphaned Firewall Rules

Problem: CNI Interfaces Not in Trusted Zone

Part 12: Production Best Practices

1. Version Control Everything

2. Test in Staging First

3. Regular Security Audits

4. Monitor Firewall Changes

5. Document Your Topology

6. Maintain Emergency Runbooks

7. Backup Current Configuration

Conclusion

Next Steps

Related Reading on Citizix

Further Resources