This guide shows how to backup and restore k3s step by step. It covers which files a restore needs and why, a script that takes a consistent snapshot of the datastore, encrypts it and ships it off-site, a cron schedule to run it nightly, an alert for when it stops working, and a restore you have practised before you need it.

It works for both datastores k3s ships with:

• SQLite: the default for a single-server cluster. The datastore is one file, /var/lib/rancher/k3s/server/db/state.db.
• Embedded etcd: what you get when the first server starts with --cluster-init, and what every HA k3s cluster with embedded servers runs.

If you run k3s against an external datastore (PostgreSQL or MySQL), back that database up with its own tools, for example pgBackRest. The token, TLS and config sections below still apply.

What a k3s Backup Has to Contain

The datastore on its own is not enough. A restore is only usable if you bring back these files with it, and all of them live on the server nodes:

What this does not back up

• Application data in PersistentVolumes. With the default local-path provisioner it sits under /var/lib/rancher/k3s/storage/ on whichever node ran the pod. The datastore backup restores the PVC objects, not the bytes inside the volumes. Use Velero or each application’s own backup tool (pg_dump, pgBackRest, mysqldump) for those.
• Agent nodes. An agent holds no cluster state. Rebuild it and rejoin it with the token.
• Container images. They are pulled again from the registry.

Why encrypt the backup

The archive holds the CA private keys, the server token and every Kubernetes Secret in the cluster. Anyone who reads it has full control of the cluster. Encrypt it before it leaves the server, to a public key whose private half is kept off the server. Then neither the bucket nor a compromised server can decrypt old backups.

Prerequisites

• Root on a k3s server node
• An S3-compatible bucket (AWS S3, Hetzner Object Storage, MinIO, Backblaze B2, and so on) and an access key that can write to it
• A workstation where you will keep the decryption key

Step 1: Install the Tools

You need rclone to talk to S3, age to encrypt and sqlite3 to take the SQLite snapshot.

On Ubuntu or Debian:

sudo apt update
sudo apt install -y age sqlite3 rclone

On Rocky Linux, AlmaLinux or RHEL, age comes from EPEL:

sudo dnf install -y epel-release
sudo dnf install -y age sqlite rclone

If your distribution’s rclone is old, install the current release instead:

curl -fsSL https://rclone.org/install.sh | sudo bash

Step 2: Create the Encryption Key

Do this on your workstation, not on the server:

age-keygen -o ~/.config/k3s-backup.key
chmod 600 ~/.config/k3s-backup.key
grep 'public key' ~/.config/k3s-backup.key

The last command prints a line like # public key: age1.... Copy only the age1... value to the server:

sudo mkdir -p /etc/k3s-backup
echo 'age1yourpublickeyhere' | sudo tee /etc/k3s-backup/age-recipient.txt

The public key can only encrypt, so it is safe on the server. Store a second copy of the private key somewhere other than your laptop, such as a password manager. Lose it and every backup is lost with it.

Step 3: Configure rclone for Your Bucket

Create /root/.config/rclone/rclone.conf. The example uses Hetzner Object Storage; change provider, endpoint and region for your provider:

[s3backup]
type = s3
provider = Other
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = https://nbg1.your-objectstorage.com
region = nbg1
acl = private

Lock it down and check that it can see the bucket:

sudo chmod 600 /root/.config/rclone/rclone.conf
sudo rclone lsd s3backup:

Step 4: Write the Backup Script

Save this as /usr/local/bin/k3s-backup.sh. It detects which datastore the node runs and does the following:

1. Collects the token, TLS, credentials, manifests, config, unit and version.
2. Takes a consistent copy of the datastore. For SQLite it uses SQLite’s online backup, so k3s keeps running. A plain cp of a live SQLite database in WAL mode can give you a torn file. For etcd it takes the newest snapshot k3s made itself.
3. Tars, compresses and encrypts the bundle with age. Plaintext never leaves the host.
4. Uploads it, downloads it again and compares SHA-256 checksums.
5. Writes a “last success” timestamp for monitoring, then deletes objects older than the retention period.

#!/usr/bin/env bash
# k3s server backup: datastore + token + TLS + config, age-encrypted, to S3.
set -euo pipefail
umask 077

DEST="s3backup:my-k3s-backups/$(hostname -s)"   # rclone remote:bucket/prefix
RECIPIENT=/etc/k3s-backup/age-recipient.txt
RETENTION_DAYS=30
SERVER=/var/lib/rancher/k3s/server
METRIC_DIR=/var/lib/node_exporter/textfile_collector

log() { echo "$(date -Iseconds) [k3s-backup] $*"; }

# One run at a time.
exec 9>/run/k3s-backup.lock
flock -n 9 || { log "another run holds the lock; exiting"; exit 0; }

T=$(mktemp -d /var/tmp/k3s-backup.XXXXXX)
trap 'rm -rf "$T"' EXIT
B="$T/data"
mkdir -p "$B/db"

# Everything a restore needs besides the datastore.
k3s --version > "$B/version.txt"
V=$(awk '{print $3; exit}' "$B/version.txt")   # e.g. v1.33.5+k3s1
cp -a /etc/rancher/k3s "$B/config"
cp -a /etc/systemd/system/k3s.service "$B/"
if [ -f /etc/systemd/system/k3s.service.env ]; then
  cp -a /etc/systemd/system/k3s.service.env "$B/"
fi
cp -a "$SERVER/token" "$SERVER/tls" "$B/"
for d in cred manifests; do
  if [ -d "$SERVER/$d" ]; then cp -a "$SERVER/$d" "$B/"; fi
done

# The datastore.
if [ -f "$SERVER/db/state.db" ]; then
  # SQLite: online, consistent snapshot; k3s keeps running.
  sqlite3 "$SERVER/db/state.db" ".backup '$B/db/state.db'"
  OK=$(sqlite3 "$B/db/state.db" "PRAGMA integrity_check;")
  [ "$OK" = "ok" ] || { log "FAIL: SQLite integrity check: $OK"; exit 1; }
  log "SQLite snapshot ok"
elif [ -d "$SERVER/db/etcd" ]; then
  # Embedded etcd: take the newest scheduled snapshot, refuse one older than 13h.
  SNAP=$(find "$SERVER/db/snapshots" -maxdepth 1 -name 'etcd-snapshot-*' -mmin -780 \
           -printf '%T@ %p\n' | sort -n | tail -1 | cut -d' ' -f2-)
  [ -n "$SNAP" ] || { log "FAIL: no etcd snapshot newer than 13h in $SERVER/db/snapshots"; exit 1; }
  cp -a "$SNAP" "$B/db/"
  log "etcd snapshot $(basename "$SNAP")"
else
  log "FAIL: no SQLite or etcd datastore under $SERVER/db (external datastore?)"
  exit 1
fi

# Encrypt, upload, verify.
OBJ="$DEST/k3s-backup-$(date -u +%Y%m%dT%H%M%SZ)-$V.tar.gz.age"
tar -C "$B" -czf - . | age -R "$RECIPIENT" > "$T/backup.age"
LOCAL_SHA=$(sha256sum "$T/backup.age" | cut -d' ' -f1)
rclone copyto "$T/backup.age" "$OBJ"
REMOTE_SHA=$(rclone cat "$OBJ" | sha256sum | cut -d' ' -f1)
if [ "$LOCAL_SHA" != "$REMOTE_SHA" ]; then
  log "FAIL: checksum mismatch for $OBJ"
  exit 1
fi
log "OK: $OBJ $(stat -c %s "$T/backup.age") bytes"

# Success timestamp for node_exporter's textfile collector.
if [ -d "$METRIC_DIR" ]; then
  echo "k3s_backup_last_success_timestamp_seconds $(date +%s)" > "$METRIC_DIR/k3s_backup.prom.tmp"
  chmod 0644 "$METRIC_DIR/k3s_backup.prom.tmp"
  mv "$METRIC_DIR/k3s_backup.prom.tmp" "$METRIC_DIR/k3s_backup.prom"
fi

# Retention.
rclone delete "$DEST/" --min-age "${RETENTION_DAYS}d" || log "WARNING: retention prune failed"

Make it executable, then run it once by hand before you schedule it:

sudo chmod 755 /usr/local/bin/k3s-backup.sh
sudo /usr/local/bin/k3s-backup.sh
sudo rclone lsl s3backup:my-k3s-backups/$(hostname -s)/

The run should end with an OK: line, and the object should show in the listing.

A few choices in the script are deliberate:

• umask 077. The temporary copy holds the CA keys and token, so only root can read it.
• flock. If a slow upload is still going when the next cron run starts, the second run exits instead of competing with the first.
• The checksum round trip. A zero exit from the upload is not proof that the object in the bucket is intact. Reading it back is.
• The 13-hour check for etcd. k3s takes etcd snapshots at 00:00 and 12:00 by default and keeps 5 on each server. If scheduled snapshots break, the script fails loudly instead of uploading the same stale file every night.

Embedded etcd: k3s can ship snapshots to S3 itself

On etcd clusters, k3s can upload its own scheduled snapshots with the --etcd-s3 flags. Add them to /etc/rancher/k3s/config.yaml on the servers:

etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 10
etcd-s3: true
etcd-s3-endpoint: nbg1.your-objectstorage.com
etcd-s3-bucket: my-k3s-backups
etcd-s3-folder: etcd
etcd-s3-access-key: YOUR_ACCESS_KEY
etcd-s3-secret-key: YOUR_SECRET_KEY

This is simpler, but it only ships the snapshot. You still need the token, TLS, credentials and config backed up somewhere, and the snapshot is not encrypted on your side. The script above covers both gaps, so it is worth running even with --etcd-s3 on.

Step 5: Schedule the Backup with Cron

Create /etc/cron.d/k3s-backup:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# m  h  dom mon dow  user  command
30   2  *   *   *    root  /usr/local/bin/k3s-backup.sh >> /var/log/k3s-backup.log 2>&1

What each part does:

• PATH: cron’s default PATH is /usr/bin:/bin. k3s and often rclone live in /usr/local/bin, so without this line the script fails under cron even though it worked in your shell. This is the most common reason a cron backup “never ran”.
• 30 2 * * *: minute 30, hour 2, every day. For etcd, pick a time just after one of k3s’s snapshot times so the newest snapshot is fresh. With the default 0 */12 * * * schedule, 02:30 picks up the 00:00 snapshot. Times are in the server’s timezone (timedatectl shows it).
• root: files in /etc/cron.d/ take a user column, unlike crontab -e. The script needs root to read the token and TLS keys.
• >> /var/log/k3s-backup.log 2>&1: keeps the output. Otherwise cron mails it or throws it away.

Files in /etc/cron.d/ must be owned by root and not group- or world-writable, or cron ignores them:

sudo chown root:root /etc/cron.d/k3s-backup
sudo chmod 644 /etc/cron.d/k3s-backup

On an HA etcd cluster, schedule it on one server. Every server’s snapshot holds the whole cluster.

Rotate the log so it doesn’t grow forever. Create /etc/logrotate.d/k3s-backup:

/var/log/k3s-backup.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root root
}

To test the cron entry without waiting until 02:30, temporarily set the time to two minutes from now, then watch the log:

sudo tail -f /var/log/k3s-backup.log

Step 6: Alert When Backups Stop

A backup that silently stopped three weeks ago is the usual way people find out they have no backup. The script writes k3s_backup_last_success_timestamp_seconds only after a verified upload. If you run Prometheus node exporter on the server with the textfile collector enabled:

node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

then two alert rules cover both failure modes, a stale backup and a metric that was never written:

groups:
  - name: k3s-backup
    rules:
      - alert: K3sBackupStale
        expr: time() - k3s_backup_last_success_timestamp_seconds > 30 * 3600
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "k3s backup on {{ $labels.instance }} is over 30h old"
      - alert: K3sBackupMissing
        expr: absent(k3s_backup_last_success_timestamp_seconds)
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "k3s backup metric is missing, so K3sBackupStale cannot fire"

30 hours gives one missed night plus a margin before it pages.

If you don’t run Prometheus, a dead-man’s-switch service such as Healthchecks.io works the same way. Add a curl to its ping URL as the script’s last line and it alerts when the ping stops arriving.

Step 7: Restore k3s From Backup

Read this whole section before you start. A restore is an outage. Always restore onto the same k3s version the backup was taken on (it’s in the object name and in version.txt), then upgrade afterwards.

7.1 Fetch and decrypt the backup

From your workstation, pick an object and stream it, decrypted, straight to the server. The plaintext is never written to your workstation’s disk:

rclone lsl s3backup:my-k3s-backups/k3s-server-01/

OBJ=k3s-backup-20261101T023000Z-v1.33.5+k3s1.tar.gz.age
rclone cat "s3backup:my-k3s-backups/k3s-server-01/$OBJ" \
  | age -d -i ~/.config/k3s-backup.key \
  | ssh root@k3s-server-01 'umask 077; mkdir -p /root/k3s-restore && tar -C /root/k3s-restore -xzf -'

Your workstation needs the same rclone remote configured.

On the server, check what you got:

R=/root/k3s-restore
ls -l "$R" "$R/db"
cat "$R/version.txt"

7.2 Restoring onto a new host

If the original server is gone, install the same version without starting it, then continue with 7.3 or 7.4:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.33.5+k3s1" INSTALL_K3S_SKIP_START=true sh -s - server

Then put back the config and unit, so the server starts with the same flags it had:

sudo mkdir -p /etc/rancher/k3s
sudo cp -a "$R/config/." /etc/rancher/k3s/
sudo cp -a "$R/k3s.service" /etc/systemd/system/k3s.service
[ -f "$R/k3s.service.env" ] && sudo cp -a "$R/k3s.service.env" /etc/systemd/system/
sudo systemctl daemon-reload

If the new host has a different IP or hostname, update node-ip, tls-san or advertise-address in config.yaml before you start it.

7.3 Restore a SQLite server

S=/var/lib/rancher/k3s/server

sudo systemctl stop k3s

# Keep the current state, in case you need to go back.
[ -d "$S/db" ] && sudo mv "$S/db" "$S/db.before-restore-$(date +%s)"

# Datastore
sudo mkdir -p "$S/db"
sudo cp -a "$R/db/state.db" "$S/db/state.db"

# Token, CAs, credentials and manifests: these must match the datastore
sudo cp -a "$R/token" "$S/token"
sudo rm -rf "$S/tls" && sudo cp -a "$R/tls" "$S/tls"
[ -d "$R/cred" ] && sudo rm -rf "$S/cred" && sudo cp -a "$R/cred" "$S/cred"
[ -d "$R/manifests" ] && sudo cp -a "$R/manifests/." "$S/manifests/"

sudo systemctl start k3s

Moving the whole db/ directory aside also removes any stale state.db-wal and state.db-shm files. Those must not be paired with a different state.db.

7.4 Restore an embedded etcd cluster

Stop k3s on every server first:

sudo systemctl stop k3s

On the first server, put back the token, CAs and credentials as in 7.3, then reset the cluster from the snapshot:

S=/var/lib/rancher/k3s/server
sudo cp -a "$R/token" "$S/token"
sudo rm -rf "$S/tls" && sudo cp -a "$R/tls" "$S/tls"
[ -d "$R/cred" ] && sudo rm -rf "$S/cred" && sudo cp -a "$R/cred" "$S/cred"

SNAP=$(ls "$R"/db/etcd-snapshot-*)
sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path="$SNAP" \
  --token="$(sudo cat "$R/token")"

It ends with Managed etcd cluster membership has been reset, restart without --cluster-reset flag now. If config.yaml also sets token:, it must match the backed-up token, or k3s will not start.

Start the first server normally:

sudo systemctl start k3s

On each of the other servers, move their old datastore aside and start them so they rejoin the restored cluster:

sudo mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.before-restore-$(date +%s)
sudo systemctl start k3s

7.5 Verify the restored cluster

sudo k3s kubectl get nodes -o wide
sudo k3s kubectl get pods -A | grep -v -E 'Running|Completed'
sudo k3s kubectl get --raw='/readyz?verbose' | tail -3

• Agents reconnect on their own because the token and CA are unchanged. If one does not, run sudo systemctl restart k3s-agent on it.
• Node objects come back from the backup. If you restored onto new hosts, delete the old nodes that no longer exist with kubectl delete node <name>.
• Your existing kubeconfigs keep working because the CA is the same one.

When you are happy with the restore, delete the plaintext copy. It holds the token and CA keys:

sudo rm -rf /root/k3s-restore

Step 8: Test the Restore on a Schedule

A backup you have never restored is a hope, not a backup. At least once a month:

1. Quick check: decrypt the newest backup into a temporary directory on your workstation, confirm token, tls/, config/, version.txt and the datastore are present, and for SQLite run sqlite3 state.db "PRAGMA integrity_check;". Then delete the directory.
2. Real check: once a quarter, do Step 7 on a throwaway VM with the same k3s version, and confirm kubectl get pods -A shows your workloads. Time it. That number is your actual recovery time.

T=$(mktemp -d)
rclone cat "s3backup:my-k3s-backups/k3s-server-01/$OBJ" | age -d -i ~/.config/k3s-backup.key | tar -C "$T" -xzf -
ls "$T" "$T/db"
[ -f "$T/db/state.db" ] && sqlite3 "$T/db/state.db" "PRAGMA integrity_check;"
rm -rf "$T"

Related Content

• How to Setup a Kubernetes Cluster with K3S in Rocky Linux 8/9
• How to Use Ansible to Setup and Upgrade K3s the Lightweight Kubernetes
• Velero Lifecycle Commands – Backup and Restore Scenarios
• pgBackRest Guide: PostgreSQL Backup, Restore and PITR
• How to Monitor Linux Servers Using Prometheus Node Exporter

Conclusion

A usable k3s backup is more than a copy of the datastore. It is the datastore taken consistently, plus the token that decrypts it, the CAs every client trusts and the config the server ran with, encrypted before it leaves the host and checked after upload. Run it from cron with an explicit PATH, alert on the last success timestamp rather than on errors, and restore it onto a throwaway VM regularly, so the first restore you do is not during an outage. Back up application data in PersistentVolumes separately, with Velero or the application’s own tooling.