Kubernetes Jobs and CronJobs: A Practical kubectl Guide

Sat, 28 Aug 2021 04:52:47 +0000

Kubernetes Jobs execute one-off tasks. A Job creates one or more Pods and keeps retrying them until a specified number of them terminate successfully. A CronJob creates Jobs on a repeating schedule, the same way one line of a crontab does.

This guide covers the fields you actually need in production: retries (backoffLimit), parallel work (completions, parallelism), timeouts (activeDeadlineSeconds), automatic cleanup (ttlSecondsAfterFinished), and for CronJobs concurrencyPolicy, timeZone, suspend and history limits. Every manifest and output below was run on a Kubernetes 1.37 cluster created with kind; the batch/v1 API used here has been stable since Kubernetes 1.21, and timeZone since 1.27.

A minimal Job

Save this as job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
 name: hello-job
spec:
 backoffLimit: 4
 activeDeadlineSeconds: 120
 ttlSecondsAfterFinished: 600
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: hello
 image: busybox:1.37
 command: ["sh", "-c", "echo Hello from $(hostname); date -u"]

What the spec fields do:

backoffLimit: how many times a failed Pod is retried before the Job is marked Failed (default 6). Retries back off exponentially: 10s, 20s, 40s, capped at six minutes.
activeDeadlineSeconds: a hard timeout for the whole Job, retries included. When it expires, running Pods are killed and the Job fails with reason DeadlineExceeded. It takes precedence over backoffLimit.
ttlSecondsAfterFinished: delete the Job, and its Pods, this many seconds after it completes or fails. Without it, finished Jobs pile up until you delete them.
restartPolicy: must be Never or OnFailure in a Job. Never creates a new Pod for each retry, which keeps the failed Pods (and their logs) around. OnFailure restarts the container in the same Pod.

Apply it, wait for it, and read the output:

kubectl apply -f job.yaml
kubectl wait --for=condition=complete job/hello-job --timeout=120s
kubectl get jobs
kubectl logs job/hello-job

job.batch/hello-job created
job.batch/hello-job condition met
NAME STATUS COMPLETIONS DURATION AGE
hello-job Complete 1/1 11s 11s
Hello from hello-job-28qnh
Tue Nov 10 08:19:49 UTC 2026

For a quick one-off you don’t need a file. kubectl create job builds the same object; add --dry-run=client -o yaml to get a manifest to start from:

kubectl create job oneoff --image=busybox:1.37 -- echo hi
kubectl create job slow-job --image=busybox:1.37 --dry-run=client -o yaml -- sleep 60 > slow.yaml

Parallel Jobs: completions and parallelism

completions is how many Pods must succeed; parallelism is how many run at once. With completionMode: Indexed, each Pod gets a unique index in the JOB_COMPLETION_INDEX environment variable, which is the easiest way to split work into shards.

apiVersion: batch/v1
kind: Job
metadata:
 name: batch-worker
spec:
 completions: 6
 parallelism: 2
 completionMode: Indexed
 backoffLimit: 3
 ttlSecondsAfterFinished: 600
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: worker
 image: busybox:1.37
 command: ["sh", "-c", "echo processing shard $JOB_COMPLETION_INDEX; sleep 5"]

kubectl apply -f parallel-job.yaml
kubectl wait --for=condition=complete job/batch-worker --timeout=120s
kubectl get pods -l batch.kubernetes.io/job-name=batch-worker
kubectl logs -l batch.kubernetes.io/job-name=batch-worker --prefix

The Pods run two at a time, and the Pod name carries the index:

NAME READY STATUS RESTARTS AGE
batch-worker-0-rm58n 0/1 Completed 0 43s
batch-worker-1-68r9z 0/1 Completed 0 43s
batch-worker-2-wcwpk 0/1 Completed 0 30s
batch-worker-3-9xwr6 0/1 Completed 0 29s
batch-worker-4-gfd2q 0/1 Completed 0 22s
batch-worker-5-fvk46 0/1 Completed 0 21s
[pod/batch-worker-0-rm58n/worker] processing shard 0
[pod/batch-worker-1-68r9z/worker] processing shard 1
[pod/batch-worker-2-wcwpk/worker] processing shard 2
[pod/batch-worker-3-9xwr6/worker] processing shard 3
[pod/batch-worker-4-gfd2q/worker] processing shard 4
[pod/batch-worker-5-fvk46/worker] processing shard 5

Leave completions unset and set only parallelism for a work-queue pattern, where the Pods pull items from a queue themselves and the Job finishes once at least one Pod has succeeded and all of them have terminated.

Timeouts with activeDeadlineSeconds

A Job whose Pod hangs will otherwise run forever. This one sleeps for five minutes but is only allowed 15 seconds:

apiVersion: batch/v1
kind: Job
metadata:
 name: deadline-job
spec:
 activeDeadlineSeconds: 15
 ttlSecondsAfterFinished: 30
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: slow
 image: busybox:1.37
 command: ["sleep", "300"]

kubectl apply -f deadline.yaml
kubectl wait --for=condition=failed job/deadline-job --timeout=60s
kubectl get job deadline-job -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

job.batch/deadline-job condition met
FailureTarget=True DeadlineExceeded: Job was active longer than specified deadline
Failed=True DeadlineExceeded: Job was active longer than specified deadline

Thirty seconds later, ttlSecondsAfterFinished removes it:

kubectl get job deadline-job

Error from server (NotFound): jobs.batch "deadline-job" not found

Debugging a failed Job

This Job always exits non-zero, so it uses up its backoffLimit of 2 (one run plus two retries):

apiVersion: batch/v1
kind: Job
metadata:
 name: failing-job
spec:
 backoffLimit: 2
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: fail
 image: busybox:1.37
 command: ["sh", "-c", "echo connecting to db...; echo 'error: connection refused' >&2; exit 1"]

Start with the Job’s status and events. kubectl describe job shows the Pod counts and why the Job stopped:

kubectl describe job failing-job

Pods Statuses: 0 Active (0 Ready) / 0 Succeeded / 3 Failed
...
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal SuccessfulCreate 47s job-controller Created pod: failing-job-295fl
 Normal SuccessfulCreate 27s job-controller Created pod: failing-job-5wtp6
 Normal SuccessfulCreate 7s job-controller Created pod: failing-job-49zcw
 Warning BackoffLimitExceeded 4s job-controller Job has reached the specified backoff limit

Then list the Pods the Job created (every Job Pod carries the batch.kubernetes.io/job-name label) and read their logs:

kubectl get pods -l batch.kubernetes.io/job-name=failing-job
kubectl logs job/failing-job
kubectl logs failing-job-49zcw

NAME READY STATUS RESTARTS AGE
failing-job-295fl 0/1 Error 0 47s
failing-job-49zcw 0/1 Error 0 7s
failing-job-5wtp6 0/1 Error 0 27s
Found 3 pods, using pod/failing-job-295fl
error: connection refused
connecting to db...

kubectl logs job/<name> picks one Pod; name a specific Pod to see a particular attempt. This is why restartPolicy: Never is easier to debug: each attempt leaves its own Pod behind. Keep in mind that ttlSecondsAfterFinished deletes those Pods too, so set it long enough to look at failures.

What the common statuses mean:

Symptom	Likely cause	Look at
`BackoffLimitExceeded`	The container exits non-zero every time	`kubectl logs` on the failed Pods
`DeadlineExceeded`	The Job ran longer than `activeDeadlineSeconds`	Raise the deadline or find what hangs
Pods stuck `Pending`	No node can fit the requests, or a PVC is unbound	`kubectl describe pod <pod>` events
`ImagePullBackOff` / `ErrImagePull`	Wrong image name or tag, or a missing `imagePullSecrets`	`kubectl describe pod <pod>`
`CreateContainerConfigError`	A referenced Secret or ConfigMap does not exist	`kubectl describe pod <pod>`

A Job’s Pod template is immutable once created. To fix one, delete the Job and apply it again:

kubectl delete job failing-job
kubectl apply -f failing-job.yaml

Deleting a Job deletes its Pods as well.

CronJobs

A CronJob creates a Job from its jobTemplate each time the schedule fires. The schedule uses the standard five-field cron format (minute, hour, day of month, month, day of week).

apiVersion: batch/v1
kind: CronJob
metadata:
 name: nightly-report
spec:
 schedule: "*/1 * * * *"
 timeZone: "Africa/Nairobi"
 concurrencyPolicy: Forbid
 startingDeadlineSeconds: 300
 successfulJobsHistoryLimit: 3
 failedJobsHistoryLimit: 1
 suspend: false
 jobTemplate:
 spec:
 backoffLimit: 2
 activeDeadlineSeconds: 600
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: report
 image: busybox:1.37
 command: ["sh", "-c", "echo generating report at $(date -u); sleep 10"]

The schedule here runs every minute so you can watch it work; for a real nightly job use something like "30 2 * * *". The CronJob-level fields:

timeZone: an IANA time zone name. Without it, the schedule is interpreted in the time zone of the kube-controller-manager, which is usually UTC. Do not put CRON_TZ= or TZ= in schedule; the API rejects it.
concurrencyPolicy: what happens when the next run is due and the last one is still going. Allow (the default) runs them side by side, Forbid skips the new run, Replace kills the running Job and starts the new one. For anything that writes to a database or a file, Forbid is usually right.
startingDeadlineSeconds: if a run could not start on time (controller down, or skipped because of Forbid), it may still start up to this many seconds late. After that the run is counted as missed.
successfulJobsHistoryLimit / failedJobsHistoryLimit: how many finished Jobs to keep (defaults 3 and 1). The CronJob controller deletes older ones, so you do not need ttlSecondsAfterFinished in the template.
suspend: true stops new runs without deleting the CronJob. Jobs already running are not affected.

Apply it and check it:

kubectl apply -f cronjob.yaml
kubectl get cronjobs

NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
nightly-report */1 * * * * Africa/Nairobi False 0 38s 57s

Each run creates a Job named after the CronJob and the scheduled time:

kubectl get jobs

NAME STATUS COMPLETIONS DURATION AGE
nightly-report-29904980 Complete 1/1 13s 58s

Run a CronJob now

To test a CronJob without waiting for the schedule, create a Job from its template:

kubectl create job --from=cronjob/nightly-report nightly-report-manual
kubectl wait --for=condition=complete job/nightly-report-manual --timeout=60s
kubectl logs job/nightly-report-manual

job.batch/nightly-report-manual created
job.batch/nightly-report-manual condition met
generating report at Tue Nov 10 08:20:38 UTC 2026

kubectl sets the CronJob as the manual Job’s owner, so it is deleted along with the CronJob.

Suspend and resume a CronJob

kubectl patch cronjob nightly-report -p '{"spec":{"suspend":true}}'
kubectl get cronjob nightly-report

cronjob.batch/nightly-report patched
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
nightly-report */1 * * * * Africa/Nairobi True 0 38s 57s

Set it back to false to resume. Runs that fall due while a CronJob is suspended count as missed. Without startingDeadlineSeconds, the missed run starts immediately when you resume; with a short deadline, it is dropped instead.

Debugging a CronJob that does not run

kubectl get cronjob shows SUSPEND True: it is suspended.
LAST SCHEDULE is empty or old: check kubectl describe cronjob <name> for events. If more than 100 runs were missed (for example after a long controller outage), the controller logs an error and does not start the Job; setting startingDeadlineSeconds limits how far back it counts.
Jobs are created but fail: debug the Job with the steps in the section above.
Runs are skipped: with concurrencyPolicy: Forbid, a run is skipped while the previous one is still active. Check the Job durations against the schedule interval.

To find out about failures without checking by hand, alert on them. With kube-state-metrics installed, kube_job_status_failed > 0 catches failed Jobs and time() - kube_cronjob_status_last_successful_time catches a CronJob that has stopped succeeding.

Cleanup

kubectl delete cronjob nightly-report
kubectl delete job hello-job batch-worker failing-job oneoff

Deleting a CronJob deletes the Jobs and Pods it created, including Jobs made with --from=cronjob/....

Conclusion

For most batch work, a Job needs four fields beyond the Pod template: backoffLimit for retries, activeDeadlineSeconds so it cannot hang forever, ttlSecondsAfterFinished so finished Jobs get cleaned up, and restartPolicy: Never so failed attempts keep their logs. For a CronJob, set timeZone explicitly and choose concurrencyPolicy on purpose, and use kubectl create job --from=cronjob/... to test it before you trust the schedule.

Kubectl on Citizix, The Best blog for Linux, Opensource, Devops, Cloud and all things awesome tech.