Job Parallelism and Completions

A single-completion Job processes one item. But what if you need to process 10 files, send 50 messages, or run a suite of integration tests? You could create 10 separate Jobs, but that is repetitive and hard to track. A single Job with completions: 10 and parallelism: 3 runs three Pods simultaneously, tracks how many have succeeded, and stops when 10 completions are recorded.

nano batch-job.yaml

Start with the Job that requires multiple completions:

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-job
spec:
  completions: 6
  parallelism: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: busybox:1.36
          command: ['sh', '-c', 'echo Processing item && sleep 3']

kubectl apply -f batch-job.yaml

What completions and parallelism control

completions is the total number of successful Pod exits required before the Job is done. parallelism is the maximum number of Pods the Job controller will run simultaneously. The controller creates new Pods as previous ones complete, never exceeding parallelism, until completions is reached.

Watch the Pods as they run:

kubectl get pods -l job-name=batch-job --watch

You will see two Pods start, reach Completed, then two more start, until six total completions are recorded. Press Ctrl+C to exit the watch. Then check the Job:

kubectl get jobs

COMPLETIONS shows how many have succeeded out of the total required. DURATION shows the elapsed time since the Job started.

Non-indexed vs indexed completion modes

By default, Jobs use non-indexed mode: each Pod is interchangeable and the Job simply counts successful exits. This works for processing items from a shared queue where each worker picks the next available item.

Kubernetes also supports completionMode: Indexed. Each Pod receives a unique index via the JOB_COMPLETION_INDEX environment variable (0 through completions-1). This lets each Pod know which specific item it is responsible for without an external queue.

nano indexed-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: indexed-job
spec:
  completions: 4
  parallelism: 2
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: busybox:1.36
          command: ['sh', '-c', 'echo "Processing index $JOB_COMPLETION_INDEX"']

kubectl apply -f indexed-job.yaml
kubectl get pods -l job-name=indexed-job

Pod names for indexed Jobs include the index: indexed-job-0-<hash>, indexed-job-1-<hash>. Each Pod processes exactly one index.

kubectl logs -l job-name=indexed-job

Each Pod prints its own index. The four outputs together cover indices 0 through 3 with no overlap and no gap.

kubectl delete job batch-job indexed-job

Parallelism and completions transform a Job from a one-shot task into a batch processing primitive. The next lesson covers what happens when Pods in a Job fail and how to configure retry limits and backoff behavior.