How to Fix a Stuck Kubernetes Job

Why the Job is stuck

kubectl get jobs shows 0/1 completions. The Job controller keeps spawning pods, each one fails, and after backoffLimit retries the Job stops trying. The Job itself stays in the namespace showing the failure count - it does not clean itself up.

The cause is always in the pod logs. Read them first.

Step 1: find the failing pod and read its logs

# See completion count and failure count
kubectl get jobs

# List the pods the Job created (they may be in Error/Completed state)
kubectl get pods -l job-name=migrate

# Read what the container actually printed
kubectl logs -l job-name=migrate

The output points directly at the problem. A typo in command: produces a classic:

sh: ehco: not found

That single line tells you ehco is not a real command - it should be echo.

Step 2: fix the manifest

Open job.yaml and correct the command in the container spec:

spec:
  template:
    spec:
      containers:
      - name: migrate
        image: busybox:1.36
        command: ["sh", "-c", "echo 'migration complete'"]

Save the file. Do not kubectl apply it yet.

Step 3: delete the old Job, then recreate it

Job pod templates are immutable - Kubernetes rejects any attempt to change spec.template on an existing Job. You must delete the old Job first:

# Delete the stuck Job (waits for full removal)
kubectl delete job migrate

# Apply the fixed manifest
kubectl apply -f job.yaml

# Wait for the Job to complete
kubectl wait --for=condition=complete job/migrate --timeout=90s

After a successful run, kubectl get jobs shows 1/1 completions.

Why immutable templates matter

Kubernetes freezes a Job's pod template at creation time by design - it prevents the spec from changing under in-flight pods. The rejection message from a naive re-apply is:

The Job "migrate" is invalid: spec.template: Invalid value:
... field is immutable

This trips up operators who are used to kubectl apply updating Deployments on the fly. For Jobs (and CronJobs), the pattern is always: delete, then recreate. For CronJobs the controller handles this automatically on the next scheduled run - but a one-off Job needs the manual delete cycle.

The investigation checklist

• kubectl get jobs - check completion count and failure count
• kubectl logs -l job-name=<name> - read the actual error output
• kubectl describe job <name> - see events and the last failure reason
• Fix the manifest, delete the Job, re-apply, then confirm completions