Python Graceful Shutdown: Handling SIGTERM Correctly

What SIGTERM means in Kubernetes

When you roll out a new version or scale down a deployment, Kubernetes sends SIGTERM to each pod's main process and waits up to terminationGracePeriodSeconds (default 30 s) for the process to exit on its own. Only after that deadline does it send SIGKILL. A worker that ignores SIGTERM runs until the hard kill - any job it was processing is dropped mid-flight.

The broken pattern

# worker.py - BUG: no SIGTERM handler
import time, os

RUNNING = True   # never set to False

def process_job(job_id):
    print(f"processing job {job_id}")
    time.sleep(5)   # simulates real work
    print(f"done job {job_id}")

while RUNNING:
    job = fetch_next_job()
    process_job(job)

Send SIGTERM to this process and it keeps running. The only way to stop it is SIGKILL, which gives it no chance to finish the current job.

The fix: register a SIGTERM handler

import signal, time, os

RUNNING = True

def _on_sigterm(signum, frame):
    global RUNNING
    print("SIGTERM received - finishing current job, then exiting", flush=True)
    RUNNING = False

signal.signal(signal.SIGTERM, _on_sigterm)

def process_job(job_id):
    print(f"processing job {job_id}")
    time.sleep(5)
    print(f"done job {job_id}")

while RUNNING:
    job = fetch_next_job()
    process_job(job)   # current job runs to completion
    # RUNNING is checked here, between jobs - not mid-job

The handler sets RUNNING = False immediately when the signal arrives, but the while RUNNING: check only fires between jobs, so the in-flight job always finishes. New jobs are not started after the flag flips.

Verify it works

python3 worker.py &
PID=$!
sleep 2
kill -TERM $PID          # send SIGTERM
# worker logs the message, finishes current job, exits with code 0
wait $PID && echo "clean exit"

Compare that to the old behavior where ps -p $PID still showed the process alive after kill -TERM and kill -9 was the only escape.

Production additions

• preStop hook - add a short sleep in a Kubernetes preStop lifecycle hook so the load balancer has time to drain connections before SIGTERM arrives.
• SIGINT too - register the same handler for signal.SIGINT so Ctrl-C in dev behaves identically.
• Async workers - in asyncio, use loop.add_signal_handler(signal.SIGTERM, shutdown_callback) and cancel running tasks inside the callback.
• Flush buffers - before exiting, flush any in-memory queues (logs, metrics, batched DB writes) to avoid silent data loss.