Apache Airflow Tutorial: Backfill Missed Days With catchup

Q: How do I test an Airflow DAG for a single date without the scheduler?

Run `airflow dags test `, for example `airflow dags test event_loader 2026-02-01`. It executes the DAG in-process with no scheduler, worker, or triggerer, so you can confirm one day works and inspect any traceback before backfilling the full range.

A data pipeline that was never deployed leaves a gap - here, 90 days of a missing events table. The fix is an Airflow DAG with catchup=True and a start_date in the past, so the scheduler replays every missed daily run from that date to now. This tutorial shows the three knobs that make backfill work and how to verify one day with airflow dags test.

Data Engineerairflowdagbackfill

Why the data is missing

The events table is short 90 days because the DAG that loads it was never deployed. Every day the scheduler should have run a load task and did not, so nothing wrote that day's slice. You cannot fix this by running the loader once - you need one run per missed day, each stamped with the correct date.

Airflow already models this. Every scheduled run carries a logical date (the ds template, YYYY-MM-DD), and a task keyed on ds writes only that day's partition. Give Airflow a start_date in the past plus catchup=True, and it enumerates every daily interval between then and now, running the task once per interval. That is a backfill.

The three knobs that make backfill work

Backfill turns on with exactly three DAG parameters:

start_date - a date in the past. Airflow will not schedule any run before this.
schedule_interval="@daily" - the cadence; each interval becomes one run.
catchup=True - the switch. With it, deploying now replays every missed interval from start_date forward. (Most steady-state DAGs use catchup=False for exactly the opposite reason - you do not want a new DAG to stampede through months of history by accident. Here you want it.)

Write the DAG

Fill in event_loader.py. The _load(ds) helper is already provided - it writes /tmp/events-<ds>.txt, standing in for a real partitioned write. You wire it to a PythonOperator and pass the logical date in via op_kwargs:

from datetime import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator


def _load(ds: str) -> None:
    with open(f"/tmp/events-{ds}.txt", "w") as f:
        f.write(f"events loaded for {ds}\n")


with DAG(
    dag_id="event_loader",
    schedule_interval="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=True,
) as dag:
    load_events = PythonOperator(
        task_id="load_events",
        python_callable=_load,
        op_kwargs={"ds": "{{ ds }}"},
    )

The "{{ ds }}" string is a Jinja template. Airflow renders it to each run's logical date before the task executes, so run for 2026-02-01 calls _load("2026-02-01") and writes /tmp/events-2026-02-01.txt. The DAG must be built at module top level (inside the with block, not hidden in a function) or the scheduler will not discover it.

Verify one day before you backfill the range

Point Airflow at your DAGs folder and run a single logical date in-process. airflow dags test needs no scheduler and no worker - it runs the DAG then and there, which is ideal for local and CI checks:

export AIRFLOW__CORE__DAGS_FOLDER=/workspace
/opt/airflow-venv/bin/airflow dags list | grep event_loader
/opt/airflow-venv/bin/airflow dags test event_loader 2026-02-01
ls /tmp/events-*.txt

You should see event_loader in the list and /tmp/events-2026-02-01.txt appear. Once one date works, the full recovery is a single command:

/opt/airflow-venv/bin/airflow dags backfill event_loader \
  --start-date 2026-01-01 --end-date 2026-03-31

In production, cap concurrency with max_active_runs so the scheduler does not fire all 90 days at once and flood the warehouse, and keep _load idempotent - writing to a partition keyed on ds means re-running any date is a safe no-op.

Want to try it hands-on? HeyDevJob gives you this exact setup in a live cloud workspace in your browser - edit it, run it, and see it work. Free, nothing to install.

Try it in a workspace →

What you'll practice

Enabling Airflow backfill with a past start_date, a daily schedule, and catchup=True
Passing the logical date into a PythonOperator task with op_kwargs and the {{ ds }} template
Verifying a single day with airflow dags test before running the full backfill range

FAQ

How do I backfill missed runs in Airflow?

Set a start_date in the past and catchup=True on the DAG, then deploy - the scheduler enumerates every daily interval from start_date to now and runs the task once per interval. To recover a specific window on demand instead, run airflow dags backfill <dag_id> --start-date <d1> --end-date <d2>.

What does catchup=True do in an Airflow DAG?

catchup controls whether Airflow schedules runs for intervals that already passed between start_date and now. With catchup=True a freshly deployed DAG replays all of those missed runs, which is exactly what you want when filling a data gap; catchup=False skips them and only runs going forward.

How do I test an Airflow DAG for a single date without the scheduler?

Run airflow dags test <dag_id> <logical_date>, for example airflow dags test event_loader 2026-02-01. It executes the DAG in-process with no scheduler, worker, or triggerer, so you can confirm one day works and inspect any traceback before backfilling the full range.

How do I pass the execution date to an Airflow task?

Pass op_kwargs with a Jinja template to the operator, for example op_kwargs={"ds": "{{ ds }}"}. Airflow renders {{ ds }} to that run's logical date (YYYY-MM-DD) before the task executes, so each backfilled day receives its own date string.

Keep learning

Schedule an Airflow DAG (ETL)Data project Make an ETL Pipeline IdempotentData project Load Data IncrementallyData project Data roadmapStep by step to hired Data interview questionsSTAR answers All Data projectsProjects hub

Learn it by doing. Open this in a live cloud workspace, make the change yourself, and keep a record of the work you can share.

Open the workspace →