How to Validate DataFrames With pandera

What pandera does

pandera is a pandas-native schema validator. You declare a DataFrameSchema that maps each column to its expected type and one or more Check constraints. When you call schema.validate(df), pandera compares every value in the DataFrame against those rules and raises SchemaErrors on the first violation - or, with lazy=True, collects all violations before raising.

This is the difference between a data-quality gate that stops a bad batch versus one that just points at the first bad row and leaves you hunting for more.

Declaring a schema

import pandas as pd
import pandera.pandas as pa

schema = pa.DataFrameSchema(
    {
        "email": pa.Column(
            str,
            nullable=False,
            checks=pa.Check.str_matches(r"^[^@]+@[^@]+\.[^@]+$"),
        ),
        "age": pa.Column(
            int,
            checks=pa.Check.in_range(18, 120),
        ),
        "order_total": pa.Column(
            float,
            checks=pa.Check.ge(0),
        ),
    },
    coerce=True,  # cast CSV strings to the declared dtype
)

coerce=True is important when reading from CSV: pandas reads ambiguous columns as object (string), so without coercion the dtype check on age (declared int) would fail immediately.

Validating lazily and handling failures

import sys

df = pd.read_csv("customers.csv")

try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as e:
    print("validation failed:")
    for _, row in e.failure_cases.iterrows():
        print(f"  FAIL {row['column']} / {row['check']}: {row['failure_case']}")
    sys.exit(1)

print("all checks passed")
sys.exit(0)

e.failure_cases is a DataFrame with columns column, check, failure_case, and index - so you can log, alert, or write the bad rows to a quarantine table rather than losing context.

Key checks at a glance

Where this fits in a pipeline

A validate.py script that exits non-zero on failure becomes a hard gate in any ETL: Airflow's BashOperator or PythonOperator marks the task failed; a shell pipeline stops on set -e; a CI check blocks the merge. The bad batch never reaches the load step.