How to Handle Parquet Schema Evolution With PyArrow

Why mixed-schema Parquet files break readers

Parquet stores the schema inside each file. When a release adds a column, older files keep their old schema and new files carry the new one. A directory ends up looking like this:

data/events_v1.parquet   # id (int64), name (string)
data/events_v2.parquet   # id (int64), name (string), category (string)

Reading these one at a time is fine, but the moment you try to stack them into one table the column sets no longer line up. Downstream code that does df["category"] raises KeyError on any v1 row, because v1 never had that column. The lake evolved; the reader did not.

Inspect the two schemas to confirm the drift before writing any code:

python3 -c "import pyarrow.parquet as pq; [print(f, pq.read_schema('/workspace/data/'+f)) for f in ['events_v1.parquet','events_v2.parquet']]"

You will see category present in v2 and absent in v1. That is the whole problem.

Unify the files with PyArrow schema promotion

PyArrow can concatenate tables with different schemas as long as you ask it to promote the union of columns. Missing columns are backfilled with nulls, which you then replace with the business default:

import os
import pyarrow as pa
import pyarrow.parquet as pq

tables = []
for name in sorted(os.listdir("/workspace/data")):
    if name.endswith(".parquet"):
        tables.append(pq.read_table(f"/workspace/data/{name}"))

unified = pa.concat_tables(tables, promote_options="permissive")

With permissive promotion the resulting table has columns [id, name, category], and the v1 rows carry null in category. Promotion is the safe direction of schema evolution: ADDING a column is compatible, so old and new files coexist in one view.

Fill the default and return one DataFrame

The pandas path is even more direct and is what the reference implementation uses - read each file, add the column if it is missing, then concatenate:

import os
import pandas as pd
import pyarrow.parquet as pq


def read_all_events() -> pd.DataFrame:
    frames = []
    for name in sorted(os.listdir("/workspace/data")):
        if not name.endswith(".parquet"):
            continue
        df = pq.read_table(f"/workspace/data/{name}").to_pandas()
        if "category" not in df.columns:
            df["category"] = "uncategorized"
        frames.append(df[["id", "name", "category"]])
    out = (
        pd.concat(frames, ignore_index=True)
        .sort_values("id")
        .reset_index(drop=True)
    )
    return out

Two details make this correct. First, fill the default BEFORE concatenating, so v1 rows get "uncategorized" instead of NaN - consumers never see a mixed null/string column. Second, project df[["id", "name", "category"]] in a fixed order so every frame lines up regardless of which version it came from. Sorting by id at the end gives a stable, deterministic result.

Which schema changes are safe

Not all evolution is equal. Adding a nullable column - the case here - is backward compatible: old readers ignore it, new readers get the default. Renaming a column is dangerous, because to a Parquet reader a rename looks like a drop plus an add, and old data silently disappears from the new name. Dropping a column and later re-adding the same name with a different type is a footgun that produces type-cast errors on read. Table formats like Apache Iceberg and Delta Lake track column lineage to make these operations safe; for raw Parquet, a unified reader like the one above is what carries you across versions.