How to Convert CSV to Parquet With PyArrow

Why Parquet instead of CSV

CSV is row-oriented text: every query reads every column of every row, decompressed. Parquet is columnar and compressed, so analytics engines only read the columns they need and skip entire files when the partition doesn't match the filter. On a 50k-row event log, switching from CSV to snappy-compressed Parquet typically cuts file size by 5x and query time even more.

The standard lakehouse layout puts data under a Hive-partitioned path:

lake/events/year=2024/month=01/part-0.parquet
lake/events/year=2024/month=02/part-0.parquet
lake/events/year=2024/month=03/part-0.parquet

Query engines like DuckDB, Spark, Athena, and Trino read this layout natively and apply partition pruning automatically - a WHERE month = '01' filter never touches the other directories.

Reading the CSV and deriving partition columns

PyArrow does NOT auto-extract year/month from a timestamp column - you derive them in pandas first, then hand the table over to PyArrow.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# parse_dates turns the string column into a real datetime
df = pd.read_csv("events.csv", parse_dates=["event_ts"])

# derive integer partition columns BEFORE converting to a PyArrow table
df["year"]  = df["event_ts"].dt.year
df["month"] = df["event_ts"].dt.month

table = pa.Table.from_pandas(df)

Writing the partitioned Parquet dataset

pq.write_to_dataset creates the directory tree and writes one or more Parquet files per partition. Pass partition_cols in the order you want the directories nested:

pq.write_to_dataset(
    table,
    root_path="lake/events",
    partition_cols=["year", "month"],
    compression="snappy",
)

Snappy is the default compromise: fast decompression (readable with minimal CPU) and decent size reduction. It is built into PyArrow - no extra package needed.

Checking the result

# compare total sizes
ls -lh events.csv
du -sh lake/events/

# inspect a single file's metadata (row count, schema, compression)
python3 -c "
import pyarrow.parquet as pq
m = pq.read_metadata('lake/events/year=2024/month=01/part-0.parquet')
print(m)
"

The partition columns (year, month) are pulled OUT of the row data and encoded into the directory names - they won't appear as columns inside the Parquet file itself, but query engines reconstruct them from the path.

Reading it back

Any tool that understands Hive partitioning reads the whole dataset at once:

dataset = pq.read_table("lake/events")
# or filter to one partition - only that directory is read
dataset = pq.read_table("lake/events", filters=[("year", "=", 2024), ("month", "=", 1)])

DuckDB is even simpler: SELECT * FROM 'lake/events/**/*.parquet' WHERE year = 2024.