How to Migrate a pandas Pipeline to Polars

Why Polars is faster than pandas

pandas executes eagerly row by row using a single thread. Polars builds a lazy query plan - every scan_csv, join, group_by, and with_columns call adds a node to the plan without touching data. When you call .collect(), the Rust-based query planner optimizes the order, pushes predicates down, and runs the physical plan parallelized across all cores.

For join-heavy ETL on 100k-1M rows, Polars is typically 5-30x faster than pandas with minimal code changes.

The pandas starting point

import pandas as pd

orders = pd.read_csv("orders.csv", parse_dates=["created_at"])
customers = pd.read_csv("customers.csv")

merged = orders.merge(customers, left_on="customer_id", right_on="id")
merged["week"] = merged["created_at"].dt.isocalendar().week
weekly = merged.groupby(["region", "week"])["amount"].sum().reset_index(name="revenue")
weekly = weekly.sort_values(["region", "week"])
weekly["revenue_4w_avg"] = (
    weekly.groupby("region")["revenue"].transform(lambda s: s.rolling(4, min_periods=1).mean())
)
weekly.to_csv("revenue.csv", index=False)

The Polars lazy port

Replace every pandas call with its lazy Polars equivalent. The structure is almost identical - the key differences are scan_csv (lazy) instead of read_csv (eager), column expressions inside .agg(), and .over() for per-group window functions.

import polars as pl

orders = pl.scan_csv("orders.csv", try_parse_dates=True)
customers = pl.scan_csv("customers.csv")

result = (
    orders
    .join(customers, left_on="customer_id", right_on="id", how="inner")
    .with_columns(pl.col("created_at").dt.week().alias("week"))
    .group_by(["region", "week"])
    .agg(pl.col("amount").sum().alias("revenue"))
    .sort(["region", "week"])
    .with_columns(
        pl.col("revenue")
          .rolling_mean(window_size=4, min_samples=1)
          .over("region")
          .alias("revenue_4w_avg")
    )
)

df = result.collect()   # query plan executes here
df.write_csv("revenue.csv")

Key migration patterns

• pl.scan_csv vs pl.read_csv - scan_csv returns a LazyFrame and reads nothing until .collect(). Use it for any file you will filter, join, or aggregate - the planner can skip rows and columns it doesn't need.
• Column expressions in .agg() - Polars uses pl.col("amount").sum() inside .agg(...), not the pandas df["col"].sum() Series method.
• Window functions with .over() - pl.col("revenue").rolling_mean(4).over("region") computes the rolling average partitioned by region without a groupby + transform dance.
• ISO week - pl.col("created_at").dt.week() (pass try_parse_dates=True to scan_csv so the timestamp column is already a Date/Datetime type).

When to reach for Polars

Polars is the right default for single-machine ETL in the 1 MB-50 GB range. Below 1 MB, pandas is fine. Above 50 GB, add result.lazy().collect(streaming=True) for constant-memory processing - or reach for a distributed system.