Read Partitioned Parquet in PySpark (and Filter the Right Partition)

A report that reads one hard-coded partition folder keeps returning old data long after new partitions land. Read the table root so PySpark discovers every partition, then filter on the partition column.

Get started - freeMid and Senior open as you complete Junior projects
Data Engineerpythonpysparkparquet

How Hive-style partitioning works

A partitioned table is a folder tree where the partition value is in the path:

lake/orders/
  year=2024/part-0.parquet
  year=2025/part-0.parquet
  year=2026/part-0.parquet

When you read lake/orders, the engine walks the tree, turns year=2026 into a real year column, and can skip whole folders when you filter on it. That skipping is called partition pruning.

The bug: a path that pins one partition

df = spark.read.parquet("/workspace/lake/orders/year=2024")
df = df.filter(F.col("year") == 2024)

Reading a single partition folder means new years never show up, and the filter hides the problem because it agrees with the path. Both lines have to change together.

Read the root, filter the column

df = spark.read.parquet("/workspace/lake/orders")
df = df.filter(F.col("year") == 2026)

The reader finds every partition; the filter picks the one you need, and pruning still means only that folder is scanned.

Keep the year out of the code

In a scheduled job, compute the partition from the run date (or pass it as a parameter) instead of hard-coding it, so the job does not quietly go stale next year.

What you do in the project

You trace why the report shows only 2024 rows, change the read path and the filter, and confirm the 12 current-year rows come back. The job uses the PySpark DataFrame API.

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free

What you'll practice

FAQ

How do I read partitioned Parquet files in PySpark?

Point spark.read.parquet at the root folder of the table, not at a single partition folder. Spark discovers the key=value folders and adds the partition columns to the DataFrame.

What is partition pruning?

Partition pruning is when the engine skips partition folders that cannot match a filter, for example reading only year=2026 for WHERE year = 2026. It only works when you filter on the partition column.

Why does my Spark job keep returning old data?

A common cause is a hard-coded path or filter pinned to an old partition or date. Read the table root and compute the partition value from the run date.

Keep learning

Pandas to Parquet: Partition an S3 ExportData projectHow to Query Parquet Files With DuckDBData projectPySpark StructType: Define a CSV SchemaData projectData roadmapStep by step to hiredData interview questionsSTAR answersAll Data projectsProjects hub

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free
Get started - free