How to Fix a PySpark groupBy Column Name Error

What goes wrong

PySpark (and the duckdb.experimental.spark backend) resolves column names at execute time, not at the point you write df.groupBy(...). If you pass a name that isn't in the DataFrame, you get an error like:

BinderException: Referenced column "region_id" not found in FROM clause!
Candidates: region, order_id, amount

Or, depending on the engine version, the query runs but groups on nothing useful - producing one row per input row instead of one row per group.

Read the schema first

Before chasing the aggregation logic, confirm the actual column names:

from duckdb.experimental.spark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.read.csv("/workspace/orders.csv", header=True)
df.printSchema()
# root
#  |-- region: string (nullable = true)
#  |-- order_id: string (nullable = true)
#  |-- amount: double (nullable = true)

printSchema() is the fastest way to catch a name mismatch before the job runs.

Fix the groupBy argument

Once you know the real column name, update the groupBy call to match:

from duckdb.experimental.spark.sql import SparkSession
from duckdb.experimental.spark.sql import functions as F

spark = SparkSession.builder.appName("regional-sales").getOrCreate()
orders = spark.read.csv("/workspace/orders.csv", header=True)

# Wrong - region_id does not exist in this DataFrame
# revenue = orders.groupBy("region_id").agg(F.sum("amount").alias("revenue"))

# Correct - use the actual column name
revenue = (
    orders.groupBy("region")
          .agg(F.sum("amount").alias("revenue"))
          .orderBy("region")
)

for row in revenue.collect():
    print(f"{row['region']}: {row['revenue']:.2f}")
# apac: 125.00
# eu: 400.00
# us: 325.00

The output collapses from one row per order to one row per region - exactly what the rollup is supposed to produce.

Prevent it in production

• Call df.printSchema() or df.columns once at the top of every notebook - stale schema assumptions are the most common source of groupBy bugs when the upstream CSV or table changes.
• Use a StructType schema when reading CSVs in production code so Spark rejects unexpected schemas at read time, not at group time.
• Static checkers like PySpark Pandera or dbt column contracts catch column-name drift before the job ever runs in a pipeline.