PySpark Schema: Read a CSV With an Explicit StructType
When PySpark infers a CSV schema, one bad value such as TBD can turn a whole numeric column into strings, and the aggregation fails. Declare the schema with StructType so every column has the type you expect.
Why schema inference bites
df = spark.read.csv("transactions.csv", header=True, inferSchema=True)
df.groupBy("category").agg(F.sum("amount")) # sum(VARCHAR) error
Inference samples the data and picks the narrowest type that fits every
value. One row with amount = TBD means the column no longer fits a number,
so it becomes a string, and sum() on a string column fails. Inference also
costs an extra pass over the file.
Declare the schema with StructType
from pyspark.sql import types as T
schema = T.StructType([
T.StructField("id", T.IntegerType()),
T.StructField("category", T.StringType()),
T.StructField("amount", T.DoubleType()),
])
df = spark.read.csv("transactions.csv", header=True, schema=schema)
Now amount is always a DoubleType. The schema is also documentation: the
next person reading the job sees exactly what the file is supposed to hold.
Then deal with the bad row
A schema does not make bad data good. Decide what an unknown amount means:
load it as NULL (an empty field), drop the row, or route it to a reject
file. Do not let a placeholder like TBD sit in the source.
What you do in the project
You reproduce the sum(VARCHAR) error, find the bad row, add a
StructType with DoubleType for amount, clean the data, and get exact
per-category totals. The job uses the PySpark DataFrame API.
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - freeWhat you'll practice
- Reproducing a schema inference failure on a CSV
- Defining a schema with StructType, StructField and DoubleType
- Deciding how to handle a row that does not fit the schema
FAQ
How do I define a schema in PySpark?
Build a StructType from a list of StructField(name, type) entries, for example StructField("amount", DoubleType()), and pass it to spark.read.csv(..., schema=schema).
Should I use inferSchema in PySpark?
It is fine for exploring data. For production jobs an explicit schema is safer and faster, because inference needs an extra pass and can pick the wrong type when one value is bad.
What happens to values that do not match the schema?
It depends on the read mode. By default Spark sets unparseable values to null; FAILFAST raises an error; DROPMALFORMED drops the row. Choose the behavior on purpose.
Keep learning
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - free