PySpark Schema: Read a CSV With an Explicit StructType

When PySpark infers a CSV schema, one bad value such as TBD can turn a whole numeric column into strings, and the aggregation fails. Declare the schema with StructType so every column has the type you expect.

Get started - freeMid and Senior open as you complete Junior projects
Data Engineerpythonpysparkschema

Why schema inference bites

df = spark.read.csv("transactions.csv", header=True, inferSchema=True)
df.groupBy("category").agg(F.sum("amount"))   # sum(VARCHAR) error

Inference samples the data and picks the narrowest type that fits every value. One row with amount = TBD means the column no longer fits a number, so it becomes a string, and sum() on a string column fails. Inference also costs an extra pass over the file.

Declare the schema with StructType

from pyspark.sql import types as T

schema = T.StructType([
    T.StructField("id", T.IntegerType()),
    T.StructField("category", T.StringType()),
    T.StructField("amount", T.DoubleType()),
])
df = spark.read.csv("transactions.csv", header=True, schema=schema)

Now amount is always a DoubleType. The schema is also documentation: the next person reading the job sees exactly what the file is supposed to hold.

Then deal with the bad row

A schema does not make bad data good. Decide what an unknown amount means: load it as NULL (an empty field), drop the row, or route it to a reject file. Do not let a placeholder like TBD sit in the source.

What you do in the project

You reproduce the sum(VARCHAR) error, find the bad row, add a StructType with DoubleType for amount, clean the data, and get exact per-category totals. The job uses the PySpark DataFrame API.

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free

What you'll practice

FAQ

How do I define a schema in PySpark?

Build a StructType from a list of StructField(name, type) entries, for example StructField("amount", DoubleType()), and pass it to spark.read.csv(..., schema=schema).

Should I use inferSchema in PySpark?

It is fine for exploring data. For production jobs an explicit schema is safer and faster, because inference needs an extra pass and can pick the wrong type when one value is bad.

What happens to values that do not match the schema?

It depends on the read mode. By default Spark sets unparseable values to null; FAILFAST raises an error; DROPMALFORMED drops the row. Choose the behavior on purpose.

Keep learning

PySpark groupBy: Fix a Wrong Column NameData projectPySpark Handle NULLs: .isNull() vs == NoneData projectRead Partitioned Parquet in PySparkData projectData roadmapStep by step to hiredData interview questionsSTAR answersAll Data projectsProjects hub

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free
Get started - free