How to Handle NULLs in PySpark (Use .isNull(), Not == None)

Why == None always returns nothing

In standard Python, x == None works as you'd expect. In PySpark (and any SQL-based engine including DuckDB's Spark-compatible API), the comparison follows SQL NULL semantics:

NULL == NULL   -> NULL   (not True)
NULL == 'foo'  -> NULL   (not False)

The filter() call drops any row where the condition evaluates to anything other than True - including NULL. So F.col("end_date") == None evaluates to NULL for every row that has a NULL end_date, and those rows are filtered out instead of kept. The result: zero rows, and no error message.

The correct pattern: .isNull() and .isNotNull()

PySpark's Column class has dedicated methods for NULL checks:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder.getOrCreate()

df = spark.read.option("header", True).csv("subscriptions.csv")

# Wrong - always returns 0 rows when end_date can be NULL
# active = df.filter(F.col("end_date") == None)

# Correct - returns rows where end_date IS NULL (active subscriptions)
active = df.filter(F.col("end_date").isNull())

active.show()

To find rows where the column is NOT NULL (expired subscriptions):

expired = df.filter(F.col("end_date").isNotNull())

The same rule applies in SQL expressions

If you write PySpark queries using spark.sql() or .filter() with a SQL string, the same rule applies - use IS NULL, not = NULL:

# SQL string form - also correct
active = df.filter("end_date IS NULL")
expired = df.filter("end_date IS NOT NULL")

WHERE end_date = NULL in SQL is a no-op for the same reason: the comparison returns NULL, so no rows match.

How to spot this bug quickly

The signature of the == None bug is a filter that returns zero rows when you know the column has NULL values. Cross-check with:

# Count NULLs in the column - should be > 0 if the bug is present
df.filter(F.col("end_date").isNull()).count()

If that count is positive but your equality filter returned zero, you've found the cause.

Linter coverage

pyspark-stubs and ruff (with the pandas-vet plugin) flag == None comparisons on DataFrame columns. Adding these to your CI catches the pattern before it reaches production.