How to Build an Exactly-Once Kafka Pipeline

Why Kafka delivers duplicates

Kafka's default delivery semantics are at-least-once: a producer retries on network failures, and the broker may deliver the same message more than once. Even with enable.idempotence=True on the producer (which prevents broker-side duplicates), application retries and redeliveries can still send the same logical event twice. The consumer is responsible for making its downstream effects idempotent.

The producer: simulate a duplicate publish

import json
from kafka import KafkaProducer

BROKER = "localhost:9092"
TOPIC = "orders"

def main():
    p = KafkaProducer(
        bootstrap_servers=BROKER,
        value_serializer=lambda v: json.dumps(v).encode(),
    )
    # Send each order_id twice - simulating a retry/redelivery
    for _ in range(2):
        for order_id in range(1, 101):
            p.send(TOPIC, {"order_id": order_id})
    p.flush()
    p.close()

200 messages land in the topic (100 unique order IDs, each sent twice).

The consumer: dedup with a seen-set

Track processed IDs in an in-memory set. Skip any message whose key has already been handled:

import json
from kafka import KafkaConsumer

BROKER = "localhost:9092"
TOPIC = "orders"

def main():
    consumer = KafkaConsumer(
        TOPIC,
        bootstrap_servers=BROKER,
        auto_offset_reset="earliest",
        consumer_timeout_ms=5000,
        value_deserializer=lambda b: json.loads(b.decode()),
    )
    seen = set()
    with open("processed.txt", "w") as f:
        for msg in consumer:
            oid = msg.value["order_id"]
            if oid in seen:
                continue        # duplicate - skip
            seen.add(oid)
            f.write(f"{oid}\n") # downstream effect fires exactly once
    consumer.close()

processed.txt ends up with exactly 100 lines despite 200 messages in the topic. The if oid in seen: continue guard is the entire exactly-once mechanism for downstream effects.

What makes this production-ready

The in-memory set works for a single consumer session, but restarts lose the state. Production setups persist the seen-IDs to survive crashes and restarts:

• Redis SETNX - atomic "set if not exists"; returns 0 on a duplicate so you skip in one round-trip.
• Postgres INSERT ... ON CONFLICT DO NOTHING - idempotent upsert; the row already existing is the dedup signal.
• Kafka Streams RocksDB - persistent state store that survives consumer restarts inside the same consumer group.

Pair the dedup store with consumer-group offset management so the consumer resumes from its last committed offset on restart rather than re-reading the whole topic. Those two pieces together - persistent seen-set plus committed offsets - give you reliable exactly-once semantics for any downstream effect.

Check your work

wc -l processed.txt          # should print 100
sort -u processed.txt | wc -l  # should also print 100 - no duplicates