The Outbox Pattern (Reliable Events)

The problem it solves (dual write)

A common flow: save an order to the database, then publish an order_created event to a queue. But that's two systems, and there's no transaction across them:

# BROKEN: if the process dies between these, the order exists but no event is sent
db.insert_order(order)
sqs.publish("order_created", order)   # crash here -> event lost forever

Downstream consumers (notifications, analytics) silently miss the event. Retrying naively risks the opposite - publishing twice, or publishing for an order that rolled back.

The fix: write the event to an outbox table, in the same transaction

Instead of publishing inline, insert an outbox row in the same database transaction as the business write. Either both commit or neither does - atomic:

def create_order(customer, amount):
    with conn:                      # one transaction: commit or rollback together
        cur = conn.cursor()
        cur.execute("INSERT INTO orders (customer, amount) VALUES (%s, %s) RETURNING id",
                    (customer, amount))
        order_id = cur.fetchone()[0]
        cur.execute(
            "INSERT INTO outbox (event_type, payload) VALUES ('order_created', %s)",
            (json.dumps({"order_id": order_id, "customer": customer, "amount": amount}),))
    return order_id

No queue call here - just two inserts that commit together.

The poller publishes the outbox

A separate process reads unpublished outbox rows and sends them to the queue, marking each as sent only after the publish succeeds:

def publish_outbox():
    rows = db.query("SELECT id, event_type, payload FROM outbox WHERE published_at IS NULL")
    for r in rows:
        sqs.publish(r["event_type"], r["payload"])           # publish
        db.execute("UPDATE outbox SET published_at = now() WHERE id = %s", (r["id"],))

Why this gives at-least-once delivery

• If the app crashes after the transaction, the event is still in the outbox - the poller sends it on the next run. Nothing is lost.
• If the poller crashes after publishing but before marking the row sent, it will publish that row again next run. So delivery is at-least-once, not exactly-once.
• Therefore consumers must be idempotent - dedupe on the event/order id so a re-delivered event is harmless.

In the real world

Tools like Debezium read the outbox table's change log directly (change-data-capture) so you don't even run a poller. The guarantee is the same: write once, atomically; deliver at least once; dedupe on the consumer.