Archive AWS DynamoDB Streams to S3 With Python (boto3)

DynamoDB Streams publishes every write to a table as a change record. You write the consumer in Python - walk each shard from the oldest record, convert the typed NewImage to plain JSON, and archive every order to S3.

Get started - freeMid and Senior open as you complete Junior projects
Data Engineerawsdynamodbdynamodb-streams

DynamoDB Streams is change data capture

With Streams turned on, every insert, update and delete on a table produces a record in an ordered, 24-hour log. That makes it a CDC (change data capture) source: you can copy changes into S3 for a data lake, an audit archive or a warehouse without touching the application that writes them.

Reading a stream with boto3

Streams has its own client (dynamodbstreams). The read loop is: shards, then an iterator per shard, then pages of records.

desc = streams.describe_stream(StreamArn=arn)["StreamDescription"]
for shard in desc["Shards"]:
    it = streams.get_shard_iterator(
        StreamArn=arn, ShardId=shard["ShardId"],
        ShardIteratorType="TRIM_HORIZON")["ShardIterator"]
    while it:
        page = streams.get_records(ShardIterator=it)
        if not page["Records"]:
            break
        for rec in page["Records"]:
            item = to_plain(rec["dynamodb"]["NewImage"])
            s3.put_object(Bucket="orders-archive",
                          Key=f'{item["id"]}.json', Body=json.dumps(item))
        it = page.get("NextShardIterator")

TRIM_HORIZON starts at the oldest record still in the stream; LATEST would only see new writes.

The typed-JSON gotcha

Stream images use DynamoDB's typed format: {"amount": {"N": "10"}}. Numbers arrive as strings inside an N key. Convert them back, or your archive ends up with "10" instead of 10 and every downstream sum breaks.

In production

A Lambda function with a DynamoDB Streams trigger or Kinesis Data Firehose usually does this, with retries and checkpointing handled for you. Writing the loop yourself is how you learn what those services do.

What you do in the project

You implement archive_stream() against a table with Streams enabled and confirm all five orders land in S3 as plain JSON with numeric amounts.

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free

What you'll practice

FAQ

What is a DynamoDB Stream?

A DynamoDB Stream is an ordered log of item-level changes to a table, kept for 24 hours. Each record can include the new image, the old image, or both, depending on the stream view type.

What does TRIM_HORIZON mean?

TRIM_HORIZON starts reading at the oldest record still available in the shard. LATEST starts after the most recent record, so it only sees changes made after you start reading.

How do I send DynamoDB changes to S3 in production?

Common options are a Lambda function triggered by the stream, or Kinesis Data Streams plus Firehose to batch records into S3. Both handle checkpointing and retries for you.

Keep learning

AWS DynamoDB Batch Write From a CSVData projectHow to Stream Data With AWS KinesisData projectPandas to Parquet: Partition an S3 ExportData projectData roadmapStep by step to hiredData interview questionsSTAR answersAll Data projectsProjects hub

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free
Get started - free