Archive AWS DynamoDB Streams to S3 With Python (boto3)
DynamoDB Streams publishes every write to a table as a change record. You write the consumer in Python - walk each shard from the oldest record, convert the typed NewImage to plain JSON, and archive every order to S3.
DynamoDB Streams is change data capture
With Streams turned on, every insert, update and delete on a table produces a record in an ordered, 24-hour log. That makes it a CDC (change data capture) source: you can copy changes into S3 for a data lake, an audit archive or a warehouse without touching the application that writes them.
Reading a stream with boto3
Streams has its own client (dynamodbstreams). The read loop is: shards,
then an iterator per shard, then pages of records.
desc = streams.describe_stream(StreamArn=arn)["StreamDescription"]
for shard in desc["Shards"]:
it = streams.get_shard_iterator(
StreamArn=arn, ShardId=shard["ShardId"],
ShardIteratorType="TRIM_HORIZON")["ShardIterator"]
while it:
page = streams.get_records(ShardIterator=it)
if not page["Records"]:
break
for rec in page["Records"]:
item = to_plain(rec["dynamodb"]["NewImage"])
s3.put_object(Bucket="orders-archive",
Key=f'{item["id"]}.json', Body=json.dumps(item))
it = page.get("NextShardIterator")
TRIM_HORIZON starts at the oldest record still in the stream;
LATEST would only see new writes.
The typed-JSON gotcha
Stream images use DynamoDB's typed format: {"amount": {"N": "10"}}.
Numbers arrive as strings inside an N key. Convert them back, or your
archive ends up with "10" instead of 10 and every downstream sum breaks.
In production
A Lambda function with a DynamoDB Streams trigger or Kinesis Data Firehose usually does this, with retries and checkpointing handled for you. Writing the loop yourself is how you learn what those services do.
What you do in the project
You implement archive_stream() against a table with Streams enabled and
confirm all five orders land in S3 as plain JSON with numeric amounts.
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - freeWhat you'll practice
- Reading DynamoDB Streams shards with get_shard_iterator and get_records
- Converting DynamoDB typed images (N, S) to plain JSON
- Archiving change records to S3 with put_object
FAQ
What is a DynamoDB Stream?
A DynamoDB Stream is an ordered log of item-level changes to a table, kept for 24 hours. Each record can include the new image, the old image, or both, depending on the stream view type.
What does TRIM_HORIZON mean?
TRIM_HORIZON starts reading at the oldest record still available in the shard. LATEST starts after the most recent record, so it only sees changes made after you start reading.
How do I send DynamoDB changes to S3 in production?
Common options are a Lambda function triggered by the stream, or Kinesis Data Streams plus Firehose to batch records into S3. Both handle checkpointing and retries for you.
Keep learning
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - free