How to Batch-Load a CSV Into AWS DynamoDB With batch_write_item

Why put_item in a loop is slow

DynamoDB's put_item writes exactly one item per API call. Loading a CSV by looping over rows and calling put_item for each one means one HTTPS round-trip per row - a 100-row file is 100 requests, a 10,000-row file is 10,000. Every call pays the network latency again, and on a provisioned table each write consumes capacity units serially, so a large load can throttle itself or exhaust the day's budget.

Here is the slow version - correct, but one round-trip per row:

import csv, os, boto3

dynamodb = boto3.resource(
    "dynamodb",
    endpoint_url=os.environ.get("AWS_ENDPOINT_URL"),
    region_name=os.environ.get("AWS_DEFAULT_REGION", "us-east-1"),
)
table = dynamodb.Table("products")

rows = list(csv.DictReader(open("products.csv")))
for row in rows:                       # 1 request per row - slow
    table.put_item(Item={
        "sku":   row["sku"],
        "name":  row["name"],
        "price": int(row["price"]),
    })

Use batch_write_item in chunks of 25

DynamoDB's bulk write, batch_write_item, accepts up to 25 items per call. Chunk the CSV rows into groups of 25 and send each group as a list of PutRequest entries keyed by the table name. That turns 100 single writes into 4 batch calls:

client = dynamodb.meta.client

for i in range(0, len(rows), 25):
    chunk = rows[i:i+25]
    req = {"products": [
        {"PutRequest": {"Item": {
            "sku":   r["sku"],
            "name":  r["name"],
            "price": int(r["price"]),
        }}} for r in chunk
    ]}
    while req:
        resp = client.batch_write_item(RequestItems=req)
        req = resp.get("UnprocessedItems") or {}

Two details are load-bearing. First, the 25-item cap is a hard limit - send more and the call is rejected, so the range(0, len(rows), 25) chunking is required, not an optimization. Second, batch_write_item does not raise on partial failure: under per-partition throttling it returns the items it could not write in the response's UnprocessedItems field. Without the while req: retry loop those rows are dropped silently - the script exits cleanly having lost data.

Verify every row landed

Trusting the exit code is not enough - confirm the item count in the table itself:

cd /workspace
python3 loader.py
aws dynamodb scan --table-name products --select COUNT --output text --query Count
# 100

A scan COUNT of 100 for a 100-row CSV means every row was written and no UnprocessedItems were quietly dropped.

The production shortcut

Once you understand the raw API, boto3 ships a helper that hides the chunking and the retry: the batch_writer() context manager. It buffers writes, flushes them in batches of 25, and re-submits unprocessed items automatically:

with table.batch_writer() as batch:
    for row in rows:
        batch.put_item(Item={
            "sku":   row["sku"],
            "name":  row["name"],
            "price": int(row["price"]),
        })

This is what most production loaders use. Knowing the underlying batch_write_item call is still worth it - it is what the helper does under the hood, and it is what you reach for when you need custom retry or backoff logic.