How to Stream a Large File to S3 With AWS S3 Multipart Upload

Why put_object OOMs on large files

The naive uploader loads the whole file into memory before it sends a single byte:

def upload(path: str, key: str) -> None:
    with open(path, "rb") as f:
        s3.put_object(Bucket=BUCKET, Key=key, Body=f.read())

f.read() returns the entire file as one bytes object in RAM. A 12MB file needs 12MB of memory; a 4GB file needs 4GB. In a container with a modest memory limit, the process is killed by the OOM killer long before the upload finishes. There is a hard ceiling too: a single put_object cannot exceed 5GB, so above that the call fails outright regardless of memory.

The right tool is S3's multipart upload - split the object into parts, upload each part independently, and let S3 stitch them back together. You do not have to orchestrate that by hand.

Fix: use upload_file, the transfer manager

boto3 ships a high-level transfer manager. Replace the whole body of upload with one line:

def upload(path: str, key: str) -> None:
    s3.upload_file(path, BUCKET, key)

upload_file reads the file from disk in chunks - it never materializes the whole thing in memory. Above the default 8MB threshold it automatically switches to S3's multipart API: it splits the file into parts, uploads them concurrently, retries any part that fails, and completes the multipart upload for you. Memory use stays flat whether the file is 12MB or 12GB.

The full working uploader:

import os
import boto3

BUCKET = "lake"
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ.get("AWS_ENDPOINT_URL"),
    region_name=os.environ.get("AWS_DEFAULT_REGION", "us-east-1"),
)

def upload(path: str, key: str) -> None:
    s3.upload_file(path, BUCKET, key)

if __name__ == "__main__":
    upload("/workspace/sample.bin", "uploads/sample.bin")
    print("uploaded sample.bin")

Run it and confirm the object landed at full size:

python3 uploader.py
aws s3api head-object --bucket lake --key uploads/sample.bin \
  --query ContentLength --output text
# -> 12582912   (matches the local 12MB file, byte for byte)

Tune multipart with TransferConfig

The defaults are good, but you can control the multipart behavior when you need to. TransferConfig sets the threshold at which multipart kicks in, the chunk size for each part, and how many parts upload in parallel:

from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,   # switch to multipart above 8MB
    multipart_chunksize=8 * 1024 * 1024,   # 8MB per part
    max_concurrency=4,                     # 4 parts in flight at once
)
s3.upload_file(path, BUCKET, key, Config=config)

Larger max_concurrency speeds up big transfers on a fat network link; a larger multipart_chunksize cuts the part count (S3 allows up to 10,000 parts per object). The point is you rarely need to touch these - s3.upload_file(path, BUCKET, key) gets multipart, streaming, parallelism, and retries right by default, which is exactly why "read the whole file into memory" is the wrong reflex and this one-liner is the right one.