Buffer Events and Batch Writes to AWS S3 (Avoid the Small Files Problem)

An archiver that calls put_object once per event fills the bucket with thousands of tiny files that are slow and expensive to read. Buffer events in memory and flush them as one JSON Lines object per batch, by size or by time.

Get started - freeMid and Senior open as you complete Junior projects
Data Engineerawss3batching

The small files problem

At 100 events a second, one object per event is 8.6 million objects a day. Every query engine that later reads the bucket (Athena, Spark, DuckDB) pays a request and some overhead per file, and S3 charges per PUT. Many small files is one of the most common reasons a data lake gets slow and costly.

Buffer, then flush on size or time

class EventWriter:
    def __init__(self):
        self._buffer = []
        self._last_flush = time.monotonic()

    def write(self, event):
        self._buffer.append(event)
        if (len(self._buffer) >= FLUSH_AT_EVENTS
                or time.monotonic() - self._last_flush >= FLUSH_AT_SECONDS):
            self._flush()

    def _flush(self):
        if not self._buffer:
            return
        body = "\n".join(json.dumps(e) for e in self._buffer) + "\n"
        key = f"events/{int(time.time() * 1000)}.jsonl"
        s3.put_object(Bucket="events", Key=key, Body=body)
        self._buffer = []
        self._last_flush = time.monotonic()

    def close(self):
        self._flush()

JSON Lines (one JSON document per line) is easy to append, split and load.

The trade-off

A buffer lives in memory, so a crash loses whatever has not been flushed. Production systems accept that window, or put a durable queue (Kinesis, Kafka) in front and let Firehose or a consumer do the batching.

What you do in the project

You change EventWriter to buffer and flush, then write 250 events and confirm exactly three objects with 100, 100 and 50 lines.

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free

What you'll practice

FAQ

What is the small files problem?

It is the slowdown and extra cost caused by storing data as many tiny files. Each file adds request and metadata overhead for every reader and every write, so a lake with millions of small objects is slow to query and expensive to run.

How big should files in S3 be for analytics?

A common target is tens to hundreds of megabytes per file for columnar formats like Parquet. For raw event archives, batch by count and time so files are large enough to be efficient but data is never too stale.

What is JSON Lines?

JSON Lines (.jsonl) stores one JSON object per line. It is easy to append to, stream and split, and most data tools can load it directly.

Keep learning

Stream a Large File With AWS S3 MultipartData projectPandas to Parquet: Partition an S3 ExportData projectHow to Stream Data With AWS KinesisData projectData roadmapStep by step to hiredData interview questionsSTAR answersAll Data projectsProjects hub

Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.

Get started - free
Get started - free