Buffer Events and Batch Writes to AWS S3 (Avoid the Small Files Problem)
An archiver that calls put_object once per event fills the bucket with thousands of tiny files that are slow and expensive to read. Buffer events in memory and flush them as one JSON Lines object per batch, by size or by time.
The small files problem
At 100 events a second, one object per event is 8.6 million objects a day. Every query engine that later reads the bucket (Athena, Spark, DuckDB) pays a request and some overhead per file, and S3 charges per PUT. Many small files is one of the most common reasons a data lake gets slow and costly.
Buffer, then flush on size or time
class EventWriter:
def __init__(self):
self._buffer = []
self._last_flush = time.monotonic()
def write(self, event):
self._buffer.append(event)
if (len(self._buffer) >= FLUSH_AT_EVENTS
or time.monotonic() - self._last_flush >= FLUSH_AT_SECONDS):
self._flush()
def _flush(self):
if not self._buffer:
return
body = "\n".join(json.dumps(e) for e in self._buffer) + "\n"
key = f"events/{int(time.time() * 1000)}.jsonl"
s3.put_object(Bucket="events", Key=key, Body=body)
self._buffer = []
self._last_flush = time.monotonic()
def close(self):
self._flush()
- Size trigger keeps objects a useful size.
- Time trigger bounds how stale the archive can get when traffic is low.
close()flushes the remainder so the last partial batch is not lost.
JSON Lines (one JSON document per line) is easy to append, split and load.
The trade-off
A buffer lives in memory, so a crash loses whatever has not been flushed. Production systems accept that window, or put a durable queue (Kinesis, Kafka) in front and let Firehose or a consumer do the batching.
What you do in the project
You change EventWriter to buffer and flush, then write 250 events and
confirm exactly three objects with 100, 100 and 50 lines.
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - freeWhat you'll practice
- Buffering events in memory and flushing on count or elapsed time
- Writing JSON Lines batches to S3 with put_object
- Flushing the final partial batch on close
FAQ
What is the small files problem?
It is the slowdown and extra cost caused by storing data as many tiny files. Each file adds request and metadata overhead for every reader and every write, so a lake with millions of small objects is slow to query and expensive to run.
How big should files in S3 be for analytics?
A common target is tens to hundreds of megabytes per file for columnar formats like Parquet. For raw event archives, batch by count and time so files are large enough to be efficient but data is never too stale.
What is JSON Lines?
JSON Lines (.jsonl) stores one JSON object per line. It is easy to append to, stream and split, and most data tools can load it directly.
Keep learning
Do real production work, free. Start with Junior projects in a live cloud workspace - every project you complete lands on your portfolio.
Get started - free