Data Engineering Projects
Every project below is a real data problem teams hit in production - broken pipelines, dropped records, slow queries, messy models. Fix it in a live cloud workspace and it lands on your portfolio as a clickable proof point.
34 projects to build
Make an ETL Pipeline IdempotentSwitched from INSERT to UPSERT (INSERT...ON CONFLICT) so re-running the same CSV produces the same row count, Speed Up a Slow SQL Report QueryRewrote the correlated subqueries as JOINs and added indexes on grouping columns - report time dropped from 10Make a CSV Importer Resilient to Bad DataWrapped the row-parsing loop in try/except so bad rows get logged and skipped instead of crashing the whole imHandle Orphan Records in an ETL JoinChecked each customer_id exists before inserting the order so FK violations go to a skip-log instead of crashiDebug a Broken SQL Revenue AggregationIdentified a one-to-many JOIN creating duplicate rows before SUM, fixed by narrowing the join or moving it to Convert Timestamps to Local Time (Timezones)Applied AT TIME ZONE to UTC timestamps before grouping by date so East Coast events at 11pm don't shift to theImplement Incremental Loading in an ETLAdded a watermark table tracking last_processed_ts so each run only loads `WHERE created_at > watermark`, neveWrite a Top-5 Revenue SQL ReportShipped a SQL query aggregating orders by customer, returning top 5 by total spend with order count and customValidate DataFrames With panderaDeclared a pandera DataFrameSchema that catches null emails, out-of-range ages, and bad order totals - validatConvert CSV to Parquet for a LakehouseBuilt a pyarrow-based CSV → Parquet converter with snappy compression and partitioning by year/month - the resQuery a Parquet Lake With DuckDBWrote a DuckDB analytics query against the partitioned Parquet lake - uses partition pruning + columnar projecRank Rows Per Group With SQL Window FunctionsReplaced the GROUP BY + correlated subquery with a single ROW_NUMBER() OVER (PARTITION BY region ORDER BY spenPartition an AWS S3 Data Export by DateRewrote the S3 key from a flat `orders.parquet` to a Hive-style `orders/year=YYYY/month=MM/day=DD/orders.parquFix a Wrong-Column PySpark GroupByFixed a groupBy that aggregated by `region_id` (which doesn't exist on the input) so the regional totals collaFix a PySpark Inner Join That Should Be LeftSwitched an `orders.join(customers, 'cust_id')` to a left join so orders whose customer was deleted still appeHandle NULLs in a PySpark PipelineReplaced `F.col('end_date') == None` (which evaluates to NULL, never True) with `F.col('end_date').isNull()` sBatch-Load a CSV Into AWS DynamoDBReplaced a Python `put_item`-in-a-loop with `batch_write_item` (25-item chunks + UnprocessedItems retry) - samStream a Large File With AWS S3 Multipart UploadReplaced `put_object(Body=open(path).read())` with `upload_file` so the file streams via S3's multipart API inCut a v1.0.0 Release Tag in GitCreated the annotated tag `v1.0.0` on the second-to-last commit (the "release commit" - the one before the WIPLoad Data Into AWS Redshift With dbtWrote the dbt model that reads raw_orders, filters to completed orders, normalizes cents-to-dollars on total, Repair a dbt Model That Doubles RowsSwitched the orders model from `materialized='incremental'` (no unique_key, no is_incremental guard) to `materBuild a Data Quality Monitoring APIBuilt automated data quality monitoring system with null detection, duplicate checking, anomaly detection, andSchedule a Pipeline as an Airflow DAGWrapped the ad-hoc ETL script in an Airflow DAG with three tasks (extract → transform → load) wired with depenBuild a Staging-to-Marts dbt ProjectStood up a dbt project with a sources block, a staging model that cleans the raw orders table, and a customer_Implement SCD2 With dbt SnapshotsWired a dbt snapshot that captures change history on the customers table - running it twice across an update pMigrate a pandas Pipeline to PolarsPorted the pandas join+groupby+rolling-window pipeline to Polars' lazy API - same output, ~5x faster on the 20Stream a Kafka Topic to ParquetBuilt a streaming sink that consumes events from a Kafka topic, batches 100 at a time, and writes each batch aImplement the Outbox Pattern (At-Least-Once)Replaced "INSERT order; then SQS send" (lossy on partial failure) with "INSERT order + INSERT outbox row in onBackfill 90 Days of Events With AirflowCreated an Airflow DAG (event_loader) with catchup=True and a 2026-01-01 start_date so the scheduler backfillsBuild a Kimball Star SchemaModeled the orders mart as a star schema - one fact table (fct_orders) and three dimensions (dim_customers, diWire Data Quality Tests Across a dbt LineageLayered dbt tests across the entire DAG - schema tests on sources, generic + custom tests on staging models, rHandle Parquet Schema Evolution (v1 + v2)Built `read_all_events()` that scans the directory, reads v1 + v2 Parquet files into a single PyArrow table viBuild an Exactly-Once Kafka PipelineBuilt the producer/consumer pair against Redpanda - producer publishes 200 messages (100 unique order_ids sentSwitch Batch ETL to Streaming on AWS KinesisBuilt produce() + consume() in streamer.py - producer puts 50 events to the `events` Kinesis stream; consumer
Plus 20+ more Data projects in the workspace, with new ones added regularly.
More for Data engineers
Build your Data portfolio free. Fix real systems in a live cloud workspace - every fix is yours to keep.
Start free →