Data Engineering Projects
Every project below is a real data problem teams hit in production - broken pipelines, dropped records, slow queries, messy models. Fix it in a live cloud workspace and it lands on your portfolio as a clickable proof point.
24 projects to build
Make an ETL Pipeline IdempotentSwitched from INSERT to UPSERT (INSERT...ON CONFLICT) so re-running the same CSV produces the same row count, Speed Up a Slow SQL Report QueryRewrote the correlated subqueries as JOINs and added indexes on grouping columns - report time dropped from 10Make a CSV Importer Resilient to Bad DataWrapped the row-parsing loop in try/except so bad rows get logged and skipped instead of crashing the whole imConvert Timestamps to Local Time (Timezones)Applied AT TIME ZONE to UTC timestamps before grouping by date so East Coast events at 11pm don't shift to theImplement Incremental Loading in an ETLAdded a watermark table tracking last_processed_ts so each run only loads `WHERE created_at > watermark`, neveValidate DataFrames With panderaDeclared a pandera DataFrameSchema that catches null emails, out-of-range ages, and bad order totals - validatConvert CSV to Parquet for a LakehouseBuilt a pyarrow-based CSV → Parquet converter with snappy compression and partitioning by year/month - the resQuery a Parquet Lake With DuckDBWrote a DuckDB analytics query against the partitioned Parquet lake - uses partition pruning + columnar projecRank Rows Per Group With SQL Window FunctionsReplaced the GROUP BY + correlated subquery with a single ROW_NUMBER() OVER (PARTITION BY region ORDER BY spenFix a Wrong-Column PySpark GroupByFixed a groupBy that aggregated by `region_id` (which doesn't exist on the input) so the regional totals collaFix a PySpark Inner Join That Should Be LeftSwitched an `orders.join(customers, 'cust_id')` to a left join so orders whose customer was deleted still appeHandle NULLs in a PySpark PipelineReplaced `F.col('end_date') == None` (which evaluates to NULL, never True) with `F.col('end_date').isNull()` sCut a v1.0.0 Release Tag in GitCreated the annotated tag `v1.0.0` on the second-to-last commit (the "release commit" - the one before the WIPLoad Data Into AWS Redshift With dbtWrote the dbt model that reads raw_orders, filters to completed orders, normalizes cents-to-dollars on total, Repair a dbt Model That Doubles RowsSwitched the orders model from `materialized='incremental'` (no unique_key, no is_incremental guard) to `materBuild a Data Quality Monitoring APIBuilt automated data quality monitoring system with null detection, duplicate checking, anomaly detection, andSchedule a Pipeline as an Airflow DAGWrapped the ad-hoc ETL script in an Airflow DAG with three tasks (extract → transform → load) wired with depenBuild a Staging-to-Marts dbt ProjectStood up a dbt project with a sources block, a staging model that cleans the raw orders table, and a customer_Implement SCD2 With dbt SnapshotsWired a dbt snapshot that captures change history on the customers table - running it twice across an update pMigrate a pandas Pipeline to PolarsPorted the pandas join+groupby+rolling-window pipeline to Polars' lazy API - same output, ~5x faster on the 20Implement the Outbox Pattern (At-Least-Once)Replaced "INSERT order; then SQS send" (lossy on partial failure) with "INSERT order + INSERT outbox row in onBuild a Kimball Star SchemaModeled the orders mart as a star schema - one fact table (fct_orders) and three dimensions (dim_customers, diBuild an Exactly-Once Kafka PipelineBuilt the producer/consumer pair against Redpanda - producer publishes 200 messages (100 unique order_ids sentSwitch Batch ETL to Streaming on AWS KinesisBuilt produce() + consume() in streamer.py - producer puts 50 events to the `events` Kinesis stream; consumer
Plus 30+ more Data projects in the workspace, with new ones added regularly.
More for Data engineers
Build your Data portfolio free. Fix real systems in a live cloud workspace - every fix is yours to keep.
Start free →