Dagster Tutorial: Build Your First Asset Pipeline

What is Dagster?

Dagster is an open-source orchestrator for data pipelines. Where older tools think in terms of tasks ("run this script, then that one"), Dagster thinks in terms of assets - the tables, files, dashboards, and machine-learning models your pipeline actually produces. You write a Python function that computes an asset, annotate it, and Dagster builds a dependency graph from the relationships between those assets.

That shift matters in practice. Because Dagster knows what each asset is, not just that a task ran, you get data lineage, the ability to materialize a single asset on demand, partitioned backfills, and a live catalog of everything your platform owns. It is built for the full lifecycle: local development with hot-reloading, testing without a running scheduler, and production deployment to Kubernetes or Dagster+.

Dagster is licensed Apache 2.0 and is widely used for ELT, analytics engineering (it has a built-in dbt integration), and ML feature pipelines. This tutorial covers the modern workflow built around the dg CLI and software-defined assets.

How Dagster works (key concepts)

A handful of concepts carry most of the framework:

• Asset - a persistent object your pipeline produces, declared with the @dg.asset decorator on a function. The function body computes the asset; the return value (and any I/O manager) handles storage.
• Dependencies - you express them either by accepting an upstream asset as a function argument, or with the deps=[...] parameter when you only need ordering, not the data in memory.
• Definitions - the top-level container that tells Dagster which assets, jobs, schedules, sensors, and resources exist in your project. With the dg workflow, definitions are discovered automatically from your defs/ folder.
• Resources - shared, configurable connections (a database client, an S3 bucket, an API token) injected into assets so the same code runs against dev and prod.
• Schedules and sensors - schedules run on a cron cadence; sensors react to external events (a new file, an upstream completion). Both trigger materializations.
• Partitions - slice an asset by date or category so you can backfill or rematerialize one slice at a time.

The mental model: define assets, let Dagster infer the graph, then materialize - either manually in the UI, on a schedule, or in response to a sensor.

Installing Dagster

Dagster needs Python 3.10 or newer. The fastest path is the create-dagster CLI, which scaffolds a proper Python package the Dagster UI can load. Using the uv package manager:

uvx create-dagster@latest project dagster-quickstart
cd dagster-quickstart
source .venv/bin/activate    # macOS/Linux; use .venv\Scripts\activate on Windows
uv add pandas

If you prefer pip, install the CLI first, then scaffold and set up the editable package:

pip install create-dagster
create-dagster project dagster-quickstart
cd dagster-quickstart
python -m venv .venv
source .venv/bin/activate
pip install pandas
pip install --editable . --group dev

You can also add the libraries to an existing project directly:

pip install dagster dagster-webserver dagster-dg-cli

Verify the toolchain with dg --version. The scaffolded project puts your code under src/dagster_quickstart/defs/, which is where Dagster looks for definitions.

Hands-on: define and materialize an asset graph

Let's build a tiny two-step pipeline: download raw records, then transform them. Create the file with the scaffold helper so it lands in the right place:

dg scaffold defs dagster.asset assets.py

That writes src/dagster_quickstart/defs/assets.py. Replace its contents with two connected assets:

import pandas as pd
import dagster as dg


@dg.asset
def raw_people() -> pd.DataFrame:
    """Source data: a small set of people with ages."""
    return pd.DataFrame(
        {
            "name": ["Ada", "Linus", "Grace", "Dennis"],
            "age": [28, 35, 52, 41],
        }
    )


@dg.asset
def people_with_cohort(
    context: dg.AssetExecutionContext,
    raw_people: pd.DataFrame,
) -> pd.DataFrame:
    """Derive an age cohort from the raw data."""
    df = raw_people.copy()
    df["cohort"] = pd.cut(
        df["age"],
        bins=[0, 30, 40, 100],
        labels=["Young", "Middle", "Senior"],
    )
    context.log.info(f"Computed {len(df)} rows of cohort data")
    return df

Notice that people_with_cohort takes raw_people as an argument. That argument name is the dependency declaration - Dagster wires raw_people upstream of people_with_cohort automatically, and passes the returned DataFrame straight in. No manual DAG wiring, no string task IDs.

Start the development server from the project root:

dg dev

Open http://localhost:3000 and you will see both assets in the graph, raw_people feeding people_with_cohort. Click Materialize all to run them. You can also do this from the terminal without the UI:

dg list defs        # confirm Dagster discovered both assets
dg check defs       # validate the definitions load cleanly
dg launch --assets "*"   # materialize every asset

To run the pipeline on a cadence, add a schedule in your definitions. Schedules are definitions that Dagster picks up the same way it picks up assets, and they target a job built from your asset selection - giving you a daily refresh without leaving Python.

When to use Dagster

Reach for Dagster when your pipeline is fundamentally about producing and refreshing data assets, and you want lineage, observability, and selective rematerialization without extra work. It works well for ELT and analytics stacks (especially alongside dbt), for ML feature and training pipelines where you want to track exactly which dataset produced which model, and for teams that need a shared catalog of what every pipeline owns.

It is also a strong choice when local testing matters: assets are plain Python functions, so you can unit-test transformation logic without standing up a scheduler. If you are migrating off Airflow but want to keep existing DAGs running during the transition, Dagster can orchestrate them while you port logic to assets incrementally.

Dagster is less of a fit when you only need a generic cron runner for arbitrary, non-data tasks, or when a one-off script is all the situation calls for - the asset model adds structure you will not use. But for any pipeline you will run repeatedly and need to reason about over time, the asset-centric approach is worth the extra structure.