Example

This section introduces an advanced Kedro project with Dagster, inspired by the Kedro-Dagster Example Repository.

Danger

This documentation section is a work in progress. Please check back later for a more complete guide!

Project Overview

This repo builds on the Kedro Spaceflights tutorial, augmented with dynamic pipelines following the GetInData blog post.

Note

Here, parameters for dynamic pipelines are namespaced via YAML inheritance rather than a custom merge resolver.

Additionally, the project features:

• Multi-environment support: Easily switch between local, dev, staging, and prod environments. Each environment has its own dagster.yml and catalog.yml in conf/<ENV_NAME>/.
• MLflow integration: kedro-mlflow is used for experiment tracking and model registry. Configure MLflow in your Kedro project and it will be available as a Dagster resource.
• Hyperparameter tuning with Optuna: Integrate Optuna for distributed hyperparameter optimization via the optuna.StudyDataset Kedro dataset.

Quick Start

1. Install dependencies (using uv for reproducible environments):

uv sync
source .venv/bin/activate

1. Run Kedro pipelines as usual:

uv run kedro run --env <KEDRO_ENV>

Replace <KEDRO_ENV> with your target environment (e.g., local).

1. Explore pipelines in Dagster UI:

export KEDRO_ENV=local
kedro dagster dev

Your Kedro datasets appear as Dagster assets and pipelines as Dagster jobs.

Dagster Asset Lineage Graph generated from the example Kedro project.

Dagster Asset List generated from the example Kedro project.

---

Next Steps

• Explore the Technical Documentation for advanced configuration and customization.
• See the API Reference.