Dagster × Modal Demo

Orchestrate your flexible compute workloads with the combined forces of Dagster and Modal.

In this example project we show how you can write a pipeline that automatically detects newly published podcasts, transcribes them using the power of GPUs, and notifies you with the summary so that you can focus on the things that truly matter (not listening to people talk into fancy microphones).

Architecture

Example Asset Lineage

Getting started

Install the project dependencies:

pip install -e ".[dev]"

Run Dagster:

dagster dev

Open http://localhost:3000 in your browser.

References

Modal

• Modal Docs
• Modal Docs: Parallel podcast transcription using Whisper
• Modal Docs: Mounting Cloudflare R2 Buckets

Dagster

• Dagster Docs
• Dagster Docs: Pipes Subprocess Reference
• Dagster Docs: OpenAI Integration
• Dagster Docs: AWS Integration

OpenAI

• OpenAI Whisper
• OpenAI Cookbook: Summarizing Long Documents

Miscellaneous

• yagmail for Sending Email from Python

More Screenshots

Screenshot of R2 Storage

Screenshot of Summary Asset

Example Podcast Feeds

Podcasts are sourced from RSS feeds as; a table containing a sample of feeds is below:

Title	RSS Feed
Practical AI: Machine Learning, Data Science, LLM	RSS Feed
The Data Exchange with Ben Lorica	RSS Feed
Hub & Spoken: Data	RSS Feed
Making Data Simple	RSS Feed
The Data Chief	RSS Feed