Prefect

Prefect is an orchestrator for data-intensive workflows. It's the simplest way to transform any Python function into a unit of work that can be observed and orchestrated. With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes. With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more. Every activity is tracked and can be monitored with the Prefect server or Prefect Cloud dashboard.

from prefect import flow, task
from typing import List
import httpx


@task(retries=3)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
    for repo in repos:
        get_stars(repo)


# run the flow!
github_stars(["PrefectHQ/Prefect"])

After running some flows, fire up the Prefect UI to see what happened:

prefect server start

From here, you can continue to use Prefect interactively or deploy your flows to remote environments, running on a scheduled or event-driven basis.

Getting Started

Prefect requires Python 3.8 or later. To install Prefect, run the following command in a shell or terminal session:

pip install prefect

Start by then exploring the core concepts of Prefect workflows, then follow one of our friendly tutorials to learn by example.

Join the community

Prefect is made possible by the fastest growing community of thousands of friendly data engineers. Join us in building a new kind of workflow system. The Prefect Slack community is a fantastic place to learn more about Prefect, ask questions, or get help with workflow design. The Prefect Discourse is a community-driven knowledge base to find answers to your Prefect-related questions. All community forums, including code contributions, issue discussions, and slack messages are subject to our Code of Conduct.

Contribute

See our documentation on contributing to Prefect.

Thanks for being part of the mission to build a new kind of workflow system and, of course, happy engineering!