Feature Request: Add visualizer for tracking workflows and file inputs/outputs

[lbeckman314]

Overview

Currently there is no visualizer for the workflows submitted in Funnel. It could be beneficial to add a way to allow users to track:

• The history of workflows and whether they succeeded or failed
• Location of the workflow inputs and outputs
• The declarative workflow definition files themselves

Dagu

One possible option for implementing this feature would be to add the Golang program dagu as an extension to Funnel:

Dagu is a powerful Cron alternative that comes with a Web UI. It allows you to define dependencies between commands as a Directed Acyclic Graph (DAG) in a declarative YAML format. Dagu simplifies the management and execution of complex workflows. It natively supports running Docker containers, making HTTP requests, and executing commands over SSH.

• Repo: https://github.com/daguflow/dagu
• Docs: https://dagu.readthedocs.io/
• License: GPL-3.0 license

Airflow

Alternatively a more established program like Apache Airflow could be used instead:

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

• Repo: https://github.com/apache/incubator-airflow
• Docs: https://airflow.apache.org/
• License: Apache-2.0 license

Additional Resources

• awesome-workflow-engines: A curated list of awesome open source workflow engines