Export to CSV for tasks profiling and stats

[marcocrasso]

Hi all,
I have been working on a tool that receives as input a workflow name, and a range of times in Epoch, to generate two CSV files, one with workflows execution details and another with their tasks details. For the last two months, this tool has been really really useful for our team to identify bottle necks in our pipelines. The output files can be easily massaged to feed R or old-good GNUplot.

The tool is written in Kotlin and uses Conductor client library (in particular WorkflowClient) to retrieve data from workflows (and their tasks) executed at a given range and which name and version match inputs. The tool support Conductor server behind basic auth proxies.

I've checked internally, and get permission to distribute it under Apache 2.0. I think this can be a good contribution to the "community projects". If that the case, please let me know how to procede, my guess is that a I should create a PR modifying README.md with a link to the repo of the tool, but you might want try it first.

Have a nice day, Marco

[aravindanr]

hello, thanks for choosing to share it with us. If you could point us to the repo hosting the tool and give us instructions to run it, that would be helpful. We would like to at least have a basic understanding of community created tools before adding them to the README. Thanks.

[marcocrasso]

Hi, sure, please find it at https://github.com/marcocrasso/conductor-exporter .

[marcocrasso]

Hi! during the weekend I refactored the component to support other exporters rather than only CSV. For example, I included one that summarises executed workflows in the given time range. Therefore, now it is named conductor-exporter, since it supports plug-in exporters through configuration and dependencies, which can target any format.

[marcocrasso]

@aravindanr have you get a chance to take a look? The instructions are in main README.md, but also here:

git clone [email protected]:marcocrasso/conductor-exporter.git
cd conductor-exporter
export CONDUCTOR_WORKFLOW_NAME=your_wf_name
export CONDUCTOR_WORKFLOW_VERSION=your_wf_version (Optional, default is 1)
export CONDUCTOR_URL=http://[user:password]@host:port/api/ (Don't forget the /api/ ) 
./gradlew bootRun --args="--startTime=1621518731077 --endTime=1621518838410"

endTime is optional, ignore it to use "now". The epoch dates are in your workflows instances. After editing the above instructions and running the tool you will have the new reports at conductor-exporter folder.

Please let me know whether I could help.