Consider unifying config.yml across steps

[imagejan]

Having a single config.yml for all steps of a workflow would have several benefits:

• Avoid redundant definition of common arguments (e.g., zarr plate directory, project folder, etc.).
• Reuse the config file for possible (prefect, nextflow, ...) workflows/pipelines built across all steps.

[tibuch]

Having only a single config file is dangerous in combination with nextflow. Nextlfow checks, among other things, the last modification date of the config file to determine if it can re-use cached results. This can lead to a situation where you would only want to re-run the last step of a workflow with a different input parameter, but nextflow will start from scratch because all steps are invalid with the changed config file.

However, I agree that we should carefully think about config files and probably have multiple ones, but without parameter repetition. For example we could think of having a dataset-config.yaml which contains all information about the dataset and for each processing step we have a separate config containing the respective parameters and a link to the dataset-config.yaml.

When we create these config files we should ensure that users only have to input each parameter once and that we move the parameters to the correct config file.