guide: using metrics/plots and params without pipelines

[tupui]

UPDATE: Related/part of #2815

My use case involves the inability to really use pipelines. I have models and data, but I am not (or cannot) do the training. So I don't have experiments per se. Still, I have metrics or other metadata I would want to track.

Apparently I can do this using commands like dvc metrics diff --targets metrics.json. I think this could be highlighted in the documentation that it is not required to have a dvc.yaml file. Although I would prefer that the cmd argument would be optional.

UPDATE: Jump to #2216 (comment)

I can do a PR with guidance about where I should put this if wanted. I thought about putting this in a use case (versioning) or in the user guide.

Reference to this proposal is here . cc @jorgeorpinel

[shcheklein]

Although I would prefer that the cmd argument would be optional.

cc @skshetry

[jorgeorpinel]

prefer that the cmd argument would be optional

Not sure having stages without an action would make sense. But how about accepting params, metrics, and plots in regular .dvc files? They already accept deps and outs which are the parents of the aforementioned fields.

[jorgeorpinel]

where I should put this if wanted

@tupui The command reference would be the first place that comes to mind.

Already mentioned in:

• https://dvc.org/doc/command-reference/params/diff
• https://dvc.org/doc/command-reference/metrics/diff
• https://dvc.org/doc/command-reference/plots/show
• https://dvc.org/doc/command-reference/plots/diff

Not mentioned yet in:

• https://dvc.org/doc/command-reference/params
• https://dvc.org/doc/command-reference/metrics
• https://dvc.org/doc/command-reference/metrics/show
• https://dvc.org/doc/command-reference/plots

Not supported in:

• https://dvc.org/doc/command-reference/plots/modify

One strategy would be to make sure the mentions are emphatic enough e.g. instead of parenthesis or grayed out notes, make a paragraph (starting with the 💡 emoji perhaps). Also review that it's mentioned in the --targets options for commands that have those.

Another idea would be to just leave minimal mentions (double check they're consistent), but emphasize in the Options section of each ref: for targets argument create an entry that fully describes the behavior and accepted values, like in https://dvc.org/doc/command-reference/repro#options. For --targets options similarly expand as much as needed.

[tupui]

Not sure having stages without an action would make sense. But how about accepting params, metrics, and plots in regular .dvc files? They already accept deps and outs which are the parents of the aforementioned fields.

But the value of the main yaml is that the information is centralised. If you just want to give general metadata, there should be an easy way to do so without needing to have stages or anything else. A top level option in the yaml would be better IMO. But maybe this is calling for a dedicated file as this yaml is move about pipelines.

[tupui]

One strategy would be to make sure the mentions are emphatic enough e.g. instead of parenthesis or grayed out notes, make a paragraph (starting with the 💡 emoji perhaps). Also review that it's mentioned in the --targets options for commands that have those.

Another idea would be to just leave minimal mentions (double check they're consistent), but emphasize in the Options section of each ref: for targets argument create an entry that fully describes the behavior and accepted values, like in https://dvc.org/doc/command-reference/repro#options. For --targets options similarly expand as much as needed.

Reading at the metrics one for instance, I think the explanation is OK. What I was missing is the fact that I could use all these function without the need to have the yaml file. So in Project structure that could maybe be highlighted. Currently it is saying:

dvc.yaml pipelines files define stages that form the pipeline(s) of a project. All stage-based features such as dvc params, dvc metrics, and dvc plots are specified here.

Which can lead to the conclusion that you have to use this yaml in order to use these functionalities.

[jorgeorpinel]

If you just want to give general metadata, there should be an easy way to do so without needing to have stages or anything else. A top level option in the yaml would be better IMO.

There's a separate discussion around that. See iterative/dvc#3936 (comment)
For now file metadata needs it's own .dvc file so I think that's where params/metrics support could be added. If/when tracking data gets incorporated to dvc.yaml that would carry over too.

BTW this part of the discussion should be in the iterative/dvc repo to be considered. Let's see if @skshetry has no objection on opening that issue (thanks).

[jorgeorpinel]

the metrics one for instance, I think the explanation is OK

Which one? It's not even mentioned in https://dvc.org/doc/command-reference/metrics or https://dvc.org/doc/command-reference/metrics/show.

in Project structure that could maybe be highlighted

Feel free to send a proposed change (via a PR). Thanks

[tupui]

the metrics one for instance, I think the explanation is OK

Which one? It's not even mentioned in https://dvc.org/doc/command-reference/metrics or https://dvc.org/doc/command-reference/metrics/show.

The one for diff. But true it is missing from the general metric discussion.

in Project structure that could maybe be highlighted

Feel free to send a proposed change (via a PR). Thanks

Ok I will try to propose something 😃

[skshetry]

Btw, .dvc file does support specifying metrics, params and plots. This is a pre-1.0 compatibility feature though.

We have also plans for supports different sections in dvc.yaml file. Eg:

metrics:
  - metrics.json
stages:
  ...

[jorgeorpinel]

Merged into #2815.

[jorgeorpinel]

#2572, rather.