-
Notifications
You must be signed in to change notification settings - Fork 386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ref: clarify about types of metrics/plots #2956
Comments
Maybe we need to put the types of metrics in a proper metrics/plots guide and not explain the concepts in the ref... @dberenbaum do you think we the core team should address this for now, or plan for a guide? Thanks |
👍 I do think the current plots command ref doesn't incorporate image support enough at the moment. It still refers mostly to "plots metrics," and I don't think the "Types of Metrics" section at the top makes much sense now. Also, it doesn't look like it's addressed in any of |
When the time comes, maybe we need some visual aid to explain how we handle different metrics?
|
I like the tree, but do we consider image plots as metrics at all? They could be actual images, like computer vision examples. I'm not sure it's intuitive to think of an image file as a type of metric. |
Also... parallel coordinates plots 🤔 (would be part of |
I think this is the key question. By "plots" we refer both to metrics that come in "data series" form (term "data" here is also not ideal) which can be visualized by Conceptually I'd organize it like: metrics
Separately,
|
Since we are still clarifying the types of metrics and plots and we already have the discussion here, why don't we decide how to describe them here and then decide what we need to edit in help text, command reference, user guide, etc.? I would only describe scalars as metrics because:
I don't think we should describe any kind of plots as metrics. |
OK in this case we mainly have to edit the command refs. for both metrics and plots, and unlink them (don't call plots metrics). Then we need to incorporate this approach to #2925 (Get Started) Cc @iesahin Finally, to consider it as well for #2572 (User Guide) |
I have separated plots from metrics/params to describe them more independently in #3050 A visualization section can describe plots as CSV files or plots as images, as you see fit. |
Should we expect a PR from the core team on this? Or do we consider it docs work as the feature is already documented (in a way we don't love) @dberenbaum |
We need to actually update some of the help text in the core repo, and then to update the command ref. So at least some of it needs to happen from the core team, and then we can see whether it makes sense to have the docs team take over some of it. @pared Do you want me to create an issue and/or PR in the core repo, or do you want to do it yourself? |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Waiting on iterative/dvc#7086 |
Migrating main points from #1442:
|
@dberenbaum @pared it's not obvious how that issue relates to the terminology discussion here. Could someone briefly explain the plan? Thanks |
BTW the guide part should be covered in #2572 so we can limit this issue to the cmd ref. indeed + maybe mentions in the Get Started if needed. |
I thought the consensus from the discussion above was to cleanly separate these concepts. Mixing all metrics and plots functions into one command looks like the opposite direction to me.
As mentioned in #2956 (comment), there is some help text to update, but otherwise I'm not sure how it's a core discussion? It seems the main action point is to separate metrics and plots in the docs. |
Oh. No, I don't necessarily agree that the solution is to separate the concepts in docs because plots are conceptually types of metrics (general meaning of the word "metrics"). Again thinking of the example of an AUC-ROC curve: These are clearly modeling metrics (right?). But if that's what you decide then sure, we can just call |
My thoughts were drifting in a similar direction: I was actually thinking about refactoring |
Maybe I'm odd, but I wouldn't call an AUC-ROC curve or anything other than a scalar value a metric 🤷 . Edit: I would call the AUC-ROC scalar value a metric, but I wouldn't describe the ROC curve as a metric.
Sure. We can discuss further if needed once we have a PR to review. |
Ah then indeed we should try a more general term like "performance measure" to separate the terminology 🙂 I updated the labels as this may involve more than just the cmd ref but no need to work on anything outside of small/strategic word changes in the cmd ref on your end @pared (we'll take it over from there). Thanks! |
I don't see a big problem either, though I'm biased already. I remember initially I also had a lot of doubts. I think partially this confusion comes from the fact that in loggers like W&B you would log metrics and then plot them online over time. Plots is something that is built on top of data that you log, including scalars, etc. But they also have a way to log plots - https://docs.wandb.ai/guides/track/log/plots . Even though in the description they would say: "These simple charts make it easy to construct basic visualizations of metrics and results.". Not sure if that affects how this data is stored internally though or it's just a bunch of convenience functions. And we mix a bit files with data (scalar or not) with plots (spec of a graphical representation). It means that we can't make a plot on top of metrics files? To show its trend for example? Or a plot that takes data from multiple outputs. Etc. It might make sense to detach plot specs from the outputs. |
@shcheklein |
That's the connection between metrics and plots I had on the back of my mind as well, but not something DVC does at the moment. |
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
I think this will no longer be a concern after the changes in #3691 are merged. |
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
Related: iterative/dvc#7477 Related: iterative#2956
* cmd-ref: plots: flexible plots docs Related: iterative/dvc#7477 Related: #2956 * Update content/docs/user-guide/project-structure/dvcyaml-files.md * Apply suggestions from code review * Update content/docs/command-reference/plots/index.md * plots: examples: move to subcommands * plots: refactor top-level plots definition * plots: review refactor * Update content/docs/command-reference/plots/index.md * ref: fix a link (2/2) per #3691 (review) * ref: remove concept of type of metrics * ref: term "plots files" (consistency) * ref: wrap `plots index` usage block * plots: top-level plots edits * plots: improve motivation for top-level plots per #3691 (review) * ref: edit `plots show` desc * ref: return plot template examples from `plots show` to index * guide: move top-lv plot mention from stage entry to desc * ref: clean up new `plots show` examples * ref: more copy edits around plots * Update content/docs/user-guide/project-structure/dvcyaml-files.md * Update content/docs/command-reference/plots/index.md * Update content/docs/command-reference/plots/index.md * Update content/docs/command-reference/plots/index.md * Restyled by prettier Co-authored-by: Paweł Redzyński <[email protected]> Co-authored-by: Jorge Orpinel <[email protected]> Co-authored-by: Jorge Orpinel Perez <[email protected]> Co-authored-by: Dave Berenbaum <[email protected]> Co-authored-by: Restyled.io <[email protected]> Co-authored-by: Dave Berenbaum <[email protected]>
Both the metrics and plots cmd refs. explain 2 types of "metrics" (metrics and plots) -- done to to keep both references connected.
Calling plots a kind of metric is already confusing. Furthermore, in #2839 the concept of "2 types of plots files" was introduced (data series rendered with Vega vs. arbitrary image files) to the latter, which may be confusing.
Also, the plots usage block needs wrapping to avoid scroll
UPDATE: Jump to #2956 (comment)
The text was updated successfully, but these errors were encountered: