Plots: View tooltip with table of metric values across selected experiments

[aschuh-hf]

When hovering over a metric plot, I can see tooltips of the metric value for the curve under the cursor (though this takes some patience to get the cursor in the right place also due to lack of zoom, cf. #4530). What I often use and find quite useful with TensorBoard is that this would show me the metric values of all curves either at this or the latest available step (x coordinate). This is quite useful to compare the curves at different timepoints. Would be great if this was also supported by the DVC Plots towards replacing my use of TensorBoard.

Example screenshot from TensorBoard:

[mattseddon]

@daavoo @dberenbaum the on hover vertical line was dropped from dvc-render in iterative/dvc-render#99. Do you remember the reason for that change?

@aschuh-hf you should be able to achieve this by using a custom template. In order to achieve this add the template to your project and specify the custom template in your dvc.yaml. I've provided a quick example below. I am sure you'd be able to extend this to get the desired behaviour.

custom template

  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": { "values": "<DVC_METRIC_DATA>" },
  "title": "<DVC_METRIC_TITLE>",
  "width": 300,
  "height": 300,
  "layer": [
    {
      "encoding": {
        "x": {
          "field": "<DVC_METRIC_X>",
          "type": "quantitative",
          "title": "<DVC_METRIC_X_LABEL>"
        },
        "y": {
          "field": "<DVC_METRIC_Y>",
          "type": "quantitative",
          "title": "<DVC_METRIC_Y_LABEL>",
          "scale": { "zero": false }
        },
        "color": { "field": "rev", "type": "nominal" }
      },
      "layer": [
        { "mark": "line" },
        {
          "selection": {
            "label": {
              "type": "single",
              "nearest": true,
              "on": "mouseover",
              "encodings": ["x"],
              "empty": "none",
              "clear": "mouseout"
            }
          },
          "mark": "point",
          "encoding": {
            "opacity": {
              "condition": {
                "selection": "label",
                "value": 1
              },
              "value": 0
            }
          }
        }
      ]
    },
    {
      "transform": [{ "filter": { "selection": "label" } }],
      "layer": [
        {
          "mark": { "type": "rule", "color": "gray" },
          "encoding": {
            "x": {
              "field": "<DVC_METRIC_X>",
              "type": "quantitative"
            }
          }
        },
        {
          "encoding": {
            "text": {
              "type": "quantitative",
              "field": "<DVC_METRIC_Y>"
            },
            "x": {
              "field": "<DVC_METRIC_X>",
              "type": "quantitative"
            },
            "y": {
              "field": "<DVC_METRIC_Y>",
              "type": "quantitative"
            }
          },
          "layer": [
            {
              "mark": {
                "type": "text",
                "align": "left",
                "dx": 5,
                "dy": -5
              },
              "encoding": {
                "color": {
                  "type": "nominal",
                  "field": "rev"
                }
              }
            }
          ]
        }
      ]
    }
  ]
}

dvc.yaml entry

plots:
  - training/plots/metrics/train/acc.tsv:
      template: ./template.json
      x: step
      y: acc
      y_label: accuracy

plot

Screen.Recording.2023-08-21.at.11.22.13.am.mov

[aschuh-hf]

Now that flexibility of customizing the plots is pretty neat! I'll give it a try.

[shcheklein]

Related repo and an issue in the Vega repo vega/vega-tooltip#215. Overall we could try to utilize the vega-tooltip repo to do custom TB-like tooltip (which I agree are important). Most likely though we'll prioritize the migration / additional support for the Plotly engine (since Vega is too complicate to customize based on the user feedback and has some limitations in ML scenarios). So, let's us get back to this issue a bit later please @aschuh-hf as soon as we migrate to Plotly.

[dberenbaum]

AFAICT from the PR, it was an attempt to copy tensorboard's ability to show multiple values per point, but it looks like it came at the expense of showing multiple points at once. From a quick look I'm not sure how easy it is to do both simultaneously even in plotly.

[aschuh-hf]

Most likely though we'll prioritize the migration / additional support for the Plotly engine (since Vega is too complicate to customize based on the user feedback and has some limitations in ML scenarios). So, let's us get back to this issue a bit later please @aschuh-hf as soon as we migrate to Plotly.

Sure, I can work with the template @mattseddon provided for now (and anyway often still use TensorBoard in parallel).

I also find Vega (or Lite), despite how powerful it may be quite difficult to get used to. So far I stayed away from it. I found https://vega.github.io/vega-lite/examples/interactive_multi_line_pivot_tooltip.html to be closest to what I would like, though I wasn't able to figure out how to have the tooltip list populated by the experiment revs rather than a hard-coded list of experiments as in the example.

show multiple values per point, but it looks like it came at the expense of showing multiple points at once

Indeed, I was wondering if that was possible. Maybe it's not the highest priority to have that ability, though I did find it quite useful to have the last available metric for any curve prior to the selected timepoint listed alongside others. One might interrupt training runs because its clear they won't perform as expected, especially in case of a hyper-parameter search (e.g., using Ray Tune). Or during live training runs compare the current metric with later times of other curves.

[shcheklein]

@aschuh-hf added to the same task list / requirements iterative/dvc-render#7

[mattseddon]

WDYT about the tooltip shown here:

Screen.Recording.2023-09-07.at.3.33.40.pm.mov

[aschuh-hf]

This looks great!

[aschuh-hf]

Is it correct, though, that similar to the template I found in the Vega-Lite examples, this requires to be able to set the list of revs in the template. Is there a <DVC_*> variable that can be used for this?

I.e., to replace this hard coded list of experiment revs in your example:

"encoding": {
    "x": {"field": "step", "type": "quantitative", "title": "step"},
    "color": {
      "legend": {"disable": false},
      "scale": {
        "domain": ["pagan-puns", "soppy-debs", "workspace"],
        "range": ["#945dd6", "#13adc7", "#f46837"]
      }
    },
    "strokeDash": {
      "field": "filename",
      "scale": {
        "domain": ["test/acc.tsv", "train/acc.tsv"],
        "range": [[1, 0], [8, 8]]
      },
      "legend": {
        "disable": false,
        "symbolFillColor": "transparent",
        "symbolStrokeColor": "grey"
      }
    },

I tried to adapt this to a template, and replaced the color specification above by

"color": {
    "field": "rev",
    "type": "nominal"
}

but I only get a single row with "undefined" in the tooltip. Also the numerical value displayed doesn't match the y axis / expected value [EDIT: I just realized the value shown is the sum of the values across selected points]. Could you help me adapt the template to get the result you show in your example? Thanks!

Adapted metrics plot template

{
    "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
    "data": {
        "values": "<DVC_METRIC_DATA>"
    },
    "title": "<DVC_METRIC_TITLE>",
    "width": "container",
    "height": "container",
    "params": [
        {
            "name": "smooth",
            "value": 0.001,
            "bind": {"input": "range", "min": 0.001, "max": 1, "step": 0.001}
        }
    ],
    "layer": [
        {
            "transform": [
                {
                    "pivot": "rev-filename",
                    "value": "<DVC_METRIC_Y>",
                    "groupby": ["<DVC_METRIC_X>"]
                }
            ],
            "mark": {
                "type": "rule",
                "tooltip": {
                    "content": "data"
                }
            },
            "encoding": {
                "opacity": {
                    "condition": {
                        "value": 0.3,
                        "param": "hover",
                        "empty": false
                    },
                    "value": 0
                }
            },
            "params": [
                {
                    "name": "hover",
                    "select": {
                        "type": "point",
                        "fields": [
                            "<DVC_METRIC_X>"
                        ],
                        "nearest": true,
                        "on": "mouseover",
                        "clear": "mouseout"
                    }
                }
            ]
        },
        {
            "encoding": {
                "y": {
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            },
            "layer": [
                {
                    "mark": "line"
                },
                {
                    "transform": [
                        {
                            "filter": {
                                "param": "hover",
                                "empty": false
                            }
                        }
                    ],
                    "mark": "point"
                }
            ],
            "transform": [
                {
                    "loess": "<DVC_METRIC_Y>",
                    "on": "<DVC_METRIC_X>",
                    "groupby": [
                        "rev",
                        "filename",
                        "rev-filename",
                        "field",
                        "filename::field"
                    ],
                    "bandwidth": {"signal": "smooth"}
                }
            ]
        },
        {
            "mark": {
                "type": "line",
                "opacity": 0.2
            },
            "encoding": {
                "x": {
                    "field": "<DVC_METRIC_X>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_X_LABEL>"
                },
                "y": {
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            }
        },
        {
            "mark": {
                "type": "circle",
                "size": 10
            },
            "encoding": {
                "x": {
                    "aggregate": "max",
                    "field": "<DVC_METRIC_X>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_X_LABEL>"
                },
                "y": {
                    "aggregate": {
                        "argmax": "<DVC_METRIC_X>"
                    },
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            }
        }
    ],
    "encoding": {
        "x": {
            "field": "<DVC_METRIC_X>",
            "type": "quantitative",
            "title": "<DVC_METRIC_X_LABEL>"
        },
        "color": {
            "field": "rev",
            "type": "nominal",
            "legend": {
                "disable": false
            }
        },
        "shape": {
            "legend": {
                "disable": false,
                "symbolFillColor": "grey"
            }
        }
    }
}

[aschuh-hf]

Also note that with the template above, when I click on a plot to show the detail and test the new zoom and pan (which wasn't working for me with the previous template yet using v1.0.52 of the DVC extension), no plot is shown, just an empty canvas. Certainly because there's something off with my attempt to adjust the template.

[aschuh-hf]

I checked the output index.html of dvc plots show / dvc plots diff when using the --open flag and realized that rev-filename as used in the example is not a data field of <DVC_METRIC_DATA>. So I replaced it with rev. The tooltip works when I view this index HTML page in the browser (there is an odd undefined rev, but that's likely a different issue).

I wasn't yet able to test it in the extension because dvc still uses the wrong template (not the one in the current modified dvc.yaml in the workspace). For the other dvc plots commands, I used the --template option to force a specific one I wanted to test.

Plot template

{
    "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
    "data": {
        "values": "<DVC_METRIC_DATA>"
    },
    "title": "<DVC_METRIC_TITLE>",
    "width": 800,
    "height": 600,
    "params": [
        {
            "name": "smooth",
            "value": 0.001,
            "bind": {"input": "range", "min": 0.001, "max": 1, "step": 0.001}
        }
    ],
    "layer": [
        {
            "transform": [
                {
                    "pivot": "rev",
                    "value": "<DVC_METRIC_Y>",
                    "groupby": ["<DVC_METRIC_X>"]
                }
            ],
            "mark": {
                "type": "rule",
                "tooltip": {
                    "content": "data"
                }
            },
            "encoding": {
                "opacity": {
                    "condition": {
                        "value": 0.3,
                        "param": "hover",
                        "empty": false
                    },
                    "value": 0
                }
            },
            "params": [
                {
                    "name": "hover",
                    "select": {
                        "type": "point",
                        "fields": [
                            "<DVC_METRIC_X>"
                        ],
                        "nearest": true,
                        "on": "mouseover",
                        "clear": "mouseout"
                    }
                }
            ]
        },
        {
            "encoding": {
                "y": {
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            },
            "layer": [
                {
                    "mark": "line"
                },
                {
                    "transform": [
                        {
                            "filter": {
                                "param": "hover",
                                "empty": false
                            }
                        }
                    ],
                    "mark": "point"
                }
            ],
            "transform": [
                {
                    "loess": "<DVC_METRIC_Y>",
                    "on": "<DVC_METRIC_X>",
                    "groupby": [
                        "rev",
                        "filename",
                        "field",
                        "filename::field"
                    ],
                    "bandwidth": {"signal": "smooth"}
                }
            ]
        },
        {
            "mark": {
                "type": "line",
                "opacity": 0.2
            },
            "encoding": {
                "x": {
                    "field": "<DVC_METRIC_X>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_X_LABEL>"
                },
                "y": {
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            }
        },
        {
            "mark": {
                "type": "circle",
                "size": 10
            },
            "encoding": {
                "x": {
                    "aggregate": "max",
                    "field": "<DVC_METRIC_X>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_X_LABEL>"
                },
                "y": {
                    "aggregate": {
                        "argmax": "<DVC_METRIC_X>"
                    },
                    "field": "<DVC_METRIC_Y>",
                    "type": "quantitative",
                    "title": "<DVC_METRIC_Y_LABEL>",
                    "scale": {
                        "zero": false
                    }
                },
                "color": {
                    "field": "rev",
                    "type": "nominal"
                }
            }
        }
    ],
    "encoding": {
        "x": {
            "field": "<DVC_METRIC_X>",
            "type": "quantitative",
            "title": "<DVC_METRIC_X_LABEL>"
        },
        "color": {
            "field": "rev",
            "type": "nominal",
            "legend": {
                "disable": false
            }
        },
        "shape": {
            "legend": {
                "disable": false,
                "symbolFillColor": "grey"
            }
        }
    }
}

[mattseddon]

I'm sorry, the linked template was not designed to be reused or adapted. I was looking for feedback on the appearance/contents of the tooltip.

Seeing as you are only using rev and not filename or field in the template this one should work for you:

Custom template

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {
    "values": "<DVC_METRIC_DATA>"
  },
  "title": "<DVC_METRIC_TITLE>",
  "width": "container",
  "height": "container",
  "params": [
    {
      "name": "smooth",
      "value": 0.001,
      "bind": {
        "input": "range",
        "min": 0.001,
        "max": 1,
        "<DVC_METRIC_X>": 0.001
      }
    }
  ],
  "layer": [
    {
      "encoding": {
        "y": {
          "field": "<DVC_METRIC_Y>",
          "type": "quantitative",
          "title": "<DVC_METRIC_Y_LABEL>",
          "scale": {
            "zero": false
          }
        },
        "color": {
          "field": "rev",
          "type": "nominal"
        }
      },
      "layer": [
        { "mark": "line" },
        {
          "transform": [{ "filter": { "param": "hover", "empty": false } }],
          "mark": "point"
        }
      ],
      "transform": [
        {
          "loess": "<DVC_METRIC_Y>",
          "on": "<DVC_METRIC_X>",
          "groupby": ["rev", "filename", "field", "filename::field"],
          "bandwidth": {
            "signal": "smooth"
          }
        }
      ]
    },
    {
      "params": [{ "bind": "scales", "name": "grid", "select": "interval" }],
      "mark": {
        "type": "line",
        "opacity": 0.2
      },
      "encoding": {
        "x": {
          "field": "<DVC_METRIC_X>",
          "type": "quantitative",
          "title": "<DVC_METRIC_X_LABEL>"
        },
        "y": {
          "field": "<DVC_METRIC_Y>",
          "type": "quantitative",
          "title": "<DVC_METRIC_Y_LABEL>",
          "scale": {
            "zero": false
          }
        },
        "color": {
          "field": "rev",
          "type": "nominal"
        }
      }
    },
    {
      "mark": {
        "type": "circle",
        "size": 10,
        "tooltip": {
          "content": "encoding"
        }
      },
      "encoding": {
        "x": {
          "aggregate": "max",
          "field": "<DVC_METRIC_X>",
          "type": "quantitative",
          "title": "<DVC_METRIC_X_LABEL>"
        },
        "y": {
          "aggregate": {
            "argmax": "step"
          },
          "field": "<DVC_METRIC_Y>",
          "type": "quantitative",
          "title": "<DVC_METRIC_Y_LABEL>",
          "scale": {
            "zero": false
          }
        },
        "color": {
          "field": "rev",
          "type": "nominal"
        }
      }
    },
    {
      "transform": [
        {
          "pivot": "rev",
          "value": "acc",
          "groupby": ["<DVC_METRIC_X>"]
        }
      ],
      "mark": { "type": "rule", "tooltip": { "content": "data" } },
      "encoding": {
        "opacity": {
          "condition": { "value": 0.3, "param": "hover", "empty": false },
          "value": 0
        }
      },
      "params": [
        {
          "name": "hover",
          "select": {
            "type": "point",
            "fields": ["<DVC_METRIC_X>"],
            "nearest": true,
            "on": "mouseover",
            "clear": "mouseout"
          }
        }
      ]
    }
  ],
  "encoding": {
    "x": {
      "field": "<DVC_METRIC_X>",
      "type": "quantitative",
      "title": "<DVC_METRIC_X_LABEL>"
    }
  }
}

☝🏻 the above template should have zoom on scroll and the previously shown tooltips. Something similar should be the default template soon.

I wasn't yet able to test it in the extension because dvc still uses the wrong template (not the one in the current modified dvc.yaml in the workspace)

The only way to force DVC to use the updated template is to select the workspace as a revision for plotting.

[aschuh-hf]

Thanks, @mattseddon. I just got around to update to this new template and all is working fine! Loving it.

[aschuh-hf]

Just FYI, I had to update one occurrence of acc in the provided template by <DVC_METRIC_Y>:

"transform": [
                {
                    "pivot": "rev",
                    "value": "<DVC_METRIC_Y>",
                    "groupby": [
                        "<DVC_METRIC_X>"
                    ]
                }
            ],