Render stored Matplotlib png output as a fallback when opening a notebook

[asteppke]

Actual behaviour

With the current ipympl and using %matplotlib widget as a backend VS code can render this example without issues:

%matplotlib widget

from matplotlib import pyplot as plt

fig, ax = plt.subplots()
ax.plot([0,1,2], [2,3,4] ,"o-")

Though, when saving and reloading the notebook file no image is shown (although a static image is available):

Expected behaviour

The small </> icon on the left allows to change the mimetype representation. When this is changed manually to png the alternative widget representation that was previously saved is shown correctly:

So VS code does not recognize that without a running/current widget available that the png static representation should be shown instead.

Environment data

• VS Code version: 1.63.2
• Jupyter Extension version v2021.11.1001550889
• Python Extension version v2021.12.1559732655
• OS (Windows | Mac | Linux distro) and version: Windows 10
• Python and/or Anaconda version: 3.9.9
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Local

[martinRenou]

See jupyter/notebook#6181 for the approach we took in Jupyter Notebook: do a try/catch of the mimetype rendering logic, so that if the mimetype renderer fails to render, we fallback to another available mimetype.

[rchiodo]

Thanks for the bug. VS code doesn't allow for a fallback mimetype. On reload it just uses the widget data as that's what was selected before.

We don't support reloading widgets at the moment (see this bug for more info: #5424 (comment))

[DonJayamanne]

Solution

• When loading nb without widget state give pref to other mime types
• When executing a cell, use current mime type priority setting

[DonJayamanne]

Summary of discussion with @mjbvz

• Todo for @mjbvz
  • Provide some kind of a way for render to inform VS Code whether rendering failed/not possible.
  • E.g. have renderOutputItem return a promise
  • Return value will be a promise so that the renderer can wait till all scripts are loaded,
  • One of the arguments would be a cancellation token to ensure we abort unnecessary processing.

[DonJayamanne]

@mjbvz I can see another case for fallbacks:

• Run the following cell in Jupyter

from vega import VegaLite 
from IPython.display import display 
display({
	"application/vnd.vegalite.v4+json": {
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "A simple bar chart with embedded data.",
  "data": {
    "values": [
      {"a": "A", "b": 28}, {"a": "B", "b": 55}, {"a": "C", "b": 43},
      {"a": "D", "b": 91}, {"a": "E", "b": 81}, {"a": "F", "b": 53},
      {"a": "G", "b": 19}, {"a": "H", "b": 87}, {"a": "I", "b": 52}
    ]
  },
  "mark": "bar",
  "encoding": {
    "x": {"field": "a", "type": "nominal", "axis": {"labelAngle": 0}},
    "y": {"field": "b", "type": "quantitative"}
  }
}}, raw=True)

• We will get one output with two mime types, application/vnd.vegalite.v4+json and image/png. The preference is given to the complex mime, however if there's no renderer for that then the fallback is the image.
  E.g. if you were to open the above ipynb on github, then you get to see the image. This is what a user would expect, as the vega mime renderer isn't supported on github and there's a simple image mime type that can be easily displayed.

TLDR:

I believe we (VS Code) should also do the same thing, instead of displaying a message saying no renderer could be found.... search marketplace we should fall back to a mime type thats supported, so users can see the output.

[mjbvz]

@rebornix To support this, we need to transfer the entire output to the webview. Today I believe we just send over the output for the selected item mime type

That would allow us to fallback to a different renderer if we can't find one for the current mime type

Let's look into this for October

[DonJayamanne]

Scenraios

1. Sharing Notebook

• User executes a notebook with a multi-mime output containing ipywidget output & HTML
• IPyWidget is displayed
• User shares this notebook with someone else and they don't see anything

2. Opening existing notebook

• User executes a notebook with a multi-mime output containing ipywidget output & HTML
• IPyWidget is displayed
• Users closes the notebook and re-opens it & nothing is displayed

Expectation

If IPYWidget otuput cannot be displayed, then display the HTML or next available mime

Problems encountered

For scenario 1, things work most of the time with the new API provided by Matt. However in Scenario 2 it will fail to work 100% of the time.
This is because of the way IPyWidgets are rendered, here's a summary of how Widgets are rendered:

• Cell is executed and we get an IPyWidget output in a message message
• IPyWidget output is added to the Cell Output (the only content in the cell output is some JSON with an IPyWidget Id)
• The renderer then sees this Id and asks the Widget manager to render the IPyWidget output
• Note: Widget manager is listening to messages from the Kernel
• At this point the widget manager now waits for the ipywidget data to become available, and once it gets that data the widget is rendered
• I.e. the widget manager will wait indefinitely for the widget data to arrive

In the case of Scenario 2, when the notebook is re-opened

• We auto start kernels in the background (this is done to ensure we pre-warm kernels for a speedy startup of kernels)
• As a result the Widget manager is now hooked up on the renderer side and we have a like kernel connection
• Bearing this in mind, when the user opens an existing notebook, the ipywidget will be sent to the widget manager for rendering, and widget manager waits indefinitely for the widget data to arrive.
• However since this widget was never generated in the new session, no data will arrive, hence nothing will get renderered

Solution

We need to check whether the IPyWidget Id belongs to the active kernel session or not, and if it does not, then assume that it will not get rendered.
Couple of approaches

• Monitor the kernle messages and keep track of the widget ids generated
• Query the kernel for widget state (careful, this could result in a dead lock where widget manager is waiting for output to be rendered and then we send a request and wait for that message which now blocks the renderer)