Support two-way communication in ipywidgets #3276

seeM · 2024-05-27T13:55:25Z

Currently, if you create a widget with ipywidgets:

import ipywidgets as widgets
a = widgets.FloatSlider()
display(a)

And adjust the slider, the state is not updated in the kernel:

print(a.value)
# 0

Similarly, if you set the state in the kernel, the widget view doesn't update:

a.value = 100
# Nothing happens in the frontend

This is because we're not handling messages over the widget comm between its frontend and backend.

A few more tiny steps toward #3276 trying to avoid One Massive PR. Commits are self-contained: 1. Include `metadata` in language runtime messages. IPyWidgets JS libs assume comm open messages include `metadata['version']` specifying the widget protocol version. 2. Add optional `id` param to `session.createClient`. We'll need to create comms with a given UUID. 3. Make runtime client `sendMessage` method public. We'll need to send fire-and-forget messages to widget comms. 4. Allow creating `jupyter.widget.control` comms. We'll use them to fetch widget state from the kernel. ### QA Notes No functional changes. Nothing should break!

A few of the notebook preloads and notebook renderers contributed by the `ms-toolsai.jupyter` and `ms-toolsai.jupyter-renderers` extensions are incompatible with Positron's language runtimes. This PR stops them from being contributed. Another step toward #3276. In a follow-up PR, I'll contribute our own preloads/renderers. Note that the blocked preloads/renderers didn't work in Positron anyway, so this PR should have no effect. #### QA Notes - IPyWidgets should work as before with the same limitations (no communication with runtimes) - R HTMLWidgets should continue to work - Python Plotly widgets should continue to work - Notebooks should be able to render most output types, as before

Another step toward #3276. This PR moves the rendering of our notebook output webview service to inside the webview, giving the notebook renderer access to features which previously did not work, including (from the code): > Some features known to be NYI: > - Message passing between the renderer and the host (RenderContext) > - Extending another renderer (RenderContext) > - State management (RenderContext) > - Raw Uint8Array data and blobs We do this by listening for a new `positronRender` message inside the webview. I also changed our `render_complete` message to `positronRenderComplete` and added it to the known message types. I also made some changes to how we determine the required preloads and resource roots to bring it more in line with the backlayer webview: 1. **Preloads:** We get preloads by view type instead of by extension. 2. **Resource roots:** 1. Removed the renderer's extension folder. 2. Added each renderer's parent folder. 3. Added each preload's parent folder. 4. Added each preload's specified `localResourceRoots`. ### QA Notes This should not break any existing widgets/plots/HTML outputs. Here are some tests for different code paths: 1. R html content: `gt::gt(mtcars)` 2. Python notebook renderer: `import plotly.express as px; px.bar(x=["a", "b", "c"], y=[1, 3, 2])` 3. R notebook renderer: `library(plotly); plot_ly(data = iris, x = ~Sepal.Length, y = ~Petal.Length)` 4. Python custom ipywidgets webview: `from ipyleaflet import Map; display(Map(center=(52.204793, 360.121558), zoom=12))`

This PR adds support for fully interactive IPyWidgets, addressing #3276. For example, create and display a widget: ```python import ipywidgets w = ipywidgets.IntSlider() display(w) ``` Interact with the widget in the UI, then check its value in the console: ```python w.value ``` Set the value in the console: ```python w.value = 13 ``` Note the updated UI. --- We've already been using VSCode notebook renderers to render interactive plots in the Plots pane. This PR adds a notebook renderer for IPyWidgets, and routes messages between the renderer (widget frontend) and the runtime (widget backend) via the updated Positron IPyWidgets service. ### Unit Tests The new implementation of the IPyWidgets service is unit tested. This PR adds a few mocks to make that possible. Most useful are `TestLanguageRuntimeSession ` and `TestRuntimeClientInstance`, although they're only implemented as far as needed for this PR. I used the convention of starting Positron test suites with `Positron - `. There's also a new script to run only Positron's unit tests excluding VSCode's: ```sh ./scripts/test-positron.sh ``` Note that this script is not included in our CI since we already run `./scripts/test.sh` which is a superset. ### QA Notes These widgets/libraries should work: * All of `ipywidgets` except `Output` and `Image` * `ipytree` * `ipyleaflet` * `anywidget` * `drawdata` * `pythreejs` These widgets/libraries will not yet work: * `bqplot` * `ipycanvas` * `ipydatagrid` * `ipympl` Most of the remaining widgets/libraries require us to support `buffers` in the Jupyter messaging protocol (#3974) which I intend to work on next.

seeM · 2024-07-31T13:04:07Z

For verification, the following widget libraries should work in the latest release:

ipywidgets – all widget types except Output and Image
ipytree
ipyleaflet
anywidget
drawdata
pythreejs

For all of these libraries, you should be able to manipulate the widget frontend and see some state on the widget object change (it will not automatically update in the variables pane). You should also be able to change some attribute on the widget object and see the frontend change.

These widgets libraries will not yet work:

bqplot
ipycanvas
ipydatagrid
ipympl

testlabauto · 2024-08-01T18:29:12Z

Seeing
Error loading renderer 'positron-ipywidgets'
Failed to fetch dynamically imported module: https://file+.vscode-resource.vscode-cdn.net/Applications/Positron.app/Contents/Resources/app/extensions/positron-ipywidgets/renderer/dist/index.js

Note that the t/Applications/Positron.app/Contents/Resources/app/extensions/positron-ipywidgets/renderer/dist dir doesn't exist.

Addresses the issue in #3276 (comment). I confirmed that this fixed the issue by building a macOS arm64 release locally.

testlabauto · 2024-08-07T19:18:24Z

Sorry to report that a similar error is still appearing:

Error loading renderer 'positron-ipywidgets'
Failed to fetch dynamically imported module: https://file+.vscode-resource.vscode-cdn.net/Applications/Positron.app/Contents/Resources/app/extensions/positron-ipywidgets/renderer/media/index.js

If I check the filesystem:

ls /Applications/Positron.app/Contents/Resources/app/extensions/positron-ipywidgets/renderer/
package.json  src           tsconfig.json yarn.lock

testlabauto · 2024-08-09T16:00:43Z

Verified Fixed

Positron Version(s) : 2024.08.0-24
OS Version          : OSX

Test scenario(s)

import ipywidgets as widgets
... a = widgets.FloatSlider()
... display(a)

Manually move slider

print(a.value)
62.7

Link(s) to TestRail test cases run or created:
N/A

manzt · 2024-08-09T20:29:24Z

Sweet!!! Thanks so much for implementing this. Can confirm anywidget is working:

seeM added enhancement New feature or request lang: python labels May 27, 2024

seeM self-assigned this May 27, 2024

seeM mentioned this issue May 27, 2024

Allow passing metadata on client creation for ipywidgets support [Merged to incorrect base] #3277

Merged

juliasilge added this to the Future milestone May 28, 2024

seeM mentioned this issue May 29, 2024

Allow passing metadata on client creation for ipywidgets support #3297

Merged

seeM mentioned this issue Jul 11, 2024

Language runtime changes required for IPyWidgets #3966

Merged

This was referenced Jul 11, 2024

Use the correct notebook cell output item type for widgets #3973

Merged

Block incompatible notebook preloads and renderers #3975

Merged

seeM mentioned this issue Jul 17, 2024

Provide context to notebook output webview renderers #4050

Merged

petetronic modified the milestones: Future, Release Candidate Jul 22, 2024

seeM modified the milestones: Release Candidate, 2024.09.0 Pre-Release Jul 29, 2024

seeM added theme: app builder area: widgets Issues related to Widgets. labels Jul 29, 2024

seeM mentioned this issue Jul 29, 2024

Fully interactive IPyWidgets #4172

Merged

seeM mentioned this issue Aug 1, 2024

Fix renderer bundle being excluded from builds #4211

Merged

seeM added a commit that referenced this issue Aug 1, 2024

Fix renderer bundle being excluded from builds (#4211)

6d96097

Addresses the issue in #3276 (comment). I confirmed that this fixed the issue by building a macOS arm64 release locally.

testlabauto closed this as completed Aug 9, 2024

github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support two-way communication in ipywidgets #3276

Support two-way communication in ipywidgets #3276

seeM commented May 27, 2024

seeM commented Jul 31, 2024 •

edited

Loading

testlabauto commented Aug 1, 2024

testlabauto commented Aug 7, 2024

testlabauto commented Aug 9, 2024

manzt commented Aug 9, 2024 •

edited

Loading

Support two-way communication in ipywidgets #3276

Support two-way communication in ipywidgets #3276

Comments

seeM commented May 27, 2024

seeM commented Jul 31, 2024 • edited Loading

testlabauto commented Aug 1, 2024

testlabauto commented Aug 7, 2024

testlabauto commented Aug 9, 2024

Verified Fixed

Test scenario(s)

Manually move slider

manzt commented Aug 9, 2024 • edited Loading

seeM commented Jul 31, 2024 •

edited

Loading

manzt commented Aug 9, 2024 •

edited

Loading