Panel embed not working with more than one pane in the same panel/layout

[julioasotodv]

Tested in OSX and Linux, Panel versions from 0.8.3 to 0.9.4 (and Bokeh 1.4.0 and 2.0.0, according to Panel requirements)

The issue

Hi! I am trying to embed a simple Panel layout (in this case in the notebook, but it could very well be with my_layout.save(embed=True) that contains:

• 2 widgets (in this case pn.widgets.Select, but it doesn't matter)
• 2 panes (in this case pn.pane.Markdown)

Where one pane is created through a reactive function involving one widget, and the other pane through another reactive function involving the other widget, as follows:

import panel as pn

pn.extension()

# First widget+pane:
widget_1 = pn.widgets.Select(options=["A", "B", "C"])

@pn.depends(widget_1.param.value)
def write_markdown_1(wid_val):
    return pn.pane.Markdown(object="You selected %s!" % wid_val)


# Second widget+pane:
widget_2 = pn.widgets.Select(options=["D", "E", "F"])

@pn.depends(widget_2.param.value)
def write_markdown_2(wid_val):
    return pn.pane.Markdown(object="You selected %s!" % wid_val)

Nothing special, it works as expected. However, I plan to generate a standalone HTML with these elements (no Bokeh Server, as my client does not know what is Python), so the easiest way would be to just wrap the elements inside a pn.Column, embed the column and call it a day:

# Embed all in column, in order to show all elements:

column = pn.Column(widget_1, write_markdown_1, widget_2, write_markdown_2)

column.embed()

However, the embedding is not responding as it should be. For some reason, the first Markdown does not work; whereas the second one works perfectly. Please take a look at this video:

I believe there is some problem with pn.io.embed(). I have tried to look at the code, but it has been really hard for me to figure out how the element tree is transversed in order to find what has to been transformed into a JsLink.

I mean, I believe it is a very first-world problem and not a whole lot of users will ever need to live without the Bokeh/Panel Server, but I have spent hours looking for the bug (the code above is just an example; the real code I'm writing is way more obfuscated).

If instead of placing everything in one Column we use two (one for each widget+pane), the problem disappears. However, exporting the whole thing as one HTML instead of two is way harder in that way (in fact, I have not been able to do so).

Thank you again for Panel. It's awesome.

[horatiubota]

Can I ask why this is labeled as an enhancement and not a bug? Is this the expected behavior when embedding multiple widget/panel combinations?

[philippjfr]

I guess it's both in fact. I'd expect this to work (so it's technically a bug) but at the same time at present I'd expect it to cause computing the cross-product of all widget options when ideally it would independently evaluate the two interactive elements and store them separetly.

[horatiubota]

Thanks!

I'm trying to debug this and I noticed that in the example above (and my own code), if the second Select widget has the last value selected (in this example, if you select "F"), the first widget updates its corresponding markdown as expected. If you select anything other than the last value in the second widget, then the first widget does not work anymore.

Update:
The state_model at the end of executing panel.io.embed for the example above looks like:

{
   "id":"1008",
   "js_event_callbacks":{

   },
   "js_property_callbacks":{

   },
   "json":False,
   "name":"None",
   "state":{
      "A":{
         "D":{
            "content":"{"events": [
                {"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected D!</p>"}], "references": []}",
            "header":"{"msgid": "1019", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "E":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected E!</p>"}], "references": []}",
            "header":"{"msgid": "1018", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "F":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1004"}, "new": "<p>You selected A!</p>"}, {"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected F!</p>"}], "references": []}",
            "header":"{"msgid": "1017", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         }
      },
      "B":{
         "D":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected D!</p>"}], "references": []}",
            "header":"{"msgid": "1016", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "E":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected E!</p>"}], "references": []}",
            "header":"{"msgid": "1015", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "F":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1004"}, "new": "<p>You selected B!</p>"}, {"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected F!</p>"}], "references": []}",
            "header":"{"msgid": "1014", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         }
      },
      "C":{
         "D":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected D!</p>"}], "references": []}",
            "header":"{"msgid": "1013", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "E":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected E!</p>"}], "references": []}",
            "header":"{"msgid": "1012", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         },
         "F":{
            "content":"{"events": [{"attr": "text", "kind": "ModelChanged", "model": {"id": "1004"}, "new": "<p>You selected C!</p>"}, {"attr": "text", "kind": "ModelChanged", "model": {"id": "1007"}, "new": "<p>You selected F!</p>"}], "references": []}",
            "header":"{"msgid": "1011", "msgtype": "PATCH-DOC"}",
            "metadata":"{}"
         }
      }
   },
   "subscribed_events":[

   ],
   "tags":[

   ],
   "values":[
      "A",
      "D"
   ],
   "widgets":{
      "1002":0,
      "1005":1
   }
}

Notice that only value F has events for both first and second Markdown widgets.

[horatiubota]

The issue seems to be caused by cross_product:

panel/panel/io/embed.py

Line 296 in 4bf8756

cross_product = list(product(*[vals[::-1] for _, _, vals, _ in values]))

And the fact that cross_product contains all combinations of widget values, in order -- for the example above, cross_product has the following value :

[('C', 'F'), ('C', 'E'), ('C', 'D'), ('B', 'F'), ('B', 'E'), ('B', 'D'), ('A', 'F'), ('A', 'E'), ('A', 'D')]

The problem is when iterating over the widget values (using cross_product) and updating them:

panel/panel/io/embed.py

Lines 311 to 318 in 4bf8756

	for key in (tqdm(cross_product, leave=False, file=sys.stdout) if progress else cross_product):
	sub_dict = state_dict
	skip = False
	for i, k in enumerate(key):
	w, m, _, g = values[i]
	try:
	with always_changed(config.safe_embed):
	w.value = k

When setting w.value = k, the first widget value will be set to C three times, which does not trigger a ModelChanged event (i.e., it triggers it once, the first time you set the value). In turn, this does not update the state model to include two of the values on the first widget (e.g., (C, E) and (C, D)), only the first value (e.g., (C, F)), which is why selecting the last value (F) on the second widget makes the first widget work (the same happens with B and A).

I quickly verified this by setting:

w.value = w.values[0]
w.value = w.values[1]
w.value = k

which triggers the change events. Is there a better way to manually trigger the ModelChanged event after setting w.value = k? I assume always_changed is controlling this somehow but I couldn't figure it out (tried setting pn.config.safe_embed = True but had the same outcome).

[chmielcode]

I've just encountered this bug and spent some time trying to figure out, what was going on. At least for me exporting interactive html files is the best part of Panel, so I really care about this issue not being forgotten. Thank you for your work.

[julioasotodv]

It looks like this is still happening in version 0.13.1; I believe that it will be hard to fix...

[philippjfr]

Honestly, it's more of a feature than a fix. The embed functionality was only ever designed for very simple applications, indeed it came out of the idea of HoloViews HoloMap components which embedded their contents which really consist only of a plot and a number of widgets that drive that plot. I think it wouldn't be that difficult to implement this. The main issue to solve is how you indicate which components are linked together, it should be possible in theory to figure out which widgets have effects on which other components but it's a very hard problem. A clean API that lets you express "these widgets control these components and these widgets control these other components" would make the problem a ton easier. If anyone has suggestions on what that might look like that'd be very helpful.

[aeantipov]

cross_product indeed seems to be doing something weird.
here's an example

pn.interact(
    lambda a,b : print(a,b),
    a = [1,2,3],
    b = [4,5,6],
).embed(max_states=9, progress=False)

The output is

1 4
3 4
3 6
3 5
3 4
2 4
2 6
2 5
2 4
1 4
1 6
1 5
1 4

(13 times against the total of 9). So some values will not be updated.

[aeantipov]

Even the single panel with two widgets actually exhibits the same behavior.
This code

import panel as pn
import numpy as np
pn.extension()

from matplotlib import pyplot as plt

def _plot(a,b):
    fig = plt.figure()
    x = np.linspace(0, np.pi, 101)
    plt.plot(x, np.sin(x*a + b))
    plt.title(f"a={a}, b={b}")
    plt.close(fig)
    return pn.pane.Matplotlib(fig)

pn.interact(
    _plot,
    a = [1,2,3],
    b = [4,5,6],
).embed(max_states=9, progress=False)

Gives a panel with all plots except of the last one with a=3 and b=6.

[yanovs]

I was wondering if there was any update on this?

I think I'm running into the same thing: when using a Matplotlib pane and embed(), the last rendering never updates. For me, it's always the last one, no matter how many widget choices I make. However, if I don't call embed() and just leave it in dynamic mode, it always works as expected.

FYI, I've tried some things and none have worked, including:

• Setting safe_embed via pn.extension("ipywidgets", safe_embed=True)
• Using matplotlib.figure.Figure() per https://panel.holoviz.org/FAQ.html
• Calling panel.param.trigger("object") per Matplotlib pane not updating output #3501

Possibly related:

• Older versions: matplotlib chart not updating  #399
• I was already returning plt.gcf(): Unable to redraw matplotlib plot when using interact #159
• Plot in jupyter using matplotlib is not updating #938
• DateSlider does not seem to update its value property #753
• Older versions: Introductory example doesn't seem to work #415

My notebook:

#!/usr/bin/env python
# coding: utf-8

# In[ ]:


import matplotlib as mpl
import matplotlib.backends.backend_agg
import matplotlib.pyplot as plt
import pandas as pd
import panel as pn


# In[ ]:


# pn.extension(safe_embed=True)
pn.extension("ipywidgets", safe_embed=True)
plt.ioff()
mpl.rcParams["figure.max_open_warning"] = 0


# In[ ]:


def func0(mult=2, title="abc"):
    ax = None

    df = pd.DataFrame({"a": [1, 2, 3]}) * mult
    df.plot(title=f"mult={mult}, title={title!r}", ax=ax)

    effective_fig = plt.gcf()
    pane = pn.pane.Matplotlib(effective_fig)
    return pane


def func1(mult=2, title="abc"):
    fig = mpl.figure.Figure()
    matplotlib.backends.backend_agg.FigureCanvas(fig)  # Init canvas
    ax = fig.subplots()

    df = pd.DataFrame({"a": [1, 2, 3]}) * mult
    df.plot(title=f"mult={mult}, title={title!r}", ax=ax)

    effective_fig = fig
    pane = pn.pane.Matplotlib(effective_fig)
    return pane


def func2(mult=2, title="abc"):
    fig = mpl.figure.Figure()
    matplotlib.backends.backend_agg.FigureCanvas(fig)  # Init canvas
    ax = fig.subplots()

    df = pd.DataFrame({"a": [1, 2, 3]}) * mult
    df.plot(title=f"mult={mult}, title={title!r}", ax=ax)

    effective_fig = fig
    pane = pn.pane.Matplotlib(effective_fig)
    pane.param.trigger("object")
    return pane


# None of these work
func = func0
# func = func1
# func = func2


# In[ ]:


# None of these work
# interact_view = pn.interact(func, mult=[1.0, 2.0, 3.0], title=["abc", "def"])
interact_view = pn.interact(func, mult=[1, 2, 3])
# interact_view = pn.interact(func, mult=(1, 3))
# interact_view = pn.interact(func, title=["abc", "def"])
interact_view


# In[ ]:


interact_view.embed(max_states=500, max_opts=500, progress=True)

Here are some versions of interest (I can provide more):

ipykernel          5.1.4
ipython            7.16.3
ipython-genutils   0.2.0
ipywidgets         7.5.1
jupyter-client     6.0.0
jupyter-core       4.6.3
matplotlib         2.2.5
notebook           6.0.3
pandas             1.5.2
panel              0.14.1
param              1.12.2

Thanks!

[m-maggi]

The issue seems to be caused by cross_product:

panel/panel/io/embed.py

Line 296 in 4bf8756

cross_product = list(product(*[vals[::-1] for _, _, vals, _ in values]))

And the fact that cross_product contains all combinations of widget values, in order -- for the example above, cross_product has the following value :

[('C', 'F'), ('C', 'E'), ('C', 'D'), ('B', 'F'), ('B', 'E'), ('B', 'D'), ('A', 'F'), ('A', 'E'), ('A', 'D')]

The problem is when iterating over the widget values (using cross_product) and updating them:

panel/panel/io/embed.py

Lines 311 to 318 in 4bf8756

	for key in (tqdm(cross_product, leave=False, file=sys.stdout) if progress else cross_product):
	sub_dict = state_dict
	skip = False
	for i, k in enumerate(key):
	w, m, _, g = values[i]
	try:
	with always_changed(config.safe_embed):
	w.value = k

When setting w.value = k, the first widget value will be set to C three times, which does not trigger a ModelChanged event (i.e., it triggers it once, the first time you set the value). In turn, this does not update the state model to include two of the values on the first widget (e.g., (C, E) and (C, D)), only the first value (e.g., (C, F)), which is why selecting the last value (F) on the second widget makes the first widget work (the same happens with B and A).

I quickly verified this by setting:

w.value = w.values[0]
w.value = w.values[1]
w.value = k

which triggers the change events. Is there a better way to manually trigger the ModelChanged event after setting w.value = k? I assume always_changed is controlling this somehow but I couldn't figure it out (tried setting pn.config.safe_embed = True but had the same outcome).

I also observed the behaviour described by @horatiubota, namely when one chooses the last option from the second selector, the interactivity using the first selector works, otherwise not.

Following the observation from @horatiubota I can make the interactivity work by changing

panel/panel/io/embed.py

Line 318 in 4bf8756

w.value = k

to

w.value = w.values[0]
w.value = w.values[1]
w.value = k