Allow Overlay returned in DynamicMap to return varying number of items

[thoth291]

In the notebook here I'm building a simple peakfinder widget.

So far 3 problems:

1. Bokeh point size
   Here is what I get for the initial stage: You can clearly see the Blue point

Then I move to the parameter for which that Blue point disappears.

Then I move it back to the initial space - but the point size is reduced to the minimal

2. Matplotlib: point size is 100 and seems to be depend on fig_size - it's counterintuitive to me

3. My calculation may take a while - how to delay the recalculation to the time when the slider is released?

P.S. I also want to update this DynamicMap to HoloMap so that when I save it goes to HTML, but when I wrap dmap with holomap - it does nothing... I'm sure I'm doing something wrong...

[thoth291]

1 more issue.

4. In the same notebook (scroll to the bottom) DynamicMap is not responding at all:

[thoth291]

I think the issues 1. and 4. are connected.
In 4. apparently my lowess method was buggy and produced empty list which caused empty picks and etc. It is somehow affecting other methods to appear.
In 1. all of a sudden point is disappears and then reappears but with different style settings.

So if one of the plots in DynamicMap is changing something dramatic in one of the Overlays then it produces wrong image: with bad Style or Plot options...

This is example when 4. is not working

ldata=lowess[:, 0]
lxs=lowess[:, 1]

This is an example when 4. is working

ldata=lowess[:, 1]
lxs=lowess[:, 0]

[philippjfr]

Sorry for being so slow to respond.

then I move it back to the initial space - but the point size is reduced to the minimal

Definitely seems like a bug I'll start investigating right away.

Matplotlib: point size is 100 and seems to be depend on fig_size - it's counterintuitive to me

This is an overall issue with sizing in matplotlib, the fig_size simply scales the fig_inches of the figure and leaves text and point sizes unaffected. I agree this is unintuitive and would like a scaling mechanism that scales everything in the figure.

how to delay the recalculation to the time when the slider is released?

Our widgets have some debouncing by default but disabling continuous updates seems like a good feature request.

I also want to update this DynamicMap to HoloMap so that when I save it goes to HTML, but when I wrap dmap with holomap - it does nothing... I'm sure I'm doing something wrong...

The way to achieve this is to explicitly sample the DynamicMap first like this:

hv.HoloMap(dmap[[0, 10, 20], [1, 2, 3]])

all of a sudden point is disappears and then reappears but with different style settings.

This does seem related to your first issue, I'll investigate.

[philippjfr]

Okay I've finally had a chance to go through your notebook. It's quite hard to follow complex examples like this and self-contained, minimal examples are much preferred when tracking down bugs. I've now tracked down the issue to the fact that your DynamicMap is returning an Overlay with varying numbers of Points. The number of items in an Overlay must be fixed when working with a DynamicMap because the plot cannot create new glyphs/artists on the fly. In bokeh you may use an NdOverlay with varying numbers of items but in matplotlib the number of Elements returned by a DynamicMap callable must remain fixed. I'm going to leave this issue open to better document these limitations.

[jlstevens]

Sounds like something we want to detail in the User Guide. Not sure which one though...

[thoth291]

Thanks, @philippjfr. I'm sorry for the big notebook - will try to be better next time.
Yeah - inconsistency in-between the backends are very surprising and mostly unpredicted. It would be nice not to have them at all - but that's impossible to achieve...

Will you create a request on this issue which you mentioned above:

Our widgets have some debouncing by default but disabling continuous updates seems like a good feature request.

I'll switch to NdOverlay for bokeh, but will have to implement static plots using pure matplotlib. Which means that instead of reducing libraries from (bokeh, matplotlib) to (holoviews) - I add one more library dependency and have to keep 3 libraries (bokeh, matplotlib and holoviews) in mind while working with my data.

[stevievb]

@philippjfr , I'm trying to use NdOverlay in bokeh with a varying number of items as you mentioned above but it doesn't seem to be working.

import holoviews as hv
import holoviews.plotting.bokeh # noqa (Activate backend)
from bokeh.layouts import column
from bokeh.plotting import curdoc
from pandas import DataFrame
from holoviews.streams import Stream
from bokeh.layouts import widgetbox
from bokeh.models.widgets import CheckboxGroup

doc = curdoc()

df1 = DataFrame({'x': [1, 2, 3], 'y': [2, 5, 8], 'desc': ['A', 'b', 'C']})
df2 = DataFrame({'x': [1.2, 2.2, 3.2], 'y': [2, 5, 8], 'desc': ['A', 'b', 'C']})
df3 = DataFrame({'x': [1.4, 2.4, 3.4], 'y': [2, 5, 8], 'desc': ['A', 'b', 'C']})

ShowMoreStream = Stream.define('Show More Stream', show_more=False)
show_more_stream = ShowMoreStream()

points1 = hv.Points(df1, kdims=['x', 'y'], vdims=['desc'])
points2 = hv.Points(df2, kdims=['x', 'y'], vdims=['desc'])
points3 = hv.Points(df3, kdims=['x', 'y'], vdims=['desc'])

renderer = hv.Store.renderers['bokeh'].instance(mode='server', holomap='server')
options = hv.Store.options(backend='bokeh')

def points_callback(show_more):
    if show_more:
        return hv.NdOverlay({'1': points1,
                             '2': points2,
                             '3': points3})
    else:
        return hv.NdOverlay(hv.NdOverlay({'1': points1,
                                        '2': points2}))

points_dmap = hv.DynamicMap(points_callback, streams=[show_more_stream])

def show_more_checkbox_change_callback(attr, old, new):
    if len(new):
        points_dmap.event(show_more=True)
    else:
        points_dmap.event(show_more=False)

checkbox_group = CheckboxGroup(labels=["Show More"])
checkbox_group.on_change("active", show_more_checkbox_change_callback)

plot = renderer.get_plot(points_dmap)

layout = column(plot.state, widgetbox(checkbox_group))

doc.add_root(layout)

When I hit the checkbox I get the message " OverlayPlot01231: Some Elements returned by the dynamic callback were not initialized correctly and could not be rendered."

If reverse the order so that the overlay with the most number of elements is displayed first I don't get and error message and I can toggle the element on and off but the legend does not update.

def points_callback(show_more):
    if show_more:
        return hv.NdOverlay(hv.NdOverlay({'1': points1,
                                        '2': points2}))
    else:
        return hv.NdOverlay({'1': points1,
                             '2': points2,
                             '3': points3})

[philippjfr]

Thanks for the nice reproducible example, that message is unfortunately less informative than it could be, so I'll make sure to fix that and I'll have to document this stuff. Currently to be able to update NdOverlays easily we have something called "batched" mode, which collapses all the elements in your NdOverlay into a single ColumnDataSource. Batched mode doesn't support legends so we have decided that any NdOverlay with more than 10 items will be batched by default. You can reduce the batching limit with the legend_items plot option. In your example you can either reduce the legend_limit globally or for those NdOverlays in particular e.g.:

def points_callback(show_more):
    if show_more:
        items = {'1': points1, '2': points2})
    else:
        items = {'1': points1, '2': points2, '3': points3})
    return hv.NdOverlay(items).opts(plot=dict(legend_limit=0))

This is quite an annoyance so I'll make sure to document it. In future we can likely lift this restriction at least on bokeh server.

[shaginhekvs]

@philippjfr the problem partly still persists, if you flip the order such that

def points_callback(show_more):
    if not show_more:
        items = {'1': points1, '2': points2})
    else:
        items = {'1': points1, '2': points2, '3': points3})
    return hv.NdOverlay(items).opts(plot=dict(legend_limit=0))

you can only see the initially plotted 2 points now and the 3rd point cannot be enabled.

[philippjfr]

Since there is now a clear path forward toward allowing DynamicMaps to return Overlays with varying numbers of items I've repurposed the issue and assigned it to the next milestone.

[philippjfr]

This is now supported with some caveats (namely in bokeh the toolbar will disappear but this is an issue that must be addressed in bokeh).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.