Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panel-based Datashader dashboard #676

Merged
merged 23 commits into from
Jan 28, 2019
Merged

Panel-based Datashader dashboard #676

merged 23 commits into from
Jan 28, 2019

Conversation

jbednar
Copy link
Member

@jbednar jbednar commented Dec 23, 2018

This PR closes #358, #618, and #221 by providing a Panel-based dashboard to replace the current pure Bokeh version that has been broken for several releases now, with the Intake support provided by @jsignell. The old Bokeh dashboard was over 500 lines of code that I was never able to update or even keep running, hence the long period of breakage. This new version fits into 50 lines of code (so far) and has nearly the same functionality, once the to-do items below are addressed:

  • opacity (of the data) and spread_size are not hooked up to the control
  • the widgets aren't in any order yet
  • it would be nice to separate some widgets to be on top of the plot and others down the side, if there is a clear semantic distinction
  • colormaps should be updated when the plot changes
  • opensky data is not tested
  • osm data is not tested and is not yet listed in the .yml file
  • the basemap argument doesn't update the map tiles, due to an open Bokeh bug in caching tiles (now fixed in Invalidate TileSource cache on property changes bokeh/bokeh#8594)
  • should probably add a logo
  • Need to allow field (fare, etc.) to be selected

There are also other to-do items that weren't supported in the last working version of the dashboard:

@jbednar
Copy link
Member Author

jbednar commented Dec 23, 2018

Old (hacked to be runnable, but doesn't show the data):

image

New:
image

@jbednar jbednar added the wip label Dec 23, 2018
examples/dashboard.yml Outdated Show resolved Hide resolved
counts:
label: Ride counts
plot:
xlim: !!python/tuple [-8240227.037, -8231283.905]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've arranged it so you can simply provide lists and they are cast appropriately.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I'm probably wrong about that, I think that was explicitly for dimension ranges.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to provide support for such casting if you can, to avoid this awkwardness.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think tuple is right and the awkwardness is yaml's fault.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's definitely YAML's fault, or at least JSON's fault (from which YAML inherits its scope). But the fact remains that it's awkward for Python users to write code like the above, so we should probably accept a list as well as a tuple unless we are already using the list/tuple distinction for some semantic purpose in this case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in holoviz/hvplot#248.

examples/dashboard.yml Outdated Show resolved Hide resolved
@philippjfr
Copy link
Member

I've pushed updates to apply opacity and spread_size, however control over global opacity required a small fix in holoviews, which we can push out as part of a 1.11.1 release (see holoviz/holoviews#3371).

@jbednar jbednar force-pushed the panel_dashboard branch 3 times, most recently from f4498be to ebd7cf6 Compare January 16, 2019 21:23
@jbednar
Copy link
Member Author

jbednar commented Jan 16, 2019

Current version:

image

Still just 54 lines. Need to add support for selecting which field to plot.

examples/dashboard.ipynb Outdated Show resolved Hide resolved
examples/dashboard.ipynb Outdated Show resolved Hide resolved
@jbednar jbednar force-pushed the panel_dashboard branch 2 times, most recently from 255a883 to 04ff6ca Compare January 22, 2019 15:08
@jbednar
Copy link
Member Author

jbednar commented Jan 25, 2019

I've made a number of changes in the above commits:

  • Added nyc_taxi_50k to datasets.yml as a quick test case to use
  • Updated the .yml file to consistently list "counts" as the first item in the list of plottable fields, as it makes a good default.
  • Removed any mention of "fare" from the .yml, because the nyc_taxi parq files don't currently include the fare column. (At some point I need to make new Parquet files including fare and also adding JFK to the visible range of points.)
  • Added an "agg_fn" parameter that allows the user to select min, max, mean, etc.

All of that seems fine so far. I've also made some changes that don't quite work yet:

  • I changed the .points method to .rasterize so that it could specify the agg_fn, which works but required me to make rasterize no longer be dynamic, which means that RangeXY is no longer tied to this object and zooming no longer updates the datashaded plot.
  • I added a "field" parameter, which is populated correctly but doesn't seem to have any effect yet, despite being applied in the same method as the agg_fn and having the same dependency structure.

@jbednar
Copy link
Member Author

jbednar commented Jan 28, 2019

Thanks, @philippjfr for re-connecting RangeXY and for realizing that the field parameter is working, it just has no noticeable effect for a counts aggregator; e.g. switch to Mean to see the field having an effect. I've renamed .points to .elem because the plot isn't always a points plot; depends what's declared in the data catalog.

I also reorganized the code a bit and added a very extensive section at the end unpacking how it works. It would be nice not to have to use three different mechanisms for getting the DynamicMaps involved, but I'll open a separate issue about that. Meanwhile, I think it's a very concise and mostly understandable way to express some quite complicated relationships between user-visible controls and computations, and it's a massive improvement over the old dashboard, so it should be ready to merge when tests pass.

@jbednar jbednar merged commit cd033d5 into master Jan 28, 2019
weiji14 added a commit to weiji14/deepicedrain that referenced this pull request Aug 21, 2020
Improve our HvPlot/Panel dashboard with some new bells and whistles! Like a proper GIS desktop tool, the xy_dhdt dashboard plot can now keep the zoom level when changing between variables (thanks to https://discourse.holoviz.org/t/keep-zoom-level-when-changing-between-variables-in-a-scatter-plot)! Supersedes e4874b0. This is a major refresh of my old IceSatExplorer code at https://github.com/weiji14/cryospheric-data-lakes/blob/master/code/scripts/h5_to_np_icesat.ipynb, which uses ICESat-1 instead of ICESat-2. The dashboard also takes a lot of cues from the example at https://examples.pyviz.org/datashader_dashboard/dashboard.html, implemented in holoviz/datashader#676.

Other significant improvements include a categorical colourmap for the 'referencegroundtrack' variable, and being able to see the height and time of an ICESat-2 measurement at a particular cycle on hover over the points! Oh, and did I mention that the rendering now happens on the GPU?!! Data transformed to and from Parquet is fast! Note that this is a work in progress, and that there are more sweeping improvements to come. I've also split out the crossover analysis code into a separate atlxi_lake.ipynb file since atlxi_dhdt.ipynb was getting too long.
weiji14 added a commit to weiji14/deepicedrain that referenced this pull request Aug 22, 2020
Improve our HvPlot/Panel dashboard with some new bells and whistles! Like a proper GIS desktop tool, the xy_dhdt dashboard plot can now keep the zoom level when changing between variables (thanks to https://discourse.holoviz.org/t/keep-zoom-level-when-changing-between-variables-in-a-scatter-plot)! Supersedes e4874b0. This is a major refresh of my old IceSatExplorer code at https://github.com/weiji14/cryospheric-data-lakes/blob/master/code/scripts/h5_to_np_icesat.ipynb, which uses ICESat-1 instead of ICESat-2. The dashboard also takes a lot of cues from the example at https://examples.pyviz.org/datashader_dashboard/dashboard.html, implemented in holoviz/datashader#676.

Other significant improvements include a categorical colourmap for the 'referencegroundtrack' variable, and being able to see the height and time of an ICESat-2 measurement at a particular cycle on hover over the points! Oh, and did I mention that the rendering now happens on the GPU?!! Data transformed to and from Parquet is fast! Note that this is a work in progress, and that there are more sweeping improvements to come. I've also split out the crossover analysis code into a separate atlxi_lake.ipynb file since atlxi_dhdt.ipynb was getting too long.
weiji14 added a commit to weiji14/deepicedrain that referenced this pull request Aug 24, 2020
Improve our HvPlot/Panel dashboard with some new bells and whistles! Like a proper GIS desktop tool, the xy_dhdt dashboard plot can now keep the zoom level when changing between variables (thanks to https://discourse.holoviz.org/t/keep-zoom-level-when-changing-between-variables-in-a-scatter-plot)! Supersedes e4874b0. This is a major refresh of my old IceSatExplorer code at https://github.com/weiji14/cryospheric-data-lakes/blob/master/code/scripts/h5_to_np_icesat.ipynb, which uses ICESat-1 instead of ICESat-2. The dashboard also takes a lot of cues from the example at https://examples.pyviz.org/datashader_dashboard/dashboard.html, implemented in holoviz/datashader#676.

Other significant improvements include a categorical colourmap for the 'referencegroundtrack' variable, and being able to see the height and time of an ICESat-2 measurement at a particular cycle on hover over the points! Oh, and did I mention that the rendering now happens on the GPU?!! Data transformed to and from Parquet is fast! Note that this is a work in progress, and that there are more sweeping improvements to come. I've also split out the crossover analysis code into a separate atlxi_lake.ipynb file since atlxi_dhdt.ipynb was getting too long.
weiji14 added a commit to weiji14/deepicedrain that referenced this pull request Sep 15, 2020
Improve our HvPlot/Panel dashboard with some new bells and whistles! Like a proper GIS desktop tool, the xy_dhdt dashboard plot can now keep the zoom level when changing between variables (thanks to https://discourse.holoviz.org/t/keep-zoom-level-when-changing-between-variables-in-a-scatter-plot)! Supersedes e4874b0. This is a major refresh of my old IceSatExplorer code at https://github.com/weiji14/cryospheric-data-lakes/blob/master/code/scripts/h5_to_np_icesat.ipynb, which uses ICESat-1 instead of ICESat-2. The dashboard also takes a lot of cues from the example at https://examples.pyviz.org/datashader_dashboard/dashboard.html, implemented in holoviz/datashader#676.

Other significant improvements include a categorical colourmap for the 'referencegroundtrack' variable, and being able to see the height and time of an ICESat-2 measurement at a particular cycle on hover over the points! Oh, and did I mention that the rendering now happens on the GPU?!! Data transformed to and from Parquet is fast! Note that this is a work in progress, and that there are more sweeping improvements to come. I've also split out the crossover analysis code into a separate atlxi_lake.ipynb file since atlxi_dhdt.ipynb was getting too long.
@maximlt maximlt deleted the panel_dashboard branch December 25, 2021 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Simplify dashboard example
3 participants