Html repr #3425

jsignell · 2019-10-21T21:08:54Z

This PR supersedes #1820 - see that PR for original discussion. See this gist to try out the new MultiIndex and options functionality.

Closes html repr of xarray object (for the notebook) #1627, closes WIP: html repr #1820
Tests added
Passes black . && mypy . && flake8
Fully documented, including whats-new.rst for all changes and api.rst for new API

TODO:

Add support for Multi-indexes
Probably good to have some opt-in or fail back system in case where we (or users) know that the rendering will not work
Add some tests

pep8speaks · 2019-10-21T21:09:06Z

Hello @jsignell! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-10-24 15:59:14 UTC

jsignell · 2019-10-21T22:18:14Z

I don't really like how the attrs look without the alignment of keys and values. I will think about how to make that more legible

dcherian · 2019-10-21T22:50:40Z

Honestly this is looking super good and ready to merge already!

Thanks @jsignell

The only thing that looks funny to me is the alternating black-gray text coloring below:

jsignell · 2019-10-22T01:01:22Z

Yeah I agree that is a strange part of the original PR. The version in the fiddle doesn't do that, so I'd really like to hear back from @benbovy about whether the fiddle css is more uptodate.

rabernat · 2019-10-22T01:05:40Z

Julia, it's fantastic to see you working on this! Thanks so much!

shoyer · 2019-10-22T01:15:19Z

This looks fantastic! Really great to have you working on this!

Some specific feedback:

Yes, I agree it would be nice to align the Attributes text, maybe this could make use of an HTML table?
Maybe bold would look better than underline for coordinates that are dimensions? I find the underline makes text a little hard to read. Happy to leave this up to you, though.
On my Ubuntu desktop, the "+" icon is bright yellow, which I'm guessing is not what was intended. Maybe it would better to stick with the three circle "data" icon?
What happens if there is a really long variable name? I had a version of the jsfiddle that expanded names when you hovered over them.

xarray/core/formatting_html.py

shoyer · 2019-10-22T01:19:12Z

xarray/core/formatting_html.py

+    d['id'] = 'section-' + str(uuid.uuid4())
+
+    # TODO: no value preview if not in memory
+    d['preview'] = format_values_preview(obj.values, max_char=70)


same concern as above

xarray/core/formatting_html.py

benbovy · 2019-10-22T07:31:46Z

This is really nice @jsignell !

I'd really like to hear back from @benbovy about whether the fiddle css is more uptodate.

We did a lot of experiments, even some after starting the implementation in #1820, so I admit this is quite messy. I also did some clean-up and tweaks directly in that PR, so unfortunately there is no fiddle of reference that exactly corresponds to what's in here. Sorry for that!

The html/css in this PR is the cleanest one, I think.

The latest fiddles (for Dataset) are these two ones:

https://jsfiddle.net/Lmqq7yzz/9/
https://jsfiddle.net/1g04ykum/ (@shoyer's try for handling long variable names)

the only thing that looks funny to me is the alternating black-gray text coloring

In the HTML repr, the width of the space character is less large that for the monospace font used in the text repr, so I originally used this trick to better distinguish between the values in the inline data repr. Now that each value is encapsulated in it's own span, we could tweak the padding instead (or both... I don't have strong opinion on this).

benbovy · 2019-10-22T07:35:37Z

Just one small issue in the gist with the vertical alignment within a variable row, and the height of the icon container (I'm using Firefox):

jsignell · 2019-10-22T13:17:28Z

Yeah the data preview is also shifted vertically.

jsignell · 2019-10-22T14:08:00Z

I just updated the gist with these changes:

got rid of alternating data colors in preview
fixed icon alignment in row, and rest of data in row
got rid of some not-useful spans
changed from underline to bold for index coords

xarray/core/formatting_html.py

jsignell · 2019-10-22T17:42:03Z

Is there ever a situation in which dimensions would be expandable? It seems like it is hard-coded to collapsed.

I am trying to generalize the css to reduce the length.

shoyer · 2019-10-22T19:19:20Z

Is there ever a situation in which dimensions would be expandable? It seems like it is hard-coded to collapsed.

I don't think it would ever be expanded. The collapsed icon is just there for visual consistency.

jsignell · 2019-10-22T22:08:49Z

Ok I think I've addressed all the comments. I just updated the gist with the latest output. Tomorrow I'll start adding tests. The css is now ~300 lines and it gets injected everytime the dataset is rendered. I'm not sure if that is considered too long and if it is I don't quite know what to do about it.

shoyer · 2019-10-22T23:37:46Z

The css is now ~300 lines and it gets injected everytime the dataset is rendered. I'm not sure if that is considered too long and if it is I don't quite know what to do about it.

You could consider running the CSS through some sort of preprocessing tool to minify it, e.g., to remove all the spaces. But my guess is that it's probably fine.

It would be informative to measure the size of the output HTML encoded as UTF-8 bytes for some example datasets. For context, a simple line chart from matplotlib is ~20 KB when saved as a PNG image. Anything in that ball-park would be OK, up to perhaps a few hundred KB for really big datasets with lots of variables.

shoyer · 2019-10-22T23:39:03Z

xarray/core/common.py

@@ -134,6 +134,12 @@ def __array__(self: Any, dtype: DTypeLike = None) -> np.ndarray:
    def __repr__(self) -> str:
        return formatting.array_repr(self)

+    def _repr_html_(self):
+        if OPTIONS["display_style"] == "classic":
+            classic = repr(self).replace("<", "&lt;").replace(">", "&gt;")


Use html.escape here: https://docs.python.org/3/library/html.html#html.escape

shoyer · 2019-10-22T23:39:36Z

xarray/core/common.py

+    def _repr_html_(self):
+        if OPTIONS["display_style"] == "classic":
+            classic = repr(self).replace("<", "&lt;").replace(">", "&gt;")
+            return "<pre>{repr}</pre>".format(repr=classic)


Could use an f-string here:

Suggested change

return "<pre>{repr}</pre>".format(repr=classic)

return f"<pre>{classic}</pre>"

I didn't realize that xarray was already supporting >=3.6 👍

setup.py

shoyer · 2019-10-22T23:59:53Z

I just gave this a try in Chrome / Google Colab. It's looking great!

Here's the rasm dataset. It weighs in at exactly 20 KB!

The alignment of "Dimensions" line does looks a little off -- would that be easy to fix?

There's also a strange alignment thing with the three circle icon in array repr, which maybe could be moved to the right side?

shoyer · 2019-10-23T01:13:41Z

We probably should be careful to use html.escape() pervasively to protect against malformed data. We can trust user code, but probably should not trust all data -- it should be safe to load and view arbitrary netCDF files in xarray.

codecov · 2019-10-23T13:07:59Z

Codecov Report

Merging #3425 into master will decrease coverage by 1.09%.
The diff coverage is 23.12%.

@@            Coverage Diff            @@
##           master    #3425     +/-   ##
=========================================
- Coverage   96.16%   95.06%   -1.1%     
=========================================
  Files          65       66      +1     
  Lines       13436    13587    +151     
=========================================
- Hits        12921    12917      -4     
- Misses        515      670    +155

Impacted Files	Coverage Δ
xarray/core/options.py	`97.91% <100%> (+0.09%)`	⬆️
xarray/core/dataset.py	`95.89% <20%> (-0.23%)`	⬇️
xarray/core/formatting_html.py	`21.05% <21.05%> (ø)`
xarray/core/common.py	`91.92% <42.85%> (-1.29%)`	⬇️
xarray/convert.py	`100% <0%> (ø)`	⬆️
xarray/plot/facetgrid.py	`96.73% <0%> (ø)`	⬆️
xarray/plot/plot.py	`96.38% <0%> (ø)`	⬆️
xarray/backends/rasterio_.py	`96.85% <0%> (ø)`	⬆️
xarray/plot/utils.py	`97.74% <0%> (ø)`	⬆️
xarray/core/resample_cftime.py	`91.48% <0%> (ø)`	⬆️
... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0c336f...3a9edae. Read the comment docs.

shoyer · 2019-10-23T16:47:28Z

Here's what we're currently looking at!

jsignell · 2019-10-23T16:59:38Z

I'm still worried about size, but minifying the css and even minifying the html did not make a big difference. It looks like the html_repr for rasm is 38kb .

shoyer · 2019-10-23T17:06:59Z

It looks like the html_repr for rasm is 38kb .

How are you measuring this?

I see:

>>> len(rasm._repr_html_().encode('utf8'))
19624

jsignell · 2019-10-23T17:07:33Z

~~Oh I was looking at bytes not len.~~ I didn't have .encode('utf8')

jsignell · 2019-10-23T17:11:51Z

Ok I am going to write some tests now. I updated the gist. Do people like the names "classic" and "html" for the display_style options? Another option would be "classic" and "rich".

benbovy · 2019-10-23T18:35:23Z

Or maybe "text" and "html"?

Hopefully "html" will eventually become the classic display :)

jsignell · 2019-10-23T19:23:16Z

I added some tests. I can add more if anyone has a test case that they want to be sure is covered.

jsignell · 2019-10-24T15:59:31Z

Ok I think it's fixed now

benbovy · 2019-10-24T16:28:22Z

Yes it looks nice!

jsignell · 2019-10-24T16:44:55Z

It isn't too hard to get it working with dark theme, but requires slight tweaks to the colors to align with those provided as vars by jlab.

dcherian · 2019-10-24T16:48:41Z

Awesome. Thanks @jsignell Let's leave dark theme compatibility for a future PR.

shoyer · 2019-10-24T16:57:45Z

Should we expand the .data section of the DataArray HTML repr by default? That would look a little more consistent with the text repr.

benbovy · 2019-10-24T16:57:57Z

Thanks a lot @jsignell ! It is great to see this merged finally!

mrocklin · 2019-10-24T18:52:08Z

Thanks @jsignell (and all). I'm really jazzed about this.

choldgraf · 2019-10-24T19:16:27Z

this is awesome! I was so excited about it that I updated my little blog post with the latest xarray master https://predictablynoisy.com/xarray-explore-ieeg

I was a little sad at the gigantic repr that came out of my DataArray, and now it is nice and tidy html!

benbovy · 2019-10-24T19:42:09Z

Cool!

I guess this post about how you use notebooks in your blog is still up to date?

I looks like we'll need to fix a couple of issues (the channel coordinate looks weird), but maybe it is not related to the front-end.

Besides variable names, we probably need to somehow handle overflow for variables that have a long list of dimension labels.

choldgraf · 2019-10-24T19:54:44Z

Actually, no that's totally out of date now haha. Thanks for reminding me I should update that

edit: realized I should have said what I'm actually using - I'm using this:https://jupyterbook.org/features/page.html

jsignell · 2019-10-24T20:09:44Z

I looks like we'll need to fix a couple of issues (the channel coordinate looks weird), but maybe it is not related to the front-end.

I am opening a PR. I didn't escape dtypes and this dtype is <U5 which html does terribly with

shoyer · 2019-10-24T20:16:13Z

To verify proper escaping, we could validate the generated HTML as XML:
https://stackoverflow.com/questions/13742538/how-to-validate-xml-using-python-without-third-party-libs

This could be a good use case for Hypothesis. Or you could manually build a dataset with variables of every dtype.

jsignell · 2019-10-24T20:26:17Z

I opened a PR and included a small test. It can probably go in while we work on a more complete testing infrastructure.

jsignell · 2019-10-24T20:29:22Z

Besides variable names, we probably need to somehow handle overflow for variables that have a long list of dimension labels.

These are handled in the same way as long variable names, they truncate with an ellipsis and then expand on hover.

benbovy · 2019-10-25T07:00:26Z

These are handled in the same way as long variable names, they truncate with an ellipsis and then expand on hover.

Oh yes I missed it.

* upstream/master: Escaping dtypes (pydata#3444) Html repr (pydata#3425)

…e-multiple-dims * upstream/master: change ALL_DIMS to equal ellipsis (pydata#3418) Escaping dtypes (pydata#3444) Html repr (pydata#3425)

* upstream/master: Another groupby.reduce bugfix. (pydata#3403) add icomoon license (pydata#3448) change ALL_DIMS to equal ellipsis (pydata#3418) Escaping dtypes (pydata#3444) Html repr (pydata#3425)

* upstream/master: upgrade black verison to 19.10b0 (pydata#3456) Remove outdated code related to compatibility with netcdftime (pydata#3450) Remove deprecated behavior from dataset.drop docstring (pydata#3451) jupyterlab dark theme (pydata#3443) Drop groups associated with nans in group variable (pydata#3406) Allow ellipsis (...) in transpose (pydata#3421) Another groupby.reduce bugfix. (pydata#3403) add icomoon license (pydata#3448) change ALL_DIMS to equal ellipsis (pydata#3418) Escaping dtypes (pydata#3444) Html repr (pydata#3425)

commit bc39877 Merge: 507b1f6 278d2e6 Author: dcherian <[email protected]> Date: Tue Oct 29 09:36:30 2019 -0600 Merge remote-tracking branch 'upstream/master' into dask-tokenize * upstream/master: upgrade black verison to 19.10b0 (pydata#3456) Remove outdated code related to compatibility with netcdftime (pydata#3450) Remove deprecated behavior from dataset.drop docstring (pydata#3451) jupyterlab dark theme (pydata#3443) Drop groups associated with nans in group variable (pydata#3406) Allow ellipsis (...) in transpose (pydata#3421) Another groupby.reduce bugfix. (pydata#3403) add icomoon license (pydata#3448) change ALL_DIMS to equal ellipsis (pydata#3418) Escaping dtypes (pydata#3444) Html repr (pydata#3425) commit 507b1f6 Author: dcherian <[email protected]> Date: Tue Oct 29 09:34:47 2019 -0600 Fix window test commit 4ab6a66 Author: dcherian <[email protected]> Date: Thu Oct 24 14:30:57 2019 -0600 Implement __dask_tokenize__

shoyer reviewed Oct 22, 2019

View reviewed changes

xarray/core/formatting_html.py Outdated Show resolved Hide resolved

xarray/core/formatting_html.py Outdated Show resolved Hide resolved

shoyer reviewed Oct 22, 2019

View reviewed changes

xarray/core/formatting_html.py Outdated Show resolved Hide resolved

crusaderky mentioned this pull request Oct 22, 2019

v0.14.1 Release #3434

Closed

12 tasks

shoyer reviewed Oct 22, 2019

View reviewed changes

setup.py Outdated Show resolved Hide resolved

dcherian merged commit ba48fbc into pydata:master Oct 24, 2019

jsignell deleted the html_repr branch October 24, 2019 16:58

jsignell mentioned this pull request Oct 24, 2019

jupyterlab dark theme #3443

Merged

4 tasks

jsignell mentioned this pull request Oct 24, 2019

Escaping dtypes #3444

Merged

4 tasks

SimonHeybrock mentioned this pull request Oct 25, 2019

_repr_html_ for improved rendering in Jupyter notebooks scipp/scipp#496

Closed

dcherian added a commit to dcherian/xarray that referenced this pull request Oct 25, 2019

Merge remote-tracking branch 'upstream/master' into interp-na-maxgap

fde2c14

* upstream/master: Escaping dtypes (pydata#3444) Html repr (pydata#3425)

ahuang11 mentioned this pull request Nov 15, 2019

collapsible=True / expandable=True for pn.Column holoviz/panel#792

Closed

dtasev mentioned this pull request Dec 11, 2019

Implements to_html for Dataset scipp/scipp#743

Merged

jsignell mentioned this pull request Feb 27, 2020

Turn on _repr_html_ by default? #3806

Closed

thomasjpfan mentioned this pull request Apr 23, 2020

ENH Adds HTML visualizations for estimators scikit-learn/scikit-learn#14180

Merged

juanfcocontreras mentioned this pull request May 22, 2020

Support for xarray man-group/dtale#200

Closed

	return "<pre>{repr}</pre>".format(repr=classic)
	return f"<pre>{classic}</pre>"

Html repr #3425

Html repr #3425

Conversation

jsignell commented Oct 21, 2019 • edited by dcherian Loading

pep8speaks commented Oct 21, 2019 • edited Loading

Comment last updated at 2019-10-24 15:59:14 UTC

jsignell commented Oct 21, 2019

dcherian commented Oct 21, 2019

jsignell commented Oct 22, 2019

rabernat commented Oct 22, 2019

shoyer commented Oct 22, 2019

shoyer Oct 22, 2019

Choose a reason for hiding this comment

benbovy commented Oct 22, 2019

benbovy commented Oct 22, 2019 • edited Loading

jsignell commented Oct 22, 2019

jsignell commented Oct 22, 2019

jsignell commented Oct 22, 2019

shoyer commented Oct 22, 2019

jsignell commented Oct 22, 2019

shoyer commented Oct 22, 2019

shoyer Oct 22, 2019

Choose a reason for hiding this comment

shoyer Oct 22, 2019

Choose a reason for hiding this comment

jsignell Oct 23, 2019

Choose a reason for hiding this comment

shoyer commented Oct 22, 2019 • edited Loading

shoyer commented Oct 23, 2019

codecov bot commented Oct 23, 2019 • edited Loading

Codecov Report

shoyer commented Oct 23, 2019

jsignell commented Oct 23, 2019

shoyer commented Oct 23, 2019

jsignell commented Oct 23, 2019 • edited Loading

jsignell commented Oct 23, 2019

benbovy commented Oct 23, 2019

jsignell commented Oct 23, 2019

jsignell commented Oct 24, 2019

benbovy commented Oct 24, 2019

jsignell commented Oct 24, 2019

dcherian commented Oct 24, 2019

shoyer commented Oct 24, 2019

benbovy commented Oct 24, 2019

mrocklin commented Oct 24, 2019

choldgraf commented Oct 24, 2019 • edited Loading

benbovy commented Oct 24, 2019

choldgraf commented Oct 24, 2019 • edited Loading

jsignell commented Oct 24, 2019

shoyer commented Oct 24, 2019

jsignell commented Oct 24, 2019

jsignell commented Oct 24, 2019

benbovy commented Oct 25, 2019

jsignell commented Oct 21, 2019 •

edited by dcherian

Loading

pep8speaks commented Oct 21, 2019 •

edited

Loading

benbovy commented Oct 22, 2019 •

edited

Loading

shoyer commented Oct 22, 2019 •

edited

Loading

codecov bot commented Oct 23, 2019 •

edited

Loading

jsignell commented Oct 23, 2019 •

edited

Loading

choldgraf commented Oct 24, 2019 •

edited

Loading

choldgraf commented Oct 24, 2019 •

edited

Loading