Add better and more data grids

[MarcSkovMadsen]

My Pain

I believe the current DataFrame pane and widget built on SlickGrid are not up to the task.

There are several issues

• Lack of support for other types of data than DataFrame like Arrow arrow or Bokeh ColumnDataSource
• Lack of support for streaming data (which ColumnDataSource) could provide
• Lack of support for responsive and many columns
• Lack of advanced customization and styling

I believe nice and easy to use tables are just as important as nice plots. So I really believe the lack of nice tables/ grids is holding Panel back.

Solution

Improve current implementation or provide additional grids.

Additional Context

Below I will provide some grids for inspiration.

[MarcSkovMadsen]

Tabulator

http://tabulator.info/

Tabulator is a MIT licensed grid which seems to look+feel great and have a very, very nice api.

I believe the implementation could be something very similar to how the ECharts pane is implemented.

At it's core you just need a .configuration parameter which holds the configuration dictionary used by tabulator.

Then you could provide an additional .value parameter which could be used to transfer larger amounts of data. The user should be able to provide value as DataFrame, ColumnDataSource or maybe some day in the future Arrow. This value will just be added to the .data configuration on the js. side.

Then you could provide some kind of functionality to provide (parts of) the configuration with js functions for advanced formatting, callbacks etc. Either the user can specify the configuration entirely as a string or he/ she can specify the js part as a .configuration_js string parameter which gets merged into the configuration on the js side. I prefer the latter.

Then we should provide selection and selected_values parameters. The selection provides a list of row numbers or indexes. The selected_values provides the selected rows as a DataFrame, ColumnDataSource or Arrow table depending on the value type provided.

We should also enable streaming. Either via the stream api of ColumnDataSource, by providing a .value_append parameter or by providing a value_change_type (replace, append, prepend, ...).

I believe it would be nice to provide some helper function which takes a DataFrame or similar and converts it into the columns list. Then the user has an easy starting point.

I also believe it would be nice to provide the user with an easy way to append one of the different available themes to pn.configure.css_files.

Take a look at the documentation. It's really, really awesome and understandable.

I have started experimenting and coding for the awesome-panel-extensions package. So reach out before you start to not repeat work already done.

[MarcSkovMadsen]

JExcel

My users are often very familiar with Excel. So being able to provide an excel like grid would be awesome.

I've found jExcel https://github.com/paulhodel/jexcel which at a first look looks awesome and could be implemented an a way very similar to Tabulator above. Its MIT licensed.

[MarcSkovMadsen]

I have a poc of Tabulator working. I've based it on a bokeh .ts and bokeh .py model.

It works really, really great: It's responsive, Has different css styles out of the box, Is very easy to configure with lots of examples.

---

Cannot prebuild

Currently I refer to the implementation in the bokeh .py model

The problem is that if I want to use it as a prebuilt model using panel build awesome_panel_extensions it does not register.

It's strange because if I look at the dist files I can see the TabulatorModel is being registered.

Dataframe vs ColumnDataSource

I am in doubt on the api. Currently the user can provide a Dataframe or a ColumnDataSource. If he she provides a ColumnDataSource the he/ she can use the stream and patching api of the CDS and it works great (see example above).

But I am in doubt if the users will think the ColumnDataSource is friction and whether I should just provide functionality to work with DataFrames.

I also don't know how the api should be if I really want to integrate with HoloViews, LinkedBrushing and Streamz. That would be nice to learn.

Pane or Widget

Currently I've implemented this as a Widget with configuration, data and selected_indicies parameters. I'm in doubt if this is a Widget or Pane because I don't really know what the difference is. I'm also in doubt on how much I should mimic the api of panel.widgets.DataFrame for easy of use in Panel.

Code

class Tabulator(Widget):
    configuration = param.Dict(doc="""The Tabulator configuration""")
    data = param.Parameter(
        doc="""One of pandas.DataFrame or bokeh.models.ColumnDataSource.
        If specified it will transfered efficiently to the browser and added to the configuration
        """
    )
    _data = param.ClassSelector(
        class_=ColumnDataSource, doc="Used to transfer data efficiently to frontend" ""
    )
    selected_indicies = param.List(doc="The list of selected row indexes")

    height = param.Integer(default=300, bounds=(0, None))

    _rename = {
        "data": None,
        "_data": "data",
    }
    _widget_type = _BkTabulator

    def __init__(self, **params):
        if "configuration" not in params:
            params["configuration"] = _DEFAULT_CONFIGURATION.copy()
        params["selection"] = []

        super().__init__(**params)

        self._update_column_data_source()

    @param.depends("data", watch=True)
    def _update_column_data_source(self, *events):
        if self.data is None:
            self._data = None
        elif isinstance(self.data, pd.DataFrame):
            self._data = ColumnDataSource(self.data)
        elif isinstance(self.data, ColumnDataSource):
            self._data = self.data
        else:
            raise ValueError("The `data` provided is not of a supported type!")

    @classmethod
    def to_columns_configuration(
        cls, data: Union[pd.DataFrame, ColumnDataSource]
    ) -> List[Dict[str, str]]:
        """Returns a relevant configuration dictionary of the specified data source

        Args:
            data (Union[pd.DataFrame, ColumnDataSource]): The data source

        Returns:
            Dict: The configuration

        Example:

        >>> import pandas as pd
        >>> data = {"name": ["python", "panel"]}
        >>> df = pd.DataFrame(data)
        >>> Tabulator.to_columns_configuration(df)
        [{'title': 'Name', 'field': 'name', 'sorter': 'string', 'formatter': 'plaintext', 'hozAlign': 'left'}]
        """
        col_conf = []
        for field in data.columns:
            dtype = str(data.dtypes[field])
            conf = cls._core(field=field, dtype=dtype)
            col_conf.append(conf)
        return col_conf

    @classmethod
    def _core(cls, field: str, dtype: str) -> Dict[str, str]:
        dtype_str = str(dtype)
        return {
            "title": cls._to_title(field),
            "field": field,
            "sorter": _SORTERS.get(dtype_str, "string"),
            "formatter": _FORMATTERS.get(dtype_str, "plaintext"),
            "hozAlign": _HOZ_ALIGNS.get(dtype_str, "left"),
        }

    @staticmethod
    def _to_title(field: str) -> str:
        return field.replace("_", " ").title()

    @staticmethod
    def config(css: str="default", momentjs: bool=True):
        # pn.config.js_files["tabulator"]=JS_SRC
        # if momentjs:
        #     pn.config.js_files["moment"]=MOMENT_SRC
        if css:
            href = CSS_HREFS[css]
            if href not in pn.config.css_files:
                pn.config.css_files.append(href)

    @property
    def selected_data(self) -> Union[pd.DataFrame, ColumnDataSource]:
        """Returns the selected rows of the data based

        Raises:
            NotImplementedError: [description]

        Returns:
            Union[pd.DataFrame, ColumnDataSource]: [description]
        """
        # Selection is a list of row indices. For example [0,2]
        if self.data is None:
            return None
        if isinstance(self.data, pd.DataFrame):
            return self.data.iloc[self.selected_indicies,]
        if isinstance(self.data, ColumnDataSource):
            # I could not find a direct way to get a selected ColumnDataSource
            selected_data = self.data.to_df().iloc[self.selected_indicies,]
            return ColumnDataSource(selected_data)
        raise NotImplementedError()

FYI. @philippjfr .

[xavArtley]

Did you tried this?
https://discourse.holoviz.org/t/how-do-i-create-and-use-a-custom-bokeh-model-in-panel/122/6

[MarcSkovMadsen]

An alternative approach to providing more grids would be to improve the existing. Maybe it is just better documentation that we need.

• How to make sure it looks and feels nice and is responsive. There are reports/ issues of the opposite?
• How to do complex formatting, sorting, editing etc.
• How to style? Where can I find stylesheets or create my own?
  • https://reebalazs.github.io/SlickGrid-bootstrap/examples/example-bootstrap.html
  • http://jsfiddle.net/eierfiddle/bRF3d/
• How to stream?
• How to patch?

[MarcSkovMadsen]

Hi @xavArtley

Thanks so much for reaching out.

Yes. When using __implementation__ i use actually I need to import my model before calling pn.extension() to make it works. And that works.

The problem is when I want to remove implemention and prebuild things with panel build awesome_panel_extensions. Then it does not register. The project is already a bokeh extension providing the WebComponent. So I thought everything would work. But it does not.

[xavArtley]

I succeed to make prebuilt extensions here
https://github.com/xavArtley/pnbk-extensions/tree/master/pnbkext/lumino

It may be outdated but it used to work

[MarcSkovMadsen]

My code is in the repo https://github.com/MarcSkovMadsen/awesome-panel-extensions.git in the branch tabulator-extension.

panel model: https://github.com/MarcSkovMadsen/awesome-panel-extensions/blob/tabulator-extension/awesome_panel_extensions/widgets/tabulator.py
bokeh .py model: https://github.com/MarcSkovMadsen/awesome-panel-extensions/blob/tabulator-extension/awesome_panel_extensions/bokeh_extensions/tabulator_model.py
bokeh. ts model: https://github.com/MarcSkovMadsen/awesome-panel-extensions/blob/tabulator-extension/awesome_panel_extensions/bokeh_extensions/tabulator_model.ts

I panel build awesome_panel_extensions to build the extensions.

If i look into dist/awesome_panel_extensions.js the TabulatorModel is there.

I've also tested that the WebComponent model is still registered and working. So some parts of the build works.

But if I `python 'tests\widgets\test_tabulator\test_tabulator_designer.py' it does not work

@mattpap or @philippjfr . If you have some thoughts I would really appreciate it because I've run out of random changes/ ideas to try out.

Additional Context

[mattpap]

You need to set __module__ in the model's implementation to match Python's full module path. Fully qualified model paths from bokeh and bokehjs must match, otherwise bokehjs won't be able to resolve a model, as it doesn't assume anything about the structure of your extension.

[mattpap]

We need to make this clearer in bokehjs, either at compilation or usage time, because skipping __module__ in an extension is almost always incorrect.

[MarcSkovMadsen]

Thanks so much @mattpap .

One day I would really like to learn from you or somebody else how to efficiently develop bokeh extensions. How to use integrated debugging, setup automated testing, live reload etc. Right now my development flow is too slow.

One thing I should do is add some snippets in VS Code for generating the bokeh .ts and .py models. I have some for the Panel .py extensions. And it makes things much more easy.

One thing I've thought about is that some of the template you can get in a front end framework like angular via ng generate component new-model could also be awesome to do for Bokeh/ Panel. That would speed up things.

[philippjfr]

Tabulator was merged as part of the 0.11 release and has been great!