Add an IbisInterface

[philippjfr]

Adds an IbisInterface built on the Ibis project which will let us interface directly with various database backends.

So far this is a basic skeleton which we can fill in bit by bit. The first item for discussion is what object a HoloViews Dataset should wrap. To me it seems most appropriate for it to wrap any Expr object which represents some expression on a Database, e.g. the selection of an individual Table or any other operation. The question then is what types of expressions should be supported, e.g. can or should we support a ColumnExpr or must the basic unit always represent a table.

Implementation status

Core methods

• init: Basic constructor
• range: Computes the range along a column
• values: Returns all values along a column (or only unique values)
• length: Returns number of rows in the table
• nonzero: Whether there are any rows in the table
• dtype: Return the dtype of the column
• isscalar: Checks whether column value has a single unique value
• select: Select given boolean mask or given scalar, range or list selection(s)
• groupby: Returns one or more Datasets grouped by the values along one or more columns
• redim: Rename columns in the table
• iloc: Integer indexing of rows and columns
• sort: Sort by one or more columns ascending/descending
• aggregate: Aggregate along one or more columns with given function (e.g. mean, min, max)
• sample: A list of selections

Write methods (these methods write back to the data)

• add_dimension: Add a column to the table (can this be supported?)
• mask: Mask out values given a boolean mask
• assign: Add or overwrite one or more columns

Conversion methods

• array: Return a NumPy array of requested columns

Tests Status

Passing: 73
Error: 34
Fail: 0
Skip: 16

Cc: @kcpevey @tonyfast

[MarcSkovMadsen]

I have been thinking of HoloViews/ Panel and Ibis lately on the back of a blog by quantstack.

I cannot read from the above the why. So i hope you will write a bit about the why or vision.

For me the vision would be to make it very easy to use a wide range of data sources within HoloViz. Similarly to how easy it is to use BI tools like Tableau with All kinds of data sources.

And maybe this interface could be used to only load the data needed to display in the plot or do the aggregation on the backend server.

For me unfortunately Ibis does not support sql server or snowflake which means i cannot use it in my Daily Work.

[jbednar]

Yes, the vision is to be able to instantiate a HoloViews element where the "data" is lazy, just a query and not the actual data, with the data fetched dynamically when it is actually needed for plotting or analysis. That does indeed set HoloViews up to do some more of the things that BI tools do.

If you have any funding, I would imagine that Quansight would add support to Ibis for sql server, and maybe snowflake...

[kcpevey]

@MarcSkovMadsen Adding additional support in Ibis is definitely something we at Quansight would be open to discussing.

[tonyfast]

There is some WIP on a microsoft sql server interface ibis-project/ibis#1997

[philippjfr]

This is starting to look good. I've got a few comments but will hold off until you tell me it's ready.

[tonyfast]

I'd love to take some comments soon. I'm nearing the end of the skeleton and starting to understand the model better.

Is my PR what is breaking the travis tests?

[philippjfr]

I'll do a high-level review today then. While I do think this PR is probably failing tests, the tests are also generally broken today. I'll merge a PR to get the tests running again and then you can rebase.

…

On Tue, 4 Aug 2020, 04:44 Tony Fast, ***@***.***> wrote: I'd love to take some comments soon. I'm nearing the end of the skeleton and starting to understand the model better. Is my PR what is breaking the travis tests? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4517 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL2TM7EOTNLY2NVHKXKAE3R65YZTANCNFSM4OWCXQ2Q> .

[philippjfr]

Will have a go at this now.

[philippjfr]

Huge amount of progress, 73 tests passing and 34 errors.

[philippjfr]

Okay the only real problems remaining:

• iloc doesn't work
• groupby and aggregate do not preserve ordering (this is okay, just needs to be documented since the same is true for cuDFs).

[tonyfast]

Any ideas why the appveyor ci is getting mad at the sqlite connections? I can't seem to find a solution.

[tonyfast]

I don't think it is possible to avoid sorting for some ibis backends. what should be done in this case? is it likely that test_dataset_groupby_second_dim is failing because of sorting problems?

[philippjfr]

That's fine, we just have to document it and update the test. The cuDF interface has the same problem currently.

[dharhas]

Looks like travis_wait 30 doit develop_install $CHANS_DEV -o $HV_REQUIREMENTS is failing, If I understand right this is waiting 30 mins and then dying so 30 mins is isn't long enough for this command to execute. Not sure why this command is taking greater than 30 mins though.

[philippjfr]

Looks like travis_wait 30 doit develop_install $CHANS_DEV -o $HV_REQUIREMENTS is failing, If I understand right this is waiting 30 mins and then dying so 30 mins is isn't long enough for this command to execute. Not sure why this command is taking greater than 30 mins though.

I'll rebase, I fixed that one master. Pulling from conda-forge made it very slow.

[goanpeca]

I'll rebase, I fixed that one master. Pulling from conda-forge made it very slow.

Hi @philippjfr, I am looking with @tonyfast into the failing tests. Had to reset a commit.

[philippjfr]

I had dropped conda-forge because it was taking forever to solve in the Py2 build. Will see if we can get around that somehow, otherwise let's just increase the timeout. Looks like we're still getting the NoWindowOp errors, still the wrong version of Ibis?

[tonyfast]

naw. i'm doing something wrong. i had to add ibis builds from conda-forge cause i couldn't get it from anywhere the other channels. I've got to keep trying things otherwise I'll likely have to duck some of the iloc tests until a new version of ibis shows up.

[philippjfr]

'll likely have to duck some of the iloc tests until a new version of ibis shows up.

Totally fine, I'd suggest adding a decorator that checks for a specific version:

from unittest import skipIf
ibis_skip = skipIf(ibis is None or ibis_version < '...', 'Ibis not available or wrong version')

[kcpevey]

The sister PR to add rowid support in Ibis has been merged ibis-project/ibis#2345

[philippjfr]

Thanks for all your efforts here @kcpevey and @tonyfast! I'm merging now and hoping that we can get some docs in before release.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.