add support for sparse X

[apoorvalal]

Adds support for CSR matrix inputs for all learners [not supported elsewhere, e.g. this open issue on econml]. Turned out to be not too bad since Kfold, integer slicing all work as usual with CSR matrices.

I added a safe_len method in utils and replaced all instances of len(X) with safe_len(X) (hence the many files touched - turned out no major substantive changes were required). This also means that the Matrix class definition in typing now unions over array, dataframe, and csr_matrix.

Also added an example notebook that documents massive (~8-10x) speed gains from using sparse inputs. I can't think of any obvious side-effects, but I'd defer to you folks on whether there are edge cases I haven't considered here.

Checklist

• Added a CHANGELOG.rst entry

[apoorvalal]

For completeness, I also tried the same estimation routine with pandas.arrays.SparseArray returned by pd.get_dummies when sparse=True but was unreasonably slow [I terminated the job after ~5 mins]. LGBM may not even support it, but if so the failure was silent.

[apoorvalal]

docs build was previously failing due to mypy getting annoyed at cell magics, is now failing due to

happens only on the build system; local run of postinstall works fine.

will likely need some metadata tweaking; will try to take care of this later but feel free to change things around [you have write access to my fork and can make changes directly]

edit: fixed ; made the sparse problem quite small (which makes the runtime difference smaller, but that's nbd)

[kklein]

Just minor nitpicks; looking great! :)

Would you mind adding the following to the top of changelog.rst?

0.11.0 (2024-09-xx)
-------------------

**New features**

* Add support for using ``scipy.sparse.csr_matrix`` as datastructure for covariates ``X``. 

[kklein]

LGTM - thanks a lot! :)

Feel free to squash and merge

[apoorvalal]

don't think i have the privileges; would be great if you could.

thanks again; excited to use this.

[kklein]

We might add 1-2 changes at the beginning of next week and will probably release version 0.11.0 by end of next week. :)