Python API 1.0 acceptance tests are in place and can be executed easily.

[pablo-gar]

It is left to the discretion of the engineer working on this to assess which ones should also be converted to unit tests. Please reach out to me if you'd like to discuss

Initial candidate of tests:

• For both organisms ,full obs, var and X can be iterated e.g. for chunk in ["obs"].read(): ...
• For both organisms, full obs and var can be loaded in a pandas.DataFrame e.g. ["obs"].read().concat().to_pandas
• Experiment query operations can be done over a large slice of data, e.g. with .ms["RNA"].axis_query(tiledbsoma.AxisQuery(value_filter = "tissue == 'brain') then perform to completion the main ExperimentQuery operations:
  • for chunk in obs(): ...
  • for chunk in var(): ...
  • for chunk in X().tables(): ...
• Large AnnDatas can be loaded, document computing requirements:
  • largest possible in <20 min @bkmartinjr has made it happened in an EC2 instance r6id.32xlarge get_anndata(census = census, organism = "Homo sapiens",)
  • most common cell type get_anndata(census = census, organism = "Homo sapiens", obs_value_filter = "cell_type == 'neuron'")
  • most common tissue get_anndata(census = census, organism = "Homo sapiens", obs_value_filter = "cell_type == 'brain'")

[atolopko-czi]

These are all "acceptance" tests, in fact. Even if not run in an automated fashion, it would seem good to have them all scripted so they can be run on an arbitrarily-sized EC2 instance as a single step.

[bkmartinjr]

Agree. These are not stress tests. For an example issue found by a stress test, see: single-cell-data/TileDB-SOMA#1169 Our census builder/validator is a pretty good stress test all by itself.

@atolopko-czi - if we do automate the above, do you have any recommendations about how we record performance/behavior so that regressions/changes can be tracked?

[atolopko-czi]

@atolopko-czi - if we do automate the above, do you have any recommendations about how we record performance/behavior so that regressions/changes can be tracked?

I'd be fine with committing a log file to the repo. It would contain the relevant context info (instance type, OS, s/w versions for Cell Census, TileDB-SOMA, Python, etc.) and the test results (pass/fail, applicable result size counts, timings).

We could potentially wrap this all up in a notebook that gets run at the same time as the pedagogical notebooks, but honestly I'm not sure that's a great fit.