Move to keyset pagination (#220)

* move to keyset pagination

* linting

* adjustments for further data integration

* test adjustments

* linting corrections

* run dev workflow

* fix docs

* remove custom sql join for large data test

* test with rowcount update

* linting, docs, cleanup

* cleanup

* remove test branch

* rework pageset generation

* linting

* fix types and docstring

Co-Authored-By: Faisal Alquaddoomi <[email protected]>

* docs and types

* update to use pageset instead of chunk fxn names

Co-Authored-By: Jenna Tomkinson <[email protected]>

* update variable names

Co-Authored-By: Gregory Way <[email protected]>

* better exception docs

* better docstring parameter description

Co-Authored-By: Gregory Way <[email protected]>

* better documentation and exception msg

* remove exception

* point to the example in docs

Co-Authored-By: Gregory Way <[email protected]>

* add further notes about when/why to customize

* linting

* remove test raise

* sorted results from concatenation by natural sort

---------

Co-authored-by: Faisal Alquaddoomi <[email protected]>
Co-authored-by: Jenna Tomkinson <[email protected]>
Co-authored-by: Gregory Way <[email protected]>

cytotable/constants.py

@@ -68,13 +68,6 @@
     ],
 }
 
-# metadata column names and types for internal use within CytoTable
-CYOTABLE_META_COLUMN_TYPES = {
-    "cytotable_meta_source_path": "VARCHAR",
-    "cytotable_meta_offset": "BIGINT",
-    "cytotable_meta_rownum": "BIGINT",
-}
-
 CYTOTABLE_DEFAULT_PARQUET_METADATA = {
     "data-producer": "https://github.com/cytomining/CytoTable",
     "data-producer-version": str(_get_cytotable_version()),