Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyTorch DataLoader #499

Merged
merged 35 commits into from
May 30, 2023
Merged

PyTorch DataLoader #499

merged 35 commits into from
May 30, 2023

Commits on May 18, 2023

  1. add PyTorch DataLoader support

    introduce this under a new "incubation" sub-package
    atolopko-czi committed May 18, 2023
    Configuration menu
    Copy the full SHA
    fa1024c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2080d40 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    95ba2bb View commit details
    Browse the repository at this point in the history

Commits on May 19, 2023

  1. instance var init fix

    atolopko-czi committed May 19, 2023
    Configuration menu
    Copy the full SHA
    abcc380 View commit details
    Browse the repository at this point in the history
  2. rm explicit timeout param handling

    user to specify as normal DataLoader param
    atolopko-czi committed May 19, 2023
    Configuration menu
    Copy the full SHA
    cb37af4 View commit details
    Browse the repository at this point in the history
  3. minor fixes, comments

    atolopko-czi committed May 19, 2023
    Configuration menu
    Copy the full SHA
    2e4b0bd View commit details
    Browse the repository at this point in the history

Commits on May 22, 2023

  1. Configuration menu
    Copy the full SHA
    895fcb5 View commit details
    Browse the repository at this point in the history
  2. swap order of obs and labels in DataLoader rows/batches

    "X, y" is common ordering in ML training usage
    atolopko-czi committed May 22, 2023
    Configuration menu
    Copy the full SHA
    e33af14 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. support access to ExperimentAccessQuery from ExperimentDataPipe

    allows model training code access query dim sizes
    
    required refactoring to initialize ExperimentDataPipe when needed
    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    dd55c7d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    60e0b9d View commit details
    Browse the repository at this point in the history
  3. refactoring

    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    1084482 View commit details
    Browse the repository at this point in the history
  4. take Experiment obj instead of URI

    cleans up experiment open logic, providing s3 region param
    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    d766785 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    fdeb52c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c66bdb3 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    080e5ec View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    70aec84 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    e0968b1 View commit details
    Browse the repository at this point in the history
  10. support split and shuffle via PyTorch DataPipes

    separate the ExperimentDataPipe instantation from the experiment_data_loader()
    method, to allow user to specify additional pipe operations, per her use case.
    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    ad0356e View commit details
    Browse the repository at this point in the history
  11. TODO

    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    891684c View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    503a7dc View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    b51b4a9 View commit details
    Browse the repository at this point in the history
  14. refactor var name

    atolopko-czi committed May 23, 2023
    Configuration menu
    Copy the full SHA
    94778ea View commit details
    Browse the repository at this point in the history

Commits on May 24, 2023

  1. fix GHA builds

    atolopko-czi committed May 24, 2023
    Configuration menu
    Copy the full SHA
    7f617f2 View commit details
    Browse the repository at this point in the history
  2. invert sparse_X option to dense_X

    Sparse Tensors are experimental and are currently known not to work with
    multiprocessing or LR models. So let's make use of dense Tensors the
    default. Use of sparse Tensors now requires the explicit option
    `sparse_X=True`.
    
    Also fix bug where dense, non-batched results were returned as 1-row batches,
    instead of being unwrapped.
    atolopko-czi committed May 24, 2023
    Configuration menu
    Copy the full SHA
    f6c0f78 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6313ee9 View commit details
    Browse the repository at this point in the history
  4. run experimental tests conditionally (#501)

    * experimental tests only run with explicit pytest option
    * only run experimental tests with Python 3.10 on GHA, since not all experimental package dependencies are available for lower version (e.g. pytorch 2.0.1 not available for Python 3.7)
    atolopko-czi authored May 24, 2023
    Configuration menu
    Copy the full SHA
    4bbb3b8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c027b7d View commit details
    Browse the repository at this point in the history

Commits on May 25, 2023

  1. docs

    atolopko-czi committed May 25, 2023
    Configuration menu
    Copy the full SHA
    1c3a736 View commit details
    Browse the repository at this point in the history
  2. use API's default buffer_bytes value

    rename buffer_bytes to soma_buffer_bytes for clarity
    atolopko-czi committed May 25, 2023
    Configuration menu
    Copy the full SHA
    8cfc59b View commit details
    Browse the repository at this point in the history
  3. refactoring

    - rename ms_name to measurement_name
    - remove ExperimentDataPipe num_workers param (unused except for an invalid usage check)
    atolopko-czi committed May 25, 2023
    Configuration menu
    Copy the full SHA
    8ce019f View commit details
    Browse the repository at this point in the history
  4. refactoring

    rename layer_name to X_name, to align with Census API get_anndata
    atolopko-czi committed May 25, 2023
    Configuration menu
    Copy the full SHA
    553423d View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    51ba4bb View commit details
    Browse the repository at this point in the history
  6. update TODO

    atolopko-czi committed May 25, 2023
    Configuration menu
    Copy the full SHA
    ce3abe7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    067893b View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. Configuration menu
    Copy the full SHA
    78040c5 View commit details
    Browse the repository at this point in the history