Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using new parquet in train #104

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from
Draft

Using new parquet in train #104

wants to merge 27 commits into from

Commits on Sep 10, 2024

  1. removed process_parquet function to utils

    Christina Butsko committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    523014f View commit details
    Browse the repository at this point in the history

  2. addeded augment parameter

    Christina Butsko committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5c74855 View commit details
    Browse the repository at this point in the history

  3. added function for timeseries subsetting, so that it is centered arou… 

    …nd valid_date, with the possibility to augment the latter
    Christina Butsko committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    cb197a3 View commit details
    Browse the repository at this point in the history

  4. added augment parameter; replaced default link to new parquet file; a… 

    …dded looping through individual parquets for performance reasons
    Christina Butsko committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    9e9ac9d View commit details
    Browse the repository at this point in the history

  5. major rework of process_parquet function; minimal viable functionality

    Christina Butsko committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    b3e0284 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. moved MIN_EDGE_BUFFER parameter from utils to dataset.py

    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    1d37536 View commit details
    Browse the repository at this point in the history

  2. added logger message about enabled augmentation

    Christina Butsko committed Sep 11, 2024
    1 Configuration menu
    Copy the full SHA
    ae27e25 View commit details
    Browse the repository at this point in the history

  3. removed augment=False parameter from evaluate function, since it is a… 

    …nyway False by default
    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    8d7e4c1 View commit details
    Browse the repository at this point in the history

  4. rephrased checking if valid_date is too close to the edge without mes… 

    …sing with augment parameter
    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b2f1aa3 View commit details
    Browse the repository at this point in the history

  5. bugs and typos fixes

    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ff25509 View commit details
    Browse the repository at this point in the history

  6. moved NODATA and MIN_EGDE parameters to dataops.py to avoid circular … 

    …imports and failing tests
    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    e338c09 View commit details
    Browse the repository at this point in the history

  7. updated test dataset to use new ong parquet format

    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    c9ffa01 View commit details
    Browse the repository at this point in the history

  8. updated tests

    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f6e1a9e View commit details
    Browse the repository at this point in the history

  9. created separate test file for process_parquet function

    Christina Butsko committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    407cec9 View commit details
    Browse the repository at this point in the history

  10. an attempt to make time_token shift more general than just for months

    Christina Butsko committed Sep 11, 2024
    1 Configuration menu
    Copy the full SHA
    ad05b3d View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. merging changes from main

    Christina Butsko committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2945e9d View commit details
    Browse the repository at this point in the history

  2. black fix

    Christina Butsko committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ef06f94 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. adding test long parquet file

    Christina Butsko committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    30ab19c View commit details
    Browse the repository at this point in the history

  2. fixed test file path

    Christina Butsko committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    96dbc0d View commit details
    Browse the repository at this point in the history

  3. isort fix

    Christina Butsko committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    55dbbbe View commit details
    Browse the repository at this point in the history

  4. fixed test and commented lines that will not be needed after merge

    Christina Butsko committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    a7eedd8 View commit details
    Browse the repository at this point in the history

  5. Formatting

    @kvantricht
    kvantricht committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6f7646f View commit details
    Browse the repository at this point in the history

  6. Formatting

    @kvantricht
    kvantricht committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    203c4ac View commit details
    Browse the repository at this point in the history

  7. making GT values binary crop/nocrop

    Christina Butsko committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    465d65a View commit details
    Browse the repository at this point in the history

  8. Test with 1 epoch finetuning

    @kvantricht
    kvantricht committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    54cc2be View commit details
    Browse the repository at this point in the history

  9. Merge branch 'using-new-parquet-in-train' of github.com:WorldCereal/p… 

    …resto-worldcereal into using-new-parquet-in-train
    @kvantricht
    kvantricht committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e44445a View commit details
    Browse the repository at this point in the history

  10. Bump einops version

    @kvantricht
    kvantricht committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    1a957a2 View commit details
    Browse the repository at this point in the history