-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Croptype #71
Open
gabrieltseng
wants to merge
193
commits into
main
Choose a base branch
from
croptype
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Croptype #71
Changes from 4 commits
Commits
Show all changes
193 commits
Select commit
Hold shift + click to select a range
bf5dda2
added code for multiclass finetuning Presto for croptype task
7fd1dde
fixed result collection for sklearn models
6090b9f
added hierarchical classifier v.0 to downstream models
ea4b2b6
added patches for handling valid_date as token; added more updated de…
7285c38
major change: spatial prediction for croptype + looots of minor changes
1601dbc
replaced confusing argument name; improved formatting
586db66
added valid_month parameters to default config
532ec67
added placeholder for loading finetuned model
a350280
class name constructed from task_type; cleaned unused pieces
0cf1e60
added a line to mask latlons for hackey generalizability test
725e2af
formatting & cleaning
5f5d296
updated test split files
6ef5c14
bug fix
2440203
bug fixes
9b6088d
bug fixes and default argument updates
4417a3b
implemented simple balancing for croptype; CAUTION: makes training MU…
38907b0
switched class balancing from finetune_class to a new balanced_class;…
406fb16
Merge branch 'main' into croptype
kvantricht ee1b2c5
black fixes
kvantricht 85e336e
isort fixes
kvantricht be27740
Black fix
kvantricht 6f25105
Black fix
kvantricht 8795d27
changed default from None to empty string, according to mypy suggesti…
019f37c
added missing imports
e682da2
changed conflicting variable name
f7790a4
fixed expected output types
e173157
removed duplicated function
6a9b07f
edited model loading to use updated loading function
e691a32
bug fix and mypy fixes
4ff8846
mypy typing fixes
d1c1687
another round of black fixes
c53c1b8
flake fixes and additional cleanup
4d9c39d
minor changes to pass ruff checks
4f0d8c7
formatting fixes
c684c2a
added balance as an argument
297b44d
added other_class to CROPTYPE19
b511938
added valid_time attribute handling for compatibility of Phase I and …
0c9e34b
fixed parsing balance as a bool
2c5e778
bug fixes
788d444
isort version fixes 🤦♀️
3afa345
another version black fix 😪
aead839
removed unused import
f79a904
removed unnecessary assignment
d5f4206
Merge branch 'main' into croptype
kvantricht 3a27024
Merge branch 'main' into croptype
kvantricht 4bc5995
Merge branch 'main' into croptype
kvantricht c0c06ee
fixes to pass test_dataset tests
d9cbc0c
fixes for test_eval tests
29edb75
fixes to pass test_presto tests
959269c
fixed reversed mapping of VV and VH 🤦♀️
748c40e
new asserts for new output format
5ac1448
added croptype eval test
66751c2
modified lr to be different for cropland/croptype training
3a68645
changed to cleaner json handling
f6a5cec
updated test_df to include more croptype samples for croptype tests
bf35cf4
isort fixes
ed85037
changing lr for croptype properly
f82be5a
added spatial inference test for croptype prediction
57b5099
formatting
753dc9a
resolving black version formatting
4f40b1f
resolving black version formatting v.2 😔
df8dcb8
added hiclass package to requirements
d72d819
Merge branch 'main' into croptype
kvantricht db561fa
uncommented model saving 🤦♀️ + minor changes
8a37d6e
substituted ifs with elifs as per Gabis suggestion
f5df76b
completely ignore catboost_info folder
751a919
removed catboost info folder
158012f
removed unnecessary commented lines
643c129
moved target_crop method into the WorldCerealLabelledDataset class as…
74525fc
breaking long lines
9043e8c
isort fixes
d0ada7a
putting target_crop back into WorldCerealBase class ☹
19e51b6
formatting fixes
59d2b3d
fixed computing valid_date_ind so that it's more robust; added fillin…
181ae8b
introduced MIN_SAMPLES_PER_CLASS parameter so that it can be reused i…
12ce57c
added additional balancing parameters; optimal values TBD
80b8313
add nans handling in metrics calculation
48c17e8
disentangled the device confusion in tests. Thanks Gabi!
75ff5f6
moved target_crop into WorldCerealLabelledDataset
47f581c
replaced model_mode parameter with a more transparent one; done some …
0f6df1d
formatting fixes
523014f
removed process_parquet function to utils
5c74855
addeded augment parameter
cb197a3
added function for timeseries subsetting, so that it is centered arou…
9e9ac9d
added augment parameter; replaced default link to new parquet file; a…
b3e0284
major rework of process_parquet function; minimal viable functionality
1d37536
moved MIN_EDGE_BUFFER parameter from utils to dataset.py
ae27e25
added logger message about enabled augmentation
8d7e4c1
removed augment=False parameter from evaluate function, since it is a…
b2f1aa3
rephrased checking if valid_date is too close to the edge without mes…
ff25509
bugs and typos fixes
e338c09
moved NODATA and MIN_EGDE parameters to dataops.py to avoid circular …
c9ffa01
updated test dataset to use new ong parquet format
f6e1a9e
updated tests
407cec9
created separate test file for process_parquet function
ad05b3d
an attempt to make time_token shift more general than just for months
91bfd27
merging main to croptype
e1b90f7
black fix
2945e9d
merging changes from main
ef06f94
black fix
30ab19c
adding test long parquet file
96dbc0d
fixed test file path
55dbbbe
isort fix
a7eedd8
fixed test and commented lines that will not be needed after merge
6f7646f
Formatting
kvantricht 203c4ac
Formatting
kvantricht 465d65a
making GT values binary crop/nocrop
54cc2be
Test with 1 epoch finetuning
kvantricht e44445a
Merge branch 'using-new-parquet-in-train' of github.com:WorldCereal/p…
kvantricht 1a957a2
Bump einops version
kvantricht c83f7dc
created different py files for ss training and finetuning
5bf3743
fixed plotting functionality for new patches format
713a15a
removed unnecessary line
f8d9f84
fixed masking bug
c702081
fixed usage of time token during finetuning
100e606
added milder handling for lower mask_ratios
eda6d37
added logger messaging about balancing
eed2463
fixed bug in plotting
36f75a4
bug fixes and cleanup
2c0c325
added logging for masking and time token usage
e205514
bug fixes and cleanup
b216d3f
fixed SSL
aba07b0
added basic test to check balancing
9dab52d
fixed timestep_positions function for ssl
579da2f
added tests for temporal shift
1f6cb27
test fixes
8215473
isort fixes
14b1abe
formatting fixes
14cdce0
another version of black fixes 🤦♀️
767b187
Bump version
kvantricht b69eb3f
dont import matplotlib globally
kvantricht 75cea4b
#108 avoid global import of `CLASS_MAPPINGS`
kvantricht 08313a7
Remove unused import
kvantricht b25537b
Run tests with less CatBoost iterations
kvantricht e7147f3
Formatting fix
kvantricht 094fc2b
added handling of corner case when during SSL we only have 12 timeste…
7eaa8ba
added a slightly better explanation of valid_position variable
fb255cf
fixed ndvi masking
bf76d57
formatting fixes
64c84fc
Allow running inference without valid_date token
kvantricht 188f093
bug fix
f37314a
added proper NDVI masking to InferenceDataset + test
c566f37
formatting fixes
891a7ac
formatting fixes
6100d33
added corrected patch that starts with first day of month
ea28bf5
regenerated test features file
6c8c0a0
#109 pass `augment` argument
kvantricht c1eadb5
Add location_id and ref_id to processed parquet
kvantricht f8b0807
Add `ref_id` to test parquet
kvantricht d47ddfb
avoid if-else
kvantricht 5cb8898
Avoid if else
kvantricht 923340a
Avoid if-else
kvantricht 8c88c56
Formatting
kvantricht ce3fae1
Run actions on PR to croptype
kvantricht ed60335
reintroduced ref_id into dataset and made cleaner logger message abou…
373e872
fixing the number of available_timesteps
f03649e
fixed available_timesteps computation for corner cases
c59c066
cleanup
3ba98a3
formatting
e806c28
additional check on the available_timesteps + descr
eee7dc5
isort fix, hopefully the correct version
f03c1cd
Check nr of timesteps in inference
kvantricht b051404
Merge branch 'timestep-position-debugging' of github.com:WorldCereal/…
kvantricht 865ab1e
Attempt to auto-format
kvantricht add376a
Should be f-string
kvantricht e5e0109
Moved import to top
kvantricht 89d72a4
Merge pull request #114 from WorldCereal/timestep-position-debugging
cbutsko e163e9f
removed unnecessary lines that double the size of embeddings
6780a83
added loading of finetuned model
6b904e8
slightly cleaner handling of valid_month token
66e0c72
changed strict to True during model loading
069ede7
enhanced plotting
8702c7f
updated masking not to take into account existing mask
f96f78d
turning on augmentation for downstream model
80bae7b
Bugfix: use `valid_month_as_token` kwarg
kvantricht 2ef8dde
Formatting fixes
kvantricht ea042e7
Formatting fix bis
kvantricht 205fd76
added tests for both for using valid_month token and not
8cd4d15
reverting masking changes for now; need to make sure it does not affe…
e4fdac4
changed default value of valid_month_as_token to False when loading m…
059bd6b
added valid_month related tests
641c8c5
commented lines that create ref feature files
e97fc70
added new reference feature files for with and without valid_month
2d889ee
removed unnecessary prints
6c43ceb
formatting
024ffc3
fixed test for valid_month token
6f05047
create ref feature files
34a026b
fixed tests
6b09d83
a very brave attempt to mess with encoder compile 🙈
e9cbfa8
removing redundant creation of valid_month token when the flag is False
080e20c
removed obsolete TODOs
a75adbf
Merge pull request #115 from WorldCereal/valid_month-and-mask-debugging
cbutsko aa2f74d
Bump version number to 0.1.6
kvantricht File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,4 +17,6 @@ gha-creds-*.json | |
| .idea | ||
| scrap | ||
| output/* | ||
| imgs/* | ||
| imgs/* | ||
| # don't track catboost training info | ||
| */catboost_info | ||
Large diffs are not rendered by default.
Oops, something went wrong.
Binary file not shown.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
catboost_infolooks like it is in git - should that folder be removed from git?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should indeed not be in git.