Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1388 some ecmwf products have invalid default parameters #1389

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

1388 some ecmwf products have invalid default parameters #1389

wants to merge 9 commits into from

Conversation

amarandon
Copy link
Collaborator

@amarandon amarandon commented Nov 6, 2024 •
edited
Loading

Fix metadata configuration of some ECMWF products to match what is actually available on the data stores. In 2 cases the start date when wrong. In other cases, 2 other parameters were invalid for the given start date.

@amarandon amarandon linked an issue Nov 6, 2024 that may be closed by this pull request
Some ECMWF products have invalid default parameters #1388
Open
@amarandon amarandon force-pushed the 1388-some-ecmwf-products-have-invalid-default-parameters branch from cc294ad to 772734d Compare November 6, 2024 12:48
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024 •
edited
Loading

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      429      27  93.71%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      69  89.85%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 976, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   159-161, 208-209, 235-237
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       253      51  79.84%   136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       537     108  79.89%   235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324
plugins/download/s3rest.py                     116      24  79.31%   118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      13  90.15%   100, 104, 115, 274, 294, 350-351, 371, 374-382
plugins/search/build_search_result.py          188      24  87.23%   100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 512, 514
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     774     101  86.95%   458, 516, 530, 534-540, 568-572, 681-693, 737, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1987-1990, 1995-1998, 2007, 2037-2041, 2047
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   251      63  74.90%   216-217, 273, 281, 299-316, 331-369, 460, 502-533, 680, 687-735
rest/errors.py                                  69       5  92.75%   106, 116, 127, 143-144
rest/server.py                                 188      24  87.23%   89, 112-114, 277-282, 310, 495-497, 514-519, 548, 550, 554-555, 559-560
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176       5  97.16%   225-229, 282, 285, 353
rest/types/queryables.py                        55       1  98.18%   163
rest/types/stac_search.py                      125       7  94.40%   138, 184, 199-201, 209, 213
rest/utils/__init__.py                          93      12  87.10%   108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              114      14  87.72%   65, 82, 141-144, 211, 225-234, 244, 265, 278
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      38  92.43%   85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1027-1028, 1151
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9734    1593  83.63%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 8bff4bd

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024 •
edited
Loading

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      429      28  93.47%   83-85, 94, 102, 106-108, 179, 190, 695-697, 810-813, 856-857, 866-867, 972, 1003, 1035-1040, 1042
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    767      69  91.00%   374, 664, 708-711, 749, 793, 827, 872-877, 903, 994, 1062, 1200, 1285-1297, 1333, 1335, 1363, 1367-1378, 1391-1397, 1480-1483, 1516-1536, 1588, 1605-1609, 1621-1624, 1980, 2004-2010, 2261, 2265-2269, 2278-2280, 2312
api/search_result.py                            58       4  93.10%   92, 101, 108, 122
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      69  89.85%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 976, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             172      15  91.28%   116-121, 171, 209, 231, 235, 259, 281, 399-402, 414-415
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   159-161, 208-209, 235-237
plugins/apis/usgs.py                           180      31  82.78%   155, 257, 291, 326-328, 333, 359-360, 365, 395-402, 413-418, 440-446, 448-454, 477
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   51, 61
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   151-154, 175-180
plugins/authentication/oauth.py                 13       7  46.15%   42-44, 48-51
plugins/authentication/openid_connect.py       206      27  86.89%   80-81, 93-111, 149, 155-183, 191, 323-326, 352
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 92      16  82.61%   114, 143, 145, 176-189, 245-249
plugins/authentication/token_exchange.py        36      14  61.11%   75, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 102-104, 113-115, 122
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       7  76.67%   55-60, 63-64, 80-84
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   282, 295, 362-365, 379-383, 425-427, 431, 463-464, 470-474, 503, 535, 539, 546, 576-584, 588, 620-628, 639-641, 672-746, 764-824, 835-840, 852-865, 890, 905-907, 910, 920-928, 936-949, 959-990, 997-1009, 1047, 1073, 1118-1120, 1340
plugins/download/base.py                       253      53  79.05%   136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   53-67
plugins/download/http.py                       537     109  79.70%   235, 331-334, 336-337, 344-349, 367-382, 399-401, 413, 461, 468-474, 492, 506, 520, 528-530, 546-551, 562, 580, 623-626, 648, 688, 733, 747-753, 782-846, 864, 894-903, 925-926, 955-960, 966, 969, 986, 1003-1004, 1017, 1034-1035, 1042, 1104, 1119, 1178-1179, 1185, 1195, 1231, 1267, 1287, 1322-1324
plugins/download/s3rest.py                     116      24  79.31%   118, 154, 161, 196, 223-230, 233-235, 239, 250-256, 264-265, 268-272, 295, 316-319
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      14  89.39%   100, 104, 115, 274, 294, 350-351, 371, 374-382, 384
plugins/search/build_search_result.py          188      31  83.51%   100, 141-142, 148, 159, 292-295, 324, 359-376, 425, 454, 457, 467, 484, 504-519
plugins/search/cop_marine.py                   240      47  80.42%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 218, 224, 228, 232, 245, 256-257, 265, 293, 297, 312, 316, 320, 324, 328-332, 338-341, 344-358, 375, 424-428, 433, 445
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          202      69  65.84%   188-191, 207, 218, 222-223, 234, 239, 244, 251, 264-267, 321-322, 326, 336-342, 347, 373-376, 384-395, 412, 414, 421-424, 426-427, 445-449, 482, 492, 503, 516, 522-537, 542
plugins/search/qssearch.py                     774     135  82.56%   458, 516, 530, 534-540, 568-572, 681-693, 737, 740, 753, 763, 782-797, 834-837, 908-909, 957, 976, 983, 995, 1052, 1073, 1076-1077, 1086-1087, 1096-1097, 1106-1107, 1134, 1205-1210, 1214-1223, 1257, 1279, 1356, 1446, 1533, 1536-1537, 1619-1623, 1685, 1688, 1694-1695, 1716, 1743-1755, 1762, 1794-1796, 1806-1812, 1842, 1865, 1870-1871, 1886, 1902, 1970-2076
plugins/search/static_stac_search.py            75      13  82.67%   98-124, 140, 153
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   251     139  44.62%   162, 164, 166, 169-170, 184-194, 206-207, 209-210, 216-217, 220, 223, 264-318, 331-369, 400-434, 449-465, 481-490, 502-533, 550, 592-641, 680, 687-735
rest/errors.py                                  69      49  28.99%   60, 65-100, 105-108, 115-118, 126-147, 155-160, 175-181
rest/server.py                                 188     188  0.00%    18-573
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176      16  90.91%   225-229, 262-264, 282, 285, 291, 295, 353, 370-380
rest/types/queryables.py                        55      13  76.36%   50-51, 58-59, 66-67, 93-98, 107-108, 163
rest/types/stac_search.py                      125      11  91.20%   136-138, 184, 199-201, 209, 213, 261, 264
rest/utils/__init__.py                          93      30  67.74%   79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              114      39  65.79%   65, 78-82, 93-105, 132-134, 141-144, 184, 211, 221-237, 242, 244, 265, 270, 278, 288
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             80       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      38  92.43%   85, 193-194, 203-230, 233, 248, 328-332, 405-409, 428-430, 509, 524, 564-565, 929-932, 940-941, 979-980, 1027-1028, 1151
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             41       0  100.00%
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9734    2042  79.02%

Diff against develop

Filename      Stmts    Miss  Cover
----------  -------  ------  --------
TOTAL             0       0  +100.00%

Results for commit: 8bff4bd

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024 •
edited
Loading

Test Results

    4 files  ±0      4 suites  ±0   5m 59s ⏱️ -1s
  573 tests ±0    570 ✅ ±0   3 💤 ±0  0 ❌ ±0 
2 292 runs  ±0  2 196 ✅ ±0  96 💤 ±0  0 ❌ ±0 

Results for commit 8bff4bd. ± Comparison against base commit 2473ac1.

♻️ This comment has been updated with latest results.

@jlahovnik jlahovnik marked this pull request as draft November 6, 2024 15:22
@amarandon
fix(prodiver): fix version formats for wekeo_ecmwf
de6be6c 
fix version formats for SATELLITE_CARBON_DIOXIDE and SATELLITE_METHANE
@amarandon amarandon marked this pull request as ready for review November 7, 2024 14:35
@alambare alambare added the dedl DEDL related label Nov 8, 2024
@alambare alambare marked this pull request as draft November 8, 2024 16:23
@amarandon
fix(provider): change default variable for CAMS_GAC_FORECAST
ab81315 
and set start date to an earlier date
@amarandon
fix(provider): change default variable for CAMS_GFE_GFAS
b4e9fdd 
Change default variable for CAMS_GFE_GFAS to a variable that is
available since 2003
@amarandon amarandon force-pushed the 1388-some-ecmwf-products-have-invalid-default-parameters branch from 705ec13 to b4e9fdd Compare November 12, 2024 11:03
@amarandon amarandon force-pushed the 1388-some-ecmwf-products-have-invalid-default-parameters branch from 1118988 to 3dc8d99 Compare November 12, 2024 13:37
@amarandon amarandon force-pushed the 1388-some-ecmwf-products-have-invalid-default-parameters branch from 3dc8d99 to 8bff4bd Compare November 12, 2024 13:56
@amarandon amarandon marked this pull request as ready for review November 12, 2024 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlahovnik jlahovnik jlahovnik approved these changes

@sbrunato sbrunato Awaiting requested review from sbrunato

Assignees

@amarandon amarandon

Labels
dedl DEDL related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Some ECMWF products have invalid default parameters
3 participants
@amarandon @jlahovnik @alambare