Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): shared and multiple auth per provider #1292

Merged
merged 15 commits into from
Oct 2, 2024
Merged

Conversation

sbrunato
Copy link
Collaborator

@sbrunato sbrunato commented Aug 9, 2024

Fixes #1265

  • Shared credentials between providers using matching_url or matching_conf new Authentication plugin parameters.
  • New Authentication plugins topics keys in providers configuration: search_auth and download_auth, allowing to have multiple authentication plugins per provider

See Developer documentation / Authentication plugins


These new features also bring the following changes:

  • Changes in plugins and providers configuration:
    • AwsDownload and CreodiasS3Download base_uri to s3_endpoint
    • aws_eos configuration update:
      • auth.credentials.api_key moved to search_auth.credentials.api_key
      • auth.credentials.aws_* moved to download_auth.credentials.aws_*
  • Updated plugin manager cache model and get_*plugins* methods APIs
  • Providers registration page update and sorted providers list in documentation

@sbrunato sbrunato marked this pull request as draft August 9, 2024 12:59
Copy link
Contributor

github-actions bot commented Aug 9, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      384      27  92.97%   83-85, 94, 102, 106-108, 179, 190, 581-583, 696-699, 742-743, 752-753, 858, 921-926, 928
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    763      69  90.96%   360, 650, 694-697, 735, 779, 813, 858-863, 889, 980, 1048, 1186, 1271-1283, 1319, 1321, 1349, 1353-1364, 1377-1383, 1466-1469, 1502-1522, 1574, 1591-1595, 1607-1610, 1962, 1986-1992, 2243, 2247-2251, 2260-2262, 2294
api/search_result.py                            59       4  93.22%   83, 92, 99, 113
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             173      15  91.33%   116-121, 171, 209, 231, 235, 259, 281, 388-391, 403-404
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              48       4  91.67%   134, 160-165
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       187      17  90.91%   114, 128-154, 162, 316-319, 345
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 89      16  82.02%   79, 108, 110, 133-146, 202-206
plugins/authentication/token_exchange.py        36      20  44.44%   74-80, 92-122
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   51-52, 68, 77-80, 82, 89-92
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   266, 279, 346-349, 363-367, 409-411, 415, 447-448, 454-458, 487, 519, 523, 530, 560-568, 572, 604-612, 623-625, 656-730, 748-808, 819-824, 836-849, 874, 889-891, 894, 904-912, 920-933, 943-974, 981-993, 1031, 1057, 1102-1104, 1324
plugins/download/base.py                       253      51  79.84%   136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       534     127  76.22%   195-207, 209-210, 306-309, 311-312, 319-324, 342-357, 374-376, 388, 436, 443-449, 467, 481, 495, 503-505, 521-526, 537, 555, 597-601, 623, 663, 708, 722-728, 757-821, 839, 869-878, 900-901, 928-933, 939, 942, 958, 975-976, 1006-1007, 1014, 1075-1081, 1136-1137, 1143, 1153, 1189, 1225, 1243-1256, 1282-1284
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 218-225, 228-230, 234, 245-251, 259-260, 263-267, 290, 311-314
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      13  90.15%   102, 106, 117, 276, 296, 352-353, 373, 376-384
plugins/search/build_search_result.py          189      24  87.30%   96, 137-138, 144, 155, 289-292, 321, 356-373, 422, 451, 454, 464, 481, 509, 511
plugins/search/cop_marine.py                   236      50  78.81%   55, 63-65, 75-76, 81, 86-87, 103, 105, 108, 143-145, 157-158, 200, 206, 210, 214, 227, 238-239, 247, 275, 279, 294, 298, 302, 306, 310-314, 320-323, 326-340, 357, 406-410, 415, 427
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          202      69  65.84%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-326, 328-329, 347-351, 384, 394, 405, 418, 424-439, 444
plugins/search/qssearch.py                     718      94  86.91%   390, 394-400, 515-527, 571, 587, 597, 616-631, 668-671, 742-743, 791, 810, 817, 829, 886, 907, 910-911, 920-921, 930-931, 940-941, 968, 1039-1044, 1048-1057, 1091, 1113, 1173, 1222, 1286, 1289-1290, 1372-1376, 1438, 1441, 1447-1448, 1469, 1496-1508, 1515, 1547-1549, 1559-1565, 1595, 1618, 1633, 1649, 1724-1727, 1732-1735, 1762-1763, 1778
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  26       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   251      64  74.50%   216-217, 273, 281, 298-316, 331-369, 460, 502-533, 680, 687-735
rest/errors.py                                  67       5  92.54%   106, 116, 127, 140-141
rest/server.py                                 188      24  87.23%   89, 112-114, 277-282, 310, 495-497, 514-519, 548, 550, 554-555, 559-560
rest/stac.py                                   319      63  80.25%   306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 778-782, 789, 843-844, 905, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176       5  97.16%   225-229, 282, 285, 353
rest/types/queryables.py                        56       1  98.21%   164
rest/types/stac_search.py                      126       7  94.44%   129, 175, 190-192, 200, 204
rest/utils/__init__.py                          93      12  87.10%   108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              114      14  87.72%   53, 70, 129-132, 199, 213-222, 232, 253, 266
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      37  92.63%   85, 90, 194-195, 204-231, 234, 248, 328-332, 406-410, 429-431, 510, 525, 561-562, 926-929, 937-938, 976-977, 1148
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             40       2  95.00%   98-99
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9603    1599  83.35%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  -------
cli.py                           +2       0  +0.12%
config.py                       +22       0  +0.43%
api/core.py                      -6      -5  +0.58%
plugins/manager.py              +43      +3  +0.56%
plugins/download/http.py          0      -2  +0.38%
plugins/search/qssearch.py        0      -2  +0.28%
utils/__init__.py                +2       0  +0.03%
TOTAL                           +63      -6  +0.17%

Results for commit: f6d6d6f

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Aug 9, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         314      59  81.21%   657-716, 818-869, 873
config.py                                      384      28  92.71%   83-85, 94, 102, 106-108, 179, 190, 581-583, 696-699, 742-743, 752-753, 858, 889, 921-926, 928
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    763      69  90.96%   360, 650, 694-697, 735, 779, 813, 858-863, 889, 980, 1048, 1186, 1271-1283, 1319, 1321, 1349, 1353-1364, 1377-1383, 1466-1469, 1502-1522, 1574, 1591-1595, 1607-1610, 1962, 1986-1992, 2243, 2247-2251, 2260-2262, 2294
api/search_result.py                            59       4  93.22%   83, 92, 99, 113
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                680      68  90.00%   130-132, 227, 259-260, 306-307, 317-329, 331, 342, 348-360, 405-406, 443, 464-467, 490, 498-499, 575-576, 600-601, 607-610, 625-626, 775, 821, 992-997, 1124, 1138-1158, 1178, 1183, 1312, 1326, 1397, 1449, 1489-1493, 1508
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             173      15  91.33%   116-121, 171, 209, 231, 235, 259, 281, 388-391, 403-404
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   43, 56
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              48       4  91.67%   134, 160-165
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       187      17  90.91%   114, 128-154, 162, 316-319, 345
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 89      16  82.02%   79, 108, 110, 133-146, 202-206
plugins/authentication/token_exchange.py        36      20  44.44%   74-80, 92-122
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   48-53, 66-111
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        491     163  66.80%   266, 279, 346-349, 363-367, 409-411, 415, 447-448, 454-458, 487, 519, 523, 530, 560-568, 572, 604-612, 623-625, 656-730, 748-808, 819-824, 836-849, 874, 889-891, 894, 904-912, 920-933, 943-974, 981-993, 1031, 1057, 1102-1104, 1324
plugins/download/base.py                       253      53  79.05%   136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       534     128  76.03%   195-207, 209-210, 306-309, 311-312, 319-324, 342-357, 374-376, 388, 436, 443-449, 467, 481, 495, 503-505, 521-526, 537, 555, 597-601, 623, 663, 708, 722-728, 757-821, 839, 869-878, 900-901, 928-933, 939, 942, 958, 975-976, 989, 1006-1007, 1014, 1075-1081, 1136-1137, 1143, 1153, 1189, 1225, 1243-1256, 1282-1284
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 218-225, 228-230, 234, 245-251, 259-260, 263-267, 290, 311-314
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         132      14  89.39%   102, 106, 117, 276, 296, 352-353, 373, 376-384, 386
plugins/search/build_search_result.py          189      31  83.60%   96, 137-138, 144, 155, 289-292, 321, 356-373, 422, 451, 454, 464, 481, 501-516
plugins/search/cop_marine.py                   236      50  78.81%   55, 63-65, 75-76, 81, 86-87, 103, 105, 108, 143-145, 157-158, 200, 206, 210, 214, 227, 238-239, 247, 275, 279, 294, 298, 302, 306, 310-314, 320-323, 326-340, 357, 406-410, 415, 427
plugins/search/creodias_s3.py                   55       3  94.55%   58, 76, 110
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          202      69  65.84%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-326, 328-329, 347-351, 384, 394, 405, 418, 424-439, 444
plugins/search/qssearch.py                     718     123  82.87%   390, 394-400, 515-527, 571, 574, 587, 597, 616-631, 668-671, 742-743, 791, 810, 817, 829, 886, 907, 910-911, 920-921, 930-931, 940-941, 968, 1039-1044, 1048-1057, 1091, 1113, 1173, 1222, 1286, 1289-1290, 1372-1376, 1438, 1441, 1447-1448, 1469, 1496-1508, 1515, 1547-1549, 1559-1565, 1595, 1618, 1633, 1649, 1707-1807
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  26       1  96.15%   36
rest/constants.py                                6       0  100.00%
rest/core.py                                   251     139  44.62%   162, 164, 166, 169-170, 184-194, 206-207, 209-210, 216-217, 220, 223, 264-318, 331-369, 400-434, 449-465, 481-490, 502-533, 550, 592-641, 680, 687-735
rest/errors.py                                  67      47  29.85%   60, 65-100, 105-108, 115-118, 126-144, 152-157, 172-178
rest/server.py                                 188     188  0.00%    18-573
rest/stac.py                                   319      68  78.68%   240, 306, 328, 380-383, 410-437, 468-470, 493, 525-526, 608-648, 670-686, 713, 778-782, 789, 843-844, 850, 905, 943, 976, 995-997
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     176      16  90.91%   225-229, 262-264, 282, 285, 291, 295, 353, 370-380
rest/types/queryables.py                        56      13  76.79%   51-52, 59-60, 67-68, 94-99, 108-109, 164
rest/types/stac_search.py                      126      11  91.27%   127-129, 175, 190-192, 200, 204, 252, 255
rest/utils/__init__.py                          93      30  67.74%   79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              114      39  65.79%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-225, 230, 232, 253, 258, 266, 276
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      37  92.63%   85, 90, 194-195, 204-231, 234, 248, 328-332, 406-410, 429-431, 510, 525, 561-562, 926-929, 937-938, 976-977, 1148
utils/constraints.py                           119      38  68.07%   62, 89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             40       2  95.00%   98-99
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 86, 88, 90, 92, 94, 110, 118-120, 128
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9603    2040  78.76%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  -------
cli.py                           +2       0  +0.12%
config.py                       +22       0  +0.44%
api/core.py                      -6      -5  +0.58%
plugins/manager.py              +43      +3  +0.56%
plugins/download/http.py          0      -2  +0.37%
plugins/search/qssearch.py        0      -2  +0.28%
utils/__init__.py                +2       0  +0.03%
TOTAL                           +63      -6  +0.21%

Results for commit: f6d6d6f

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Aug 9, 2024

Test Results

    4 files  ± 0      4 suites  ±0   6m 11s ⏱️ -1s
  565 tests + 3    562 ✅ + 3   3 💤 ±0  0 ❌ ±0 
2 260 runs  +12  2 166 ✅ +12  94 💤 ±0  0 ❌ ±0 

Results for commit f6d6d6f. ± Comparison against base commit 0d7ff3c.

♻️ This comment has been updated with latest results.

@sbrunato sbrunato force-pushed the multiple_auth branch 3 times, most recently from f5b9630 to df62d99 Compare October 2, 2024 07:16
@sbrunato sbrunato marked this pull request as ready for review October 2, 2024 14:21
@sbrunato sbrunato merged commit d148e50 into develop Oct 2, 2024
10 checks passed
@sbrunato sbrunato deleted the multiple_auth branch October 2, 2024 14:23
@sbrunato sbrunato added this to the 3.0.0.dev milestone Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

shared auth between providers
1 participant