🎉 CI secrets: full migration to GSM #8561

antixar · 2021-12-06T22:50:16Z

What

All secrets must be loaded from GSM only

How

Rewrite all logic scripts to Python
migrate of exist GitHub secrets to GSM
remove GitHub connectors' secrets from CI flow

migrated values: github_secrets_keys.txt

…-migration-to-GSM

antixar · 2021-12-06T22:54:19Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1546975050
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1546975050
🐛

antixar · 2021-12-06T23:16:59Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547042160
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547042160
🐛

antixar · 2021-12-06T23:23:25Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547063702
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547063702
🐛

antixar · 2021-12-06T23:54:03Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547148411
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547148411
🐛

antixar · 2021-12-07T00:07:01Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547186710
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547186710
🐛

antixar · 2021-12-07T00:15:31Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547215788
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1547215788
🐛

antixar · 2021-12-14T17:05:48Z

/test connector=connectors/source-lemlist

🕑 connectors/source-lemlist https://github.com/airbytehq/airbyte/actions/runs/1578981273
✅ connectors/source-lemlist https://github.com/airbytehq/airbyte/actions/runs/1578981273
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 source_lemlist/__init__.py       2      0   100%
	 source_lemlist/auth.py           8      0   100%
	 source_lemlist/source.py        19      2    89%
	 source_lemlist/streams.py       49     11    78%
	 ------------------------------------------------
	 TOTAL                           78     13    83%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 13 passed, 1 skipped in 13.81s ========================

antixar · 2021-12-14T17:15:11Z

/performance-test connector=connectors/source-mysql

antixar · 2021-12-15T10:38:07Z

/test connector=connectors/source-marketo

🕑 connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582296444
✅ connectors/source-marketo https://github.com/airbytehq/airbyte/actions/runs/1582296444
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                         Stmts   Miss  Cover
	 ------------------------------------------------
	 source_marketo/__init__.py       2      0   100%
	 source_marketo/source.py       237    151    36%
	 source_marketo/utils.py         31      8    74%
	 ------------------------------------------------
	 TOTAL                          270    159    41%

Phlair

Good stuff, mostly minor comments.

Main point would be, if we're removing github secrets entirely, doesn't that make the github secrets code in this PR redundant?

tools/ci_common_utils/ci_common_utils/google_api.py

tools/ci_credentials/ci_credentials/secrets_loader.py

tools/ci_credentials/ci_credentials/main.py

tools/lib/gcp-token.sh

antixar · 2021-12-15T12:02:55Z

/test connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582600282
❌ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582600282
🐛

antixar · 2021-12-15T12:11:03Z

/test connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582632940
✅ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582632940
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_intercom/__init__.py       2      0   100%
	 source_intercom/source.py       215     40    81%
	 -------------------------------------------------
	 TOTAL                           217     40    82%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_intercom/__init__.py       2      0   100%
	 source_intercom/source.py       215    105    51%
	 -------------------------------------------------
	 TOTAL                           217    105    52%

Phlair

lgtm

antixar · 2021-12-15T13:01:28Z

/test connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582812515
✅ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/1582812515
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_intercom/__init__.py       2      0   100%
	 source_intercom/source.py       215     40    81%
	 -------------------------------------------------
	 TOTAL                           217     40    82%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                          Stmts   Miss  Cover
	 -------------------------------------------------
	 source_intercom/__init__.py       2      0   100%
	 source_intercom/source.py       215    105    51%
	 -------------------------------------------------
	 TOTAL                           217    105    52%

* add python packages for CI scripts * add tox config for all subpackages * draft version * init venv for scripts * fix venv * remove used comments * fix run test * change base folder * update secret format * update docs * remove an unused file * remove github secrets logic fully * fix base_folder balue * add functions desc

antixar added 4 commits December 1, 2021 09:53

add python packages for CI scripts

8cba67a

add tox config for all subpackages

27b6eb8

draft version

5dba506

Merge remote-tracking branch 'origin/master' into antixar/7774-finish…

272d894

…-migration-to-GSM

antixar temporarily deployed to more-secrets December 6, 2021 22:50 Inactive

antixar self-assigned this Dec 6, 2021

antixar linked an issue Dec 6, 2021 that may be closed by this pull request

Completion of migration to GSM #7774

Closed

jrhizor temporarily deployed to more-secrets December 6, 2021 22:56 Inactive

fix pyenv installation

b193885

antixar temporarily deployed to more-secrets December 6, 2021 23:16 Inactive

jrhizor temporarily deployed to more-secrets December 6, 2021 23:19 Inactive

fix pyenv installation

f6a7b48

antixar temporarily deployed to more-secrets December 6, 2021 23:24 Inactive

jrhizor temporarily deployed to more-secrets December 6, 2021 23:25 Inactive

init venv for scripts

d0ed630

antixar temporarily deployed to more-secrets December 6, 2021 23:55 Inactive

jrhizor temporarily deployed to more-secrets December 6, 2021 23:56 Inactive

init venv for scripts

9f2d3b8

antixar temporarily deployed to more-secrets December 7, 2021 00:08 Inactive

jrhizor temporarily deployed to more-secrets December 7, 2021 00:09 Inactive

init venv for scripts

bfb76e6

antixar temporarily deployed to more-secrets December 7, 2021 00:16 Inactive

jrhizor temporarily deployed to more-secrets December 7, 2021 00:17 Inactive

merge with master

97c331c

antixar requested a review from sherifnada December 14, 2021 17:05

antixar marked this pull request as ready for review December 14, 2021 17:05

jrhizor temporarily deployed to more-secrets December 14, 2021 17:07 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 10:39 Inactive

remove an unused file

19bcf79

antixar temporarily deployed to more-secrets December 15, 2021 10:43 Inactive

Phlair reviewed Dec 15, 2021

View reviewed changes

remove github secrets logic fully

675ee62

antixar temporarily deployed to more-secrets December 15, 2021 12:03 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 12:05 Inactive

fix base_folder balue

76ef7f3

antixar requested a review from Phlair December 15, 2021 12:11

antixar temporarily deployed to more-secrets December 15, 2021 12:12 Inactive

jrhizor temporarily deployed to more-secrets December 15, 2021 12:12 Inactive

add functions desc

5d3283f

antixar temporarily deployed to more-secrets December 15, 2021 12:32 Inactive

Phlair approved these changes Dec 15, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets December 15, 2021 13:03 Inactive

antixar merged commit d3b0c99 into master Dec 15, 2021

antixar deleted the antixar/7774-finish-migration-to-GSM branch December 15, 2021 13:08

jrhizor mentioned this pull request Dec 16, 2021

Bump Airbyte version from 0.33.12-alpha to 0.34.0-alpha #8834

Merged

alafanechere mentioned this pull request Jan 19, 2023

ci_credentials: disable tooling test run by tox #21580

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 CI secrets: full migration to GSM #8561

🎉 CI secrets: full migration to GSM #8561

antixar commented Dec 6, 2021 •

edited

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 7, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 7, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 14, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 14, 2021

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

Phlair left a comment

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

Phlair left a comment

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

🎉 CI secrets: full migration to GSM #8561

🎉 CI secrets: full migration to GSM #8561

Conversation

antixar commented Dec 6, 2021 • edited Loading

What

How

antixar commented Dec 6, 2021 • edited by github-actions bot Loading

antixar commented Dec 6, 2021 • edited by github-actions bot Loading

antixar commented Dec 6, 2021 • edited by github-actions bot Loading

antixar commented Dec 6, 2021 • edited by github-actions bot Loading

antixar commented Dec 7, 2021 • edited by github-actions bot Loading

antixar commented Dec 7, 2021 • edited by github-actions bot Loading

antixar commented Dec 14, 2021 • edited by github-actions bot Loading

antixar commented Dec 14, 2021

antixar commented Dec 15, 2021 • edited by github-actions bot Loading

Phlair left a comment

Choose a reason for hiding this comment

antixar commented Dec 15, 2021 • edited by github-actions bot Loading

antixar commented Dec 15, 2021 • edited by github-actions bot Loading

Phlair left a comment

Choose a reason for hiding this comment

antixar commented Dec 15, 2021 • edited by github-actions bot Loading

antixar commented Dec 6, 2021 •

edited

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 6, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 7, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 7, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 14, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading

antixar commented Dec 15, 2021 •

edited by github-actions bot

Loading