🎉 CI secrets: full migration to GSM (#8561)

* add python packages for CI scripts * add tox config for all subpackages * draft version * init venv for scripts * fix venv * remove used comments * fix run test * change base folder * update secret format * update docs * remove an unused file * remove github secrets logic fully * fix base_folder balue * add functions desc
airbytehq · Dec 15, 2021 · d3b0c99 · d3b0c99
1 parent 0fd129e
commit d3b0c99
Show file tree

Hide file tree

Showing 20 changed files with 747 additions and 394 deletions.
diff --git a/.github/workflows/publish-command.yml b/.github/workflows/publish-command.yml
@@ -66,16 +66,28 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: ${{github.event.pull_request.head.repo.full_name}} # always use the branch's repository
-      # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559.
-      - name: Install Pyenv
-        run: python3 -m pip install virtualenv==16.7.9 --user
-      - uses: actions/setup-java@v1
+      - name: Install Java
+        uses: actions/setup-java@v1
         with:
           java-version: '17'
-      - name: Write Integration Test Credentials # TODO DRY this with test-command.yml
-        run: ./tools/bin/ci_credentials.sh ${{ github.event.inputs.connector }}
+      - name: Install Pyenv and Tox
+        # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
+        run: |
+          python3 -m pip install --quiet virtualenv==16.7.9 --user
+          python3 -m virtualenv venv
+          source venv/bin/activate
+          pip install --quiet tox==3.24.4
+      - name: Test and install CI scripts
+        # all CI python packages have the prefix "ci_"
+        run: |
+          source venv/bin/activate
+          tox -r -c ./tools/tox_ci.ini
+          pip install --quiet -e ./tools/ci_*
+      - name: Write Integration Test Credentials for ${{ github.event.inputs.connector }}
+        run: |
+          source venv/bin/activate
+          ci_credentials ${{ github.event.inputs.connector }}
         env:
-          GITHUB_PROVIDED_SECRETS_JSON: ${{ toJson(secrets) }}
           GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
       - run: |
           echo "$SPEC_CACHE_SERVICE_ACCOUNT_KEY" > spec_cache_key_file.json && docker login -u airbytebot -p ${DOCKER_PASSWORD}

diff --git a/.github/workflows/test-command.yml b/.github/workflows/test-command.yml
@@ -61,20 +61,33 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: ${{ github.event.inputs.repo }}
-      # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
-      - name: Install Pyenv
-        run: python3 -m pip install virtualenv==16.7.9 --user
-      - uses: actions/setup-java@v1
+      - name: Install Java
+        uses: actions/setup-java@v1
         with:
           java-version: '17'
-      - name: Write Integration Test Credentials
-        run: ./tools/bin/ci_credentials.sh ${{ github.event.inputs.connector }}
+      - name: Install Pyenv and Tox
+        # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
+        run: |
+          python3 -m pip install --quiet virtualenv==16.7.9 --user
+          python3 -m virtualenv venv
+          source venv/bin/activate
+          pip install --quiet tox==3.24.4
+      - name: Test and install CI scripts
+        # all CI python packages have the prefix "ci_"
+        run: |
+          source venv/bin/activate
+          tox -r -c ./tools/tox_ci.ini
+          pip install --quiet -e ./tools/ci_*
+      - name: Write Integration Test Credentials for ${{ github.event.inputs.connector }}
+        run: |
+          source venv/bin/activate
+          ci_credentials ${{ github.event.inputs.connector }}
         env:
-          GITHUB_PROVIDED_SECRETS_JSON: ${{ toJson(secrets) }}
           GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
-      - run: |
+
+      - name: test ${{ github.event.inputs.connector }}
+        run: |
           ./tools/bin/ci_integration_test.sh ${{ github.event.inputs.connector }}
-        name: test ${{ github.event.inputs.connector }}
         id: test
         env:
           ACTION_RUN_ID: ${{github.run_id}}

diff --git a/.github/workflows/test-performance-command.yml b/.github/workflows/test-performance-command.yml
@@ -67,13 +67,24 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: ${{ github.event.inputs.repo }}
-      # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
-      - name: Install Pyenv
-        run: python3 -m pip install virtualenv==16.7.9 --user
-      - name: Write Integration Test Credentials
-        run: ./tools/bin/ci_credentials.sh ${{ github.event.inputs.connector }}
+      - name: Install Pyenv and Tox
+        # Beside PyEnv, this does not set any runtimes up because it uses an AMI image that has everything pre-installed. See https://github.com/airbytehq/airbyte/issues/4559/
+        run: |
+          python3 -m pip install --quiet virtualenv==16.7.9 --user
+          python3 -m virtualenv venv
+          source venv/bin/activate
+          pip install --quiet tox==3.24.4
+      - name: Test and install CI scripts
+        # all CI python packages have the prefix "ci_"
+        run: |
+          source venv/bin/activate
+          tox -r -c ./tools/tox_ci.ini
+          pip install --quiet -e ./tools/ci_*
+      - name: Write Integration Test Credentials for ${{ github.event.inputs.connector }}
+        run: |
+          source venv/bin/activate
+          ci_credentials ${{ github.event.inputs.connector }}
         env:
-          GITHUB_PROVIDED_SECRETS_JSON: ${{ toJson(secrets) }}
           GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
       - run: |
           ./tools/bin/ci_performance_test.sh ${{ github.event.inputs.connector }} ${{ github.event.inputs.cpulimit }} ${{ github.event.inputs.memorylimit }}

diff --git a/docs/connector-development/README.md b/docs/connector-development/README.md
@@ -138,7 +138,8 @@ In order to run integration tests in CI, you'll often need to inject credentials
 2. **Add the GSM secret's labels**:
     * `connector` (required) -- unique connector's name or set of connectors' names with '_' as delimiter i.e.: `connector=source-s3`, `connector=destination-snowflake`
     * `filename` (optional) -- custom target secret file. Unfortunately Google doesn't use '.' into labels' values and so Airbyte CI scripts will add '.json' to the end automatically. By default secrets will be saved to `./secrets/config.json` i.e: `filename=config_auth` => `secrets/config_auth.json`
-3. That should be it.
+3. **Save a necessary JSON value** [Example](https://user-images.githubusercontent.com/11213273/146040653-4a76c371-a00e-41fe-8300-cbd411f10b2e.png).
+4. That should be it.
 
 #### Access CI secrets on GSM
 Access to GSM storage is limited to Airbyte employees. To give an employee permissions to the project:
@@ -148,9 +149,4 @@ Access to GSM storage is limited to Airbyte employees. To give an employee permi
 - select the role `Development_CI_Secrets`
 3. Save
 
-#### How to migrate to the new secrets' logic:
-1. Create all necessary secrets according to the instructions above
-2. Remove all lines with old connector's Github secrets from this file: tools/bin/ci_credentials.sh
-3. Remove all old secrets from Github repository secrets.
-4. That should be it.