Setup the new auto-release CircleCI Release #43

noklam · 2022-07-04T14:22:35Z

Description

To adapt the monorepo structure and support auto-release for plugin

The old process

Every time main is updated -> Check if __version__.py is updated, if so trigger a sync job
Check if the version is on PyPI - if not - trigger - Send a POST request to CircleCI to trigger publish_kedro pipeline
Twine for PyPI / Another request for GitHub Release

The Current Flow

Whenever main get updated, it checked if any of the repositories need to do a release (By looping all __version__ and compare to PyPI)
Another CircleCI pipeline is triggered with the pipeline parameter (release_version and release_package) which triggers the build and publish workflow.

Development notes

release_package & release_version replace the release_kedro :True to handle multiple repositories
Refactor and combine a few bash scripts into 1 Python script
Put some documentation back into code about how to create the token
A successful CI build on CCI
Update documentation on Confluence for plugins

The credential on CCI are token generated on PyPI, I don't know where's the best place to document this information, maybe I will document how I do the testing locally on Confluence.

Extra tips:
I only realise I can trigger a pipeline on main with the UI, which should help you to "unit test" some individual workflow. (So you don't have to make a thousand dummy commits to trigger the job 😅)

Testings are done locally with Personal Token, we have a separate set of tokens saved in CircleCI - some links are written in docstring to guide how to create these tokens.

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes

Signed-off-by: Nok Chan <[email protected]>

…icd-auto-release Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>

This reverts commit eb6ade0.

Signed-off-by: Nok Chan <[email protected]>

This reverts commit 547b938.

Signed-off-by: Nok Chan <[email protected]>

noklam · 2022-07-07T10:51:43Z

kedro-datasets/features/steps/dummy.feature

@@ -0,0 +1 @@
+# A dummy file to keep CI behave correctly


The kedro-datasets didn't really have an e2e test. But keeping a dummy test here simplifies the conditions with CI config and it doesn't have much overhead, so I create a dummy file.

noklam · 2022-07-07T10:52:02Z

kedro-datasets/pyproject.toml

@@ -33,7 +33,7 @@ min-public-methods = 1
 [tool.coverage.report]
 fail_under = 100
 show_missing = true
-omit = ["tests/*"]
+omit = ["tests/*", "kedro_datasets/datasets/holoviews/*"]


Carries from kedro

Signed-off-by: Nok Chan <[email protected]>

noklam · 2022-07-07T12:25:33Z

tools/circleci/circleci_release.py

+        payload = {
+            "branch": CIRCLE_BRANCH,
+            "parameters": {
+                "release_package": package_name,


We used to check the version twice in two different jobs, I think it's better to keep a single source of truth and just pass this information in the requests.

noklam · 2022-07-07T12:27:57Z

tools/circleci/circleci_release.py

+from utils.package_version import get_package_version
+
+PACKAGE_PATHS = (
+    "kedro-datasets/kedro_datasets",


In theory, we can just use kedro-datasets and parse it, but I am not sure if this is a convention. So I just keep the longer path here, not too important anyway.

noklam · 2022-07-07T12:30:17Z

tools/circleci/circleci_release.py

+        }
+
+        print(package_name, package_version)
+        if check_no_version_pypi(pypi_endpoint, package_name, package_version):


Everytime main is updated, it will check all repositories and see if there is a need to do a release. So it is possible that it triggers more than 1 release job (for different repository) but in practice, we shouldn't have one commit changing 2 different repositories.

Signed-off-by: Nok Chan <[email protected]>

noklam · 2022-07-07T12:55:30Z

No idea why getting PermissionError only on Python3.10 + Windows, @MerelTheisenQB any idea? I remember you have fixed some bug in CI that only fails on Python3.10 + Windows+ before.

E PermissionError: Forbidden
../../miniconda/envs/kedro_plugins/lib/python3.10/site-packages/s3fs/core.py:548: PermissionError

merelcht

Some minor questions from me, but in general this looks good to me! 👍

merelcht · 2022-07-08T09:52:02Z

.circleci/config.yml

@@ -1,5 +1,13 @@
 version: 2.1

+parameters:


Why do you need these parameters here?

It's a CircleCI thing, otherwise, it will just throw an error. We called these 2 parameters in the POST request.

merelcht · 2022-07-08T09:53:04Z

.circleci/continue_config.yml

+      # - run:
+      #     name: Maybe merge main into develop or raise a PR
+      #     command: ./tools/circleci/github_scripts/merge.sh . "main" "develop" "${GITHUB_TAGGING_TOKEN}"


I guess we can remove this as I don't see us adding a develop branch for this repo.

merelcht · 2022-07-08T09:56:51Z

tools/circleci/circleci_release.py

+        print("Creating CircleCI Pipeline successfully")
+        print(resp.content)
+    else:
+        print("Failed to create CircleCI Pipeline")


Do we need any more info here? Error code or something like that to help us figure out what went wrong?

I agree with Merel, maybe put the response code in the print statement as well.

merelcht · 2022-07-08T10:02:03Z

No idea why getting PermissionError only on Python3.10 + Windows, @MerelTheisenQB any idea? I remember you have fixed some bug in CI that only fails on Python3.10 + Windows+ before.

E PermissionError: Forbidden
../../miniconda/envs/kedro_plugins/lib/python3.10/site-packages/s3fs/core.py:548: PermissionError

Hmm strange.. I think I solved those issues at the time by running tests sequentially instead of in parallel. But it's odd it's just the one test now..

SajidAlamQB

Awesome work on this just a minor comment! 🎉

SajidAlamQB · 2022-07-08T10:49:30Z

tools/circleci/circleci_release.py

+        print("Creating CircleCI Pipeline successfully")
+        print(resp.content)
+    else:
+        print("Failed to create CircleCI Pipeline")


I agree with Merel, maybe put the response code in the print statement as well.

noklam · 2022-07-12T15:05:31Z

Let's solve this unittest issue for 3.10 separately

noklam added 30 commits June 28, 2022 10:33

put back the circle ci config

ecffa72

Signed-off-by: Nok Chan <[email protected]>

initial setup for datasets cI

79df35b

Signed-off-by: Nok Chan <[email protected]>

Add window config

009c3d6

Signed-off-by: Nok Chan <[email protected]>

dummy changes to trigger CI

6284409

Signed-off-by: Nok Chan <[email protected]>

Bug fix the path-filtering, wrong repository

a55ff3d

Signed-off-by: Nok Chan <[email protected]>

Merge branch 'main' of github.com:kedro-org/kedro-plugins into feat/c…

5e11bd6

…icd-auto-release Signed-off-by: Nok Chan <[email protected]>

More attempts to fix CI - wish I can test locally

11a6a25

Signed-off-by: Nok Chan <[email protected]>

Need to cd to the plugins directory before install

3616dda

Signed-off-by: Nok Chan <[email protected]>

Add window setup for tensorflow test

6c79e13

Signed-off-by: Nok Chan <[email protected]>

fixing Spark related test

77f4566

Signed-off-by: Nok Chan <[email protected]>

Fix config

63f0002

Signed-off-by: Nok Chan <[email protected]>

Trigger CI

d3fa5b5

Signed-off-by: Nok Chan <[email protected]>

reverse the boolean condition...

fffdec9

Signed-off-by: Nok Chan <[email protected]>

fix conda env

935af8c

Signed-off-by: Nok Chan <[email protected]>

Fix Window Test and some linting

ee6581f

Signed-off-by: Nok Chan <[email protected]>

fix coverage issue

e4a0e54

Signed-off-by: Nok Chan <[email protected]>

add snippets to check env variable

b857f20

Signed-off-by: Nok Chan <[email protected]>

Comment out invalid release config for testing

8afecfb

Signed-off-by: Nok Chan <[email protected]>

testing

9c20426

Signed-off-by: Nok Chan <[email protected]>

use older python syntax

42258b8

Signed-off-by: Nok Chan <[email protected]>

remove auto-release change & split PR

bc3c510

Signed-off-by: Nok Chan <[email protected]>

remove empty lines

e53ab6f

Signed-off-by: Nok Chan <[email protected]>

add CI scripts

eb6ade0

Signed-off-by: Nok Chan <[email protected]>

Revert "add CI scripts"

315d236

This reverts commit eb6ade0.

add test to cover loading csv in remote storage

547b938

Signed-off-by: Nok Chan <[email protected]>

Revert "add test to cover loading csv in remote storage"

4ee29fe

This reverts commit 547b938.

add build scripts

2a0ee5b

Signed-off-by: Nok Chan <[email protected]>

update file for permission issue

7ac9451

Signed-off-by: Nok Chan <[email protected]>

setup env

cea2141

Signed-off-by: Nok Chan <[email protected]>

update config

8914067

Signed-off-by: Nok Chan <[email protected]>

noklam and others added 9 commits July 5, 2022 17:15

more bug fix

6df8cc8

Signed-off-by: Nok Chan <[email protected]>

fix pypi folder

6795635

Signed-off-by: Nok Chan <[email protected]>

remove the TWINE_REPOSITORY_URL argument

2aaa207

Signed-off-by: Nok Chan <[email protected]>

update the CI config

0eb5611

Signed-off-by: Nok Chan <[email protected]>

trigger a proper release

a0b5efe

Signed-off-by: Nok Chan <[email protected]>

fix config

fa9395e

Signed-off-by: Nok Chan <[email protected]>

Merge branch 'main' into feat/refactor-auto-release

ebbe3ac

add dummy e2etest

55a57e6

Signed-off-by: Nok Chan <[email protected]>

add comments

40ea125

Signed-off-by: Nok Chan <[email protected]>

noklam commented Jul 7, 2022

View reviewed changes

noklam added 2 commits July 7, 2022 12:03

fix lint

b802213

Signed-off-by: Nok Chan <[email protected]>

format nicer

022bbe1

Signed-off-by: Nok Chan <[email protected]>

noklam marked this pull request as ready for review July 7, 2022 12:23

noklam commented Jul 7, 2022

View reviewed changes

Clean up for review

d036af2

Signed-off-by: Nok Chan <[email protected]>

noklam requested review from merelcht and SajidAlamQB July 7, 2022 12:37

merelcht approved these changes Jul 8, 2022

View reviewed changes

SajidAlamQB approved these changes Jul 8, 2022

View reviewed changes

apply review comments

ce5cbf2

noklam merged commit 5813362 into main Jul 12, 2022

noklam deleted the feat/refactor-auto-release branch July 12, 2022 15:05

AhdraMeraliQB mentioned this pull request Nov 11, 2022

Fix telemetry release CI #71

Merged

4 tasks

noklam mentioned this pull request Apr 5, 2023

Update Release to fail when release isn't triggered properly #158

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup the new auto-release CircleCI Release #43

Setup the new auto-release CircleCI Release #43

noklam commented Jul 4, 2022 •

edited

Loading

noklam Jul 7, 2022

noklam Jul 7, 2022

noklam Jul 7, 2022

noklam Jul 7, 2022

noklam Jul 7, 2022

noklam commented Jul 7, 2022

merelcht left a comment

merelcht Jul 8, 2022

noklam Jul 12, 2022

merelcht Jul 8, 2022

merelcht Jul 8, 2022

SajidAlamQB Jul 8, 2022

merelcht commented Jul 8, 2022

SajidAlamQB left a comment

SajidAlamQB Jul 8, 2022

noklam commented Jul 12, 2022

Setup the new auto-release CircleCI Release #43

Setup the new auto-release CircleCI Release #43

Conversation

noklam commented Jul 4, 2022 • edited Loading

Description

The old process

The Current Flow

Development notes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented Jul 7, 2022

merelcht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merelcht commented Jul 8, 2022

SajidAlamQB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented Jul 12, 2022

noklam commented Jul 4, 2022 •

edited

Loading