tools: Migrate from CircleCi to GitHub Actions #679

shahzadlone · 2022-07-27T07:31:24Z

Relevant issue(s)

Resolves #570

Description

Moves all our CircleCi workflows to GitHub Actions (Change Detection, Test Runs).
Change detection will only run in [PRs] or [pushes to tags, master, or develop].
Change detection for GitHub Action Workflow needed explicit private and public ssh keys (generated and added the corresponding keys to secret section, and deploy key sections), there is an authentication step that takes care of the SSH authentication for that workflow.
Added a new Workflow to build defradb binary and call defradb start on it to ensure the start up command doesn't crash.
Removed CircleCi config file validation tool.

Merge Process

In order for a clean merge, once we decide we are ready to merge this will need to disable CircleCi completely so the failing CI check doesn't show up. In that amount of time would strongly suggest no other merges happen (since we are in code freeze anyways it should be less of an issue).

Pre-Merge Steps:

Remove required CI checks from CircleCi
Stop CircleCi from building this repository.
Merge once all checks pass.

Post-Merge Steps:

Add / Update required CI checks with all Github Actions that run on pushes to develop, maybe for master too (in branch protection rules).
Remove the CircleCi Deploy key (was gone already, perhaps circleci removed it?).

Limitations

Through this change found that CircleCi was actually not properly setting these environment variables:

      DEFRA_BADGER_MEMORY: true
      DEFRA_BADGER_FILE: true
      DEFRA_MAP: true

So for now the CI will run without them and have made these issues:

Tasks

I made sure the code is well commented, particularly hard-to-understand areas.
I made sure the repository-held documentation is changed accordingly.
I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

This PR's checks should show if these workflows work or not.

Specify the platform(s) on which this was tested:

Manjaro

orpheuslummis · 2022-07-27T12:48:18Z

It LGTM so I'm not sure why it fails. Any insight?

shahzadlone · 2022-07-27T21:33:46Z

It LGTM so I'm not sure why it fails. Any insight?

Have a few hunches, but nothing certain. Will let you know when I fix it and convert to non-draft.

Here are the finding @orpheuslummis :

Running Tests Workflow was failing because of:

Change Detection was failing because GitHub Action SSH Authentication was needed.

fredcarle

LGTM but I do have one question/concern before you merge.

fredcarle · 2022-07-28T20:41:16Z

Makefile

 .PHONY: test\:changes
 test\:changes:
-	env DEFRA_DETECT_DATABASE_CHANGES=true gotestsum --junitfile /tmp/defradb-dev/changes.xml -- ./... -shuffle=on -p 1
+	env DEFRA_DETECT_DATABASE_CHANGES=true gotestsum -- ./... -shuffle=on -p 1


question: Trying to run this will fail because of the issue we raised yesterday. Should be remove it for now?

No this will not fail. Infact we have an entire GitHub Action that is running this and passing (Detect Change).

We had two problems yesterday:

SSH authentication was not being done, i.e. the git command would fail. (solved by adding private and public ssh secrets and deployment key and doing the authentication in the runner).

Setting this env variable to false was still running the detection tests (solved by not setting this env to false explicitly anymore).

Moreover, we do have an other off case (issue #637) where with race this always fails (but rarely should this fail without race).

Ah dam... Missed the SSH auth part in the PR description. Sorry about that. If the keys are there then all good 🙂

orpheuslummis

LGTM

.github/workflows/start-binary.yml

shahzadlone · 2022-07-29T05:07:01Z

Since we are in "code freeze" and this seems to be not v0.3.0 specific, we can hold this off until develop is ready for v0.3.1.

properly.

- Resolves sourcenetwork#570 - Description: * Moves all our CircleCi workflows to GitHub Actions (Change Detection, Test Runs). * Change detection will only run in [PRs] or [pushes to tags, master, or develop]. * Change detection for GitHub Action Workflow needed explicit private and public ssh keys (generated and added the corresponding keys to secret section, and deploy key sections), there is an authentication step that takes care of the SSH authentication for that workflow. * Added a new Workflow to build defradb binary and call `defradb start` on it to ensure the start up command doesn't crash. * Removed CircleCi config file validation tool.

shahzadlone added the ci/build This is issue is about the build or CI system, and the administration of it. label Jul 27, 2022

shahzadlone self-assigned this Jul 27, 2022

This comment was marked as off-topic.

Sign in to view

shahzadlone added the action/no-benchmark Skips the action that runs the benchmark. label Jul 27, 2022

sourcenetwork deleted a comment from source-devs Jul 27, 2022

shahzadlone force-pushed the lone/tools/migrate-all-circli-pipelines-to-github-actions branch from ce440ab to 2fe3e2f Compare July 27, 2022 08:26

shahzadlone force-pushed the lone/tools/migrate-all-circli-pipelines-to-github-actions branch 18 times, most recently from 683d5a0 to 575aa6a Compare July 28, 2022 09:04

shahzadlone changed the title ~~tools: Migrate fully from CircleCi to GitHub Actions~~ tools: Migrate from CircleCi to GitHub Actions Jul 28, 2022

shahzadlone force-pushed the lone/tools/migrate-all-circli-pipelines-to-github-actions branch from 575aa6a to a39b49b Compare July 28, 2022 09:39

shahzadlone marked this pull request as ready for review July 28, 2022 09:46

shahzadlone requested a review from orpheuslummis July 28, 2022 09:49

shahzadlone requested review from jsimnz and fredcarle July 28, 2022 09:49

shahzadlone added the area/testing Related to any test or testing suite label Jul 28, 2022

fredcarle approved these changes Jul 28, 2022

View reviewed changes

shahzadlone force-pushed the lone/tools/migrate-all-circli-pipelines-to-github-actions branch from a39b49b to 702ff82 Compare July 28, 2022 21:21

orpheuslummis approved these changes Jul 28, 2022

View reviewed changes

.github/workflows/start-binary.yml Show resolved Hide resolved

shahzadlone added the DO NOT MERGE label Jul 29, 2022

shahzadlone added this to the DefraDB v0.3.1 milestone Jul 29, 2022

shahzadlone added 6 commits August 3, 2022 01:31

wip: Remove circli config.

4e7b728

wip: Migrate the change detect workflows.

d52b9d3

wip: Migrate the Workflow for running all the tests.

c481b87

wip: Clean up previous actions and make more consistent to the new ones.

a0d1da9

wip: Add a new Workflow in charge of ensuring the binary can start

89c02b1

properly.

wip: Update and Cleanup Makefile

9e52dc8

shahzadlone force-pushed the lone/tools/migrate-all-circli-pipelines-to-github-actions branch from 702ff82 to 9e52dc8 Compare August 3, 2022 05:33

shahzadlone merged commit 3973c2d into develop Aug 3, 2022

shahzadlone deleted the lone/tools/migrate-all-circli-pipelines-to-github-actions branch August 3, 2022 05:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: Migrate from CircleCi to GitHub Actions #679

tools: Migrate from CircleCi to GitHub Actions #679

shahzadlone commented Jul 27, 2022 •

edited

Loading

This comment was marked as off-topic.

orpheuslummis commented Jul 27, 2022

shahzadlone commented Jul 27, 2022 •

edited

Loading

fredcarle left a comment

fredcarle Jul 28, 2022

shahzadlone Jul 28, 2022 •

edited

Loading

fredcarle Jul 28, 2022

orpheuslummis left a comment

shahzadlone commented Jul 29, 2022

tools: Migrate from CircleCi to GitHub Actions #679

tools: Migrate from CircleCi to GitHub Actions #679

Conversation

shahzadlone commented Jul 27, 2022 • edited Loading

Relevant issue(s)

Description

Merge Process

Limitations

Tasks

How has this been tested?

This comment was marked as off-topic.

orpheuslummis commented Jul 27, 2022

shahzadlone commented Jul 27, 2022 • edited Loading

fredcarle left a comment

Choose a reason for hiding this comment

fredcarle Jul 28, 2022

Choose a reason for hiding this comment

shahzadlone Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

fredcarle Jul 28, 2022

Choose a reason for hiding this comment

orpheuslummis left a comment

Choose a reason for hiding this comment

shahzadlone commented Jul 29, 2022

shahzadlone commented Jul 27, 2022 •

edited

Loading

shahzadlone commented Jul 27, 2022 •

edited

Loading

shahzadlone Jul 28, 2022 •

edited

Loading