ci: Add workflow that builds DefraDB AMI upon tag push #1304

shahzadlone · 2023-04-06T17:03:09Z

Relevant issue(s)

Resolves #1241

Description

Move the action from: https://github.com/sourcenetwork/defradb-cloud inside this repo as it was intended.

Enables an automated creation of a defradb AMI whenever a tag is pushed.

The action won't be visible in the PR as it is to only be triggered on tag push.
However I did test it in this PR and the triggered CI action run can be found here: https://github.com/sourcenetwork/defradb/actions/runs/4631156601/jobs/8193666739?pr=1304

Note:

(1) I didn't write everything included in the pr, this was an outcome of our partnership with AWS 3rd team that was helping us.
(2) The packer workflow (i.e. ami creation) uses and might seem familiar to: https://github.com/sourcenetwork/defradb/blob/develop/tools/scripts/deploy_defradb.sh
(3) The creation of ami was tested and works properly.
(4) I spotted some bugs in the terraform workflow (mostly how the secret was being handled, which is now fixed).
(5) Due to (4) not as confident that the deployment of an image to spin up an ec2 instance will go smoothly as the workflow currently stands (I would not worry about that yet, as we can fix it when we get approved on marketplace and can actually test it end-to-end).

NEED FEEDBACK:

Should the AMI be created for any new tag that is pushed? Currently it will only run for tags with full valid semver: i.e. v0.5.1, but won't run on v0.5 or v0.4.1-alpha

Tasks

I made sure the code is well commented, particularly hard-to-understand areas.
I made sure the repository-held documentation is changed accordingly.
I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

Triggering the workflow and ensuring the AMI is created on amazon.

Specify the platform(s) on which this was tested:

WSL 2 (Manjaro)

AndrewSisley · 2023-04-06T17:40:57Z

tools/cloud/aws/terraform/.terraform.lock.hcl

question: Should this file not be excluded from the repo? (I have no idea what best practice is with these, but is common enough to do with similar)

No, it should stay.

More context: https://developer.hashicorp.com/terraform/language/files/dependency-lock#lock-file-location

"
Lock File Location
The dependency lock file is a file that belongs to the configuration as a whole, rather than to each separate module in the configuration. For that reason Terraform creates it and expects to find it in your current working directory when you run Terraform, which is also the directory containing the .tf files for the root module of your configuration.
...
Terraform automatically creates or updates the dependency lock file each time you run the terraform init command. You should include this file in your version control repository so that you can discuss potential changes to your external dependencies via code review, just as you would discuss potential changes to your configuration itself.
"

cheers for the explanation! :)

fredcarle

I'm approving this with the caveat that I'm not an expert and if something was wrongly configured I'm not sure I would spot it. I do have a few questions however.

Maybe give others the day to chime in in case they know more than I do.

fredcarle · 2023-04-06T17:39:17Z

tools/cloud/aws/terraform/workspaces/dev-backend.conf

+# DEV BACKEND CONFIG SETTINGS
+####################################################################################
+# S3 backend
+bucket                  = "source-tfstate-dev" # REPLACE WITH YOUR BUCKET NAME


question: should the comment still be there?

Yes, due to (5) from PR description. Don't want to lose any help / hint until then.

fredcarle · 2023-04-06T17:39:22Z

tools/cloud/aws/terraform/workspaces/prod-backend.conf

+# DEV BACKEND CONFIG SETTINGS
+####################################################################################
+# S3 backend
+bucket                  = "source-tfstate-prod" # REPLACE WITH YOUR BUCKET NAME


question: should the comment still be there?

Yes, due to (5) from PR description. Don't want to lose any help / hint until then.

fredcarle · 2023-04-06T17:39:43Z

tools/cloud/aws/terraform/workspaces/test-backend.conf

+# TEST BACKEND CONFIG SETTINGS
+####################################################################################
+# S3 backend
+bucket                  = "source-tfstate-test" # REPLACE WITH YOUR BUCKET NAME


question: should the comment still be there?

Yes, due to (5) from PR description. Don't want to lose any help / hint until then.

fredcarle · 2023-04-06T17:40:21Z

.github/workflows/build-ami-with-packer.yml

+
+env:
+  PACKER_LOG: 1
+  # RELEASE_VERSION: v0.4.0


question: Should this line be uncommented or removed?

Would rather leave it in, is there for testing manually.

fredcarle · 2023-04-06T17:40:25Z

.github/workflows/build-ami-with-packer.yml

+
+      - name: Environment version target
+        run: echo "RELEASE_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
+        # run: echo ${{ env.RELEASE_VERSION }}


question: Should this line be uncommented or removed?

Would rather leave it in, is there for testing manually.

AndrewSisley · 2023-04-06T17:46:31Z

I've skimmed over it, but I have zero experience with terraform, limited and out of date aws experience, and only a very vague notion of what this is for :)

shahzadlone · 2023-04-06T17:59:35Z

I've skimmed over it, but I have zero experience with terraform, limited and out of date aws experience, and only a very vague notion of what this is for :)

No worries @AndrewSisley was in a similar boat a couple of weeks back haha. Thanks for taking a look at it nonetheless =D

Ensured the action passes and makes a new AMI, that also shows up on AWS console.

codecov · 2023-04-09T09:00:24Z

Codecov Report

Merging #1304 (3b5dfac) into develop (b08fe6f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #1304   +/-   ##
========================================
  Coverage    70.47%   70.47%           
========================================
  Files          184      184           
  Lines        17823    17823           
========================================
  Hits         12561    12561           
  Misses        4310     4310           
  Partials       952      952

see 1 file with indirect coverage changes

- Resolves #1241 - Description: Enables an automated creation of a DefraDB AMI whenever a tag is pushed.

…#1304) - Resolves sourcenetwork#1241 - Description: Enables an automated creation of a DefraDB AMI whenever a tag is pushed.

shahzadlone added ci/build This is issue is about the build or CI system, and the administration of it. meta/dev These issues related to the meta elements of the code base like dev flow, repo logistics, and PM action/no-benchmark Skips the action that runs the benchmark. labels Apr 6, 2023

shahzadlone added this to the DefraDB v0.5 milestone Apr 6, 2023

shahzadlone self-assigned this Apr 6, 2023

shahzadlone requested a review from a team April 6, 2023 17:18

AndrewSisley reviewed Apr 6, 2023

View reviewed changes

fredcarle approved these changes Apr 6, 2023

View reviewed changes

shahzadlone force-pushed the lone/ci/creation-of-defra-ami-on-aws branch 2 times, most recently from 8c2bc54 to 3b5dfac Compare April 9, 2023 08:59

shahzadlone added 4 commits April 9, 2023 04:59

Add the recommended ignores to .gitignore

5d259a9

PR: Add packer - Build the AMI image

7fe7d74

PR: Add terraform - Deploy the AMI image

8f077ee

PR: Change to trigger on tag push only.

3b5dfac

Ensured the action passes and makes a new AMI, that also shows up on AWS console.

shahzadlone changed the title ~~ci: Add workflow to that builds defradb AMI upon tag push~~ ci: Add workflow that builds DefraDB AMI upon tag push Apr 9, 2023

shahzadlone merged commit 01bfd49 into develop Apr 9, 2023

shahzadlone deleted the lone/ci/creation-of-defra-ami-on-aws branch April 9, 2023 10:54

shahzadlone added a commit that referenced this pull request Apr 13, 2023

ci: Add workflow that builds DefraDB AMI upon tag push (#1304)

2e24fdf

- Resolves #1241 - Description: Enables an automated creation of a DefraDB AMI whenever a tag is pushed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Add workflow that builds DefraDB AMI upon tag push #1304

ci: Add workflow that builds DefraDB AMI upon tag push #1304

shahzadlone commented Apr 6, 2023 •

edited

Loading

AndrewSisley Apr 6, 2023

shahzadlone Apr 6, 2023 •

edited

Loading

AndrewSisley Apr 6, 2023

fredcarle left a comment •

edited

Loading

fredcarle Apr 6, 2023

shahzadlone Apr 6, 2023

fredcarle Apr 6, 2023

shahzadlone Apr 6, 2023

fredcarle Apr 6, 2023

shahzadlone Apr 6, 2023

fredcarle Apr 6, 2023

shahzadlone Apr 6, 2023

fredcarle Apr 6, 2023

shahzadlone Apr 6, 2023

AndrewSisley commented Apr 6, 2023

shahzadlone commented Apr 6, 2023

codecov bot commented Apr 9, 2023 •

edited

Loading

ci: Add workflow that builds DefraDB AMI upon tag push #1304

ci: Add workflow that builds DefraDB AMI upon tag push #1304

Conversation

shahzadlone commented Apr 6, 2023 • edited Loading

Relevant issue(s)

Description

Note:

NEED FEEDBACK:

Tasks

How has this been tested?

Choose a reason for hiding this comment

shahzadlone Apr 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fredcarle left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewSisley commented Apr 6, 2023

shahzadlone commented Apr 6, 2023

codecov bot commented Apr 9, 2023 • edited Loading

Codecov Report

shahzadlone commented Apr 6, 2023 •

edited

Loading

shahzadlone Apr 6, 2023 •

edited

Loading

fredcarle left a comment •

edited

Loading

codecov bot commented Apr 9, 2023 •

edited

Loading