README.md

Installation guide

This guide describes how to set up Four Keys with your GitHub or GitLab project. The main steps are:

1. Forking or cloning this repository
2. Building required images with Cloud Build
3. Providing values for required Terraform variables
4. Executing Terraform to deploy resources
5. Generating sample data (optional)

Alternatively, to deploy Four Keys as a remote Terraform module, see terraform/modules/fourkeys/README.md

Before you begin

To deploy Four Keys with Terraform, you will first need:

• A Google Cloud project with billing enabled
• The owner role assigned to you on the project
• The Google Cloud CLI and Terraform installed on your local machine. We recommend deploying from Cloud Shell on your Google Cloud project.

Deploying with Terraform

1. Set an environment variable indicating your Google Cloud project ID:

   export PROJECT_ID="YOUR_PROJECT_ID"

2. Clone the fourkeys git repository and change into the root directory

   git clone https://github.com/dora-team/fourkeys.git && cd fourkeys
   

3. Use Cloud Build to build and push containers to Google Container Registry for the dashboard, event-handler:

   gcloud builds submit dashboard --config=dashboard/cloudbuild.yaml --project $PROJECT_ID && \
   gcloud builds submit event-handler --config=event-handler/cloudbuild.yaml --project $PROJECT_ID
   

4. Use Cloud Build to build and push containers to Google Container Registry for the parsers you plan to use. See the bq-workers for available options. GitHub for example:

   gcloud builds submit bq-workers --config=bq-workers/parsers.cloudbuild.yaml --project $PROJECT_ID --substitutions=_SERVICE=github
   

5. Change your working directory to terraform/example and rename terraform.tfvars.example to terraform.tfvars

   cd terraform/example && mv terraform.tfvars.example terraform.tfvars
   

6. Edit terraform.tfvars with values for the required variables. See variables.tf for a list of the variables, along with their descriptions and default values. Values not defined in terraform.tfvars will use default values defined in variables.tf

7. Run the following commands from the example directory:

   terraform init to inialize Terraform and download the module

   terraform plan to preview changes.

   terraform apply to deploy the resources.

Generating mock data

To test your Four Keys deployment, you can generate mock data that simulates events from a GitHub repository.

1. Export your event handler URL an environment variable. This is the webhook URL that will receive events:

   export WEBHOOK=`gcloud run services list --project $PROJECT_ID | grep event-handler | awk '{print $4}'`

2. Export your event handler secret to an environment variable. This is the secret used to authenticate events sent to the webhook:

   export SECRET=`gcloud secrets versions access 1 --secret=event-handler --project $PROJECT_ID`

3. From the root of the fourkeys project run:

   python3 data-generator/generate_data.py --vc_system=github

   You can see these events being run through the pipeline:

   • The event handler logs show successful requests
   • The Pub/Sub topic show messages posted
   • The BigQuery GitHub parser show successful requests

4. View the generated data in the events_raw table in with bq:

   bq query --project_id $PROJECT_ID 'SELECT * FROM four_keys.events_raw WHERE source = "githubmock";'

   Or query the table directly in BigQuery:

   SELECT * FROM four_keys.events_raw WHERE source = 'githubmock';