How to Test an API Pull Request

The API is a tightly-coupled codebase and that means even simple changes can result breaking bugs for users. Follow this playbook to ensure we can add new code.

IMPORTANT: All testing should be performed by a human from the selected environment's runnable-angular app.

Setup

The author of the PR is responsible for setting up the test by doing the following:

1. Merge Latest Master and Fix Conflicts

• By doing this you can ensure that fixes to other problems do not persist in your branch

2. Setup a testing environment for the PR's branch

• Choose the Environment: staging, gamma, or epsilon
• Deploy the PR branch
• Sanity Check that the environment works (login, click things, ensure nothing errors out)

3. Find a Tester

• Slack #general: @channel Need a tester on Gamma for my PR: <link to PR>
• Walk up to someone's desk and personally ask them
• Ask a trusted Engineer/Manager to identify someone

4. Update the PR Description to indicate that someone is responsible for testing

Once the tester has committed to doing the test, they should proceed with the following steps.

The QA Tests

The following section has a list of tests that a tester should perform when evaluating a PR for merge. Here are the ground rules:

• The tester should not be asked to test until all commits are considered final.
• The tester should fail-fast, meaning if one of the below checks fails they should stop testing an inform the author
• If a new commit is made to the PR branch, then all tests must be redone to ensure a breaking change was not introduced.

Once all tests below have been verified as working by the tester the author should immediately merge, tag, and deploy the changes to production.

Test 1: New Repository Containers

1. From the configuration view add a new repository project
2. Enter the correct information in the helper wizard
3. Build the project
4. Check: Do the build logs appear after starting the container build?
5. Wait for the build to complete
6. Check: Can you get terminal access an run commands for the new container?

Test 2: New Non-Repository Containers

1. From the configuration view add a new non-repo container (Redis, MongoDB, etc.)
2. Build the container
3. Check: Do the build logs appear after starting the container build?
4. Wait for the build to complete
5. Check: Can you get terminal access an run commands for the new container?

Test 3: Rebuild Repo Container

1. From the containers view choose a running container
2. Rebuild the container without cache
3. Check: Do the build logs appear after initiating the rebuild?
4. Wait for the build to complete
5. Check: Can you get terminal access an run commands for the rebuilt container?

Test 4: GitHub Webhooks

1. Choose a running container to which you have GitHub write permissions
2. Create a new branch and push the branch to origin
3. Check: Does the branch appear as "building" in the UI?
4. Wait for the build to complete
5. Check: Can you use the terminal to run commands on the new branch container?

Test 5: Container-to-Container DNS (If applicable)

1. Choose a running container that is connected to another running container via an DNS rules
2. From the terminal attempt to CURL the url to the remote container
3. Check: Ensure the request was routed to the correct container (e.g. check remote container logs at the time of curl)

Test 6: Navi URLs (If applicable)

1. Check: Ensure master-to-master URL works as expected
2. Check: Ensure master-to-branch URL works as expected
3. Check: Ensure branch-to-master URL works as expected
4. Check: Ensure branch-to-branch URL works as expected