Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

workflow for publishing artifacts for cloud #14199

Merged
merged 1 commit into from
Jun 28, 2022
Merged

workflow for publishing artifacts for cloud #14199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 40 additions & 21 deletions .github/workflows/publish-oss-for-cloud.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Publish OSS Artifacts for Cloud
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
group: ${{ github.workflow }}-${{ inputs.oss_ref || github.sha }}
davinchia marked this conversation as resolved.
Show resolved Hide resolved

on:
workflow_dispatch:
Expand Down Expand Up @@ -46,7 +46,7 @@ jobs:
github-token: ${{ needs.find_valid_pat.outputs.pat }}

generate-tags:
name: "Generate Tags"
name: "Generate Dev and Master Tags"
runs-on: ubuntu-latest
outputs:
dev_tag: ${{ steps.set-outputs.outputs.dev_tag }}
Expand All @@ -69,16 +69,17 @@ jobs:
echo "::set-output name=dev_tag::dev-${commit_sha}"

# If this commit is on the master branch, also set master_tag
if test 0 -eq $(git merge-base --is-ancestor "${commit_sha}" master); then
if $(git merge-base --is-ancestor "${commit_sha}" master); then
echo "::set-output name=master_tag::${commit_sha}"
fi

oss-branch-build:
name: "Build and Push Images from Branch"
name: "Gradle Build and Publish"
needs:
- start-runner
- generate-tags
runs-on: ${{ needs.start-runner.outputs.label }}
environment: more-secrets
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
Expand All @@ -90,37 +91,55 @@ jobs:
with:
branch_version_tag: ${{ needs.generate-tags.outputs.dev_tag }}

- name: Publish Dev Jars
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be missing something, why move the jar publishing to be before the Docker image publishing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't actually matter, but I just wanted to try splitting up the steps into different jobs. To do this, the publish step depends on some java setup steps in the Build Branch step, so I had to keep these within the same job, whereas the docker stuff I could split out into a separate job.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

understand, thanks.

env:
CLOUDREPO_USER: ${{ secrets.CLOUDREPO_USER }}
CLOUDREPO_PASSWORD: ${{ secrets.CLOUDREPO_PASSWORD }}
run: VERSION=${{ needs.generate-tags.outputs.dev_tag }} SUB_BUILD=PLATFORM ./gradlew publish
shell: bash

- name: Publish Master Jars
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remind me again why we are publishing master jars with this if condition?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was in the tech spec https://docs.google.com/document/d/1Os-GMcDwJuqVNKI8yUmSbOevYt92N1W3BWY-Dj8DqdY/edit#heading=h.pyzrac8d5gsg where commits from the master branch would create different tags/jar versions from non master commits. I think we can actually drop this feature though because I don't actually see how we'd use these tags. I think it'd be easier to just assume the tag will always be dev-<short_sha>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Peter. I do think it's useful though agree this is an incremental. I think it's worth adding a comment here explaining why. also think it's fine to remove if it seems confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I think we can just leave it as is for now, but if publishing master jars seems to noticeably slow down this workflow then we can probably remove this step and comment on why

if: needs.generate-tags.outputs.master_tag != ''
env:
CLOUDREPO_USER: ${{ secrets.CLOUDREPO_USER }}
CLOUDREPO_PASSWORD: ${{ secrets.CLOUDREPO_PASSWORD }}
run: VERSION=${{ needs.generate-tags.outputs.master_tag }} SUB_BUILD=PLATFORM ./gradlew publish
shell: bash

docker-push:
name: "Push Docker Images"
needs:
- start-runner
- generate-tags
- oss-branch-build
runs-on: ${{ needs.start-runner.outputs.label }}
steps:
- name: Login to Docker (on Master)
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_PASSWORD }}

- name: Push Dev Docker Images
- name: Prepare Docker buildx
run: |
GIT_REVISION=$(git rev-parse HEAD)
[ [ -z "$GIT_REVISION" ] ] && echo "Couldn't get the git revision..." && exit 1
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx create --name oss-buildx --driver docker-container --use
VERSION=${{ needs.generate-tags.outputs.dev_tag }}
VERSION=$VERSION GIT_REVISION=$GIT_REVISION docker buildx bake --platform=linux/amd64,linux/arm64 -f docker-compose-cloud.build.yaml --push
docker buildx rm oss-buildx
shell: bash

- name: Push Master Docker Images
if: needs.generate-tags.outputs.master_tag != ""
- name: Set Git Revision
run: |
GIT_REVISION=$(git rev-parse HEAD)
[ [ -z "$GIT_REVISION" ] ] && echo "Couldn't get the git revision..." && exit 1
docker buildx create --name oss-buildx --driver docker-container --use
VERSION=${{ needs.generate-tags.outputs.master_tag }}
VERSION=$VERSION GIT_REVISION=$GIT_REVISION docker buildx bake --platform=linux/amd64,linux/arm64 -f docker-compose-cloud.build.yaml --push
docker buildx rm oss-buildx
echo "GIT_REVISION=${GIT_REVISION}" >> $GITHUB_ENV
shell: bash

- name: Publish Dev Jars
- name: Push Docker Images
env:
VERSION: ${{ needs.generate-tags.outputs.dev_tag }}
ALT_TAG: ${{ needs.generate-tags.outputs.master_tag }}
run: GIT_REVISION=$GIT_REVISION docker buildx bake -f docker-compose-cloud.buildx.yaml --push
shell: bash
run: VERSION=${{ needs.generate-tags.outputs.dev_tag }} SUB_BUILD=PLATFORM ./gradlew publish
- name: Publish Master Jars
if: needs.generate-tags.outputs.master_tag != ""

- name: Cleanup Docker buildx
run: docker buildx rm oss-buildx
shell: bash
run: VERSION=${{ needs.generate-tags.outputs.master_tag }} SUB_BUILD=PLATFORM ./gradlew publish
70 changes: 70 additions & 0 deletions docker-compose-cloud.buildx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Defines the minimum set of images needed to run Cloud.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally we could just modify the existing docker-compose-cloud.build.yaml instead of creating a new (mostly duplicate) file here. However, the workflow runner amis contain an outdated version of docker-compose that errors out when encountering x- fields.
Based on the latest spec, docker-compose is supposed to ignore all unknown extensions without failing (and this is the case when running docker-compose locally).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove the old docker-compose build yaml file?

from my quick search, I think we should be able to.

Screenshot 2022-06-27 at 5 36 10 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old yaml is used by the build-and-push-branch action which is called from deploy-command.yml in cloud

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right. I should be clearer. I mean if the docker compose ignores unknown extensions, can we replace all usages of the old docker compose file with the new buildx one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh right yes, if we can update docker compose to ignore the unknown extensions then we can replace all usages of the old compose file with this new one

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeap. can we either create an issue or note this down somewhere so we don't forget?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created https://github.com/airbytehq/airbyte-cloud/issues/1949

# Used to push OSS images that Cloud depends on.

version: "3.7"

services:
worker:
image: airbyte/worker:${VERSION}
build:
dockerfile: Dockerfile
context: airbyte-workers/build/docker
labels:
io.airbyte.git-revision: ${GIT_REVISION}
args:
VERSION: ${VERSION}
x-bake:
tags:
- airbyte/worker:${VERSION}
- airbyte/worker:${ALT_TAG:-${VERSION}}
platforms:
- linux/amd64
- linux/arm64
webapp:
image: airbyte/webapp:${VERSION}
build:
dockerfile: Dockerfile
context: airbyte-webapp/build/docker
labels:
io.airbyte.git-revision: ${GIT_REVISION}
args:
VERSION: ${VERSION}
x-bake:
tags:
- airbyte/webapp:${VERSION}
- airbyte/webapp:${ALT_TAG:-${VERSION}}
platforms:
- linux/amd64
- linux/arm64
metric-reporter:
image: airbyte/metrics-reporter:${VERSION}
build:
dockerfile: Dockerfile
context: airbyte-metrics/reporter/build/docker
labels:
io.airbyte.git-revision: ${GIT_REVISION}
args:
VERSION: ${VERSION}
x-bake:
tags:
- airbyte/metrics-reporter:${VERSION}
- airbyte/metrics-reporter:${ALT_TAG:-${VERSION}}
platforms:
- linux/amd64
- linux/arm64
container-orchestrator:
image: airbyte/container-orchestrator:${VERSION}
build:
dockerfile: Dockerfile
context: airbyte-container-orchestrator/build/docker
labels:
io.airbyte.git-revision: ${GIT_REVISION}
args:
VERSION: ${VERSION}
x-bake:
tags:
- airbyte/container-orchestrator:${VERSION}
- airbyte/container-orchestrator:${ALT_TAG:-${VERSION}}
platforms:
- linux/amd64
- linux/arm64