Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for publishing SQL schema from scratch, and publishing from a branch #1751

Merged
merged 1 commit into from
Aug 21, 2023

Conversation

inahga
Copy link
Contributor

@inahga inahga commented Aug 18, 2023

Fixes #1735.

Appropriately handle the case where we're starting a new release and need to have the migrations directory created. The force flag is not required for this operation (nor should it IMO).

I also fix the job to allow us deploying from a branch, in which case the directory will just be the branch name. This is mainly for testing this PR but I can see it being useful in niche circumstances.

To cleanly accomplish this I replace gcloud with rclone, for a few reasons:

  • rclone is a stable interface, gcloud alpha is not.
  • rclone has more predictable and favorable semantics, i.e. we don't need any special handling for the case where the remote migrations directory doesn't exist.
  • gcloud alpha doesn't have a directory existence test function, nor does it provide any sensible way of parsing the kind of error that occurs. So to ignore a missing remote directory we have to do || true or do if grep "not found" <<<$OUTPUT, neither of which are very good solutions.
Expand for details on the testing I did for this:
  • Setting up schema from scratch https://github.com/divviup/janus/actions/runs/5906546433/job/16022784957.
    🐧 rclone ls --verbose gs:/janus-artifacts-sql-schemas | grep inahga/test
         1734 inahga/test-1735/db/00000000000001_initial_schema.down.sql
        24634 inahga/test-1735/db/00000000000001_initial_schema.up.sql
    
  • Add new migration to existing schema https://github.com/divviup/janus/actions/runs/5906561538/job/16022835739
    🐧 rclone ls --verbose gs:/janus-artifacts-sql-schemas | grep inahga/test
         1734 inahga/test-1735/db/00000000000001_initial_schema.down.sql
        24634 inahga/test-1735/db/00000000000001_initial_schema.up.sql
            4 inahga/test-1735/db/00000000000002_foo.down.sql
            4 inahga/test-1735/db/00000000000002_foo.up.sql
    
  • Attempt modifying existing schema, fails as expected. https://github.com/divviup/janus/actions/runs/5906574446/job/1602287765. (output was same as above).
  • I assert that I know what I'm doing, and force the migration to continue anyway with the force flag. https://github.com/divviup/janus/actions/runs/5906628503/job/16023031038
    🐧 rclone ls --verbose gs:/janus-artifacts-sql-schemas | grep inahga/test
         1734 inahga/test-1735/db/00000000000001_initial_schema.down.sql
            4 inahga/test-1735/db/00000000000001_initial_schema.up.sql
            4 inahga/test-1735/db/00000000000002_foo.down.sql
            4 inahga/test-1735/db/00000000000002_foo.up.sql
    
  • Isolation test that asserts the version parsing changes haven't broken anything:
    🐧 cat test.sh
    #!/usr/bin/env bash
    
    if grep -P "^refs/tags/" <<<"$GITHUB_REF" >/dev/null; then
            VERSION=${GITHUB_REF/refs\/tags\//}
            MINOR_VERSION="$(echo "$VERSION" | awk -F'.' '{if (NF != 3) {exit 1}; printf "%s.%s", $1, $2}')"
            echo "version=$MINOR_VERSION"
    elif grep -P "^refs/heads/" <<<"$GITHUB_REF" >/dev/null; then
            BRANCH=${GITHUB_REF/refs\/heads\//}
            echo "version=$BRANCH"
    else
            echo "Unable to parse version information"
            exit 1
    fi
    
    🐧 GITHUB_REF=refs/tags/0.1.2 ./test.sh
    version=0.1
    🐧 GITHUB_REF=refs/tags/0.subscriber-01.2 ./test.sh
    version=0.subscriber-01
    🐧 GITHUB_REF=refs/heads/inahga/test ./test.sh
    version=inahga/test
    🐧 GITHUB_REF=refs/heads/inahga/test/foo/bar ./test.sh
    version=inahga/test/foo/bar
    🐧 GITHUB_REF=refs/heads/branch ./test.sh
    version=branch
    🐧 GITHUB_REF=idk/wtf ./test.sh
    Unable to parse version information

@inahga inahga requested a review from a team as a code owner August 18, 2023 20:15
@inahga inahga merged commit d3182ac into main Aug 21, 2023
7 checks passed
@inahga inahga deleted the inahga/fix-1735 branch August 21, 2023 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

publish-sql-schema doesn't handle case where there are no existing migrations for a minor version
2 participants