[EPIC] Connector Release Confidence and Automation

[evantahler]

Overview

In order to increase the speed-to-release for connectors, this epic will contain work that will allow the team to have higher confidence that underlying changes to the CDK and related tools like the testing framework (SAT) are working as expected.

Stories

• Multiple connectors can be tested at once by a single Github comment (both internal unit tests and SAT)
• Ensure that every connector is tested weekly (both internal unit tests and SAT)
• Fix any currently broken or flaky connectors
• Multiple connectors can be version-bumped at once by a single comment
• Multiple connectors can be released at once by a single Github comment

Rationale

The workflow in question comes from #11668, in which we intend to change many connectors from using JSON spec files to YML files. These many connectors will all need:

1. To run any unit tests within the connector package
2. To run the SAT tests against any of the changed connectors
3. To automatically update the version (patch version +1)
4. To automatically update the release notes
5. To automatically publish a new version of the connector
6. To automatically update any related docs
7. To automatically update any metadata (connectors maniefest?)

We anticipate more of these types of bulk changes in the future, and having good tooling around this process will help the team's velocity.

In addition to needing to test and publish in bulk, we should also be testing all connectors regularly to ensure that there are no regressions.

Assumptions

• The plumbing that runs the / commands from Github comments for both /publish and /test is similar, and we can reuse our work
• We are comfortable auto-generating release notes for patch/minor changes from either commit messages or Github comments

Links
Possibly solves https://github.com/airbytehq/airbyte-internal-issues/issues/509

[girarda]

We should also think about how to propagate the upgrades to cloud to automated the end-to-end process (could be as a follow up ticket)

[evantahler]

I wonder what the ergonomics of releasing or testing many connectors at once looks like. At minimum I propose we support:

1. single connector (like we do today): /bump-patch-and-publish connector=connectors/source-postgres
2. multiple connectors via list: /bump-patch-and-publish connector=connectors/source-postgres,connectors/source-mysql
3. multiple connectors via glob or regular expression: /bump-patch-and-publish connector=*

[noahkawasaki-airbyte]

Additionally on the point of the GHA comments, some of connectors/source-* and connectors= stuff is redundant/unnecessary to type out (unless there's something I'm missing). Would be nice to just say /bump-patch-and-publish-connectors source-postgres,source-s3,destination-something just reduce some of the repetitive strings

[noahkawasaki-airbyte]

Two issues related:

https://github.com/airbytehq/airbyte-internal-issues/issues/562
#12031

[noahkawasaki-airbyte]

I've filed an issue about your point on number 5 here: #12108

[noahkawasaki-airbyte]

I'm going to experiment on tackling number 3 and 5 here: #12405 FYI!

Just want to make sure in case anyone else was also working on the same issues