Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump and Publish all destinations #14472

Closed
benmoriceau opened this issue Jul 6, 2022 · 4 comments
Closed

Bump and Publish all destinations #14472

benmoriceau opened this issue Jul 6, 2022 · 4 comments
Assignees

Comments

@benmoriceau
Copy link
Contributor

benmoriceau commented Jul 6, 2022

Destinations can now support per-stream state, with the main change being that destinations now need to keep track of the latest state message they have received and committed per-stream. Previously, destinations only needed to store the most recent state message in general, without knowing about the stream(s) involved.

We may need to re-publish all destinations to be sure that the shared base code is in use.

  1. The shared database base code was updated in this PR - Update destinations to handle new state messages #13670
  2. Are there any destinations that do not share the same Java base code and need to be updated manually?

We should compile a list of minimum version for each destination that supports per-stream state. This will likely need to be shared with the community in some way, probably via our docs.

@benmoriceau benmoriceau self-assigned this Jul 6, 2022
@evantahler evantahler changed the title Update all dest? Update all destinations? Jul 7, 2022
@evantahler evantahler changed the title Update all destinations? Bump and Publish all destinations? Jul 7, 2022
@evantahler
Copy link
Contributor

evantahler commented Jul 7, 2022

Note: Thanks to #13864, we can publish lots of connectors at once via a single slash command now!

@edgao
Copy link
Contributor

edgao commented Jul 7, 2022

There are a few non-java destinations (if you grep the build.gradle for airbyte-python you can find them) so they'll need manual work.

also, does the Java code update work for non-database connectors? E.g. destination-s3 doesn't use the JDBC logic at all.

@evantahler
Copy link
Contributor

evantahler commented Jul 12, 2022

This Google Sheeet contains an audit of all of our destinations. It has been determined that 10 connectors need to be bumped and re-published to use the new Java destination code introduced via #13670. Most Java destinations have been published recently due to the normal flow of work.

None of the Python destinations seem to be doing anything complex with STATE messages.
They either just echo the STATE message back, or do a full flush than echo the message. Either way, there is no logic that would differ with per-stream state.

cc @sherifnada and @grishick

@evantahler evantahler changed the title Bump and Publish all destinations? Bump and Publish all destinations Jul 12, 2022
@evantahler
Copy link
Contributor

@benmoriceau I think this story is complete? We will be re-publishing the destinations again via #14851

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants