-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
destination-s3: add file transfer #46302
base: master
Are you sure you want to change the base?
destination-s3: add file transfer #46302
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @stephane-airbyte and the rest of your teammates on Graphite |
5649507
to
0a94310
Compare
@@ -36,6 +36,11 @@ object UploadFormatConfigFactory { | |||
FileUploadFormat.PARQUET -> { | |||
UploadParquetFormatConfig(formatConfig) | |||
} | |||
FileUploadFormat.RAW_FILES -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh nice. This sidesteps any questions w/r/t conversion.
0a94310
to
7baeb75
Compare
6f87147
to
52c0fe1
Compare
7baeb75
to
2dd1a87
Compare
52c0fe1
to
95a7d03
Compare
2dd1a87
to
75cc841
Compare
95a7d03
to
721ddfa
Compare
75cc841
to
f0f2536
Compare
e2bb0c0
to
e1dd9ce
Compare
721ddfa
to
8a78c22
Compare
a371928
to
cd74813
Compare
8a78c22
to
48a9e9e
Compare
9456da1
to
76dcbe6
Compare
/publish-java-cdk --force=true
|
/publish-java-cdk force=true
|
76dcbe6
to
f654a2b
Compare
eaee8a1
to
8fa5bff
Compare
010e67d
to
25ad892
Compare
8fa5bff
to
ba25b14
Compare
25ad892
to
8921fd6
Compare
8921fd6
to
cc6fccc
Compare
ba25b14
to
45446b2
Compare
45446b2
to
48f7929
Compare
cc6fccc
to
5abe9d2
Compare
48f7929
to
cb1bfbe
Compare
5abe9d2
to
c66040f
Compare
c66040f
to
367e2f0
Compare
} | ||
val flushFunction = | ||
if (featureFlags.useFileTransfer()) { | ||
FileTransferDestinationFlushFunction( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What determines whether the feature flag is set? The fact that the source is flagged as a file source? Explicit opt-in at the sync level?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair question.
Basically, we need the source configuration to have a specific parameter enabled (I don't know the details of the parameter) AND the destination needs to have supportsFileTransfer
set to true
in its metadata.yaml
. If those 2 conditions are true, then the 2 variables are set accordingly, a common volume is mounted on both containers, and it's expected that all records are file-based instead of record-based.
If the source config has the parameter set to true
and the destination doesn't support file transfer, the platform will throw an exception
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. The shim seems like it's in the best place, and the file flush function is straightforward. I didn't have enough time to go over the tests in detail, but high-level how we're adding the file option to the docker env is clear.
One question about the env variables just to help me plan for the new CDK, but that's my own curiosity.
adding file transfer to destinaiton-s3
file transfer and record-based sync are exclusive. The platform will set the environment variables
USE_FILE_TRANSFER
totrue
andAIRBYTE_STAGING_DIRECTORY
to the mounting point of the staging directory when the destination supports file transfer and the source enabled it in its config.destination-s3 will check the
USE_FILE_TRANSFER
to decide whether to enable file transfer or record-based sync.Record-based integration tests are all passing, and there's an extra test that makes sure file-based transfer is disabled.