destination-s3: add file transfer #46302
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @stephane-airbyte and the rest of your teammates on |
octavia-squidington-iii
added
area/connectors
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
5649507
to
0a94310
Compare
| @@ -36,6 +36,11 @@ object UploadFormatConfigFactory { | |||
| FileUploadFormat.PARQUET -> { | |||
| UploadParquetFormatConfig(formatConfig) | |||
| } | |||
| FileUploadFormat.RAW_FILES -> | |||
There was a problem hiding this comment.
Oh nice. This sidesteps any questions w/r/t conversion.
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
0a94310
to
7baeb75
Compare
stephane-airbyte
changed the base branch from
stephane/09-30-cdk-java_add_file_transfer_mount_to_destinationacceptancetest
to
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
stephane-airbyte
force-pushed
the
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
branch
from
6f87147
to
52c0fe1
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
7baeb75
to
2dd1a87
Compare
stephane-airbyte
force-pushed
the
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
branch
from
52c0fe1
to
95a7d03
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
2dd1a87
to
75cc841
Compare
stephane-airbyte
force-pushed
the
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
branch
from
95a7d03
to
721ddfa
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
75cc841
to
f0f2536
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
8 times, most recently
from
e2bb0c0
to
e1dd9ce
Compare
stephane-airbyte
force-pushed
the
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
branch
from
721ddfa
to
8a78c22
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
2 times, most recently
from
a371928
to
cd74813
Compare
stephane-airbyte
force-pushed
the
stephane/10-02-cdk-java_reorganize_the_destinationaccptancetest_to_split_out_the_actual_tests_from_all_the_util_methods
branch
from
8a78c22
to
48a9e9e
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
9456da1
to
76dcbe6
Compare
|
/publish-java-cdk --force=true
|
|
/publish-java-cdk force=true
|
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
76dcbe6
to
f654a2b
Compare
stephane-airbyte
force-pushed
the
stephane/10-09-simple_split_of_destinationacceptancetest
branch
from
eaee8a1
to
8fa5bff
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
2 times, most recently
from
010e67d
to
25ad892
Compare
stephane-airbyte
force-pushed
the
stephane/10-09-simple_split_of_destinationacceptancetest
branch
from
8fa5bff
to
ba25b14
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
25ad892
to
8921fd6
Compare
stephane-airbyte
changed the base branch from
stephane/10-09-simple_split_of_destinationacceptancetest
to
graphite-base/46302
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
8921fd6
to
cc6fccc
Compare
stephane-airbyte
force-pushed
the
graphite-base/46302
branch
from
ba25b14
to
45446b2
Compare
stephane-airbyte
force-pushed
the
graphite-base/46302
branch
from
45446b2
to
48f7929
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
cc6fccc
to
5abe9d2
Compare
stephane-airbyte
force-pushed
the
graphite-base/46302
branch
from
48f7929
to
cb1bfbe
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
5abe9d2
to
c66040f
Compare
stephane-airbyte
force-pushed
the
stephane/10-01-destination-s3_add_file_transfer
branch
from
c66040f
to
367e2f0
Compare
| } | ||
| val flushFunction = | ||
| if (featureFlags.useFileTransfer()) { | ||
| FileTransferDestinationFlushFunction( |
There was a problem hiding this comment.
What determines whether the feature flag is set? The fact that the source is flagged as a file source? Explicit opt-in at the sync level?
There was a problem hiding this comment.
Fair question.
Basically, we need the source configuration to have a specific parameter enabled (I don't know the details of the parameter) AND the destination needs to have supportsFileTransfer set to true in its metadata.yaml. If those 2 conditions are true, then the 2 variables are set accordingly, a common volume is mounted on both containers, and it's expected that all records are file-based instead of record-based.
If the source config has the parameter set to true and the destination doesn't support file transfer, the platform will throw an exception
There was a problem hiding this comment.
Looks good. The shim seems like it's in the best place, and the file flush function is straightforward. I didn't have enough time to go over the tests in detail, but high-level how we're adding the file option to the docker env is clear.
One question about the env variables just to help me plan for the new CDK, but that's my own curiosity.
adding file transfer to destinaiton-s3
file transfer and record-based sync are exclusive. The platform will set the environment variables
USE_FILE_TRANSFERtotrueandAIRBYTE_STAGING_DIRECTORYto the mounting point of the staging directory when the destination supports file transfer and the source enabled it in its config.destination-s3 will check the
USE_FILE_TRANSFERto decide whether to enable file transfer or record-based sync.Record-based integration tests are all passing, and there's an extra test that makes sure file-based transfer is disabled.