✨Source S3 (v4): Set decimal_as_float to True for parquet files #29342

girarda · 2023-08-10T18:51:17Z

What

Set decimal_as_float on ParquetFormat for backwards compatibility
Closes S3 source: Legacy parquet configs should set decimal_as_float to True #29227

github-actions · 2023-08-10T18:51:44Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-08-11T14:56:16Z

source-s3 test report (commit `f9fc565b24`) - ❌

⏲️ Total pipeline duration: 22mn21s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	❌
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

octavia-squidington-iii · 2023-08-11T16:18:07Z

source-s3 test report (commit `8c2572823f`) - ❌

⏲️ Total pipeline duration: 11mn05s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

octavia-squidington-iii · 2023-08-11T16:27:53Z

source-s3 test report (commit `8c2572823f`) - ❌

⏲️ Total pipeline duration: 03mn00s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

girarda · 2023-08-11T16:32:46Z

Noting that CATs are failing because the PR isn't based off an updated master (see https://airbytehq-team.slack.com/archives/C02UF50V9HA/p1691765570710209)

clnoll

⛴!

octavia-squidington-iii · 2023-08-14T19:17:19Z

source-s3 test report (commit `5b0c44fa98`) - ❌

⏲️ Total pipeline duration: 11mn42s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

octavia-squidington-iii · 2023-08-14T23:18:24Z

source-s3 test report (commit `408f9605bc`) - ❌

⏲️ Total pipeline duration: 13mn38s

Step	Result
Validate airbyte-integrations/connectors/source-s3/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-s3 docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-s3 test

girarda · 2023-08-15T01:13:14Z

/approve-and-merge reason="not updating connector version because this is a non-visible change"

octavia-approvington · 2023-08-15T01:13:49Z

Our work here is done

…ytehq#29342) * [ISSUE airbytehq#28893] infer csv schema * [ISSUE airbytehq#28893] align with pyarrow * Automated Commit - Formatting Changes * [ISSUE airbytehq#28893] legacy inference and infer only when needed * [ISSUE airbytehq#28893] fix scenario tests * [ISSUE airbytehq#28893] using discovered schema as part of read * [ISSUE airbytehq#28893] self-review + cleanup * [ISSUE airbytehq#28893] fix test * [ISSUE airbytehq#28893] code review part #1 * [ISSUE airbytehq#28893] code review part #2 * Fix test * formatcdk * [ISSUE airbytehq#28893] code review * FIX test log level * Re-adding failing tests * [ISSUE airbytehq#28893] improve inferrence to consider multiple types per value * set decimal_as_float to True * update * Automated Commit - Formatting Changes * add file adapters for avro, csv, jsonl, and parquet * fix try catch * update * format * pr feedback with a few additional default options set --------- Co-authored-by: maxi297 <[email protected]> Co-authored-by: maxi297 <[email protected]> Co-authored-by: brianjlai <[email protected]>

maxi297 and others added 18 commits August 4, 2023 13:11

[ISSUE #28893] infer csv schema

eaba483

[ISSUE #28893] align with pyarrow

60757c1

Automated Commit - Formatting Changes

f49235d

[ISSUE #28893] legacy inference and infer only when needed

a394666

[ISSUE #28893] fix scenario tests

4f9d162

[ISSUE #28893] using discovered schema as part of read

0617c82

[ISSUE #28893] self-review + cleanup

d157aa3

[ISSUE #28893] fix test

57b011f

[ISSUE #28893] code review part #1

71cdca9

Merge branch 'master' into issue-28893/infer-schema-csv

ef8f5f5

[ISSUE #28893] code review part #2

f651d03

Fix test

a573a89

formatcdk

0ce37e5

[ISSUE #28893] code review

82db6c3

FIX test log level

3027a4f

Re-adding failing tests

ac91730

[ISSUE #28893] improve inferrence to consider multiple types per value

f1a60ba

set decimal_as_float to True

53c9248

octavia-squidington-iii added area/connectors Connector related issues connectors/source/s3 labels Aug 10, 2023

girarda and others added 7 commits August 10, 2023 11:55

update

c71bb95

Merge branch 'master' into issue-28893/infer-schema-csv

7113603

Automated Commit - Formatting Changes

c9e2004

add file adapters for avro, csv, jsonl, and parquet

e6a0b4d

fix try catch

2f827c0

update

cbbfe76

merge

f9fc565

octavia-squidington-iii added the CDK Connector Development Kit label Aug 11, 2023

girarda changed the base branch from master to brian/s3_csv_format_adapter August 11, 2023 14:28

girarda changed the title ~~set decimal_as_float to True~~ ✨Source S3: Set decimal_as_float to True for parquet files Aug 11, 2023

girarda changed the title ~~✨Source S3: Set decimal_as_float to True for parquet files~~ ✨Source S3 (v4): Set decimal_as_float to True for parquet files Aug 11, 2023

girarda requested review from brianjlai, maxi297 and clnoll August 11, 2023 14:31

girarda marked this pull request as ready for review August 11, 2023 14:32

format

8c25728

octavia-squidington-iii removed the CDK Connector Development Kit label Aug 11, 2023

clnoll approved these changes Aug 11, 2023

View reviewed changes

brianjlai and others added 2 commits August 11, 2023 15:07

pr feedback with a few additional default options set

868a597

merge

5b0c44f

Base automatically changed from brian/s3_csv_format_adapter to master August 14, 2023 22:47

brianjlai requested a review from a team as a code owner August 14, 2023 22:47

merge

408f960

girarda added checklist-action-run and removed checklist-action-run labels Aug 14, 2023

octavia-approvington approved these changes Aug 15, 2023

View reviewed changes

octavia-approvington merged commit 690479d into master Aug 15, 2023
17 of 18 checks passed

octavia-approvington deleted the alex/s3_parquet_adapter branch August 15, 2023 01:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨Source S3 (v4): Set decimal_as_float to True for parquet files #29342

✨Source S3 (v4): Set decimal_as_float to True for parquet files #29342

girarda commented Aug 10, 2023 •

edited

Loading

github-actions bot commented Aug 10, 2023 •

edited by girarda

Loading

octavia-squidington-iii commented Aug 11, 2023

octavia-squidington-iii commented Aug 11, 2023

octavia-squidington-iii commented Aug 11, 2023

girarda commented Aug 11, 2023

clnoll left a comment

octavia-squidington-iii commented Aug 14, 2023

octavia-squidington-iii commented Aug 14, 2023

girarda commented Aug 15, 2023

octavia-approvington commented Aug 15, 2023

✨Source S3 (v4): Set decimal_as_float to True for parquet files #29342

✨Source S3 (v4): Set decimal_as_float to True for parquet files #29342

Conversation

girarda commented Aug 10, 2023 • edited Loading

What

github-actions bot commented Aug 10, 2023 • edited by girarda Loading

Before Merging a Connector Pull Request

octavia-squidington-iii commented Aug 11, 2023

source-s3 test report (commit f9fc565b24) - ❌

octavia-squidington-iii commented Aug 11, 2023

source-s3 test report (commit 8c2572823f) - ❌

octavia-squidington-iii commented Aug 11, 2023

source-s3 test report (commit 8c2572823f) - ❌

girarda commented Aug 11, 2023

clnoll left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 14, 2023

source-s3 test report (commit 5b0c44fa98) - ❌

octavia-squidington-iii commented Aug 14, 2023

source-s3 test report (commit 408f9605bc) - ❌

girarda commented Aug 15, 2023

octavia-approvington commented Aug 15, 2023

girarda commented Aug 10, 2023 •

edited

Loading

github-actions bot commented Aug 10, 2023 •

edited by girarda

Loading

source-s3 test report (commit `f9fc565b24`) - ❌

source-s3 test report (commit `8c2572823f`) - ❌

source-s3 test report (commit `8c2572823f`) - ❌

source-s3 test report (commit `5b0c44fa98`) - ❌

source-s3 test report (commit `408f9605bc`) - ❌