Source Amazon S3: Refactored docs #12534

lazebnyi · 2022-05-03T13:04:34Z

What

Refactored docs

How

Updated doc to standard doc teamplate

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

lazebnyi · 2022-05-03T13:05:49Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2263965560
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2263965560
🐛

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2263965560
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2263965560
🐛 https://gradle.com/s/4jt52hviavxb6
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_match_expected[inputs0] - AssertionError:...
=================== 1 failed, 43 passed in 91.82s (0:01:31) ====================

lazebnyi · 2022-05-04T13:55:12Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2270193926
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2270193926
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  28      1    96%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/s3file.py                                                  37      2    95%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           184     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   29     10    66%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              59     35    41%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               682    149    78%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  28      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      3    95%
source_s3/source_files_abstract/storagefile.py                       23      5    78%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/utils.py                                                   29      8    72%
source_s3/source_files_abstract/file_info.py                         26     10    62%
source_s3/source_files_abstract/source.py                            37     15    59%
source_s3/s3file.py                                                  37     18    51%
source_s3/source_files_abstract/stream.py                           184     91    51%
source_s3/source_files_abstract/spec.py                              59     35    41%
source_s3/s3_utils.py                                                19     13    32%
source_s3/stream.py                                                  43     30    30%
-------------------------------------------------------------------------------------
TOTAL                                                               682    246    64%

CLAassistant · 2022-05-05T13:08:08Z

All committers have signed the CLA.

lazebnyi · 2022-05-09T11:31:17Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2293865944
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2293865944
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  28      1    96%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/s3file.py                                                  37      2    95%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           184     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   29     10    66%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              42     22    48%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               665    136    80%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  28      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      3    95%
source_s3/source_files_abstract/storagefile.py                       23      5    78%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/utils.py                                                   29      8    72%
source_s3/source_files_abstract/file_info.py                         26     10    62%
source_s3/source_files_abstract/source.py                            37     15    59%
source_s3/s3file.py                                                  37     18    51%
source_s3/source_files_abstract/stream.py                           184     91    51%
source_s3/source_files_abstract/spec.py                              42     22    48%
source_s3/s3_utils.py                                                19     13    32%
source_s3/stream.py                                                  43     30    30%
-------------------------------------------------------------------------------------
TOTAL                                                               665    233    65%

* Refactored spec and docs * Updated spec.json * Rollback spec fromating * Rollback spec fromating * Rollback spec fromating

Refactored spec and docs

e2d631f

lazebnyi requested review from grubberr, midavadim and annalvova05 May 3, 2022 13:04

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 3, 2022

Updated spec.json

27c8d23

midavadim approved these changes May 6, 2022

View reviewed changes

lazebnyi changed the title ~~Source Amazon S3: Refactored spec and docs~~ Source Amazon S3: Refactored docs May 9, 2022

lazebnyi added 3 commits May 9, 2022 14:27

Rollback spec fromating

066731e

Rollback spec fromating

d0d030d

Rollback spec fromating

d8e6f66

lazebnyi removed request for grubberr and annalvova05 May 9, 2022 11:31

lazebnyi marked this pull request as ready for review May 9, 2022 11:56

lazebnyi merged commit 27e6ce2 into master May 9, 2022

lazebnyi deleted the lazebnyi/refactor-amazon-s3-spec-and-doc-to-checklist branch May 9, 2022 11:56

octavia-squidington-iii mentioned this pull request May 9, 2022

Bump Airbyte version from 0.37.0-alpha to 0.37.1-alpha #12703

Merged

suhomud pushed a commit that referenced this pull request May 23, 2022

Source Amazon S3: Refactored docs (#12534)

c7a3ab5

* Refactored spec and docs * Updated spec.json * Rollback spec fromating * Rollback spec fromating * Rollback spec fromating

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Amazon S3: Refactored docs #12534

Source Amazon S3: Refactored docs #12534

lazebnyi commented May 3, 2022 •

edited

Loading

lazebnyi commented May 3, 2022 •

edited by github-actions bot

Loading

lazebnyi commented May 4, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

lazebnyi commented May 9, 2022 •

edited by github-actions bot

Loading

Source Amazon S3: Refactored docs #12534

Source Amazon S3: Refactored docs #12534

Conversation

lazebnyi commented May 3, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

lazebnyi commented May 3, 2022 • edited by github-actions bot Loading

lazebnyi commented May 4, 2022 • edited by github-actions bot Loading

CLAassistant commented May 5, 2022 • edited Loading

lazebnyi commented May 9, 2022 • edited by github-actions bot Loading

lazebnyi commented May 3, 2022 •

edited

Loading

lazebnyi commented May 3, 2022 •

edited by github-actions bot

Loading

lazebnyi commented May 4, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

lazebnyi commented May 9, 2022 •

edited by github-actions bot

Loading