Small spec fixes to make sure they work with connector form UI #21587

flash1293 · 2023-01-19T11:46:20Z

#21451 is making the SATs more strict w.r.t. UI limitations - some schemas are valid JSON schema, but will cause small issues in the UI. This PR is fixing the connectors causing issues:

s3

Enum was encoded within an allOf (see pydantic/pydantic#2088) - added a rewrite logic to fix the schema so it works correctly in the UI.

As this changed the previous schema I had to disable the compatibility check

sendgrid

Date had either integer or string set as value - the UI will always produce strings anyway, reflected this in the schema

flash1293 · 2023-01-19T16:18:17Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3960337377
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3960337377
🐛 https://gradle.com/s/qw4q7g6gupumu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_config_match_spec[inputs0] - Failed: Conf...
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs1]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
=================== 4 failed, 15 passed, 15 errors in 18.18s ===================

flash1293 · 2023-01-19T16:19:00Z

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3960342266
❌ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3960342266
🐛 https://gradle.com/s/txvty7wqpmbzc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============= 2 failed, 28 passed, 1 skipped in 354.72s (0:05:54) ==============

flash1293 · 2023-01-19T16:19:08Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3960342704
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3960342704
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                20      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/stream.py                           242     17    93%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      4    90%
source_s3/source.py                                                  27      4    85%
source_s3/source_files_abstract/formats/csv_parser.py               102     30    71%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/exceptions.py                                              10      4    60%
source_s3/source_files_abstract/source.py                            39     16    59%
source_s3/source_files_abstract/spec.py                              55     31    44%
source_s3/source_files_abstract/formats/jsonl_parser.py              53     32    40%
source_s3/source_files_abstract/formats/avro_parser.py               39     25    36%
source_s3/source_files_abstract/formats/parquet_parser.py            64     44    31%
-------------------------------------------------------------------------------------
TOTAL                                                               897    229    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              55      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  27      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/exceptions.py                                              10      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            64      2    97%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      2    95%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/avro_parser.py               39      3    92%
source_s3/source_files_abstract/formats/jsonl_parser.py              53      5    91%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/source.py                            39      5    87%
source_s3/source_files_abstract/stream.py                           242     40    83%
source_s3/source_files_abstract/formats/csv_parser.py               102     20    80%
source_s3/s3_utils.py                                                20      4    80%
source_s3/utils.py                                                   31     10    68%
-------------------------------------------------------------------------------------
TOTAL                                                               897     97    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.1.28.
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
=========== 104 passed, 7 skipped, 31 warnings in 245.16s (0:04:05) ============

flash1293 · 2023-01-19T16:19:40Z

/test connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/3960345719
❌ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/3960345719
🐛 https://gradle.com/s/vlay6sfz5sifu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestConnection::test_check[inputs1] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
============= 3 failed, 22 passed, 31 warnings, 8 errors in 49.37s =============

flash1293 · 2023-01-19T16:20:09Z

/test connector=connectors/source-zoho-crm

🕑 connectors/source-zoho-crm https://github.com/airbytehq/airbyte/actions/runs/3960350471
❌ connectors/source-zoho-crm https://github.com/airbytehq/airbyte/actions/runs/3960350471
🐛 https://gradle.com/s/dorzgs35gkrpi

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED integration_tests/test_stream_factory.py::test_stream_factory - Except...
	 �[31m============================== �[31m�[1m1 failed�[0m�[31m in 0.71s�[0m�[31m ===============================�[0m

flash1293 · 2023-01-19T18:15:02Z

/test connector=connectors/source-amazon-ads

🕑 connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3961220500
❌ connectors/source-amazon-ads https://github.com/airbytehq/airbyte/actions/runs/3961220500
🐛 https://gradle.com/s/z7nf5oh7drcum

Build Failed

Test summary info:

Could not find result summary

flash1293 · 2023-01-19T18:15:15Z

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3961223881
✅ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3961223881
Python tests coverage:

Name                               Stmts   Miss  Cover
------------------------------------------------------
source_linkedin_ads/analytics.py      45      0   100%
source_linkedin_ads/__init__.py        2      0   100%
source_linkedin_ads/utils.py          91      5    95%
source_linkedin_ads/source.py        195     27    86%
------------------------------------------------------
TOTAL                                333     32    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.1.12.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
================== 29 passed, 2 skipped in 432.62s (0:07:12) ===================

flash1293 · 2023-01-19T18:15:24Z

/test connector=connectors/source-sendgrid

girarda · 2023-01-19T19:16:21Z

airbyte-integrations/connectors/source-amazon-ads/integration_tests/spec.json

@@ -51,7 +51,7 @@
        "order": 6,
        "type": "array",
        "items": {
-          "type": "integer"
+          "type": "string"


I did a quick check in the prod db and these items appear to be integers so we'll need to migrate existing cloud configurations

Thanks for flagging this - I wondered how these configs got created and realized that I missed the “array of integers” support in the UI which means I can revert this change. I will update this PR and the SAT extension.

…c-fixes

flash1293 · 2023-01-20T10:00:06Z

@girarda For the sendgrid one I checked and right now users have to use the integer format (specifying a unix timestamp). So we should run a migration for these. Alternatively we could just take away the string format (because so far users can't use that), but it seems like only allowing strings for a start time is a much better choice than only allowing unix timestamps like we effectively do right now.

…c-fixes

flash1293 · 2023-01-23T17:55:50Z

/test connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/3989140476
✅ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/3989140476
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_sendgrid/__init__.py       2      0   100%
source_sendgrid/source.py        26      3    88%
source_sendgrid/streams.py      171     21    88%
-------------------------------------------------
TOTAL                           199     24    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.16.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============ 31 passed, 2 skipped, 31 warnings in 83.83s (0:01:23) =============

girarda

approved assuming the change is S3 is not breaking

girarda · 2023-01-23T23:12:41Z

airbyte-integrations/connectors/source-s3/Dockerfile

@@ -17,5 +17,5 @@ COPY source_s3 ./source_s3
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

-LABEL io.airbyte.version=0.1.28
+LABEL io.airbyte.version=0.1.29


should this be a minor update / is this a breaking change?

no, that's actually a patch change - it doesn't change what's allowed for this field, it just makes sure the UI understands correctly.

…c-fixes

flash1293 · 2023-01-25T15:22:30Z

/test connector=connectors/source-sendgrid

🕑 connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/4007363227
✅ connectors/source-sendgrid https://github.com/airbytehq/airbyte/actions/runs/4007363227
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_sendgrid/__init__.py       2      0   100%
source_sendgrid/source.py        26      3    88%
source_sendgrid/streams.py      171     21    88%
-------------------------------------------------
TOTAL                           199     24    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.2.16.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============ 31 passed, 2 skipped, 31 warnings in 89.12s (0:01:29) =============

flash1293 · 2023-01-25T15:22:34Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4007363542
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/4007363542
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3_utils.py                                                20      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/stream.py                           248     20    92%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      4    90%
source_s3/source.py                                                  27      4    85%
source_s3/source_files_abstract/formats/csv_parser.py               102     30    71%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/exceptions.py                                              10      4    60%
source_s3/source_files_abstract/source.py                            39     16    59%
source_s3/source_files_abstract/spec.py                              55     31    44%
source_s3/source_files_abstract/formats/jsonl_parser.py              53     32    40%
source_s3/source_files_abstract/formats/avro_parser.py               39     25    36%
source_s3/source_files_abstract/formats/parquet_parser.py            64     44    31%
-------------------------------------------------------------------------------------
TOTAL                                                               907    235    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              55      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  27      0   100%
source_s3/exceptions.py                                              10      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            64      2    97%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      2    95%
source_s3/stream.py                                                  43      3    93%
source_s3/s3file.py                                                  41      3    93%
source_s3/source_files_abstract/formats/avro_parser.py               39      3    92%
source_s3/source_files_abstract/formats/jsonl_parser.py              53      5    91%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/source.py                            39      5    87%
source_s3/source_files_abstract/stream.py                           248     45    82%
source_s3/source_files_abstract/formats/csv_parser.py               102     20    80%
source_s3/s3_utils.py                                                20      4    80%
source_s3/utils.py                                                   31     10    68%
-------------------------------------------------------------------------------------
TOTAL                                                               907    105    88%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:103: Backward compatibility tests are disabled for version 0.1.29.
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
================== 102 passed, 7 skipped in 241.68s (0:04:01) ==================

flash1293 · 2023-01-25T15:42:24Z

/publish connector=connectors/source-sendgrid

🕑 Publishing the following connectors:
connectors/source-sendgrid
https://github.com/airbytehq/airbyte/actions/runs/4007536439

Connector	Did it publish?	Were definitions generated?
connectors/source-sendgrid	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

flash1293 · 2023-01-25T16:00:15Z

/publish connector=connectors/source-s3

🕑 Publishing the following connectors:
connectors/source-s3
https://github.com/airbytehq/airbyte/actions/runs/4007675692

Connector	Did it publish?	Were definitions generated?
connectors/source-s3	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-01-25T16:17:20Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	23.98%	❌

Joe Reuter added 6 commits January 19, 2023 11:29

switch array type to reflect actual config

9af17a6

switch array type to reflect actual config

984e664

rewrite allOf in spec

9ba3fb8

remove array type

58f2853

remove array type

1a5d40c

convert to string

7f60f35

octavia-squidington-iv added area/connectors Connector related issues connectors/source/amazon-ads connectors/source/linkedin-ads connectors/source/s3 connectors/source/sendgrid connectors/source/zoho-crm labels Jan 19, 2023

flash1293 mentioned this pull request Jan 19, 2023

Extend SATs to capture UI limitations #21451

Merged

adjust changelogs

ac4e35b

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 19, 2023

flash1293 marked this pull request as ready for review January 19, 2023 12:06

flash1293 added team/extensibility team/connectors-python labels Jan 19, 2023

disable backwards compatible check for other connectors

894b458

girarda reviewed Jan 19, 2023

View reviewed changes

Joe Reuter added 2 commits January 20, 2023 10:34

Merge remote-tracking branch 'origin/master' into flash1293/small-spe…

232132b

…c-fixes

revert some changes

1ee8072

flash1293 removed connectors/source/amazon-ads connectors/source/linkedin-ads labels Jan 20, 2023

Joe Reuter added 4 commits January 23, 2023 17:44

Merge remote-tracking branch 'origin/master' into flash1293/small-spe…

73d2e0c

…c-fixes

revert zoho-crm changes as build is broken

36bd5c9

make sendgrid take strings for starttime

6159404

adjust version

5a9029c

girarda approved these changes Jan 23, 2023

View reviewed changes

Joe Reuter added 3 commits January 24, 2023 10:51

Merge remote-tracking branch 'origin/master' into flash1293/small-spe…

1661357

…c-fixes

Merge branch 'master' into flash1293/small-spec-fixes

7a650ac

bump version and adjust date

a903524

flash1293 removed the connectors/source/zoho-crm label Jan 25, 2023

auto-bump connector version

4471f40

octavia-squidington-iii temporarily deployed to more-secrets January 25, 2023 16:08 — with GitHub Actions Inactive

update seed

cdbc7e2

flash1293 temporarily deployed to more-secrets January 25, 2023 17:12 — with GitHub Actions Inactive

flash1293 merged commit 6e37343 into master Jan 25, 2023

flash1293 deleted the flash1293/small-spec-fixes branch January 25, 2023 18:43

github-actions bot mentioned this pull request Jan 25, 2023

Bump helm chart version reference to 0.43.24 #21887

Closed

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21954

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Small spec fixes to make sure they work with connector form UI #21587

Small spec fixes to make sure they work with connector form UI #21587

flash1293 commented Jan 19, 2023 •

edited

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023

girarda Jan 19, 2023

flash1293 Jan 19, 2023

flash1293 commented Jan 20, 2023 •

edited

Loading

flash1293 commented Jan 23, 2023 •

edited by github-actions bot

Loading

girarda left a comment

girarda Jan 23, 2023

flash1293 Jan 24, 2023

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 25, 2023

Small spec fixes to make sure they work with connector form UI #21587

Small spec fixes to make sure they work with connector form UI #21587

Conversation

flash1293 commented Jan 19, 2023 • edited Loading

s3

sendgrid

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Passed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

flash1293 commented Jan 19, 2023 • edited by github-actions bot Loading

Build Passed

flash1293 commented Jan 19, 2023

girarda Jan 19, 2023

Choose a reason for hiding this comment

flash1293 Jan 19, 2023

Choose a reason for hiding this comment

flash1293 commented Jan 20, 2023 • edited Loading

flash1293 commented Jan 23, 2023 • edited by github-actions bot Loading

Build Passed

girarda left a comment

Choose a reason for hiding this comment

girarda Jan 23, 2023

Choose a reason for hiding this comment

flash1293 Jan 24, 2023

Choose a reason for hiding this comment

flash1293 commented Jan 25, 2023 • edited by github-actions bot Loading

Build Passed

flash1293 commented Jan 25, 2023 • edited by github-actions bot Loading

Build Passed

flash1293 commented Jan 25, 2023 • edited by github-actions bot Loading

flash1293 commented Jan 25, 2023 • edited by github-actions bot Loading

github-actions bot commented Jan 25, 2023

Airbyte Code Coverage

flash1293 commented Jan 19, 2023 •

edited

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 19, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 20, 2023 •

edited

Loading

flash1293 commented Jan 23, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading

flash1293 commented Jan 25, 2023 •

edited by github-actions bot

Loading