🎉 New source: SFTP Bulk [python cdk] #17691

henriblancke · 2022-10-06T19:22:29Z

What

Source that let's you bulk sync files from an SFTP server and write them to your destination as a single stream.

I named this source source-ftp since source-sftp already exists

Main advantages of this FTP source over source-sftp:

Fetch files from an FTP server matching a folder path and define an optional file pattern to bulk ingest files into a single stream
- source-sftp sees every file matching a folder path and optional file pattern as separate streams
Incrementally load files into your destination from an FTP server based on when files were last added or modified
Optionally load only the latest file matching a folder path and optional pattern and overwrite the data in your destination (helpful when a snapshot file gets added to your FTP server on a regular basis containing the latest data)
You can still achieve loading single files into single streams by creating multiple sources where the file pattern matches a single file.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Integration

The integration tests spin up a temporary sftp server, secrets/config.json can be a copy of integration_tests/valid_config.json

Acceptance

The acceptance tests spin up a temporary sftp server, secrets/config.json can be a copy of integration_tests/valid_config.json

Signed-off-by: Henri Blancke <[email protected]>

marcosmarxm · 2022-10-11T16:49:38Z

@henriblancke any reasons why not improve the File connector?

henriblancke · 2022-10-11T16:54:31Z

@marcosmarxm ooh wow, I had no idea the file connector also supported SFTP, let me give that a try and see if that satisfies my use case! Thanks for pointing that out!

henriblancke · 2022-10-11T17:44:54Z

@marcosmarxm it looks like the File connector uses smart_open. smart_open doesn't have an easy way to iterate over an sftp folder/path and seems to be designed (mostly) to open single files only. There is also no easy way to access the ssh/scp/sftp connection created by smart_open and reuse that to add logic to support iterating over files and matching file patterns before retrieving them from the server with smart_opens built in functionality.

Since the File connector heavily relies on smart_open for almost all it's functionality changing that for ssh/scp/sftp may be too much of a change to its design? Let me know what you think? I'm open to exploring whatever you think the best solution is here. Thanks again for all your help!

henriblancke · 2022-10-13T13:19:23Z

@marcosmarxm any advice on the best way forward here?

marcosmarxm · 2022-10-17T12:42:39Z

@henriblancke sorry the delay here I'll take a look today on this!

marcosmarxm · 2022-10-18T23:34:07Z

We can go foward with this connector, maybe rename to Bulk FTP to make sure why it is different frmo others.

Signed-off-by: Henri Blancke <[email protected]>

henriblancke · 2022-10-19T16:42:08Z

@marcosmarxm awesome! Everything should be renamed

marcosmarxm · 2022-10-19T23:50:52Z

I added this to my to-do list for tomorrow sorry the delay here.

Signed-off-by: Henri Blancke <[email protected]>

marcosmarxm · 2022-10-20T19:33:14Z

There are some tests are not passing. Please run ./gradlew airbyte-integrations:connectors:source-sftp-bulk:integrationTest and correct them.

…rce-ftp Signed-off-by: Henri Blancke <[email protected]>

Signed-off-by: Henri Blancke <[email protected]>

henriblancke · 2022-10-21T14:22:59Z

@marcosmarxm should be fixed

marcosmarxm · 2022-10-21T16:49:17Z

/test connector=connectors/source-sftp-bulk

🕑 connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3299089115
❌ connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3299089115
🐛 https://gradle.com/s/7zrx7urtq2gem

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_core.py::TestSpec::test_config_match_spec[inputs0] - docker.errors...
ERROR test_core.py::TestSpec::test_match_expected[inputs0] - docker.errors.AP...
ERROR test_core.py::TestSpec::test_docker_env[inputs0] - docker.errors.APIErr...
ERROR test_core.py::TestSpec::test_oneof_usage[inputs0] - docker.errors.APIEr...
ERROR test_core.py::TestSpec::test_required[inputs0] - docker.errors.APIError...
ERROR test_core.py::TestSpec::test_optional[inputs0] - docker.errors.APIError...
ERROR test_core.py::TestSpec::test_has_secret[inputs0] - docker.errors.APIErr...
ERROR test_core.py::TestSpec::test_secret_never_in_the_output[inputs0] - dock...
ERROR test_core.py::TestSpec::test_defined_refs_exist_in_json_spec_file[inputs0]
ERROR test_core.py::TestSpec::test_oauth_flow_parameters[inputs0] - docker.er...
ERROR test_core.py::TestSpec::test_backward_compatibility[inputs0] - docker.e...
ERROR test_core.py::TestSpec::test_additional_properties_is_true[inputs0] - d...
ERROR test_core.py::TestConnection::test_check[inputs0] - docker.errors.APIEr...
ERROR test_core.py::TestConnection::test_check[inputs1] - docker.errors.APIEr...
ERROR test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.API...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.APIErro...
ERROR test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs0]
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
ERROR test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
============================== 29 errors in 2.33s ==============================

marcosmarxm · 2022-10-21T17:03:38Z

@marcosmarxm should be fixed

@henriblancke I ran locally and here in CI and both have the same errors. Looks the container is getting error 500. Can you show the output of your ./gradlew integrationTest ran locally?

henriblancke · 2022-10-21T18:01:20Z

@marcosmarxm hmm weird, seems like docker doesn't like us using port 22 (judging from the CI logs): listen tcp4 0.0.0.0:22: bind: address already in use. It should only spin up 1 container mapped to port 22 though 🤔?

The test seem to run ok locally for me. Here's the output of my .gradlew integrationTest:

Let me try to change the port to another random port and see if that resolves the issue?

Signed-off-by: Henri Blancke <[email protected]>

marcosmarxm · 2022-10-24T23:17:51Z

Sorry the delay here @henriblancke I'll review this tomorrow morning.

marcosmarxm · 2022-10-25T18:15:07Z

/test connector=connectors/source-sftp-bulk

🕑 connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3323269934
✅ connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3323269934
Python tests coverage:

Name                                                                                                                                         Stmts   Miss  Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------
source_sftp_bulk/__init__.py                                                                                                                     2      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/__init__.py                 6      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/__init__.py                       2      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/config.py                        83      6    93%
source_sftp_bulk/client.py                                                                                                                     131     24    82%
source_sftp_bulk/source.py                                                                                                                      72     15    79%
source_sftp_bulk/streams.py                                                                                                                     43      9    79%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/base.py                          10      4    60%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/plugin.py                        48     21    56%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/compare.py                 62     38    39%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/asserts.py                 37     24    35%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/common.py                  77     52    32%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/connector_runner.py       112     84    25%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py     105     87    17%
----------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                                          790    364    54%
Name                           Stmts   Miss  Cover
--------------------------------------------------
source_sftp_bulk/__init__.py       2      0   100%
source_sftp_bulk/source.py        72     39    46%
source_sftp_bulk/streams.py       43     26    40%
source_sftp_bulk/client.py       131     96    27%
--------------------------------------------------
TOTAL                            248    161    35%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
======================== 27 passed, 2 skipped in 34.74s ========================

marcosmarxm · 2022-10-27T11:55:34Z

airbyte-integrations/connectors/source-sftp-bulk/integration_tests/ssh/id_rsa

+-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn


@henriblancke last probably is remove this key, even this been used only for integration tests I don't think it is a good idea to add the keys. I'm looking a way to create programatically.

@marcosmarxm that makes sense! I pushed up a change to generate the keys programmatically. Thanks for the suggestion

Signed-off-by: Henri Blancke <[email protected]>

marcosmarxm · 2022-10-28T02:22:45Z

/test connector=connectors/source-sftp-bulk

🕑 connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3342323702
✅ connectors/source-sftp-bulk https://github.com/airbytehq/airbyte/actions/runs/3342323702
Python tests coverage:

Name                                                                                                                                         Stmts   Miss  Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------
source_sftp_bulk/__init__.py                                                                                                                     2      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/__init__.py                 6      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/__init__.py                       2      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/config.py                        83      6    93%
source_sftp_bulk/client.py                                                                                                                     131     24    82%
source_sftp_bulk/source.py                                                                                                                      72     15    79%
source_sftp_bulk/streams.py                                                                                                                     43      9    79%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/base.py                          10      4    60%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/plugin.py                        48     21    56%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/compare.py                 62     38    39%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/asserts.py                 37     24    35%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/common.py                  77     52    32%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/connector_runner.py       112     84    25%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py     105     87    17%
----------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                                          790    364    54%
Name                           Stmts   Miss  Cover
--------------------------------------------------
source_sftp_bulk/__init__.py       2      0   100%
source_sftp_bulk/source.py        72     39    46%
source_sftp_bulk/streams.py       43     26    40%
source_sftp_bulk/client.py       131     96    27%
--------------------------------------------------
TOTAL                            248    161    35%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
======================== 27 passed, 2 skipped in 34.55s ========================

marcosmarxm · 2022-10-28T02:38:55Z

/publish connector=connectors/source-sftp-bulk

🕑 Publishing the following connectors:
connectors/source-sftp-bulk
https://github.com/airbytehq/airbyte/actions/runs/3342380143

Connector	Did it publish?	Were definitions generated?
connectors/source-sftp-bulk	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* [INIT] setup source-ftp Signed-off-by: Henri Blancke <[email protected]> * [ADD] add logic Signed-off-by: Henri Blancke <[email protected]> * [ADD] unit, integration, acceptance tests Signed-off-by: Henri Blancke <[email protected]> * [UPD] update docs Signed-off-by: Henri Blancke <[email protected]> * [UPD] clean up crew Signed-off-by: Henri Blancke <[email protected]> * [UPD] update host to localhost Signed-off-by: Henri Blancke <[email protected]> * [UPD] rename from ftp to sftp bulk Signed-off-by: Henri Blancke <[email protected]> * [INIT] setup source-ftp Signed-off-by: Henri Blancke <[email protected]> * [ADD] add logic Signed-off-by: Henri Blancke <[email protected]> * [ADD] unit, integration, acceptance tests Signed-off-by: Henri Blancke <[email protected]> * [UPD] update docs Signed-off-by: Henri Blancke <[email protected]> * [UPD] clean up crew Signed-off-by: Henri Blancke <[email protected]> * [UPD] update host to localhost Signed-off-by: Henri Blancke <[email protected]> * [UPD] rename from ftp to sftp bulk Signed-off-by: Henri Blancke <[email protected]> * update branch * [FIX] integration tests Signed-off-by: Henri Blancke <[email protected]> * [FIX] acceptance test fixture Signed-off-by: Henri Blancke <[email protected]> * [UPD] change ftp port Signed-off-by: Henri Blancke <[email protected]> * add uuid to image build * add source def * [FIX] generate ssh keys Signed-off-by: Henri Blancke <[email protected]> * auto-bump connector version Signed-off-by: Henri Blancke <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

* Fix broken link to config in .env comment (#18287) * Add message migration to discover schema (#18205) * upgrade commons-text to 1.10.0 (#18275) * test * fix * fix other modules * version bumps + changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * [low-code] Propagate options to InterpolatedRequestInputProvider (#18050) * properly propagate options * cleanup * turn into dataclass * rename * no need for deepcopy * fix test * bump * cleaner * Add message translation to GetSpec (#18130) * Update SpecActivityImpl to build a VersionedStreamFactory * Enable Protocol Version detection from Stream for SPEC * Print log before the action for better debugging * Fix buffer size for protocol detection * Improve detectVersion error handling * extract constan * Rename attribute for clarity * only rm file if it exists (#18310) * remove extra check (#10554) * remove extra check * simplify logic * Google ads: Upgrade CDK to incorporate config error handling (#18309) * Bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * :tada: Source Linkedin Ads - for adDirectSponsoredContents stream skip accounts which are not part of organization (#18111) * for adDirectSponsoredContents stream skip accounts which are part of organization * updated PR number * updated log message, added unit tests * additional formatting * updated connector version in source_definitions.yaml * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Add SchemaChange to WebBackendConnectionRead object (#17969) * add schemaChange field to WebBackendConnectionRead and breakingChange to ConnectionRead * Destinations BigQuery and GCS: update docs about bucket encryption (#18315) * Bump Airbyte version from 0.40.16 to 0.40.17 (#18316) Co-authored-by: edgao <[email protected]> * Updates `getPrivateKeyPair` to throw exception when private key pair does not exist (#18263) * Changes method from returning null to throwing Exception to match method structure * Added javadoc explaining that KeyPair should not be returning null * Moved ConnectionErrorException to commons package * Adds error handling for SshWrappedDestinations and markdown to include changes * Bumps version numbers for Postgres, MySQL, MSSQL * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: xkcd [python cdk] (#18049) * Add new source: xkcd * Add documentation for xckd connector * Fix unit test test_parse_response * Fix xkcd changelog * Improve iteration method * Update tests test_next_page_token and test_parse_response * add timeout for full read * add seed config and eof * correct source def id * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Source pinterest: Fix type of `start_date` (#18285) Signed-off-by: Sergey Chvalyuk <[email protected]> * requester = MagicMock(use_cache=False) (#18289) Signed-off-by: Sergey Chvalyuk <[email protected]> * fix Connector Base build (#18351) * SAT: new `test_strictness_level` field in config (#18218) * 🪟 🐛 Fixed static <ConditionSection /> width (#18233) * fixed ConditionSection width * fixed relative scss colors import * 🎉 Source Shopify: update API version to `2022-10` (#18298) * Update airbyte-db reference (#18368) * 🪟 🐛 Load global CSS in storybook (#18353) * Bump helm chart version reference to 0.40.33 (#18371) Co-authored-by: xpuska513 <[email protected]> Co-authored-by: Kyryl Skobylko <[email protected]> * enable support for publishing datadog metrics from a local environment (#18282) * local datadog docker-compose * add datadog docker-compose * update comments * Simplify the OSS documentation deploy system (#2670) (#18377) + Adds a placeholder for 'Deploy Docs' Github Workflow. * fix: add task to list all dependencies (#18304) * fix: add task to list all dependencies * fix: remove newline * remove useless json_size metric (#18247) * 🎉Destination-elasticsearch: added custom sertificate support (#18177) * [11356] Destination-elasticsearch: added custom certificate support * update spotbugs to latest (#18208) * Source Jira - Pagination fix on jira users stream (#16944) * pagination fix on jira users stream * Update spec.json * Update spec.json * Update streams.py * bump docker version and changelog * Update spec.json * Update streams.py * update jsons and source * update source * Update source.py * auto-bump connector version Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 Source Google Ads: Add fields to the campaign stream 🚨 (#18069) * feat: add new fields to campaign stream * docs: updated docs and bumped Dockerfile version * Update campaigns.json * Update Dockerfile * auto-bump connector version Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * fixed check-style (#18390) * Readded xkcd to source def yaml (#18386) * readded xkcd to source def yaml * remove changes in klavyio * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Trace errors in orchestrator workers (#18381) * Trace errors in orchestrator workers * Trace cancel method of workers * Formatting * Restore final modifier * Source Shopify: add unit tests (#17944) * Export python type information for airbyte-cdk (#18387) Fixes #18384 * Fix error highlight of select boxes (#18396) * Source Hubspot: update `contacts` scope in docs (#18380) Signed-off-by: Sergey Chvalyuk <[email protected]> * remove extra ' in version (#18400) - rm extra `'` in version string; `"2.1.1'"` -> `"2.1.1"` * Improve listSourcesForWorkspace route (#18207) * 🪟 🐛 Fix additional buttons for connection creation (#18372) * make button enabled regardless credit balance * remove FeatureItem.AllowCreateConnection at all, since we don't need it anymore * update tests * Replace "AllowCreateConnection" feature flag with "AllowDBTCloudIntegration" in tests * Change port for acceptance test destination db (#18391) * Bmoric/update connection list with breaking (#18125) * add schemaChange * merge conflict * frontend tests * tests * l * fix source catalog id * test * formatting * move schema change to build backend web connection * check if actor catalog id is different * fix * tests and fixes * remove extra var * remove logging * continue to pass back new catalog id * api updates * fix mockdata * tests * add schemaChange * merge conflict * frontend tests * tests * l * fix source catalog id * test * formatting * move schema change to build backend web connection * check if actor catalog id is different * fix * tests and fixes * remove extra var * remove logging * continue to pass back new catalog id * api updates * fix mockdata * tests * tests * optional of nullable * Tmp * For diff * Add test * More test * Fix test and add some * Fix merge and test * Fix PMD * Fix test * Rm dead code * Fix pmd * Address PR comments * RM unused column Co-authored-by: alovew <[email protected]> * Generate connector builder api client (#18274) * update openapi for connector builder to work with orval for now * add orval configuration for generating the connector builder client * create ConnectorBuilderApiService with methods to call the generated connector builder API client code * rename connector definition to manifest * refactor builder api service to match existing patterns * fix name of pages folder * improve comment * modify structure of StreamRead * fix path * :window: :wrench: Generify connector documentation layout into resizable panels component (#18272) * generify connector documentation layout into resizable panels component * add intl string for connector builder expand message * update name of formatted download yaml message * fix stylelint * Update airbyte-webapp/src/components/ui/ResizablePanels/ResizablePanels.module.scss Co-authored-by: Joey Marshment-Howell <[email protected]> * move splitter styles into scss module * simplify ResizablePanels component * reorder * use variables * remove units on 0 * Update airbyte-webapp/src/views/Connector/ConnectorDocumentationLayout/ConnectorDocumentationLayout.module.scss Co-authored-by: Joey Marshment-Howell <[email protected]> * Update airbyte-webapp/src/components/ui/ResizablePanels/ResizablePanels.module.scss Co-authored-by: Joey Marshment-Howell <[email protected]> * Update airbyte-webapp/src/components/ui/ResizablePanels/ResizablePanels.module.scss Co-authored-by: Joey Marshment-Howell <[email protected]> Co-authored-by: Joey Marshment-Howell <[email protected]> * SAT: declare `bypass_reason` in `acceptance-test-config.yml` (#18364) * Source Iterable - better processing of 401 and 429 errors (#18292) * #829 source iterable - better processing of 401 and 429 errors * source iterable: upd changelog * #829 source iterable: fix unit tests * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Source Asana: fix request-cache test (#18375) * Source Asana: fix request-cache test * Source Asana: Bump version * 🐛 Source Asana: revert docker label; update CDK * 🐛 Source Freshdesk: fix test; update CDK (#18389) * 🐛 Source Freshdesk: fix test; update CDK * 🐛 Source FreshDesk: revert docker label; update CDK * Update cdk-speedrun.md (#18416) * Update README.md * Update Facebook-marketing.md with warning about ongoing setup issue (#18422) There is an ongoing issue blocking anyone with the most recent API update from being able to setup a Facebook Marketing Source Connector. Put this warning at the top of the setup page until it is fixed. * Source Greenhouse: extend users stream schema (#18154) * #17506 source greenhouse: extend users stream schema * #17506 source greenhouse: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Source Hubspot: update CDK; fix expected test result according to 0.2.2 update (#18424) * Additional worker APM tracing (#18427) * Publish new CDK version with typehints exported (#18398) * Extract the AttemptApi out of the ConfigurationApi (#18406) * Tmp * Extract the Attempt API from the V1 API * Add comments * format * Rename to Controller * 🐛 Source Salesforce: update CDK; fix test supported_sync_modes is required for AirbyteStream (#18429) * :window: :wrench: Introduce useAuthentication hook (#17965) * Introduce useAuthentication hook * Work for both auth mechanism * Finalize implementation * Add documentation * Fix unit tests * Add unit tests * Remove getValues call * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.tsx Co-authored-by: Lake Mossman <[email protected]> * Update airbyte-webapp/src/views/Connector/ServiceForm/useAuthentication.test.tsx Co-authored-by: Lake Mossman <[email protected]> * Change variable name Co-authored-by: Lake Mossman <[email protected]> * Update: aws ec2 deployment guide (#18360) * Updated Deploy on GCP Doc (#18286) * Updated Deploy on GCP Doc * Added formatting changes * Updated deploying Airbyte on Azure section (#18331) * feat: deploy airbyte on kubernetes doc (#18339) * New Source: Zoom [low-code CDK] (#18179) * feat: implement meetings & users API in zoom connector * feat: add support fot all zoom GET APIs * fix: unhandled error cases by adding default handler and minor refactor * feat: sync the catalog with the latest zoom API responses * docs: add the new zoom connector built with low-code CDK * chore: remove unnecessary files, and rename connector in definitions * fix: forgot to add source_definitions.yaml earlier 😅 * remove some empty streams * add eof * correct spec yaml * docs: update docs/integrations and replace zoom with low code CDK zoom * fix: definition generation err * udpate seed file and remove zoom-singer * remove zoom singer spec * add icon Co-authored-by: marcosmarxm <[email protected]> * 🐛 Source SurveyMonkey: fix expected test result according to 0.2.2 update (#18431) * Update postgres.md (#18432) * Update README.md * SAT: make all tests mandatory when `test_strictness_level` == `high` (#18414) * Refactor SourceHandler:buildSourceRead function (#18430) This function takes as input a `sourceId` and performs a database call to retrieve the full source information. This refactoring ensures that call sites that already have the full object can bypass that database call. This is especially important for methods that fetch and return a list of sources. * Make YouTube login button Google styled (#18443) * Edited Notion Doc (#18405) * edited doc * some wording changes * edited step * SAT: enforce `bypass_reason` declaration on `empty_streams` when `test_strictness_level == high` (#18425) * Fix the tag in openAPI (#18445) * Fix the tag * remove unused * use api to do jobpersistence query (#18308) * use api to do jobpersistence query * renaming some variables * fix test * RefreshSchema Activity (#18388) * Add RefreshSchema Activity * Follow-up fix for SSHExceptions (#18383) * Follow-up fix for SSHExceptions * Catch RuntimeException instead of augmenting connection error messaging for now * Update tests * Bump versions + docs * auto-bump connector version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Refactor the buildDestinationRead function (#18446) Create a new buildDestinationRead function, that function can be called by top-level functions in the handler that already have a DestinationConnection object. * 🐛 Fix S3 destination integration tests to use bucket path (#18031) * Fix S3 destination tests * Add Changelog * CR Fixes * add unit test * version bump * fix dockerfile * auto-bump connector version Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * fix formatting + imports (#18457) * RefreshSchema method on RefreshSchema activity (#18460) * Add refreshSchema method to RefreshSchema activity * 🐛 Source Tiktok Marketing: fix test timeout (#18433) * Postgres Source : updated docs for CDC setup (#18291) * Postgres Source updated docs for CDC setup * fixed typo * fixed typo * rephrased * rephrased * Source postgres: disable allow and prefer ssl modes in CDC mode (#18256) * Source postgres: disable ssl_mode allow and prefer in CDC mode * Source postgres: code review fixes * Source postgres: code review fixes * Source postgres: improve ssl mode handling * Source postgres: bump version * auto-bump connector version * Source postgres: bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Twilio: implement slicing (#18423) * #804 source twilio: implement slicing * #804 source twilio - format code * #804 source twilio - implement slicing * #804 source twilio: configure incremental streams * #804 fix expected records * #804 extend Calls stream schema * #804 source twilio: allow configurable slice for SATs * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Fix broken table in the docs (#18466) * 🎉 New Connector: Clockify [python cdk] (#17767) * new connector source-clockify * feat: enable caching for streams (users, projects) * chore: pep8 changes at unit_tests * chore: pep8 changes at integration_tests * chore: update schema * Adds python formatting, removes unused import. * Makes the task duration field nullable to pass integration test. * fix: add second type to null values to the schema files * Adds a null fallback value to task duration. * Updates airbyte-cdk dependency. * Adds UUID in source definitions. * auto-bump connector version * Requested changes. * add clockify to source def seed * correct spec.json add titles * add icon * run format * remove source spec * correct spec * add eof gitignore * auto-bump connector version Co-authored-by: nataly <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Nataly Merezhuk <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * Optimize listDestinationsForWorkspace route (#18456) Add a function to fetch all destinations in a workspace * Add new streams table components (#18262) * Add new streams table components * Update CatalogTree and Catalog section to render new components based on env-based flag * Add basic Bulk edit panel, stream panel, table row, table headers * Split the overlay out of the modal and make it its own component. Use in StreamPanel * Add Fields to en.json * StreamPanel -> StreamDetailsPanel * Add new stream fields table components * Use new stream fiels table props iface on Stream details panel * add basic auth for Lever (#17996) * add basic auth for Lever * Update lever-hiring.md * Update spec.json * Update source.py * Revert "Update spec.json" This reverts commit 77b10a653b1056c959664416a6c89eafb6f49e07. * Revert "Update source.py" This reverts commit 3d6d83c6c63597d0b8ed46038d17989b38ba9712. * Update spec.json * update spec.json, replace sample config, change check to auth_type * unit tests with parametrize * gradlew formatting python + fixing bugs to ensure it works * modifs as suggested by reviewer * Update test_source.py * Update test_source.py * fixing specs in order for integration tests to pass * fix acceptance test TestSpec.test_oauth_flow_parameters * bump the docker image version to 0.1.3 * auto-bump connector version Co-authored-by: Yiyang Li <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Source File: fix check method (#18481) * 🐛 Source File: fix check method * 🐛 Source File: bump version * 🐛 Source File: test for wrong format * 🐛 Source File: docs update * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Make script both Unix/Linux and OSX compatible (#18493) * Update to latest Temporal SDK release (#18492) * Trace launcher worker run method (#18441) * Bmoric/extract connection api (#18409) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * Fix PMD errors * Add explicit path to the controller * 🎉 New Source: Auth0 [python cdk] (#18338) * 🎉 New Source: Auth0 [python cdk] - Full import and incremental are supported - Only users stream is added - Added unit tests and ensure all acceptance tests are passed locally - It's free to create a new account in Auth0, so it's free to setup an integration account. * fix: remove unused import in source.py * auto-bump connector version Co-authored-by: sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Destination Redshift: handle empty s3 bucket_path (#18434) * make bucket_path required * check for nonempty path * not required * handle null/empty path * version bump + changelog * fix comment * auto-bump connector version * format Co-authored-by: Octavia Squidington III <[email protected]> * Use Google business mails by default for login (#18462) * support custom error messaging for error response + retryable errors (#18204) * support custom error messaging for error response + retryable errors * remove changed backoff i was using for testing * refactor filter to construct response status internally * pr feedback * bump version and update changelog * Alternate way to view temp/CSV files on Windows. (#18482) Document alternate method, using Docker intrinsic functionality, to copy temp files locally in the situation that cannot be browsed through WSL2's file explorer. * docs: fix link reference (#18503) * 🐛Destination-elasticsearch: enforce ssl connection on cloud (#18341) * [16992] Destination-elasticsearch: enforce ssl connection on cloud * Bmoric/extract destination api (#18502) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * Fix PMD errors * Add explicit path to the controller * Extract destiantion API * bump to v15.0 (#18467) * bump to v15.0 * disable backwards compatible test for 0.2.69 * fix spec * skip comprehensive * bump * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Persist geography updates (#18501) * persist geography in connection update * use ApiPojoConverters * add test coverage for persisting geography update * persist geography column for workspace * use US instead of AUTO in mock data so that tests don't pass due to the database-level default column value * use ApiPojoConverters instead of Enum.convertTo in workspace handler * format * Add SSH tunnel for MongoDB Destination (#18280) * Add SSH tunnel for MongoDB Destination * Fixed host/port resolution issues with Docker containers * Formatting changes * Bump version number and adds documentation for SSH tunnel * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Improve performance of the operation that retrieve all connector for a given definition (#18499) * 🎉 New Source: News API (#18301) * Initial commit * Add streams * Simplify using options in request_parameters * Address comments * Add examples to query * Add docs * Change authenticator to ApiKeyAuthenticator * add seed config * remove icon * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Source Hubspot: acceptance fix (#18498) * 🎉 New Source: Workable (low-code SDK) (#18033) * Generate initial Workable connector * Implement jobs route * Make acceptance tests pass * Implement pagination * Try to fix pagination (not yet working) * Make pagination work * Add created_after parameter * Add Workable docs * Move unused test * Add candidates stream * Fix acceptance tests * Remove unused files * Add stages stream * Add recruiters stream * Remove sample_state file * Add Workable to docs/integrations/README.md * Update changelog * Change Sentry page size back * Rename created_after_date to start_date * Remove data type map * Add start_date to sample_config.json * Add streams documentation to yaml * add title to params inspec * format files * fix: added source to source_definitions Co-authored-by: Harshith Mullapudi <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * source-airtable: improve discovery logic (#18491) * 🎉 Destination Meilisearch: migrating to python and fix integration (#18036) * Add meilisearch python destionation * add destination * first working version * Writting tests * writer more resilient * Adding unit test * correct integration test stream param * update doc * remove unused variables * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🐛 Source File: add retry logic for `Connection reset by peer - 104` error (#18428) * 🪟 🐛 Split Heading and Text components (#17992) * split Text and Heading components * replace Text with Heading where appropriate * 🐛 Source Facebook Marketing: increase timeout (#18536) * Bmoric/extract db migration api (#18459) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * tmp * Fix PMD errors * Extract DB migrator * Add something that I forgot * restore destination factory initialization * format * 🪟 Apply default sync mode logic to new streams (#18451) * brute force solution * cleanup * testing and comment * review cleanup * Bmoric/extract destination definition api (#18515) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * tmp * Fix PMD errors * Extract DB migrator * Add something that I forgot * extract destination definition api * restore destination factory initialization * format * format * explicit path * 🪟 🎨 Add `PillSelect` UI component (#18376) * Add PillButton UI component Update Text component to support xs size * Update PillButton prop names * Remove outline and tint from PillButton until it's resolved with design * Add PillButton example to popout * PillButton -> PillSelect Remove story from Popout Create PillSelect component that manages select logic Make PillButton part of PillSelect * Support text overflow in pill button * Show tooltip on pill select multiple items * Use border-thin var in PillButton style * Update gap in PillButton to use spacing var * extract destination definition specification api (#18521) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * tmp * Fix PMD errors * Extract DB migrator * Add something that I forgot * extract destination definition api * restore destination factory initialization * extract destination definition specification api * format * format * format * explicit path * Restore docs build (#18551) + Corrects two errors, unblocks docs build * add authentication info to local development guide (#18509) * add authentication info to local development guide * rephrase * 🪟 🐛 Fix UI issues in connection streams table (#18544) * Fix PathPopout styling * Ensure that text overflow still works * Only show tooltip when there is more than 1 * Use caret instad of sort icon so that it's centered * Fix option content for SyncSettignsDropdown so that it fits * Fix @forward on StreamHeader scss * Rollback tooltip updates * Simplify the OSS documentation deploy system (#2670) (#18436) + Updates tools/bin/deploy_docusaurus to work on Github Runners + Adds 'Deploy docs.airbyte.com' Github workflow * Simplify the OSS documentation deploy system (#2670) (#18448) + Updates Airbyte Docs: informs about automated Docs site build and deployment. * Source Iterable: distinguish 401 from empty stream (#18537) * #829 source iterable: distinguish 401 from empty stream * #829 source iterable: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Databricks destination: introduce personal staging location (#18557) * Update s3 parquet writer - Rename gcsFileLocation. - Merge local variables. * Implement staging location getter * Add integration test and fix post request * Add comment and format code * Ensure that restart delay is set before any other activity is executed (#18547) * Ensure that restart delay is set before any other activity is executed * Add version check * Formatting * 🎉 New Source: ActiveCampaign (#18335) * Init source ActiveCampaign * Added documentation files * Added contacts and campaigns streams * Added deals stream * added segments stream * Added lists stream * Formatted schemas * Added references to API docs * added forms stream * Added pagination * Updated documentation * Formatted schemas * Removed unnecessary files * Added changelog * Updated base URL * Added sample config * add source seed * run format * generate uuid for activecampaign * auto-bump connector version Co-authored-by: Rytis Mickevičius <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * normalization: Fix failing mypy checks (#18534) * Use Protocol Versions in Syncs (#18324) * Pass protocol version into IntegrationLauncherConfig * Use VersionedStreamStreamFactory in AirbyteSource/Destination * Add AirbyteMessageBufferedWriter * Use VersionedBufferedWriter * 🪟 🎉 Client-side validation of cron expressions <= 1 hr for cloud (#18044) * add feature for cron expression format * add new validation function for sub one hr cron * add feature-dependent validation of cronExpression * add test coverage for create & edit connection forms * enable sub one hour cron expressions on cloud temporarily * Lowcode connectors: Fix few issues with stream slicers (#18529) * Fix a few bugs * bump * Protocol change: Define a set of well-known data types (#17486) * add WellKnownTypes.yaml * rename to snakecase + put in airbyte-protocol * add examples * more descriptoins * descriptions, more restrictions, better regex * update documentation * explicitly call out BC support * 🎉 New Source: Coin API (#18302) * Initial commit * Add streams * Correct acceptance test config * Make secret * Update docsurl * Remove catalog.json * Add second stream * Add default to spec * Change to start_date and end_date * Add links to CoinAPI docs in spec * Change to start_date and end_date * Add docs * add coin api to source seed file * remove icon * generate source spec for coin api Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: marcosmarxm <[email protected]> * 🎉 New Source: TVMaze Schedule (#18333) * Initial commit * Add streams * Remove file and update country code description * Add docs * format * correct schema * add tvmaze schedule to source spec seed * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Marcos Marx <[email protected]> * 🎉 New Source: MailerLite (#18336) * Init source MailerLite * Added documentation * Removed unnecessary files and a comment * Added changelog * Added sample config * Updated base URL * format * add mailerlite to source def seed * update docker image tag version * auto-bump connector version Co-authored-by: Dainius Salkauskas <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: Nasa APOD [python cdk] (#18394) * NASA APOD connector first implementation * Fix Acceptance and Unit tests * Fix DecrecationWarning on NoAuth * Update changelog with PR id * `date` input removal in favor of `start_date` + Minor fixes * Connector renaming to support different Nasa streams * Update docs and source definitions with connector rename * Support incremental updates * Fix typo on abnormal_state.json, making acceptance tests to fail * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Add missing generated python files (#18580) * 🎉 New Source - Lokalise [low-code SDK] (#18522) * Add keys stream * Fix keys stream for now * Start to add languages stream * Add docs * Start to add pagination * Remove paginator again * Revert "Remove paginator again" This reverts commit 3f8daace174473be767ef78d0fa852be81925203. * Pagination seems to work * Clean up yaml * Increase max page size * Docs improvements * Add comments stream * Add contributors stream * Fix contributors stream * Add translations source * Add new acceptance test format * Add abnormal state * Add PR number * run format * simplify selector * add eof * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: Oura (#18224) * Initial commit * Add streams * Final tidying * Make secret * Remove start_date and end_date parameters * Add docs * update cdk version * run format + add source def for oura * run config process manually Co-authored-by: marcosmarxm <[email protected]> * 🎉 New Source: Ashby (#18334) * create boilerplate * configure connector * add ashby documentation * update airbyte docs * configure tests * Revert "update airbyte docs" This reverts commit 7dcf2b2dad1a77007b64aa8453aeb1c27e74ae58. * update airbyte documentation * update catalogs * add start date for specific streams * update config files * add ashby to source def seed * update doc * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source - Sonar Cloud (#18475) * init commit * add docs * add docs * fix comments * change string to array * 🎉 New source: SFTP Bulk [python cdk] (#17691) * [INIT] setup source-ftp Signed-off-by: Henri Blancke <[email protected]> * [ADD] add logic Signed-off-by: Henri Blancke <[email protected]> * [ADD] unit, integration, acceptance tests Signed-off-by: Henri Blancke <[email protected]> * [UPD] update docs Signed-off-by: Henri Blancke <[email protected]> * [UPD] clean up crew Signed-off-by: Henri Blancke <[email protected]> * [UPD] update host to localhost Signed-off-by: Henri Blancke <[email protected]> * [UPD] rename from ftp to sftp bulk Signed-off-by: Henri Blancke <[email protected]> * [INIT] setup source-ftp Signed-off-by: Henri Blancke <[email protected]> * [ADD] add logic Signed-off-by: Henri Blancke <[email protected]> * [ADD] unit, integration, acceptance tests Signed-off-by: Henri Blancke <[email protected]> * [UPD] update docs Signed-off-by: Henri Blancke <[email protected]> * [UPD] clean up crew Signed-off-by: Henri Blancke <[email protected]> * [UPD] update host to localhost Signed-off-by: Henri Blancke <[email protected]> * [UPD] rename from ftp to sftp bulk Signed-off-by: Henri Blancke <[email protected]> * update branch * [FIX] integration tests Signed-off-by: Henri Blancke <[email protected]> * [FIX] acceptance test fixture Signed-off-by: Henri Blancke <[email protected]> * [UPD] change ftp port Signed-off-by: Henri Blancke <[email protected]> * add uuid to image build * add source def * [FIX] generate ssh keys Signed-off-by: Henri Blancke <[email protected]> * auto-bump connector version Signed-off-by: Henri Blancke <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: Mailjet Mail API (#18332) * init commit * add streams * add schemas * add docs * add docs * fix comments * restore vscode config * restore vscode config * clean * lint * lint * lint * lint * fix comments Co-authored-by: Marcos Marx <[email protected]> * Sonar Cloud: add to source spec and publish connector (#18584) * solve conflict * solve conflict spec.yaml * add sonar cloud to seed + format * add eof * generate config source spec * Source Mailjet Mail: publish connector and additional format (#18586) * init commit * add streams * add schemas * solve conflict * add docs * fix comments * restore vscode config * restore vscode config * clean * lint * lint * lint * lint * fix comments * run format * update acceptance test * update schema * update acceptance test * auto-bump connector version Co-authored-by: Haithem SOUALA <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Implement Connector Builder testing panel (first iteration) (#18449) * [SPIKE] add Builder page with editor * add YamlEditor component * export colors from CSS * move template into dedicated file * fix initial load from local storage * move download button into separate component * fix stylelint * remove console log * add todo * make monaco background transparent and apply gradient to parent div background * clarify comment * remove unnecessary 180deg * lock the builder UI behind a feature * use rgb instead of hex to fix stylelint * use _colors.scss and disable hex length stylelint rule * disable rule in file * generify ConnectorDocumentationLayout into TwoPanelLayout * use TwoPanelLayout for BuilderPage * add darker gradient color to _colors.scss and use in gradient * move /builder to /connector-builder * rename TwoPanelLayout to ResizablePanels * put gradient on parent container instead * make lightOverlay actually overlay instead of un-rendering children * add testing panel component and stream selector * fix rotated text * fix more styling * remove unnecessary comment * add starting flex to right panel * replace react-select dropdown with headless-ui listbox * add test controls and fix button icon styling * add Configure Test Input button and modal * add connector builder client * move connector builder state into dedicated context * fix bold prop in Text component to not throw error * pass parsed yaml->json into query * store config in connector builder state context * add result display * fix styles of result display * move config menu into testing panel * capitalize stream names * undo changes to Text component, since that is being addressed in a separate PR * fix more merge issues and fix generated schema usage * capitalize header * remove console log * rename data to streamRead * fix stylelint and use variables * use another variable for spacing * use stream name in state data object * move ConnectorBuilderStateProvider into ConnectorBuilderPage * fix sizing of modal * fix styling of active stream selector option * Update airbyte-webapp/src/components/StreamTestingPanel/ResultDisplay.tsx Co-authored-by: Tim Roes <[email protected]> * Update airbyte-webapp/src/components/StreamTestingPanel/ResultDisplay.tsx Co-authored-by: Tim Roes <[email protected]> * hide result display panel if no data * remove button specificity styling hacks, use size prop instead * Update airbyte-webapp/src/components/StreamTestingPanel/StreamSelector.tsx Co-authored-by: Tim Roes <[email protected]> * change logs to errors Co-authored-by: Tim Roes <[email protected]> * Source: Google Search Console: validate custom_reports.dimension (#18568) Signed-off-by: Sergey Chvalyuk <[email protected]> * SAT: make `expect_records` mandatory in `high` `test_strictness_level` (#18497) * 🎉 New destination: Yugabytedb (#18039) * add yugabytedb destination connector * add dest def + add changelog * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * fix worker volumeMounts (#18604) * 🎉 New Source: RD Station Marketing [python cdk] (#18348) * Add new source: RD Station Marketing * Update catalog with all streams * Update changelog with the rigth PR reference number * Minor fixes to RD Station docs * Change field replication_start_date to start_date * Fix unit tests after changing replication_start_date name * Fix typo * Refact next_page_token logic * Fix unit tests * Remove print function * Update airbyte-cdk version to 0.2 * Update airbyte-integrations/connectors/source-rd-station-marketing/source_rd_station_marketing/streams.py Co-authored-by: Marcos Marx <[email protected]> * Change from yield to yield from * Apply format * add rd station to source def * update doc changelog * auto-bump connector version Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Simplify the OSS documentation deploy system (#2670) (#18598) + Adds Slack notification about failed Docs builds (slack notifications fall into "#oss-master-build-failure" for now) + (while here) Unbreaks docs build * improved the digitalocean deployment guide (#18566) * fixed failed build (#18610) * 🎉 New Source: ConvertKit (#18455) * Init source omnisend * Removed unnecessary files * Init source convertkit * Added forms, sequences streams * Added tags stream * Added subscribers, broadcasts streams * Added documentation * Removed unnecessary files * Updated pull request information in documentation * Added sample config * Updated invalid config * Formatting, removed abnormal state, uncommented full_refresh acceptance test * Added pagination for subscribers stream * fix: add source definition for convertkit * auto-bump connector version Co-authored-by: Sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉New Source: Google Webfonts [low-code cdk] (#18496) * New source: Google webfonts * chore: Add Docs * chore: update changelog * chore: resolved given comments for PR * chore: unwanted files removed * fix: generate and add source definitions Co-authored-by: sajarin <[email protected]> * Clean up build.gradle. (#18555) Upstream has fixed this bug so we no longer need to host it internally. Remove this to clean up build.gradle. * 🎉 New Destination: Typesense (#18349) * Initial boilerplate * 🎉 New Destination: Typesense * remove .java-version * fix doc * add typesense to dest def * add release stage * add requirement to main * auto-bump connector version * add changelog Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: Waiteraid [low-code cdk] (#18165) * added source waiteraid * 🎉 New Source: Waiteraid * add searchBookings stream * add P/R number * add SUMMARY entry * add docs/integrations/README.md entry * add builds.md entry * add docs to each endpoint * fix schema ~ * Update airbyte-integrations/builds.md added web address Co-authored-by: Marcos Marx <[email protected]> * Delete catalog.json * Delete TODO.md * Update spec.yaml * add waiteraid to source def seed * auto-bump connector version Co-authored-by: Sebastian Brickel <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Replace `recipesLink` link with `tutorialsLink` (#18616) * Protocol Change: `AirbyteControlMessage.ConnectorConfig` (#17907) * Protocol Change: AirbyteConfigMessage * update PR link in docs * Lint * Update python files * Update docs/understanding-airbyte/airbyte-protocol.md Co-authored-by: Sherif A. Nada <[email protected]> * Update docs/understanding-airbyte/airbyte-protocol.md Co-authored-by: Sherif A. Nada <[email protected]> * `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage` * AirbyteOrchestratorMessage * Update docs * `AirbyteControlConnectorConfigMessage` Co-authored-by: Sherif A. Nada <[email protected]> * Remove workflow version check (#18613) * Improved the Oracle cloud deployment guide (#18615) * Update OCI VM deployment guide * Update OCI deployment guide * Update on oci-vm.md * 🐛 Lowcode: ListStreamSlicer and SubstreamSlicer should get the stream_slice from the arguments (#18574) * fix list stream slicer * get the stream slice from the parameters * delete unused code * Add comments * missing test * newline * with pytest.raise * reset to master * newline * Update __init__.py * Update __init__.py * 🎉 New Source: ConvertKit (#18455) * Init source omnisend * Removed unnecessary files * Init source convertkit * Added forms, sequences streams * Added tags stream * Added subscribers, broadcasts streams * Added documentation * Removed unnecessary files * Updated pull request information in documentation * Added sample config * Updated invalid config * Formatting, removed abnormal state, uncommented full_refresh acceptance test * Added pagination for subscribers stream * fix: add source definition for convertkit * auto-bump connector version Co-authored-by: Sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉New Source: Google Webfonts [low-code cdk] (#18496) * New source: Google webfonts * chore: Add Docs * chore: update changelog * chore: resolved given comments for PR * chore: unwanted files removed * fix: generate and add source definitions Co-authored-by: sajarin <[email protected]> * Clean up build.gradle. (#18555) Upstream has fixed this bug so we no longer need to host it internally. Remove this to clean up build.gradle. * 🎉 New Destination: Typesense (#18349) * Initial boilerplate * 🎉 New Destination: Typesense * remove .java-version * fix doc * add typesense to dest def * add release stage * add requirement to main * auto-bump connector version * add changelog Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * 🎉 New Source: Waiteraid [low-code cdk] (#18165) * added source waiteraid * 🎉 New Source: Waiteraid * add searchBookings stream * add P/R number * add SUMMARY entry * add docs/integrations/README.md entry * add builds.md entry * add docs to each endpoint * fix schema ~ * Update airbyte-integrations/builds.md added web address Co-authored-by: Marcos Marx <[email protected]> * Delete catalog.json * Delete TODO.md * Update spec.yaml * add waiteraid to source def seed * auto-bump connector version Co-authored-by: Sebastian Brickel <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Replace `recipesLink` link with `tutorialsLink` (#18616) * Protocol Change: `AirbyteControlMessage.ConnectorConfig` (#17907) * Protocol Change: AirbyteConfigMessage * update PR link in docs * Lint * Update python files * Update docs/understanding-airbyte/airbyte-protocol.md Co-authored-by: Sherif A. Nada <[email protected]> * Update docs/understanding-airbyte/airbyte-protocol.md Co-authored-by: Sherif A. Nada <[email protected]> * `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage` * AirbyteOrchestratorMessage * Update docs * `AirbyteControlConnectorConfigMessage` Co-authored-by: Sherif A. Nada <[email protected]> * Remove workflow version check (#18613) * Improved the Oracle cloud deployment guide (#18615) * Update OCI VM deployment guide * Update OCI deployment guide * Update on oci-vm.md * bump Co-authored-by: Dainius Salkauskas <[email protected]> Co-authored-by: Sajarin <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: Balasubramanian T K <[email protected]> Co-authored-by: Davin Chia <[email protected]> Co-authored-by: Cirdes <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: sbrickel-nimble <[email protected]> Co-authored-by: Sebastian Brickel <[email protected]> Co-authored-by: Evan Tahler <[email protected]> Co-authored-by: Sherif A. Nada <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Victor Ikechukwu <[email protected]> * Add versioning logging (#18618) * Bmoric/extract healt api (#18523) * Tmp * Extract the Attempt API from the V1 API * Add comments * Move Connection API out of configuration API * format * format * Rename to Controller * Rename to Controller * Add values to the factory * Change the constructor to use hadler instead of objects needed by the handler * Update with new tags. * tmp * Fix PMD errors * Extract DB migrator * Add something that I forgot * extract destination definition api * restore destination factory initialization * extract destination definition specification api * format * format * format * extract health check api * fix test * format * Fix ConfiguredCatalog for Resets (#18625) We should only force (FULL_REFRESH,OVERWRITE) for the streams to reset. For the other streams, we only want to replace OVERWRITE with APPEND to avoid having destination clear the stream. The other cases should be left as is. * :tada: New Source: Mailjet SMS (#18345) * init commit * add doc * add doc * change authentication methode * fix doc * fix doc * add docs * fix comments * use request_options_provider * lint * fix comments Co-authored-by: Marcos Marx <[email protected]> * More comprehensive temporal error message (#18608) * To be remove * Remove the signal for waiting after a failed activity and ensure we are waiting the expected time * Revert "To be remove" This reverts commit 3a5f7b4f729724e6943450af15b324ca05ac6c4d. * Remove unused and move failure reason to the helper * Avoid repetitive new Set() * :tada: New Connector: Zendesk Sell [python cdk] (#17888) * first working connector - overwrite only * edit readme and source definitions * Update source_definitions.yaml * changing catalog and adding all available streams * remove custom fields * remove documents * manage incremental syncs * add unit tests ; add start date to specs * try catch for next page token method * Create source_to_test.py * Update source.py * Update source_to_test.py * working source * Delete TODO.md * Update source.py * Update spec.yaml * Update test_streams.py * give up on incremental and have something that works * removing start date necessity in spec and sample configs * add connector to source def * run format * update airbyte cdk version * correct unit test * run format * auto-bump connector version Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Source Mailjet SMS: publish PR and add source def (#18620) * init commit * add docs * fix comments * use request_options_provider * solve conflict * solve doc conflict * Source Mailjet Mail: publish connector and additional format (#18586) * init commit * add streams * add schemas * solve conflict * add docs * fix comments * restore vscode config * restore vscode config * clean * lint * lint * lint * lint * fix comments * run format * update acceptance test * update schema * update acceptance test * auto-bump connector version Co-authored-by: Haithem SOUALA <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * add source def + correct tests * update schema Co-authored-by: Haithem SOUALA <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Add endpoint to retrieve manifest template (#18578) * add endpoint to retrieve manifest template * remove 400 error from endpoint * Source S3: use AirbyteTracedException (#18602) * #750 # 837 #904 Source S3: use AirbyteTracedException * source s3: upd changelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Fix source spec diff (#18645) * Rename maxCursor and add comment (#18570) * Rename maxCursor to make the code easier to understand * Rename and update comment * ami-0d648081937c75a73 -> ami-06cf12549e3d9c522 (#18667) Signed-off-by: Sergey Chvalyuk <[email protected]> * 🐛 Source Google Analytics (Universal Analytics): add `Custom Reports` schema validation on `Check Connection` (#18670) * 🪟 🐛 Fix trailing whitespace and ending with a forward slash transformation GitHub URL issue (#18299) * add regex matcher for custom transformation GitHub url * move validationSchema to separate file * add tests for validationSchema * 🎉 New Source: Omnisend (#18577) * Init source omnisend * Removed unnecessary files * added campaigns, carts, orders, products streams * Fix acceptance tests * Added documentation * Added pagination * Removed abnormal state, uncommented acceptance tests * Added docs references * Added reference to PR * Added stop condition for pagination * Added title to API Key in spec.yaml * add omnisend source to def seed * run format * auto-bump connector version Co-authored-by: Rytis Mickevičius <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Added a note about chained dbt transformations (#18678) * Source metabase: allow using only https urls (#18607) * #16519 source metabase: allow using only https urls * source metabase: upd changelog * #16519 fix url regexp * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * 🪟 🎉 Add sync mode, primary key, cursor select components to new streams table (#18627) * Add SyncModeSelect component Add button type to pill button and stop propagation Stop propagation on popoout outside click listener and migrate to scss module * Update DropDown option to handle click event directly and stop propagation * Add StreamPathSelect component to select the primary key and cursor paths Update PillSelect to handle nil values * Add SyncMode, Primary Key, and Cursor selects to CatalogTreeTableRow * Replace popup click catcher with Overlay component Add Overlay color variant and onClick handler option Fix Overlay import * Source Intercom: change airbyte-cdk version to 0.2 (#18681) * Source Intercom: change airbyte-cdk version to 0.2 * Source Intercom: update chagelog * Source Intercom: bump version and update chagelog * Source Intercom: Fix PR id in chagelog * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * feat: Add airbyte-cron Helm chart (#18542) * feat: Add airbyte-cron Helm chart * New chart resides in `charts/airbyte-cron` * Updated the parent chart (`airbyte`) `values.yaml` to include config key for `cron` * Add airbyte-cron to release workflow * chore: remove TEMPORAL_* vars Co-authored-by: Kyryl Skobylko <[email protected]> * fix: Typo in semver format for chart (#18686) This looks to have been a mistaken change that was made to the workflow. Reverting back to the original value. * 🐛 Source Amplitude: fix empty `series` for `AverageSessionLength` stream (#18684) * Destination Redis: TLS support (#18358) * 16249 Destination Redis TLS support * Refactoring based on pr review * Remove duplicate paragraph (#18662) Removes a duplicate of the paragraph that begins "Note that we're also setting the `stream_cursor_field` in the stream's `$options`..." which followed an outdated schema. * Simplify the job persistence method. (#18698) Method interface: - has generic type that isn't needed. Confusing because no generic is happening here. - has two existing parameters that are data fields of one of the already existing parameters. Confusing since the two additional parameters aren't actually supposed to be passed in separately from the existing parameter. Remove both. * chore: Add airbyte-cron chart to .bumpversion.cfg (#18688) * fix helm release workflow (#18716) * AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689) Signed-off-by: Sergey Chvalyuk <[email protected]> * Remove unused test script. (#18715) * ci: replace set-output with new github actions syntax (#18691) * Sources postgres, mysql, alloydb: encode database name (#18538) * Source postgres: encode database name * Source postgres, mysql: move encoding in util class, apply for mysql * Source postgres, mysql: make var final * Source postgres, mysql: bump version * Source postgres, mysql: format code * auto-bump connector version * Source mysql: bump version * auto-bump connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]> * Source Prestashop: prevent users from using http url (#18599) Signed-off-by: Sergey Chvalyuk <[email protected]> * add nonBreakingChangePreference and notifySchemaChanges (#18636) * add nonBreakingChangePreference and notifySchemaChanges * Changes docs deploy failures to show up in #docs (#18738) * Green house docs (#18314) * corrected the documentation for FreshDesk Connector * corrected the documentation for Greenhouse Connector * Resurfaces hidden cloud connectors within UI (#18748) * Update helpers functions in subcharts (#18757) * Update helpers functions in subcharts; Refactor component.imageTag function call * fix typo in _helpers.tpl * Update chart.yaml to reference right subchart (#18718) * Update chart.yaml to reference right subchart * Update airbyte-cron chart name * 🎉 New Destination: Heap Analytics [python cdk] (#18530) * 🎉 New Desination: Heap Analytics [python cdk] - implement a heap client to load data via the server-side API: https://developers.heap.io/reference/server-side-apis-overview - the connector supports a generic data source, and the api_type determines the output schema. The output schema is dynamic. - users pick the columns that will be loaded to the destination - Consequently, each configured catalog only includes one stream * add a bootstrap to illustrate the connector * add destination dest def * run format all files * correct unit test * auto-bump connector version Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]> * Source Salesforce: add get_error_display_message for ConnectionError (#18753) Signed-off-by: Sergey Chvalyuk <[email protected]> * fix helm release workflow (#18776) * Update to latest Micronaut release (#18774) * fix cron resources spec (#18778) * ci: use oldgithub actions notation for output in publish-helm-charts worklow (#18780) * improve connector dependency github action (#18480) * Use more specific variable name * Write affected sources and destinations to files * Use Markdown template to format results * Update workflow to edit existing comment instead of making a new one * Check whether changed_files.txt is empty before proceeding * move template to .github/ * Use different quotes for true * Move logic about continuing back to after dependency report because of ignores * Use correct filename in conditional * Use if to avoid early exit * Create folder for comment templates and use constant for file path * Add handling for non-source non-destination info, pull out markdown listing * 🐛 Add a drop table hook to drop scd tables in case of overwrite sync (#18015) * Add a drop table hook to drop scd tables in case of overwrite sync * Add an integration test for dropping SCD table on overwrite * skip new test for Oracle and TiDB * Add normalization run after initial reset * Bump normalization version * Move components to `components/common` (#18728) * Move ApiErrorboundary to components/common and cleanup * Move PageViewContainer to…

henriblancke added 5 commits October 6, 2022 12:56

[INIT] setup source-ftp

65cc070

Signed-off-by: Henri Blancke <[email protected]>

[ADD] add logic

919f120

Signed-off-by: Henri Blancke <[email protected]>

[ADD] unit, integration, acceptance tests

2160aca

Signed-off-by: Henri Blancke <[email protected]>

[UPD] update docs

a50912c

Signed-off-by: Henri Blancke <[email protected]>

[UPD] clean up crew

fb14d03

Signed-off-by: Henri Blancke <[email protected]>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 6, 2022

octavia-squidington-iv added bounty community connectors/source/ftp labels Oct 6, 2022

[UPD] update host to localhost

39f21c1

Signed-off-by: Henri Blancke <[email protected]>

sajarin added internal and removed bounty labels Oct 17, 2022

henriblancke added 2 commits October 19, 2022 11:20

Merge branch 'master' into source-ftp

02b04ac

[UPD] rename from ftp to sftp bulk

79580cd

Signed-off-by: Henri Blancke <[email protected]>

henriblancke changed the title ~~🎉 New source: FTP~~ 🎉 New source: SFTP Bulk Oct 19, 2022

octavia-squidington-iv added the connectors/source/sftp-bulk label Oct 19, 2022

marcosmarxm added the hacktober label Oct 19, 2022

marcosmarxm changed the title ~~🎉 New source: SFTP Bulk~~ 🎉 New source: SFTP Bulk [python cdk] Oct 19, 2022

henriblancke added 3 commits October 20, 2022 16:21

[INIT] setup source-ftp

d518ba1

Signed-off-by: Henri Blancke <[email protected]>

[ADD] add logic

b4c0d41

Signed-off-by: Henri Blancke <[email protected]>

[ADD] unit, integration, acceptance tests

b929aed

Signed-off-by: Henri Blancke <[email protected]>

marcosmarxm force-pushed the source-ftp branch from 79580cd to 8d6c95e Compare October 20, 2022 19:32

henriblancke added 3 commits October 20, 2022 15:44

Merge branch 'source-ftp' of github.com:henriblancke/airbyte into sou…

0c92a5a

…rce-ftp Signed-off-by: Henri Blancke <[email protected]>

[FIX] integration tests

50acaa5

Signed-off-by: Henri Blancke <[email protected]>

[FIX] acceptance test fixture

d052014

Signed-off-by: Henri Blancke <[email protected]>

[UPD] change ftp port

4e00b97

Signed-off-by: Henri Blancke <[email protected]>

RealChrisSean mentioned this pull request Oct 21, 2022

SFTP Bulk [Python CDK] airbytehq/connector-contest#172

Closed

add uuid to image build

a78c1aa

add source def

7ca3385

marcosmarxm approved these changes Oct 27, 2022

View reviewed changes

[FIX] generate ssh keys

9e8798a

Signed-off-by: Henri Blancke <[email protected]>

auto-bump connector version

ee895d7

marcosmarxm merged commit c469ea8 into airbytehq:master Oct 28, 2022

henriblancke deleted the source-ftp branch October 28, 2022 13:11

octavia-squidington-iii mentioned this pull request Nov 1, 2022

Bump Airbyte version from 0.40.17 to 0.40.18 #18827

Merged

RealChrisSean added the python Pull requests that update Python code label Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New source: SFTP Bulk [python cdk] #17691

🎉 New source: SFTP Bulk [python cdk] #17691

henriblancke commented Oct 6, 2022 •

edited

Loading

marcosmarxm commented Oct 11, 2022

henriblancke commented Oct 11, 2022

henriblancke commented Oct 11, 2022 •

edited

Loading

henriblancke commented Oct 13, 2022

marcosmarxm commented Oct 17, 2022

marcosmarxm commented Oct 18, 2022

henriblancke commented Oct 19, 2022

marcosmarxm commented Oct 19, 2022

marcosmarxm commented Oct 20, 2022

henriblancke commented Oct 21, 2022

marcosmarxm commented Oct 21, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 21, 2022 •

edited

Loading

henriblancke commented Oct 21, 2022

marcosmarxm commented Oct 24, 2022

marcosmarxm commented Oct 25, 2022 •

edited by github-actions bot

Loading

marcosmarxm Oct 27, 2022

henriblancke Oct 27, 2022

marcosmarxm commented Oct 28, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 28, 2022 •

edited by github-actions bot

Loading

		-----BEGIN OPENSSH PRIVATE KEY-----
		b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn

🎉 New source: SFTP Bulk [python cdk] #17691

🎉 New source: SFTP Bulk [python cdk] #17691

Conversation

henriblancke commented Oct 6, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Oct 11, 2022

henriblancke commented Oct 11, 2022

henriblancke commented Oct 11, 2022 • edited Loading

henriblancke commented Oct 13, 2022

marcosmarxm commented Oct 17, 2022

marcosmarxm commented Oct 18, 2022

henriblancke commented Oct 19, 2022

marcosmarxm commented Oct 19, 2022

marcosmarxm commented Oct 20, 2022

henriblancke commented Oct 21, 2022

marcosmarxm commented Oct 21, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Oct 21, 2022 • edited Loading

henriblancke commented Oct 21, 2022

marcosmarxm commented Oct 24, 2022

marcosmarxm commented Oct 25, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm Oct 27, 2022

Choose a reason for hiding this comment

henriblancke Oct 27, 2022

Choose a reason for hiding this comment

marcosmarxm commented Oct 28, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Oct 28, 2022 • edited by github-actions bot Loading

henriblancke commented Oct 6, 2022 •

edited

Loading

henriblancke commented Oct 11, 2022 •

edited

Loading

marcosmarxm commented Oct 21, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 21, 2022 •

edited

Loading

marcosmarxm commented Oct 25, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 28, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 28, 2022 •

edited by github-actions bot

Loading