🎉 New Source: Qonto [python cdk] #17452

leilaballouard · 2022-09-30T11:02:20Z

What

Creation of a new source connector : Qonto API , following this issue

How

It uses the python HTTP API source connector template provided by Airbyte.
There is currently 3 streams :

Authentication is made using a basic method, Oauth can be implemented.

🚨 User Impact 🚨

No breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-09-30T11:02:29Z

All committers have signed the CLA.

leilaballouard · 2022-10-04T18:48:09Z

Closing this PR in order to avoid the CI running for nothing.Will re-open later.

leilaballouard · 2022-10-25T13:23:15Z

There is two main points I think still need some improvement:

I'm not sure I'm testing the connector the right way : Qonto is a paid application, but you can retrieve data using a sandbox ( https://api-doc.qonto.com/docs/business-api/cdc1ae4176df5-sandbox-testing ) but it requires registration of the application. There is also "mocked servers" that can be requested (as we can see in the right panel in the documentation here )
Is there another connector you can advise that is working the same way so I can get inspiration ? Or should we register an airbyte app ?
Regarding authentication to Qonto : it can be done either with a token , either using OAuth. I have very few understanding of Oauth using airbyte so I went with the token authetication, but do you advise something in particular ?

Thanks !

sajarin · 2022-10-25T15:16:20Z

Thanks @leilaballouard for the PR. We'll have someone for our community maintainer program review this PR!

vincentkoc · 2022-11-03T09:25:53Z

@leilaballouard are you able to contact me on Airbyte Slack and share test credentials so i can validate the tests?

vincentkoc

Hi @leilaballouard thanks for your contribution, im one of the Airbyte maintainers and will be reviewing your PR. If you can post a copy of your integration tests so i can validate that would be great.

vincentkoc

Changes required to spec and need to add an "endpoint variable" see other examples like Zuora connector which has various endpoints selectable

airbyte-integrations/connectors/source-qonto/source_qonto/spec.yaml

airbyte-integrations/connectors/source-qonto/acceptance-test-config.yml

airbyte-integrations/connectors/source-qonto/integration_tests/invalid_config.json

airbyte-integrations/connectors/source-qonto/source_qonto/spec.yaml

vincentkoc · 2022-11-08T13:20:01Z

=============================== warnings summary ===============================
../usr/local/lib/python3.9/site-packages/source_acceptance_test/config.py:246: 26 warnings
test_core.py: 1 warning
  /usr/local/lib/python3.9/site-packages/source_acceptance_test/config.py:246: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
    logging.warn("The acceptance-test-config.yml file is in a legacy format. Please migrate to the latest format.")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============ 24 passed, 3 skipped, 27 warnings in 61.96s (0:01:01) =============

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD SUCCESSFUL in 6m 48s
48 actionable tasks: 32 executed, 1 from cache, 15 up-to-date

vincentkoc · 2022-11-08T13:20:44Z

@leilaballouard build is passing, we just need to do a few more things to finalise and merge this PR. Thanks again for making the suggested changes!

leilaballouard · 2022-11-08T14:57:02Z

@koconder Regarding integration tests, it seems ok ✅ :

(.venv) ➜  source-qonto git:(leilaballouard/17235-source-qonto) python -m pytest -p integration_tests           
Test session starts (platform: darwin, Python 3.9.11, pytest 6.1.2, pytest-sugar 0.9.5)
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/leilaballouard/airbyte-fork/airbyte-integrations/connectors/source-qonto/.hypothesis/examples')
rootdir: /Users/leilaballouard/airbyte-fork, configfile: pytest.ini
plugins: hypothesis-6.54.6, sugar-0.9.5, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2, cov-3.0.0
collecting ... 
 airbyte-integrations/connectors/source-qonto/unit_tests/test_auth.py::test_authenticator ✓                                                                                 6% ▋         HTTPStatus.OK

 airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_check_connection[HTTPStatus.OK--expected_result0] ✓                                          12% █▍        HTTPStatus.NOT_FOUND

 airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_check_connection[HTTPStatus.NOT_FOUND- -expected_result1] ✓                                  19% █▉        HTTPStatus.UNAUTHORIZED

 airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_check_connection[HTTPStatus.UNAUTHORIZED-Invalid credentials-expected_result2] ✓             25% ██▌       
 airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams ✓                                                                                    31% ███▎      
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_request_params ✓                                                                            38% ███▊      
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_next_page_token ✓                                                                           44% ████▍     
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_parse_response ✓                                                                            50% █████     
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_request_headers ✓                                                                           56% █████▋    
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_http_method ✓                                                                               62% ██████▍   
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓                                                         69% ██████▉   
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓                                                75% ███████▌  
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓                                           81% ████████▎ 
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓                                       88% ████████▊ 
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_backoff_time ✓                                                                              94% █████████▍
 airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_transactions_request_params ✓                                                              100% ██████████
=================================================================================== warnings summary ====================================================================================
airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_transactions_request_params
  /Users/leilaballouard/airbyte-fork/airbyte-integrations/connectors/source-qonto/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:43: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-qonto/unit_tests/test_streams.py::test_transactions_request_params
  /Users/leilaballouard/airbyte-fork/airbyte-integrations/connectors/source-qonto/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.22s):
      16 passed

Also , I have issues running the acceptance tests (I'm not sure If i'm the one that should run them) :

(.venv) ➜  source-qonto git:(leilaballouard/17235-source-qonto)  python -m pytest -p integration_tests.acceptance 
.....
....
Results (17.63s):
      11 passed
       3 failed
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:215 TestConnection.test_check[inputs0]
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:215 TestConnection.test_check[inputs1]
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:252 TestDiscovery.test_discover[inputs0]
      10 error
       2 skipped

All are failing with this errror :

docker.errors.ContainerError: Command 'discover --config /data/tap_config.json' in image '<Image: 'airbyte/source-qonto:dev'>' returned non-zero exit status 1: 
{
  "type":"TRACE",
  "trace":{
     "type":"ERROR",
     "emitted_at":1667917568625.158,
     "error":{
        "message":"Something went wrong in the connector. See the logs for more details.",
        "internal_message":"[Errno 2] No such file or directory: '/data/tap_config.json'",
        "stack_trace":"Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 85, in run\n    raw_config = self.source.read_config(parsed_args.config)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py\", line 51, in read_config\n    with open(config_path, \"r\") as file:\nFileNotFoundError: [Errno 2] No such file or directory: '/data/tap_config.json'\n",
        "failure_type":"system_error"
     }
  }
}

vincentkoc · 2022-11-08T22:39:59Z

These are your failures, i will check my results and report back.

       3 failed
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:215 TestConnection.test_check[inputs0]
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:215 TestConnection.test_check[inputs1]
         - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:252 TestDiscovery.test_discover[inputs0]

vincentkoc · 2022-11-10T21:00:14Z

/test connector=connectors/source-qonto

🕑 connectors/source-qonto https://github.com/airbytehq/airbyte/actions/runs/3440379042
✅ connectors/source-qonto https://github.com/airbytehq/airbyte/actions/runs/3440379042
Python tests coverage:

Name                       Stmts   Miss  Cover
----------------------------------------------
source_qonto/endpoint.py       4      0   100%
source_qonto/auth.py          12      0   100%
source_qonto/__init__.py       2      0   100%
source_qonto/source.py        78      7    91%
----------------------------------------------
TOTAL                         96      7    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              353    110    69%   53, 64-72, 77-84, 88-89, 93-94, 192, 230-247, 256-264, 268-273, 279, 312-317, 355-362, 405-407, 410, 475-483, 495-498, 503, 559-560, 566, 569, 605-615, 628-653
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1516    348    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:65: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:257: The previous connector image could not be retrieved.
================= 25 passed, 3 skipped, 28 warnings in 22.62s ==================

vincentkoc · 2022-11-10T21:54:01Z

/publish connector=connectors/source-qonto run-tests=false

🕑 Publishing the following connectors:
connectors/source-qonto
https://github.com/airbytehq/airbyte/actions/runs/3440711174

Connector	Did it publish?	Were definitions generated?
connectors/source-qonto	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* Creation of the new Qonto source connector * Update setup.py * Support production and mocked server endpoints * finalise mocked api calls * Update source_definitions.yaml * lint * auto-bump connector version Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

Creation of the new Qonto source connector

75e4eee

github-actions bot added the area/connectors Connector related issues label Sep 30, 2022

Merge branch 'master' into leilaballouard/17235-source-qonto

c0f275b

octavia-squidington-iv added community connectors/source/qonto labels Sep 30, 2022

leilaballouard mentioned this pull request Sep 30, 2022

New Source: Qonto #17235

Closed

leilaballouard closed this Oct 4, 2022

leilaballouard reopened this Oct 25, 2022

Merge branch 'master' into leilaballouard/17235-source-qonto

609f0e7

leilaballouard marked this pull request as ready for review October 25, 2022 13:00

octavia-squidington-iv added the bounty label Oct 25, 2022

Merge branch 'master' into leilaballouard/17235-source-qonto

93d7f8c

sajarin added the bounty-XL Maintainer program: claimable extra large bounty PR label Oct 25, 2022

sajarin changed the title ~~Creation of the new Qonto source connector~~ 🎉 New Source: Qonto Oct 25, 2022

vincentkoc self-assigned this Oct 27, 2022

vincentkoc self-requested a review October 27, 2022 14:29

Update setup.py

ea8f8ad

bleonard added the reward-200 label Oct 31, 2022

Merge branch 'master' into leilaballouard/17235-source-qonto

037b5c5

vincentkoc changed the title ~~🎉 New Source: Qonto~~ 🎉 New Source: Qonto [python cdk] Nov 3, 2022

vincentkoc suggested changes Nov 4, 2022

View reviewed changes

vincentkoc and others added 2 commits November 4, 2022 22:52

Merge branch 'master' into leilaballouard/17235-source-qonto

4f60249

Support production and mocked server endpoints

76e97aa

leilaballouard force-pushed the leilaballouard/17235-source-qonto branch from 2f4ea09 to 76e97aa Compare November 7, 2022 15:12

vincentkoc added 3 commits November 9, 2022 00:11

finalise mocked api calls

bcf53c7

Update source_definitions.yaml

fa5f18d

lint

c7480c8

Merge branch 'master' into leilaballouard/17235-source-qonto

5c76211

vincentkoc approved these changes Nov 8, 2022

View reviewed changes

vincentkoc added 2 commits November 11, 2022 07:43

Merge branch 'master' into leilaballouard/17235-source-qonto

d81a240

Merge branch 'master' into leilaballouard/17235-source-qonto

93b43af

Merge branch 'master' into leilaballouard/17235-source-qonto

5a4ccd6

auto-bump connector version

858b735

sajarin merged commit d59275d into airbytehq:master Nov 14, 2022

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

rach-r mentioned this pull request Nov 17, 2022

🎉 New Source: Apple Search Ads [low-code CDK] #19557

Merged

11 tasks

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Qonto [python cdk] #17452

🎉 New Source: Qonto [python cdk] #17452

leilaballouard commented Sep 30, 2022 •

edited

Loading

CLAassistant commented Sep 30, 2022 •

edited

Loading

leilaballouard commented Oct 4, 2022

leilaballouard commented Oct 25, 2022

sajarin commented Oct 25, 2022

vincentkoc commented Nov 3, 2022

vincentkoc left a comment

vincentkoc left a comment

vincentkoc commented Nov 8, 2022

vincentkoc commented Nov 8, 2022

leilaballouard commented Nov 8, 2022

vincentkoc commented Nov 8, 2022

vincentkoc commented Nov 10, 2022 •

edited by github-actions bot

Loading

vincentkoc commented Nov 10, 2022 •

edited by github-actions bot

Loading

🎉 New Source: Qonto [python cdk] #17452

🎉 New Source: Qonto [python cdk] #17452

Conversation

leilaballouard commented Sep 30, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Sep 30, 2022 • edited Loading

leilaballouard commented Oct 4, 2022

leilaballouard commented Oct 25, 2022

sajarin commented Oct 25, 2022

vincentkoc commented Nov 3, 2022

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc commented Nov 8, 2022

vincentkoc commented Nov 8, 2022

leilaballouard commented Nov 8, 2022

vincentkoc commented Nov 8, 2022

vincentkoc commented Nov 10, 2022 • edited by github-actions bot Loading

Build Passed

vincentkoc commented Nov 10, 2022 • edited by github-actions bot Loading

leilaballouard commented Sep 30, 2022 •

edited

Loading

CLAassistant commented Sep 30, 2022 •

edited

Loading

vincentkoc commented Nov 10, 2022 •

edited by github-actions bot

Loading

vincentkoc commented Nov 10, 2022 •

edited by github-actions bot

Loading