🎉 New Source: Alpha Vantage #18320

isaacharrisholt · 2022-10-21T21:05:42Z

What

Alpha Vantage is one of the most popular stock price APIs, and it's completely free, making it a crucial part of many FinTech business' stacks. This PR adds an Alpha Vantage source to Airbyte.

It also contains an extended DpathExtractor that supports a new data format. This is explained in the docstring.

Hacktoberfest: airbytehq/connector-contest#222

How

Adds a source with the following streams:

time_series_intraday
time_series_daily
time_series_daily_adjusted
time_series_weekly
time_series_weekly_adjusted
time_series_monthly
time_series_monthly_adjusted

AV always returns the full history of an asset, so I have not implemented an incremental read for this source.

🚨 User Impact 🚨

No changes to existing code.

Pre-merge Checklist

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Integration

Acceptance

marcosmarxm

Looks good, it is missing the documentation can you add it? A few comments, but overal the implementatino is great.

airbyte-integrations/connectors/source-alpha-vantage/source_alpha_vantage/spec.yaml

airbyte-integrations/connectors/source-alpha-vantage/unit_tests/test_object_dpath_extractor.py

airbyte-integrations/connectors/source-alpha-vantage/integration_tests/catalog.json

marcosmarxm

Some comments and it is ready to go! :)
Are you able to run ./gradlew airbyte-integrations:connectors:source-alpha-vantage:integrationTest my tests are failing.

airbyte-integrations/connectors/source-alpha-vantage/integration_tests/sample_config.json

airbyte-integrations/connectors/source-alpha-vantage/integration_tests/invalid_config.json

marcosmarxm

Using the hardcode value the connector works. Let's see what is causing the issue with the options['function'] after that this is ready to be merged.

marcosmarxm · 2022-10-28T11:24:34Z

airbyte-integrations/connectors/source-alpha-vantage/source_alpha_vantage/alpha_vantage.yaml

+      request_parameters:
+        apikey: "{{ config['api_key'] }}"
+        symbol: "{{ config['symbol'] }}"
+        function: "{{ options['function'] }}"
+        datatype: json
+        outputsize: "{{ config['outputsize'] }}"
+        interval: "{{ config['interval'] }}"
+        adjusted: "{{ config['adjusted'] }}"


Small detail @isaacharrisholt.

@girarda I saw one bug with this connector:
https://www.alphavantage.co/query?apikey=****&symbol=IBM&datatype=json&outputsize=compact&interval=60min
The request parameter function is not added to the url. If I change the parameter to a hardcoded string it works fine, example TIME_SERIES_DAILY. Any idea?

@marcosmarxm I think this is an issue that was solved recently. You might need to reinstall the CDK with the newest version.

I'll give a try, maybe I was using the wrong pyenv =p

marcosmarxm · 2022-11-01T13:30:24Z

/test connector=connectors/source-alpha-vantage

🕑 connectors/source-alpha-vantage https://github.com/airbytehq/airbyte/actions/runs/3369781521
❌ connectors/source-alpha-vantage https://github.com/airbytehq/airbyte/actions/runs/3369781521
🐛 https://gradle.com/s/wyeh3wxvnr4la

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_core.py::TestSpec::test_config_match_spec[inputs0] - FileNotFoundE...
ERROR test_core.py::TestConnection::test_check[inputs0] - FileNotFoundError: ...
ERROR test_core.py::TestDiscovery::test_discover[inputs0] - FileNotFoundError...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - Fil...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestBasicRead::test_read[inputs0] - FileNotFoundError: [E...
ERROR test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs0]
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
================== 11 passed, 3 skipped, 13 errors in 12.68s ===================

marcosmarxm · 2022-11-01T16:11:17Z

/test connector=connectors/source-alpha-vantage

🕑 connectors/source-alpha-vantage https://github.com/airbytehq/airbyte/actions/runs/3370960427
✅ connectors/source-alpha-vantage https://github.com/airbytehq/airbyte/actions/runs/3370960427
Python tests coverage:

Name                                             Stmts   Miss  Cover
--------------------------------------------------------------------
source_alpha_vantage/__init__.py                     2      0   100%
source_alpha_vantage/object_dpath_extractor.py      31      2    94%
source_alpha_vantage/source.py                       4      1    75%
--------------------------------------------------------------------
TOTAL                                               37      3    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 377-379, 382, 442-451, 480-481, 487, 490, 523-533, 546-571, 576-580
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
======================== 24 passed, 3 skipped in 24.13s ========================

marcosmarxm · 2022-11-01T16:28:25Z

/publish connector=connectors/source-alpha-vantage

🕑 Publishing the following connectors:
connectors/source-alpha-vantage
https://github.com/airbytehq/airbyte/actions/runs/3371080658

Connector	Did it publish?	Were definitions generated?
connectors/source-alpha-vantage	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm

thanks @isaacharrisholt

* master: (38 commits) New Source: Gridly (#18342) 🎉 New Source: Alpha Vantage (#18320) ci_integration_test.sh: cut GITHUB_STEP_SUMMARY (#18895) 🎉 New Source: Datadog [python cdk] (#18150) Hide Reject all button in consent dialog (#18596) feat: add doc url to track event (#18690) fix: install java in oss catalog deploy action (#18887) [CI] Speed up check_images_exist (#18873) Extract open API (#18879) Remove unused interfaces (#18880) add action for deploying oss connector catalog to GCS (#18633) feat: generate full connector catalog json (#18562) Add unsupported_protocol_version column to Connection (#18876) Extract OAuth API (#18818) update images to have non-transparent background (#18874) DiscoverSchema endpoints calculates diff and breaking change (#18571) Validate protocol version on connector update (#18639) Bmoric/extract notification api (#18812) Show version and changelog status for affected connectors (#18845) Bmoric/extract logs api (#18621) ...

isaacharrisholt and others added 4 commits October 13, 2022 18:50

Initial commit

f34699e

Merge branch 'master' into alpha_vantage

da05fb4

Tidy and add custom dpath extractor

48df82e

Get acceptance tests passing

804efe2

github-actions bot added the area/connectors Connector related issues label Oct 21, 2022

isaacharrisholt added 3 commits October 23, 2022 10:34

Improve dict-checking for ObjectDpathExtractor

d364eea

Improve dict-checking for ObjectDpathExtractor

6aca05f

Add more tests to ObjectDpathExtractor

7c3092e

marcosmarxm added community hacktober labels Oct 24, 2022

marcosmarxm requested changes Oct 24, 2022

View reviewed changes

bleonard added bounty connectors/source/alpha-vantage labels Oct 25, 2022

Add docs

e28a43b

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 25, 2022

isaacharrisholt added 2 commits October 25, 2022 09:46

Address comments

6db283d

Update docs

bcbe895

marcosmarxm requested changes Oct 26, 2022

View reviewed changes

airbyte-integrations/connectors/source-alpha-vantage/integration_tests/sample_config.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-alpha-vantage/integration_tests/invalid_config.json Outdated Show resolved Hide resolved

Add sample and invalid configs

d3240c2

marcosmarxm reviewed Oct 28, 2022

View reviewed changes

RealChrisSean mentioned this pull request Oct 29, 2022

🎉 New Source: Alpha Vantage #18320 airbytehq/connector-contest#222

Closed

format

61d6738

add alpha vantage to source def

6fd936b

octavia-squidington-iii and others added 2 commits November 1, 2022 16:49

auto-bump connector version

6ac1a3d

Merge branch 'master' into alpha_vantage

2abf7e6

marcosmarxm approved these changes Nov 3, 2022

View reviewed changes

marcosmarxm merged commit 606534d into airbytehq:master Nov 3, 2022

RealChrisSean added the low-code label Nov 15, 2022

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Alpha Vantage #18320

🎉 New Source: Alpha Vantage #18320

isaacharrisholt commented Oct 21, 2022 •

edited by RealChrisSean

Loading

marcosmarxm left a comment

marcosmarxm left a comment

marcosmarxm left a comment

marcosmarxm Oct 28, 2022

isaacharrisholt Oct 28, 2022

marcosmarxm Oct 28, 2022

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading

marcosmarxm left a comment

🎉 New Source: Alpha Vantage #18320

🎉 New Source: Alpha Vantage #18320

Conversation

isaacharrisholt commented Oct 21, 2022 • edited by RealChrisSean Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Oct 28, 2022

Choose a reason for hiding this comment

isaacharrisholt Oct 28, 2022

Choose a reason for hiding this comment

marcosmarxm Oct 28, 2022

Choose a reason for hiding this comment

marcosmarxm commented Nov 1, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Nov 1, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Nov 1, 2022 • edited by github-actions bot Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

isaacharrisholt commented Oct 21, 2022 •

edited by RealChrisSean

Loading

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 1, 2022 •

edited by github-actions bot

Loading