🎉 Source Github: IssueReactions - re-implemented using GraphQL #21457

grubberr · 2023-01-16T12:12:52Z

Signed-off-by: Sergey Chvalyuk [email protected]

What

Trying to partially improve this oncall https://github.com/airbytehq/oncall/issues/1303

Performance for stream IssueReactions

Type	HTTP requests	Sync time	Number of records
GraphQL	102	102 seconds	2400
REST API	18056	3022 seconds	2400

In my case real sync time of REST API was about 14000 seconds because about 11000 seconds were sleeping. I have noted only pure working time.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2023-01-17T08:12:42Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3937272010
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3937272010
🐛 https://gradle.com/s/moidw7exrzu5i

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============= 2 failed, 38 passed, 1 skipped in 1390.97s (0:23:10) =============

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2023-01-17T09:52:09Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3938027690
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3938027690
🐛 https://gradle.com/s/vnjra2cgmph4u

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============= 1 failed, 39 passed, 1 skipped in 329.50s (0:05:29) ==============

grubberr · 2023-01-17T10:33:16Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3938365288
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3938365288
Python tests coverage:

Name                             Stmts   Miss  Cover
----------------------------------------------------
source_github/utils.py              14      0   100%
source_github/github_schema.py    8807      0   100%
source_github/__init__.py            2      0   100%
source_github/streams.py           849     90    89%
source_github/graphql.py           167     22    87%
source_github/source.py            106     25    76%
----------------------------------------------------
TOTAL                             9945    137    99%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================== 40 passed, 1 skipped in 351.74s (0:05:51) ===================

grubberr · 2023-01-17T23:35:18Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3944239017
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3944239017
🐛 https://gradle.com/s/l6grd3zwk24wy

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream issue...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============= 1 failed, 39 passed, 1 skipped in 325.35s (0:05:25) ==============

evantahler

Awesome work - one question below about the state format change

evantahler · 2023-01-17T23:50:44Z

airbyte-integrations/connectors/source-github/acceptance-test-config.yml

-          issue_reactions: ["airbytehq/integration-test", "11", "created_at"]
+          issue_reactions: ["airbytehq/integration-test", "created_at"]


Does this change imply that the format of state changed for an incremental stream? If so, that's probably a major (breaking) semver change

yes state format changed, state object simplified
Can you advice how to be in such cases? Are you proposing to increase version number 0.3.11 -> 0.4.0?

Thank you

To answer the question, the user experience needs to be considered:

If the sync will continue as it has in the past (maintain it's incremental position) - patch change (no user intervention needeed)

If the sync will succeed, but require a full import again because the state format is changed - minor change (no user intervention needed, but a slow sync will happen)

If the sync will fail until the user takes action or the records returned are changed - major change.

I think we are in category 2, like you suggest - is that correct?

answer - 2
because the state format is changed
it will re-read all issue-reaction records again from "start_date"

Thanks! Then as you suggest, this should be v0.4.0 of the connector.
We are trying to get good at semver versioning of connectors from the user's point of view so we can automate warnings and breaking change announcements in the app.

I have bumped to 0.4.0

@evantahler can you please look can we proceed with this PR?

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2023-01-19T22:11:34Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3962814551
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3962814551
🐛 https://gradle.com/s/klvhgmhhjvqiq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream issue...
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
============= 2 failed, 38 passed, 1 skipped in 1573.37s (0:26:13) =============

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2023-01-20T00:25:55Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3963584092
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/3963584092
Python tests coverage:

Name                             Stmts   Miss  Cover
----------------------------------------------------
source_github/utils.py              14      0   100%
source_github/github_schema.py    8807      0   100%
source_github/__init__.py            2      0   100%
source_github/streams.py           851     90    89%
source_github/graphql.py           167     22    87%
source_github/source.py            106     25    76%
----------------------------------------------------
TOTAL                             9947    137    99%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================== 40 passed, 1 skipped in 338.22s (0:05:38) ===================

grubberr · 2023-01-26T22:49:45Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/4019831738
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/4019831738
Python tests coverage:

Name                             Stmts   Miss  Cover
----------------------------------------------------
source_github/utils.py              14      0   100%
source_github/github_schema.py    8807      0   100%
source_github/__init__.py            2      0   100%
source_github/streams.py           851     90    89%
source_github/graphql.py           167     22    87%
source_github/source.py            106     25    76%
----------------------------------------------------
TOTAL                             9947    137    99%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================== 40 passed, 1 skipped in 341.28s (0:05:41) ===================

grubberr · 2023-01-26T23:28:19Z

/publish connector=connectors/source-github

🕑 Publishing the following connectors:
connectors/source-github
https://github.com/airbytehq/airbyte/actions/runs/4020063176

Connector	Did it publish?	Were definitions generated?
connectors/source-github	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-01-27T00:23:33Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24%	❌

IssueReactions use GraphQL API

559a6e5

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr self-assigned this Jan 16, 2023

octavia-squidington-iv added area/connectors Connector related issues connectors/source/github labels Jan 16, 2023

grubberr added 7 commits January 16, 2023 16:09

Merge branch 'master' into grubberr/issue-reactions-graphql

e905862

_get_reactions_from_issue added

de8a1c9

Signed-off-by: Sergey Chvalyuk <[email protected]>

user.json -> user_graphql.json

13bb21f

Signed-off-by: Sergey Chvalyuk <[email protected]>

simplify code, use page_size

69b58e4

Signed-off-by: Sergey Chvalyuk <[email protected]>

Merge branch 'master' into grubberr/issue-reactions-graphql

6106723

Merge branch 'master' into grubberr/issue-reactions-graphql

8051874

github.md updated

1806b54

Signed-off-by: Sergey Chvalyuk <[email protected]>

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 17, 2023

grubberr requested a review from evantahler January 17, 2023 08:19

simplify state

2adefe8

Signed-off-by: Sergey Chvalyuk <[email protected]>

Merge branch 'master' into grubberr/issue-reactions-graphql

f7e33b3

grubberr requested review from alafanechere, artem1205 and roman-yermilov-gl January 17, 2023 15:34

Merge branch 'master' into grubberr/issue-reactions-graphql

5183348

evantahler reviewed Jan 17, 2023

View reviewed changes

Merge branch 'master' into grubberr/issue-reactions-graphql

514c9fc

Signed-off-by: Sergey Chvalyuk <[email protected]>

octavia-squidington-iv added the area/frontend Related to the Airbyte webapp label Jan 19, 2023

revert manifestToBuilderForm.ts to master

0047ced

Signed-off-by: Sergey Chvalyuk <[email protected]>

octavia-squidington-iv removed the area/frontend Related to the Airbyte webapp label Jan 19, 2023

Merge branch 'master' into grubberr/issue-reactions-graphql

8a74f90

issue_reactions fixed

35547e7

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr requested a review from brianjlai January 23, 2023 18:01

evantahler approved these changes Jan 26, 2023

View reviewed changes

Merge branch 'master' into grubberr/issue-reactions-graphql

1997f74

auto-bump connector version

09cf60f

octavia-squidington-iii temporarily deployed to more-secrets January 27, 2023 00:14 — with GitHub Actions Inactive

grubberr merged commit 65fc1bc into master Jan 27, 2023

grubberr deleted the grubberr/issue-reactions-graphql branch January 27, 2023 00:45

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21954

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Github: IssueReactions - re-implemented using GraphQL #21457

🎉 Source Github: IssueReactions - re-implemented using GraphQL #21457

grubberr commented Jan 16, 2023 •

edited

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

evantahler left a comment

evantahler Jan 17, 2023

grubberr Jan 19, 2023

evantahler Jan 19, 2023

grubberr Jan 19, 2023

evantahler Jan 19, 2023

grubberr Jan 20, 2023

grubberr Jan 26, 2023

grubberr commented Jan 19, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 26, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 26, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 27, 2023

		issue_reactions: ["airbytehq/integration-test", "11", "created_at"]
		issue_reactions: ["airbytehq/integration-test", "created_at"]

🎉 Source Github: IssueReactions - re-implemented using GraphQL #21457

🎉 Source Github: IssueReactions - re-implemented using GraphQL #21457

Conversation

grubberr commented Jan 16, 2023 • edited Loading

What

Performance for stream IssueReactions

Pre-merge Checklist

Community member or Airbyter

Airbyter

grubberr commented Jan 17, 2023 • edited by github-actions bot Loading

Build Failed

grubberr commented Jan 17, 2023 • edited by github-actions bot Loading

Build Failed

grubberr commented Jan 17, 2023 • edited by github-actions bot Loading

Build Passed

grubberr commented Jan 17, 2023 • edited by github-actions bot Loading

Build Failed

evantahler left a comment

Choose a reason for hiding this comment

evantahler Jan 17, 2023

Choose a reason for hiding this comment

grubberr Jan 19, 2023

Choose a reason for hiding this comment

evantahler Jan 19, 2023

Choose a reason for hiding this comment

grubberr Jan 19, 2023

Choose a reason for hiding this comment

evantahler Jan 19, 2023

Choose a reason for hiding this comment

grubberr Jan 20, 2023

Choose a reason for hiding this comment

grubberr Jan 26, 2023

Choose a reason for hiding this comment

grubberr commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

grubberr commented Jan 20, 2023 • edited by github-actions bot Loading

Build Passed

grubberr commented Jan 26, 2023 • edited by github-actions bot Loading

Build Passed

grubberr commented Jan 26, 2023 • edited by github-actions bot Loading

github-actions bot commented Jan 27, 2023

Airbyte Code Coverage

grubberr commented Jan 16, 2023 •

edited

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 17, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 19, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 20, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 26, 2023 •

edited by github-actions bot

Loading

grubberr commented Jan 26, 2023 •

edited by github-actions bot

Loading