Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source File: specify pyxlsb as engine for Excel file parsing #15568

Merged
merged 6 commits into from
Aug 26, 2022
Merged

Source File: specify pyxlsb as engine for Excel file parsing #15568

merged 6 commits into from
Aug 26, 2022

Conversation

arsenlosenko
Copy link
Contributor

@arsenlosenko arsenlosenko commented Aug 11, 2022 •
edited
Loading

Resolving:
#5622

@arsenlosenko arsenlosenko self-assigned this Aug 11, 2022
@CLAassistant
Copy link

CLAassistant commented Aug 11, 2022 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 11, 2022
@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 11, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2842132710
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2842132710
Python tests coverage:

	 Name                                                     Stmts   Miss  Cover   Missing
	 --------------------------------------------------------------------------------------
	 source_acceptance_test/base.py                              10      4    60%   15-18
	 source_acceptance_test/config.py                            82      6    93%   75-77, 81-83
	 source_acceptance_test/conftest.py                         153    153     0%   6-263
	 source_acceptance_test/plugin.py                            48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py                  320    105    67%   40, 60-68, 73-80, 84-85, 89-90, 174, 216-233, 240-245, 251, 284-289, 314-321, 361-363, 366, 426-435, 464-465, 471, 474, 507-517, 530-555, 560-564
	 source_acceptance_test/tests/test_full_refresh.py           52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py           121     25    79%   21-23, 29-31, 36-43, 48-61, 208-214
	 source_acceptance_test/utils/asserts.py                     37      2    95%   57-58
	 source_acceptance_test/utils/backward_compatibility.py      77      8    90%   66-73, 93, 95, 105, 152-153
	 source_acceptance_test/utils/common.py                      77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                     62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py           110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py         105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 --------------------------------------------------------------------------------------
	 TOTAL                                                     1266    454    64%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       259     41    84%
source_file/source.py        50     35    30%
---------------------------------------------
TOTAL                       311     76    76%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/source.py        50     27    46%
source_file/client.py       259    161    38%
---------------------------------------------
TOTAL                       311    188    40%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 25 passed, 1 skipped in 60.13s (0:01:00) ===================

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2846695838
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2846695838
Python tests coverage:

Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       265     41    85%
source_file/source.py        50     35    30%
---------------------------------------------
TOTAL                       317     76    76%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       265    126    52%
source_file/source.py        50     27    46%
---------------------------------------------
TOTAL                       317    153    52%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-214
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1321    463    65%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 26 passed, 1 skipped in 64.56s (0:01:04) ===================

@arsenlosenko arsenlosenko linked an issue Aug 12, 2022 that may be closed by this pull request
Document & add tests for XLSB format in source-file #5622
Closed
@arsenlosenko arsenlosenko marked this pull request as ready for review August 12, 2022 12:27
bazarnov
bazarnov approved these changes Aug 12, 2022
View reviewed changes
Copy link
Collaborator

@bazarnov bazarnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have the unit_test to cover this Excel file parser?

@arsenlosenko arsenlosenko force-pushed the arsenlosenko/5622-xlsb-format-support-and-tests branch from ae7401c to 304ab63 Compare August 26, 2022 11:34
@github-actions github-actions bot removed the area/documentation Improvements or additions to documentation label Aug 26, 2022
@arsenlosenko
Copy link
Contributor Author

/test connector=connectors/source-file

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Aug 26, 2022
@lazebnyi
Copy link
Collaborator

lazebnyi commented Aug 26, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2933469312
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2933469312
🐛 https://gradle.com/s/phioqqbklmiza

Build Failed

Test summary info:

Could not find result summary

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2933605683
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/2933605683
Python tests coverage:

Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       273     40    85%
source_file/source.py        51     27    47%
---------------------------------------------
TOTAL                       326     67    79%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/source.py        51      0   100%
source_file/__init__.py       2      0   100%
source_file/client.py       273     34    88%
---------------------------------------------
TOTAL                       326     34    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 26 passed, 1 skipped in 64.57s (0:01:04) ===================

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2933708105


Connector Did it publish? Were definitions generated?
connectors/source-file ❌ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file-secure

🕑 Publishing the following connectors:
connectors/source-file-secure
https://github.com/airbytehq/airbyte/actions/runs/2933712304


Connector Did it publish? Were definitions generated?
connectors/source-file-secure ❌ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/2933782764


Connector Did it publish? Were definitions generated?
connectors/source-file ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file-secure

🕑 Publishing the following connectors:
connectors/source-file-secure
https://github.com/airbytehq/airbyte/actions/runs/2933917231


Connector Did it publish? Were definitions generated?
connectors/source-file-secure ❌ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@arsenlosenko arsenlosenko force-pushed the arsenlosenko/5622-xlsb-format-support-and-tests branch from 5751823 to 7b95f5f Compare August 26, 2022 13:28
@arsenlosenko
Copy link
Contributor Author

arsenlosenko commented Aug 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-file-secure

🕑 Publishing the following connectors:
connectors/source-file-secure
https://github.com/airbytehq/airbyte/actions/runs/2934045331


Connector Did it publish? Were definitions generated?
connectors/source-file-secure ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@arsenlosenko arsenlosenko merged commit 35257a0 into master Aug 26, 2022
@arsenlosenko arsenlosenko deleted the arsenlosenko/5622-xlsb-format-support-and-tests branch August 26, 2022 14:34
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Aug 26, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bazarnov bazarnov bazarnov approved these changes

@lazebnyi lazebnyi lazebnyi approved these changes

@grubberr grubberr Awaiting requested review from grubberr

Assignees

@arsenlosenko arsenlosenko

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/file connectors/source/file-secure
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Document & add tests for XLSB format in source-file
5 participants
@arsenlosenko @CLAassistant @lazebnyi @bazarnov @octavia-squidington-iii