-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛Destination-Bigquery: Added an explicit error message if sync fails due to a config issue #21144
🐛Destination-Bigquery: Added an explicit error message if sync fails due to a config issue #21144
Conversation
… fails due to a config issue
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (2)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-bigquery |
1.2.11 |
✅ | ✅ |
destination-bigquery-denormalized |
1.2.11 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
👀 Other Modules (1)
- base-normalization
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
final TableDataWriteChannel writer = bigQuery.writer(job, writeChannelConfiguration); | ||
final TableDataWriteChannel writer; | ||
|
||
try{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: space between try
and {
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated, thanks. Please ignore the code-style issues for now. Once everything is approved I usually run the gradle's "format" command along with version bumping. So such issues would be automatically fixed.
|
||
try{ | ||
writer = bigQuery.writer(job, writeChannelConfiguration); | ||
}catch (final BigQueryException e){ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: spaces
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated, thanks. Please ignore the code-style issues for now. Once everything is approved I usually run the gradle's "format" command along with version bumping. So such issues would be automatically fixed.
try{ | ||
writer = bigQuery.writer(job, writeChannelConfiguration); | ||
}catch (final BigQueryException e){ | ||
throw new ConfigErrorException(CONFIG_ERROR_MSG + e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a way to make this catch
clause more specific, so that only permission and 404 errors result in ConfigErrorException
, but other problems (whatever they may be) are not translated into ConfigErrorException
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To tell the truth, this is what I did from the very beginning - i.e. added a catch to check the only exceptions with 404 in message. But after playing a bit more I noticed that all other possible issues were also related to config issues. So then updated code to catch everything as ConfigError. At this part of code, we just create a new connection. So whatever will go wrong - will be more or less related to the config issue. Should I anyway update it to catch the 404 only? Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally, we'd catch specific error codes and provide error messages specific to those codes. Here is the list of error codes (https://cloud.google.com/bigquery/docs/error-messages). I don't think we can run into all of these here. We can probably assume that we will mostly run into 404
and 403
which we should report as config errors. Occasionally, we might run into 500
and 503
, which we should not report as config errors and we should provide a different error message when we run into 500
and 503
@@ -30,6 +32,26 @@ | |||
|
|||
public class BigQueryUploaderFactory { | |||
|
|||
private static final String CONFIG_ERROR_MSG = """ | |||
\n | |||
******************************************************************************************** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm..., so I like the formatting in the expanded log view, but I think I would prefer removing the long line of stars so that the message is easier to read in the non-expanded view (judging by the screenshot attached to the PR).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed stars, thanks. Previously just added it to make the error message more noticeable and fancy :)
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
/publish connector=connectors/destination-bigquery
if you have connectors that successfully published but failed definition generation, follow step 4 here |
Airbyte Code Coverage
|
/publish connector=connectors/destination-bigquery-denormalized
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…due to a config issue (#21144) * [19998] Destination-Bigquery: Added an explicit error message in sync fails due to a config issue
* api changes for writing discover catalog * api changes * format * worker change 1 * change return type of the API to return catalogId * worker to call api * typo * 🎉 Source GoogleSheets - migrated SAT to strictness level (#21399) * migrated SAT to strictness level * fixed expected records * revert file from another source * changed extension to txt * changed extension to txt * 🐛Destination-Bigquery: Added an explicit error message if sync fails due to a config issue (#21144) * [19998] Destination-Bigquery: Added an explicit error message in sync fails due to a config issue * ci-connector-ops: split workflows(#21474) * CI: nightly build alpha sources and destinations (#21562) * Revert "Change main class in strict-encrypt destination and bump versions on both destinations to keep them in sync (#21509)" (#21567) This reverts commit 1d202d1. * Fixes webhook updating logic (#21519) * ci_credentials: disable tooling test run by tox (#21580) * disable tox * rename steps * revert changes on experimental workflow * do not install tox * Revert "CI: nightly build alpha sources and destinations (#21562)" (#21589) This reverts commit 61f88f3. * Security update of default docker images (#21407) Because there is a lot of CVEs in those releases. Co-authored-by: Topher Lubaway <[email protected]> * 📝 add docs for how to add normalization (#21563) * add docs * add schema link * update based on feedback * 🪟 🚦 E2E tests: clean up matchers (#20887) * improve serviceTypeDropdownOption selector * add test ids to PathPopout component(s) * add unique id's to table dropdowns * extend submitButtonClick to support optional click options * update dropdown(pathPopout) matchers * add test-id to Overlay component * remove redundant function brackets * revert changes onSubmit button click * fix dropDown overlay issue * move all duplicated intercepters to beforeEach * add test id's to Connections, Sources and Destinations tables * add table helper functions * update source page actions * intercepter fixes * update createTestConnection function with optional replication settings * remove extra Connection name check * replace "cypress-postgres" with "pg-promise" npm package * update cypress config * Revert "update createTestConnection function with optional replication settings" This reverts commit 8e47c78. * Revert "remove extra Connection name check" This reverts commit dfb19c7. * replace openSourceDestinationFromGrid with specific selector * replace openSourceDestinationFromGrid with specific selector * turn on test * add test-id's * fix selectors * update test * update test snapshots * fix lost data-testid after resolve merge conflicts * remove extra check * move clickOnCellInTable helper to common.ts file * remove empty line and comments * fix dropdownType * replace partial string check with exact * extract interceptors and waiters to separate file * fix selector for predefined PK * fix selector * add comment regarding dropdown * 🪟 🎨 [Free connectors] Update modal copy (#21600) * move start/end time options out of optional block (#21541) * lingering fix * reflecting api changes * test fix * worker to call api to do discover work * recovered deleted html * self review * more converters refactor * fix connector test * fix test * fix * fix integration test * add unit test for converter * static fix * api client needs to have a timeout in case request does not get responded --------- Co-authored-by: midavadim <[email protected]> Co-authored-by: Eugene <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: Yatsuk Bogdan <[email protected]> Co-authored-by: Hervé Commowick <[email protected]> Co-authored-by: Topher Lubaway <[email protected]> Co-authored-by: Pedro S. Lopez <[email protected]> Co-authored-by: Vladimir <[email protected]> Co-authored-by: Joey Marshment-Howell <[email protected]> Co-authored-by: Lake Mossman <[email protected]>
* api changes for writing discover catalog * api changes * format * worker change 1 * change return type of the API to return catalogId * worker to call api * typo * 🎉 Source GoogleSheets - migrated SAT to strictness level (#21399) * migrated SAT to strictness level * fixed expected records * revert file from another source * changed extension to txt * changed extension to txt * 🐛Destination-Bigquery: Added an explicit error message if sync fails due to a config issue (#21144) * [19998] Destination-Bigquery: Added an explicit error message in sync fails due to a config issue * ci-connector-ops: split workflows(#21474) * CI: nightly build alpha sources and destinations (#21562) * Revert "Change main class in strict-encrypt destination and bump versions on both destinations to keep them in sync (#21509)" (#21567) This reverts commit 1d202d1. * Fixes webhook updating logic (#21519) * ci_credentials: disable tooling test run by tox (#21580) * disable tox * rename steps * revert changes on experimental workflow * do not install tox * Revert "CI: nightly build alpha sources and destinations (#21562)" (#21589) This reverts commit 61f88f3. * Security update of default docker images (#21407) Because there is a lot of CVEs in those releases. Co-authored-by: Topher Lubaway <[email protected]> * 📝 add docs for how to add normalization (#21563) * add docs * add schema link * update based on feedback * 🪟 🚦 E2E tests: clean up matchers (#20887) * improve serviceTypeDropdownOption selector * add test ids to PathPopout component(s) * add unique id's to table dropdowns * extend submitButtonClick to support optional click options * update dropdown(pathPopout) matchers * add test-id to Overlay component * remove redundant function brackets * revert changes onSubmit button click * fix dropDown overlay issue * move all duplicated intercepters to beforeEach * add test id's to Connections, Sources and Destinations tables * add table helper functions * update source page actions * intercepter fixes * update createTestConnection function with optional replication settings * remove extra Connection name check * replace "cypress-postgres" with "pg-promise" npm package * update cypress config * Revert "update createTestConnection function with optional replication settings" This reverts commit 8e47c78. * Revert "remove extra Connection name check" This reverts commit dfb19c7. * replace openSourceDestinationFromGrid with specific selector * replace openSourceDestinationFromGrid with specific selector * turn on test * add test-id's * fix selectors * update test * update test snapshots * fix lost data-testid after resolve merge conflicts * remove extra check * move clickOnCellInTable helper to common.ts file * remove empty line and comments * fix dropdownType * replace partial string check with exact * extract interceptors and waiters to separate file * fix selector for predefined PK * fix selector * add comment regarding dropdown * 🪟 🎨 [Free connectors] Update modal copy (#21600) * move start/end time options out of optional block (#21541) * lingering fix * reflecting api changes * test fix * reset to master * routing changes * remove unexpected merge * resolve dependency micronaut * resolve dependency * format * fix test * rename and refactor location of test files * test * redo the routing service * add test to config repository * query workspace not connection for discover/check * remove unused bean --------- Co-authored-by: midavadim <[email protected]> Co-authored-by: Eugene <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Greg Solovyev <[email protected]> Co-authored-by: Yatsuk Bogdan <[email protected]> Co-authored-by: Hervé Commowick <[email protected]> Co-authored-by: Topher Lubaway <[email protected]> Co-authored-by: Pedro S. Lopez <[email protected]> Co-authored-by: Vladimir <[email protected]> Co-authored-by: Joey Marshment-Howell <[email protected]> Co-authored-by: Lake Mossman <[email protected]>
What
BigQuery destination runs into 404 error in
STANDARD
mode. This usually happens in "Mirror source structure" replication mode. We don't know the destination schema name at the check connection stage so can't test it.How
This PR just adds an explicit log message with possible advice on how to fix the error in additionally to already existing logs. But the issue itself would be fixed along the way once this one #20561 is fixed
Tested locally:
🚨 User Impact 🚨
No breaking changes expected.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.