BigQuery source: "Could not connect with provided configuration. Error: null"

[PS-istarostenko]

Enviroment

• Airbyte version: 0.29.4-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: source-bigquery:0.1.1
• Severity: Critical
• Step where error happened: Setup new connection

Current Behavior

When setting up BigQuery as a new source I get the following error:

Could not connect with provided configuration. Error: null

The same configuration worked just fine when used to set up BigQuery as a destination

Expected Behavior

Source connected successfully

Logs

2021-08-11 04:27:50 INFO () TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.29.4-alpha
2021-08-11 04:27:50 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-bigquery:0.1.1 exists...
2021-08-11 04:27:50 INFO () LineGobbler(voidCall):85 - airbyte/source-bigquery:0.1.1 was found locally.
2021-08-11 04:27:50 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4bf35597-e260-4815-8173-21ef35323e20/0 --network host --log-driver none airbyte/source-bigquery:0.1.1 check --config source_config.json
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.s.b.BigQuerySource(main):183 - {} - starting source: class io.airbyte.integrations.source.bigquery.BigQuerySource
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.source.bigquery.BigQuerySource
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {check=null, config=source_config.json}
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: CHECK
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-08-11 04:27:52 INFO i.a.i.s.r.AbstractRelationalDbSource(check):154 - {} - Exception while checking connection: 
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - java.lang.NullPointerException: null
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 	at io.airbyte.integrations.source.bigquery.BigQuerySource.toDatabaseConfig(BigQuerySource.java:73) ~[io.airbyte.airbyte-integrations.connectors-source-bigquery-0.28.1-alpha.jar:?]
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 	at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.createDatabaseInternal(AbstractRelationalDbSource.java:490) ~[io.airbyte.airbyte-integrations.connectors-source-relational-db-0.28.1-alpha.jar:?]
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 	at io.airbyte.integrations.source.relationaldb.AbstractRelationalDbSource.check(AbstractRelationalDbSource.java:147) [io.airbyte.airbyte-integrations.connectors-source-relational-db-0.28.1-alpha.jar:?]
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:122) [io.airbyte.airbyte-integrations.bases-base-java-0.28.1-alpha.jar:?]
2021-08-11 04:27:52 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 	at io.airbyte.integrations.source.bigquery.BigQuerySource.main(BigQuerySource.java:184) [io.airbyte.airbyte-integrations.connectors-source-bigquery-0.28.1-alpha.jar:?]
2021-08-11 04:27:52 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...

[69mb]

Just ran into the same. NPE is from this line since Default Dataset ID is an optional parameter. Should be changed to .get(CONFIG_DATASET_ID).asText(null)?

[prasrvenkat]

Hitting the same problem, would love to have that PR #6051 reviewed/merged (Thanks @69mb ).
Our use case is potentially configuring to pull from multiple datasets and forcing a dataset on config limits us from doing that (even though its marked optional, empty does not work screenshot attached)