JDBC sources: Improve error message when an invalid cursor column is selected

[bleonard]

When a non-sortable column is used as a cursor, we currently produce an error message that isn't super useful (<column type> can't be used as a cursor). Ideally, this message should include the table and column names, to assist users in tracking down which stream is causing the problem.

---

"java.lang.IllegalArgumentException: OTHER is not supported.
 io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1"

Something interesting on this cloud account.
More details: https://docs.google.com/spreadsheets/d/1ja8deCtFHtFVP3QrWELsMpVCMfPB1kz0QfLG1szns1I/edit#gid=408762974

[sentry-io]

Sentry issue: CONNECTOR-INCIDENT-MANAGEMENT-8

[grishick]

It appears that the user was able to set up an incremental sync with an unsupported field type.

[edgao]

after #14714 this will be reported with a slightly more informative error message (OTHER cannot be used as a cursor).

Hopefully sentry is able to recognize that as the same stacktrace, but leaving a breadcrumb comment in case it doesn't.

[bleonard]

after #14714 this will be reported with a slightly more informative error message (OTHER cannot be used as a cursor).

Hopefully sentry is able to recognize that as the same stacktrace, but leaving a breadcrumb comment in case it doesn't.

It would be slightly nicer if OTHER was better explained. Is it possible to do "first_name column can not be used as a cursor. Please use integers or timestamps" or something that effect?

[edgao]

yeah, that's fair. I updated the issue title and description.

It requires a bit of refactoring to get the table+column name down to where the exception is thrown (or maybe to catch the exception where the those names are known?) so going to leave this be for now.

[bleonard]

yeah, that's fair. I updated the issue title and description.

It requires a bit of refactoring to get the table+column name down to where the exception is thrown (or maybe to catch the exception where the those names are known?) so going to leave this be for now.

Agreed, just noting a future ideal. Thanks.

[grishick]

This started happening again two days ago for another customer

[grishick]

This connection has a lot of tables, but from the error log, I cannot tell which table is failing. Would be great if the log included the name of the table, name of the field, field value, and as much information about field type and why JDBC driver cannot map it to any known type.

2022-08-03 08:33:00 source > java.lang.IllegalArgumentException: OTHER cannot be used as a cursor.
2022-08-03 08:33:00 source > 	at io.airbyte.integrations.source.postgres.PostgresSourceOperations.setStatementField(PostgresSourceOperations.java:110) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at io.airbyte.integrations.source.postgres.PostgresSourceOperations.setStatementField(PostgresSourceOperations.java:42) ~[io.airbyte.airbyte-integrations.connectors-source-postgres-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$16(AbstractJdbcSource.java:283) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:62) ~[io.airbyte.airbyte-db-db-lib-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$17(AbstractJdbcSource.java:273) ~[io.airbyte.airbyte-integrations.connectors-source-jdbc-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:37) ~[io.airbyte-airbyte-commons-0.39.41-alpha.jar:?]
2022-08-03 08:33:00 source > 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.0.1-jre.jar:?]
2022-08-03 08:33:00 source > 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.0.1-jre.jar:?]
2022-08-03 08:33:00 source > 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.0.1-jre.jar:?]

[grishick]

Related alpha-beta issue: https://github.com/airbytehq/alpha-beta-issues/issues/164

[tuliren]

There are already two PRs related to this issue:
#14356 - prevent users from selecting invalid columns
#15317 - improve error message

[subodh1810]

My PR has been approved but the build is failing, the remaining thing is to fix the build and get it merged

[grishick]

@subodh1810 builds should be OK now