🎉 New source: TiDB #11283

Daemonxiao · 2022-03-21T07:16:16Z

What

Add new source TiDB.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Tests

Unit

Integration

Acceptance

CLAassistant · 2022-03-21T07:16:22Z

All committers have signed the CLA.

airbyte-integrations/connectors/source-tidb/src/main/resources/spec.json

…/spec.json uptdate doc Co-authored-by: Xiang Zhang <[email protected]>

airbyte-integrations/connectors/source-tidb/README.md

docs/integrations/sources/tidb.md

Co-authored-by: Xiang Zhang <[email protected]>

docs/integrations/sources/tidb.md

Co-authored-by: Xiang Zhang <[email protected]>

Daemonxiao · 2022-03-22T03:32:32Z

refer issue #10891

marcosmarxm · 2022-03-23T01:31:02Z

Thanks for the contribution @Daemonxiao I'll review it tomorrow

…marcos/test-pr-11283

marcosmarxm · 2022-03-23T22:15:45Z

@Daemonxiao is it not possible to use TiDB using the MySQL connector? The implementation looks is duplicating code from MySQL connector.

marcosmarxm · 2022-03-23T22:18:54Z

...ons/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java

+import org.slf4j.LoggerFactory;
+
+
+public class TiDBSource extends AbstractJdbcSource<MysqlType> implements Source {


Please check the implementation of mysql-strict-encryt connector. So far I didn't find any change from Mysql that is specifically applied to TiDB.

zhangyangyu · 2022-03-24T01:26:57Z

@Daemonxiao is it not possible to use TiDB using the MySQL connector? The implementation looks is duplicating code from MySQL connector.

Thanks for your review @marcosmarxm. TiDB is a MySQL compatible database, just like CockroachDB vs postgresql. So in most cases using MySQL connector or even JDBC connector works. But the compatibility is not 100%. For example, TiDB has its own internal system databases(present in code), TiDB doesn't support spatial types(present in doc). And the biggest incompatibility is TiDB doesn't support MySQL binlog at all so currently MySQL connector CDC would not function at all(present in both code and doc). And I expect in future the differences might vary. So we'd like to develop a dedicated TiDB source connector although currently it replicate many MySQL connector code.

marcosmarxm · 2022-03-24T18:11:44Z

@Daemonxiao is it not possible to use TiDB using the MySQL connector? The implementation looks is duplicating code from MySQL connector.

Thanks for your review @marcosmarxm. TiDB is a MySQL compatible database, just like CockroachDB vs postgresql. So in most cases using MySQL connector or even JDBC connector works. But the compatibility is not 100%. For example, TiDB has its own internal system databases(present in code), TiDB doesn't support spatial types(present in doc). And the biggest incompatibility is TiDB doesn't support MySQL binlog at all so currently MySQL connector CDC would not function at all(present in both code and doc). And I expect in future the differences might vary. So we'd like to develop a dedicated TiDB source connector although currently it replicate many MySQL connector code.

But it's possible to reduce the duplicate code right. The SQL operations can be the same because it's a subset of MySQL. You only need to create the spec and the building spec (see the connector I commented previously). Did you try to use only creating the TiDBSource.java file? Can you point in your code where is different from the MySQL implementation? Why not use the MySQL connector and only create a TiDB documentation explaining you can use the MySQL connector with ONLY the default method?

zhangyangyu · 2022-03-25T08:34:15Z

But it's possible to reduce the duplicate code right.

Definitely. We change the implementation from extending JDBC to MySQL, so no need to duplicate much code.

The SQL operations can be the same because it's a subset of MySQL. You only need to create the spec and the building spec (see the connector I commented previously).

Simply extending the spec is not ideal since the difference is not only in configuration but also code behavior. The difference is small for now, but we also want to leave the possibility for future. For example, add CDC ability, directly accessing the storage layer of TiDB.

Can you point in your code where is different from the MySQL implementation?

Please review the new implementation. :-) I think it's more clear now.

marcosmarxm · 2022-03-25T10:47:28Z

...tors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java

+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import static io.airbyte.db.jdbc.JdbcConstants.*;


Please convert to single import here.

Thanks for you review. I've changed it.
convert to single import

marcosmarxm

some comments

marcosmarxm · 2022-03-25T10:49:10Z

...ons/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java

+    return Set.of(
+        "information_schema",
+        "metrics_schema",
+        "performance_schema",
+        "mysql");
+  }


The internal of TiDB is exactly as MySQL ones?

No. TiDB is MySQL compatible but it's implemented from scratch, in golang and rust. So at least for these internal databases, it's definitely not 100% compatible. It doesn't support SYS database https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features and has its own METRICS_SCHEMA database https://docs.pingcap.com/tidb/stable/metrics-schema#metrics-schema.

marcosmarxm · 2022-03-25T10:54:38Z

...tors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java

+  @Override
+  public MysqlType getFieldType(JsonNode field) {
+    try {
+      final MysqlType literalType = MysqlType.getByName(field.get(INTERNAL_COLUMN_TYPE_NAME).asText());
+      final int columnSize = field.get(INTERNAL_COLUMN_SIZE).asInt();
+
+      switch (literalType) {
+        // TINYINT(1) are interpreted as boolean
+        case TINYINT, TINYINT_UNSIGNED -> {
+          if (columnSize == 1) {
+            return MysqlType.BOOLEAN;
+          }
+        }
+        // When CHAR[N] and VARCHAR[N] columns have binary character set, the returned
+        // types are BINARY[N] and VARBINARY[N], respectively. So we don't need to
+        // convert them here. This is verified in MySqlSourceDatatypeTest.
+      }
+
+      return literalType;
+    } catch (final IllegalArgumentException ex) {
+      LOGGER.warn(String.format("Could not convert column: %s from table: %s.%s with type: %s (type name: %s). Casting to VARCHAR.",
+          field.get(INTERNAL_COLUMN_NAME),
+          field.get(INTERNAL_SCHEMA_NAME),
+          field.get(INTERNAL_TABLE_NAME),
+          field.get(INTERNAL_COLUMN_TYPE),
+          field.get(INTERNAL_COLUMN_TYPE_NAME)));
+      return MysqlType.VARCHAR;
+    }
+  }


You can remove this function for now, you're not overwritten it. If in the future will be a change let this happens with it, but today the behavior is equal to MySQL.

Not exactly same. bit(1) is not interpreted as boolean.

marcosmarxm · 2022-03-25T10:55:38Z

...tors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java

+      case BIT -> {
+        putBinary(json, columnName, resultSet, colIndex);
+      }


Can you link to TiDB docs showing BIT works only a binary?

The formal doc of BIT is https://docs.pingcap.com/tidb/stable/data-type-numeric#bit-type. It explicitly documents TINYINT(1) as an alias of boolean https://docs.pingcap.com/tidb/stable/data-type-numeric#boolean-type. And for TiDB replication tools, none of its supported protocol interprets BIT(1) as boolean: https://docs.pingcap.com/tidb/stable/ticdc-canal-json#sql-type-field.

TiDB is built from scratch to be compatible with MySQL 5.7 so don't suffer history debts.

marcosmarxm · 2022-03-30T16:29:40Z

Thanks @Daemonxiao I requested to the Connector team to give a final review of your contribution.

marcosmarxm · 2022-04-19T00:34:25Z

/test connector=connectors/source-tidb

marcosmarxm

thanks @Daemonxiao

Daemonxiao · 2022-04-21T03:49:28Z

@marcosmarxm Thanks for your work. BTW, do we need to provide the TiDB logo for Airbyte UI?

marcosmarxm · 2022-04-25T20:37:48Z

@marcosmarxm Thanks for your work. BTW, do we need to provide the TiDB logo for Airbyte UI?

you can add into airbyte-config/init/src/main/resources/icons
also update the seed file to receive the icon

* add new source tidb * formate java code style and add item in SUMMARY.md * update doc * Update airbyte-integrations/connectors/source-tidb/src/main/resources/spec.json uptdate doc Co-authored-by: Xiang Zhang <[email protected]> * Update airbyte-integrations/connectors/source-tidb/README.md * Update docs/integrations/sources/tidb.md Co-authored-by: Xiang Zhang <[email protected]> * Update docs/integrations/sources/tidb.md Co-authored-by: Xiang Zhang <[email protected]> * add seed and doc changelog * run format * regenerate seed file Co-authored-by: Xiang Zhang <[email protected]> Co-authored-by: marcosmarxm <[email protected]>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Mar 21, 2022

octavia-squidington-iii added the community label Mar 21, 2022

Daemonxiao added 3 commits March 21, 2022 08:38

add new source tidb

430df4e

formate java code style and add item in SUMMARY.md

cceec7f

update doc

b05d5d5

zhangyangyu reviewed Mar 22, 2022

View reviewed changes

airbyte-integrations/connectors/source-tidb/src/main/resources/spec.json Outdated Show resolved Hide resolved

Update airbyte-integrations/connectors/source-tidb/src/main/resources…

e138f8b

…/spec.json uptdate doc Co-authored-by: Xiang Zhang <[email protected]>

Daemonxiao commented Mar 22, 2022

View reviewed changes

airbyte-integrations/connectors/source-tidb/README.md Outdated Show resolved Hide resolved

Update airbyte-integrations/connectors/source-tidb/README.md

23ccb9c

zhangyangyu reviewed Mar 22, 2022

View reviewed changes

docs/integrations/sources/tidb.md Outdated Show resolved Hide resolved

Update docs/integrations/sources/tidb.md

3bb36ef

Co-authored-by: Xiang Zhang <[email protected]>

zhangyangyu reviewed Mar 22, 2022

View reviewed changes

docs/integrations/sources/tidb.md Outdated Show resolved Hide resolved

Update docs/integrations/sources/tidb.md

4b65720

Co-authored-by: Xiang Zhang <[email protected]>

marcosmarxm self-assigned this Mar 23, 2022

Merge branch 'new-source-tidb' of github.com:Daemonxiao/airbyte into …

aa4109e

…marcos/test-pr-11283

marcosmarxm requested changes Mar 23, 2022

View reviewed changes

marcosmarxm requested changes Mar 25, 2022

View reviewed changes

marcosmarxm added 2 commits April 19, 2022 18:31

solve conflict

b7f78fb

add seed and doc changelog

605158d

run format

c22969a

marcosmarxm approved these changes Apr 20, 2022

View reviewed changes

marcosmarxm temporarily deployed to more-secrets April 20, 2022 00:18 Inactive

regenerate seed file

16b3746

marcosmarxm temporarily deployed to more-secrets April 20, 2022 00:21 Inactive

marcosmarxm merged commit 5164e12 into airbytehq:master Apr 20, 2022

octavia-squidington-iii mentioned this pull request Apr 20, 2022

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12194

Closed

octavia-squidington-iii mentioned this pull request Apr 21, 2022

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12258

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New source: TiDB #11283

🎉 New source: TiDB #11283

Daemonxiao commented Mar 21, 2022

CLAassistant commented Mar 21, 2022 •

edited

Loading

Daemonxiao commented Mar 22, 2022

marcosmarxm commented Mar 23, 2022

marcosmarxm commented Mar 23, 2022

marcosmarxm Mar 23, 2022

zhangyangyu commented Mar 24, 2022 •

edited

Loading

marcosmarxm commented Mar 24, 2022

zhangyangyu commented Mar 25, 2022 •

edited

Loading

marcosmarxm Mar 25, 2022

Daemonxiao Mar 25, 2022

marcosmarxm left a comment

marcosmarxm Mar 25, 2022

zhangyangyu Mar 25, 2022 •

edited

Loading

marcosmarxm Mar 25, 2022

zhangyangyu Mar 25, 2022

marcosmarxm Mar 25, 2022

zhangyangyu Mar 25, 2022 •

edited

Loading

marcosmarxm commented Mar 30, 2022

marcosmarxm commented Apr 19, 2022

marcosmarxm left a comment

Daemonxiao commented Apr 21, 2022

marcosmarxm commented Apr 25, 2022

		import org.slf4j.LoggerFactory;


		public class TiDBSource extends AbstractJdbcSource<MysqlType> implements Source {

🎉 New source: TiDB #11283

🎉 New source: TiDB #11283

Conversation

Daemonxiao commented Mar 21, 2022

What

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Tests

CLAassistant commented Mar 21, 2022 • edited Loading

Daemonxiao commented Mar 22, 2022

marcosmarxm commented Mar 23, 2022

marcosmarxm commented Mar 23, 2022

marcosmarxm Mar 23, 2022

Choose a reason for hiding this comment

zhangyangyu commented Mar 24, 2022 • edited Loading

marcosmarxm commented Mar 24, 2022

zhangyangyu commented Mar 25, 2022 • edited Loading

marcosmarxm Mar 25, 2022

Choose a reason for hiding this comment

Daemonxiao Mar 25, 2022

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Mar 25, 2022

Choose a reason for hiding this comment

zhangyangyu Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

marcosmarxm Mar 25, 2022

Choose a reason for hiding this comment

zhangyangyu Mar 25, 2022

Choose a reason for hiding this comment

marcosmarxm Mar 25, 2022

Choose a reason for hiding this comment

zhangyangyu Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

marcosmarxm commented Mar 30, 2022

marcosmarxm commented Apr 19, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

Daemonxiao commented Apr 21, 2022

marcosmarxm commented Apr 25, 2022

CLAassistant commented Mar 21, 2022 •

edited

Loading

zhangyangyu commented Mar 24, 2022 •

edited

Loading

zhangyangyu commented Mar 25, 2022 •

edited

Loading

zhangyangyu Mar 25, 2022 •

edited

Loading

zhangyangyu Mar 25, 2022 •

edited

Loading