📝 MySql source: make it clear about tinyint (size > 1)

[tuliren]

What

• Make it clear that tinyint(>1) is converted to number, not all tinyint.

[danieldiamond]

@tuliren - i realise this is just a docs PR - i'm running into issues for boolean fields in mysql 1.0.1 source that are tinyint(1) unsigned

Numeric value 'true' is not recognized

see issue https://discuss.airbyte.io/t/mysql-1-0-1-ga-source-normalization-failure/2773

[tuliren]

The MySQL source has not changed in parsing tinyint.

Source code:
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java#L92-L99

case TINYINT, TINYINT_UNSIGNED -> {
  if (field.getLength() == 1L) {
    // TINYINT(1) is boolean
    putBoolean(json, columnName, resultSet, colIndex);
  } else {
    putShortInt(json, columnName, resultSet, colIndex);
  }
}

Unit test:
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mysql/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractMySqlSourceDatatypeTest.java#L84-L101

// tinyint without width
addDataTypeTestData(
    TestDataHolder.builder()
        .sourceType("tinyint")
        .airbyteType(JsonSchemaType.INTEGER)
        .addInsertValues("null", "-128", "127")
        .addExpectedValues(null, "-128", "127")
        .build());

// tinyint(1) is equivalent to boolean
addDataTypeTestData(
    TestDataHolder.builder()
        .sourceType("tinyint")
        .fullSourceDataType("tinyint(1)")
        .airbyteType(JsonSchemaType.BOOLEAN)
        .addInsertValues("null", "1", "0")
        .addExpectedValues(null, "true", "false")
        .build());

which both confirmed that tinyint(1) is interpreted as boolean, and tinyint(>1) is small integer.

@edgao, what Daniel mentioned in this thread may be a regress in Snowflake normalization. Do you know there is any recent change that lead to this?

[edgao]

Not after 0.6.12. #14362 could be related, but that's from late July; source-mysql 0.6.12 was released in mid-September. The type-detection changes in there don't look problematic either.

@danieldiamond did you upgrade the platform in addition to source-mysql? Or did you only upgrade the source-mysql connector?

[danieldiamond]

I updated from 6.12 to 1.0.1 and the issue arose.
I've since updated the platform as well.

[edgao]

got it. probably not a change in normalization then, if the issue started happening immediately after the connector upgrade but before platform upgrade.

When was the last time you refreshed your schema? Wondering if you're just running with a schema that was generated under a really old source-mysql version, which could explain this behavior

[danieldiamond]

I've just deployed the latest airbyte version, mysql source, snowflake destination and synced for the first time meaning the latest schema from source table and still running into the issue:

I've created a new issue and attached logs: https://discuss.airbyte.io/t/mysql-to-snowflake-normalization-issue-all-latest-versions/2883

[edgao]

hm - @subodh1810 I'm not really sure what's happening here, do you have some thoughts? latest version of platform + source-mysql (CDC) is failing to sync a boolean column: (schema was generated using latest source-mysql version, so shouldn't be a version conflict)

2022-10-12 06:13:23 normalization > Database Error in model TEST_mytable_SCD (models/generated/airbyte_incremental/scd/my_schema/TEST_mytable_SCD.sql)
2022-10-12 06:13:23 normalization >   100038 (22018): Numeric value 'false' is not recognized

Seems like it's getting discovered as an integer (or number?) but read is outputting a true/false string, so normalization fails. But I don't understand how this is slipping past source-mysql's datatype tests

mysql schema info in #17510 (comment)

[edgao]

derp. I forgot that Subodh is OOO right now - @rodireich tagging you as a random db-sources member for this question ^

[danieldiamond]

bump @edgao @rodireich @subodh1810

[edgao]

I poked db the sources team for help here - unfortunately I'm a bit out of the loop with this connector but will make sure we figure this out

[rodireich]

@danieldiamond Can you please try adding tinyInt1isBit=false as a jdbc url param.
Just to confirm, are you using logical replication (CDC) or standard?

[danieldiamond]

CDC

[danieldiamond]

@rodireich adding the jdbc url param made no difference

[rodireich]

Thanks Daniel, I'm looking if there's some discrepancy between how CDC and standard deal with tinyint(1) columns.

[rodireich]

Hi Daniel
I'm replicating a mysql db tables with TINYINT(1) and TINYINT(4) with CDC and see no problem.
Can you share information on how you connection is set?
In my case it's source-mysql(1.0.7) <> postgres

[danieldiamond]

mysql DB version is 5.7.38
source-mysql(1.0.7) <> destination-snowflake(0.4.38)
Syncing CDC

COL A: tinyint(3) unsigned
COL B: tinyint(3) unsigned
COL C: tinyint(1) unsigned

I think column C is the offending column

[rodireich]

And may I ask what the sync mode for this table is? (incremental+dedup etc.)

[danieldiamond]

incremental+dedup

[subodh1810]

@danieldiamond I tried to reproduce your issue.
Airbyte : v0.40.17
MySQL Source : v1.0.6 (CDC mode)
Snowflake destination : v0.4.38 (Internal Staging)

Connection was setup using INCREMENTAL | DEDUPED + HISTORY.

Source table data and structure :

mysql> select * from tiny_int_test;
+----+----------+----------+
| id | column_1 | column_2 |
+----+----------+----------+
|  1 |        0 |       99 |
|  2 |        1 |      100 |
|  3 |        2 |      101 |
+----+----------+----------+
3 rows in set (0.00 sec)

mysql> desc tiny_int_test;
+----------+------------+------+-----+---------+-------+
| Field    | Type       | Null | Key | Default | Extra |
+----------+------------+------+-----+---------+-------+
| id       | int        | NO   | PRI | NULL    |       |
| column_1 | tinyint(1) | YES  |     | NULL    |       |
| column_2 | tinyint    | YES  |     | NULL    |       |
+----------+------------+------+-----+---------+-------+
3 rows in set (0.00 sec)

as you can see above the column_1 is marked as tinyint(1) and it is being considered as a boolean column by Airbyte.

The sync along with normalization was successful, the column was created as a boolean in snowflake destination and the data was also correct.
column type

data

[subodh1810]

I am not sure why is your sync failing. I would advice to clean up the scd tables manually and then use the latest versions of the platform and the source/destination and try again.

[danieldiamond]

@subodh1810 thank you so much for your investigation. Unfortunately I am using the latest version of everything (airbyte, source, destination) and removing the SCD table is not an option - it doesn't exist, this is the first sync.

Airbyte version: 0.40.17
airbyte/source-mysql:1.0.8
airbyte/destination-snowflake:0.4.38

Let me try provide as much information as possible so that you might notice some inconsistencies.

CDC sync: incremental+dedupe
Staging: AWS S3 Staging

Source schema:

[
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "PRI",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "MUL",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "MUL",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "MUL",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "mediumint(8) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "tinyint(3) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": "0",
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "smallint(3) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": "0",
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "tinyint(3) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": "0",
    "Extra": ""
  },
  {
    "OFFENDING_FIELD": "OFFENDING",
    "Type": "tinyint(1) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": "0",
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "MUL",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "NO",
    "Key": "",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "smallint(5) unsigned",
    "Null": "NO",
    "Key": "MUL",
    "Default": "0",
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "enum('a','b','c')",
    "Null": "NO",
    "Key": "",
    "Default": "a",
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "mediumint(9)",
    "Null": "YES",
    "Key": "",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "YES",
    "Key": "",
    "Default": null,
    "Extra": ""
  },
  {
    "Field": "<redacted>",
    "Type": "int(10) unsigned",
    "Null": "YES",
    "Key": "",
    "Default": null,
    "Extra": ""
  }
]

See OFFENDING_FIELD definition above

Every column in airbyte is defined as Integer (none are boolean), except for the enum one which is string.

The raw table in snowflake has the offending column as false, which is why it fails in the staging view which is trying to cast it as:

cast(OFFENDING_COLUMN as 
    bigint
) as OFFENDING_COLUMN

[danieldiamond]

https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-boolean-values

Map all TINYINT(1) or TINYINT(1) UNSIGNED columns to BOOLEAN types.

Can you try TINYINT(1) UNSIGNED (specifically unsigned as i've shown in my schema above?)

[subodh1810]

Thanks @danieldiamond I was able to reproduce the problem. You mentioned correctly that the issue is specifically with TINYINT(1) unsigned. Working on a fix!

[danieldiamond]

Really appreciate the double take @subodh1810 thanks for your investigation and help!

[subodh1810]

I have raised PR #18619 to tackle this. Go through the description of the PR for detailed understanding of what went wrong