🐛 Normalization: Fix sync from HubSpot to MySQL fails with "Row size too large" on create table #10485
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bump docker version. Update basic-normalization.md docs.
htrueman
requested review from
sergei-solonitcyn,
Zirochkaa and
yevhenii-ldv
github-actions
bot
added
area/documentation
|
@sergei-solonitcyn tested backward comparability. Everything works fine. |
|
Also it's possible to get another issue: |
sergei-solonitcyn
approved these changes
Feb 22, 2022
|
/test connector=bases/base-normalization
|
htrueman
changed the title
|
/publish connector=bases/base-normalization
|
Merged
|
Since the generated sql files from normalization-mysql is being changed in this PR, the integration tests outputs should also be included in the PR: they will reflect what the actual change in terms of final native SQL queries would be. See #10837 |
Comment on lines
+543
to
+545
| elif self.destination_type == DestinationType.MYSQL: | ||
| # Cast to `text` datatype. See https://github.com/airbytehq/airbyte/issues/7994 | ||
| sql_type = f"{sql_type}(1024)" |
There was a problem hiding this comment.
The proper way of changing a datatype for a certain destination should be done in dbt macros, not directly in the python code:
see
There was a problem hiding this comment.
Could this be causing this error?
Failure Origin: normalization, Message: Normalization failed during the dbt run. This may indicate a problem with the data itself.
1292 (22007): Truncated incorrect CHAR(1024) value:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Closes #7994.
Changes default string casting from
varchar(512)totext.How
cast(field as char)which leaded to field varchar(512).varchar(512)may use as much as 512 * 4 = 2048 bytes in utf8mb4 encoding.marketing_emailshas 38 of those fields).So as a fix, I've updated the mysql create table query to cast sting fields as
cast(field as char(1024)).Which leads to field text type on created table (
cast(field as text)is not a valid statement, see https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html#function_cast).Experimentally found that values that is less then 1024 are converted to varchar types and values larger may lead to mediumtext or longtext, which are too large.
Recommended reading order
stream_processor.py🚨 User Impact 🚨
This would change
varcharfield in mysql tables totext.