Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Base Normalization: quote schema name to allow reserved keywords #14683

Merged
merged 14 commits into from
Jul 27, 2022
Merged

🎉 Base Normalization: quote schema name to allow reserved keywords #14683

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
907bbe4
quote this.schema
grubberr Jul 13, 2022
f114106
basic-normalization.md updated
grubberr Jul 13, 2022
759ea2e
NormalizationRunnerFactory.java updated
grubberr Jul 13, 2022
546bef5
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 14, 2022
cc87c28
format code
grubberr Jul 14, 2022
0bcd4b7
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 20, 2022
1a752fe
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 24, 2022
392863d
lint code
grubberr Jul 24, 2022
743163d
lint code (2)
grubberr Jul 24, 2022
c2bfd09
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 24, 2022
7fed9d1
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 27, 2022
755598a
basic-normalization.md updated
grubberr Jul 27, 2022
ab4a8aa
Merge branch 'master' into grubberr/14674-base-normalization
grubberr Jul 27, 2022
675248f
NormalizationRunnerFactory.java updated
grubberr Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/bases/base-normalization/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@ WORKDIR /airbyte
ENV AIRBYTE_ENTRYPOINT "/airbyte/entrypoint.sh"
ENTRYPOINT ["/airbyte/entrypoint.sh"]

LABEL io.airbyte.version=0.2.12
LABEL io.airbyte.version=0.2.13
LABEL io.airbyte.name=airbyte/normalization
10 changes: 6 additions & 4 deletions .../bases/base-normalization/normalization/transform_catalog/destination_name_transformer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,13 +181,15 @@ def __normalize_identifier_name(
return f"'{result}'"
return result

def apply_quote(self, input: str) -> str:
def apply_quote(self, input: str, literal=True) -> str:
if literal:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I understand this change - previously all of our quotes were using quote or adapter.quote; why do we need to switch them to just wrapping in single-quote marks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I agree not very obvious

before my change this function generated only such output:

adapter.quote('table_name')

as you know on jinja2 "level" it means: pass String literal 'table_name' to jinaj2 function adapter.quote

but I have a new requirement I need to get:

adapter.quote(this.schema)

on jinja2 level it means pass jinja2 variable to jinja2 function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah that makes sense (also I misread the code, thought it was doing a return f"'{input}'"). So basically this is behaving exactly the same for literal=False.

input = f"'{input}'"
if self.destination_type == DestinationType.ORACLE:
# Oracle dbt lib doesn't implemented adapter quote yet.
return f"quote('{input}')"
edgao marked this conversation as resolved.
Show resolved Hide resolved
return f"quote({input})"
elif self.destination_type == DestinationType.CLICKHOUSE:
return f"quote('{input}')"
return f"adapter.quote('{input}')"
return f"quote({input})"
return f"adapter.quote({input})"

def __normalize_naming_conventions(self, input_name: str, is_column: bool = False) -> str:
result = input_name
Expand Down
5 changes: 4 additions & 1 deletion ...integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1242,7 +1242,10 @@ def add_to_outputs(
quoted_unique_key=self.get_unique_key(in_jinja=True),
active_row_column_name=active_row_column_name,
normalized_at_incremental_clause=self.get_incremental_clause_for_column(
"this.schema + '.' + " + self.name_transformer.apply_quote(final_table_name),
"{} + '.' + {}".format(
self.name_transformer.apply_quote("this.schema", literal=False),
self.name_transformer.apply_quote(final_table_name),
),
self.get_normalized_at(in_jinja=True),
),
unique_key_reference=unique_key_reference,
Expand Down
2 changes: 1 addition & 1 deletion ...te-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
public class NormalizationRunnerFactory {

public static final String BASE_NORMALIZATION_IMAGE_NAME = "airbyte/normalization";
public static final String NORMALIZATION_VERSION = "0.2.12";
public static final String NORMALIZATION_VERSION = "0.2.13";

static final Map<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>> NORMALIZATION_MAPPING =
ImmutableMap.<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>>builder()
Expand Down
4 changes: 2 additions & 2 deletions docs/understanding-airbyte/basic-normalization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,8 +353,9 @@ Therefore, in order to "upgrade" to the desired normalization version, you need

| Airbyte Version | Normalization Version | Date | Pull Request | Subject |
|:----------------|:----------------------|:-----------| :--- |:---------------------------------------------------------------------------|
| | 0.2.11 | 2022-07-26 | [\#13591](https://github.com/airbytehq/airbyte/pull/13591) | Updated support for integer columns. |
| | 0.2.13 | 2022-07-27 | [\#14683](https://github.com/airbytehq/airbyte/pull/14683) | Quote schema name to allow reserved keywords |
| | 0.2.12 | 2022-07-26 | [\#14362](https://github.com/airbytehq/airbyte/pull/14362) | Handle timezone in date-time format. Parse date correct in clickhouse. |
| | 0.2.11 | 2022-07-26 | [\#13591](https://github.com/airbytehq/airbyte/pull/13591) | Updated support for integer columns. |
| | 0.2.10 | 2022-07-18 | [\#14792](https://github.com/airbytehq/airbyte/pull/14792) | Add support for key pair auth for snowflake |
| | 0.2.9 | 2022-07-06 | [\#14485](https://github.com/airbytehq/airbyte/pull/14485) | BigQuery partition pruning otimization |
| | 0.2.8 | 2022-07-13 | [\#14522](https://github.com/airbytehq/airbyte/pull/14522) | BigQuery replaces `NULL` array entries with the string value `"NULL"` |
Expand Down Expand Up @@ -400,4 +401,3 @@ Therefore, in order to "upgrade" to the desired normalization version, you need
| 0.28.2-alpha | 0.1.38 | 2021-07-28 | [\#5027](https://github.com/airbytehq/airbyte/pull/5027) | Handle quotes in column names when parsing JSON blob |
| 0.27.5-alpha | 0.1.37 | 2021-07-22 | [\#3947](https://github.com/airbytehq/airbyte/pull/4881/) | Handle `NULL` cursor field values when deduping |
| 0.27.2-alpha | 0.1.36 | 2021-07-09 | [\#3947](https://github.com/airbytehq/airbyte/pull/4163/) | Enable normalization for MySQL destination |