Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: define a maxLength limit for varchar at the connector level #1812

Open
pnadolny13 opened this issue Jul 6, 2023 · 3 comments
Open
Labels
Accepting Pull Requests evergreen good first issue Good for newcomers kind/Feature New feature or request SQL Support for SQL taps and targets
Milestone

Comments

@pnadolny13
Copy link
Contributor

pnadolny13 commented Jul 6, 2023

Originally raised in slack https://meltano.slack.com/archives/C01TCRBBJD7/p1688652938423919

The meltanolabs target-snowflake was throwing an error because its accepting the max length 4294967295 from the schema message as the varchar length which is larger than what snowflake accepts 16777216. It looks like currently we just pass that value through to sqlalchemy with no validation to limit it from being too large and ultimately the DDL fails.

I'm going to fix it in target-snowflake but it does feel like something that should be handled by the SDK because its going to be relevant for most ever target. If we had a property method that needed to be implemented or a default class property that could be overridden. Another thing that might be worth checking out is if the sqlalchemy dialect package could tell us this generically somehow.

@pnadolny13
Copy link
Contributor Author

This was solved in target-snowflake in MeltanoLabs/target-snowflake#68. That has some logic specific to target-snowflake so using something like Edgar suggested in MeltanoLabs/target-snowflake#68 (comment) would probably be best for the SDK.

Copy link

stale bot commented Aug 9, 2024

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@edgarrmondragon
Copy link
Collaborator

edgarrmondragon commented Aug 29, 2024

We'll be able to do this once #2618 is in

That's not it, since that's for taps. This is for targets, and a little more involved so punting to a later milestone.

@edgarrmondragon edgarrmondragon added the SQL Support for SQL taps and targets label Aug 29, 2024
@edgarrmondragon edgarrmondragon modified the milestones: v0.41.0, v0.42.0 Sep 6, 2024
@edgarrmondragon edgarrmondragon modified the milestones: v0.42.0, v0.43.0 Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Accepting Pull Requests evergreen good first issue Good for newcomers kind/Feature New feature or request SQL Support for SQL taps and targets
Projects
None yet
Development

No branches or pull requests

2 participants