Provide path for SQL target developers to "max out" varchar declarations in columnarly compressed datasets

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/373

Originally created by @aaronsteers on 2022-05-04 23:09:50

---

~~"urgency::low"

Now that https://gitlab.com/meltano/sdk/-/issues/371+ is resolved and merged, I want to open a quick stub issue to track the work to add a recommendation for those target developers who prefer a length-unlimited or length-maxed varchar declaration.

• Columnar databases may not have any storage or performance differences based on length, for instance between varchar(10) and varchar(max).
• Some databases may also have a high cost of running alter column and/or have alter column disallowed.

Rough implementation

One path to implement:

• Developer overrides to_sql_type(jsonschema_type: dict) -> sqlalchemy.types.TypeEngine, calling the base class and then converting any varchar instances with varchar(max).
• Depending upon the target implementation, and whether it makes sense to make this a user-configurable implementation choice, the developer could optionally tie this behavior to a setting like prefer_varchar_max or similar.

The above could be a simple docs update or an addition to code samples, or it could be an built-in option or implementation flag within the SDK's python code.

[MeltyBot]

View 2 previous comments from the original issue on GitLab

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Superseded by #1812, which can be ported from a reference implementation