Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADAP-865: Parameterize where clause, add option to supply list of relations #758

Merged
merged 21 commits into from
Oct 11, 2023

Conversation

mikealfare
Copy link
Contributor

@mikealfare mikealfare commented Aug 29, 2023

resolves #801
docs dbt-labs/docs.getdbt.com/#

Problem

The current implementation of get_catalog only supports schema-level limitation. This can take a long time when schemas have a lot of objects in them.

Solution

Provide a way to pass in a list of relations instead of schemas. The new relations parameter can be a list of relations, a list of schema relation objects, or a mix.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@mikealfare mikealfare self-assigned this Aug 29, 2023
@cla-bot cla-bot bot added the cla:yes label Aug 29, 2023
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide.

@mikealfare

This comment was marked as outdated.

@mikealfare mikealfare changed the title parameterize where clause, add option to supply list of relations ADAP-865: Parameterize where clause, add option to supply list of relations Aug 30, 2023
dev-requirements.txt Outdated Show resolved Hide resolved
@mikealfare mikealfare marked this pull request as ready for review October 3, 2023 22:29
@mikealfare mikealfare requested a review from a team as a code owner October 3, 2023 22:29
@dataders
Copy link
Contributor

@mikealfare do you predict that there will be relevant adapter zone tests here?

@mikealfare
Copy link
Contributor Author

@mikealfare do you predict that there will be relevant adapter zone tests here?

The idea is that once this feature gets turned on (which it is for dbt-snowflake) then all existing catalog tests get routed through this new workflow. We should test this macro directly at some point, but we're fairly confident that we're at least at where we were before this change.

@colin-rogers-dbt, I know you had the same question. Does this address your concern as well?

@mikealfare mikealfare merged commit d44e80e into main Oct 11, 2023
23 checks passed
@mikealfare mikealfare deleted the feature/applied-state/get-catalog-by-object branch October 11, 2023 22:13
philippe-boyd-maxa pushed a commit to maxa-ai/dbt-snowflake that referenced this pull request Nov 27, 2023
…ations (dbt-labs#758)

* parameterize where clause, add option to supply list of relations

* parameterize where clause, add option to supply list of relations

* revert whitespace fix

* revert whitespace fix

* revert whitespace fix

* fix missing macro keyword

* point to the dev branch on core, revert before pushing to main

* add new macro get_catalog_relations, update get_catalog to share common logic with get_catalog_relations

* fixed reference in get_catalog_relations, added original dict version of relations temporarily for testing

* remove dict version of relations based get_catalog, point to List[BaseRelation] version

* point dev reqs back to main on core

* fix typo in schemas argument

* add feature flag to turn on relation filtering for get_catalog

* update changelog to point to the PR instead of a broken url

* Support changes to dbt-core capability system

---------

Co-authored-by: Peter Allen Webb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ADAP-943] [ADAP-928] [Feature] Implement relation filtering on get_catalog macro
6 participants