Add SemanticModel Node Type #7769
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With the integration with MetricFlow we're taking a dependency on `dbt-semantic-interfaces` which acts as the source of truth for protocols which MetricFlow and dbt-core need to agree on. Additionally we're hard pinning to 0.1.0.dev3 for now. We plan on having a less restrictive specification when dbt-core 1.6 hits GA.
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
gshank
reviewed
Jun 5, 2023
There was a problem hiding this comment.
In general it looks like a perfectly reasonable starting point. I did have a few nits...
The discussion about our various Relation classes might be a larger thing and I'm not actually asking for a change there, those classes just seem to be proliferating lately.
QMalcolm
reviewed
Jun 6, 2023
peterallenwebb
requested review from
emmyoop and
aranke
and removed request for
a team
MichelleArk
reviewed
Jun 7, 2023
| def process_semantic_models(self) -> None: | ||
| for semantic_model in self.manifest.semantic_models.values(): | ||
| if semantic_model.model: | ||
| statically_parsed = py_extract_from_source(f"{{{{ {semantic_model.model} }}}}") |
There was a problem hiding this comment.
this call can throw an ExtractionError (defined in dbt_extractor) -- would be good to catch it and give a throw user-facing error instead since it is likely to be a typo
There was a problem hiding this comment.
Agree, and we'll follow up in the current sprint with #7822.
MichelleArk
reviewed
Jun 7, 2023
| input_package_name, input_model_name = None, ref[0] | ||
|
|
||
| refd_node = self.manifest.ref_lookup.find( | ||
| input_model_name, input_package_name, None, self.manifest |
There was a problem hiding this comment.
could we add a #TODO comment here that links to #7688? refs with the version keyword won't be supported until 7688 is closed. At that point we can pass in the parsed version instead of always passing None to the ref lookup call
There was a problem hiding this comment.
Agree, and we'll follow up in the current sprint with #7822.
MichelleArk
reviewed
Jun 7, 2023
MichelleArk
reviewed
Jun 7, 2023
| fqn=fqn, | ||
| model=unparsed.model, | ||
| name=unparsed.name, | ||
| node_relation=None, # Resolved from the value of "model" after parsing |
There was a problem hiding this comment.
for my own understanding -- we need to resolve the relation later on because not all model nodes are available in the manifest for lookup at this point?
There was a problem hiding this comment.
Should we make the node relation evaluate lazily, so we can put its value in here?
There was a problem hiding this comment.
SemanticModels come from yaml only, and the referenced models should always be sql based and so already resolvable when this is parsed.
There was a problem hiding this comment.
@MichelleArk Yes, I discussed this with Gerda and she thought we could possibly handle it immediately, since models would likely have been created, but doing it at the end would be most likely to succeed without complications.
@aranke I'm can follow up the idea of lazy evaluation in the current sprint. If you have a specific method of doing it in mind can you add it as a comment to #7822?
MichelleArk
reviewed
Jun 7, 2023
| class NodeRelation(dbtClassMixin): | ||
| alias: str | ||
| schema_name: str # TODO: Could this be called simply "schema" so we could reuse StateRelation? | ||
| database: Optional[str] = None |
There was a problem hiding this comment.
From this comment -- does this still need to include a unified relation_name that respects quoting + include policies?
There was a problem hiding this comment.
if we do -- we'll probably need the adapter.Relation to do something like this during compilation:
adapter = get_adapter(self.config)
relation_cls = adapter.Relation
relation_name = str(relation_cls.create_from(self.config, node))(from https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/compilation.py#L486-L488)
There was a problem hiding this comment.
I don't think the SemanticModels actually get compiled, since they're yaml-only. There is a question of whether they need the individual pieces (identifier/schema/database) or just the relation_name...
There was a problem hiding this comment.
@QMalcolm I'm not sure of the answer to this, but I suspect the answer may be yes. What makes sense from the perspective of MetricFlow integration?
There was a problem hiding this comment.
@MichelleArk Yep! We need to include the unified relation_name, and it's desirable that it respects quoting + include policies.
As for whether we need both the individual pieces as well as the relation_name, for this PR yes, long term we don't have to. MetricFlow is fine with having the parts or the relation_name. It was suggested that we use this node_relation object as it was a pattern that was already used elsewhere, thus the shape of the object currently. Though MetricFlow currently uses the relation_name attribute as seen here, and ignores the individual properties. However, the individual properties as well as the relation_name are currently required by the protocol. So for the scope of this PR, we should include the individual properties as well as the relation_name. Otherwise we'd first need to update and release a new DSI (which would include a number of other schema changes that have been made), and then propagate all those changes in core. Although the node_relation changes might be trivial in core, propagating the other changes with the new DSI could eat a fair bit of time.
|
Looks like a good starting point! Most of my comments are around error handling -- those aren't necessarily blocking this chunk of work but we should create follow-up issues to track the ones we want to address after this is merged. |
aranke
reviewed
Jun 7, 2023
|
|
||
|
|
||
| @dataclass | ||
| class NodeRelation(dbtClassMixin): |
There was a problem hiding this comment.
Stu added in a StateRelation class, I think we can use that instead?
There was a problem hiding this comment.
That one doesn't actually have the relation_name in it, and I think they want that. Settling on a common relation class is the "larger issue" that I mentioned we didn't need to solve at this particular moment.
| fqn=fqn, | ||
| model=unparsed.model, | ||
| name=unparsed.name, | ||
| node_relation=None, # Resolved from the value of "model" after parsing |
There was a problem hiding this comment.
Should we make the node relation evaluate lazily, so we can put its value in here?
| current_timestamp as created_at""" | ||
|
|
||
|
|
||
| class TestSemanticModelParsing: |
There was a problem hiding this comment.
What are the 3-5 most common ways this could go wrong?
Can we test for them?
There was a problem hiding this comment.
I agree that testing is too light in this changeset. The directive Jeremy has given Quigley and I is to get the MetricFlow team unblocked ASAP and clean things up in future issues. So I've opened this one to address testing: #7820, and we will work on it in the current sprint.
gshank
reviewed
Jun 7, 2023
core/dbt/contracts/graph/manifest.py
Outdated
| @@ -706,6 +707,7 @@ class Manifest(MacroMethods, DataClassMessagePackMixin, dbtClassMixin): | |||
| public_nodes: MutableMapping[str, PublicModel] = field(default_factory=dict) | |||
| project_dependencies: Optional[ProjectDependencies] = None | |||
| publications: MutableMapping[str, PublicationConfig] = field(default_factory=dict) | |||
| semantic_models: MutableMapping[str, SemanticModel] = field(default_factory=dict) | |||
There was a problem hiding this comment.
The existing pattern is the node objects use "Model" and the dictionaries use "nodes", that is Model nodes are found in "nodes" dictionary, "PublicModel" nodes are found in public_nodes. Could this be the "semantic_nodes" to preserve that pattern?
There was a problem hiding this comment.
In that case, do you think we just rely on the existing nodes dictionary, and not have a special case for SemanticModel objects? Would it matter that the SemanticModels are not yet linked into the compiled graph?
There was a problem hiding this comment.
The original "metrics" went in a "metrics" dictionary... This has been a bit ad-hoc and not closely thought through, but it feels like the existing pattern is that SQL things go in nodes (plus seeds...) and yaml-generated things go in their own dictionaries. There are some assumptions in the rest of the code that match, such as the links in the file objects, etc. So I think it still makes sense to put semantic models in their own dictionary. If anything I'd be tempted to separate out some of the existing things that are in the nodes dictionary into their own dictionaries and maybe make some combined "indexes" (for cases where we don't want to loop over "nodes" all the time...)
There was a problem hiding this comment.
Just wanted to mention that one of the reasons for going in the direction of more individual dictionaries rather than less, is that currently jsonschema and deserializers can't always correctly guess the classes of the objects in the "nodes" dictionary, leading to other hacky things like the big deserialization if statement in the _deserialize method in ParsedNode.
There was a problem hiding this comment.
This is all really good to know. It seems like the right direction to me.
core/dbt/contracts/files.py
Outdated
| @@ -228,6 +228,7 @@ class SchemaSourceFile(BaseSourceFile): | |||
| groups: List[str] = field(default_factory=list) | |||
| # node patches contain models, seeds, snapshots, analyses | |||
| ndp: List[str] = field(default_factory=list) | |||
| semantic_models: List[str] = field(default_factory=list) | |||
There was a problem hiding this comment.
semantic_nodes? (see other comment)
There was a problem hiding this comment.
@gshank To clarify, do you think we should change the name in the yml file, or just for our internal property names?
There was a problem hiding this comment.
Just for for the internal property names. I think "semantic_models" in the yaml files is fine, it matches the "models" section we already have. Naming is so hard.
|
|
||
|
|
||
| @dataclass | ||
| class NodeRelation(dbtClassMixin): |
There was a problem hiding this comment.
That one doesn't actually have the relation_name in it, and I think they want that. Settling on a common relation class is the "larger issue" that I mentioned we didn't need to solve at this particular moment.
| class NodeRelation(dbtClassMixin): | ||
| alias: str | ||
| schema_name: str # TODO: Could this be called simply "schema" so we could reuse StateRelation? | ||
| database: Optional[str] = None |
There was a problem hiding this comment.
I don't think the SemanticModels actually get compiled, since they're yaml-only. There is a question of whether they need the individual pieces (identifier/schema/database) or just the relation_name...
| fqn=fqn, | ||
| model=unparsed.model, | ||
| name=unparsed.name, | ||
| node_relation=None, # Resolved from the value of "model" after parsing |
There was a problem hiding this comment.
SemanticModels come from yaml only, and the referenced models should always be sql based and so already resolvable when this is parsed.
gshank
approved these changes
Jun 7, 2023
6 tasks
This was referenced Jun 9, 2023
leahwicz
added a commit
to leahwicz/dbt-core
that referenced
this pull request
Sep 12, 2023
* break out a large test suite as a separate execution to avoid memory issues on windows CI runs (#7669) * Remove `DelayedFileHandler` (#7661) * remove DelayedFileHandler * Changelog * set_path to no-op * more no-ops for rpc * Clearer comments * Model Deprecation (#7562) * CT-2461: Work toward model deprecation * CT-2461: Remove unneeded conversions * CT-2461: Fix up unit tests for new fields, correct a couple oversights * CT-2461: Remaining implementation and tests for model/ref deprecation warnings * CT-2461: Changelog entry for deprecation warnings * CT-2461: Refine datetime handling and tests * CT-2461: Fix up unit test data * CT-2461: Fix some more unit test data. * CT-2461: Fix merge issues * CT-2461: Code review items. * CT-2461: Improve version -> str conversion * fix #7502: write run_results.json for run operation (#7655) * Honor `--skip-profile-setup` parameter when inside an existing project (#7609) * Honor `--skip-profile-setup` parameter when inside an existing project * Use project name as the profile name * Use separate file connections for reading and writing * Raise a custom exception when no adapters are installed * Test skipping interactive profile setup when inside a dbt project * Replace `assert_not_called()` since it does not work * Verbose CLI argument for skipping profile setup * Use separate file connections for reading and writing * Check empty list in a Pythonic manner * Include null checks in utils test base (#7672) * Include null checks in utils test base * Add tests for the schema test * Add tests for this macro --------- Co-authored-by: Mila Page <[email protected]> * Bump ubuntu from 23.04 to 23.10 (#7675) * Bump ubuntu from 23.04 to 23.10 Bumps ubuntu from 23.04 to 23.10. --- updated-dependencies: - dependency-name: ubuntu dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Add automated changelog yaml from template for bot PR --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <[email protected]> * Adding performance modeling for 1.3.4 to refs/heads/main (#7525) * adding performance baseline for 1.3.4 * Add newline --------- Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: leahwicz <[email protected]> * Adding performance modeling for 1.4.1 to refs/heads/main (#7527) * adding performance baseline for 1.4.1 * Add newline --------- Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: leahwicz <[email protected]> * Adding performance modeling for 1.3.0 to refs/heads/main (#7530) * adding performance baseline for 1.3.0 * Add newline --------- Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: leahwicz <[email protected]> * Adding performance modeling for 1.4.6 to refs/heads/main (#7532) * adding performance baseline for 1.4.6 * Add newline --------- Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: leahwicz <[email protected]> * CT 2516 ensure that paths in Jinja context flags object are strings (#7678) * Relaxed the pyyaml dependency to >=5.3. (#7681) * Updating CODEOWNERS to consolidate (#7693) Consolidating to remove Language and Execution and instead default to the `core-team` * CT 2590 write pub artifact to log (#7686) * Bumping version to 1.6.0b2 and generate changelog * Adding link to 1.5 release notes (#7707) * Add `%` to adapter suite test cases for `persist_docs` (#7699) * Test table/view/column-level comments with `%` symbol * Test docs block with `%` symbol * Changelog entry * Update to reusable workflow for branch testing (#7676) * fix overlooked node12 case with abandonded marketplace action * update slack notification * remove spaces per formatting * replace with cli dispatch * move conditional * add explicit token, temp comment out slack * add checkout * checkout teh right branch * switch to PAT * add back repo checkout * manually check workflow status * fix notification * swap to reusable workflow * fix path * swap permissions * remove trigger * fix secrets * point to main * fix #7300: Enable state for deferral to be separate from state for selectors (#7690) * Target path should be relative to project dir, rather than current working directory (#7706) * Adding performance modeling for 1.2.0 to refs/heads/main (#7560) * adding performance baseline for 1.2.0 * Adding newline --------- Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: leahwicz <[email protected]> * Readd exp_path for config deprecation warnings (#7536) * Dropped support for Python 3.7 (#7623) * Template rendering optimization (#7451) * CT-2478: Template rendering optimization * CT-2478: Fix type annotation, and accomodate non-string unit test cases. * fix StopIteration error when publication not found (#7710) * add ability to select models by access (#7739) * add ability to select models by access * changie * Update core/dbt/graph/selector_methods.py * pass optional sql_header to empty subquery sql rendering (#7734) * Fix doc link in selector.py (#7755) * Fix doc link in selector.py * Ran changie to modify changelog entry * Fix constraint rendering for expressions and foreign key constraint types (#7512) * add project_name to manifest metadata (#7754) * Version 0 for model works for latest_version (#7712) * force dependency between test models (#7767) * remove entire changes folder (#7766) Co-authored-by: Kshitij Aranke <[email protected]> * log PublicationArtifactAvailable even when partially parsing & public models unchanged (#7783) * Fixes #7299: dbt retry (#7763) * Fix not equals comparison to be null-safe for adapters/utils tests (#7776) * Fix names within functional test * Changelog entry * Test for implementation of null-safe equals comparison * Remove duplicated where filter * Fix null-safe equals comparison * Fix tests for `concat` and `hash` by using empty strings () instead of `null` * Remove macro namespace interpolation * Improve warnings for constraints and materialization types (#7696) * first pass * debugging * regen proto types * refactor to use warn_supported flag * PR feedback * Package-namespaced generate_x_name macro resolution (#7509) * generalize BaseModelConstraintsRuntimeEnforcement (#7805) * Fixes #7551: Create `add_from_artifact` to populate `state_relation` field of nodes (#7796) * Validate public models are not materialized as ephemeral (#7794) * fix error message for empty/None: --warn-error-options handling (#7735) * Adap 496/add test connection mode to debug (#7741) * --connection-flag * Standardize the plugin functions used by DebugTask * Cleanup redundant code and help logic along. * Add more output tests to add logic coverage and formatting. * Code review --------- Co-authored-by: Mila Page <[email protected]> * fix: removing dependency from dependencies.yml (#7743) * ADAP-2: Materialized Views (#7239) * changie * ADAP-387: Stub materialized view as a materialization (#7211) * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-387: reverting db_api implementation (#7322) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * added tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * add enum to relation for comparison in jinja --------- Co-authored-by: Mike Alfare <[email protected]> * ADAP-391: Add configuration change option (#7272) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * move up pre-existing drop of backup * change relation type to view to be consistent * add base test case * fix jinja exception message expression, basic test passing * align names between postgres and default to same convention * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integration tests * add on_configuration_change to expected_manifest to pass functional tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * add alter materialized view macro * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * updated default option for on_configuration_change on NodeConfig * removed explicit calls to enum values * add test setup for testing fail config option * updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name * setup configuration change framework * skipped tests that are expected to fail without adapter implementation * cleaned up log checks --------- Co-authored-by: Mike Alfare <[email protected]> * ADAP-388: Stub materialized view as a materialization - postgres (#7244) * move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-402: Add configuration change option - postgres (#7334) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * add drop and show macros for indexes * add drop and show macros for indexes * add logic to determine the indexes to create or drop * pulled index updates through the workflow properly * convert configuration changes to fixtures, implement index changes into tests * created Model dataclass for readability, added column to swap index columns for testing * fixed typo --------- Co-authored-by: Matthew McKnight <[email protected]> * ADAP-395: Implement native materialized view DDL (#7336) * changie * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * updating to feature main * updating to feature main * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * fixed list being passed as string bug * removed explicit calls to enum values * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395 # Conflicts: # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # plugins/postgres/dbt/include/postgres/macros/adapters.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py # tests/functional/materializations/test_materialized_view.py * moved postgres implemention into plugin directory * update index methods to align with the configuration update macro * added native ddl to postgres macros * removed extra docstring * updated references to View, now references MaterializedView * decomposed materialization into macros * refactor index create statement parser, add exceptions for unexpected formats * swapped conditional to check for positive state * removed skipped test now that materialized view is being used * return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test * add check for indexes in particular for apply on configuration scenario * removed extra argument * add materialized views to get_relations / list_relations * typos in index change logic * moved full refresh check inside the build sql step --------- Co-authored-by: Matthew McKnight <[email protected]> * removing returns from tests to stop logs from printing * moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures * fixed overwrite issue, simplified assertion method * updated import order to standard * fixed test import paths * updated naming convention for proper test collection with the test runner * still trying to make the test runner happy * rewrite index updates to use a better source in Postgres * break out a large test suite as a separate run * update `skip` and `fail` scenarios with more descriptive results * typo * removed call to skip status * reverting `exceptions_jinja.py` * added FailFastError back, the right way * removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports * removed assumed models in method calls, removed odd insert records and replaced with get row count * fixed index issue, removed some indirection in testing * made test more readable * remove the "apply" from the tests and put it on the base as the default * generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility * fixed type to be more generic to accommodate adapters with their own relation types * fixed all the broken index stuff * updated on_configuration_change to use existing patterns * updated on_configuration_change to use existing patterns * reflected update in tests and materialization logic * reflected update in tests and materialization logic * reverted the change to create a config object from the option object, using just the option object now * reverted the change to create a config object from the option object, using just the option object now * modelled database objects to support monitoring all configuration changes * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * reverted centralized framework, retained a few reusable base classes * updated names to be more consistent * readability updates * added readme specifying that `relation_configs` only supports materialized views for now --------- Co-authored-by: Matthew McKnight <[email protected]> Co-authored-by: Matthew McKnight <[email protected]> * Add SemanticModel Node Type (#7769) * Add dbt-semantic-interfaces as a dependency With the integration with MetricFlow we're taking a dependency on `dbt-semantic-interfaces` which acts as the source of truth for protocols which MetricFlow and dbt-core need to agree on. Additionally we're hard pinning to 0.1.0.dev3 for now. We plan on having a less restrictive specification when dbt-core 1.6 hits GA. * Add implementations of DSI Metadata protocol to nodes.py * CT-2521: Initial work on adding new SemanticModel node * CT-2521: Second rough draft of SemanticModels * CT-2521: Update schema v10 * CT-2521: Update unit tests for new SemanticModel collection in manifest * CT-2521: Add changelog entry * CT-2521: Final touches on initial implementation of SemanticModel parsing * Change name of Metadata class to reduce potential for confusion * Remove "Replaceable" inheritance, per review * CT-2521: Rename internal variables from semantic_models to semantic_nodes * CT-2521: Update manifest schema to reflect change --------- Co-authored-by: Quigley Malcolm <[email protected]> * Update spec MetricNode for dbt x MetricFlow integration and begin outputting semantic manifest artifact (#7812) * Refactor MetricNode definition to satisfy DSI Metric protocol * Fix tests involving metrics to have updated properties * Update UnparsedMetricNode to match new metric yaml spec * Update MetricParser for new unparsed and parsed MetricNodes * Remove `rename_metric_attr` We're intentionally breaking the spec. There will be a separate tool provided for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming things like `type` to `calculation_method`. This is problematic because `type` is on the new spec, while `calculation_method` is not. Additionally, since we're intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be used for any property renaming. * Fix tests for Metrics (1.6) changes * Regenerated v10 manifest schema and associated functional test artifact state * Remove no longer needed tests * Skip / comment out tests for metrics functionality that we'll be implementing later * Begin outputting semantic manifest artifact on every run * Drop metrics during upgrade_manifest_json if manifest is v9 or before * Update properties of `minimal_parsed_metric_dict` to match new metric spec * Add changie entry for metric node breaking changes * Add semantic model nodes to semantic manifest * Bumping version to 1.6.0b3 and generate changelog * Rm spaces from NodeType strings (#7842) * Rm space from NodeType strings * Add changelog entry * Add new index.html and changelog yaml files from dbt-docs (#7836) * Move remaining unit tests to `/tests` directory (#7843) * moved remaining unit tests to directory * fix path * Delete profiles.yml * rename remaining test refs * Use project directory in path selector instead of cwd (#7829) * Use contextvar to store and get project_root for path selector method * Changie * Modify test to check Path selector with project-dir * Don't set cv_project_root in base task if no config * update adapters url (#7779) * update adapters url in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed. old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli thank you @dbeatty10 for your 🦅 👀 ! * adding changie entry * Update .changes/unreleased/Breaking Changes-20230612-161159.yaml --------- Co-authored-by: Emily Rockman <[email protected]> * Allow ProjectDependency to have extra fields (#7834) * Allow ProjectDependency to have extra fields * changelog * Respect column `quote` config in model contracts (#7537) * Update SemanticModel node to match DSI 0.1.0dev3 protocols (#7848) * Add tests to ensure our semantic layer nodes satisfy the DSI protocols These tests create runtime checkable versions of the protocols defined in DSI. Thus we can instantiate instances of our semantic layer nodes and use `isinstance` to check that they satisfy the protocol. These `runtime_checkable` versions of the protocols should only exist in testing and should never be used in the actual package code. * Update the `Dimension` object of `SemanticModel` node to match DSI protocol * Make `UnparsedDimension` more strict and update schema readers accordingly * Update the `Entity` object of `SemanticModel` node to match DSI protocol * Make `UnparsedEntity` more strict and update schema readers accordingly * Update the `Measure` object of `SemanticModel` node to match DSI protocol * Make `UnparsedMeasure` more strict and update schema readers accordingly * Update the `SemanticModel` node to match DSI protocol A lot of the additions are helper functions which we don't actually use in core. This is a known issue. We're in the process of removing a fair number of them from the DSI protocol spec. However, in the meantime we need to implement them to satisfy the protocol unfortunately. * Make `UnparsedSemanticModel` more strict and update schema readers accordingly * Changie entry for updating SemanticModel node * Bumping version to 1.6.0b4 and generate changelog * Fix: `dbt show --inline` with `private` models (#7838) * Add functional test * Check resource_type before DbtReferenceError * Changelog entry * Bump mypy from 0.981 to 1.0.1 (#7027) * Bump mypy from 0.981 to 1.0.1 Bumps [mypy](https://github.com/python/mypy) from 0.981 to 1.0.1. - [Release notes](https://github.com/python/mypy/releases) - [Commits](https://github.com/python/mypy/compare/v0.981...v1.0.1) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Add automated changelog yaml from template for bot PR * upgrade mypy and fix all errors * fixing some duplicate imports from conflict resolution * fix mypy errors from merging in main --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: Emily Rockman <[email protected]> * Add AdapterRegistered event log message (#7862) * Add AdapterRegistered event log message * Add AdapterRegistered to unit test * make versioning and logging consistent * make versioning and logging consistent * add to_version_string * remove extra equals * format fire_event * Allow setting packages in dependencies.yml and move dependencies to runtime config (#7857) * [CT-2696] Skip jinia parsing of metric filters (#7885) * Update metric filters in testing fixtures I incorrectly wrote the tests such that they didn't include curly braces, `{{..}}`, around things like `dimension(..)` for filters. This updates the tests fixtures to have proper filter specifications * Skip jinja rendering of `filter` key of metrics Note that `filter` can show up in multiple places: as a root key on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`), denominator (`metric.type_params.denominator.filter`), numerator (`metric.type_params.numerator.filter`), and a metric input measure (`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`). In this commit we skip all of them :) * Add changie doc for skipping jinja parsing for metric filters * Update yaml renderer test for metrics * Ct-2690 Support dbt-semantic-interfaces 0.1.0dev5 (#7888) * Upgrade to dbt-semantic-interfaces v0.1.0dev5 This is a fairly simple upgrade. Literally it's just pointing at the the new versions. The v3 schemas are directly compatible with v5 because there were no protocol level changes from v3 to v5. All the changers were updates to tools MetricFlow uses from DSI, not tools that we ourselves are using in core (yet). * Add changie doc for DSI version bump * Upgrade core to support dbt-semantic-interfaces `0.1.0dev7` (#7903) * Bump DSI dependency version to 0.1.0dev7 * Cleaner DSI type enum importing Previoulsy we had to use individual import paths for each type enum that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces has been updated to allow for importing all the type enums from a singular path. * Cleaner DSI protocol importing Previoulsy we had to use individual import paths for each protocol that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces has been updated to allow for importing all the protocols from a singular path. * Add semantic protocol satisifcation test for metric type params * Replace `metric.type_params.measures` with `metric.type_params.input_measures` In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`. Additionally `input_measures` should not be user specified but something we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`. Finally, actually populating `input_measures` is somewhat complicated due to the existance of derived metrics, thus that work is being pushed off to CT-2707. * Update metric numerator/denominator to be `MetricInput`s In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator` switched from being `MetricInputMeasure`s to `MetricInput`s. This commit reflects that change. Additionally, some helper functions on metric type params were removed related to the numerator and denominator. Thus we've removed them respectively in this commit. * Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure` * Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure` DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol, thus we've added it to our implementation in core. Additionally, we had a typo in a method name in our implementation of `MetricInputMeasure`, ironically a similar function to the one we've added for `MetricInput` * Changie doc for upgraded to DSI 0.1.0dev7 * Fix parsing of metric numerator and denominator in schema_yaml_readers Previously numerator and denominator of a metric were `MetricInputMeasure`s, now they're `MetricInput`s. Changing the typing isn't enough though. We have parsing functions in `schema_yaml_readers` which were specifically parsing the numerator and denominator as if they were `MetricInputMeasure`s. Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s. During this we had some logic in a parsing function `_get_metric_inputs` which could be abstracted to newly added functions. * Fixes #7753: Fix regression in `run-operation` to not require the name of the package to run (#7811) * Reorganize, annotate, and revise dependency pins (#7368) * Reorganize + annotate dependencies * Loosen pins on other dbt-labs packages * Minor pins, no patch pins * Rm support for py37 * Fix protobuf pin * Bump networkx upper bound #6551 * Do not jinja render packages from dependencies yml (#7910) * Add some comments to methods constructing Project/RuntimeConfig * Save flag that packages dict came from dependencies.yml * Test for not rendering packages_dict * Changie * Ensure packages_yml_dict and dependencies_yml_dict are dictionaries * Ensure "packages" passed to render_packages is a dict * Remove --config-dir instead of fixing #7774 (#7793) * Stringfy the dir always to solve the bug. * Add changelog --------- Co-authored-by: Mila Page <[email protected]> * Refactor/unify public and model nodes (#7891) * Further Integrate Semantic Models (#7917) * CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code * CT-2651: Finish integrating semantic models into the partial parsing system * CT-2651: More semantic model details for partial parsing * CT-2651: Remove merged references to project_dependencies * CT-2651: Revise changelog entry * CT-2651: Disable unit test until partial parsing of semantic models is complete. * CT-2651: Temporarily disable an apparently-flaky test. * Resolve SemanticModel refs in the same way as other refs (#7895) * UninstalledPackagesFoundError references correct packages specified path (#7886) * Fix: safe remove of external nodes from nodes.depends_on (#7923) * Bumping version to 1.6.0b5 and generate changelog * CT-2719: Rename the semantic_nodes collection on the manifest to semantic_models (#7927) * Improved Semantic expr specification handling (#7876) * Update semantic model parsing test to check different measure expr types * Allow semantic model measure exprs to be defined with ints and bools in yaml Sometimes the expr for a measure can defined in yaml with a bool or an int. However, we were only allowing for strings. There was a work around for this, which was wrapping your bool or int in double quotes in the yaml, but this can be fairly annoying for the end user. * Changie doc for fixing measure expr yaml specification * Improved Semantic Model Measure Percentile defaults (#7877) * Update semantic model parsing test to check measure agg params * Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false This was a mistake in our implementation of the MeasureAggregationParams. We had defined them as optional and defaulting to `None`. However, as the protocol states, they cannot be `None`, they must be a boolean value. Thus now we now ensure them. * Add changie doc for measure percentile fixes * CT-2732: Fix selector methods to include semantic models (#7936) * Bumping version to 1.6.0b6 and generate changelog * update mashumaro to 3.8.1 (#7951) * Update mashumaro to 3.8 * Change to 3.8.1 * Changie * Fix comment for `dbt retry` (#7932) * Factor Out Repeated Logic in the PartialParsing Class (#7952) * CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization. * CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes() * CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing() * Fix macro namespace search packages (#5804) * allow setting enabled and depends_on_nodes from ModelNodeArgs (#7930) * Set access model node args (#7966) * Add Partial Parsing Support for Semantic Models (#7964) * CT-2711: Add partial parsing support for semantic models * CT-2711: Fix typo identified in code review * Add merge incremental strategy for postgres (#6951) * Add merge incremental strategy * Expect merge to be a valid strategy for Postgres --------- Co-authored-by: Anders Swanson <[email protected]> Co-authored-by: Doug Beatty <[email protected]> * Handle state:modified for external nodes (#7925) * Add thread_id context var (#7942) * Add thread_id context var * Changie * Fix context test * Update .changes/unreleased/Features-20230623-173357.yaml Co-authored-by: Doug Beatty <[email protected]> * Fix tests --------- Co-authored-by: Doug Beatty <[email protected]> * Add invocation command flag (#7939) * Add invocation command flag * Add changie entry * Update .changes/unreleased/Features-20230623-111254.yaml * consolidate cross-project ref entrypoint + plugin framework (#7955) * Add restrict-access to dbt_project.yml (#7962) * Bumping version to 1.6.0b7 and generate changelog * Fix CTE insertion position when the model uses WITH RECURSIVE (#7350) (#7414) * Replace space with underscore in NodeType strings (#7947) * Use events.contextvar because of multiprocessing unable to pickle ContextVar (#7949) * Add task contextvars to events/contextvars.py * Use events.contextvars instead of task.contextvars * Changie * CT-2757: Fix unit test which broke due to merge issues (#7978) * `dbt clone` (#7881) Co-authored-by: Matthew McKnight <[email protected]> * CT-2707: Populate metric input measures (#7984) * Fix tests fixtures which were using measures for metric numerator/denominators In our previous upgrade to DSI dev7, numerators and denominators for metrics switched from being `MetricInputMeasure`s to `MetricInput`s. I.e. metric numerators and denominators should references other metrics, not semantic model measures. However, at that time, we weren't actually doing anything with numerators and denominators in core, so no issue got raised. The changes we are about to make though are going to surface these issues.. * Add tests for ensuring a metric's `input_measures` gets properly populated * Begin populating `metric.type_params.input_measures` This isn't my favorite bit of code. Mostly because there are checks for existence which really should be handled before this point, however a good point for that to happen doesn't exist currently. For instance, in an ideal world by the time we get to `_process_metric_node`, if a metric is of type `RATIO` and the nominator and denominator should be guaranteed. * Update test checking that disabled metrics aren't added to the manifest metrics We updated from the metric `number_of_people` to `average_tenure_minus_people` for this test because disabling `number_of_people` raised other exceptions at parse time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people` is a leaf metric, and so for this test, it is a better candidate. * Update `test_disabled_metric_ref_model` to have more disabled metrics There are metrics which depend on the metric `number_of_people`. If `number_of_people` is disabled without the metrics that depend on it being disabled, then a different (expected) exception would be raised than the one this test is testing for. Thus we've disabled those downstream metrics. * Add test which checks that metrics depending on disabled metrics raise an exception * Add changie doc for populating metric input measures * Unified to UTC (#7665) * UnifiedToUTC * Check proximity of dbt_valid_to and deleted time * update the message to print if the assertion fails * add CHANGELOG entries * test only if naive * Added comments about naive and aware * Generalize comparison of datetimes that are "close enough" --------- Co-authored-by: Doug Beatty <[email protected]> * type + fix typo in ModelNodeArgs.unique_id (#7992) * Remove pin of sqlparse, minor refactoring, add tests (#7993) * [CT-2551] Make state selection MECE (#7773) * ct-2551: adds old and unmodified state selection methods * ct-2551: update check_unmodified_content to simplify * add unit and integration tests for unmodified and old * add changelog entry * ct-2551: reformatting of contingent adapter assignment list * Added note before running integration tests (#7657) Co-authored-by: Kshitij Aranke <[email protected]> * Add time spine table configuration to semantic manifest (#7996) * Bump dbt-semantic-interface to dev8 * Create time_spline_table_configuration in semantic manifest * Add metricflow_time_spin to semantic_models tests * Remove skip from test * Changie * Update exception message Co-authored-by: Quigley Malcolm <[email protected]> --------- Co-authored-by: Quigley Malcolm <[email protected]> * allow on_schema_change: fail for incremental models with contracts (#8006) * versioned node selection with underscore delimiting (#7995) * Bumping version to 1.6.0b8 and generate changelog * fire proper event for inline query error (#7960) * CT-2752: Extract methods to new SemanticManifest class for better encapsulation of details. (#8012) * CT-2780 [Docs] Fix-toc-links-in-contributing-md (#8017) * docs(contributing): fix-toc-link-in-contributing-md * docs(contributing-md): fix-link2 * Fix-typo * Remove backtick from href * changie new * Cough commit / trigger CI * Nicer error message for contracted model missing 'columns' (#8024) * ModelNodeArgs.unique_id - include v (#8038) * pin click (#8050) * pin click * changelog * add workflow for flaky test testing (#8044) * add workflow for flaky test testing * improve docs * rename workflow * update default input * add min passing tests * partial parse file path (#8032) * Update the PR template (#7892) * add interface changes section to the PR template * update entire template * split up choices for tests and interfaces * minor formatting change * add line breaks * actually put in line breaks * revert split choices in checklist * add line breaks to top * move docs link * typo * Semantic Model Validation (#8049) * Use dbt-semantic-interface validations on semantic models and metrics defined in Core. * Remove empty test, since semantic models don't generate any validation warnings. * Add changelog entry. * Temporarily remove requirement that there must be semantic models definied in order to define metrics * Fix accidental propagation of log messages to root logger (#7882) * Fix accidental propagation of log messages to root logger. * Add changelog entry * Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used. * add negative part number test case for split part cross db util (#7200) * add negative test case * changie * missed a comma * Update changelog entry * Add a negative number (rather than subtract a positive number) --------- Co-authored-by: Doug Beatty <[email protected]> Co-authored-by: Doug Beatty <[email protected]> * Remove `create_metric` as a public facing `SemanticModel.Measure` property (#8068) * Remove `create_metric` as a public facing `SemanticModel.Measure` property We want to add `create_metric`. The `create_metric` property will be incredibly useful. However, at this time it is not hooked up, and we don't have time to hook it up before the code freeze for 1.6.0rc of core. As it doesn't do anything, we shouldn't allow people to specify it, because it won't do what one would expect. We plan on making the implementation of `create_metric` a priority for 1.7 of core * Changie doc for the removal of create_metric property * Flaky Test Workflow (#8055) * add permissions * replace db setup * try with bash instead of just pytest flags * fix test command * remove spaces * remove force-flaky flag * add starting vlaues * add mac and windows postgres isntall * define use bash * fix typo * update output report * tweak last if condition * clarify failures/successful runs * print running success and failure tally * just output pytest instead of capturing it * set shell to not exit immediately on exit code * add formatting around results for easier scanning * more output formatting * add matrix to unlock parallel runners * increase to ten batches * update debug * add comment * clean up comments * Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release (#8071) * Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release * Changelog entry * Fixes #7785: fail-fast behavior (#8066) * Add validate_sql method to base adapter with implementation for SQLAdapters (#8001) * Add dry_run method to base adapter with implementation for SQLAdapters resolves #7839 In the CLI integration, MetricFlow will issue dry run queries as part of its warehouse-level validation of the semantic manifest, including all semantic model and metric definitions. In most cases, issuing an `explain` query is adequate, however, BigQuery does not support the `explain` keyword and so we cannot simply pre-pend `explain` to our input queries and expect the correct behavior across all contexts. This commit adds a dry_run() method to the BaseAdapter which mirrors the execute() method in that it simply delegates to the ConnectionManager. It also adds a working implementation to the SQLConnectionManager and includes a few test cases for adapter maintainers to try out on their own. The current implementation should work out of the box with most of our adapters. BigQuery will require us to implement the dry_run method on the BigQueryConnectionManager, and community-maintained adapters can opt in by enabling the test and ensuring their own implementations work as expected. Note - we decided to make these concrete methods that throw runtime exceptions for direct descendants of BaseAdapter in order to avoid forcing community adapter maintainers to implement a method that does not currently have any use cases in dbt proper. * Switch dry_run implementation to be macro-based The common pattern for engine-specific SQL statement construction in dbt is to provide a default macro which can then be overridden on a per-adapter basis by either adapter maintainers or end users. The advantage of this is users can take advantage of alternative SQL syntax for performance or other reasons, or even to enable local usage if an engine relies on a non-standard expression and the adapter maintainer has not updated the package. Although there are some risks here they are minimal, and the benefit of added expressiveness and consistency with other similar constructs is clear, so we adopt this approach here. * Improve error message for InvalidConnectionError in test_invalid_dry_run. * Rename dry_run to validate_sql The validate_sql name has less chance of colliding with dbt's command nomenclature, both now and in some future where we have dry-run operations. * Rename macro and test files to validate_sql * Fix changelog entry * Update validate sql test classes to new nomenclature (#8076) The original implementation of validate_sql was called dry_run, but in the rename the test classes and much of their associated documentation still retained the old naming. This is mainly cosmetic, but since these test classes will be imported into adapter repositories we should fix this now before the wrong name proliferates. * Remove volume declaration (#8069) * Remove volume declaration * Changelog entry --------- Co-authored-by: Doug Beatty <[email protected]> * update contributing.md reference to test/integration (#8073) * Er/ct 2675 test custom target (#8079) * remove skip * fix retry test * 8030/fix contract checksum (#8072) * Add semantic_models to tracked resource counts (#8078) * Add semantic_models to tracked resource counts * Add changelog entry * Simplify node statistic tabulation. * Remove review comment. Replace with explanation. --------- Co-authored-by: Peter Allen Webb <[email protected]> * CT-2691 Fix the populating of a `Metric`'s `depends_on` property (#8015) * Add metrics from metric type params to a metric's depends_on * Add Lookup utility for finding `SemanticModel`s by measure names * Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on` * Add `SemanticModelConfig` to `SemanticModel` Some tests were failing due to `Metric`'s referencing `SemanticModels`. Specifically there was a check to see if a referenced node was disabled, and because `SemanticModel`'s didn't have a `config` holding the `enabled` boolean attr, core would blow up. * Checkpoint on test fixing * Correct metricflow_time_spine_sql in test fixtures * Add check for `SemanticModel` nodes in `Linker.link_node` Now that `Metrics` depend on `SemanticModels` and `SemanticModels` have their own dependencies on `Models` they need to be checked for in the `Linker.link_node`. I forget the details but things blow up without it. Basically it adds the SemanticModels to the dependency graph. * Fix artifacts/test_previous_version_state.py tests * fix access/test_access.py tests * Fix function metric tests * Fix functional partial_parsing tests * Add time dimension to semantic model in exposures fixture * Bump DSI version to a minimum of 0.1.0dev10 DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension` setting. This incoherence was that `measure.agg_time_dimension` was being required, even though it was no longer supposed to be a required attribute (it's specificially typed as optional in the protocol). This was causing a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension` value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes the issue. Interestingly after bumping the DSI version, the integration tests were still failing. If I ran the tests individually they passed though. To get `make integration` to run properly I ended up having to clear my `.tox` cache, as it seems some outdated state was being persisted. * Add test specifically for checking the `depends_on` of `Metric` nodes * Re-enable test asserting calling metric nodes in models * Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure` DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure` protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`. This finishes a change where for a given measure either the `Measure.agg_time_dimension` or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be set, instead of always require the measure's `Measure.agg_time_dimension`. * Add changie doc for populating metric --------- Co-authored-by: Gerda Shank <[email protected]> * add __test__ = False to non-test classes that start with Test (#8086) * gitignore user.yml and profiles.yml (#8087) * [CT-2821] Support `dbt-semantic-interfaces~=0.1.0rc1` (#8085) * Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1` * Add tests for asserting WhereFilter satisfies protocol * Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol * Changie doc for moving to DSI 0.1.0rc1 * [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089) * Add test to ensure `NonAdditiveDimension` implementation satisfies protocol * Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings` * Add changie doc for typo fix in NonAdditiveDimension * Fix query comment tests (#7928) (#7928) * add env vars for datadog ci visibility (#8097) * add env vars for datadog ci visibility * modify pytest command for tracing * fix posargs * move env vars to job that needs them * add test repeater to DD * swap flags * Apply new integration tests to existing framework to identify supported features (#8099) * applied new integration tests to existing framework * applied new integration tests to existing framework * generalized tests for reusability in adapters; fixed drop index issue * generalized tests for reusability in adapters; fixed drop index issue * removed unnecessary overrides in tests * adjusted import to allow for usage in adapters * adjusted import to allow for usage in adapters * removed fixture artifact * generalized the materialized view fixture which will need to be specific to the adapter * unskipped tests in the test runner package * corrected test condition * corrected test condition * added missing initial build for the relation type swap tests * bumpversion 1.7.0a1 (#8111) * Cleanup main after cutting new 1.6.latest branch (#8102) * Publish coverage results to codecov.io (#8127) * wrap deprecation warnings in warn_or_error calls (#8129) * Add new index.html and changelog yaml files from dbt-docs (#8131) * Rm docs changelog entry (#8147) * Rearrange pp_versioned_models test (#8150) * format exception from dbtPlugin.initialize (#8143) * Hotfix for 372: Use JSONEncoder in json.dumps (#8151) * Correcting spelling of partition (#8101) * Correcting spelling of partition * Changie entry * add a node status (#8174) * Add implementation issue template (#8176) * [CT-1849] `_connection_exception_retry` handles `EOFError` exceptions (#8182) * Add test for checking that `_connection_exception_retry` handles `EOFError`s * Update `_connection_exception_retry` to handle `EOFError` exceptions * Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions * Fixed double underline (#7944) * Fixed double-underline * backward compatibility postgres_get_relations * Remove invalid comments * compatibility adapter and get_relation * fix generic for call * fix adapter dispatch grammar issue * [CT-1483] Let macro names include word materialization (#8181) * Add test asserting that a macro with the work materializtion doesn't cause issues * Let macro names include the word `materialization` Previously we were checking if a macro included a materialization based on if the macro name included the word `materialization`. However, a macro name included the word `materialization` isn't guarnteed to actually have a materialization, and a macro that doesn't have `materialization` in the name isn't guaranteed to not have a materialization. This change is to detect macros with materializations based on the detected block type of the macro. * Add changie doc materialization in macro detection * [CT-2594] Fix serialization of `warn_error_options` on Contexts (#8180) * Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts * Add dictification specific to `warn_error_options` in `args_to_dict` * Changie doc for serialization changes of warn_error_options * Check for existing_relation immediately prior to renaming (#8193) * add param to control maxBytes for single dbt.log file (#8200) * add param to control maxBytes for single dbt.log file * nits * nits * Update core/dbt/cli/params.py Co-authored-by: Peter Webb <[email protected]> --------- Co-authored-by: Peter Webb <[email protected]> * Update Core for Latest dbt-extractor with Version Parsing (#8206) * Update dbt-extractor requirement, and adjust ref handling accordingly * Add changelog entry. * Update implementation-ticket.yml (#8208) * up timeout (#8218) * Bump mypy from 1.3.0 to 1.4.0 (#7912) * Bump mypy from 1.3.0 to 1.4.0 Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.0. - [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Add automated changelog yaml from template for bot PR * add to pre-commit config --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: Emily Rockman <[email protected]> * Initialize sqlparse lexer and tweak order of setting compilation fields (#8215) * Split integration tests into parallel groups / jobs (#6346) * Ensure that target_schema from snapshot config is promoted to node level (#8117) * pin upper bound for sqlparse (#8236) * pin upper bound for sqlparse * changelog * [CT-2879] Fix unbound variable error in `checked_agg_time_dimension_for_measure` (#8235) * Fix unbound variable error in `checked_agg_time_dimension_for_measure` * Improve assertion error message in `checked_agg_time_dimension_for_measure` * Add changie doc for checked_agg_time_dimension_for_measure unbound variable fix …
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #7499
resolves #7503
Description
Checklist
changie newto create a changelog entry