Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation Cleanup #1602

Closed
beckjake opened this issue Jul 11, 2019 · 2 comments
Closed

Compilation Cleanup #1602

beckjake opened this issue Jul 11, 2019 · 2 comments

Comments

@beckjake
Copy link
Contributor

Feature

Clean up the compilation process and remove compilation artifacts from the results

Feature description

This is an add-on for #1589.

Currently, compiled nodes store some information on them that varies based on node type (compiled test nodes have all the same fields, so ignore the differences between them for now):

  • extra_ctes: This is changed during ref resolution (which happens in the "runtime" context, while compiled_sql is being created). Refs to ephemeral nodes are turned into appends on this array.
  • extra_ctes_injected: This is always False until extra_ctes has been processed during compilation.
  • compiled_sql: This is None at first, and then is rendered from the runtime context and the raw_sql.
  • compiled: This is always True if compiled_sql is a value, and False if it's None. This should be a property, if it even exists.
  • injected_sql: This is None at first, and is built from compiled_sql + extra_ctes (+ extra_ctes_injected, but that's just bookkeeping)
  • wrapped_sql: This is intially None on all nodes, and on data tests is 'select (count*) from (\n{injected_sql}\n) sbq'. For schema tests and hooks, it is just set to the same as wrapped_sql. Finally, all values of wrapped_sql are passed through rendering. So None becomes 'None'.

I propose we change this so that only the current wrapped_sql (for data tests) and injected_sql (for everything else) is included, under the name compiled_sql, and make everything use that. During compilation we should use a wrapper type of some sort that holds the extra bookkeeping information, instead of attaching that to the actual node spec. We can also remove the cte-related methods from the CompiledNode object, which seems good.

Who will this benefit?

Developers, people looking at the manifest

@cmcarthur
Copy link
Member

we should also write accompanying documentation for the various "states" a node can be in -- before this, "parsed", "compiled", "injected", "wrapped" were valid states though the intermediate states ("compiled", "injected") were not actually useful. "parsed", "compiled" seems like a better set of states

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jan 4, 2021

With the removal of injected_sql in favor of overwriting compiled_sql (#2762), and standardizing the artifact schemas in v0.19.0, I think we've accomplished the spirit of this issue.

We're still overdue to document the distinction between a node being parsed vs. compiled.

@jtcohen6 jtcohen6 closed this as completed Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants