Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve rpc server compilation performance #1824

Closed
drewbanin opened this issue Oct 10, 2019 · 0 comments · Fixed by #1830
Closed

Improve rpc server compilation performance #1824

drewbanin opened this issue Oct 10, 2019 · 0 comments · Fixed by #1830
Labels
enhancement New feature or request rpc Issues related to dbt's RPC server

Comments

@drewbanin
Copy link
Contributor

drewbanin commented Oct 10, 2019

Describe the feature

When the rpc server triggers a re-compilation of a dbt project, all of the nodes in the project are re-compiled. This recompilation process can be slow (especially when models need to hit the database), and yet the compiled node results are largely unused.

This compilation process is currently used to inject ephemeral models as CTEs into queries submitted via run_sql and compile_sql. Can we extricate ephemeral model injection from the rest of compilation? Or, are we otherwise able to limit the set of models that are compiled in a given re-compilation of the entire project?

Let's try to pursue an approach where the rpc server maintains a parsed, not compiled, representation of the manifest in memory. When the run_sql or compile_sql methods are invoked, only the ephemeral ancestors of the "just-in-time" rpc node should be compiled. Any "concrete" models will not need to be compiled, as their parsed representations provide all the information needed for effective compilation of the rpc node.

@drewbanin drewbanin added enhancement New feature or request rpc Issues related to dbt's RPC server labels Oct 10, 2019
@drewbanin drewbanin added this to the Louisa May Alcott milestone Oct 10, 2019
beckjake added a commit that referenced this issue Oct 14, 2019
…compile-performance

Feature/improve rpc compile performance (#1824)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request rpc Issues related to dbt's RPC server
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant