[Tracking] Static / Experimental Parser (Tree Sitter)

[nathaniel-may]

Proof of Concept (Completed 04/20/21)

• Write tree-sitter grammar for subset of jinja and unit test.
• Extract refs sources and configs from model file.
• Detect false positives and misses with sample projects- both must be zero.
• Determine if the percent of model files affected is worth it to include in the main product. (result = 60%)

0.20.0-RC1 (Completed 05/20/21)

• dbt with the experimental parser passes unit tests
• Run internal analytics project with and without the experimental parser. Manifest files must compare as exactly the same.
• dbt with the experimental parser passes integration tests
• Make a packaging strategy for the new parser + dbt.
• Split tree-sitter repository from extractor
• Create separate CI pipeline for tree-sitter repository to create wheels and upload to pypi (Thank you, Kyle!)
• Rerun integration tests once the tree-sitter dependency is installable from pypi.
• Add --use-experimental-parser flag to core, defaulting to false.

0.20.0-RC2 (Completed 06/29/21)

• rewrite extractor and tests in rust. (PR)
• write PyO3 wrapper to expose to python ABI (write PyO3 wrapper to expose to python ABI dbt-extractor#19)
• depend on tree-sitter-jinja2 not via private repo branch (publish on crates.io, or public repo (easiest)
• add github actions to package extractor (add github actions to package extractor and publish to pypi dbt-extractor#18)
• add step to publish to pypi to github actions (add github actions to package extractor and publish to pypi dbt-extractor#18)
• rerun project test runner with rust extractor ( rewrite project test runner in rust dbt-extractor#20)
• in dbt remove extractor.py, both tree-sitter dependencies, and instead depend on the rust extractor package from pypi PR
• Add snowplow reporting to sometimes run both and send back if they match or not. (Add experimental parser correctness checks to tracking #3503)
• Add counter for how many model files were "experimentally parsed" vs "jinja rendered" to the perf file. (Update project load tracking for v0.20 parsing changes #3438)
• Add snowplow reporting to track how many models can be parsed without jinja rendering. (Update project load tracking for v0.20 parsing changes #3438)

0.20.1-RC1 (Completed 08/02/21)

• Fix merge behavior (make experimental parser respect config merge behavior #3637)

0.21.0-B2 (Completed 08/19/21)

• Make sure the model parser can detect when ref source or config has been overridden with a macro and can skip static parsing. New test for this necessary. Hard prereq for turning on by default.

Remaining Work

• Add a new flag --no-static-parser to explicitly disable the static parser
• [Optional] Infrequently sample the stable static parser against Jinja for correctness feedback.
• Turn on by default