Recipe rendering very slow in certain cases

[mbargull]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Recipe rendering can be very slow in certain cases.
Recently, I was confronted with the case of https://github.com/conda-forge/arrow-cpp-feedstock for which a conda-smithy rerender apparently takes between 10 and 30 minutes.
To put things into context:
The recipe

1. is rendered 6 times by conda-smithy (for linux-64/linux-aarch64/linux-ppc64le/osx-64/osx-arm64/win-64),
2. has outputs for 12 separate packages,
3. produces 200 package builds (due to variants defined in conda-forge-pinning+.ci_support/migrations),
4. includes 68 pin_subpackage calls via jinja.

Which resulted in roughly

• 60'000 calls to .metadata.MetaData.get_recipe_text,
• 50'000 calls to .jinja_context.pin_subpackage,
• 120'000 calls to .metadata.select_lines,
• 10'000'000 calls to .metadata.eval_selector.

(I stopped the tracing profiler at somepoint, so those numbers are extrapolated; the order of magnitude should match actual numbers, though.)

Higher call counts are to be expected due to conda-build's repeated partial parsing of the recipe.
They are additionally unproportionally increased due to some oddities like .metadata.MetaData.build_id not being cached for .jinja_context.pin_subpackage at
https://github.com/conda/conda-build/blob/24.1.2/conda_build/jinja_context.py#L406
which means each pin_subpackage_against_outputs call will re-extract the recipe text over and over again due to
https://github.com/conda/conda-build/blob/24.1.2/conda_build/metadata.py#L1660 .

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

[mbargull]

cc @xhochy, @h-vetinari since you have the "pleasure" dealing with this issue when maintaining arrow-cpp-feedstock.

I've proposed gh-5225 to treat some of the symptoms of this issue.
It's not doing doing "much" (as in, the changes do not make the process >100 times faster as would actually fixing it), but at least we'd get around 4 times speedup for the arrow-cpp-feedstock case.

[mbargull]

This issue is currently being investigated.
Quick update on the recent progress:

• CodSpeed benchmarks got enabled in Enable CodSpeed and add select lines benchmark #5233 so we can better track these performance issues.
• A (considerably stripped down version of) the arrow-cpp-feedstock example was added as such a CodSpeed benchmark in Add benchmark for high pin_subpackage count recipe #5246.
• The smaller, (seemingly) non-controversial changes from Miscellaneous render speedups #5225 have been cherry-picked into Speed up rendering by avoiding costly/redundant function calls #5248 .
• Another (saner) approach with a split globally-memoized/locally-cached version of .metadata.select_lines has been proposed in Add cached _split_line_selector to avoid redundant parsing in select_lines #5237 .

[beeankha]

Closing this issue since most of the changes proposed here have been merged:

• Remove duplicate extract_package_and_build_text call #5253
• Short-circuit get_output_dicts_from_metadata's if clause for faster evaluation #5278
• Memoize conda_build.metadata._filter_recipe_text #5279
• Remove/avoid redundant function calls #5280
• Properly re.escape variant variable in find_used_variables_* #5283
• Add cached _split_line_selector to avoid redundant parsing in select_lines #5237

The only one that has not been merged yet is the one linked directly below, pending discussions and testing:

• Use pickle's loads/dumps as a faster deepcopy #5281