Fix for #89645, stack overflow in Crossgen2 #90229
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After David Wrighton's refactoring of type loadability check in dotnet#89415 we started seeing stack overflow in Crossgen2 compilation of the outerloop test Loader/classloader/generics/regressions/DD117522/Test.csproj This is because the test is a negative test that exercises runtime behavior in the presence of a non-loadable type with recursive definition. David's stricter descent into the type ends up in an infinite recursion when presented with this invalid type. I haven't found any easy way to incorporate the additional check for recursive types into the loadability algorithm - in fact I'm not even sure whether that's generally doable. As a very simple way to protect against the infinite recursion I propose adding a heuristic limit for the type analysis stack size. I assume the proposed value 1024 to be more than enough for both Crossgen2 and NativeAOT, if it's realistic that NativeAOT can encounter deeper types than this, I can make the check specific for Crossgen2. Thanks Tomas Fixes: dotnet#89645
ghost
assigned 
It is doable. The algorithm is described in ECMA spec and implemented in CoreCLR by this call: runtime/src/coreclr/vm/methodtable.cpp Line 5283 in 3b21b90 |
It is likely still going to crash on macOS since the default stacktrace on macOS is much smaller currently. (We have a few other places running into this problem - tracked by #87879.) |
jkotas
approved these changes
Aug 10, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After David Wrighton's refactoring of type loadability check in #89415 we started seeing stack overflow in Crossgen2 compilation of the outerloop test
Loader/classloader/generics/regressions/DD117522/Test.csproj
This is because the test is a negative test that exercises runtime behavior in the presence of a non-loadable type with recursive definition. David's stricter descent into the type ends up in an infinite recursion when presented with this invalid type.
I haven't found any easy way to incorporate the additional check for recursive types into the loadability algorithm - in fact I'm not even sure whether that's generally doable.
As a very simple way to protect against the infinite recursion I propose adding a heuristic limit for the type analysis stack size. I assume the proposed value 1024 to be more than enough for both Crossgen2 and NativeAOT, if it's realistic that NativeAOT can encounter deeper types than this, I can make the check specific for Crossgen2.
Thanks
Tomas
Fixes: #89645
/cc @dotnet/crossgen-contrib