[Backport release_3_8] Try harder to find layers when deciding whether to cache them or not.

[Backport release_3_8] Try harder to find layers when deciding whether to cache them or not. #3021