-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable all configurations inner loop #108848
Enable all configurations inner loop #108848
Conversation
This reverts commit 7b69459.
Testing to determine if building tests were the source of OOMs in CI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fine by me. It probably is still worth investigating if there are leaks.
I agree, that's why I didn't want to resolve #108821. The point of this was to try to see if a "normal" workload would work correctly. I suspect the thing that's OOMing is the test build that's running the HotReload tools during the build. If that's the problem then we could isolate that and even test it with different SDKs to see if it's regressed in RC1. |
We don't have libs.tests -allconfigurations coverage when building on i.e. Windows which is why I enabled it in this leg couple months ago. See #106474 for more details. I wouldn't want to loose coverage again if possible. |
Yep, keep that issue open - if you'd like to investigate it @ViktorHofer have at it. It could be a 9.0 regression causing the OOM. |
FWIW we've already got a complete build of the failing leg: https://dev.azure.com/dnceng-public/public/_build/results?buildId=841921&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857 Done in 22 minutes, no memory warnings. |
Ran a couple more instances of this pipeline to test: |
Both of my reruns passed cleanly. |
/ba-g build analysis is flagging a build I canceled and isn't replacing that with additional runs of the same pipeline. |
Related: #108821
Reverts #108581
Rather than disable the entire leg, let's try just disabling the build of tests. It's possible the build of tests concurrent with the product was what was causing the OOMs. I see the tests running lots of resource intensive tools which could cause this problem.
We do have other coverage of
libs.tests
on linux so we don't necessarily need that here.We'll want to run
runtime_dev_innerloop
enough to convince ourselves this is safe.@jkotas @steveisok