Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Differences in required test baseline files between stage 1 and stage 2 builds #3389

Closed
mthalman opened this issue Apr 14, 2023 · 3 comments
Closed
Labels
area-testing Improvements in CI and testing

Comments

@mthalman
Copy link
Member

The release of 6.0.116 exposed an issue in the tests that make use of baseline files. That release caused a need to update the MissingXmlDoc.txt baseline file to include a new entry. But updating that file then caused the corresponding VerifyTargetingPacksHaveDoc test to fail in the stage 2 build (internal link) because the XML file was not missing in that build. So there's a difference in the output of the stage 1 and stage 2 build such that a single baseline file cannot describe the expected output of both scenarios.

This is expected to occur for other baseline files such as SDK diff and poison leaks. We should have a general solution that can apply to all these baseline file types.

One potential solution for this is to have two baseline files: one for the stage 1 build and the other for the stage 2 build. The pipeline would indicate which one to use via an environment variable. The baseline for the stage 2 build could be optionally such that if it does not exist, the test falls back to the stage 1 baseline.

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@MichaelSimons
Copy link
Member

@mthalman - Is this issue still relevant?

  1. XmlDocTests were removed from all supported branches (XmlDocTests are redundant with SdkContentTests #3031)
  2. PoisonTests baseline is intended to be empty. Only reason for entries is temporary and we should not be releasing with leaks
  3. SdkContentTests now only run against stage 1 legs

#2 and #3 can affect servicing branches but IIRC they haven't been problematic. It was really #1 that prompted this issue.

@mthalman
Copy link
Member Author

The only known occurrence of this issue currently is SDK content diff in servicing. It's fine to ignore that for servicing.

@mthalman mthalman closed this as not planned Won't fix, can't repro, duplicate, stale Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-testing Improvements in CI and testing
Projects
Archived in project
Development

No branches or pull requests

2 participants