[tests] Add support for running tests on helix #1917
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # This is a modified copy of eng/common/templates/steps/send-to-helix.yml | ||
| # Please remember to update the documentation if you make changes to these parameters! | ||
|
There was a problem hiding this comment. Loved the docs :). Did we get these from some other repo? Mainly asking to see if we need to try to keep in sync. There was a problem hiding this comment. The template is from There was a problem hiding this comment. mmm I see, is there a way to instead use that template file and pass in some parameters to accommodate for our small modifications? If possible, I'd love to reduce the amount of yaml that we have to maintain. There was a problem hiding this comment. This There was a problem hiding this comment. Not sure what you mean when you say a "sample template". Tipically the templates under eng\common\templates are there so that repos can use them if they want/need as part of their build process. I don't want to block this PR on this, but I do want to make sure this is fixed. Can we at least log a tracking bug so that we go and add the necessary hooks in dotnet/arcade repo so that we can later ingest those changes here, and remove this file entirely so that we instead leverage the files from eng\common? There was a problem hiding this comment. Opened dotnet/arcade#14470 . |
||
| parameters: | ||
| HelixSource: 'pr/default' # required -- sources must start with pr/, official/, prodcon/, or agent/ | ||
| HelixType: 'tests/default/' # required -- Helix telemetry which identifies what type of data this is; should include "test" for clarity and must end in '/' | ||
| HelixBuild: $(Build.BuildNumber) # required -- the build number Helix will use to identify this -- automatically set to the AzDO build number | ||
| HelixTargetQueues: '' # required -- semicolon-delimited list of Helix queues to test on; see https://helix.dot.net/ for a list of queues | ||
| HelixAccessToken: '' # required -- access token to make Helix API requests; should be provided by the appropriate variable group | ||
| HelixProjectPath: '' # required -- path to the project file to build | ||
| HelixProjectArguments: '' # optional -- arguments passed to the build command | ||
| HelixConfiguration: '' # optional -- additional property attached to a job | ||
| HelixPreCommands: '' # optional -- commands to run before Helix work item execution | ||
| HelixPostCommands: '' # optional -- commands to run after Helix work item execution | ||
| WorkItemDirectory: '' # optional -- a payload directory to zip up and send to Helix; requires WorkItemCommand; incompatible with XUnitProjects | ||
| WorkItemCommand: '' # optional -- a command to execute on the payload; requires WorkItemDirectory; incompatible with XUnitProjects | ||
| WorkItemTimeout: '' # optional -- a timeout in TimeSpan.Parse-ready value (e.g. 00:02:00) for the work item command; requires WorkItemDirectory; incompatible with XUnitProjects | ||
| CorrelationPayloadDirectory: '' # optional -- a directory to zip up and send to Helix as a correlation payload | ||
| XUnitProjects: '' # optional -- semicolon-delimited list of XUnitProjects to parse and send to Helix; requires XUnitRuntimeTargetFramework, XUnitPublishTargetFramework, XUnitRunnerVersion, and IncludeDotNetCli=true | ||
| XUnitWorkItemTimeout: '' # optional -- the workitem timeout in seconds for all workitems created from the xUnit projects specified by XUnitProjects | ||
| XUnitPublishTargetFramework: '' # optional -- framework to use to publish your xUnit projects | ||
| XUnitRuntimeTargetFramework: '' # optional -- framework to use for the xUnit console runner | ||
| XUnitRunnerVersion: '' # optional -- version of the xUnit nuget package you wish to use on Helix; required for XUnitProjects | ||
| IncludeDotNetCli: false # optional -- true will download a version of the .NET CLI onto the Helix machine as a correlation payload; requires DotNetCliPackageType and DotNetCliVersion | ||
| DotNetCliPackageType: '' # optional -- either 'sdk', 'runtime' or 'aspnetcore-runtime'; determines whether the sdk or runtime will be sent to Helix; see https://raw.githubusercontent.com/dotnet/core/main/release-notes/releases-index.json | ||
| DotNetCliVersion: '' # optional -- version of the CLI to send to Helix; based on this: https://raw.githubusercontent.com/dotnet/core/main/release-notes/releases-index.json | ||
| WaitForWorkItemCompletion: true # optional -- true will make the task wait until work items have been completed and fail the build if work items fail. False is "fire and forget." | ||
| IsExternal: false # [DEPRECATED] -- doesn't do anything, jobs are external if HelixAccessToken is empty and Creator is set | ||
| HelixBaseUri: 'https://helix.dot.net/' # optional -- sets the Helix API base URI (allows targeting https://helix.int-dot.net ) | ||
| Creator: '' # optional -- if the build is external, use this to specify who is sending the job | ||
| DisplayNamePrefix: 'Run helix tests' # optional -- rename the beginning of the displayName of the steps in AzDO | ||
| IsWindows: false # optional | ||
| condition: succeeded() # optional -- condition for step to execute; defaults to succeeded() | ||
| continueOnError: false # optional -- determines whether to continue the build if the step errors; defaults to false | ||
|
|
||
| steps: | ||
| - ${{ if and(parameters.condition, eq(parameters.IsWindows, 'true')) }}: | ||
| - powershell: 'powershell "$env:BUILD_SOURCESDIRECTORY\eng\common\msbuild.ps1 ${{ parameters.HelixProjectPath }} ${{ parameters.HelixProjectArguments }} /restore /p:TreatWarningsAsErrors=false /t:Test /bl:$env:BUILD_SOURCESDIRECTORY\artifacts\log\$env:BuildConfig\SendToHelix.binlog"' | ||
| displayName: ${{ parameters.DisplayNamePrefix }} | ||
| env: | ||
| BuildConfig: $(_BuildConfig) | ||
| HelixSource: ${{ parameters.HelixSource }} | ||
| HelixType: ${{ parameters.HelixType }} | ||
| HelixBuild: ${{ parameters.HelixBuild }} | ||
| HelixConfiguration: ${{ parameters.HelixConfiguration }} | ||
| HelixTargetQueues: ${{ parameters.HelixTargetQueues }} | ||
| HelixAccessToken: ${{ parameters.HelixAccessToken }} | ||
| HelixPreCommands: ${{ parameters.HelixPreCommands }} | ||
| HelixPostCommands: ${{ parameters.HelixPostCommands }} | ||
| WorkItemDirectory: ${{ parameters.WorkItemDirectory }} | ||
| WorkItemCommand: ${{ parameters.WorkItemCommand }} | ||
| WorkItemTimeout: ${{ parameters.WorkItemTimeout }} | ||
| CorrelationPayloadDirectory: ${{ parameters.CorrelationPayloadDirectory }} | ||
| XUnitProjects: ${{ parameters.XUnitProjects }} | ||
| XUnitWorkItemTimeout: ${{ parameters.XUnitWorkItemTimeout }} | ||
| XUnitPublishTargetFramework: ${{ parameters.XUnitPublishTargetFramework }} | ||
| XUnitRuntimeTargetFramework: ${{ parameters.XUnitRuntimeTargetFramework }} | ||
| XUnitRunnerVersion: ${{ parameters.XUnitRunnerVersion }} | ||
| IncludeDotNetCli: ${{ parameters.IncludeDotNetCli }} | ||
| DotNetCliPackageType: ${{ parameters.DotNetCliPackageType }} | ||
| DotNetCliVersion: ${{ parameters.DotNetCliVersion }} | ||
| WaitForWorkItemCompletion: ${{ parameters.WaitForWorkItemCompletion }} | ||
| HelixBaseUri: ${{ parameters.HelixBaseUri }} | ||
| Creator: ${{ parameters.Creator }} | ||
| SYSTEM_ACCESSTOKEN: $(System.AccessToken) | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
| - ${{ if and(parameters.condition, ne(parameters.IsWindows, 'true')) }}: | ||
| - script: $BUILD_SOURCESDIRECTORY/eng/common/msbuild.sh ${{ parameters.HelixProjectPath }} ${{ parameters.HelixProjectArguments }} /restore /p:TreatWarningsAsErrors=false /t:Test /bl:$BUILD_SOURCESDIRECTORY/artifacts/log/$BuildConfig/SendToHelix.binlog | ||
| displayName: ${{ parameters.DisplayNamePrefix }} | ||
| env: | ||
| BuildConfig: $(_BuildConfig) | ||
| HelixSource: ${{ parameters.HelixSource }} | ||
| HelixType: ${{ parameters.HelixType }} | ||
| HelixBuild: ${{ parameters.HelixBuild }} | ||
| HelixConfiguration: ${{ parameters.HelixConfiguration }} | ||
| HelixTargetQueues: ${{ parameters.HelixTargetQueues }} | ||
| HelixAccessToken: ${{ parameters.HelixAccessToken }} | ||
| HelixPreCommands: ${{ parameters.HelixPreCommands }} | ||
| HelixPostCommands: ${{ parameters.HelixPostCommands }} | ||
| WorkItemDirectory: ${{ parameters.WorkItemDirectory }} | ||
| WorkItemCommand: ${{ parameters.WorkItemCommand }} | ||
| WorkItemTimeout: ${{ parameters.WorkItemTimeout }} | ||
| CorrelationPayloadDirectory: ${{ parameters.CorrelationPayloadDirectory }} | ||
| XUnitProjects: ${{ parameters.XUnitProjects }} | ||
| XUnitWorkItemTimeout: ${{ parameters.XUnitWorkItemTimeout }} | ||
| XUnitPublishTargetFramework: ${{ parameters.XUnitPublishTargetFramework }} | ||
| XUnitRuntimeTargetFramework: ${{ parameters.XUnitRuntimeTargetFramework }} | ||
| XUnitRunnerVersion: ${{ parameters.XUnitRunnerVersion }} | ||
| IncludeDotNetCli: ${{ parameters.IncludeDotNetCli }} | ||
| DotNetCliPackageType: ${{ parameters.DotNetCliPackageType }} | ||
| DotNetCliVersion: ${{ parameters.DotNetCliVersion }} | ||
| WaitForWorkItemCompletion: ${{ parameters.WaitForWorkItemCompletion }} | ||
| HelixBaseUri: ${{ parameters.HelixBaseUri }} | ||
| Creator: ${{ parameters.Creator }} | ||
| SYSTEM_ACCESSTOKEN: $(System.AccessToken) | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| <?xml version="1.0" encoding="utf-8"?> | ||
| <RunSettings> | ||
| <RunConfiguration> | ||
| <!-- Timeout in ms, 5 minutes --> | ||
| <TestSessionTimeout>300000</TestSessionTimeout> | ||
| <!-- Filter out failing (wrong framework, platform, runtime or activeissue) tests --> | ||
| <TestCaseFilter>category!=failing</TestCaseFilter> | ||
| </RunConfiguration> | ||
| <LoggerRunSettings> | ||
| <Loggers> | ||
| <Logger friendlyName="trx"> | ||
| <Configuration> | ||
| <LogFileName>TestResults.trx</LogFileName> | ||
| </Configuration> | ||
| </Logger> | ||
| <Logger friendlyName="console"> | ||
| <Configuration> | ||
| <Verbosity>normal</Verbosity> | ||
| </Configuration> | ||
| </Logger> | ||
| </Loggers> | ||
| </LoggerRunSettings> | ||
| </RunSettings> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| <Project> | ||
| <Import Project="..\Directory.Build.targets" /> | ||
| <PropertyGroup> | ||
| <DeployRunSettingsFile Condition="'$(DeployRunSettingsFile)' == ''">true</DeployRunSettingsFile> | ||
|
|
||
| <RunTestsOnHelix Condition="'$(RunTestsOnHelix)' == ''">false</RunTestsOnHelix> | ||
| <SkipTests Condition="'$(SkipTests)' == '' and ('$(IsTestSupportProject)' == 'true' or '$(RunTestsOnHelix)' == 'true')">true</SkipTests> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <None Include="$(RepositoryEngineeringDir)testing\.runsettings" CopyToOutputDirectory="PreserveNewest" Condition="'$(DeployRunSettingsFile)' == 'true'" /> | ||
| </ItemGroup> | ||
|
|
||
| <Target Name="ZipTestArchive" AfterTargets="Build" Condition="'$(ArchiveTests)' == 'true' and '$(RunTestsOnHelix)' == 'true'"> | ||
| <Error Condition="'$(TestArchiveTestsDir)' == ''" Text="TestArchiveTestsDir property to archive the test folder must be set." /> | ||
|
|
||
| <MakeDir Directories="$(TestArchiveTestsDir)" /> | ||
| <ZipDirectory SourceDirectory="$(OutDir)" | ||
joperezr marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| DestinationFile="$([MSBuild]::NormalizePath('$(TestArchiveTestsDir)', '$(MSBuildProjectName).zip'))" | ||
| Overwrite="true" /> | ||
| </Target> | ||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😞
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, 30 mins was already pretty low and super close to the time it actually takes to build the repo in AzDO, so we were going to have to do this sooner rather than later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's from a recent CI run.