Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation for CoreCLR testing. #34674

Merged
merged 3 commits into from
Apr 8, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/workflow/testing/coreclr/test-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Test cases are categorized by priority level. The most important subset should b
1. Indicate the success of the test by returning `100`. Failure can be indicated by any non-`100` value.

Example:
```
```CSharp
static public int Main(string[] notUsed)
{
try
Expand Down
66 changes: 34 additions & 32 deletions docs/workflow/testing/coreclr/windows-test-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,90 +3,92 @@ Building and running tests on Windows

## Building Tests

To build the tests simply navigate to the tests directory above the repo and run,
```
C:\git\runtime>src\coreclr\build-test.cmd
```

## Cleaning Tests
Building coreclr tests must be done using a specific script as follows:

**Note:** Cleaning should be done before all tests to be sure that the test assets are initialized correctly. To do a clean build of the tests, in a clean command prompt, issue the following command:
```
C:\git\runtime>src\coreclr\build-test.cmd -rebuild
> src\coreclr\build-test.cmd
```

## Building Tests that will be precompiled
## Building Precompiled Tests

```
C:\git\runtime>src\coreclr\build-test.cmd crossgen
> src\coreclr\build-test.cmd crossgen
```

This will use `crossgen.exe` to precompile the test executables before they are executed.
This will use `crossgen.exe` to precompile test executables before they are executed.

## Building Other Priority Tests
## Building specific Priority Tests
AaronRobinsonMSFT marked this conversation as resolved.
Show resolved Hide resolved

```
C:\git\runtime>src\coreclr\build-test.cmd -priority=1
> src\coreclr\build-test.cmd -priority=1
```

The number '1' is just an example. The default value (if no priority is specified) is 0. To clarify, if '1' is specified, all tests with CLRTestPriorty 0 **and** 1 will be built and consequently run.
The above is an example of requesting priority '1' and below be built. The default priority value is '0'. If '1' is specified, all tests with `CLRTestPriorty` `0` **and** `1` will be built and run.
AaronRobinsonMSFT marked this conversation as resolved.
Show resolved Hide resolved

## Examples

To run a clean, priority 1, crossgen test pass:
To run a priority '0' and '1' and crossgen'd test pass:

```
C:\git\runtime>src\coreclr\build-test.cmd -rebuild crossgen -priority=1
> src\coreclr\build-test.cmd crossgen -priority=1
```

`src\coreclr\build-test.cmd -?` - will list additional supported parameters.
For additional supported parameters use the following:

```
> src\coreclr\build-test.cmd -?
```

### Building Individual Tests

Note: `build-test.cmd` or `build.cmd skipnative` needs to be run at least once
**Note:** `build-test.cmd skipmanaged [Any additional flags]` needs to be run at least once if the individual test has native assets.

* Native Test: Build the generated Visual Studio solution or makefile corresponding to test cmake file.

* Managed Test: You can invoke msbuild on the project directly from Visual Studio Command Prompt.
* Managed Test: Use `dotnet.cmd` from the root of the repo on the test project directly.

### Running Tests

`src\coreclr\tests\runtest.cmd /?` - will list supported parameters.
Will list supported parameters.

```
> src\coreclr\tests\runtest.cmd /?
```

For example to run all of the tests using your checked build:
In order to run all of the tests using your checked build:

```
<repo_root>\src\coreclr\tests\runtest.cmd checked
> src\coreclr\tests\runtest.cmd checked
```

This will generate a report named as `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT_x64_Checked.html`) in the subdirectory `<repo_root>\artifacts\log`. Any tests that failed will be listed in `TestRunResults_Windows_NT_x64_Checked.err`.
This will generate a report named `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT_x64_Checked.html`) in the subdirectory `<repo_root>\artifacts\log`. Any tests that failed will be listed in `TestRunResults_Windows_NT_x64_Checked.err`.

### Investigating Test Failures

Upon completing a test run, you may find one or more tests have failed.

The output of the Test will be available in Test reports directory, but by default the directory would be something like `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Reports\Exceptions\Finalization`.
The output of the test will be available in `Test` reports directory, but by default the directory will be something like `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Reports\Exceptions\Finalization`.

There are 2 files of interest:

- `Finalizer.output.txt` - Contains all the information logged by the test.
- `Finalizer.error.txt` - Contains the information reported by CoreRun.exe (which executed the test) when the test process crashes.
- `Finalizer.error.txt` - Contains the information reported by CoreRun.exe (which executed the test) when the test process crashes.

### Rerunning a failed test
### Re-run a failed test

If you wish to re-run a failed test, please follow the following steps:
If you wish to re-run a failed test, follow the following steps:

1. Set an environment variable, `CORE_ROOT`, pointing to the path to product binaries that was passed to runtest.cmd.
1) Set an environment variable, `CORE_ROOT`, pointing to the path to product binaries that was passed to runtest.cmd.
For example using a checked build the location would be: `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Tests\Core_Root`

2. Next, run the failed test, the command to which is also present in the test report for a failed test. It will be something like `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd`.
2) Run the failed test, the command to which is also present in the test report for a failed test. It will be something like `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd`.

If you wish to run the test under a debugger (e.g. [WinDbg](http://msdn.microsoft.com/en-us/library/windows/hardware/ff551063(v=vs.85).aspx)), append `-debug <debuggerFullPath>` to the test command. For example:
If you wish to run the test under a debugger (e.g. [WinDbg](http://msdn.microsoft.com/library/windows/hardware/ff551063(v=vs.85).aspx)), append `-debug <debuggerFullPath>` to the test command. For example:

```
<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd -debug <debuggerFullPath>
> artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd -debug <debuggerFullPath>
```

### Modifying a test

If test changes are needed, make the change and build the test project. This will binplace the binaries in test binaries folder (e.g. `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization`). At this point, follow the steps to re-run a failed test to re-run the modified test.
If test changes are needed, make the change, and build the test project. This will binplace the binaries in the test binaries folder (e.g. `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization`). Then re-run the test following the instructions above.