Skip to content

Commit

Permalink
Merge branch 'upstream-main' into ritchxu/support-actions-summary
Browse files Browse the repository at this point in the history
  • Loading branch information
ritchxu committed Jun 25, 2024
2 parents 1db4305 + e052c7d commit 84e60ba
Show file tree
Hide file tree
Showing 37 changed files with 2,135 additions and 1,342 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/check-dist.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
id: diff

# If index.js was different than expected, upload the expected version as an artifact
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
if: ${{ failure() && steps.diff.conclusion == 'failure' }}
with:
name: dist
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:

- name: Upload test results
if: success() || failure()
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-results
path: __tests__/__results__/*.xml
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
v18.19.0
v20
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This [Github Action](https://github.com/features/actions) displays test results
|:--:|:--:|:--:|:--:|

**Supported languages / frameworks:**
- .NET / [xUnit](https://xunit.net/) / [NUnit](https://nunit.org/) / [MSTest](https://github.com/Microsoft/testfx-docs)
- .NET / [dotnet test](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test#examples) ( [xUnit](https://xunit.net/) / [NUnit](https://nunit.org/) / [MSTest](https://github.com/Microsoft/testfx-docs) )
- Dart / [test](https://pub.dev/packages/test)
- Flutter / [test](https://pub.dev/packages/test)
- Java / [JUnit](https://junit.org/)
Expand Down Expand Up @@ -43,7 +43,7 @@ jobs:
name: Build & Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3 # checkout the repo
- uses: actions/checkout@v4 # checkout the repo
- run: npm ci # install packages
- run: npm test # run tests (configured to use jest-junit reporter)

Expand Down Expand Up @@ -74,10 +74,10 @@ jobs:
build-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3 # checkout the repo
- uses: actions/checkout@v4 # checkout the repo
- run: npm ci # install packages
- run: npm test # run tests (configured to use jest-junit reporter)
- uses: actions/upload-artifact@v3 # upload test results
- uses: actions/upload-artifact@v4 # upload test results
if: success() || failure() # run this step even if previous step failed
with:
name: test-results
Expand Down Expand Up @@ -137,11 +137,13 @@ jobs:
# Format of test results. Supported options:
# dart-json
# dotnet-nunit
# dotnet-trx
# flutter-json
# java-junit
# jest-junit
# mocha-json
# rspec-json
reporter: ''
# Allows you to generate only the summary.
Expand Down Expand Up @@ -315,16 +317,22 @@ Configuration of `uniqueOutputName`, `suiteNameTemplate`, `classNameTemplate`, `
- Mocha version [v7.2.0](https://github.com/mochajs/mocha/releases/tag/v7.2.0) or higher
- Usage of [json](https://mochajs.org/#json) reporter.

You can use the following example configuration in `package.json`:
For Mocha >= [v9.1.0](https://github.com/mochajs/mocha/releases/tag/v9.1.0), you can use the following example configuration in `package.json`:
```json
"scripts": {
"test": "mocha --reporter json > test-results.json"
"test": "mocha --reporter json --reporter-option output=test-results.json"
}
```

Test processing might fail if any of your tests write anything on standard output.
Mocha, unfortunately, doesn't have the option to store `json` output directly to the file, and we have to rely on redirecting its standard output.
There is a work in progress to fix it: [mocha#4607](https://github.com/mochajs/mocha/pull/4607)
For Mocha < v9.1, the command should look like this:
```json
"scripts": {
"test": "mocha --reporter json > test-results.json"
}
```
Additionally, test processing might fail if any of your tests write anything on standard output.
Before version [v9.1.0](https://github.com/mochajs/mocha/releases/tag/v9.1.0), Mocha doesn't have the option to store `json` output directly to the file, and we have to rely on redirecting its standard output ([mocha#4607](https://github.com/mochajs/mocha/pull/4607)).
Please update Mocha to version [v9.1.0](https://github.com/mochajs/mocha/releases/tag/v9.1.0) or above if you encounter this issue.
</details>

<details>
Expand All @@ -339,7 +347,7 @@ Unfortunately, there are some known issues and limitations caused by GitHub API:

- Test report (i.e. Check Run summary) is markdown text. No custom styling or HTML is possible.
- Maximum report size is 65535 bytes. Input parameters `list-suites` and `list-tests` will be automatically adjusted if max size is exceeded.
- Test report can't reference any additional files (e.g. screenshots). You can use `actions/upload-artifact@v3` to upload them and inspect them manually.
- Test report can't reference any additional files (e.g. screenshots). You can use `actions/upload-artifact@v4` to upload them and inspect them manually.
- Check Runs are created for specific commit SHA. It's not possible to specify under which workflow test report should belong if more
workflows are running for the same SHA. Thanks to this GitHub "feature" it's possible your test report will appear in an unexpected place in GitHub UI.
For more information, see [#67](https://github.com/dorny/test-reporter/issues/67).
Expand Down
31 changes: 31 additions & 0 deletions __tests__/__outputs__/dotnet-nunit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
![Tests failed](https://img.shields.io/badge/tests-3%20passed%2C%205%20failed%2C%201%20skipped-critical)
|Report|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|fixtures/dotnet-nunit.xml|3 ✅|5 ❌|1 ⚪|230ms|
## ❌ <a id="user-content-r0" href="#r0">fixtures/dotnet-nunit.xml</a>
**9** tests were completed in **230ms** with **3** passed, **5** failed and **1** skipped.
|Test suite|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|[DotnetTests.NUnitV3Tests.dll.DotnetTests.XUnitTests](#r0s0)|3 ✅|5 ❌|1 ⚪|69ms|
### ❌ <a id="user-content-r0s0" href="#r0s0">DotnetTests.NUnitV3Tests.dll.DotnetTests.XUnitTests</a>
```
CalculatorTests
✅ Is_Even_Number(2)
❌ Is_Even_Number(3)
Expected: True
But was: False
❌ Exception_In_TargetTest
System.DivideByZeroException : Attempted to divide by zero.
❌ Exception_In_Test
System.Exception : Test
❌ Failing_Test
Expected: 3
But was: 2
✅ Passing_Test
✅ Passing_Test_With_Description
⚪ Skipped_Test
❌ Timeout_Test
```
19 changes: 19 additions & 0 deletions __tests__/__outputs__/rspec-json.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
![Tests failed](https://img.shields.io/badge/tests-1%20passed%2C%201%20failed%2C%201%20skipped-critical)
|Report|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|fixtures/rspec-json.json|1 ✅|1 ❌|1 ⚪|0ms|
## ❌ <a id="user-content-r0" href="#r0">fixtures/rspec-json.json</a>
**3** tests were completed in **0ms** with **1** passed, **1** failed and **1** skipped.
|Test suite|Passed|Failed|Skipped|Time|
|:---|---:|---:|---:|---:|
|[./spec/config/check_env_vars_spec.rb](#r0s0)|1 ✅|1 ❌|1 ⚪|0ms|
### ❌ <a id="user-content-r0s0" href="#r0s0">./spec/config/check_env_vars_spec.rb</a>
```
CheckEnvVars#call when all env vars are defined behaves like success load
❌ CheckEnvVars#call when all env vars are defined behaves like success load fails in assertion
(#ActiveSupport::BroadcastLogger:0x00007f1007fedf58).debug("All config env vars exist")
expected: 0 times with arguments: ("All config env vars exist")
received: 1 time with arguments: ("All config env vars exist")
✅ CheckEnvVars#call when all env vars are defined behaves like success load logs success message
⚪ CheckEnvVars#call when all env vars are defined behaves like success load skips the test
```
1 change: 1 addition & 0 deletions __tests__/__outputs__/swift-xunit.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ AcmeLibTests.AcmeLibTests
✅ test_always_pass
✅ test_always_skip
❌ test_always_fail
failed
```
107 changes: 107 additions & 0 deletions __tests__/__snapshots__/dotnet-nunit.test.ts.snap
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`dotnet-nunit tests report from ./reports/dotnet test results matches snapshot 1`] = `
TestRunResult {
"path": "fixtures/dotnet-nunit.xml",
"suites": [
TestSuiteResult {
"groups": [
TestGroupResult {
"name": "CalculatorTests",
"tests": [
TestCaseResult {
"error": undefined,
"name": "Is_Even_Number(2)",
"result": "success",
"time": 0.622,
},
TestCaseResult {
"error": {
"details": " at DotnetTests.XUnitTests.CalculatorTests.Is_Even_Number(Int32 i) in C:\\Users\\Michal\\Workspace\\dorny\\test-reporter\\reports\\dotnet\\DotnetTests.NUnitV3Tests\\CalculatorTests.cs:line 61
",
"line": undefined,
"message": " Expected: True
But was: False
",
"path": undefined,
},
"name": "Is_Even_Number(3)",
"result": "failed",
"time": 1.098,
},
TestCaseResult {
"error": {
"details": " at DotnetTests.Unit.Calculator.Div(Int32 a, Int32 b) in C:\\Users\\Michal\\Workspace\\dorny\\test-reporter\\reports\\dotnet\\DotnetTests.Unit\\Calculator.cs:line 9
at DotnetTests.XUnitTests.CalculatorTests.Exception_In_TargetTest() in C:\\Users\\Michal\\Workspace\\dorny\\test-reporter\\reports\\dotnet\\DotnetTests.NUnitV3Tests\\CalculatorTests.cs:line 33",
"line": undefined,
"message": "System.DivideByZeroException : Attempted to divide by zero.",
"path": undefined,
},
"name": "Exception_In_TargetTest",
"result": "failed",
"time": 22.805,
},
TestCaseResult {
"error": {
"details": " at DotnetTests.XUnitTests.CalculatorTests.Exception_In_Test() in C:\\Users\\Michal\\Workspace\\dorny\\test-reporter\\reports\\dotnet\\DotnetTests.NUnitV3Tests\\CalculatorTests.cs:line 39",
"line": undefined,
"message": "System.Exception : Test",
"path": undefined,
},
"name": "Exception_In_Test",
"result": "failed",
"time": 0.528,
},
TestCaseResult {
"error": {
"details": " at DotnetTests.XUnitTests.CalculatorTests.Failing_Test() in C:\\Users\\Michal\\Workspace\\dorny\\test-reporter\\reports\\dotnet\\DotnetTests.NUnitV3Tests\\CalculatorTests.cs:line 27
",
"line": undefined,
"message": " Expected: 3
But was: 2
",
"path": undefined,
},
"name": "Failing_Test",
"result": "failed",
"time": 28.162,
},
TestCaseResult {
"error": undefined,
"name": "Passing_Test",
"result": "success",
"time": 0.23800000000000002,
},
TestCaseResult {
"error": undefined,
"name": "Passing_Test_With_Description",
"result": "success",
"time": 0.135,
},
TestCaseResult {
"error": undefined,
"name": "Skipped_Test",
"result": "skipped",
"time": 0.398,
},
TestCaseResult {
"error": {
"details": "",
"line": undefined,
"message": "",
"path": undefined,
},
"name": "Timeout_Test",
"result": "failed",
"time": 14.949,
},
],
},
],
"name": "DotnetTests.NUnitV3Tests.dll.DotnetTests.XUnitTests",
"totalTime": undefined,
},
],
"totalTime": 230.30800000000002,
}
`;
4 changes: 2 additions & 2 deletions __tests__/__snapshots__/java-junit.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ at java.lang.Thread.run(Thread.java:748)
",
"line": 29,
"message": undefined,
"message": "java.lang.AssertionError: expected [1.2.1] but found [1.2.0]",
"path": "pulsar-common/src/test/java/org/apache/pulsar/AddMissingPatchVersionTest.java",
},
"name": "testVersionStrings",
Expand Down Expand Up @@ -100,7 +100,7 @@ at java.lang.Thread.run(Thread.java:748)
",
"line": 29,
"message": undefined,
"message": "java.lang.AssertionError: expected [1.2.1] but found [1.2.0]",
"path": "pulsar-common/src/test/java/org/apache/pulsar/AddMissingPatchVersionTest.java",
},
"name": "testVersionStrings",
Expand Down
49 changes: 49 additions & 0 deletions __tests__/__snapshots__/rspec-json.test.ts.snap
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`rspec-json tests report from ./reports/rspec-json test results matches snapshot 1`] = `
TestRunResult {
"path": "fixtures/rspec-json.json",
"suites": [
TestSuiteResult {
"groups": [
TestGroupResult {
"name": "CheckEnvVars#call when all env vars are defined behaves like success load",
"tests": [
TestCaseResult {
"error": {
"details": "/usr/local/bundle/ruby/3.3.0/gems/net-http-0.4.1/lib/net/http.rb:1603:in \`initialize'
./config/check_env_vars.rb:11:in \`call'
./spec/config/check_env_vars_spec.rb:7:in \`block (3 levels) in <top (required)>'
./spec/config/check_env_vars_spec.rb:19:in \`block (4 levels) in <top (required)>'",
"line": 11,
"message": "(#ActiveSupport::BroadcastLogger:0x00007f1007fedf58).debug("All config env vars exist")
expected: 0 times with arguments: ("All config env vars exist")
received: 1 time with arguments: ("All config env vars exist")",
"path": "./config/check_env_vars.rb",
},
"name": "CheckEnvVars#call when all env vars are defined behaves like success load fails in assertion",
"result": "failed",
"time": 0.004411051,
},
TestCaseResult {
"error": undefined,
"name": "CheckEnvVars#call when all env vars are defined behaves like success load logs success message",
"result": "success",
"time": 0.079159625,
},
TestCaseResult {
"error": undefined,
"name": "CheckEnvVars#call when all env vars are defined behaves like success load skips the test",
"result": "skipped",
"time": 0.000023007,
},
],
},
],
"name": "./spec/config/check_env_vars_spec.rb",
"totalTime": undefined,
},
],
"totalTime": 0.19118387,
}
`;
2 changes: 1 addition & 1 deletion __tests__/__snapshots__/swift-xunit.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ TestRunResult {
"error": {
"details": undefined,
"line": undefined,
"message": undefined,
"message": "failed",
"path": undefined,
},
"name": "test_always_fail",
Expand Down
29 changes: 29 additions & 0 deletions __tests__/dotnet-nunit.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import * as fs from 'fs'
import * as path from 'path'

import {DotnetNunitParser} from '../src/parsers/dotnet-nunit/dotnet-nunit-parser'
import {ParseOptions} from '../src/test-parser'
import {getReport} from '../src/report/get-report'
import {normalizeFilePath} from '../src/utils/path-utils'

describe('dotnet-nunit tests', () => {
it('report from ./reports/dotnet test results matches snapshot', async () => {
const fixturePath = path.join(__dirname, 'fixtures', 'dotnet-nunit.xml')
const outputPath = path.join(__dirname, '__outputs__', 'dotnet-nunit.md')
const filePath = normalizeFilePath(path.relative(__dirname, fixturePath))
const fileContent = fs.readFileSync(fixturePath, {encoding: 'utf8'})

const opts: ParseOptions = {
parseErrors: true,
trackedFiles: ['DotnetTests.Unit/Calculator.cs', 'DotnetTests.NUnitV3Tests/CalculatorTests.cs']
}

const parser = new DotnetNunitParser(opts)
const result = await parser.parse(filePath, fileContent)
expect(result).toMatchSnapshot()

const report = getReport([result])
fs.mkdirSync(path.dirname(outputPath), {recursive: true})
fs.writeFileSync(outputPath, report)
})
})
Loading

0 comments on commit 84e60ba

Please sign in to comment.