Skip to content

Commit

Permalink
Move sections into details (#297)
Browse files Browse the repository at this point in the history
  • Loading branch information
@EnricoMi
EnricoMi authored Jun 2, 2022
1 parent dda9390 commit edf3cdc
Show file tree
Hide file tree
Showing 3 changed files with 93 additions and 50 deletions.
105 changes: 74 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ and ![Windows](https://badgen.net/badge/icon/Windows?icon=windows&label) (e.g. `

See the [notes on running this action as a composite action](#running-as-a-composite-action) if you run it on Windows or macOS.

Also see the [notes on supporting pull requests from fork repositories and branches](#support-fork-repositories-and-dependabot-branches)
created by [Dependabot](https://docs.github.com/en/github/administering-a-repository/keeping-your-dependencies-updated-automatically).
Also see the [notes on supporting pull requests from fork repositories and branches created by Dependabot](#support-fork-repositories-and-dependabot-branches).

The `if: always()` clause guarantees that this action always runs, even if earlier steps (e.g., the test step) in your workflow fail.

Expand Down Expand Up @@ -169,46 +168,64 @@ with:
!config.xml
```

See the complete list of options below.
The list of most notable options:

|Option|Default Value|Description|
|:-----|:-----:|:----------|
|`files`|`*.xml`|File patterns to select the test result XML files, e.g. `"test-results/**/*.xml"`. Use multiline string for multiple patterns. Supports `*`, `**`, `?`, `[]`. Excludes files when starting with `!`. |
|`time_unit`|`seconds`|Time values in the XML files have this unit. Supports `seconds` and `milliseconds`.|
|`check_name`|`"Unit Test Results"`|An alternative name for the check result.|
|`comment_title`|same as `check_name`|An alternative name for the pull request comment.|
|`comment_mode`|`update last`|The action posts comments to a pull request that is associated with the commit. Set to `create new` to create a new comment on each commit, `update last` to create only one comment and update later on, `off` to not create pull request comments.|
|`job_summary`|`true`| Set to `true`, the results are published as part of the [job summary page](https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/) of the workflow run.|
|`hide_comments`|`"all but latest"`|Configures which earlier comments in a pull request are hidden by the action:<br/>`"orphaned commits"` - comments for removed commits<br/>`"all but latest"` - all comments but the latest<br/>`"off"` - no hiding|

<details>
<summary>Options related to Git and GitHub</summary>

|Option|Default Value|Description|
|:-----|:-----:|:----------|
|`commit`|`${{env.GITHUB_SHA}}`|An alternative commit SHA to which test results are published. The `push` and `pull_request`events are handled, but for other [workflow events](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#push) `GITHUB_SHA` may refer to different kinds of commits. See [GitHub Workflow documentation](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows) for details.|
|`github_token`|`${{github.token}}`|An alternative GitHub token, other than the default provided by GitHub Actions runner.|
|`github_retries`|`10`|Requests to the GitHub API are retried this number of times. The value must be a positive integer or zero.|
|`seconds_between_github_reads`|`0.25`|Sets the number of seconds the action waits between concurrent read requests to the GitHub API.|
|`seconds_between_github_writes`|`2.0`|Sets the number of seconds the action waits between concurrent write requests to the GitHub API.|
|`commit`|`${{env.GITHUB_SHA}}`|An alternative commit SHA to which test results are published. The `push` and `pull_request`events are handled, but for other [workflow events](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#push) `GITHUB_SHA` may refer to different kinds of commits. See [GitHub Workflow documentation](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows) for details.|
|`json_file`|no file|Results are written to this JSON file.|
|`fail_on`|`"test failures"`|Configures the state of the created test result check run. With `"test failures"` it fails if any test fails or test errors occur. It never fails when set to `"nothing"`, and fails only on errors when set to `"errors"`.|
|`pull_request_build`|`"merge"`|GitHub builds a merge commit, which combines the commit and the target branch. If tests ran on the actual pushed commit, then set this to `"commit"`.|
|`pull_request_build`|`"merge"`|As part of pull requests, GitHub builds a merge commit, which combines the commit and the target branch. If tests ran on the actual pushed commit, then set this to `"commit"`.|
|`event_file`|`${{env.GITHUB_EVENT_PATH}}`|An alternative event file to use. Useful to replace a `workflow_run` event file with the actual source event file.|
|`event_name`|`${{env.GITHUB_EVENT_NAME}}`|An alternative event name to use. Useful to replace a `workflow_run` event name with the actual source event name: `${{ github.event.workflow_run.event }}`.|
|`test_changes_limit`|`10`|Limits the number of removed or skipped tests listed on pull request comments. This can be disabled with a value of `0`.|
</details>

<details>
<summary>Options related to reporting test results</summary>

|Option|Default Value|Description|
|:-----|:-----:|:----------|
|`time_unit`|`seconds`|Time values in the XML files have this unit. Supports `seconds` and `milliseconds`.|
|`job_summary`|`true`| Set to `true`, the results are published as part of the [job summary page](https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/) of the workflow run.|
|`comment_mode`|`update last`|The action posts comments to a pull request that is associated with the commit. Set to `create new` to create a new comment on each commit, `update last` to create only one comment and update later on, `off` to not create pull request comments.|
|`hide_comments`|`"all but latest"`|Configures which earlier comments in a pull request are hidden by the action:<br/>`"orphaned commits"` - comments for removed commits<br/>`"all but latest"` - all comments but the latest<br/>`"off"` - no hiding|
|`compare_to_earlier_commit`|`true`|Test results are compared to results of earlier commits to show changes:<br/>`false` - disable comparison, `true` - compare across commits.'|
|`test_changes_limit`|`10`|Limits the number of removed or skipped tests reported on PR comments. This report can be disabled with a value of `0`.|
|`report_individual_runs`|`false`|Individual runs of the same test may see different failures. Reports all individual failures when set `true`, and the first failure only otherwise.|
|`deduplicate_classes_by_file_name`|`false`|De-duplicates classes with same name by their file name when set `true`, combines test results for those classes otherwise.|
|`check_run_annotations`|`all tests, skipped tests`|Adds additional information to the check run. This is a comma-separated list of any of the following values:<br>`all tests` - list all found tests,<br>`skipped tests` - list all skipped tests<br> Set to `none` to add no extra annotations at all.|
|`check_run_annotations_branch`|`event.repository.default_branch` or `"main, master"`|Adds check run annotations only on given branches. If not given, this defaults to the default branch of your repository, e.g. `main` or `master`. Comma separated list of branch names allowed, asterisk `"*"` matches all branches. Example: `main, master, branch_one`|
|`ignore_runs`|`false`|Does not process test run information by ignoring `<testcase>` elements in the XML files, which is useful for very large XML files. This disables any check run annotations.|
|`compare_to_earlier_commit`|`true`|Test results are compared to results of earlier commits to show changes:<br/>`false` - disable comparison, `true` - compare across commits.'|
|`check_run_annotations`|`all tests, skipped tests`|Adds additional information to the check run (comma-separated list):<br>`all tests` - list all found tests,<br>`skipped tests` - list all skipped tests,<br>`none` - no extra annotations at all|
|`check_run_annotations_branch`|default branch|Adds check run annotations only on given branches. If not given, this defaults to the default branch of your repository, e.g. `main` or `master`. Comma separated list of branch names allowed, asterisk `"*"` matches all branches. Example: `main, master, branch_one`|
|`json_file`|no file|Results are written to this JSON file.|
|`fail_on`|`"test failures"`|Configures the state of the created test result check run. With `"test failures"` it fails if any test fails or test errors occur. It never fails when set to `"nothing"`, and fails only on errors when set to `"errors"`.|

Pull request comments highlight removal of tests or tests that the pull request moves into skip state.
Those removed or skipped tests are added as a list, which is limited in length by `test_changes_limit`,
which defaults to `10`. Listing these tests can be disabled entirely by setting this limit to `0`.
which defaults to `10`. Reporting these tests can be disabled entirely by setting this limit to `0`.
This feature requires `check_run_annotations` to contain `all tests` in order to detect test addition
and removal, and `skipped tests` to detect new skipped and un-skipped tests, as well as
`check_run_annotations_branch` to contain your default branch.
</details>

## JSON result

The gathered test information are accessible as JSON. The `json` output of the action can be accessed
through the expression `steps.<id>.outputs.json`.
The gathered test information are accessible as JSON via [GitHub Actions steps outputs](https://docs.github.com/en/actions/learn-github-actions/contexts#steps-context) string or JSON file.

<details>
<summary>Access JSON via step outputs</summary>

The `json` output of the action can be accessed through the expression `steps.<id>.outputs.json`.

```yaml
- name: Publish Test Results
Expand Down Expand Up @@ -258,9 +275,13 @@ Here is an example JSON:
"annotations": 31
}
```
</details>

<details>
<summary>Access JSON via file</summary>

The optional `json_file` allows to configure a file where extended JSON information are to be written.
Compared to above, `errors` and `annotations` contain more information than just the number of errors and annotations, respectively:
Compared to `"Access JSON via step outputs"` above, `errors` and `annotations` contain more information than just the number of errors and annotations, respectively:

```json
{
Expand Down Expand Up @@ -291,6 +312,7 @@ Compared to above, `errors` and `annotations` contain more information than just
]
}
```
</details>

See [Create a badge from test results](#create-a-badge-from-test-results) for an example on how to create a badge from this JSON.

Expand All @@ -301,6 +323,9 @@ the action should run only once over all test results. For this, put the action
that depends on all your test environments. Those need to upload the test results as artifacts, which
are then all downloaded by your publish job.

<details>
<summary>Example workflow YAML</summary>

```yaml
name: CI
Expand Down Expand Up @@ -364,12 +389,17 @@ jobs:
with:
files: "artifacts/**/*.xml"
```
</details>

Please consider to [support fork repositories and dependabot branches](#support-fork-repositories-and-dependabot-branches)
together with your matrix strategy.

## Support fork repositories and dependabot branches
[comment]: <> (This heading is linked to from main method in publish_unit_test_results.py)

Getting test results of pull requests created by [Dependabot](https://docs.github.com/en/github/administering-a-repository/keeping-your-dependencies-updated-automatically)
or by contributors from fork repositories requires some additional setup. Without this, the action will fail with the
Getting test results of pull requests created by contributors from fork repositories or by
[Dependabot](https://docs.github.com/en/github/administering-a-repository/keeping-your-dependencies-updated-automatically)
requires some additional setup. Without this, the action will fail with the
`"Resource not accessible by integration"` error for those situations.

In this setup, your CI workflow does not need to publish test results anymore as they are **always** published from a separate workflow.
Expand All @@ -380,7 +410,10 @@ In this setup, your CI workflow does not need to publish test results anymore as
This workflow publishes the test results for pull requests from fork repositories and dependabot,
as well as all "ordinary" runs of your CI workflow.

Add the following job to your CI workflow to upload the event file as an artifact:
<details>
<summary>Step-by-step instructions</summary>

1. Add the following job to your CI workflow to upload the event file as an artifact:

```yaml
event_file:
Expand All @@ -394,7 +427,7 @@ event_file:
path: ${{ github.event_path }}
```

Add the following action step to your CI workflow to upload test results as artifacts.
2. Add the following action step to your CI workflow to upload test results as artifacts.
Adjust the value of `path` to fit your setup:

```yaml
Expand All @@ -407,23 +440,23 @@ Adjust the value of `path` to fit your setup:
test-results/*.xml
```

If you run tests in a [strategy matrix](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix),
3. If you run tests in a [strategy matrix](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix),
make the artifact name unique for each job, e.g.:
```yaml
with:
name: Test Results (${{ matrix.python-version }})
path: …
```

Add the following workflow that publishes test results. It downloads and extracts
4. Add the following workflow that publishes test results. It downloads and extracts
all artifacts into `artifacts/ARTIFACT_NAME/`, where `ARTIFACT_NAME` will be `Upload Test Results`
when setup as above, or `Upload Test Results (…)` when run in a strategy matrix.
It then runs the action on files matching `artifacts/**/*.xml`.

It then runs the action on files matching `artifacts/**/*.xml`.
Change the `files` pattern with the path to your test artifacts if it does not work for you.
The publish action uses the event file of the CI workflow.

Also adjust the value of `workflows` (here `"CI"`) to fit your setup:

Also adjust the value of `workflows` (here `"CI"`) to fit your setup:

```yaml
name: Test Results
Expand Down Expand Up @@ -482,12 +515,17 @@ jobs:
```

Note: Running this action on `pull_request_target` events is [dangerous if combined with code checkout and code execution](https://securitylab.github.com/research/github-actions-preventing-pwn-requests).
This event is therefore not use here intentionally!
</details>

## Create a badge from test results

This is an example how to use the [JSON](#json-result) output of this action to create a badge like this:
Here is an example how to use the [JSON](#json-result) output of this action to create a badge like this:
[![Test Results](https://gist.githubusercontent.com/EnricoMi/612cb538c14731f1a8fefe504f519395/raw/badge.svg)](https://gist.githubusercontent.com/EnricoMi/612cb538c14731f1a8fefe504f519395/raw/badge.svg)

<details>
<summary>Example worklow YAML</summary>

```yaml
steps:
- …
Expand Down Expand Up @@ -538,6 +576,7 @@ You have to create a personal access toke (PAT) with `gist` permission only. Add
Set the `gistURL` to the Gist that you want to write the badge file to, in the form of `https://gist.githubusercontent.com/{user}/{id}`.

You can then use the badge via this URL: https://gist.githubusercontent.com/{user}/{id}/raw/badge.svg
</details>

## Running as a composite action

Expand All @@ -559,7 +598,8 @@ Self-hosted runners for Windows require Bash shell to be installed. Easiest way
Git for Windows, which comes with Git BASH. Make sure that the location of `bash.exe` is part of the `PATH`
environment variable seen by the self-hosted runner.

### Isolating composite action from your workflow
<details>
<summary>Isolating composite action from your workflow</summary>

Note that the composite action modifies this Python environment by installing dependency packages.
If this conflicts with actions that later run Python in the same workflow (which is a rare case),
Expand Down Expand Up @@ -604,8 +644,10 @@ publish-test-results:
with:
files: "artifacts/**/*.xml"
```
</details>

### Slow startup of composite action
<details>
<summary>Slow startup of composite action</summary>

In some environments, the composite action startup can be slow due to the installation of Python dependencies.
This is usually the case for **Windows** runners (in this example 35 seconds startup time):
Expand Down Expand Up @@ -662,3 +704,4 @@ Mon, 03 May 2021 16:00:00 GMT ⏵ Check for Python3
Mon, 03 May 2021 16:00:00 GMT ⏵ Install Python dependencies
Mon, 03 May 2021 16:00:11 GMT ⏵ Publish Test Results
```
</details>
10 changes: 5 additions & 5 deletions action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,14 @@ inputs:
required: false
job_summary:
description: 'Set to `true`, the results are published as part of the job summary page of the workflow run'
required: false
default: 'true'
required: false
compare_to_earlier_commit:
description: 'Test results are compared to results of earlier commits to highlight changes: false - disable comparison, true - compare across commits'
default: 'true'
required: false
pull_request_build:
description: 'Pull request events build a merge of the branch and the target branch, not just the commit that has been pushed. When your workflow deviates from this behaviour and builds the actual commit, then set this option to "commit". Defaults to "merge".'
description: 'As part of pull requests, GitHub builds a merge commit, which combines the commit and the target branch. If tests ran on the actual pushed commit, then set this to "commit". Defaults to "merge".'
default: 'merge'
required: false
event_file:
Expand All @@ -72,14 +72,14 @@ inputs:
description: 'An alternative event name to use. Useful to replace a "workflow_run" event name with the actual source event name: github.event.workflow_run.event.'
required: false
test_changes_limit:
description: 'Limits the number of removed or skipped tests listed on PR comments. This can be disabled with a value of 0. The default is 10.'
description: 'Limits the number of removed or skipped tests reported on PR comments. This report can be disabled with a value of 0. The default is 10.'
required: false
check_run_annotations:
description: 'Adds additional information to the check run (comma-separated list): all tests - list all found tests, skipped tests - list all skipped tests, none - no extra annotations at all'
description: 'Adds additional information to the check run. This is a comma-separated list of any of the following values: "all tests" - list all found tests, "skipped tests" - list all skipped tests. Set to "none" to add no extra annotations at all'
default: 'all tests, skipped tests'
required: false
check_run_annotations_branch:
description: 'Adds check run annotations only on given branches. If not given, this defaults to the default branch of your repository, e.g. main or master. Comma separated list of branch names allowed, asterisk "*" matches all branches.'
description: 'Adds check run annotations only on given branches. Comma separated list of branch names allowed, asterisk "*" matches all branches. Defaults to event.repository.default_branch or "main, master".'
required: false
seconds_between_github_reads:
description: 'Sets the number of seconds the action waits between concurrent read requests to the GitHub API. This throttles the API usage to avoid abuse rate limits: https://docs.github.com/en/rest/overview/resources-in-the-rest-api#abuse-rate-limits'
Expand Down
Loading

0 comments on commit edf3cdc

Please sign in to comment.