UnicodeDecodeError has occurred on windows-latest

[yizumi1012xxx]

Hi.

I use this action for the output of e2e-test result.
And, I has error about UnicodeDecodeError at windows-latest environment.
The error occurred only at windows-latest, but not occurred at mac-latest.

My workflow definition is below .

- uses: EnricoMi/publish-unit-test-result-action/composite@v1
   if: always()

And, error log is below.

  Traceback (most recent call last):
    File "D:\a\_actions\EnricoMi\publish-unit-test-result-action\v1\composite/../python/publish_unit_test_results.py", line 224, in <module>
      settings = get_settings(options, gha)
    File "D:\a\_actions\EnricoMi\publish-unit-test-result-action\v1\composite/../python/publish_unit_test_results.py", line 161, in get_settings
      event = json.load(f)
    File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\json\__init__.py", line 293, in load
      return loads(fp.read(),
    File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 562: character maps to <undefined>
  Error: Process completed with exit code 1.

I check GITHUB_EVENT_PATH, and I find that the text include emoji because of commit message.
Is emoji is cause of error?

I can't solve this problem by myself, so please give me some solution or something.
Thank you.

[EnricoMi]

Thanks for raising this. Where in the event file is that emoji, in the commit message or the pull request description? Can you paste the character here (not the github markdown replacement that starts and ends with a colon). I want to reproduce this.

Is your workflow running in an self-hosted environment or on GitHub-hosted runners?

[EnricoMi]

From what I can see from the GitHub runner code, it always writes that file in UTF8:

https://github.com/actions/runner/blob/be9632302ceef50bfb36ea998cea9c94c75e5d4d/src/Runner.Worker/ActionRunner.cs#L130-L136

[EnricoMi]

The stacktrace reveals that it looks like the file is being read with codec encodings\cp1252.py, so json.load does not use the default utf8 encoding but a system default.

[yizumi1012xxx]

@EnricoMi

Thank you for your reply.
Sample commit message is below.

fix: 🐛 fix ci

chore: 🤖 Add log

And, my workflow is running on GitHub-hosted runners.
Do you know what the problem is?

[EnricoMi]

On Windows runners, the default encoding for Python reading text files is cp1252, which fails to read utf-8 files. I could not reproduce this behaviour, but I changed the action to be explicit about the encoding when reading the event.json file.

Please try this on your branch that fails with above error:

uses: EnricoMi/publish-unit-test-result-action/composite@branch-windows-encoding-error

[EnricoMi]

Is your breaking workflow running on push or pull_request events?

[yizumi1012xxx]

@EnricoMi
Thank you very much.
I tried it, and solve this problem completely!
#130

Today, I tried execution workflow with various commit message.
(mainly 2 byte char , for example Japanese and Emoji)

And It turns out that sometimes it works and sometimes it doesn't.
But I can not find the occurrence condition of this problem finally.

Can you release this PR to main version ?

My workflow running on push event.
Trigger definition is below.

on:
  push:
    branches:
      - '**'
    tags:
      - '!*'

And my event json is below.
(Text containing confidential information is masked.)

{
  "after": "0000000000000000000000000000000000000000",
  "base_ref": null,
  "before": "0000000000000000000000000000000000000000",
  "commits": [
    {
      "author": {
        "email": "[email protected]",
        "name": "Yuichiro Izumi",
        "username": "yizumi1012xxx"
      },
      "committer": {
        "email": "[email protected]",
        "name": "Yuichiro Izumi",
        "username": "yizumi1012xxx"
      },
      "distinct": true,
      "id": "0000000000000000000000000000000000000000",
      "message": "chore: 🤖 Add log",
      "timestamp": "2021-05-12T16:22:59+09:00",
      "tree_id": "0000000000000000000000000000000000000000",
      "url": "https://github.com/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/commit/0000000000000000000000000000000000000000"
    }
  ],
  "compare": "https://github.com/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/compare/000000000000...000000000000",
  "created": false,
  "deleted": false,
  "forced": false,
  "head_commit": {
    "author": {
      "email": "[email protected]",
      "name": "Yuichiro Izumi",
      "username": "yizumi1012xxx"
    },
    "committer": {
      "email": "[email protected]",
      "name": "Yuichiro Izumi",
      "username": "yizumi1012xxx"
    },
    "distinct": true,
    "id": "0000000000000000000000000000000000000000",
    "message": "chore: 🤖 Add log",
    "timestamp": "2021-05-12T16:22:59+09:00",
    "tree_id": "0000000000000000000000000000000000000000",
    "pulls_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/pulls{/number}",
    "pushed_at": 0000000000,
    "releases_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/releases{/id}",
    "size": 1411,
    "ssh_url": "[email protected]:xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx.git",
    "stargazers": 0,
    "stargazers_count": 0,
    "stargazers_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/stargazers",
    "statuses_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/statuses/{sha}",
    "subscribers_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/subscribers",
    "subscription_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/subscription",
    "svn_url": "https://github.com/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx",
    "tags_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/tags",
    "teams_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/teams",
    "trees_url": "https://api.github.com/repos/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx/git/trees{/sha}",
    "updated_at": "2021-04-27T10:19:42Z",
    "url": "https://github.com/xxxxxxxxxxx/xxxxxxxxxxxxxxxxxxx",
    "watchers": 0,
    "watchers_count": 0
  },
  "sender": {
    "avatar_url": "https://avatars.githubusercontent.com/u/19519041?v=4",
    "events_url": "https://api.github.com/users/yizumi1012xxx/events{/privacy}",
    "followers_url": "https://api.github.com/users/yizumi1012xxx/followers",
    "following_url": "https://api.github.com/users/yizumi1012xxx/following{/other_user}",
    "gists_url": "https://api.github.com/users/yizumi1012xxx/gists{/gist_id}",
    "gravatar_id": "",
    "html_url": "https://github.com/yizumi1012xxx",
    "id": 00000000,
    "login": "yizumi1012xxx",
    "node_id": "XXXXXXXXXXXXXXXXXXXX",
    "organizations_url": "https://api.github.com/users/yizumi1012xxx/orgs",
    "received_events_url": "https://api.github.com/users/yizumi1012xxx/received_events",
    "repos_url": "https://api.github.com/users/yizumi1012xxx/repos",
    "site_admin": false,
    "starred_url": "https://api.github.com/users/yizumi1012xxx/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/yizumi1012xxx/subscriptions",
    "type": "User",
    "url": "https://api.github.com/users/yizumi1012xxx"
  }
}

[EnricoMi]

I could reproduce the issue with a push event. It is worrying that the issue is not persistent, lets hope the fix is. I am going to release this tomorrow morning (in about 12 hours).

[EnricoMi]

This has been released and is available under @v1. Thanks for raising this annoying issue.