Failing unit tests does not abort the action #119

ccremer · 2020-02-28T22:43:10Z

I was wondering why my unit tests seemingly passed, but no coverage report was generated.
I noticed in the job logs that my unit tests failed, but the action continued, making it seem that the check passed. But there's an error that gets ignored:

FAIL
coverage: 48.1% of statements
FAIL	1.210s
FAIL
(node:2733) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function
    at escapeData (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/node_modules/@actions/core/lib/command.js:66:10)
    at Command.toString (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/node_modules/@actions/core/lib/command.js:60:35)
    at issueCommand (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/node_modules/@actions/core/lib/command.js:23:30)
    at Object.issue (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/node_modules/@actions/core/lib/command.js:27:5)
    at Object.error (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/node_modules/@actions/core/lib/core.js:120:15)
    at /home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/lib/main.js:89:20
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.3/lib/main.js:6:65)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(node:2733) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2733) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

See here: https://github.com/ccremer/znapzend-exporter/runs/475913996?check_suite_focus=true

For now, I'll have to run the unit tests twice I guess, so the job gets properly aborted.

The text was updated successfully, but these errors were encountered:

This fixes #119.

paambaati · 2020-03-04T06:52:36Z

@ccremer v2.5.4 should fix this issue!

ccremer · 2020-03-04T09:52:33Z

Unfortunately, still not fixed: https://github.com/ccremer/znapzend-exporter/runs/484538939

jasonkarns · 2020-03-29T16:17:57Z

Same error. My coverage file is generated successfully but I received this error because the coverageLocation was empty.

Error: could not find a formatter of type undefined
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov, xccov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/runner/work/scripty/scripty")

Global Flags:
  -d, --debug   run in debug mode
(node:2717) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function
    at escapeData (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:66:10)
    at Command.toString (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:60:35)
    at issueCommand (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:23:30)
    at Object.issue (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:27:5)
    at Object.error (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/core.js:127:15)
    at /home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/lib/main.js:124:28
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/lib/main.js:6:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(node:2717) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2717) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

However, the whole action still reported a successful exit so there was no indication that anything was wrong :(

run: https://github.com/testdouble/scripty/runs/543450243?check_suite_focus=true

paambaati · 2020-03-29T16:21:33Z

Apologies, but I still haven’t been able to root-cause this properly.

Would you mind running the latest version and reporting back?

jasonkarns · 2020-03-29T16:32:00Z

I'm running 2.5.6

ccremer · 2020-03-30T13:55:04Z

If anyone needs a working solution, I moved the test command (with coverage generation) outside of this action in the meantime (in .goreleasers before.hooks to be exactly) and using 2.5.0, which seems to be stable:

    - name: Test & Publish Code Coverage
      uses: paambaati/[email protected]
      env:
        CC_TEST_REPORTER_ID: asdf
      with:
        coverageCommand: true
        prefix: (go repo name)

andreipislariu · 2020-04-07T11:23:26Z

If anyone needs a working solution, I moved the test command (with coverage generation) outside of this action in the meantime (in .goreleasers before.hooks to be exactly) and using 2.5.0, which seems to be stable:
    - name: Test & Publish Code Coverage
      uses: paambaati/[email protected]
      env:
        CC_TEST_REPORTER_ID: asdf
      with:
        coverageCommand: true
        prefix: (go repo name)

The 2.5.0 version still has some problems: I don't get the right Branch name and Commit SHA in CodeClimate, if it is getting triggered by a pull request.

I see the problem with the in 2.5.5, but just the branch name, I still have problems with the commit sha.

markkevans · 2020-04-11T19:48:04Z

Same error. My coverage file is generated successfully but I received this error because the coverageLocation was empty.

Error: could not find a formatter of type undefined
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov, xccov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/runner/work/scripty/scripty")

Global Flags:
  -d, --debug   run in debug mode
(node:2717) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function
    at escapeData (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:66:10)
    at Command.toString (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:60:35)
    at issueCommand (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:23:30)
    at Object.issue (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/command.js:27:5)
    at Object.error (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/node_modules/@actions/core/lib/core.js:127:15)
    at /home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/lib/main.js:124:28
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.6/lib/main.js:6:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(node:2717) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2717) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

However, the whole action still reported a successful exit so there was no indication that anything was wrong :(

run: https://github.com/testdouble/scripty/runs/543450243?check_suite_focus=true

@jasonkarns can you test something and add the coverageLocations to the github actions workflow file, this fixes the issue for my test repository I think see this example

paambaati · 2020-04-12T10:42:28Z

@markkevans Thanks for the workaround. If it indeed works, then it might give me a clue about the issue.

@andreipislariu @ccremer @jasonkarns Can you folks confirm that this workaround works?

ccremer · 2020-04-12T12:56:18Z

With 2.5.6 it gets interesting. Previously an unsuccessful run continued. Now it's the opposite: A successful test runs but fails to format

The workaround with specifying coverageLocations works, also for a failing unit test

In the previous comment, it also seems to be an issue in the format step

/home/runner/work/scripty/scripty/cc-reporter format-coverage [] -t undefined -o codeclimate.0.json
Error: could not find a formatter of type undefined
...
(node:2717) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function

paambaati · 2020-04-17T12:51:42Z

@markkevans @andreipislariu @ccremer @jasonkarns I've finally root-caused the issue and fixed it!

Can you folks test v2.5.7?

EDIT: Please hold off on using it as I'm validating it currently again a hobby project of mine.

EDIT 2: Should work now! phew

…imate-action#119

…i/codeclimate-action#119" This reverts commit 8214099. Because I think paambaati/codeclimate-action@473bbba might have *FINALLY* *ACTUALLY* fixed the damn issue.

ccremer · 2020-04-18T10:20:00Z

With 2.5.7 now something else broke again... https://github.com/ccremer/znapzend-exporter/runs/597634507?check_suite_focus=true:

    - name: Test & Publish Code Coverage
      uses: paambaati/[email protected]
      env:
        CC_TEST_REPORTER_ID: ...
      with:
        coverageCommand: go test -coverprofile c.out ./...
        prefix: znapzend-exporter

/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter before-build
/opt/hostedtoolcache/go/1.14.2/x64/bin/go test -coverprofile c.out ./...
ok  	znapzend-exporter	1.209s	coverage: 49.4% of statements
/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter after-build --exit-code 0
time="2020-04-18T10:17:10Z" level=error msg="failed to read file znapzend-exporter/config.go\nopen znapzend-exporter/config.go: no such file or directory" 
Error: open znapzend-exporter/config.go: no such file or directory
Usage:
  cc-test-reporter after-build [flags]

Flags:
  -s, --batch-size int               batch size for source files (default 500)
  -e, --coverage-endpoint string     endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -t, --coverage-input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov, xccov]
      --exit-code int                exit code of the test run
  -r, --id string                    reporter identifier (default "e83a1c47936e887bb37d1e37ebd3395744d4b73693a3c6e199294182be045210")
      --insecure                     send coverage insecurely (without HTTPS)
  -p, --prefix string                the root directory where the coverage analysis was performed (default "/home/runner/work/znapzend-exporter/znapzend-exporter")

Global Flags:
  -d, --debug   run in debug mode
(node:4476) UnhandledPromiseRejectionWarning: TypeError: (s || "").replace is not a function

    at escapeData (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/node_modules/@actions/core/lib/command.js:66:10)
    at Command.toString (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/node_modules/@actions/core/lib/command.js:60:35)
    at issueCommand (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/node_modules/@actions/core/lib/command.js:23:30)
    at Object.issue (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/node_modules/@actions/core/lib/command.js:27:5)
    at Object.error (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/node_modules/@actions/core/lib/core.js:127:15)
    at /home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/lib/main.js:195:20
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/paambaati/codeclimate-action/v2.5.7/lib/main.js:6:65)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

paambaati · 2020-04-18T10:22:38Z

@ccremer Can you configure your job to include the debug: true flag and the ACTIONS_STEP_DEBUG secret to true so I can get additional debugging information?

ccremer · 2020-04-18T10:24:51Z

Update: it seems the coverageLocation parameter is now mandatory, then it works:

/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter before-build
/opt/hostedtoolcache/go/1.14.2/x64/bin/go test -coverprofile c.out ./...
ok  	znapzend-exporter	1.209s	coverage: 49.4% of statements
/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter format-coverage /home/runner/work/znapzend-exporter/znapzend-exporter/c.out -t gocov -o codeclimate.0.json --prefix znapzend-exporter
/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter sum-coverage codeclimate.0.json -p 1 -o coverage.total.json
/home/runner/work/znapzend-exporter/znapzend-exporter/cc-reporter upload-coverage -i coverage.total.json
Test report uploaded successfully to Code Climate

paambaati · 2020-04-18T10:26:27Z

@ccremer Can you link me to both the runs with the additional flags turned on (see #119 (comment))?

ccremer · 2020-04-18T10:32:01Z

Run with Debug, without coverageLocations: https://github.com/ccremer/znapzend-exporter/runs/597644995?check_suite_focus=true
Run with Debug, with coverageLocations: https://github.com/ccremer/znapzend-exporter/runs/597647668?check_suite_focus=true

ccremer · 2020-04-18T10:36:02Z

A failing unit test stops the action, so thats good so far: https://github.com/ccremer/znapzend-exporter/runs/597650813?check_suite_focus=true

paambaati self-assigned this Mar 4, 2020

paambaati added a commit that referenced this issue Mar 4, 2020

main: Pass error message to @actions/core error().

bb2c677

This fixes #119.

paambaati added the bug Something isn't working label Mar 4, 2020

paambaati mentioned this issue Mar 4, 2020

Fix silently failing tests #127

Merged

paambaati closed this as completed in #127 Mar 4, 2020

paambaati reopened this Mar 4, 2020

andreipislariu mentioned this issue Apr 7, 2020

Wrong commit SHA #148

Closed

paambaati mentioned this issue Apr 17, 2020

More robust error handling & error reporting #154

Merged

paambaati closed this as completed in #154 Apr 17, 2020

paambaati mentioned this issue Apr 17, 2020

Error with 2.5.6 #150

Closed

paambaati added a commit to paambaati/websight that referenced this issue Apr 17, 2020

ci: Explicitly add coverage locations so I can debug paambaati/codecl…

8214099

…imate-action#119

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing unit tests does not abort the action #119

Failing unit tests does not abort the action #119

ccremer commented Feb 28, 2020

paambaati commented Mar 4, 2020

ccremer commented Mar 4, 2020

jasonkarns commented Mar 29, 2020 •

edited

Loading

paambaati commented Mar 29, 2020

jasonkarns commented Mar 29, 2020

ccremer commented Mar 30, 2020

andreipislariu commented Apr 7, 2020

markkevans commented Apr 11, 2020

paambaati commented Apr 12, 2020

ccremer commented Apr 12, 2020

paambaati commented Apr 17, 2020 •

edited

Loading

ccremer commented Apr 18, 2020 •

edited

Loading

paambaati commented Apr 18, 2020

ccremer commented Apr 18, 2020

paambaati commented Apr 18, 2020

ccremer commented Apr 18, 2020

ccremer commented Apr 18, 2020

Failing unit tests does not abort the action #119

Failing unit tests does not abort the action #119

Comments

ccremer commented Feb 28, 2020

paambaati commented Mar 4, 2020

ccremer commented Mar 4, 2020

jasonkarns commented Mar 29, 2020 • edited Loading

paambaati commented Mar 29, 2020

jasonkarns commented Mar 29, 2020

ccremer commented Mar 30, 2020

andreipislariu commented Apr 7, 2020

markkevans commented Apr 11, 2020

paambaati commented Apr 12, 2020

ccremer commented Apr 12, 2020

paambaati commented Apr 17, 2020 • edited Loading

ccremer commented Apr 18, 2020 • edited Loading

paambaati commented Apr 18, 2020

ccremer commented Apr 18, 2020

paambaati commented Apr 18, 2020

ccremer commented Apr 18, 2020

ccremer commented Apr 18, 2020

jasonkarns commented Mar 29, 2020 •

edited

Loading

paambaati commented Apr 17, 2020 •

edited

Loading

ccremer commented Apr 18, 2020 •

edited

Loading