stderr exec check results show in goss validate

[MichaelThamm]

Describe the feature:
have-patterns replace the *"object: bytes.Reader" in the goss validate result with something human-readable. This is great, but this only works for stdout not with stderr. A simple test explaining what I mean:

  sample-check:
    exec: echo "I need to know what this is"
    exit-status: 0
    stdout:
      have-patterns:
      - foo

This check fails as expected because foo does not equal the stdout result and running

goss v -f json| jq '.results[] ."matcher-result"'

results in:

{
  "actual": "I need to know what this is\n",
  "expected": [
    "foo"
  ],
  "extra-elements": null,
  "found-elements": null,
  "message": "to have patterns",
  "missing-elements": [
    "foo"
  ],
  "transform-chain": null,
  "untransformed-value": "I need to know what this is\n"
}

Let's try this with a check that errors.

  sample-check:
    exec: cat this_file_does_not_exist.txt
    exit-status: 0
    stdout:
      have-patterns:
      - foo

This check fails as expected because foo does not equal the stdout result (only stderr) and running

goss v -f json| jq '.results[] ."matcher-result"'

results in:

{
  "actual": "",
  "expected": [
    "foo"
  ],
  "extra-elements": null,
  "found-elements": null,
  "message": "to have patterns",
  "missing-elements": [
    "foo"
  ],
  "transform-chain": null,
  "untransformed-value": ""
}

This is not great because "actual": "" is empty. A workaround can be achieved with switching the check to:

exec: cat this_file_does_not_exist.txt -> exec: "cat this_file_does_not_exist.txt" 2>&1

Describe the solution you'd like

Ideally, the goss.yaml author should not have to consider redirecting stderr to stdout per check. Maybe add a CLI command option like goss v --include-stderr.

Describe alternatives you've considered

If change in functionality to the binary is not possible (for whatever reason), I would like to see the docs updated explaining what to do in case a check outputs to stderr because I think this is quite a common scenario for Goss users.

[aelsabbahy]

There's an stderr check in goss.

  sample-check:
    exec: cat this_file_does_not_exist.txt
    exit-status: 0
    stderr:
      have-patterns:
      - foo

Command doc

[MichaelThamm]

Ahh, I did not read enough. Thanks and I will close the issue!

[MichaelThamm]

FYI, this Pattern link is broken in Manually editing Goss files.

Also, I was not able to determine how to properly write stderr checks with have-patterns because when I write:

    stderr:
      have-patterns:
      - ""

it will always succeed since "" is always in stderr regardless of pass or fail. I cannot use inverse-checks in stderr which I tried with:

    stderr:
      have-patterns:
      - "!ERROR"

So how can I write a check that will show the stderr (in human-readable format using have-patterns) ONLY when stderr is not empty.

[aelsabbahy]

stderr: ""

[MichaelThamm]

Thanks for your help and timely replies, it sure is appreciated. I will try my best to get Goss integrated into our stack!