Crashes unless stdout/err encoding is UTF-8

[ocharles]

I've seen this on two projects now. I'm not sure if it's a problem with stack, my terminal, or hedgehog - but let's start the discussion here:

→ stack test                                                                                      ~/work/circuithub/api
circuithub-api-0.0.4: build (lib + exe + test)
circuithub-api-0.0.4: copy/register
circuithub-api-0.0.4: test (suite: circuithub-api-tests)
Completed 2 action(s).
Log files have been written to: /home/ollie/work/circuithub/api/.stack-work/logs/
Test suite failure for package circuithub-api-0.0.4
    circuithub-api-tests:  exited with: ExitFailure 1
Full log available at /home/ollie/work/circuithub/api/.stack-work/logs/circuithub-api-0.0.4-test.log


    circuithub-api-tests: setNumCapabilities: not supported in the non-threaded RTS
    circuithub-api-tests: <stdout>: commitBuffer: invalid argument (invalid character)

Yet running the binary directly works fine.

[jacobstanley]

    circuithub-api-tests: setNumCapabilities: not supported in the non-threaded RTS

^ so this is likely because the test executable is not being compiled with -threaded, but that probably isn't why it's not working, it's just a warning.

    circuithub-api-tests: <stdout>: commitBuffer: invalid argument (invalid character)

^ this looks very similar to this problem #32

[ocharles]

If anyone else lands here, adding this to main fixes things:

  hSetEncoding stdout utf8
  hSetEncoding stderr utf8

[moodmosaic]

If anyone else lands here, adding this to main fixes things:

hSetEncoding stdout utf8
hSetEncoding stderr utf8

This seems to improve the situation (no <stdout>: commitBuffer: ... error) but the output looks weird:

[moodmosaic]

Here's also a screenshot from a failing property:

[ocharles]

What is your 'locale' command output?

…

On Fri, 28 Jul 2017, 10:20 pm Nikos Baxevanis, ***@***.***> wrote: Here's also a screenshot from a failing property: [image: image] <https://user-images.githubusercontent.com/287532/28736680-aac6e2d8-73f3-11e7-8e66-c018eca6a564.png> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#110 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABRjjb6bwHhegCSCR31L-Klusix2qQxks5sSlCVgaJpZM4OfVEq> .

[moodmosaic]

According to systeminfo it is en-us;English (United States).

[moodmosaic]

In bash, locale returns these:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

[chris-martin]

I'm also seeing this problem. Runs okay on my machine, but fails on Travis CI.

[moodmosaic]

@chris-martin, you mean the output looks like the images above?

[chris-martin]

@moodmosaic Output looks like this:

Test suite failure for package loc-0.1.3.0
    hedgehog:  exited with: ExitFailure 1
Full log available at /home/travis/build/chris-martin/haskell-libraries/.stack-work/logs/loc-0.1.3.0-test.log
    Examples: 84  Tried: 84  Errors: 0  Failures: 0
    hedgehog: <stdout>: commitBuffer: invalid argument (invalid character)
The command "stack --nix --no-terminal test" exited with 1.

https://travis-ci.org/chris-martin/haskell-libraries/builds/259263874

[moodmosaic]

@chris-martin, even after trying this workaround?

[chris-martin]

@moodmosaic Ah, no, that workaround does seem to fix it.

[thumphries]

Wondering if this could be avoided by adding a main runner (as suggested in #97 ) that sets encoding accordingly?

[kirelagin]

Actually, if I understand the issue right, a proper fix would be to do something like this: https://phabricator.haskell.org/D1153. It would be especially nice not only to transliterate Haskell identifiers, but also refrain from using all the utf-8 goodness when the terminal does not support it.

[moodmosaic]

@kirelagin I tried that on Win10 (phabricator.haskell.org/D1153) and seems to be doing a better job indeed:

import           Hedgehog
import qualified Hedgehog.Gen       as Gen
import qualified Hedgehog.Range     as Range

import           GHC.IO.Encoding    (mkTextEncoding, textEncodingName)
import           System.IO          (Handle, hGetEncoding, hSetEncoding, stderr,
                                     stdout)

tests :: IO Bool
tests = do
  hSetTranslit stdout
  hSetTranslit stderr
  checkParallel $$discover

-- | Change the character encoding of the given Handle to transliterate on
--   unsupported characters instead of throwing an exception.
--   https://phabricator.haskell.org/D1153
hSetTranslit :: Handle -> IO ()
hSetTranslit h = do
  menc <- hGetEncoding h
  case fmap textEncodingName menc of
    Just name | '/' `notElem` name -> do
      enc <- mkTextEncoding $ name ++ "//TRANSLIT"
      hSetEncoding h enc
    _ -> pure ()

[moodmosaic]

So, the bare minimums to keep Windows happy are

import           System.IO (hSetEncoding, stdout, stderr, utf8)

main :: IO ()
main = do -- ^ or put those in your test runner's main
  hSetEncoding stdout utf8
  hSetEncoding stderr utf8
  ...

You might have to change the active console code page to UTF-8

chcp 65001

--

[moodmosaic]

/cc @angerman

[dmjio]

export LC_ALL=C.UTF-8 worked for me

[moodmosaic]

Thanks! Good to know that.

[zmrocze]

One can arrive at this issue when running from nix-shell. Apparently there are some problems and the solution from a similar problem worked for me. I guess it does no harm if I repeat it here:

Set LOCALE_ARCHIVE=/usr/lib/locale/locale-archive

or to different appropriate path [see links ^].