ubuntu-bionic-standard (python 3.6): make ptest fails with UnicodeDecodeError

[mkoeppe]

(from #29913)

​https://github.com/mkoeppe/sage/runs/1583194874?check_suite_focus=true

We apply a simple fix similar to the one in #30576.

To test: tox -e docker-ubuntu-bionic-standard

CC: @kliem @dimpase @jhpalmieri

Component: doctest framework

Author: Matthias Koeppe

Branch/Commit: c674c57

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/31191

[mkoeppe]

comment:1

Unclear where this is coming from - perhaps a source file without coding header?

[mkoeppe]

comment:2

Something has changed on ubuntu-bionic since the fixes we made for Sage 9.2 (#30576). Now system python3.6 misbehaves although all LC_*/LANG variables are unset.

The specific failure can be fixed like this:

diff --git a/src/sage/doctest/control.py b/src/sage/doctest/control.py
index 7d2b84b9c3..ef5ec5c2e5 100644
--- a/src/sage/doctest/control.py
+++ b/src/sage/doctest/control.py
@@ -223,7 +223,7 @@ def skipfile(filename):
     base, ext = os.path.splitext(filename)
     if ext not in ('.py', '.pyx', '.pxd', '.pxi', '.sage', '.spyx', '.rst', '.tex'):
         return True
-    with open(filename) as F:
+    with open(filename, encoding="utf-8", errors="ignore") as F:
         line_count = 0
         for line in F:
             if nodoctest_regex.match(line):

but after that some individual doctest have similar unicode errors.
Setting the locale (for example LANG=C.UTF-8 make ptest) fixes all of these errors.

[mkoeppe]

comment:3

Likely this patch has made its way into ubuntu-bionic security updates. https://github.com/python/cpython/pull/8975/files

[mkoeppe]

comment:4

That patch makes it sound as if the change is specific to FreeBSD, but I don't see that in the actual changes.

[mkoeppe]

Branch: u/mkoeppe/ubuntu_bionic_standard__python_3_6___make_ptest_fails_with_unicodedecodeerror

[mkoeppe]

Commit: c674c57

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

New commits:

b3ff185	src/bin/sage: Refactor all -t commands through a new function exec-runtests
c674c57	src/bin/sage (-t): Set locale to an UTF-8 locale if LC_CTYPE is C or POSIX, for Python 3.6

[kliem]

Reviewer: Jonathan Kliem

[kliem]

comment:8

I think it is fine.

https://github.com/kliem/sage/pull/39/checks

[mkoeppe]

comment:9

Thanks for testing! I didn't expect difficulties

[kliem]

comment:10

LGTM.

[mkoeppe]

comment:11

Thank you!

[vbraun]

Changed branch from u/mkoeppe/ubuntu_bionic_standard__python_3_6___make_ptest_fails_with_unicodedecodeerror to c674c57