Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 #30576

mkoeppe · 2020-09-15T15:18:32Z

Follow-up from #15791, #29033, #30053, the latter of which disabled use of system Python 3.6.

With Python 3.6, there is a failure building the documentation on ubuntu-bionic-standard (using /usr/bin/python3.6, https://github.com/mkoeppe/sage/runs/1106251169):

  [dochtml]   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2661: ordinal not in range(128)
  [dochtml] Full log file: logs/dochtml.log
Makefile:1876: recipe for target 'doc-html' failed

This is coming from #30002 "force C locale in docbuild and remove some obsolete stuff", which added this:

build/bin/sage-site:    export LANG=C # to ensure it is possible to scrape out non-EN locale warnings

In the present ticket, we rework the fix in #30002 so that it works with Python 3.6 too (in which setting the "C" locale sabotages UTF-8 operation.)

Supporting Python 3.6 allows Sage to use the system Python on some older Linux distributions that are still in widespread use in scientific computing, including centos-8 and fedora-{26,27,28} (although Python 3.7.x packages are also available for these). See #29033 for more details.

For the unrelated failures in the Sphinx package installation that #29033 tried to work around, see #30008.
(#30008 is a dependency of this ticket merely for the convenience of testing.)

References:

https://www.python.org/dev/peps/pep-0538/ "Coercing the legacy C locale to a UTF-8 based locale"
https://www.python.org/dev/peps/pep-0540/ "Add a new UTF-8 mode"
https://bugzilla.redhat.com/attachment.cgi?id=1233034&action=diff

Depends on #30008

CC: @antonio-rojas @dimpase @slel @orlitzky @kiwifb @embray @fchapoton

Component: build

Author: Matthias Koeppe

Branch/Commit: 945c8c5

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/30576

The text was updated successfully, but these errors were encountered:

seblabbe · 2020-09-15T15:33:46Z

comment:1

Posting my branch containing few changes that need to be done in order to build doc with ubuntu 18.04 bionic and python 3.6.

The changes are not sufficient. Other similar issues arise when sphinx is running...

New commits:

`e09d256`	`setting few utf-8 encodings`

seblabbe · 2020-09-15T15:33:46Z

Branch: public/30576

seblabbe · 2020-09-15T15:33:46Z

Commit: e09d256

seblabbe · 2020-09-16T08:44:24Z

comment:2

Two comments here:

fixing the issues in the posted branch took a while because there are many try - except clauses in the doc-building which eat up all of the useful tracebacks.
I think the remaining issues are just WARNINGS that breaks the docbuild because we decided at some point in the past to consider WARNINGS as errors (which is fine). I base this observation on some hacky modifications I did in the src/sage_setup/docbuild/sphinxbuild.py. To confirm this, do you know if there is a way to build sage documentation by considering warnings just as warnings?

dimpase · 2020-09-16T09:46:08Z

comment:3

Warnings one keeps getting are errors in processing function arguments, i.e. in introspecting Python code. Ignoring them would give you a broken doc.

error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

I think the problems come from a new(ish) sphinx which is run as a backend for our custom docbuilder, and anyway the move from Python 3.6 to 3.7 was somewhat big as far as utf-8 support is concerned.

I really think that fully supporting 3.6 is hard, and we have better things to do. Python has solved this problem by moving to 3.7, let's not reinvent the wheel here.

mkoeppe · 2020-09-16T17:21:16Z

Work Issues: rebase on #30053, revert commit be47518

sagetrac-git · 2020-09-16T23:48:45Z

Changed commit from e09d256 to 9943f25

sagetrac-git · 2020-09-16T23:48:45Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`e5f6663`	`only use locale C.UTF-8 if available, else C`
`ff6f4a6`	`assume Python 3.7 or better`
`64c3a8a`	`only test python3, to be 3.7 or 3.8`
`be47518`	`build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'`
`af0a956`	`Merge branch 't/30053/build/careful_with_C_UTF8' into t/30576/public/30576`
`8adc360`	`Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"`
`161765b`	`Revert "only test python3, to be 3.7 or 3.8"`
`c75a21b`	`setting few utf-8 encodings`
`9943f25`	`Revert "assume Python 3.7 or better"`

mkoeppe · 2020-09-16T23:49:11Z

Dependencies: #30053

mkoeppe · 2020-09-16T23:49:11Z

Changed work issues from rebase on #30053, revert commit be47518 to none

seblabbe · 2020-09-17T07:05:44Z

comment:11

Replying to @dimpase:

Warnings one keeps getting are errors in processing function arguments, i.e. in introspecting Python code. Ignoring them would give you a broken doc.
error while formatting arguments for sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular_repr.complete_primary_decomposition: 'ascii' codec can't decode byte 0xc2 in position 4774: ordinal not in range(128)

There are 2 things I don't understand here:

4774 is an integer which is larger then the length of the string of the doc of complete_primary_decomposition method

sage: len(sage.rings.polynomial.multi_polynomial_ideal.MPolynomialIdeal_singular
....: _repr.complete_primary_decomposition.__doc__)                             
3284

when I look at the html documentation of that method (after building the doc while ignoring the warnings with some hacky changes in sphinxbuild.py), I see no problem

seblabbe · 2020-09-17T08:30:23Z

comment:12

when I look at the html documentation of that method (after building the doc while ignoring the warnings with some hacky changes in sphinxbuild.py), I see no problem

With the current branch with no hacky changes, I can build the documentation with the flag --keep-going:

MAKE='make -j8' make build
MAKE='make -j8' ./sage --docbuild --no-pdf-links --keep-going all html

I am now trying to see what is wrong with the output doc...

seblabbe · 2020-09-17T14:44:38Z

Attachment: error.log

seblabbe · 2020-09-17T14:49:20Z

comment:13

the command

MAKE='make -j8' build/bin/sage-logger -p './sage --docbuild --no-pdf-links --keep-going all html ' logs/dochtml-keepgoing.log
cd logs
grep error dochtml-keepgoing.log > error.log

gives relatively few errors (see 112 lines error.log attached). Very often it is the same error, for example, 60 of them are related to crystals code either can't decode byte 0xe2 in position 6257 or can't decode byte 0xc3 in position 3272 or can't decode byte 0xcc in position 7825.

I still do no know to which doc string these positions are refering to.

slel · 2020-09-17T15:20:40Z

comment:14

Hint:

$ python3 -q
>>> b'\xe2'.decode('latin1')
'â'
>>> b'\xc3'.decode('latin1')
'Ã'
>>> b'\xcc'.decode('latin1')
'Ì'
>>>

seblabbe · 2020-09-17T15:32:15Z

comment:15

But these symbols do not occur in the crystals code:

$ cd SAGE_ROOT
$ git grep Ã *.py *pyx
$ git grep â *.py *pyx
src/sage/combinat/ncsf_qsym/tutorial.py:MacMahon, Knuth, Kreweras, Glânffrwd Thomas, Stanley. In 1984, Gessel
$ git grep Ì *.py *pyx

mkoeppe · 2020-09-22T23:36:27Z

Changed dependencies from #30053 to none

mkoeppe · 2020-09-22T23:50:16Z

Dependencies: #30008

sagetrac-git · 2020-09-22T23:57:32Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`49aa061`	`Revert "build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'"`
`a8b77cd`	`Revert "only test python3, to be 3.7 or 3.8"`
`9403629`	`build/bin/sage-system-python: Work around LC_ALL=C`
`ff0dbc6`	`build/sage_bootstrap/uncompress/tar_file.py: Fix encoding to utf-8`
`f05a110`	`Merge branch 't/30008/after__30053__sphinx_3_1_2_does_not_build_on_ubuntu__trusty_xenial_bionic___debian_jessie__centos_7__again_' into t/30576/public/30576`
`945c8c5`	`build/bin/sage-site (--docbuild): Set an English locale suuch as C.utf-8, but not the C locale, to ensure correct operation on Python 3.6`

sagetrac-git · 2020-09-22T23:57:32Z

Changed commit from 9943f25 to 945c8c5

mkoeppe · 2020-09-23T00:18:00Z

Author: Matthias Koeppe

mkoeppe · 2020-09-23T00:33:22Z

Reviewer: https://github.com/mkoeppe/sage/actions/runs/267855732, ...

seblabbe · 2020-09-23T10:58:07Z

comment:22

On Ubuntu 18.04, after uninstalling python3.8 so that python3.6 gets picked up by ./configure:

## -------------------------------------------------------- ##                        
## Checking whether SageMath should install SPKG python3... ##                        ## -------------------------------------------------------- ##                        
configure:30688: checking whether any of sqlite libpng bzip2 xz libffi is installed as
 or will be installed as SPKG                                                         
configure:30697: result: no                                                           
configure:30707: checking for python3 >= 3.6, < 3.9 with modules sqlite3, ctypes, math
, hashlib, crypt, readline, socket, zlib, distutils.core                              
configure:30713: result:                                                              
configure:30728: checking ... whether /usr/bin/python3.6 is good                                                                      
configure:30921: result: yes                                                          
configure:30923: checking for python3 >= 3.6, < 3.9 with modules sqlite3, ctypes, math
, hashlib, crypt, readline, socket, zlib, distutils.core                              
configure:30982: result: /usr/bin/python3.6                                           
configure:30997: will use system package and not install SPKG python3

I confirm that make doc now works ok. From me, it is a positive review.

I let someone change the status to positive review if no problem occurs on the github actions side.

seblabbe · 2020-09-23T10:58:53Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/267855732, ... to Sébastien Labbé, https://github.com/mkoeppe/sage/actions/runs/267855732, ...

mkoeppe · 2020-09-23T12:02:15Z

Changed reviewer from Sébastien Labbé, https://github.com/mkoeppe/sage/actions/runs/267855732, ... to Sébastien Labbé

mkoeppe · 2020-09-23T12:02:15Z

comment:24

Thanks for testing!

Also ubuntu-bionic-standard on GH Action is fixed (https://github.com/mkoeppe/sage/runs/1152418788). The failures seen in https://github.com/mkoeppe/sage/actions/runs/267855732 (for example for ubuntu-bionic-minimal) come from an unrelated ticket that I merged in the same branch.

vbraun · 2020-09-27T09:10:02Z

Changed branch from public/30576 to 945c8c5

mkoeppe added this to the sage-9.2 milestone Sep 15, 2020

mkoeppe added c: build labels Sep 15, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Python 3.6: Fix locale/encoding issues~~ Python 3.6: Fix locale/encoding issues, then re-enable Python 3.6 Sep 16, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Python 3.6: Fix locale/encoding issues, then re-enable Python 3.6~~ Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 Sep 22, 2020

mkoeppe added p: critical / 2 and removed p: major / 3 labels Sep 22, 2020

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Sep 23, 2020

mkoeppe added s: positive review and removed s: needs review labels Sep 23, 2020

vbraun removed the s: positive review label Sep 27, 2020

vbraun closed this as completed in 19be843 Sep 27, 2020

dimpase mentioned this issue Sep 21, 2020

Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6 #30053

Closed

slel mentioned this issue Oct 20, 2020

Support Python 3.9 #30184

Closed

dimpase mentioned this issue Sep 30, 2020

python3 spkg-configure: Only search for "python3", implement "configure --with-python=/PATH/TO/PYTHON" #30546

Closed

mkoeppe mentioned this issue Mar 1, 2021

ubuntu-bionic-standard (python 3.6): make ptest fails with UnicodeDecodeError #31191

Closed

Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 #30576

Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 #30576

Comments

mkoeppe commented Sep 15, 2020

seblabbe commented Sep 15, 2020

seblabbe commented Sep 15, 2020

seblabbe commented Sep 15, 2020

seblabbe commented Sep 16, 2020

dimpase commented Sep 16, 2020

This comment has been minimized.

mkoeppe commented Sep 16, 2020

sagetrac-git mannequin commented Sep 16, 2020

sagetrac-git mannequin commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

This comment has been minimized.

mkoeppe commented Sep 16, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

seblabbe commented Sep 17, 2020

seblabbe commented Sep 17, 2020

seblabbe commented Sep 17, 2020

seblabbe commented Sep 17, 2020

slel commented Sep 17, 2020

seblabbe commented Sep 17, 2020

mkoeppe commented Sep 22, 2020

This comment has been minimized.

mkoeppe commented Sep 22, 2020

This comment has been minimized.

sagetrac-git mannequin commented Sep 22, 2020

sagetrac-git mannequin commented Sep 22, 2020

mkoeppe commented Sep 23, 2020

mkoeppe commented Sep 23, 2020

seblabbe commented Sep 23, 2020

seblabbe commented Sep 23, 2020

mkoeppe commented Sep 23, 2020

mkoeppe commented Sep 23, 2020

vbraun commented Sep 27, 2020