Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6 #30053

dimpase · 2020-07-03T12:25:54Z

In #29033 in build/bin/sage-spkg LC_ALL was changed to C.UTF-8
However, not all systems have it.

See also:

Set LC_ALL=C to build PDF docs #22659

Follow-up:

After #30053, sphinx 3.1.2 does not build on ubuntu-{trusty,xenial,bionic}, debian-jessie, centos-7 (again) #30008 Fix sage-system-python
macOS: Doctest failures in some locales #30586 macOS: Doctest failures in some locales
Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 #30576 Python 3.6: Fix locale/encoding issues, then re-enable Python 3.6

CC: @antonio-rojas @mkoeppe @slel @orlitzky @kiwifb @embray @fchapoton

Component: build

Author: Dima Pasechnik, Matthias Koeppe

Branch: be47518

Reviewer: Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30053

The text was updated successfully, but these errors were encountered:

dimpase · 2020-07-04T10:04:19Z

comment:3

an easy way out is just to check whether the locale change worked, and if not, use C locale, not C.UTF-8. Perhaps print a warning.

mkoeppe · 2020-07-04T15:26:48Z

comment:5

Replying to @dimpase:

an easy way out is just to check whether the locale change worked, and if not, use C locale, not C.UTF-8.

+1

dimpase · 2020-07-05T12:13:24Z

Author: Dima Pasechnik

dimpase · 2020-07-05T12:13:24Z

Commit: 37e042c

dimpase · 2020-07-05T12:13:24Z

New commits:

`37e042c`	`only use locale C.UTF-8 if available, else C`

dimpase · 2020-07-05T12:13:24Z

Branch: u/dimpase/build/careful_with_C_UTF8

kliem · 2020-07-05T13:06:50Z

comment:7

Either this ticket is a duplicate of #30008 or it should make #30008 obsolete.

kliem · 2020-07-05T13:15:46Z

comment:8

I started test runs:

https://github.com/kliem/sage/pull/20/checks

dimpase · 2020-07-05T15:19:40Z

comment:9

this ticket was a result of a bug report on Arch, not centos. Hopefully it works for #30008 too.

kliem · 2020-07-05T15:38:15Z

comment:10

Tests did not complete, because the 9.2.beta3 tests fail everywhere.

https://github.com/sagemath/sage/actions/runs/157607524

Is this a github issue or have we broken sage?

 Step 13/18 : RUN ./bootstrap
 ---> Running in 89427ef5c1c4
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
src/doc/bootstrap:48: installing src/doc/en/installation/arch.txt and src/doc/en/installation/arch-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt and src/doc/en/installation/debian-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/fedora.txt and src/doc/en/installation/fedora-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/cygwin.txt and src/doc/en/installation/cygwin-optional.txt
src/doc/bootstrap:48: installing src/doc/en/installation/homebrew.txt and src/doc/en/installation/homebrew-optional.txt
src/doc/bootstrap:55: installing src/doc/en/reference/spkg/*.rst
src/doc/bootstrap:83: installing src/doc/en/reference/repl/options.txt
src/doc/bootstrap: line 84: src/doc/en/reference/repl/options.txt: No such file or directory
The command '/bin/sh -c ./bootstrap' returned a non-zero code: 1

kliem · 2020-07-05T15:39:49Z

comment:11

I just found #30064. Edit: I was cc on that, but I didn't realize how serious this is.

Ok. I'll run a new test then.

kliem · 2020-07-05T19:03:47Z

comment:12

This breaks building sphinx on windows.

https://github.com/kliem/sage/runs/838933940

Same error as #30008.

As far as I understand the problem is that we need some sort of UTF to make the sphinx build work.
It appears that on cygwin the default is better than C and C.UTF-8 does not work.
So maybe C is not the best alternative for C.UTF-8.

Btw, strangely centos 7 appears to work with the current beta. I don't know what happened. (And I don't know yet, if this behavior is stable).

kliem · 2020-07-06T05:53:58Z

comment:13

And it breaks centos 8.

embray · 2020-07-06T11:07:55Z

comment:14

Replying to @kliem:

This breaks building sphinx on windows.

https://github.com/kliem/sage/runs/838933940

Same error as #30008.

As far as I understand the problem is that we need some sort of UTF to make the sphinx build work.
It appears that on cygwin the default is better than C and C.UTF-8 does not work.
So maybe C is not the best alternative for C.UTF-8.

I'm not sure what you mean here. C.UTF-8 is supported on Cygwin and is in fact the default locale in absence of any other settings: https://www.cygwin.com/cygwin-ug-net/setup-locale.html

The default locale in the absence of the aforementioned locale environment variables is "C.UTF-8".

dimpase · 2020-07-06T11:13:59Z

comment:15

the error UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128):

2020-07-05T16:29:58.1676169Z [sphinx-3.0.4.p0] installing. Log file: /cygdrive/d/a/sage/sage/logs/pkgs/sphinx-3.0.4.p0.log
2020-07-05T16:30:01.7883565Z   [sphinx-3.0.4.p0] error installing, exit status 1. End of log file:
2020-07-05T16:30:01.8141086Z   [sphinx-3.0.4.p0]   Found local metadata for sphinx-3.0.4.p0
2020-07-05T16:30:01.8155620Z   [sphinx-3.0.4.p0]   Attempting to download package Sphinx-3.0.4.tar.gz from mirrors
2020-07-05T16:30:01.8169727Z   [sphinx-3.0.4.p0]   http://mirrors.mit.edu/sage/spkg/upstream/sphinx/Sphinx-3.0.4.tar.gz
2020-07-05T16:30:01.8174798Z   [sphinx-3.0.4.p0]   [......................................................................]
2020-07-05T16:30:01.8191111Z   [sphinx-3.0.4.p0]   sphinx-3.0.4.p0
2020-07-05T16:30:01.8193080Z   [sphinx-3.0.4.p0]   ====================================================
2020-07-05T16:30:01.8197343Z   [sphinx-3.0.4.p0]   Setting up build directory for sphinx-3.0.4.p0
2020-07-05T16:30:01.8217424Z   [sphinx-3.0.4.p0]   Traceback (most recent call last):
2020-07-05T16:30:01.8221790Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/sage-uncompress-spkg", line 23, in <module>
2020-07-05T16:30:01.8222267Z   [sphinx-3.0.4.p0]       run()
2020-07-05T16:30:01.8222576Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/cmdline.py", line 72, in run
2020-07-05T16:30:01.8222857Z   [sphinx-3.0.4.p0]       unpack_archive(archive, dirname)
2020-07-05T16:30:01.8223251Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/action.py", line 68, in unpack_archive
2020-07-05T16:30:01.8223583Z   [sphinx-3.0.4.p0]       archive.extractall(members=archive.names)
2020-07-05T16:30:01.8223861Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/tar_file.py", line 96, in extractall
2020-07-05T16:30:01.8224117Z   [sphinx-3.0.4.p0]       **kwargs)
2020-07-05T16:30:01.8224323Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2010, in extractall
2020-07-05T16:30:01.8224793Z   [sphinx-3.0.4.p0]       numeric_owner=numeric_owner)
2020-07-05T16:30:01.8225151Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2052, in extract
2020-07-05T16:30:01.8225442Z   [sphinx-3.0.4.p0]       numeric_owner=numeric_owner)
2020-07-05T16:30:01.8225898Z   [sphinx-3.0.4.p0]     File "/cygdrive/d/a/sage/sage/build/bin/../sage_bootstrap/uncompress/tar_file.py", line 122, in _extract_member
2020-07-05T16:30:01.8226166Z   [sphinx-3.0.4.p0]       **kwargs)
2020-07-05T16:30:01.8226601Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2122, in _extract_member
2020-07-05T16:30:01.8226883Z   [sphinx-3.0.4.p0]       self.makefile(tarinfo, targetpath)
2020-07-05T16:30:01.8227488Z   [sphinx-3.0.4.p0]     File "/usr/lib/python3.6/tarfile.py", line 2163, in makefile
2020-07-05T16:30:01.8227940Z   [sphinx-3.0.4.p0]       with bltn_open(targetpath, "wb") as target:
2020-07-05T16:30:01.8228249Z   [sphinx-3.0.4.p0]   UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128)
2020-07-05T16:30:01.8228542Z   [sphinx-3.0.4.p0]   ************************************************************************
2020-07-05T16:30:01.8228988Z   [sphinx-3.0.4.p0]   Error: failed to extract /cygdrive/d/a/sage/sage/upstream/Sphinx-3.0.4.tar.gz
2020-07-05T16:30:01.8229264Z   [sphinx-3.0.4.p0]   ************************************************************************
2020-07-05T16:30:01.8229537Z   [sphinx-3.0.4.p0] Full log file: /cygdrive/d/a/sage/sage/logs/pkgs/sphinx-3.0.4.p0.log

dimpase · 2020-07-06T11:15:38Z

comment:16

could it be that locale on Cygwin is not installed by default?

However, https://www.cygwin.com/cygwin-ug-net/setup-locale.html says:

Note
For a list of locales supported by your Windows machine, use the new locale -a command, which is part of the Cygwin package. For a description see locale(1)

kliem · 2020-07-12T19:40:31Z

comment:17

Replying to @kliem:

I started test runs:

https://github.com/kliem/sage/pull/20/checks

I just started rerunning those tests on top of the current beta. Maybe that stuff just goes away by itself.

kliem · 2020-07-12T22:31:16Z

comment:18

Still causes this error.

antonio-rojas · 2020-07-23T19:31:26Z

comment:19

If the centos issue is caused by the sphinx upgrade (according to #30008), why is it blocking this? This is meant to fix another (very annoying) issue on Arch.

kliem · 2020-07-23T20:33:09Z

comment:20

It appears that #30008 fixed itself. However, this here broke the cygwin sphinx build, last I checked.

It have no clue what is going on, but with this ticket we go from passing to failing.

mkoeppe · 2020-07-23T21:09:47Z

comment:21

It seems a default setting on Cygwin is LANG=en_US.UTF-8. Perhaps we can try to only set LC_ALL if LANG is not already set or something like this.

mkoeppe · 2020-07-23T22:42:32Z

comment:22

Also it should be investigated whether it was really necessary to add this line in #29033 to achieve Python 3.6 support. In particular note that sage-uncompress-spkg uses sage-system-python (which can even be python2) -- which really has nothing to do with Python 3.6 support (which is about PYTHON_FOR_VENV).

orlitzky · 2020-08-05T14:19:41Z

comment:25

What problems arise if we drop the locale mangling entirely? Trac #15791 doesn't mention a problem.

sagetrac-git · 2020-09-16T15:06:56Z

Changed commit from e5f6663 to ff6f4a6

dimpase · 2020-09-16T15:07:27Z

comment:81

OK, so how about this?

orlitzky · 2020-09-16T15:17:21Z

comment:82

I think that's the best solution, possibly combined with dropping 3.6 from the list of system pythons that we accept. Portability is nice and all, but we're wasting a lot of time trying to support an evolutionary dead-end with py3.6.

mkoeppe · 2020-09-16T16:16:57Z

comment:83

Fine with me - then please also adjust python3's spkg-configure...

sagetrac-git · 2020-09-16T16:46:38Z

Changed commit from ff6f4a6 to 64c3a8a

sagetrac-git · 2020-09-16T16:46:38Z

Branch pushed to git repo; I updated commit sha1. New commits:

`64c3a8a`	`only test python3, to be 3.7 or 3.8`

dimpase · 2020-09-16T16:47:47Z

comment:85

Replying to @mkoeppe:

Fine with me - then please also adjust python3's spkg-configure...

fixed, as discussed - only test python3, to be either 3.7 or 3.8.

Please review

mkoeppe · 2020-09-16T17:11:21Z

Changed branch from u/dimpase/build/careful_with_C_UTF8 to u/mkoeppe/build/careful_with_C_UTF8

mkoeppe · 2020-09-16T17:12:24Z

comment:87

I have made an adjustment. I don't like mixing the change discussed in #30546 into this ticket

New commits:

`be47518`	`build/pkgs/python3/spkg-configure.m4: Use 'python >= 3.7'; keep checking for 'python3.8, python3.7'`

mkoeppe · 2020-09-16T17:12:24Z

Changed commit from 64c3a8a to be47518

mkoeppe · 2020-09-16T17:14:37Z

comment:88

Rationale: The changed search behavior would lead to confusing reports on sage-release.

mkoeppe · 2020-09-16T17:16:57Z

Changed author from Dima Pasechnik to Dima Pasechnik, Matthias Koeppe

vbraun · 2020-09-19T13:24:05Z

Changed branch from u/mkoeppe/build/careful_with_C_UTF8 to be47518

mkoeppe · 2020-09-21T05:38:44Z

Changed commit from be47518 to none

dimpase added this to the sage-9.2 milestone Jul 3, 2020

dimpase added the p: major / 3 label Jul 3, 2020

dimpase added t: bug labels Jul 4, 2020

dimpase added the s: needs review label Jul 5, 2020

kliem added s: needs work and removed s: needs review labels Jul 5, 2020

sagetrac-git mannequin added s: needs review and removed s: positive review labels Sep 16, 2020

mkoeppe added s: positive review and removed s: needs review labels Sep 16, 2020

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests~~ Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6 Sep 16, 2020

This comment has been minimized.

Sign in to view

vbraun removed the s: positive review label Sep 19, 2020

vbraun closed this as completed in a760832 Sep 19, 2020

This comment has been minimized.

Sign in to view

orlitzky mentioned this issue Sep 21, 2020

After #30053, sphinx 3.1.2 does not build on ubuntu-{trusty,xenial,bionic}, debian-jessie, centos-7 (again) #30008

Closed

This was referenced Oct 20, 2020

Support Python 3.9 #30184

Closed

Drop Python 3.6 support #30551

Closed

mkoeppe mentioned this issue Sep 27, 2020

Python 3.6: Fix locale/encoding issues in docbuild, then re-enable Python 3.6 #30576

Closed

Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6 #30053

Python 3.7+: setlocale: LC_ALL: cannot change locale (C.UTF-8) from build/bin/sage-spkg and in doctests; disable use of system Python 3.6 #30053

Comments

dimpase commented Jul 3, 2020

dimpase commented Jul 4, 2020

mkoeppe commented Jul 4, 2020

dimpase commented Jul 5, 2020

dimpase commented Jul 5, 2020

dimpase commented Jul 5, 2020

dimpase commented Jul 5, 2020

kliem commented Jul 5, 2020

kliem commented Jul 5, 2020

dimpase commented Jul 5, 2020

kliem commented Jul 5, 2020

kliem commented Jul 5, 2020

kliem commented Jul 5, 2020

kliem commented Jul 6, 2020

embray commented Jul 6, 2020

dimpase commented Jul 6, 2020

dimpase commented Jul 6, 2020

kliem commented Jul 12, 2020

kliem commented Jul 12, 2020

antonio-rojas commented Jul 23, 2020

kliem commented Jul 23, 2020

mkoeppe commented Jul 23, 2020

mkoeppe commented Jul 23, 2020

orlitzky commented Aug 5, 2020

sagetrac-git mannequin commented Sep 16, 2020

dimpase commented Sep 16, 2020

orlitzky commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

sagetrac-git mannequin commented Sep 16, 2020

sagetrac-git mannequin commented Sep 16, 2020

dimpase commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

mkoeppe commented Sep 16, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vbraun commented Sep 19, 2020

mkoeppe commented Sep 21, 2020

This comment has been minimized.