wrong locale set by kitty

[cvlmtg]

this happens on macOS Mojave with kitty 0.13.1

 ~ kitty --version
[347 13:06:22.614967] Failed to set locale with LANG: en_IT.UTF-8
kitty 0.13.1 created by Kovid Goyal

locale en_IT is wrong and it also it breaks nvim math functions, because e.g. str2float("3.3") returns "3.0"

[kovidgoyal]

the locale comes from your OS language settings. Change them and you will be fine.

[cvlmtg]

@kovidgoyal settings are fine. this is what happens in Terminal.app

 ~ locale
LANG="it_IT.UTF-8"
LC_COLLATE="it_IT.UTF-8"
LC_CTYPE="it_IT.UTF-8"
LC_MESSAGES="it_IT.UTF-8"
LC_MONETARY="it_IT.UTF-8"
LC_NUMERIC="it_IT.UTF-8"
LC_TIME="it_IT.UTF-8"
LC_ALL=

this is what happens in kitty

 ~ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

[kovidgoyal]

that just means terminal.app is using a different technique to deduce the locale, which happens to work on your system.

[kovidgoyal]

oh and the relevant function in kitty code is cocoa_get_lang() if you think there is a bug in that, feel free to submit a patch, I can see no bug.

[eelcocramer]

I also got this issue and fixed it by setting LANG to en_US in my shell configuration (config.fish, .bashrc, etc).

cocoa_get_lang() resolved en_NL.UTF-8 on my system. I also tried setting LANG to en_US.UTF-8 but I got the same results as mentioned in the initial issue.

[tallakt]

I'm having this as well. It's pretty annoying, if I'm not able to fix this, I'll have to find another terminal. I did not find a fix for this anywhere also, so perhaps some action should be taken on behaf of kitty devs? iTerm works out of the box, as does the built in terminal app

[wookayin]

As of 0.16.0, Kitty sets LC_CTYPE='UTF-8' which is wrong. (iTerm2, Terminal do not)

[kovidgoyal]

No, it doesn't. grep the codebase for LC_CTYPE

[KimonHoffmann]

I also experience the invalid setting of LANG (en_DE in my case) and LC_TYPE (UTF-8) with kitty v0.17.4 on macOS. It is worth mentioning that the latter appears to be an implicit consequence of the former and is probably not an error on its own.

Apparently for some applications NSLocale gives strange results and I have not yet been able to pinpoint how and why it happens. In a "Swift Playground" in Xcode I observed the same behavior, while I got the correct values from a minimal Objective-C or Swift CLI application.

I have now helped myself with a dirty workaround, by explicitly setting the locale I want via:

defaults write net.kovidgoyal.kitty OSLocale -string de_DE

After doing that (and restarting kitty) the LANG variable contains the correct information and the stray and ill formed LC_CTYPE variable is gone.

[KimonHoffmann]

Even though the linked comments refer to iOS instead of macOS, they might still point in the correct direction:

• https://stackoverflow.com/a/47678964
• https://stackoverflow.com/a/46206511

[kovidgoyal]

According to those posts currentLocale should return only a locale kitty is localized for. kitty is not localized for de_DE, so I dont see how the issue could be that.

[KimonHoffmann]

You are right: It shouldn't.

This matches the problem description for me and all of the posters above that it doesn't. It returns en_* (en_DE in my case), which is not a valid locale identifier.

It does only return the correct identifier, when forced to do so via the defaults setting described above.

[kovidgoyal]

Well if you have a suggestion that works better, I'm all ears, as I said before, the relevant function is cocoa_get_lang() which is just a wrapper around currentLocale(). A possible alternative is to use preferredLanguages() instead of currentLocale(). Or alternately to validate the locale returned by currentLocale() by using newlocale() to construct a locale based on it.

[KimonHoffmann]

I already played around with an alternate implementation of cocoa_get_lang() to no avail.

In the "Swift Playground" that exhibits the same behavior as kitty, I also evaluated perferredLanguages, which does contain the desired identifier, but in practice it might be difficult to determine, which array element is the one to use.

What bothers me is that even after a few hours of investigating I don't yet have the slightest clue, what causes macOS to behave that way for certain applications.

[kovidgoyal]

On Wed, Jun 17, 2020 at 11:44:19PM -0700, Kimon Hoffmann wrote: In the "Swift Playground" that exhibits the same behavior as kitty, I also evaluated `perferredLanguages`, which *does* contain the desired identifier, but in practice it might be difficult to determine, which array element is the one to use.

According to https://developer.apple.com/documentation/foundation/nslocale/1415614-preferredlanguages the first language is the "primary language".

[KimonHoffmann]

Yes, I am aware of that, but in the case of the affected "Swift Playground" this is not the case.
The list of preferred languages has two entries and the first is en.

In an application that does not exhibit the problematic behavior, on my local system NSLocale produces the following values:

• NSLocale.autoupdatingCurrent.regionCode: DE
• NSLocale.autoupdatingCurrent.languageCode: de
• NSLocale.autoupdatingCurrent.identifier: de_DE
• NSLocale.preferredLanguages: [ "de-DE" ]

In an application that does exhibit the problematic behaviour, the values are:

• NSLocale.autoupdatingCurrent.regionCode: DE
• NSLocale.autoupdatingCurrent.languageCode: en ❗
• NSLocale.autoupdatingCurrent.identifier: en_DE ❗
• NSLocale.preferredLanguages: [ "en", "de-DE" ] ❗

This leads me to another suspicion, namely that for applications affected by this behavior the locale information is not somehow restricted - as suspected in the comments linked earlier - but instead that the language code en for some reason gets preferential treatment.

Please note: Observing this behavior is notoriously difficult, because any context, in which the environment contains a properly set LANG variable makes the problem go away. E.g. just starting kitty from the macOS Terminal application (where LANG is set to de_DE.UTF-8) via

$ /Applications/kitty.app/Contents/MacOS/kitty 

Does not exhibit the problem, while starting it via

$ env -i  /Applications/kitty.app/Contents/MacOS/kitty

or via

$ open /Applications/kitty.app

does.

[KimonHoffmann]

Found it!
Removing the following key value pair from Info.plist solves the problem:

<key>CFBundleDevelopmentRegion</key>
<string>English</string>

[kovidgoyal]

I think the best solution is to check the validity of the locale in
cocoa_get_lang() and if it is not valid, to return en_US. This can be
done via newlocale()

[KimonHoffmann]

See my latest comment above.
With this change cocoa_get_locale() works as advertised.

I attached a patch that does not change the behaviour as it is right now, but hopefully makes the function more resilient in general. Feel free to use it if you like.

patch-cocoa_get_locale.txt

[kovidgoyal]

Interesting, I wonder thought what side-effects removing that would have. I looked at for example iterms plist, and it has it set to an invalid value of "English" https://github.com/gnachman/iTerm2/blob/master/plists/iTerm2.plist#L24

…

On Thu, Jun 18, 2020 at 05:42:43AM -0700, Kimon Hoffmann wrote: Found it! Removing the following key value pair from `Info.plist` solves the problem: ``` <key>CFBundleDevelopmentRegion</key> <string>English</string> ``` -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1233 (comment)

--

_____________________________________ Dr. Kovid Goyal https://www.kovidgoyal.net https://calibre-ebook.com

_____________________________________

[kovidgoyal]

Looking at https://bugreports.qt.io/browse/QTBUG-63324?focusedCommentId=371735&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel apparently either creating an empty file at Resources/empty.lproj or adding CFBundleAllowMixedLocalizations might also work. Can you confirm?

…

On Thu, Jun 18, 2020 at 05:50:10AM -0700, Kimon Hoffmann wrote: See my latest comment above. With this change `cocoa_get_locale()` works as advertised. I attached a patch that does not change the behaviour as it is right now, but hopefully makes the function more resilient in general. Feel free to use it if you like. [patch-cocoa_get_locale.txt](https://github.com/kovidgoyal/kitty/files/4798606/locale.txt) -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1233 (comment)

--

_____________________________________ Dr. Kovid Goyal https://www.kovidgoyal.net https://calibre-ebook.com

_____________________________________

[KimonHoffmann]

I've also looked at various applications and what is "correct" (for those that have it) is not that clear cut. Some have en some have English could be that different versions of Xcode place different values there.

Removing it should make no difference as this setting corresponds to a setting in Xcode (Edit Scheme -> Run -> Options) where you can set the application locale to explicitly differ from the system locale, mostly for debugging purposes.

[KimonHoffmann]

apparently either creating an empty file at Resources/empty.lproj
or adding CFBundleAllowMixedLocalizations might also work. Can you
confirm?

I'll check that out and report back.

[KimonHoffmann]

apparently either creating an empty file at Resources/empty.lproj or adding CFBundleAllowMixedLocalizations might also work. Can you confirm?

Yes, I can confirm that both options also have the desired effect, both independently and combined.

[KimonHoffmann]

I just investigated a little bit further and realized that one statement I made earlier is not quite correct:

While all the workarounds and solutions discussed fix the incorrect setting of the LANG environment variable, the erroneous LC_CTYPE variable is still present and I have no real clue where it comes from.

[kovidgoyal]

That will be coming from the shell, kitty itself does not set LC_CTYPE,
only LANG.

[KimonHoffmann]

Yes, I didn't want to imply that kitty is explicitly setting it.
But since this variable does not get set, when using other terminal emulators[*] I'm suspecting it is some kind of side effect of some other action being performed at initialization time.

[*] other emulators tested:

• Terminal.app
• Alacritty
• xterm (does not even set LANG)

[kovidgoyal]

Like what?? The only way kitty has to communicate with child processes
is environment variables and stdin. With the standard shell startup
stdin is not used. That leaves environment variables, and kitty only
sets LANG that could possibly be relevant to locales, and that too only
on macOS, because of #90

In fact iTerm for example, actually does set LC_CTYPE https://github.com/gnachman/iTerm2/blob/faf3b633cee9d8749ca2f02a5b3226af4f93411f/sources/PTYSession.m#L1656

[KimonHoffmann]

I followed the LC_CTYPE rabbit hole a bit further and it seems to be a behavior inherent to python 3 on macOS (at least on macOS Catalina).

See:

$ env -i /usr/bin/python3
Python 3.7.3 (default, Apr 24 2020, 18:51:23)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ
environ({'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'SDKROOT': '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk', 'CPATH': '/usr/local/include', 'LIBRARY_PATH': '/usr/local/lib', '__PYVENV_LAUNCHER__': '/Applications/Xcode.app/Contents/Developer/usr/bin/python3', 'LC_CTYPE': 'UTF-8'})

Compared to:

$ env -i /usr/bin/python
Python 2.7.16 (default, Apr 17 2020, 18:29:03)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ
{'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'VERSIONER_PYTHON_VERSION': '2.7'}

According to various discussions around this topic macOSs (or python on macOSs) setting of LC_CTYPE=UTF-8 is actually conforming to POSIX, but a lot of tools have trouble with it, because they expect the locale.encoding format.

It might be an option to apply some kind of heuristic to determine whether LC_CTYPE in the environment should be considered an artifact (and not an intentional choice made by the user) that should be remedied by unsetting it.

[kovidgoyal]

Does running kitty +runpy 'import os, pprint; pprint.pprint(dict(os.environ))' also insert LC_CTYPE into the environment?

…

On Wed, Jun 24, 2020 at 05:20:30AM -0700, Kimon Hoffmann wrote: I followed the `LC_CTYPE` rabbit hole a bit further and it seems to be a behavior inherent to python 3 on macOS (at least on macOS Catalina). See: ```sh $ env -i /usr/bin/python3 Python 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.environ environ({'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'SDKROOT': '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk', 'CPATH': '/usr/local/include', 'LIBRARY_PATH': '/usr/local/lib', '__PYVENV_LAUNCHER__': '/Applications/Xcode.app/Contents/Developer/usr/bin/python3', 'LC_CTYPE': 'UTF-8'}) ``` Compared to: ```sh $ env -i /usr/bin/python Python 2.7.16 (default, Apr 17 2020, 18:29:03) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.environ {'__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3', 'VERSIONER_PYTHON_VERSION': '2.7'} ``` According to various discussions around this topic macOSs (or python on macOSs) setting of `LC_CTYPE=UTF-8` is actually conforming to POSIX, but a lot of tools have trouble with it, because they expect the `locale.encoding` format. It might be an option to apply some kind of heuristic to determine whether `LC_CTYPE` in the environment should be considered an artifact (and not an intentional choice made by the user) that should be remedied by unsetting it. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1233 (comment)

--

_____________________________________ Dr. Kovid Goyal https://www.kovidgoyal.net https://calibre-ebook.com

_____________________________________

[KimonHoffmann]

Yes, it does:

$ env -i ./kitty.app/Contents/MacOS/kitty +runpy 'import os, pprint; pprint.pprint(dict(os.environ))'
{'LC_CTYPE': 'UTF-8', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x3'}

[kovidgoyal]

Hmm not sure what a good solution is here then. Since UTF-8 is a valid
value for LC_CTYPE and we have no way to know if it existed before
python inserted it, there is no real way to filter it out.

[KimonHoffmann]

Yes, I agree that it's tricky.

One heuristic I could imagine would be (only on macOS) to see, whether

1. LC_CTYPE is equal to UTF-8 and
2. LC_ALL is unset (or alternatively no other LC_* variable is set)

[kovidgoyal]

Should be fixed by my latest commit

…

On Wed, Jun 24, 2020 at 08:10:03AM -0700, Kimon Hoffmann wrote: Yes, I agree that it's tricky. One heuristic I could imagine would be (only on macOS) to see, whether 1. `LC_CTYPE` is equal to `UTF-8` and 2. `LC_ALL` is unset (or alternatively no other `LC_*` variable is set) -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1233 (comment)

--

_____________________________________ Dr. Kovid Goyal https://www.kovidgoyal.net https://calibre-ebook.com

_____________________________________

[KimonHoffmann]

Yes, I can confirm that the erroneous LC_CTYPE setting is gone.

And i really like the way you solved it, because no guessing is involved.

[kaiz0711]

locale
if use zsh
export LANGUAGE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_TYPE=en_US.UTF-8

[garrettmaring]

Still seeing non-en_US after setting above locales on kitty 27.1 on M1 mac

[garrettmaring]

Still seeing non-en_US after setting above locales on kitty 27.1 on M1 mac

the fix by was

symbol_map

# "Nerd Fonts - Pomicons"
symbol_map  U+E000-U+E00D Symbols Nerd Font

# "Nerd Fonts - Powerline"
symbol_map U+e0a0-U+e0a2,U+e0b0-U+e0b3 Symbols Nerd Font

# "Nerd Fonts - Powerline Extra"
symbol_map U+e0a3-U+e0a3,U+e0b4-U+e0c8,U+e0cc-U+e0d2,U+e0d4-U+e0d4 Symbols Nerd Font

# "Nerd Fonts - Symbols original"
symbol_map U+e5fa-U+e62b Symbols Nerd Font

# "Nerd Fonts - Devicons"
symbol_map U+e700-U+e7c5 Symbols Nerd Font

# "Nerd Fonts - Font awesome"
symbol_map U+f000-U+f2e0 Symbols Nerd Font

# "Nerd Fonts - Font awesome extension"
symbol_map U+e200-U+e2a9 Symbols Nerd Font

# "Nerd Fonts - Octicons"
symbol_map U+f400-U+f4a8,U+2665-U+2665,U+26A1-U+26A1,U+f27c-U+f27c Symbols Nerd Font

# "Nerd Fonts - Font Linux"
symbol_map U+F300-U+F313 Symbols Nerd Font

#  Nerd Fonts - Font Power Symbols"
symbol_map U+23fb-U+23fe,U+2b58-U+2b58 Symbols Nerd Font

#  "Nerd Fonts - Material Design Icons"
symbol_map U+f500-U+fd46 Symbols Nerd Font

# "Nerd Fonts - Weather Icons"
symbol_map U+e300-U+e3eb Symbols Nerd Font

# Misc Code Point Fixes
symbol_map U+21B5,U+25B8,U+2605,U+2630,U+2632,U+2714,U+E0A3,U+E615,U+E62B Symbols Nerd Font

which fixes the problem despite showing deprecation warnings for the symbol_map key (latest versions of kitty, macos)