Problem with ipython / readline / ncurses

[maxnoe]

I did a fresh install of anaconda3 this morning, Linux x86_64.

When i first opened ipython, no colour was present and tabcompletion did not work. Along came the error (or warning):

WARNING: Readline services not available or not loaded.
WARNING: The auto-indent feature requires the readline library

Which was not gone after i did conda install readline.

I then tried to do import readline, which caused the error

>>> import readline
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libncursesw.so.5: cannot open shared object file: No such file or directory

After conda install ncurses ipython and readline worked again, besides one minor thing:
Ctrl + l does not clear the screen anymore, %clear works.

[maxnoe]

I think this is connected to #42
as i'm also having the problem with blessings 'could not find terminfo database'

[maxnoe]

I narrowed it down to the provided version of the ncurses lib, it does not finde the terminfo db:

blessings is broken:

In [3]: import blessings

In [4]: blessings.Terminal()
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-4-c5bca101e4fa> in <module>()
----> 1 blessings.Terminal()

/home/maxnoe/.local/anaconda3/lib/python3.4/site-packages/blessings/__init__.py in __init__(self, kind, stream, force_styling)
    104             # somewhere.
    105             setupterm(kind or environ.get('TERM', 'unknown'),
--> 106                       self._init_descriptor)
    107 
    108         self.stream = stream

error: setupterm: could not find terminfo database

This works with a system ncurses 5.9 library, but arch switched to 6, this is why it stopped working.

[itoijala]

Works for me after conda install ncurses and restarting ipython.
I don't have a module blessings.

[maxnoe]

Does ctrl + l clear your screen?

Does

import curses
curses.setupterm() 

work?

[itoijala]

I should learn to read better…

[itoijala]

EDIT: this is wrong. The trace is from my working native IPython.

The terminfo db is found:
strace gives:

stat("/home/ismo/.terminfo", 0x55e304e28170) = -1 ENOENT (No such file or directory)
 5345 stat("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=84, ...}) = 0
 5346 access("/usr/share/terminfo/r/rxvt-unicode-256color", R_OK) = 0
 5347 open("/usr/share/terminfo/r/rxvt-unicode-256color", O_RDONLY) = 8
 5348 fstat(8, {st_mode=S_IFREG|0644, st_size=2234, ...}) = 0
 5349 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbe3ea55000
 5350 read(8, "\32\1N\0\35\0\37\0p\1$\5rxvt-unicode-256colo"..., 4096) = 2234
 5351 read(8, "", 4096)                       = 0
 5352 close(8)                                = 0
 5353 munmap(0x7fbe3ea55000, 4096)            = 0

I guess the db is not compatible?

[maxnoe]

So the problem is, that the terminfo db is generated by ncurses6 and the provided ncurses is ncurses 5.9?

Does that mean conda has to provide its own terminfodb?

[ahmadia]

Folks, can you confirm that this is an issue tied in to having ncurses6 on your system?

[maxnoe]

Yes.

EDIT: actually no, the conda ncurses is broken. we just didn't notice before because we were using a system ncurses 5.

[itoijala]

The strace above was accidentally from my working native ipython.
Anaconda ipython does not even try to stat anything named terminfo.
Setting TERMINFO doesn't help.

[maxnoe]

Should have mentioned that, also tried to set TERMINFO.

[itoijala]

I rebuilt ncurses without --enable-termcap and it works now.
Setting TERMINFO is necessary, otherwise it looks for the files inside the anaconda tree.
This could be solved by compiling in a better default path.

1. Get the files from here: https://github.com/conda/conda-recipes/tree/master/ncurses
2. Remove --enable-termcap from build.sh
3. Get the patch for compiling with GCC 5.1: https://raw.githubusercontent.com/cloudius-systems/osv/07e2d9032dbb3f4f2b0d0133e0eccd5be05dd05d/modules/ncurses/ncurses-5.9-gcc-5.patch
4. Add it to meta.yaml
5. conda build ncurses
6. conda install -f <path output by build>
7. TERMINFO=/usr/share/terminfo ipython

EDIT: My conclusion: normally anaconda uses the system ncurses which looks for and finds terminfo. This doen't work if the system has ncurses6. The conda ncurses never looks for terminfo, only termcap.

[maxnoe]

Could you post the new meta.yaml?

[itoijala]

meta.yml:

package:
  name: ncurses
  version: 5.9

source:
  fn: ncurses-5.9.tar.gz
  url: http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.9.tar.gz
  md5: 8cb9c412e5f2d96bc6f459aa8c6282a1
  patches:
    # http://lists.gnu.org/archive/html/bug-ncurses/2011-04/txtkWQqiQvcZe.txt
    - fix.patch
    - ncurses-5.9-gcc-5.patch

build:
  number: 5 # [linux64]
  number: 6 # [linux32]
  number: 1 # [osx]
  detect_binary_files_with_prefix: true

about:
  home: http://www.gnu.org/software/ncurses/
  license:  Free software (X11 License)

ncurses-5.9-gcc-5.patch:

diff --git a/ncurses/base/MKlib_gen.sh b/ncurses/base/MKlib_gen.sh
index d8cc3c9..b91398c 100755
--- ncurses/base/MKlib_gen.sh
+++ ncurses/base/MKlib_gen.sh
@@ -474,11 +474,22 @@ sed -n -f $ED1 \
    -e 's/gen_$//' \
    -e 's/  / /g' >>$TMP

+cat >$ED1 <<EOF
+s/  / /g
+s/^ //
+s/ $//
+s/P_NCURSES_BOOL/NCURSES_BOOL/g
+EOF
+
+# A patch discussed here:
+#  https://gcc.gnu.org/ml/gcc-patches/2014-06/msg02185.html
+# introduces spurious #line markers.  Work around that by ignoring the system's
+# attempt to define "bool" and using our own symbol here.
+sed -e 's/bool/P_NCURSES_BOOL/g' $TMP > $ED2
+cat $ED2 >$TMP
+
 $preprocessor $TMP 2>/dev/null \
-| sed \
-   -e 's/  / /g' \
-   -e 's/^ //' \
-   -e 's/_Bool/NCURSES_BOOL/g' \
+| sed -f $ED1 \
 | $AWK -f $AW2 \
 | sed -f $ED3 \
 | sed \

[maxnoe]

The tests fail for me after bulding with a missings lib: libncursesw.so.5 which is indeed not in envs/_test/lib

BUILD END: ncurses-5.9-5
TEST START: ncurses-5.9-5
Fetching package metadata: ......
Solving package specifications: ............
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ncurses-5.9                |                5         711 KB

The following NEW packages will be INSTALLED:

    ncurses:    5.9-5        
    openssl:    1.0.1k-1     
    pip:        7.1.2-py34_0 
    python:     3.4.3-1      
    readline:   6.2-2        
    setuptools: 18.3.2-py34_0
    sqlite:     3.8.4.1-1    
    system:     5.8-2        
    tk:         8.5.18-0     
    wheel:      0.26.0-py34_1
    xz:         5.0.5-0      
    zlib:       1.2.8-0      

Fetching packages ...
ncurses-5.9-5. 100% |###############################| Time: 0:00:00  81.87 MB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[      COMPLETE      ]|##################################################| 100%
===== testing package: ncurses-5.9-5 =====
running run_test.py
Traceback (most recent call last):
  File "/home/maxnoe/.local/anaconda3/conda-bld/test-tmp_dir/run_test.py", line 26, in <module>
    import curses
  File "/home/maxnoe/.local/anaconda3/envs/_test/lib/python3.4/curses/__init__.py", line 13, in <module>
    from _curses import *
ImportError: libncursesw.so.5: cannot open shared object file: No such file or directory
TESTS FAILED: ncurses-5.9-5

[itoijala]

Works for me.
Was a library built at all (under anaconda3/conda-bld/work/ncurses-5.9/lib/)?

[maxnoe]

The only files there are:

libform.a     libform.so.5.9  libmenu.so.5    libncurses++.a   libncurses.so.5.9  libpanel.so.5    libtinfo.so
libform.so    libmenu.a       libmenu.so.5.9  libncurses.so    libpanel.a         libpanel.so.5.9  libtinfo.so.5
libform.so.5  libmenu.so      libncurses.a    libncurses.so.5  libpanel.so        libtinfo.a       libtinfo.so.5.9

[itoijala]

Are you sure there was no error while building?
Then it would not even have tried the tests, I guess…

It could be that wide character support was not enabled.

For reference, my build.sh:

#!/bin/bash

mkdir $PREFIX/lib

sh ./configure --prefix=$PREFIX \
    --without-debug --without-ada --without-manpages \
    --with-shared --disable-overwrite --with-termlib

make -j$(getconf _NPROCESSORS_ONLN)
make install

[ahmadia]

@itoijala - thanks for all your work on this. Please ping me if there's anything I can do to help or you'd like me to review something.

[maxnoe]

Exactly the same.

Also for refernece:

             platform : linux-64
        conda version : 3.18.1
  conda-build version : 1.17.0
       python version : 3.4.3.final.0
     requests version : 2.7.0
     root environment : /home/maxnoe/.local/anaconda3  (writable)
  default environment : /home/maxnoe/.local/anaconda3
     envs directories : /home/maxnoe/.local/anaconda3/envs
        package cache : /home/maxnoe/.local/anaconda3/pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

[itoijala]

My conda-build version is older.
Strange, it was a fresh install from today.
I'll update it and see if it changes something.

platform : linux-64
        conda version : 3.18.1
  conda-build version : 1.14.1
       python version : 3.4.3.final.0
     requests version : 2.7.0
     root environment : /home/ismo/.local/anaconda3  (writable)
  default environment : /home/ismo/.local/anaconda3
     envs directories : /home/ismo/.local/anaconda3/envs
        package cache : /home/ismo/.local/anaconda3/pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False

[maxnoe]

I also did a fresh install and had the same version like you. Tthe build was successfull and during the test it told me to update. So I did but the test failed anyways.

[itoijala]

You could try adding --enable-widec to build.sh.

It also works for me with the new version of conda-build.

[maxnoe]

With --enable-widec it does not find the libtinfo because there is just a libtinfow

[itoijala]

Right, I get the same.

[maxnoe]

Would solving compiling both libraries work? With widec and without? Then there would be all libraries

[itoijala]

The Arch package includes both types of libraries without compiling twice.

[maxnoe]

From the ncurses docs:

--enable-widec
This switch causes wide-character libraries (e.g., libncursesw.so.6.0) to be built instead of normal ones (e.g., libncurses.so.6.0). These wide-character libraries are usable in both multibyte and traditional 8-bit locales, while normal libraries work properly only in 8-bit locales. Wide-character and normal libraries are source-compatible, but not binary-compatible.

Would that mean building the widec libraries and symlinking to the names without would work?

[vamega]

I use a term of xterm-termite or xterm-256color.
Things mostly work, but I did notice that Ctrl-L in IPython was not clearing the screen.

Would I need to build a custom ncurses package to fix this?

[itoijala]

@vamega: The below works for me. The important change is to add --with-terminfo-dirs=/usr/share/terminfo to the ncurses configuration.

1. Put https://raw.githubusercontent.com/conda/conda-recipes/master/ncurses/fix.patch and https://raw.githubusercontent.com/conda/conda-recipes/master/ncurses/run_test.py in a folder ncurses.
2. Put the four files from below there also.
3. conda build ncurses from outside the folder
4. conda install -f <path output by build>
5. Ctrl-L in ipython should work

build.sh:

#!/bin/bash

mkdir $PREFIX/lib

sh ./configure --prefix=$PREFIX \
    --without-debug --without-ada --without-manpages \
    --with-shared --disable-overwrite --enable-termcap \
    --with-termlib \
    --enable-widec --with-terminfo-dirs=/usr/share/terminfo    # NEW

make -j$(getconf _NPROCESSORS_ONLN)
make install

meta.yml:

package:
  name: ncurses
  version: 5.9

source:
  fn: ncurses-5.9.tar.gz
  url: http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.9.tar.gz
  md5: 8cb9c412e5f2d96bc6f459aa8c6282a1
  patches:
    # http://lists.gnu.org/archive/html/bug-ncurses/2011-04/txtkWQqiQvcZe.txt
    - fix.patch
    - ncurses-5.9-gcc-5.patch    # NEW

build:
  number: 5 # [linux64]
  number: 6 # [linux32]
  number: 1 # [osx]
  detect_binary_files_with_prefix: true

about:
  home: http://www.gnu.org/software/ncurses/
  license:  Free software (X11 License)

ncurses-5.9-gcc-5.patch:

--- ncurses/base/MKlib_gen.sh   2011-01-22 20:47:29.000000000 +0100
+++ ncurses/base/MKlib_gen.sh   2015-11-02 18:37:35.709245338 +0100
@@ -437,11 +437,22 @@
    -e 's/gen_$//' \
    -e 's/  / /g' >>$TMP

+cat >$ED1 <<EOF
+s/  / /g
+s/^ //
+s/ $//
+s/P_NCURSES_BOOL/NCURSES_BOOL/g
+EOF
+
+# A patch discussed here:
+#  https://gcc.gnu.org/ml/gcc-patches/2014-06/msg02185.html
+# introduces spurious #line markers.  Work around that by ignoring the system's
+# attempt to define "bool" and using our own symbol here.
+sed -e 's/bool/P_NCURSES_BOOL/g' $TMP > $ED2
+cat $ED2 >$TMP
+
 $preprocessor $TMP 2>/dev/null \
-| sed \
-   -e 's/  / /g' \
-   -e 's/^ //' \
-   -e 's/_Bool/NCURSES_BOOL/g' \
+| sed -f $ED1 \
 | $AWK -f $AW2 \
 | sed -f $ED3 \
 | sed \

run_test.sh:

export TERM=xterm-256color
clear

if [ `uname` == Linux ]; then
    ls $PREFIX/lib/libtinfow.so
fi

[joseandres42]

I know this may not be the correct way to do it, but this worked for me:

sudo ln /usr/lib/libncursesw.so.6 /usr/lib/libncursesw.so.5

[matrs]

I'm a new user of Anaconda (conda-build 1.18.2) and I just have this problem with ipython today (no autocompletion, readline warning. I don't have blessings ).
I'm in Arch (ncurses 6) and I also solved the problem doing a conda install ncurses (which installs ncurses 5.9)

[chendaniely]

confirmed that @matrs 's conda install ncurses fixes the no readline in IPython. I had the same issue.

[tebeka]

I can confirm as well that conda install ncurses fixes the problem for me in archlinux.

[Carreau]

This seem to break some jupyter things (jupyter/jupyter_console#63) as pointed by @tebeka above. We have a possible workaround but that we would prefer to include only if really necessary.

Thanks.

[maxnoe]

The above mentioned patch for gcc5 didn't work for me anymore, this one did:
https://raw.githubusercontent.com/cloudius-systems/osv/master/modules/ncurses/ncurses-5.9-gcc-5.patch

[maxnoe]

I closed this unintentionally, this is still an issue. I experienced the same problems again today after up a fresh copy of anaconda

[maxnoe]

I pushed the recipe that made it work for me here: https://github.com/maxnoe/conda-recipes

[msarahan]

Thanks @maxnoe - would you mind submitting your recipe to conda-forge? Submit a PR against the staged-recipes repo here: https://github.com/conda-forge/staged-recipes

That will help make your fixes more readily available.

[maxnoe]

I opened the PR

[joelostblom]

Update I solved my problems by symlinking anaconda's readline libraries to the system ones.

Hi,
I am getting the error sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook when trying to install R packages in conda, both via install.packages and by building a conda package with the cran skeleton (detailed explanations here and here, respectively). The answers I got in this thread pointed me towards the solutions discussed here. I also occasionally see this error when running the Jupyter-Notebook with the irkernel, but it does not prevent the notebook from running.

I have tried installing different version of ncurses, including the patch from @maxnoe, and I have tried to relink the libraries as suggested above (sudo ln /usr/lib/libncursesw.so.6 /usr/lib/libncursesw.so.5), but I am still running into the rl_signal_event_hook-error.

I'm quite lost at this point and any help to solve this would be greatly appreciated. Let me know if you think that this is unrelated to the solutions posted here, and I will open a new thread for this issue.

I am on Linux (Antergos, an Arch derivative) with kernel 4.4.5-1-ARCH.

[unode]

Can someone reopen this? I'm still seeing this problem.

Installed miniconda3, created a python=2 environment and installed ipython on it.
ipython complains about lack of readline support. Importing readline fails with ImportError: libncursesw.so.5: cannot open shared object file: No such file or directory

The problem is that the envs/py2/lib/python2.7/lib-dynload/readline.so file shipped with python2.7 links to libncursesw.so.5 and this library is missing from the distribution.

$ ldd envs/py2/lib/python2.7/lib-dynload/readline.so 
    linux-vdso.so.1 (0x00007ffc50df4000)
    libreadline.so.6 => /miniconda3/envs/py2/lib/python2.7/lib-dynload/../../libreadline.so.6 (0x00007f51c6a7a000)
--> libncursesw.so.5 => not found
    libpython2.7.so.1.0 => /miniconda3/envs/py2/lib/python2.7/lib-dynload/../../libpython2.7.so.1.0 (0x00007f51c6641000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f51c6424000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f51c6073000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f51c5e6f000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f51c5c6c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f51c5967000)
    /lib64/ld-linux-x86-64.so.2 (0x0000562836482000)

Installing ncurses does not fix the problem since it only includes:

$ ls envs/py2/lib/libn*
envs/py2/lib/libncurses.a
envs/py2/lib/libncurses++.a
envs/py2/lib/libncurses.so -> libncurses.so.5*
envs/py2/lib/libncurses.so.5 -> libncurses.so.5.9*
envs/py2/lib/libncurses.so.5.9*

which doesn't include libncursesw.so.

[maxnoe]

Try conda install -c conda-forge ncurses

[unode]

Both variants of libncurses are included in that ncurses package. Works fine now. Thanks!

[RSully]

I'm using the package from conda-forge and getting "setupterm: could not find terminfo database" still (see fish-shell/fish-shell#3121)

[maxnoe]

@RSully Are you using a system where the terminfo database is not in /usr/share/terminfo?

If there are other comman places for the terminfo database, we should add it to the configure call.

[RSully]

@maxnoe Mine are in that path

edit: well, that path exists, however in my case $TERM is xterm-256color and the following files with that name exist:

/usr/share/terminfo/s/screen.xterm-256color
/lib/terminfo/x/xterm-256color
/data/anaconda3/pkgs/ncurses-5.9-7/share/terminfo/x/xterm-256color
/data/anaconda3/pkgs/ncurses-5.9-4/share/terminfo/x/xterm-256color
/data/anaconda3/share/terminfo/x/xterm-256color

it looks like the conda-forge ncurses package is installing its own terminfo that includes the file, yet it isn't using it?