Readline and bracketed-paste mode

[chronitis]

Recent versions of libreadline enable bracketed-paste mode by default, which causes issues for pexpect when wrapping a readline-enabled command (like bash or python). This results in extra escape sequences in the output of eg, replwrap.run_command():

>>> from pexpect import replwrap
>>> bash = replwrap.bash()
>>> bash.run_command("echo hello")
'\x1b[?2004l\rhello\r\n\x1b[?2004h'

This mode was enabled by default in bash 5.1/readline 8.1, so is only an issue for recently compiled binaries at the moment, but presumably will become more common. This causes the replwrap tests to fail on debian unstable.

I'm not quite sure the best solution here. I tried playing with the TERM or INPUTRC environment variables (see https://www.gnu.org/software/bash/manual/html_node/Readline-Init-File.html) but I didn't manage to find a combination which suppressed this behavior. Filtering them explicitly in pexpect might also work, but I haven't considered possible side effects.

Seen in pexpect 4.8.0, python 3.9.1, bash 5.1.4, but I'd expect this to apply to other versions of python and pexpect if the bash/readline version is sufficiently new.

[hroncok]

This seem to work as a workaround:

echo "set enable-bracketed-paste off" > .inputrc
export INPUTRC=$PWD/.inputrc

[hroncok]

I've drafted #671

[mgorny]

Thanks for the workaround! It just helped me fix terminado's tests in Gentoo.

[mgorny]

Hmm, though I guess it's not a very valuable fix given that the package will probably fail at runtime :-(.

[tgbugs]

Here is another workaround that can be applied per session. I think it might be a reasonable solution, at least for the pxssh case, not sure about the local case. The essence is to call bind 'set enable-bracketed-paste off' before doing anything else.

import os
import pathlib
from pexpect import pxssh

host = 'localhost'
session = pxssh.pxssh(options=dict(IdentityAgent=os.environ.get('SSH_AUTH_SOCK')))
session.login(host, ssh_config=LocalPath('~/.ssh/config').expanduser().as_posix())
session.sendline("bind 'set enable-bracketed-paste off'")
session.prompt()

[Red-M]

I'm against filtering output explicitly because while this may fix an issue for you, it may produce an issue for someone else and then an option to enable/disable this filtering has to be added.

As far as I'm understanding a set enable-bracketed-paste off in your environment before you start what you need to do should be enough for the terminal to turn the new feature off, because of this I'm not exactly seeing an issue with the library itself which should be trying to get the "rawest" output available.

@chronitis
If this is breaking something in the testing suite for Debian sid, please give me instructions on how to replicate the issue and I'll make a patch to correct it (or merge the PR in this issue)

(Apologies for the late response, I generally prefer to be a little more hands off with these kinds of issues where it could be configured in the environment to fix the issue and I've been a little busy lately)

[felixonmars]

@chronitis If this is breaking something in the testing suite for Debian sid, please give me instructions on how to replicate the issue and I'll make a patch to correct it (or merge the PR in this issue)

Hi, we are hitting kislyuk/argcomplete#337 when packaging for Arch Linux. The issue links back here and expected pexpect to be fixed instead. If this is not going to happen, would you kindly let me know how we should move forward? (e.g. Is the above workaround expected to be inserted into individual projects using pexpect, like argcomplete in this example?)

[LewisGaul]

I also hit this issue recently on CentOS 9. What's not clear to me is why sending commands through pexpect results in bash seeing that command as being 'pasted' (hence the start and end char sequences for bracketed paste mode) - can anyone explain this? I would have expected sending data through pexpect to be more like typing at the keyboard than pasting into a terminal, and I'm wondering if there's something that could be changed along these lines in pexpect.

[Red-M]

BASH isn't seeing input text as pasted, the extra characters are line end and starts for when user input can be entered.
I'd also like to make it clear that this is a default on feature from BASH and not a bug caused by code in pexpect.

Pexpect also needs to work on platforms that aren't Linux with terminals that aren't BASH so placing patches into pexpect to "fix" this feature from BASH might break someone else's code or someone's code who wants the characters there to use as expect-able characters.
So I don't really understand why BASH decided to introduce a feature as default and make it a pain to turn off instead of making it opt-in and allowing some time to see if people wanted it to make it a default on (aka opt-out).

[LewisGaul]

BASH isn't seeing input text as pasted, the extra characters are line end and starts for when user input can be entered.

Ahh that seems obvious now you say it :) For some reason I thought they were special characters that got included only when you actually pasted something.

I'd also like to make it clear that this is a default on feature from BASH and not a bug caused by code in pexpect.

Yep, that's clear now, and agreed that it doesn't make sense for pexpect to strip stuff off by default.

That being said, it seems like it would make sense for specifically pexpect.replwrap.bash() to handle disabling bracketed-paste mode (as per the original post), but this doesn't affect me personally as I'm not currently using it :)

[Red-M]

I'd rather have pexpect be unopinionated about your environment so it can be used however you wish but when environments change (eg, BASH adds a new feature which is opt-out and adds extra terminal control characters) there isn't really anything pexpect should do except pass that straight on to you. There is a bundle of treadmill work in adding patches for certain environments to bring everything to a single text based and uniform interface. A great example of this is the work in RANCiD or Oxidised where they need to get config backups for networking devices and have to massage the terminal into submission to make obtaining those backups easy.

On the flip side this issue can be completely bypassed when you do prompt detection by making the terminal feed you back the "right" prompt or write your own regexes to match the prompt when braketed paste characters are there or not.