conpty uses default history buffer length

[devlinb]

Description of the new feature/enhancement

Right now cmd.exe instances opened by Terminal do not seem to obey the command buffer size set on cmd.exe, nor does it offer a way to set this within Terminal. Currently cmd.exe instances opened within Terminal seems hard coded to the default of 50.

[zooba]

FYI this also affects the buffer count (for child processes that attach to the console) - see https://discuss.python.org/t/interactive-command-history-in-session-started-with-subprocess-on-windows/3701 for more. Though the original complaint in that post is about conhost.exe, it also applies to Terminal, but does not have the workaround.

[jackerghan]

Anyway to work around this? Commands churn of the short 50 item history and I end up having to dig them up again and again...

[DHowett]

As a temporary (per-session) workaround, you can use DOSKEY /LISTSIZE=900 or whatever suits your fancy.

As a quasi-permanent workaround, if you set the commandline of your Command Prompt profile to cmd /s /k doskey /listsize=900 it will always start up and expand the history list to 900 elements.

As an absolutely permanent workaround, if you use a more civilized shell you will never again be constained by the limitations of CMD

[eryksun]

As a temporary (per-session) workaround, you can use DOSKEY /LISTSIZE=900 or whatever suits your fancy.

Note that doskey.exe cannot extend the number of history buffers from the default value of 4, which is far too small. In many cases, 8 or more process end up attached to the console session. For system console sessions (conhost.exe), I change the command-history settings in the "Defaults" dialog (the default settings in "HKCU\Console") to use 32 history buffers. But for some reason, console sessions (tabs) in Windows Terminal ignore the default settings in the registry. So changing it requires calling SetConsoleHistoryInfo in each session.

As an absolutely permanent workaround, if you use a more civilized shell you will never again be constained
by the limitations of CMD

Surely you know that high-level reads via ReadConsoleW and ReadFile use the console's internal command-line editor, which provides basic support for line editing, aliases, and history. This has nothing at all to do with "cmd.exe", except for the fact it's a simple console application that calls ReadConsoleW, as are countless other console applications, including "python.exe" if pyreadline isn't installed. PowerShell uses low-level reads and PSReadLine for its own interface, but that has nothing to do with console applications that run from PowerShell and inherit its console session.

[DHowett]

Surely you know

I certainly do, and don’t call me Shirley

That joke doesn’t work so well in text form.

I staunchly believe that giving the console host ownership of the command history was a mistake the consequences of which we must forever endure. 😄

I do think this is worth fixing, but perhaps more holistically. Do you think that it would present a correctness or compatibility issue to simply offer infinite history with infinite buffers? Barring infinite buffers being compatible, we could likely collect data on buffers or entries aging out and adjust our defaults accordingly.

[eryksun]

Do you think that it would present a correctness or compatibility issue to simply offer infinite history with infinite buffers?

Allowing infinite history buffers with infinite history should be fine on the client side. GetConsoleHistoryInfo and SetConsoleHistoryInfo would work the same except the console host would interpret 0xFFFFFFFF as an infinite value for HistoryBufferSize and NumberOfHistoryBuffers. (Currently these two values are limited to 0x7FFF, i.e. 32767.)

The Defaults/Properties dialog could be updated with two check boxes that enable the infinite values. If selected, which should be the default setting (also for a conpty session), the corresponding text field in the UI should be disabled.

The registry settings "NumberOfHistoryBuffers" and "HistoryBufferSize" are REG_DWORD values, so they can store 0xFFFFFFFF. The corresponding fields in a shell link are the same size (see [MS-SHLLINK] 2.5.1 ConsoleDataBlock).

I staunchly believe that giving the console host ownership of the command history was a mistake the consequences of which we must forever endure. smile

NT's console had to provide functionality equivalent to DOSKEY (MS-DOS 5.0 TSR service, 1991) for 16-bit DOS programs such as COMMAND.COM. I guess this could have been left as a private interface for NTVDM. But the designer(s) decided to provide it for Windows console applications as well, as a consistent user interface for simple CLI programs. It probably made sense back then to implement the CLI editor centrally in the subsystem server (csrss.exe), rather than on the client side in kernel32.dll -- for the sake of efficiency given the resource constraints of PCs in the early 1990s.

[JJC1138]

I'm bumping into this issue when using pipenv shell in Windows Terminal and I thought I would post the workaround I'm using for anyone who could use one. I put the following in my PowerShell profile, which is based on @eryksun's Python code from the thread linked above by @zooba:

Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;

[StructLayout(LayoutKind.Sequential)]
public struct CONSOLE_HISTORY_INFO
{
    public UInt32 cbSize;
    public UInt32 HistoryBufferSize;
    public UInt32 NumberOfHistoryBuffers;
    public UInt32 dwFlags;
    
    [DllImport("kernel32.dll", SetLastError = true)] static extern bool SetConsoleHistoryInfo(ref CONSOLE_HISTORY_INFO ConsoleHistoryInfo);
    
    public static void Set(UInt32 HistoryBufferSize, UInt32 NumberOfHistoryBuffers, bool IgnoreDuplicates)
    {
        CONSOLE_HISTORY_INFO info;
        info.cbSize = (UInt32)Marshal.SizeOf<CONSOLE_HISTORY_INFO>();
        info.HistoryBufferSize = HistoryBufferSize;
        info.NumberOfHistoryBuffers = NumberOfHistoryBuffers;
        info.dwFlags = IgnoreDuplicates ? 1U : 0U;
        
        if (!SetConsoleHistoryInfo(ref info))
        {
            Console.WriteLine("Error setting console history info: {0}", Marshal.GetLastWin32Error());
        }
    }
}
"@

# Increase the number of command history buffers so that one will be available even for deeply-nested child console processes like the ones that you get from `pipenv shell`.
[CONSOLE_HISTORY_INFO]::Set(512, 32, $FALSE);

[jrobbins-LiveData]

Python poetry has a command (poetry shell) which essentially activates a Python "virtual environment" by running a special, e.g. .bat script in a new subprocess. The current Windows Terminal settings for the number of command history buffers is too small, and ends up meaning that the user is left in a cmd shell with no up/down arrow support.

If Windows Terminal would minimally increase the current limit of 4 to a much larger number, the Python poetry user experience would be so much better.

Or, give the Windows Terminal user a way of configuring this setting, please.

Here's a link to the effort underway right now to improve the cmd.exe ergonomics in the Poetry project: python-poetry/poetry#5053

[alexolog]

As a temporary (per-session) workaround, you can use DOSKEY /LISTSIZE=900 or whatever suits your fancy.

Doesn't work for me:
See https://pastebin.com/HTmueRe1

[zadjii-msft]

From #13515, re "Discard Old Duplicates"

When run in ConPTY (read: the terminal), the console just ignores most settings. That might have been a mis-play. We may want to keep around the history settings, but just ignore the "appearance" ones.

I wonder how much of a breaking change it would be to have ConPTY load the HKCU/Console "defaults" for HistoryBufferSize, HistoryNoDup, NumberOfHistoryBuffers. The rest I'd be wary of.

[o-sdn-o]

It is related to #16198.

This explains why it works for cmd ("cmd" string is 6 bytes in UTF-16), but does not work for cmd.exe ("cmd.exe" string is 14 bytes in UTF-16).

[pfmoore]

I've just come back to this issue because I have the following in my Powershell profile (which, if I recall, I found in a post from @eryksun somewhere):

# workaround for https://github.com/microsoft/terminal/issues/2558
py -c "import ctypes; ctypes.windll.kernel32.SetConsoleHistoryInfo(b'\x10\x00\x00\x00\x00\x02\x00\x00 \x00\x00\x00\x01\x00\x00\x00')"

Is this still needed? I'm on Windows 11, and #16198 says it's not an issue on Windows 11, but I don't know if that's relevant.

More significantly, I've been experimenting with nushell, and I'm not sure if I need to copy this snippet into my nu profile, for situations where I start Windows Terminal without ever running Powershell. And similarly, if I run (for example) Python in VS Code, where neither Windows Terminal nor Powershell are invoked, do I still need this workaround?

Essentially, if the workaround is still needed, having a global place where it can be run and will then be active for all terminal sessions would be far better than having to find a way to call SetConsoleHistoryInfo in every command line shell/REPL that might be invoked directly from the GUI.

[eryksun]

# workaround for https://github.com/microsoft/terminal/issues/2558
py -c "import ctypes; ctypes.windll.kernel32.SetConsoleHistoryInfo(b'\x10\x00\x00\x00\x00\x02\x00\x00 \x00\x00\x00\x01\x00\x00\x00')"

I need to call SetConsoleHistoryInfo() in Windows 11 23H2 with Terminal 1.19.10821. ConPTY sessions ignore the related registry settings for the current user -- "HistoryBufferSize", "NumberOfHistoryBuffers", and "HistoryNoDup". Unfortunately, doskey.exe with /exename and /listsize (i.e. undocumented SetConsoleNumberOfCommandsW) isn't sufficient. I need more than 4 history buffers.

Is this still needed? I'm on Windows 11, and #16198 says it's not an issue on Windows 11, but I don't know if that's relevant.

That was about a bug in the implementation of SetConsoleNumberOfCommandsW().

Essentially, if the workaround is still needed, having a global place where it can be run and will then be active for all terminal sessions would be far better

I don't think something like that has ever been implemented on Unix systems. It always depends on a shell executing something like "~/.profile" or "~/.bashrc".

This command-history problem would be better addressed in the long term by releasing a system Readline library. Applications would use the system library to store their own command history across console sessions instead of relying on a history buffer in the console host.