ic() prints extremely slowly when in a VS Code Jupyter Notebook

[pridkett]

This is probably a niche issue, but I saw that it was also discussed in #36 (see this comment by matthewdm0816 for a little bit of the details).

In short, when using ic() with the default configuration in a Jupyter Notebook and Visual Studio code, there is some sort of interaction that makes the output almost appear to be character by character. The following low framerate gif shows the problem:

[alexmojaki]

i'd guess this has to do with colorama...

[gruns]

@pridkett thank you for recording a video! that's extremely helpful

i'd guess this has to do with colorama...

indeed my gut says the same 🙂. let's find out!

@pridkett if you don't mind, can you please try this snippet in your jupyter notebook:

import sys
from icecream import ic

def stderrPrint(*args):
    print(*args, file=sys.stderr)

ic.configureOutput(outputFunction=stderrPrint)
for x in range(5):
    ic()

1. if ic's output with the above is still slow -> the problem isn't colorama and is related to stderr
2. if ic's output with the above is not slow -> yay! we now know the problem is colorama!

[alexmojaki]

could also try a script that prints things with colorama unrelated to icecream

[pridkett]

Using the custom stderrPrint it was zippy fast - see the 0.1s by the green checkmark:

But, then again, taking @alexmojaki's hint and writing some colorama directly, was also zippy fast:

Looks like I'll have to dig deeper here.

[gruns]

But, then again, taking @alexmojaki's hint and writing some colorama directly, was also zippy fast:

really interesting result. great testing there 🙌

im not familiar with colorama's guts. i dont know offhand why icecream's use of colorama, which in its entirety is:

@contextmanager
def supportTerminalColorsInWindows():
    # Filter and replace ANSI escape sequences on Windows with equivalent Win32
    # API calls. This code does nothing on non-Windows systems.
    colorama.init()
    yield
    colorama.deinit()

produces chunky output in jupyter notebook

Looks like I'll have to dig deeper here.

if you have the time, please do and let us know what you find! thank you 🙌

@alexmojaki depending on what @pridkett finds, this may be (yet another) nail in the colorama's coffin for us

[alexmojaki]

I think the colorama.init() is crucial to the test

[pridkett]

I've spent some time digging at this. What's happening in Jupyter notebooks is that colorama is incorrectly saying that the ANSI codes need to be stripped out because it's Jupyter web and Jupyter VS Code are not TTYs. In fact, you don't need to strip out color codes inside of Jupyter Web and VS Code, so this should have a workaround similar to how PyCharm currently has a workaround in colorama.

The net impact of this is what colorama creates a wrapped version of sys.stdout and sys.stderr that calls write_and_covert on every output. I haven't dug down deep enough to figure out which part of this code is dog slow in VS Code, but it's slow and it doesn't need to be called.

This also explains why in my gif above it looks like all the lines from ic() were output in a very short time (look at the timestamps), but it took longer to display. The writes have been sent to the wrapped version of sys.stdout which is just slowly operating.

The "fix" is to make it so colorama doesn't think it needs to strip out the colors when running in Jupyter notebooks, which can handle coloring just fine. Not a bug in icecream, a bug in colorama.

[gruns]

@pridkett awesome digging! thank you! 🙌

The "fix" is to make it so colorama doesn't think it needs to strip out the colors when running in Jupyter notebooks, which can handle coloring just fine. Not a bug in icecream, a bug in colorama.

a few options:

1. provide a more convenient way to disable coloring than icecream.configureOutput(outputFunction=)

2. replace colorama with another cross platform coloring library that doesnt suffer this deficiency

3. have a list of environments where colorama doesnt work, jupyter among them, and disable ic's coloring in these environments

4. fix the colorama bug ourselves and submit the PR. or file a bug with colorama and hope they fix this on their end expediently

5. submit a PR to colorama to add jupyter notebooks as an exception, like pycharm currently is. eg https://github.com/tartley/colorama/blob/master/colorama/ansitowin32.py#L43-L53

#2 is best overall but the most effort. #1 is a nice to have and medium effort. #3 solves the immediately problem in a hacky way but degrades ic's experience

@pridkett is there an environment variable present in jupyter notebooks, akin to the PYCHARM_HOSTED var of https://github.com/tartley/colorama/blob/master/colorama/ansitowin32.py#L45-L47?

if so, we can submit a quick PR to add jupyter notebook as an exception to colorama

[peedrr]

+1 this. I love icecream but currently unusable in notebooks.

@gruns - I ran os.environ from within a notebook and there seems to be a relevant env var: JPY_PARENT_PID

A quick google search confirmed that it's set by the notebook and I also confirmed that it disappeared when the notebook wasn't running.

[gruns]

@peedrr great digging! 🙌

can you check out a local version of icecream and a local version of colorama therein, and then add

        elif 'JPY_PARENT_PID' in os.environ:
            return False

to isatty() in https://github.com/tartley/colorama/blob/master/colorama/ansitowin32.py#L43-L53

    def isatty(self):
        stream = self.__wrapped
        if 'PYCHARM_HOSTED' in os.environ:
            if stream is not None and (stream is sys.__stdout__ or stream is sys.__stderr__):
                return True
        elif 'JPY_PARENT_PID' in os.environ:
            return False
        try:
            stream_isatty = stream.isatty
        except AttributeError:
            return False
        else:
            return stream_isatty()

and see if that correctly disables coloring in jupyter notebooks and thus resolves icecream's slow output?

if so, ill submit that PR as a quick 'fix' in the vein of #5 above

5. submit a PR to colorama to add jupyter notebooks as an exception, like pycharm currently is. eg https://github.com/tartley/colorama/blob/master/colorama/ansitowin32.py#L43-L53

if its not merged quickly (looking at how active colorama is, i doubt it will be), ill add a workaround to icecream itself

[peedrr]

@gruns - I just wanted to ack your message. I'm a bit snowed under but I'll get around to it. Thanks for the step-by-step and for your responsiveness. I'll get back to you

[gruns]

@peedrr appreciate the heads up! ❤️

[raldone01]

You can use the following code to work around the but in colorama.

if "JPY_PARENT_PID" in os.environ:
  import colorama.ansitowin32 as ansito_win32
  # https://github.com/tartley/colorama/blob/136808718af8b9583cb2eed1756ed6972eda4975/colorama/ansitowin32.py#L49
  # monkey patching colorama to work with jupyter
  # replace StreamWrapper.isatty with a lambda that returns True
  ansito_win32.StreamWrapper.isatty = lambda self: True