Complete Jupyter notebook integration

[dhruvmanila]

Meta issue to keep track of the remaining tasks before shipping Jupyter notebook native integration:

• Add --diff support for Jupyter notebooks #4727
• Add --add-noqa support for Jupyter notebooks #5189
• Add support for viewing Jupyter Notebooks in the playground
• Handle trailing newline as per black for Jupyter notebooks #5190

Formatter Support

• Add support for IpyEscapeCommand token in SimpleTokenizer (if required)
• Add support for formatting StmtIpyEscapeCommand
• Add support for formatting ExprIpyEscapeCommand
• Enable formatting for Jupyter notebooks #7749

Editor Integrations

• Support for Jupyter Notebooks ruff-vscode#256 - I think this will probably involve changes in ruff-lsp so that other editors can benefit as well.

IPython Syntax Handling (%, !, etc.)

• Add ability to handle magic commands in Jupyter notebook #5030

Ruff

• Use Jupyter mode while parsing Notebook source code #6090
• Add required changes for handling the new Jupyter AST nodes #6087

Proposed new rules

• Empty magic command (%) (Autofix: remove the statement)
• Magic command contains whitespace between the prefix and command value (% matplotlib inline) (Autofix: remove all the whitespaces) (Only valid for % magic prefix)

Existing rule changes

• Update F481 autofix for Jupyter line magic expr #6116
• F811: Message unclear in Notebook context #6672

I'll be looking at other related rules and update this issue accordingly

Parser

• Add support for help end IPython escape commands in the parser #6359
• Add support for priority handling for magic commands #6357
• Lex Jupyter line magic with Mode::Jupyter RustPython-Parser#23
• Lex Jupyter Magic in assignment value position RustPython-Parser#30
• Add line magic stmt and expr AST nodes RustPython-Parser#31

Autofix for Jupyter Notebook

• Support formatting jupyter notebooks #1218

Implementation

• Maintain consistency when deserializing to JSON #5114
• Ability to perform integration test on Jupyter notebooks #5076
• Add roundtrip support for Jupyter notebook #5028
• Add support for auto-fix in Jupyter notebooks #4665

[charliermarsh]

My main question would be: at what point do you think we can remove the feature flag, and open this up to users? (Maybe we're already past that point!)

[dhruvmanila]

I do think we're ready except that without the (1) task above Ruff might give false positives. Without that we would ignore any cell which contains magic commands. Now, we would also ignore if it contains valid Python code. It could be that a variable is defined in that cell and is being referenced in another cell. Here, we might trigger undefined variable. Similar example can be given for imports.

I see this a lot for %matplotlib inline command: https://sourcegraph.com/search?q=context:global+lang:%22Jupyter+Notebook%22+%25matplotlib+inline&patternType=standard&sm=1&groupBy=repo

[dhruvmanila]

What do you think about releasing this under experimental? I'm not sure what would that look like but maybe we don't add the *.ipynb to include and ask users to explicitly opt in to that.

[charliermarsh]

Yeah, I think it would be reasonable to release it now (even without magic handling) and omit *.ipynb in include, so users need to opt-in explicitly. We can market it as experimental to get feedback. We can include it in the next release IMO.

We should also add the magic command support, but doesn't have to block if it's opt-in :)

[charliermarsh]

(To be explicit: we could just remove the flag now.)

[jpmckinney]

I'm not sure if this deserves its own issue, but: Would it be possible to use extend-include = ["*.ipynb"] and ruff format to only re-format the Python code in cells, and not the ipynb file itself?

My users typically edit notebooks in Google Colab. Google Colab uses two-space indentation in the ipynb file, instead of the typical one-space indentation. ruff format currently re-indents with one space. As such, edit history becomes difficult/impossible to track, because effectively every line gets rewritten between the user saving from Google Colab (changing to 2 spaces) and CI running ruff format (changing to 1 space).

Google Colab also puts keys like metadata, nbformat, nbformat_minor at the top of the file, but ruff format moves these to the bottom. ruff format also changes the order of execution_count, outputs, etc.

I would love to be able to opt out of ipynb formatting, so that only the Python code inside cells is reformatted.

[dhruvmanila]

(I think it deserves its own issue, we can discuss it there :))

[dhruvmanila]

Closing this as completed. The only thing remaining is to support --add-noqa.