Use JupyterLite and Pyodide for the online shell

[jtpio]

This is an alternative to sympy/sympy-live#198

Fixes sympy/sympy-live#83
Closes sympy/sympy-live#198

Copying the PR description from sympy/sympy-live#198 to here for reference:

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels including the WebAssembly powered Pyodide kernel.

The embeddable code console is currently deployed as a static website on Vercel, and the source code is currently living in the following repository: https://github.com/jtpio/replite

It supports rich rendering of outputs just like in Jupyter Notebook and JupyterLab, for example:

TODO

• Add a new shell template to the repo
• Embed the JupyterLite REPL app in an iframe: https://jupyterlite.readthedocs.io/en/latest/applications/repl.html
• Execute starter code by default:

>>> from __future__ import division
>>> from sympy import *
>>> x, y, z, t = symbols('x y z t')
>>> k, m, n = symbols('k m n', integer=True)
>>> f, g, h = symbols('f g h', cls=Function)

This also means the https://github.com/sympy/sympy-live repo could then be archived after this PR is merged, and the Google App Engine deployment be deleted.

[jtpio]

For reference https://numpy.org is also now using JupyterLite to let folks try numpy in the browser: numpy/numpy.org#547

[jtpio]

A follow-up to this would be to make a custom JupyterLite in a repo under the sympy org on GitHub, for example https://github.com/sympy/live (or repurpose https://github.com/sympy/sympy-live).

The custom deployment can easily be made from this GitHub template: https://github.com/jupyterlite/demo

This means the URL of the IFrame could then be updated to https://sympy.org/live/repl/

[asmeurer]

Hey, this looks like a good start. I have a few comments.

• Is there some way we can have a "loading" text or something like that when it is loading. When I go to the page it is just blank for a few seconds. Presumably it would be blank for even longer if I had a slower connection.
• We're going to be missing a lot of the features from https://live.sympy.org/ (see the sidebar). I don't think we really need to care about a lot of them, like the login or "recent statements".
• Having a new "about this page" thing on the side would be useful too, to explain that this runs entirely in the browser.
• Regarding the settings:
  • we don't need the "output format" setting. I doubt anyone will want something other than LaTeX, but even if they do, they can just use init_printing(), so we could just put a link to the docs for it.
  • Submit with Enter vs. Shift Enter would be nice to configure. It seems to me that most people will want to use Enter. I'm also curious how this works on mobile.
  • Privacy should be irrelevant (right?). Nothing will be stored anymore.
  • I'm not sure why we had a tab completion setting. Is there a reason that control-space would be easier for some people to press than tab? I don't think we need it.
  • Same with the history. It just works like it does in Jupyter, which is to use the arrow keys.
• This may be a comment just for jupyterlite in general, but the "run" button being at the top is very confusing. I would expect the run button to be to the right of the entry box.
• The LaTeX in the example session box is not loading.
• It would be nice if you could click the example session and have it run, same as with SymPy Live.
• The example session should show the numbers instead of >>> as long as that is what the session shows.

I also still need to figure out how to test this on mobile. The existing site works very well on mobile, so it would be a shame to give that up if jupyterlite isn't mobile friendly.

[asmeurer]

By the way, are persistent sessions possible with jupyterlite? If the answer is "no" that's fine, but I'm curious if it's possible.

[jtpio]

Thanks for the feedback 👍

Is there some way we can have a "loading" text or something like that when it is loading. When I go to the page it is just blank for a few seconds. Presumably it would be blank for even longer if I had a slower connection.

Maybe there could be a splash screen in the app itself (would need to be added).

Having a new "about this page" thing on the side would be useful too, to explain that this runs entirely in the browser.

Sounds good.

we don't need the "output format" setting. I doubt anyone will want something other than LaTeX, but even if they do, they can just use init_printing(), so we could just put a link to the docs for it.

This could even be pre-executed by default.

Submit with Enter vs. Shift Enter would be nice to configure. It seems to me that most people will want to use Enter. I'm also curious how this works on mobile.

On mobile Enter would indeed be better. Otherwise there is always the Run button in the toolbar.

I'm not sure why we had a tab completion setting. Is there a reason that control-space would be easier for some people to press than tab? I don't think we need it.

I think Tab should work ok. This is what it looks like locally:

This may be a comment just for jupyterlite in general, but the "run" button being at the top is very confusing. I would expect the run button to be to the right of the entry box.

Agree. This is because the repl reusing the JupyterLab code console as is. jupyterlite/jupyterlite#148 has some mockups and discussions on what a single cell app would look like.

The LaTeX in the example session box is not loading.

This might require the call to init_printing() that could be executed automatically as mentioned above:

On mobile the repl takes the whole width, but the page is still scrollable so I would expect it to be usable:

[jtpio]

By the way, are persistent sessions possible with jupyterlite? If the answer is "no" that's fine, but I'm curious if it's possible.

If "presistent" means being able to reuse the same execution context (kernel), then not yet. But it might be possible via a service worker at some point (wip started in jupyterlite/jupyterlite#137).

[asmeurer]

It sounds like a lot of these issues will be fixed automatically over time as the jupyterlite console improves. So we don't need to block this on anything of that nature (including the splash screen assuming that's something that would load instantly; otherwise we should add something static to this page).

This might require the call to init_printing() that could be executed automatically as mentioned above:

Adding an explicit call to init_printing() is a good idea. SymPy expressions automatically print as LaTeX in Jupyter, but init_printing() makes it so that things like lists also print as LaTeX when they contain SymPy expressions (e.g., compare the output of solve(x**2 - 1) before and after calling init_printing()).

The examples thing presumably is just happening because MathJax isn't being loaded on the page.

I think Tab should work ok. This is what it looks like locally:

Yes, this was just me opining on the existing options at live.sympy.org. I think the only option that needs to be present is Enter vs. Shift-Enter, and that's probably another thing that should be part of jupyterlite itself rather than on the SymPy site.

And once again, I would strongly encourage you to make Enter the default (everywhere, not just on mobile). I don't know how hard that is to change, but if it can't be fixed before this is merged we will need to add some text on the page explaining to use Shift-Enter to execute.

On mobile the repl takes the whole width, but the page is still scrollable so I would expect it to be usable:

I want to test this on an actual phone too. I guess we can push this up somewhere to test it.

[certik]

@asmeurer ok, it seems this PR is the way to go. As you commented at sympy/sympy-live#198 (comment), it seems that what needs to be done is:

• we just need to clean up the text on the main page

Can you be more specific? Let's get this done soon. If it's simple enough, I can help.

[asmeurer]

Yes, see my comments above.

I'd also like to have a demo of this PR published somewhere so that I can actually test it out, including on mobile.

[jtpio]

I'd also like to have a demo of this PR published somewhere so that I can actually test it out, including on mobile.

Maybe there could be a new GitHub Action that builds the website, and uploads it as an artifact? So it could be downloaded and run locally with python -m http.server.

Probably could be done in a separate PR, since it also benefits other changes made to the website.

[ivanistheone]

@asmeurer @certik

I'd also like to have a demo of this PR published somewhere so that I can actually test it out, including on mobile.

Here is a slightly modified deployed via gh-pages branch on my fork (for testing functionality):

https://ivanistheone.github.io/sympy.github.com/en/shell.html

Initial testing comments:

• shell works well (loads fast)
• sidebar can be changed/disabled
  
• mobile viewport makes the main menu slide down too much and hides the shell
  

@jtpio Good idea about adding gh actions in general to the repo (separete from this PR).
Here is my first draft at this deploy action and some edits to templates needed to make this work.

---

I'd like work on integration and testing of this PR and the updates in sympy-live repo/website. I'll have some time in coming weeks, and put together a google doc that combines all the links from various PRs and issues + checklist for next steps:
https://docs.google.com/document/d/1UOhrxZN789tKqE19_1TH0FiQK7PzisKioOa0LFeN0_k/edit

[asmeurer]

@ivanistheone that all sounds good. Maybe you can start a new PR based off this one. We definitely need to fix the header thing on mobile. I've noticed that issue before. I don't remember it being a problem in the past, so maybe something broke it at some point.

[SylvainCorlay]

This looks amazing!

[jtpio]

Thanks all!

Let me know if there is something else to be done here.

Looks like most of the points raised above could be done in separate follow-up PRs as they are not strictly related to this change?

we just need to clean up the text on the main page

@asmeurer @certik I'll let you handle this one then.

In other words happy to merge and iterate, so we can also get feedback from end users too!

[certik]

@asmeurer why don't we merge an iterate on that?

I tested the link above in Firefox on Apple M1 max, the first time it takes about 10-20s until sympy fully loads, but the execution is immediate. For the sphinx docs, possibly we can start loading the shell immediately in the background, so by the time a user clicks some prompt to evaluate, it will be immediate.

[asmeurer]

If we merge this, it would serve as a "soft launch". The "online shell" button at the top of https://www.sympy.org/en/index.html would go to the jupyterlite shell, but live.sympy.org would still point to the GAE site.

Just looking at it, I have two issues with the demo site which I think are pretty important:

• The header on mobile blocks the console.
• The submit button in the console is Shift-Enter instead of Enter.
• There should be some sort of indicator that it is loading SymPy, since it takes a long time, even on a fast computer with fast internet.
• Better mobile support in general. Things like arrow keys, tab, and shift-enter are impossible to type on mobile.

The last three are really issues with the upstream jupyterlite. Presumably they can be fixed upstream without us having to modify the site here. Even so, they are pretty big UX problems in my view, so I'd like to at least make sure they are being tracked upstream.

And some less important improvements:

• We should add init_printing() to the initial list of executed commands
• The example session should match the console (use Jupyter inputs, MathJax).
• More text on the page describing what it is.

Some of these seem quite straightforward to fix so we should just do them now.

[asmeurer]

Actually, it looks like the header on mobile has this problem on every page on sympy.org, not just this one.

[ivanistheone]

 Actually, it looks like the header on mobile has this problem on every page on sympy.org, not just this one.

Yes I saw that. This afternoon I’ll be looking into this and if I manage to fix I’ll open a PR.

[ivanistheone]

@asmeurer Please merge sympy/sympy-web-static#13 for the mobile menu fixes and #172 for a html template restrucuring prerequisite for this PR.

Previewable here: https://ivanistheone.github.io/sympy.github.com/en/index.html (this link contains code from this branch, which contains both of the above PRs.

Testing notes:

1. mobile menu works when resizing window
2. all pages are identical to current live version, which shows the s/sidebar/sidebarextra/ change is a noop)

---

@jtpio When #172 gets merged, please rebase this PR so that we'll be able to take advantage of the overwritable sidebar block. After you rebase, I'll submit anther "patch" to your PR with init_printing() addition to initi code and updated copy of side menu text (info and examples) so we can merge this PR (PHASE 1 SHIPPED).

---

After this, I'll follow up with separate work (PHASE 3) on new sympy/live repo that will power https://live.sympy.org (a clone of https://github.com/jupyterlite/demo with possible customizations such as ENTER vs SHIFT+ENTER handlers etc.). @jtpio I'll appreciate your help/advice with this step in coming weeks if you're avail.

---

If we merge this, it would serve as a "soft launch". The "online shell" button at the top of https://www.sympy.org/en/index.html would go to the jupyterlite shell, but live.sympy.org would still point to the GAE site.

Exactly! It makes sense to have https://sympy.org/en/shell.html run in parallel (soft launch) to the future https://live.sympy.org/index.html for a little while so that we'll have some time to do testing (PHASE 2) and research before "launching" to https://live.sympy.org/

[certik]

Thank you @ivanistheone and @jtpio for all your hard work!

[asmeurer]

After this, I'll follow up with separate work (PHASE 3) on new sympy/live repo that will power live.sympy.org (a clone of jupyterlite/demo with possible customizations such as ENTER vs SHIFT+ENTER handlers etc.). @jtpio I'll appreciate your help/advice with this step in coming weeks if you're avail.

I'd rather just upstream the fixes to jupyterlite. I don't want to have a separate repo to maintain. The suggestions I made should make sense for jupyterlite in general, I would think. Any SymPy specific customizations ought to be done in this repo.

[asmeurer]

@jtpio When #172 gets merged, please rebase this PR so that we'll be able to take advantage of the overwritable sidebar block. After you rebase, I'll submit anther "patch" to your PR with init_printing() addition to initi code and updated copy of side menu text (info and examples) so we can merge this PR (PHASE 1 SHIPPED).

Maybe you should just start a new PR with a branch based off of this one, so that you don't have to wait for @jtpio.

[ivanistheone]

@asmeurer Thx for merging the two prereq. PRs. I don't see the new main.js on the website though https://www.sympy.org/en/index.html Is there some additional step required to deploy to sympy.org?

I'm guessing the static submodule needs to be updated to point to the latest commit?

For the life of me I don't understand how git submodules work (and I've tried many times!), so if someone could do what's needed (bonus points for explain what needs to be done too).

[asmeurer]

I'm not great with submodules either, but I think the process goes like this:

If you haven't already cloned the submodule you can use

git submodule init
git submodule update

To update it, just

cd static
git checkout master
git pull

and then

cd ..
git add static
git commit

[asmeurer]

By the way, sympy-live is the only other repo that uses sympy-web-static, so we really should just move everything there into this repo and get rid of the submodule.

[jtpio]

Posting here for visibility.

@ivanistheone has proposed taking over this PR, thanks!

The diff is quite minimal, so feel free to adjust and modify as needed, based on your other updates to the website.

Happy to help on the JupyterLite side if needed.

After this, I'll follow up with separate work (PHASE 3) on new sympy/live repo that will power https://live.sympy.org (a clone of https://github.com/jupyterlite/demo with possible customizations such as ENTER vs SHIFT+ENTER handlers etc.). @jtpio I'll appreciate your help/advice with this step in coming weeks if you're avail.

Yes you can easily generate your own JupyterLite static website from that template. Then use the settings system via the overrides.json file to provide the custom keyboard shortcut.

[ivanistheone]

Thanks @jtpio for posting the update. I'm continuing PHASE 1 tasks in the new PR #174 .

Sorry for delay. I will push on this more this week.

[ivanistheone]

@jtpio please close this PR since code changes are all in #174 now.

[jtpio]

Closing in favor of #174, thanks @ivanistheone!