Building a small Jupyter book is slow, and very CPU intensive: why? (compared to classical Sphinx)

[Naereen]

Hello there,

I've been using Sphinx for years, to build my personal website.
It has about 150 pages, ~100 images, and about a dozen pages use Sphinx extensions like autorun, gnuplot, graphviz etc, and thus require compile time computations. (the sources are here).
Also some pages use autorun/runblock to show examples of ping, dig, Python, OCaml and Bash code, and also command lines utilities that fetch online results (for instance from Wolfram|Alpha).
Building from scratch on my laptop takes less than 120 seconds, even with no cache and an empty _build/ folder:

$ time make preprocess html images obscure send_jarvis send_zamok send_ovh check
[...]
Searching for severes ...
No severe : very good job :)
Searching for errors ...
No error : good job :)
notify-send "Sphinx" "Generating documentation : done !"
make : on quitte le répertoire « /home/lilian/web-sphinx »

real 110,614    user 30,405     sys 3,562       pcpu 30,70

This time includes sending the pages to two web-servers!

I've started to use Jupyter Book yesterday, and I'm starting by writing a demo book, before trying to use it for my future teaching resources. So far, the example book is tiny, as I don't want to spend weeks on writing content if I'm not 100% satisfied of the result.

I love how it looks, the extensions, the MyST language, the fact that it can read and process Jupyter notebooks etc.
I use it in French, and this morning I submitted a tiny pull request to update the French translation of sphinx-book-theme.

But I'm really sad to see that it takes ages to build, even this tiny project takes much, much longer than my large and heavy websites, and my CPU even starts to blow and warns me that it's used at 100% by a process:

$ jupyter-book -W -n --keep-going build --builder html .
jupyter-book build -W -n --keep-going --builder html .                                                                              
Running Jupyter-Book v0.9.1
Source Folder: /home/lilian/publis/Info-Prepas-MP2I/Modele-de-livre-avec-Jupyter-Book.git                                           
Config Path: /home/lilian/publis/Info-Prepas-MP2I/Modele-de-livre-avec-Jupyter-Book.git/_config.yml
Output Path: /home/lilian/publis/Info-Prepas-MP2I/Modele-de-livre-avec-Jupyter-Book.git/_build/html                            
Running Sphinx v3.4.3                                                                                                               
loading translations [fr]... done
[...]
generating indices... genindex done                                                                                                 
writing additional pages... search done
copying images... [100%] _build/jupyter_execute/notebooks_2_0.png
copying downloadable files... [100%] references.bib
copying static files... done
copying extra files... done
dumping search index in French (code: fr)... done
dumping object inventory... done
build finished with problems, 36 warnings.
===============================================================================
Building your book, returns a non-zero exit code (1). Look above for the cause.
===============================================================================

real 444,780    user 442,242    sys 2,980       pcpu 100,10

The sources contain 5 small .ipynb notebooks, a couple of Markdown, rST and MyST pages, almost no images, no notebooks to build, nothing that needs to fetch results from Internet... and the displayed time is just for building the HTML pages, not even sending them or anything else.

My example book is there: https://github.com/Info-Prepas-MP2I/Modele-de-livre-avec-Jupyter-Book/

If anyone has insights to know why the Jupyter-book build is that much slower than a Sphinx build...
Thanks in advance!
-- @Naereen

[Naereen]

Apparently, it is the "writing output" phase which takes really a long time.

I know Sphinx quite well and I can't understand why it could be slowed down by the overlay of Jupyter Book...
I suspect that some part of the code is bugged and slow for no reason, but I won't have time to benchmark anything right now...

Any insights?

[chrisjsewell]

Hi again! thanks for the follow up. Hmm nothing specific comes to mind; maybe something to do with the the output rendering (https://github.com/executablebooks/MyST-NB/blob/master/myst_nb/render_outputs.py) thats in a post-transform, which may come under the writing output phase.
I'm hoping to ~rewrite that sometime soon, so I will certainly bare the in mind and see if I can find any bottlenecks.

[Naereen]

Thanks for the reply.
I quickly read this file, I have to admit it's quite long, and I don't have that much time to spend on this...
But I didn't see any part which looked very inefficient, so I don't know where the bottlenecks could be...

[choldgraf]

I think what I'd try to do is copy the configuration that Jupyter Book uses as much as possible (see https://github.com/executablebooks/jupyter-book/blob/master/jupyter_book/sphinx.py#L51 for where we call Sphinx and generate the config for it). Then see if the slow-down occurs, and finally start deactivating an extension here and there to make it happen.

A quick thought - could there also be a cacheing issue or something? I've found Jupyter Book to be a bit unreliable in whether it decides to "rebuild all pages" vs. "just rebuild updated pages"...I'm not sure if we've fixed this yet, but just an idea to look into.

[Naereen]

I don't know about this cacheing issue, but it shouldn't be a problem: when building my (pretty large) website from scratch, there is no cache.
For my example of Jupyter book, the building time doesn't seem to get shorter after a second build, thus I suspect that no caching happens.