Add pooling heated kernels and pre-rendered notebook

[trungleduc]

New features

This PR aims to accelerate the startup time of Voila by adding two features:

Pooling pre-heated kernels

• A pool of kernels is started and kept in standby, the notebook Voila started with is executed in every kernel of the pool.
• When a new client requests a kernel, the pre-heated kernel in pool is used and another kernel is started asynchronously to refill the pool.
• This feature is partially addressed by hotpot_km. Some code snippets of this PR are taken from this library with permission of the author.

Pre-rendered notebook

• The HTML version of notebook is rendered in each pre-heated kernel and stored, when a client connects to Voila, under some conditions, the cached HTML is served instead of re-rendering the notebook.

How it works

When Voila started with preheat_kernel=True, it will create multiple asyncio.Task for each notebook. Each task will start a kernel and execute the notebook in this kernel. Voila keeps a mapping of notebook name to the list of tasks started for this notebook. Each task will return a tuple which contains a list HTML string of rendered cells and an async generator which can be used to render the remaining cells.

As depicted in the image below, the content of elements in this tuple depends on the moment Voila receives a request:

• If there is a kernel with notebook fully executed, Voila will get the HTML string of all cells then send to frontend.
• If all kernel are executing notebook, Voila will pick the first started kernel and send the rendered cell to frontend while executing the remaining.

Speed comparison

• With a fully executed notebook

voila.mp4

-With a partially executed notebook

voilahp.mp4

Current status

The pre-heat kernel option works with any kernel manager, it is deactivated by default, re-activate it by setting preheat_kernel = True. For example, with this command, for each notebook Voilà started with, a pool of 5 kernels is created and will be used for new connections.

    voila --preheat_kernel=True --pool_size=5

To change the additional settings ( list of notebooks does not need pre-heated kernels, number of kernels in pool, refilling delay, environment variables for starting kernel...), users can use voila.json configuration file in the same folder containing the notebooks. Here is an example of settings with explanations for pre-heat kernel option.

   # voila.json
   {
      "VoilaConfiguration": {
         # Activate or deactivate preheat kernel option.
         "preheat_kernel": true 
      },
      "VoilaKernelManager": {
         # A list of notebook name or regex patterns to exclude notebooks from using preheat kernel.
         "preheat_blacklist": [
            "notebook-does-not-need-preheat.ipynb",
            "^.*foo.*$",
            ...
         ], 
         # Configuration for kernel pools
         "kernel_pools_config": { 
            # Setting for `voila.ipynb` notebook
            "voila.ipynb": {
               "pool_size": 3, # Size of pool
               "kernel_env_variables": { # The environment variables used to start kernel for `voila.ipynb`
                  "foo2": "bar2"
               }
            },
            # Setting for `test/sub-voila.ipynb` notebook
            "test/sub-voila.ipynb": {
               "pool_size": 1
            },
            ...
            # If a notebook does not have setting, it will use default setting
            "default": {
               "pool_size": 2,
               "kernel_env_variables": {
                  "foo": "bar"
               }
            },
         },
         # Delay time in second before filling the kernel pool.
         "fill_delay": 0
      }
   }

Notebook HTML will be pre-rendered with template and theme defined in VoilaConfiguration or in notebook metadata. The pre-heated kernel and cached HTML are used if these conditions are matched:

• There is an available pre-heated kernel in the kernel pool.
• If user overrides the template/theme with query string, it must match the template/theme used to pre-render the notebook.
• There is no other query strings than voila-theme and voila-template.

If the kernel pool is empty or the request does not match these conditions, Voila will fail back to start a normal kernel and render the notebook as usual.

Code changes

• Refactoring VoilaHandler: the notebook rendering part of VoilaHandler is extracted as a new class NotebookRenderer, this class is used to render notebook in both voila handler and pre-heated kernel
• Add new function voila_kernel_manager_factory to decorate the kernel manager class defined in VoilaConfiguration.multi_kernel_manager_class with pooling features
• Add two new options preheat_kernel and pool_size to Voila application to set up kernel pool.

Backwards-incompatible changes

Aside VoilaHandler refactoring, the notable changes come from this PR are:

• Add two new options --preheat_kernel and pool_size to Voila CLI
• Decorate the user defined kernel manager of Voila with the preheat features if preheat_kernel=True.

Since preheat_kernel is set to False by defaut, we should not have any breaking changes with 0.2.x

[trungleduc]

Ping @SylvainCorlay @maartenbreddels. Can you have a look at this?

[jtpio]

Thanks @trungleduc for starting this, it looks very promising!

[vidartf]

Would you mind adding some details about how this compares to https://github.com/voila-dashboards/hotpot_km/ ? (i.e. what extra functionality does this add; are there any functionality missing, etc).

[trungleduc]

Would you mind adding some details about how this compares to https://github.com/voila-dashboards/hotpot_km/ ? (i.e. what extra functionality does this add; are there any functionality missing, etc).

Actually I based on hotpot_km ideal and some implementations to create this PR. Three main differences with respect to hotpot_km:

• The pooling feature can be added to any kernel manager by decoration instead of a standalone kernel manager.
• The pre-executed code is the current notebook of Voila , the possibility to run user defined code is not available.
• The notebook is rendered and cached, when a user connected to voia, both pre-heated kernel and cached HTML are served.

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

Results table

Test file	basics.ipynb	bqplot.ipynb	dashboard.ipynb	gridspecLayout.ipynb	interactive.ipynb	ipympl.ipynb	ipyvolume.ipynb	multiple_widgets.ipynb	query-strings.ipynb
Render
chromium
actual	3491 <- [3563 - 3752 - 3906] -> 4255	3068 <- [3187 - 3267 - 3339] -> 3471	3218 <- [3473 - 3541 - 3591] -> 3613	3932 <- [4301 - 4331 - 4451] -> 4546	2287 <- [2356 - 2366 - 2369] -> 2396	4240 <- [4424 - 4646 - 5072] -> 6335	16259 <- [16905 - 19867 - 20340] -> 20874	14742 <- [14818 - 14879 - 14896] -> 15032	1518 <- [1767 - 1785 - 1804] -> 1832
expected	3379 <- [3442 - 3517 - 3701] -> 3876	2976 <- [3227 - 3321 - 3421] -> 3604	3608 <- [3623 - 3709 - 3793] -> 3825	4453 <- [4453 - 4523 - 4661] -> 4748	2559 <- [2655 - 2656 - 2660] -> 2674	3982 <- [4079 - 4213 - 4356] -> 4743	12183 <- [18509 - 19553 - 20811] -> 21515	15319 <- [15660 - 15796 - 15912] -> 16056	1517 <- [1920 - 1997 - 2103] -> 2113

❗ Test metadata have changed

--- /dev/fd/63	2021-10-04 17:44:28.991310708 +0000
+++ /dev/fd/62	2021-10-04 17:44:28.991310708 +0000
@@ -8,27 +8,27 @@
   },
   "systemInformation": {
     "cpu": {
-      "brand": "Xeon® E5-2673 v4",
+      "brand": "Xeon® E5-2673 v3",
       "cache": {
         "l1d": 65536,
         "l1i": 65536,
         "l2": 524288,
-        "l3": 52428800
+        "l3": 31457280
       },
       "cores": 2,
       "family": "6",
-      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear",
+      "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt md_clear",
       "governor": "",
       "manufacturer": "Intel®",
-      "model": "79",
+      "model": "63",
       "physicalCores": 2,
       "processors": 1,
       "revision": "",
       "socket": "",
-      "speed": 2.3,
+      "speed": 2.4,
       "speedMax": null,
       "speedMin": null,
-      "stepping": "1",
+      "stepping": "2",
       "vendor": "GenuineIntel",
       "virtualization": false,
       "voltage": ""
@@ -42,11 +42,11 @@
       "codename": "Focal Fossa",
       "codepage": "UTF-8",
       "distro": "Ubuntu",
-      "kernel": "5.8.0-1042-azure",
+      "kernel": "5.8.0-1040-azure",
       "logofile": "ubuntu",
       "platform": "linux",
       "release": "20.04.3 LTS",
-      "serial": "487dcaac68944409a00596acdb9a6fc0",
+      "serial": "cfc067bfcb844f35865e279a1b0e66c5",
       "servicepack": "",
       "uefi": false
     }

[jtpio]

Thanks @trungleduc for pushing on this.

Looks like there is a small conflict to fix.

[trungleduc]

Weird, I just rebase this PR on master to pick up latest changes, then the snapshot for ipympl failed

[jtpio]

then the snapshot for ipympl failed

Could it be related to the new 0.8.0 release from yesterday? https://github.com/matplotlib/ipympl/releases/tag/0.8.0

[trungleduc]

Yes partially, updated to 0.8.0 on my pc, still have difference between output from CI and my local output. Maybe we need to disable ipympl snapshot check?

[jtpio]

Or we update the screenshot with the one generated on CI?

[trungleduc]

In this case the local test won't pass

[jtpio]

Right, but I think the common practice is to use the ones generated on CI as reference since this is where the checks are running.

Or use Docker like in the JupyterLab repo to minimize differences, although that would complicate the setup.

[jtpio]

By the way, this PR is looking good and promising.

Adding to the 0.3.0 milestone so we get a chance at merging it early next week maybe, if we switch main to target 0.3.0 pre-releases.

@vidartf @maartenbreddels would you like to give it a quick review since you have been doing similar work on hotpot_km? Thanks!

[jtpio]

Thanks @trungleduc for working on this.

Let me know if you would like to push more changes. Otherwise we can wait for the tests to pass, merge, and release 0.3.0a0 from main with this.

cc @SylvainCorlay as the 0.3.0a0 release was discussed on other channels.

[trungleduc]

@jtpio I'm figuring out why the test many_iopub_messages_test.py fails randomly on this PR while it passed on recent PRs. Other than that I don't have anything left.

[jtpio]

OK thanks for heads-up 👍

Feel free to ping here when you think it's ready.

[trungleduc]

@jtpio It's all green now 😄

[SylvainCorlay]

🎉

[jtpio]

Cutting 0.3.0a0 with this change now.