sqlite3.OperationalError: database is locked

[vkatrychenko]

Issue

Jupyter notebook cannot connect to kernel. Try to use different images, but no luck.

Jupyter docker images:
kubeflownotebookswg/jupyter-scipy:v1.9.0
kubeflownotebookswg/jupyter-pytorch-full:v1.9.0

I can see the following errors in the pod of jupyter notebook:

File "/opt/conda/lib/python3.11/site-packages/traitlets/config/configurable.py", line 583, in instance inst = cls(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 125, in __init__ self.shell = self.shell_class.instance( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/configurable.py", line 583, in instance inst = cls(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 641, in __init__ self.init_history() File "/opt/conda/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 1908, in init_history self.history_manager = HistoryManager(shell=self, parent=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 544, in __init__ super(HistoryManager, self).__init__(shell=shell, config=config, File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 223, in __init__ self.init_db() File "/opt/conda/lib/python3.11/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 107, in catch_corrupt_db self.init_db() File "/opt/conda/lib/python3.11/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 105, in catch_corrupt_db self.hist_file.rename(newpath) File "/opt/conda/lib/python3.11/pathlib.py", line 1175, in rename os.rename(self, target) PermissionError: [Errno 13] Permission denied: '/home/jovyan/.ipython/profile_default/history.sqlite' -> '/home/jovyan/.ipython/profile_default/history-corrupt.sqlite' [I 2024-08-28 12:12:56.433 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports [W 2024-08-28 12:12:58.037 ServerApp] Timeout waiting for kernel_info reply from d94713c9-975c-4812-a2f6-bc9ca5d8ece1 [I 2024-08-28 12:12:58.038 ServerApp] Connecting to kernel d94713c9-975c-4812-a2f6-bc9ca5d8ece1. [W 2024-08-28 12:13:01.114 ServerApp] Replacing stale connection: d94713c9-975c-4812-a2f6-bc9ca5d8ece1:b30ac2fd-46c4-4882-8392-e40d58595022

Kubeflow verion 1.9.0
Notebook controller: 1.9.0
AKS cluster 1.28.12