Improve handling of notebooks in documentation

.envs/testenv-linkcheck.yml

@@ -12,8 +12,7 @@ dependencies:
 
 
   - pydata-sphinx-theme>=0.3.0  # docs
-  - myst-parser  # docs
-  - nbsphinx  # docs
+  - myst-nb  # docs
   - sphinx  # docs
   - sphinx-copybutton  # docs
   - sphinx-panels  # docs

.github/workflows/main.yml

@@ -77,25 +77,6 @@ jobs:
           micromamba activate estimagic
           pytest -m "not slow and not jax"
 
-  check-links-in-documentation:
-    name: Check links in documentation.
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: create build environment
-        uses: mamba-org/provision-with-micromamba@main
-        with:
-          environment-file: ./.envs/testenv-linkcheck.yml
-          environment-name: estimagic
-          cache-env: true
-          extra-specs: python=3.10
-      - name: run sphinx
-        shell: bash -l {0}
-        run: |
-          micromamba activate estimagic
-          cd docs/source
-          sphinx-build -T -b linkcheck -d doctrees . linkcheck
-
   code-in-docs:
     name: Run code snippets in documentation
     runs-on: ubuntu-latest

docs/rtd_environment.yml

@@ -18,11 +18,22 @@ dependencies:
   - sphinx-panels
   - ipython
   - ipython_genutils
-  - nbsphinx
+  - myst-nb
   - pydata-sphinx-theme>=0.3.0
   - numba
   - pybaum
-  - myst-parser
+  - matplotlib
+  - seaborn
+  - numpy
+  - pandas
+  - scipy
+  - patsy
+  - joblib
+  - plotly
+  - ipython
 
   - pip:
       - ../
+      - kaleido
+      - Py-BOBYQA
+      - DFO-LS

docs/source/algorithms.md

@@ -144,7 +144,7 @@ install estimagic.
 
     The criterion function need not be differentiable.
 
-    Powell's method is a conjugate direction method, minimising the function by a
+    Powell's method is a conjugate direction method, minimizing the function by a
     bi-directional search in each parameter's dimension.
 
     The argument ``direc``, which is the initial set of direction vectors and which
@@ -188,7 +188,7 @@ install estimagic.
     - **stopping.max_iterations** (int): If the maximum number of iterations is reached, the optimization stops,
       but we do not count this as convergence.
     - **norm** (float): Order of the vector norm that is used to calculate the gradient's "score" that
-      is compared to the gradient tolerance to determine convergence. Defaut is infinite which means that
+      is compared to the gradient tolerance to determine convergence. Default is infinite which means that
       the largest entry of the gradient vector is compared to the gradient tolerance.
 
 ```
@@ -286,7 +286,7 @@ install estimagic.
   Minimize a scalar function of one or more variables using the COBYLA algorithm.
 
   COBYLA stands for Constrained Optimization By Linear Approximation.
-  It is deriviative-free and supports nonlinear inequality and equality constraints.
+  It is derivative-free and supports nonlinear inequality and equality constraints.
 
   .. note::
       Cobyla's general nonlinear constraints is not supported yet by estimagic.
@@ -398,12 +398,12 @@ install estimagic.
     .. note::
         Its general nonlinear constraints' handling is not supported yet by estimagic.
 
-    It swiches between two implementations depending on the problem definition.
+    It switches between two implementations depending on the problem definition.
     It is the most versatile constrained minimization algorithm
     implemented in SciPy and the most appropriate for large-scale problems.
     For equality constrained problems it is an implementation of Byrd-Omojokun
     Trust-Region SQP method described in :cite:`Lalee1998` and in :cite:`Conn2000`,
-    p. 549. When inequality constraints  are imposed as well, it swiches to the
+    p. 549. When inequality constraints  are imposed as well, it switches to the
     trust-region interior point method described in :cite:`Byrd1999`.
     This interior point algorithm in turn, solves inequality constraints by
     introducing slack variables and solving a sequence of equality-constrained
@@ -586,7 +586,7 @@ install estimagic.
     The algorithm supports the following options:
 
     - **local_algorithm** (str/callable): Any scipy local minimizer: valid options are.
-    "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA".
+      "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA".
     "SLSQP". "trust-constr". "dogleg". "trust-ncg". "trust-exact". "trust-krylov".
     or a custom function for local minimization, default is "L-BFGS-B".
     - **n_local_optimizations**: (int) The number local optimizations. Default is 100 as in scipy's default.
@@ -742,7 +742,7 @@ install estimagic.
     - **local_algo_options**: (dict) Additional keyword arguments for the local
       minimizer. Check the documentation of the local scipy algorithms for details on
       what is supported.
-    - **initial_temperature** (float): The temparature algorithm starts with. The higer values lead to a wider search space. The range is (0.01, 5.e4] and defalt is 5230.0.
+    - **initial_temperature** (float): The temparature algorithm starts with. The higher values lead to a wider search space. The range is (0.01, 5.e4] and default is 5230.0.
     - **restart_temperature_ratio** (float): Reanneling starts when the algorithm is decreased to initial_temperature * restart_temperature_ratio. Default is 2e-05.
     - **visit** (float): Specify the thickness of visiting distribution's tails. Range is (1, 3] and default is scipy's default, 2.62.
     - **accept** (float): Controls the probability of acceptance. Range is (-1e4, -5] and default is scipy's default, -5.0. Smaller values lead to lower acceptance probability.

docs/source/conf.py

@@ -44,11 +44,10 @@
     "sphinx.ext.viewcode",
     "sphinx.ext.napoleon",
     "sphinx_copybutton",
-    "nbsphinx",
+    "myst_nb",
     "sphinxcontrib.bibtex",
     "sphinx_panels",
     "sphinx_copybutton",
-    "myst_parser",
 ]
 
 myst_enable_extensions = [
@@ -103,7 +102,7 @@
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = [".rst", ".md"]
+source_suffix = [".rst", ".ipynb", ".md"]
 
 # The master toctree document.
 master_doc = "index"
@@ -142,18 +141,21 @@
     todo_include_todos = True
     todo_emit_warnings = True
 
-# -- Options for nbsphinx  ----------------------------------------
-# Execute notebooks before conversion: 'always', 'never', 'auto' (default)
-nbsphinx_execute = "never"
-nbsphinx_prolog = r"""
-{% set docname = 'docs/source/' + env.doc2path(env.docname, base=None) %}
+# -- Options for myst-nb  ----------------------------------------
+nb_execution_mode = "force"
+nb_execution_allow_errors = False
+nb_merge_streams = True
 
-.. only:: html
+# Notebook cell execution timeout; defaults to 30.
+nb_execution_timeout = 1000
 
-    .. nbinfo::
-        Download the notebook :download:`here
-        <https://nbviewer.jupyter.org/github/OpenSourceEconomics/estimagic/blob/main/{{ docname }}>` # noqa: E501
-"""
+# List of notebooks that will not be executed.
+nb_execution_excludepatterns = [
+    # Problem with latex rendering
+    "how_to_generate_publication_quality_tables.ipynb",
+    # too long runtime
+    "bootstrap_montecarlo_comparison.ipynb",
+]
 
 # -- Options for HTML output ----------------------------------------------
 

docs/source/explanations/inference/bootstrap_montecarlo_comparison.ipynb

@@ -81,9 +81,7 @@
     "    cluster = []\n",
     "\n",
     "    for g in range(nclusters):\n",
-    "\n",
     "        for i in range(nobs_per_cluster):\n",
-    "\n",
     "            key = (i + 1) * (g + 1) - 1\n",
     "\n",
     "            arg = (\n",
@@ -136,7 +134,6 @@
     "    \"\"\"\n",
     "\n",
     "    def logit_wrap(df):\n",
-    "\n",
     "        y = df[\"y\"]\n",
     "        x = df[\"x\"]\n",
     "\n",
@@ -145,12 +142,10 @@
     "        return pd.Series(result, index=[\"constant\", \"x\"])\n",
     "\n",
     "    if cluster is False:\n",
-    "\n",
     "        result = em.bootstrap(data=data, outcome=logit_wrap, n_draws=sample_size)\n",
     "        estimates = pd.DataFrame(result.outcomes)[\"x\"]\n",
     "\n",
     "    else:\n",
-    "\n",
     "        result = em.bootstrap(\n",
     "            data=data,\n",
     "            outcome=logit_wrap,\n",
@@ -189,7 +184,6 @@
     "    np.zeros(nsim)\n",
     "\n",
     "    def loop():\n",
-    "\n",
     "        df = create_clustered_data(nclusters, nobs_per_cluster, true_beta)\n",
     "\n",
     "        return [get_t_values(df), get_t_values(df, cluster=True)]\n",
@@ -231,7 +225,6 @@
     "results_list = []\n",
     "\n",
     "for g, k in [[20, 50], [100, 10], [500, 2]]:\n",
-    "\n",
     "    results_list.append(monte_carlo(nsim=100, nclusters=g, nobs_per_cluster=k))"
    ]
   },

docs/source/explanations/inference/index.md

@@ -5,6 +5,6 @@
 maxdepth: 1
 ---
 bootstrap_ci
-bootstrap_montecarlo_comparison.ipynb
+bootstrap_montecarlo_comparison
 cluster_robust_likelihood_inference.md
 ```

docs/source/getting_started/estimation/index.md

@@ -6,9 +6,12 @@ parameter estimates and standard errors in a format that can be easily used to c
 publication quality latex or html tables.
 
 ```{toctree}
+
 ---
 maxdepth: 1
 ---
-first_likelihood_estimation_with_estimagic.ipynb
-first_msm_estimation_with_estimagic.ipynb
+
+first_likelihood_estimation_with_estimagic
+first_msm_estimation_with_estimagic
+
 ```

docs/source/getting_started/index.md

@@ -77,7 +77,7 @@ hidden: true
 maxdepth: 1
 ---
 installation
-first_optimization_with_estimagic.ipynb
+first_optimization_with_estimagic
 estimation/index
-first_derivative_with_estimagic.ipynb
+first_derivative_with_estimagic
 ```

docs/source/how_to_guides/differentiation/index.md

@@ -1,10 +1,12 @@
 # Differentiation
 
 ```{toctree}
+
 ---
 maxdepth: 1
 ---
-how_to_calculate_first_derivatives.ipynb
-how_to_calculate_second_derivatives.ipynb
-how_to_plot_derivatives.ipynb
+how_to_calculate_first_derivatives
+how_to_calculate_second_derivatives
+how_to_plot_derivatives
+
 ```

docs/source/how_to_guides/inference/index.md

@@ -1,10 +1,12 @@
 # Inference
 
 ```{toctree}
+
 ---
 maxdepth: 1
 ---
-how_to_calculate_likelihood_standard_errors.ipynb
-how_to_calculate_msm_standard_errors.ipynb
-how_to_do_bootstrap_inference.ipynb
+how_to_calculate_likelihood_standard_errors
+how_to_calculate_msm_standard_errors
+how_to_do_bootstrap_inference
+
 ```

docs/source/how_to_guides/miscellaneous/index.md

@@ -1,11 +1,13 @@
 # Miscellaneous Topics
 
 ```{toctree}
+
 ---
 maxdepth: 1
 ---
-how_to_generate_publication_quality_tables.ipynb
-how_to_use_batch_evaluators.ipynb
-how_to_visualize_and_interpret_sensitivity_measures.ipynb
+how_to_generate_publication_quality_tables
+how_to_use_batch_evaluators
+how_to_visualize_and_interpret_sensitivity_measures
+
 faq
 ```

docs/source/how_to_guides/optimization/index.md

@@ -12,11 +12,11 @@ how_to_specify_bounds
 how_to_specify_constraints
 how_to_use_logging
 how_to_use_the_dashboard
-how_to_handle_errors_during_optimization.ipynb
+how_to_handle_errors_during_optimization
 how_to_scale_optimization_problems
-how_to_do_multistart_optimizations.ipynb
-how_to_benchmark_optimization_algorithms.ipynb
-how_to_visualize_histories.ipynb
-how_to_visualize_an_optimization_problem.ipynb
-how_to_pick_an_optimizer.ipynb
+how_to_do_multistart_optimizations
+how_to_benchmark_optimization_algorithms
+how_to_visualize_histories
+how_to_visualize_an_optimization_problem
+how_to_pick_an_optimizer
 ```

docs/source/index.md

@@ -165,6 +165,6 @@ development/index
 algorithms
 ```
 
-[how to do multistart]: how_to_guides/optimization/how_to_do_multistart_optimizations.ipynb
-[how to use logging]: how_to_guides/optimization/how_to_use_logging.ipynb
-[msm tutorial]: getting_started/estimation/first_msm_estimation_with_estimagic.ipynb
+[how to do multistart]: how_to_guides/optimization/how_to_do_multistart_optimizations
+[how to use logging]: how_to_guides/optimization/how_to_use_logging
+[msm tutorial]: getting_started/estimation/first_msm_estimation_with_estimagic

environment.yml

@@ -35,8 +35,7 @@ dependencies:
   - sqlalchemy <2.0   # run, tests
 
   - pydata-sphinx-theme>=0.3.0  # docs
-  - myst-parser  # docs
-  - nbsphinx  # docs
+  - myst-nb  # docs
   - sphinx  # docs
   - sphinx-copybutton  # docs
   - sphinx-panels  # docs