linear_regression.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Pegp3_Ejmro"
      },
      "source": [
        "# MLE, MAP - Linear Regression Tutorial\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4UX2jxhjmrv"
      },
      "source": [
        "We consider a linear regression problem of the form\n",
        "$$\n",
        "y = \\boldsymbol x^T\\boldsymbol\\theta + \\epsilon\\,,\\quad \\epsilon \\sim \\mathcal N(0, \\sigma^2)\n",
        "$$\n",
        "where $\\boldsymbol x\\in\\mathbb{R}^D$ are inputs and $y\\in\\mathbb{R}$ are noisy observations. The parameter vector $\\boldsymbol\\theta\\in\\mathbb{R}^D$ parametrizes the function.\n",
        "\n",
        "We assume we have a training set $(\\boldsymbol x_n, y_n)$, $n=1,\\ldots, N$. We summarize the sets of training inputs in $\\mathcal X = \\{\\boldsymbol x_1, \\ldots, \\boldsymbol x_N\\}$ and corresponding training targets $\\mathcal Y = \\{y_1, \\ldots, y_N\\}$, respectively.\n",
        "\n",
        "In this tutorial, we are interested in finding good parameters $\\boldsymbol\\theta$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "NCNDwBo4jmrx",
        "outputId": "3563aaad-d73d-4ada-cfde-d208cdfc4a7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPyUlEQVR4nO3df6zdd13H8eeLbgMEFbRXGNvCnfHOMOWXniwSjZnpqoWRFRSSLUZBJQ3GQV000riExSHJiIllAgEbmQyzMAg/pNriKAMdaoY7XQbsB2vrMrLOyS6bgAvoUnj7xz3Du+7c9rP743zPj+cjOen3x6ffvr5pd1/7/jjfb6oKSZJO5ildB5AkTQYLQ5LUxMKQJDWxMCRJTSwMSVKTU7oOsFE2b95c8/PzXceQpIly8ODBr1fV3LB1U1sY8/Pz9Pv9rmNI0kRJ8tWV1nlKSpLUxMKQJDWxMCRJTSwMSZoyuw8c2pDtWhiSNGWuvvHwhmzXwpAkNbEwJElNLAxJUpOp/eKeJM2C3QcODb1mMb9r3+Pmd25Z4LKt56zpz8q0vkCp1+uV3/SWNIvmd+3j3qsuXNXvTXKwqnrD1nlKSpLUxMKQJDWxMCRJTSwMSZoyO7csbMh2LQxJmjJrvRtqJRaGJKmJhSFJajIWhZHkmiQPJrl9hfXnJ/lmktsGn7eOOqMkzbpx+ab3B4B3Ax88wZjPV9UrRxNHknS8sTjCqKqbgIe7ziFJWtlYFEajlyX5YpJPJfmpYQOS7EjST9JfXFwcdT5JmmqTUhi3As+vqhcD7wL+dtigqtpTVb2q6s3NzY0ynyRNvYkojKr6VlU9MpjeD5yaZHPHsSRppkxEYSR5bpIMps9jKfdD3aaSpNkyFndJJfkQcD6wOclR4ArgVICqeh/wGuB3kxwDvgNcXNP6XHZJGlNjURhVdclJ1r+bpdtuJUkdmYhTUpKk7lkYkqQmFoYkqYmFIUlqYmFIkppYGJKkJhaGJKmJhSFJamJhSJKaWBiSpCYWhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlqYmFIkppYGJKkJhaGJKmJhSFJamJhSJKaWBiSpCYWhiSpiYUhSWpiYUiSmoxFYSS5JsmDSW5fYX2S/EWSI0m+lORnRp1RkmbdWBQG8AFg2wnWvxxYGHx2AO8dQSZJ0jJjURhVdRPw8AmGbAc+WEtuBp6V5PTRpJMkwZgURoMzgPuWzR8dLHucJDuS9JP0FxcXRxZOkmbBpBRGk6raU1W9qurNzc11HUeSpsqkFMb9wFnL5s8cLJMkjcikFMZe4DcHd0v9HPDNqnqg61CSNEtO6ToAQJIPAecDm5McBa4ATgWoqvcB+4FXAEeAbwO/1U1SSZpdY1EYVXXJSdYX8HsjiiNJGmJSTklJkjpmYUiSmlgYkqQmFoYkqYmFIUlqYmFIkppYGJKkJhaGJKmJhSFJamJhSJKaWBiSpCYWhiSpiYUhaU12HzjUdQSNiIUhaU2uvvFw1xE0IhaGJKmJhSFJamJhSJKajMUb9yRNht0HDg29ZjG/a9/j5nduWeCyreeMKpZGJEtvP50+vV6v+v1+1zGkqTe/ax/3XnVh1zG0TpIcrKresHWekpIkNbEwJElNLAxJUhMLQ9Ka7Nyy0HUEjYiFIWlNvBtqdlgYkqQmFoYkqclYFEaSbUnuTnIkya4h61+fZDHJbYPPG7rIKUmzrPNveifZBLwH2AocBW5Jsreq7jxu6Ier6tKRB5QkAeNxhHEecKSq7qmqR4Hrge0dZ5IkHWccCuMM4L5l80cHy473a0m+lOSjSc4atqEkO5L0k/QXFxc3IqskzaxxKIwWfwfMV9WLgAPAtcMGVdWequpVVW9ubm6kASVp2o1DYdwPLD9iOHOw7Puq6qGq+t/B7F8BPzuibJKkgXEojFuAhSRnJzkNuBjYu3xAktOXzV4E3DXCfJIkxuAuqao6luRS4AZgE3BNVd2R5EqgX1V7gTcnuQg4BjwMvL6zwJI0o3wfhiTp+3wfhiRpzSwMSVITC0OS1MTCkCQ1sTAkSU0sDElSEwtDktTEwpAkNbEwJElNLAxJUhMLQ5LU5KSFkeRAkhePIowkaXy1HGG8BXhnkr8+7jHjkqQZctLCqKpbq+qXgL8H/iHJFUmevvHRJEnjpOkaRpIAdwPvBd4EHE7yGxsZTJpmuw8c6jqC9KS1XMP4F5ZembobOIOllxedD5yXZM9GhpOm1dU3Hu46gvSktbxxbwdwZz3xTUtvSuKrUiVpRpy0MKrqjhOsvnAds0iSxtiavodRVfesVxBJ0nhrOSUlaQ12Hzg09JrF/K59j5vfuWWBy7aeM6pY0pOWJ16amA69Xq/6/X7XMaSh5nft496rPKOr8ZPkYFX1hq3z0SCSpCYWhiSpiYUhSWpiYUgd2LlloesI0pM2FoWRZFuSu5McSbJryPqnJvnwYP0Xksx3EFNaN94NpUnUeWEk2QS8B3g5cC5wSZJzjxv2O8B/VdVPsPSIkneMNqUkqfPCAM4DjlTVPVX1KHA9sP24MduBawfTHwW2DB6IKEkakXEojDOA+5bNHx0sGzqmqo4B3wR+9PgNJdmRpJ+kv7i4uEFxJWk2jUNhrJuq2lNVvarqzc3NdR1HkqbKOBTG/cBZy+bPHCwbOibJKcAPAw+NJJ0kCRiPwrgFWEhydpLTgIuBvceN2Qu8bjD9GuCzQx63LknaQJ0/fLCqjiW5FLgB2ARcU1V3JLkS6FfVXuD9wN8kOQI8zFKpSJJGqPPCAKiq/cD+45a9ddn0/wCvHXUuSdL/G4dTUpKkCWBhSJKaWBiSpCYWhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlqYmFIkppYGJKkJhaGJKmJhSFJamJhSJKaWBiSpCYWhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlqYmFIkppYGJKkJhaGJKmJhSFJamJhSJKadFoYSX4kyYEkhwe/PnuFcd9Nctvgs3fUOSVJ3R9h7AJurKoF4MbB/DDfqaqXDD4XjS6eJOkxXRfGduDawfS1wKu6iyJJOpGuC+M5VfXAYPo/geesMO5pSfpJbk7yqpU2lmTHYFx/cXFxvbNK0kw7ZaP/gCSfAZ47ZNXly2eqqpLUCpt5flXdn+THgc8m+XJV/fvxg6pqD7AHoNfrrbQtSdIqbHhhVNUFK61L8rUkp1fVA0lOBx5cYRv3D369J8k/Ai8FnlAYkqSN0/Upqb3A6wbTrwM+efyAJM9O8tTB9Gbg54E7R5ZQkgR0XxhXAVuTHAYuGMyTpJfkrwZjXgD0k3wR+BxwVVVZGJI0Yht+SupEquohYMuQ5X3gDYPpfwVeOOJokqTjdH2EIUmaEBaGJKmJhaGJsfvAoa4jSDPNwtDEuPrGw11HkGaahSFJamJhSJKaWBiSpCadfg9DWsnuA4eGXrOY37XvcfM7tyxw2dZzRhVLmmmpms5n9PV6ver3+13H0Dqa37WPe6+6sOsY0lRLcrCqesPWeUpKktTEwpAkNbEwJElNLAxNjJ1bFrqOIM00C0MTw7uhpG5ZGJKkJhaGJKmJhSFJamJhSJKaWBiSpCYWhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlq0mlhJHltkjuSfC/J0Bd2DMZtS3J3kiNJdo0yoyRpSddHGLcDvwrctNKAJJuA9wAvB84FLkly7mjiSZIe0+k7vavqLoAkJxp2HnCkqu4ZjL0e2A7cueEBJUnf1/URRoszgPuWzR8dLHuCJDuS9JP0FxcXRxJOkmbFhh9hJPkM8Nwhqy6vqk+u559VVXuAPQC9Xq/Wc9uSNOs2vDCq6oI1buJ+4Kxl82cOlkmSRmgSTkndAiwkOTvJacDFwN6OM0nSzOn6ttpXJzkKvAzYl+SGwfLnJdkPUFXHgEuBG4C7gI9U1R1dZZakWdX1XVKfAD4xZPl/AK9YNr8f2D/CaJKk40zCKSlJ0hiwMCRJTSwMSVITC2MFuw8c6jqCJI0VC2MFV994uOsIkjRWLAxJUhMLQ5LUxMKQJDXp9It742L3gUNDr1nM79r3uPmdWxa4bOs5o4olSWMlVdP5UNder1f9fn/Vv39+1z7uverCdUwkSeMvycGqGvoGVE9JSZKaWBiSpCYWhiSpiYWxgp1bFrqOIEljxcJYgXdDSdLjWRiSpCYWhiSpiYUhSWoytV/cS7IIfHUNm9gMfH2d4nRpWvYD3JdxNS37Mi37AWvbl+dX1dywFVNbGGuVpL/Stx0nybTsB7gv42pa9mVa9gM2bl88JSVJamJhSJKaWBgr29N1gHUyLfsB7su4mpZ9mZb9gA3aF69hSJKaeIQhSWpiYUiSmlgYK0jytiRfSnJbkk8neV7XmVYryZ8l+cpgfz6R5FldZ1qtJK9NckeS7yWZuFsgk2xLcneSI0l2dZ1nLZJck+TBJLd3nWUtkpyV5HNJ7hz829rZdabVSvK0JP+W5IuDffmTdd2+1zCGS/JDVfWtwfSbgXOr6o0dx1qVJL8MfLaqjiV5B0BVvaXjWKuS5AXA94C/BP6wqlb/WsURS7IJOARsBY4CtwCXVNWdnQZbpSS/CDwCfLCqfrrrPKuV5HTg9Kq6NckPAgeBV03i30uSAM+oqkeSnAr8M7Czqm5ej+17hLGCx8pi4BnAxDZrVX26qo4NZm8Gzuwyz1pU1V1VdXfXOVbpPOBIVd1TVY8C1wPbO860alV1E/Bw1znWqqoeqKpbB9P/DdwFnNFtqtWpJY8MZk8dfNbtZ5eFcQJJ3p7kPuDXgbd2nWed/Dbwqa5DzKgzgPuWzR9lQn8wTask88BLgS90HGXVkmxKchvwIHCgqtZtX2a6MJJ8JsntQz7bAarq8qo6C7gOuLTbtCd2sn0ZjLkcOMbS/oytln2R1luSZwIfA37/uDMME6WqvltVL2HpTMJ5SdbtdOEp67WhSVRVFzQOvQ7YD1yxgXHW5GT7kuT1wCuBLTXmF66exN/LpLkfOGvZ/JmDZerY4Hz/x4DrqurjXedZD1X1jSSfA7YB63JjwkwfYZxIkuXvaN0OfKWrLGuVZBvwR8BFVfXtrvPMsFuAhSRnJzkNuBjY23GmmTe4UPx+4K6q+vOu86xFkrnH7oJM8nSWbrBYt59d3iW1giQfA36SpTtyvgq8saom8v8GkxwBngo8NFh08wTf8fVq4F3AHPAN4Laq+pVOQz0JSV4BvBPYBFxTVW/vNtHqJfkQcD5Lj9L+GnBFVb2/01CrkOQXgM8DX2bpv3eAP66q/d2lWp0kLwKuZenf11OAj1TVleu2fQtDktTCU1KSpCYWhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlqYmFIIzR478LWwfSfJnlX15mkVjP9LCmpA1cAVyb5MZaeinpRx3mkZn7TWxqxJP8EPBM4f/D+BWkieEpKGqEkLwROBx61LDRpLAxpRAavAr2OpacfPzJ4irA0MSwMaQSS/ADwceAPquou4G2M8ftVpGG8hiFJauIRhiSpiYUhSWpiYUiSmlgYkqQmFoYkqYmFIUlqYmFIkpr8H27NCzOyDWMIAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "\n",
        "import numpy as np# Define training set\n",
        "import matplotlib.pyplot as plt\n",
        "import scipy\n",
        "X = np.array([-3, -1, 0, 1, 3]).reshape(-1,1) # 5x1 vector, N=5, D=1\n",
        "y = np.array([-1.2, -0.7, 0.14, 0.67, 1.67]).reshape(-1,1) # 5x1 vector\n",
        "\n",
        "# Plot the training set\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+', markersize=10)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKodlxppjmrz"
      },
      "source": [
        "## 1. Maximum Likelihood\n",
        "We will start with maximum likelihood estimation of the parameters $\\boldsymbol\\theta$. In maximum likelihood estimation, we find the parameters $\\boldsymbol\\theta^{\\mathrm{ML}}$ that maximize the likelihood\n",
        "$$\n",
        "p(\\mathcal Y | \\mathcal X, \\boldsymbol\\theta) = \\prod_{n=1}^N p(y_n | \\boldsymbol x_n, \\boldsymbol\\theta)\\,.\n",
        "$$\n",
        "From the lecture we know that the maximum likelihood estimator is given by\n",
        "$$\n",
        "\\boldsymbol\\theta^{\\text{ML}} = (\\boldsymbol X^T\\boldsymbol X)^{-1}\\boldsymbol X^T\\boldsymbol y\\in\\mathbb{R}^D\\,,\n",
        "$$\n",
        "where \n",
        "$$\n",
        "\\boldsymbol X = [\\boldsymbol x_1, \\ldots, \\boldsymbol x_N]^T\\in\\mathbb{R}^{N\\times D}\\,,\\quad \\boldsymbol y = [y_1, \\ldots, y_N]^T \\in\\mathbb{R}^N\\,.\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e3jqv8GZjmrz"
      },
      "source": [
        "Let us compute the maximum likelihood estimate for a given training set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "zu_SP1Cajmr0"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def max_lik_estimate(X, y):\n",
        "    \n",
        "    # X: N x D matrix of training inputs\n",
        "    # y: N x 1 vector of training targets/observations\n",
        "    # returns: maximum likelihood parameters (D x 1)\n",
        "    \n",
        "    N, D = X.shape\n",
        "\n",
        "    # theta_ml = np.zeros((D,1)) ## <-- EDIT THIS LINE\n",
        "    # theta_ml = np.linalg.inv(X.T@X)@X.T@y\n",
        "    XX = X.T@X\n",
        "    Xy = X.T @ y\n",
        "    theta_ml = scipy.linalg.solve(XX, Xy)\n",
        "    return theta_ml"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "Y__J7q5ijmr1"
      },
      "outputs": [],
      "source": [
        "# get maximum likelihood estimate\n",
        "theta_ml = max_lik_estimate(X,y)\n",
        "\n",
        "#@ test\n",
        "assert np.any(theta_ml[0]), \"max_lik_estimate is not implemented\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nx87bQTyjmr3"
      },
      "source": [
        "Now, make a prediction using the maximum likelihood estimate that we just found"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "X_5u_Wn2jmr4"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def predict_with_estimate(Xtest, theta):\n",
        "    \n",
        "    # Xtest: K x D matrix of test inputs\n",
        "    # theta: D x 1 vector of parameters\n",
        "    # returns: prediction of f(Xtest); K x 1 vector\n",
        "    \n",
        "    prediction = None ## <-- EDIT THIS LINE\n",
        "    prediction = Xtest@theta\n",
        "\n",
        "    \n",
        "    return prediction "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PbpLwNjbjmr5"
      },
      "source": [
        "Now, let's see whether we got something useful:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "Tfk8gek1jmr6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "37dea3af-31e2-4c8c-f87b-5bff48d2c333"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbsUlEQVR4nO3dd5hV1b3G8e+PoUlVioDAMBZQESw4AgoiUgSkxd77DdeIBEGDCBq7UYk0RQ2WGK/GkqiXMrQRpIiCDChKRxAFpHeQNjPr/rEnuUoGGeCcs84++/08D8/DOVPOezJxXvZee/+WOecQEZHoKuY7gIiI+KUiEBGJOBWBiEjEqQhERCJORSAiEnHFfQc4ElWqVHEZGRm+Y4iIhMrs2bM3OueqHvh8KIsgIyODnJwc3zFERELFzL4v7HmdGhIRiTgVgYhIxKkIREQiTkUgIhJxKgIRkYhTEYiIRJyKQEQk4lQEIiJh8NNmGNsX9myL+bdWEYiIJDPnYP5HMKwxzHoFvv8s5i8RyjuLRUQiYfsaGHMfLBoNNc6Gmz6C6g1j/jIqAhGRZOMcfPk/MP5ByNsLbR+Dpt0hLT6/slUEIiLJZPN3MKonfDcF6jSDLs9D5ZPj+pIqAhGRZJCfBzP/ApMeB0uDToOg0a1QLP5LuSoCERHf1i+EEXfD6hyo2y4ogYo1E/byKgIREV9y98H0wTDlWShVHi5/FRpeCWYJjaEiEBHxYfVsGNED1s+HMy6HSwdA2SpeoqgIREQSad9PMPkp+HwYlKsG174Dp13qNZKKQEQkUVZ8CiN7wObl0OgWuORxKF3RdyoVgYhI3O3ZBtkPw+y/wnEZcPNIOOki36n+TUUgIhJPi8fB6F6wcy2cfzdc3B9KlvGd6hdUBCIi8bBrI4y9H+b9E46vD9e8BbXO9Z2qUCoCEZFYcg7mfQBj+8Ce7dCyHzTvBcVL+k52UCoCEZFY2bYasnrDknFQ81zo8gJUq+871SGpCEREjlZ+Psz5G2T/EfL2wyVPQtPfQbE038mKREUgInI0Ni0LhsStmAYZF0KXoVDpJN+pDov3IjCz2sCbQDXAAcOdc0P8phIROYT8PJjxIkx6EtJKQOeh0OjmhI+HiAXvRQDkAvc65+aYWXlgtpllO+cW+A4mIlKodQtgRHf4cQ7U6wCdBkKFE3ynOmLei8A5twZYU/D3HWa2EKgJqAhEJLnk7oNpzwV/SleEK18P5gSF8Cjg57wXwc+ZWQZwDjCzkI91A7oBpKenJzaYiMiqnGBU9IaF0PAqaP8MlK3sO1VMJE0RmFk54APgHufc9gM/7pwbDgwHyMzMdAmOJyJRtW9XsA4w48Xg9M/170O9dr5TxVRSFIGZlSAogbedcx/6ziMiAsDyKTDq97BlBWTeDm0ehdIVfKeKOe9FYGYGvAYsdM4N9J1HRITdWyH7IZjzZnAp6K1ZkNHcd6q48V4EQDPgJuAbM/uq4Ll+zrkx/iKJSGQtGhPcHbxzHVzwe7i4H5Q4xnequPJeBM65T4FwL7mLSPjt3BDMB5r/IVRrANf+HWo28p0qIbwXgYiIV87B1+/DuPuDheGLH4Tm9wQ3iUWEikBEomvbqmCvgKUToNZ5wZC440/znSrhivkOICJypAZlLzmyL8zPh1mvwrCmwfaR7f4Et4+PZAmAikBEQmzIxKWH/0WblsHfOkPWvcFGMXd9DuffFZpJofGgU0MiEg15uTBjGHzyFKSVCk4DnXNj6MdDxIKKQERS39p5wZC4NV/BaZ3g0j9DhRq+UyUNFYGIpK7cvTB1AHw6CI45Dq56A+r/RkcBB1ARiEgoDMpeUuiaQEbfrF887tm6Lr3a1oOVXwRD4jYuhrOug3ZPQZlKiYobKuZc+Oa3ZWZmupycHN8xRMSzjL5ZrHi64y+f3LsTJj0BM1+GCjWh8xCo28ZPwCRjZrOdc5kHPq8jAhFJHcs+CYbEbf0BzvsttHkYSpX3nSrpqQhEJPx2b4EJD8KXb0HlU+C2sVDnAt+pQkNFICLhtnBUcE/Aro3QvDdcdD+UKO07VaioCEQknHasY0LN1+C9iVC9YbBhzAln+04VSioCEQkX52DuuzCuL/X274bWfwzGRUdoSFysqQhEJDy2/gCj7oFlE6F2k+Du4Kr1fKcKPRWBiCS//HzIeQ0+fiQ4IugwAM77LyimcWmxoCIQkeS2cSmM7AE/fA4nt4JOg+G4Or5TpRQVgYgkp7z98NnzMPnpYKvI37wU3CGs8RAxpyIQkeSzZm4wHmLt13B6Z7j0OShfzXeqlKUiEJHksX8PTH0WPh0MZSrD1W9C/a6+U6U8FYGIJIcfZgRHAZuWwlnXQ7snNSQuQVQEIuLX3p0w8TH4YjhUrA03fgintPadKlJUBCLiz7cTg/sCtq2EJv8NrR6CUuV8p4ocFYGIJN5Pm4MhcV+9DVXqBRvHpzfxnSqyVAQikljz/xfG3BdMDL3wPmjxBw2J80xFICKJsWNtUAALR0GNs4K1gBpn+k4lqAhEJN6cC04Bje8X7CHc5hE4vwek6ddPstBPQkTiZ8v3MKonLP8E0i+ALs9DlVN8p5IDqAhEJPby8+CLV4LLQs2g43Nw7u0aEpekVAQiElsbFgdD4lbOhFPaQqdBcGxt36nkVyRFEZjZ60AnYL1zroHvPCJyBPL2w/TBMOVZKFkWLhsOZ16tIXEhkBRFALwBvAC86TmHiByJH7+EET1g3TdwxmXBfgHlqvpOJUWUFEXgnJtqZhm+c4jIYdq/OxgT/dnzULYqXPM2nN7Jdyo5TElRBEVhZt2AbgDp6eme04gI338WDInbvAwa3QxtH4djjvWdSo5AaJbwnXPDnXOZzrnMqlV1yCnizZ7tMLo3/LUD5OfCzSOCy0JVAqEVmiMCEUkCS7ODIXHbV0PT7tCqf7AwLKGmIhCRQ9u1CcY/AF+/B1VPgzuyofZ5vlNJjCRFEZjZO0BLoIqZrQIeds695jeViOAczP8IxvwB9myFFn2gxX1QvJTvZBJDSVEEzrnrfGcQkQNsXwNZ98LiLDjhHOgyAqrrNp9UlBRFICJJxDmY8yZMeAjy9sIlT0CT32lIXArTT1ZE/t/m72DU7+G7qVCnOXQZCpVP9p1K4kxFICLBkLiZf4FJj4OlBfOBGt2qIXERoZ+ySIoYlL3kyL5w/UJ47ZLgqqCMC6H7TMjUpNAo0U9aJEUMmbj08L4gdx9MfgZevhA2L4fLX4Xr34OKNeMTUJKWTg2JRNHq2cGQuPXzocEV0OFZKFvFdyrxREUgEiX7foLJT8Hnw6BcdbjuXTi1g+9U4pmKQCQqvpsWXBG0eTk0ugUueRxKV/SdSpKAikAkhAZlLyl0TSCjb9YvHvdsXZdeF1aD7Idh9l/huBPhllFwYotERZUQMOec7wyHLTMz0+Xk5PiOIZJUMvpmseLpjr98cvE4GN0Ldq6FpnfBxf2hZBk/AcU7M5vtnMs88HkdEYikol0bYez9MO+fcHx9uOYtqHWu71SSpFQEIqnEOZj3AYztE+wb0PIBaN4bipf0nUySmIpAJEVUZxO8cy0sGQc1M6HrC3D86b5jSQioCETCLj8f5rzBlLIPwvJ8aPcUNLkTiqX5TiYhoSIQCbNNy2BUT1gxjVIntoDOQ6HSib5TScioCETCKD8vuCnskychrSR0HhLcG2DmO5mEkIpAJGzWLYAR3eHHOVCvA3QaCBVO8J1KQkxFIBIWuXth2kCY9hyUrgBXvg5nXK6jADlqKgKRMFiVAyPuhg0L4cxroN2foGxl36kkRagIRJLZvl0w6UmY8WJw+uf696FeO9+pJMWoCESS1fIpwZC4LSsg8w5o80hwSkgkxlQEIslm91bIfijYQL7SSXBrFmQ0951KUpiKQCSZLBoDWb1h5zpo1jMYEVHiGN+pJMWpCESSwc4NwXyg+R9CtQZw7d+hZiPfqSQitGexSAwd9gbyzsHc92DYebBoNLR6ELpNVglIQqkIRGLosDaQ37YK/n41fNQNKp8C/z0NWvwB0krEL6BIIXRqSCTR8vNh9uvBrmEuH9o/A41/qyFx4o2KQCSRNi2DkT3g++lwUstgRtBxGb5TScSpCEQSIS8XZgyDT56C4qWg6zA4+waNh5CkoCIQOUJF3UD+8aaOm9YNgDVfwWmdoONzUL56glKKHJo2rxeJoV9sIJ+7F6YOgE8HwTHHwaV/hvpddRQg3iT15vVm1h4YAqQBrzrnnvYcSeTorPwiGBK3cTGceS20/xOUqeQ7lUihvBeBmaUBw4C2wCpglpmNdM4t8JtM5PCVYQ+M7QszX4aKteCGD6BuG9+xRH7VIe8jMLNsMzsrjhkaA98655Y75/YB7wJd4/h6IvGxbBLjS94PM1+C8+6Auz5XCUgoFOWI4H5gsJmtAPo559bEOENNYOXPHq8Cmhz4SWbWDegGkJ6eHuMIIkdh9xaY8CB8+RZly6TDdWOhzgW+U4kU2SGPCJxzc5xzFwOjgXFm9rCZJXwKlnNuuHMu0zmXWbVq1US/vEjhFoyEYU3gq3egeS8q3TtLJSChU6QRE2ZmwGLgJaAHsNTMbopRhtVA7Z89rlXwnEjy2rEO3rsJ3r8Jyh0Pv50U7BdQorTvZCKH7ZCnhsxsOnAiMB+YAdwKLAJ6mtmFzrluR5lhFlDXzE4kKIBrgeuP8nuKxIdzMPcdGPcA7N8NrR4KxkVrPpCEWFHWCLoBC9x/3nDQw8wWHm0A51yumd0NjCe4fPR159z8o/2+IjG39QcYdQ8smwi1m0KX56FqPd+pRI7aIYvgEL+UO8YihHNuDDAmFt9LJOby82HWK/Dxo8HjDgPgvP+CYhreK6nhqO4jcM4tj1UQkaS0YUkwJG7lDDi5VTAk7lhdtSapxfsNZSJJKW8/fDYUJj8TLAB3fRHOvl7jISQlqQhEDrRmbjAeYu3XwWygDgOgfDXfqUTiRkUg8i/798CUZ2D6EChbBa7+H6jfxXcqkbhTEYgAfP95sBawaSmcfSO0eyKYGCoSASoCiba9O4KrgWa9AhXT4aaPgkVhkQhREUh0Lf0YRvWE7auhyZ3BzWGlyvlOJZJwKgKJnp82w/h+wR3CVerB7eMh/T/mHIpEhopAosM5WDACxtwXTAxt8YfgT/FSvpOJeKUikGjYsRay7oVFo6HG2cFaQPWGvlOJJAUVgaQ25+Crt4NTQbl7oc2jcP7dkKb/64v8i/5rkNS1ZUWwGLx8MtRpBp2HQpVTfKcSSToqAkk9+XnwxXCY+BhYGnQcCOfepiFxIgehIpDUsmFxMB5i1RdwSlvoPDjYRF5EDkpFIKkhbz9MHwxTnoWS5eDyV6DhVRoSJ1IEKgIJvx+/DI4C1s2DMy4LhsSV077WIkWlIpDw2r8bJj8Nnz0PZavCNW/D6Z18pxIJHa2eSTitmA4vNQtOB51zA3Sf+YsSGJS9xF82kZBREUi47NkOo3vDG5dCfi7cPCLYO/iYY3/xaUMmLvWTTySEdGpIwmPJBBjdKxgS17Q7tOoPJcv6TiUSeioCSX67NsH4B+Dr96DqaXBHNtQ+z3cqkZShIpDk5RzM/xDG9IE9W6FFH2hxn4bEicSYikCS0/Y1kNUbFo+BE86BriOh2hmFfuqg7CWFrglk9M36xeOerevSq229uMQVCTNzzvnOcNgyMzNdTk6O7xgSD87BnDdhwkOQtxdaPQhNfnfYQ+Iy+max4umOcQopEk5mNts5l3ng8zoikOSxeXkwJO67qVCnOXQZCpVP9p1KJOWpCMS//DyY8RJMegKKFYdOg6HRLRoSJ5IgKgLxa/3CYDzE6hyo2w46DYKKNX2nEokUFYH4kbsPPh0EUwdA6QpwxWvQ4IqYDYnr2bpuTL6PSBSoCCTxVs8OjgLWLwgmhLZ/GspWielL6OogkaJTEUji7PsJPnkSZrwI5arDde/CqR18pxKJPBWBJMZ302BkD9jyXbBbWNtHoXRF36lEBM9D58zsKjObb2b5ZvYf17ZKCtizLbgk9G8Fk0FvGR3sGqYSEEkavo8I5gGXA3/xnEPiYfG4YEjczrVwQQ9o2Q9KlvGdSkQO4LUInHMLAUzbCaaWXRth7P0w759wfH249i2oea7vVCJyEL6PCIrMzLoB3QDS09M9p5FCOQfzPoCxfYJ9A1r2g+a9oHhJ38lE5FfEvQjM7GOgeiEf6u+cG1HU7+OcGw4Mh2DWUIziSaxsWx0MiVsyLvjXf9dhcPzpvlOJSBHEvQicc23i/RriUX4+zHkDJvwx2DGs3VPQ5E4oluY7mYgUkYa5REDc9u/dtAze7BIsCNc8B+76DM7vrhIQCRnfl49eZmargPOBLDMb7zNPqor5/r15uTB9KLx0Aaz5Otgz+OaRUOmk2L6OiCSE76uGPgI+8plBDtO6+cF4iB/nwKmXQseBUKGG71QichRCc9WQeJa7F6YNhGnPBTeDXfk6nHF5zIbEiYg/KgI5tFU5wVHAhoXQ8OqCIXGVfacSkRhREaSYmO7fu28XTCoYElfhBLj+H1DvkljGFZEkoD2LI+CI9u9dPgVG/R62rIDMO6DNI8G+ASISWtqzWIpm91bIfijYQL7SyXDrGMho5juViMSRikD+36IsGN0bdq2HZj2h5QNQ4hjfqUQkzlQEAjs3BPOB5n8I1RrAde9AzUa+U4lIgqgIIuCg+/c6B1+/D+PuDxaGWz0Ize6BtBIJzScifqkIIqDQq4O2rgxGQ3ybDbUaQ9cXoOqpiQ8nIt6pCKImPx9yXoOPHwGXH9wT0Lib5gOJRJiKIEo2fhvsG/zDZ3BSS+g8FI6r4zuViHimIoiCvFz4/Hn45E9QojR0eQHOuVHjIUQEUBGkvrXfwIjusGYunNYJOj4H5QvbJ0hEokpFkKr274GpA2D6YDjmOLjqDTjjMt+pRCQJqQhS0Q8zg7WAjYvhrOuh3ZNQppLvVCKSpFQEqWTvTpj0OMz8C1SsBTd8AHW1U6iI/DoVQar4diKMuge2rYTGv4XWf4RS5X2nEpEQUBGE3e4tML4/fPU2VK4Lt42FOuf7TiUiIaIiCLMFI2HMfbBrI1x4L7ToE1weKiJyGFQEYbRjXVAAC0dC9YZwwz+gxlm+U4lISKkIwsQ5mPsOjHsA9u+G1g/DBT00JE5EjoqKICy2fA+j74Flk6B202BIXJWDTBUVETkMKoJkl58Ps14NhsSZwaV/DraOLFbMdzIRSREqgmS2YUlwY9jKGXBya+g8GI5N951KRFKMiiAZ5e2H6UNgyjNQogz85mU461oNiRORuFARJJs1c4MhcWu/gfpdocMAKF/NdyoRSWEqgmSxf09wBDB9CJStAte8Bad39p1KRCJARZAMvv8cRt4Nm76Fs2+Edk8EE0NFRBJAReDT3h0w8TH44hU4tjbc9BGc3Mp3KhGJGBWBL0s/Du4L2LYKmtwJrR6EUuV8pxKRCPJaBGY2AOgM7AOWAbc557b6zBR3P22G8f2CO4SrnAp3TIDajX2nEpEI831XUjbQwDl3JrAEeMBznvhxDub/LwxrDN/8A1r8Ae6cphIQEe+8HhE45yb87OEM4EpfWeJqx1rIuhcWjYYaZwdrAdUb+k4lIgIk1xrB7cB7B/ugmXUDugGkp4fk7lrn4Mu3gv0C8vZC28egaXdIS6b/2UUk6uL+G8nMPgaqF/Kh/s65EQWf0x/IBd4+2Pdxzg0HhgNkZma6OESNrS0rYFRPWD4Z0i+ALs9DlVN8pxIR+Q9xLwLn3K9ummtmtwKdgNbOueT/BX8o+XnwxfDgslBLg44D4dzbNCRORJKW76uG2gN9gIuccz/5zBIT6xcFN4atmgWntA2GxFWs5TuViMiv8n2y+gWgFJBtwUC1Gc65O/1GOgK5+2D6YJg6AEqWg8tfgYZXaUiciISC76uGwn/SfPWcYFT0unlwxuXQ4VkoV9V3KhGRIvN9RBBe+3fDJ0/B5y9A2ePh2r/DaR19pxIROWwqgiOx4tPgKGDzcmh0M7R9HI451ncqEZEjoiI4HHu2w8cPQ87rcGwduHkEnNTSdyoRkaOiIiiqJRNgdC/Y8SOcfzdc3A9KlvWdSkTkqKkIDmXXJhj/AHz9HlQ9Da7OhlqZvlOJiMSMiuBgnIP5H8KYPrBnG1zUFy7sDcVL+U4mIhJTKoLCbF8DWb1h8Rg44RzoOhKqneE7lYhIXKgIfs45mPMmTHgoGBJ3yRPQ5HcaEiciKU2/4f5l8/JgSNx3U6FOc+gyFCqf7DuViEjcqQjy82DGSzDpCShWHDoNhka3aEiciERGtItg3YJgSNzq2VCvfTAptGJN36lERBIqmkWQuw8+HQhT/wylK8AVr0GDKzQkTkQiKXpFsGp2cBSwfgE0uBI6PANlq/hOJSLiTbSKYMoAmPwUlKsO170Hp7b3nUhExLtoFUGlE4OF4LaPQumKvtOIiCSFaBVBwyuDPyIi8m+6RlJEJOJUBCIiEaciEBGJOBWBiEjEqQhERCJORSAiEnEqAhGRiFMRiIhEnDnnfGc4bGa2Afjed44jUAXY6DtEAkXt/YLec1SE9T3Xcc5VPfDJUBZBWJlZjnMu03eORIna+wW956hItfesU0MiIhGnIhARiTgVQWIN9x0gwaL2fkHvOSpS6j1rjUBEJOJ0RCAiEnEqAhGRiFMReGBm95qZM7OU3yzZzAaY2SIz+9rMPjKzY31nihcza29mi83sWzPr6ztPvJlZbTP7xMwWmNl8M+vpO1MimFmamX1pZqN9Z4kVFUGCmVlt4BLgB99ZEiQbaOCcOxNYAjzgOU9cmFkaMAzoANQHrjOz+n5TxV0ucK9zrj7QFOgegfcM0BNY6DtELKkIEm8Q0AeIxCq9c26Ccy634OEMoJbPPHHUGPjWObfcObcPeBfo6jlTXDnn1jjn5hT8fQfBL8eaflPFl5nVAjoCr/rOEksqggQys67AaufcXN9ZPLkdGOs7RJzUBFb+7PEqUvyX4s+ZWQZwDjDTc5R4G0zwD7l8zzliKlqb1yeAmX0MVC/kQ/2BfgSnhVLKr71n59yIgs/pT3Aq4e1EZpP4M7NywAfAPc657b7zxIuZdQLWO+dmm1lLz3FiSkUQY865NoU9b2YNgROBuWYGwSmSOWbW2Dm3NoERY+5g7/lfzOxWoBPQ2qXujSurgdo/e1yr4LmUZmYlCErgbefch77zxFkzoIuZXQqUBiqY2VvOuRs95zpquqHMEzNbAWQ658I4wbDIzKw9MBC4yDm3wXeeeDGz4gSL4a0JCmAWcL1zbr7XYHFkwb9o/gZsds7d4zlOQhUcEdznnOvkOUpMaI1A4u0FoDyQbWZfmdnLvgPFQ8GC+N3AeIJF0/dTuQQKNANuAloV/Gy/KvjXsoSMjghERCJORwQiIhGnIhARiTgVgYhIxKkIREQiTkUgIhJxKgIRkYhTEYiIRJyKQCQGCubyty34+xNm9rzvTCJFpVlDIrHxMPCYmR1PMIWzi+c8IkWmO4tFYsTMpgDlgJYF8/lFQkGnhkRioGC6bA1gn0pAwkZFIHKUzKwGwT4LXYGdBRNXRUJDRSByFMysDPAhwd69C4HHCdYLREJDawQiIhGnIwIRkYhTEYiIRJyKQEQk4lQEIiIRpyIQEYk4FYGISMSpCEREIu7/APq+h48oBC2UAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# define a test set\n",
        "Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs\n",
        "\n",
        "# predict the function values at the test points using the maximum likelihood estimator\n",
        "ml_prediction = predict_with_estimate(Xtest, theta_ml)\n",
        "\n",
        "#@ test\n",
        "assert not np.array_equal(ml_prediction, Xtest), \"predict_with_estimate not implemented\"\n",
        "\n",
        "# plot\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+', markersize=10)\n",
        "plt.plot(Xtest, ml_prediction)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Axn856Lijmr6"
      },
      "source": [
        "#### Questions\n",
        "1. Does the solution above look reasonable?\n",
        "2. Play around with different values of $\\theta$. How do the corresponding functions change?\n",
        "3. Modify the training targets $\\mathcal Y$ and re-run your computation. What changes?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1kVL8Bdjmr8"
      },
      "source": [
        "Let us now look at a different training set, where we add 2.0 to every $y$-value, and compute the maximum likelihood estimate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "oQwhII6cjmr8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "66af22e5-1c3e-49fe-982a-ff81ee862f98"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPtElEQVR4nO3df6zddX3H8edr0CkTN5L1LnZQvUtWlk0n4E5Q47Iwu24oBrZME8yiw21pNP6oxGU2mkDEmWCWWBESWTOcsBCnEWaYhcU7JFOWgJ42BYFiaQwGCBlXGGCj01Tf++N+jLe39/bH7T3f7z33PB/JSc73fD/3nNc3be+r5/v9fL/fVBWSJP1C3wEkSauDhSBJAiwESVJjIUiSAAtBktSc2neA5Vq/fn1NT0/3HUOSxsru3bu/V1VTi60b20KYnp5mOBz2HUOSxkqS7y61zl1GkiTAQpAkNRaCJAmwECRp7OyY2T+S97UQJGnMXHPnIyN5XwtBkgRYCJKkxkKQJAFjfGKaJE2CHTP7Fz1mML1912HL2zZv4vItZ5/UZ2Vcb5AzGAzKM5UlTaLp7bt49OqLlvWzSXZX1WCxde4ykiQBFoIkqbEQJEmAhSBJY2fb5k0jeV8LQZLGzMnOJlqKhSBJAiwESVIz8kJI8sIk30hyX5IHk3xkkTGXJZlNsrc9/mbUuSRJh+viTOUfAa+vqoNJ1gF3J7mjqu5ZMO7zVfWeDvJIkhYx8kKouVOhD7bFde0xnqdHS9Ia1skxhCSnJNkLPAXMVNW9iwz78yT3J/liko1LvM/WJMMkw9nZ2VFGlqSJ00khVNVPqupc4Czg/CSvWDDk34HpqnolMAPcuMT77KyqQVUNpqamRppZkiZNp7OMqupZ4C7gwgWvP11VP2qL/wT8Xpe5JEndzDKaSnJGe34asAV4eMGYDfMWLwb2jTqXJOlwXcwy2gDcmOQU5groC1X15SRXAcOqug14X5KLgUPAM8BlHeSSJM3j/RAkaYJ4PwRJ0jFZCJIkwEKQJDUWgiQJsBAkSY2FIEkCLARJUmMhSJIAC0GS1FgIkiTAQpAkNRaCJAmwECRJjYUgSQIsBElSYyFIkgALQZLUWAiSJMBCkCQ1FoIkCbAQJEmNhSBJAiwESVJjIUiSAAtBktSMvBCSvDDJN5Lcl+TBJB9ZZMwLknw+yYEk9yaZHnUuSdLhuviG8CPg9VV1DnAucGGS1ywY89fA/1bVbwI7gI93kEuSNM/IC6HmHGyL69qjFgy7BLixPf8isDlJRp1NkvRznRxDSHJKkr3AU8BMVd27YMiZwGMAVXUIeA741UXeZ2uSYZLh7OzsiFNL0mTppBCq6idVdS5wFnB+klcs8312VtWgqgZTU1MrmlGSJl2ns4yq6lngLuDCBaueADYCJDkV+BXg6S6zSdKk62KW0VSSM9rz04AtwMMLht0G/GV7/mbgq1W18DiDJGmETu3gMzYANyY5hbkC+kJVfTnJVcCwqm4DbgD+JckB4Bng0g5ySZLmGXkhVNX9wHmLvH7FvOf/B7xl1FkkSUvzTGVJEmAhSJIaC0GSBFgIkqTGQpAkARaCJKmxECRJgIUgSWosBEkSYCFIkhoLQZIEWAiSpMZCkLSkHTP7+46gDlkIkpZ0zZ2P9B1BHbIQJEmAhSBJaiwESRLQzS00JY2BHTP7Fz1mML1912HL2zZv4vItZ3cVSx3KuN7LfjAY1HA47DuGtKZNb9/Fo1df1HcMraAku6tqsNg6dxlJkgALQZLUWAiSJMBCkHQU2zZv6juCOmQhSFqSs4kmi4UgSQIsBElSM/JCSLIxyV1JHkryYJJti4y5IMlzSfa2xxWjziVJOlwXZyofAj5QVXuSvBjYnWSmqh5aMO7rVfWmDvJIkhYx8m8IVfVkVe1pz78P7APOHPXnSpJOTKfHEJJMA+cB9y6y+rVJ7ktyR5KXL/HzW5MMkwxnZ2dHGVWSJk5nhZDkdOAW4P1V9fyC1XuAl1XVOcC1wJcWe4+q2llVg6oaTE1NjTSvJE2aTgohyTrmyuDmqrp14fqqer6qDrbntwPrkqzvIpskaU4Xs4wC3ADsq6pPLDHmJW0cSc5vuZ4edTZJ0s91McvodcDbgG8l2dte+xDwUoCquh54M/CuJIeAHwKX1rhel1uSxtTIC6Gq7gZyjDHXAdeNOoskaWmeqSxJAiwESVJjIUiSAAtBktRYCJIkwEKQJDUWgiQJsBAkSY2FIEkCLARJUmMhSJKA4yiEJDNJzukijCSpP8fzDeGDwCeT/HOSDaMOJEnqxzELoar2VNUfAl8G/iPJlUlOG300SVKXjusYQrt5zbeBTwPvBR5J8rZRBpPG2Y6Z/X1HkE7Y8RxD+G/gCWAHcCZwGXABcH6SnaMMJ42ra+58pO8I0gk7nhvkbAUeWuQOZu9Nsm8EmSRJPThmIVTVg0dZfdEKZpEk9eikzkOoqu+sVBBJUr9Gfk9laa3bMbN/0WMG09t3Hba8bfMmLt9ydlexpBOWIw8NjIfBYFDD4bDvGNKiprfv4tGr3aOq1SfJ7qoaLLbOS1dIkgALQZLUWAiSJMBCkEZi2+ZNfUeQTtjICyHJxiR3JXkoyYNJti0yJkk+leRAkvuTvGrUuaRRcjaRxlEX004PAR+oqj1JXgzsTjJTVQ/NG/MGYFN7vJq5aya9uoNskqRm5N8QqurJqtrTnn8f2MfcNZHmuwS4qebcA5zhpbYlqVudHkNIMg2cB9y7YNWZwGPzlh/nyNIgydYkwyTD2dnZkeWUpEnUWSEkOR24BXh/VT2/nPeoqp1VNaiqwdTU1MoGlKQJ10khJFnHXBncXFW3LjLkCWDjvOWz2muSpI50McsowA3Avqr6xBLDbgPe3mYbvQZ4rqqeHHU2SdLPdTHL6HXA24BvJdnbXvsQ8FKAqroeuB14I3AA+AHwjg5ySZLmGXkhVNXdQI4xpoB3jzqLJGlpnqksSQIsBElSYyFIkgALQZLUWAiSJMBCkCQ1FoIkCbAQJEmNhSBJAiwESVJjIUiSAAtBktRYCJIkwEKQJDUWgiQJsBAkSY2FIEkCLARJUmMhSJIAC0GS1FgIkiTAQpAkNRaCJAmwECRJjYUgSQIsBElSM/JCSPKZJE8leWCJ9RckeS7J3va4YtSZJElHOrWDz/gscB1w01HGfL2q3tRBFknSEkb+DaGqvgY8M+rPkSSdnNVyDOG1Se5LckeSly81KMnWJMMkw9nZ2S7zSdKatxoKYQ/wsqo6B7gW+NJSA6tqZ1UNqmowNTXVVT5Jmgi9F0JVPV9VB9vz24F1Sdb3HEuSJk7vhZDkJUnSnp/PXKan+00lSZNn5LOMknwOuABYn+Rx4EpgHUBVXQ+8GXhXkkPAD4FLq6pGnUuSdLiRF0JVvfUY669jblqqJKlHve8ykiStDhaCJAmwELSK7JjZ33cEaaJZCFo1rrnzkb4jSBPNQpAkARaCJKmxECRJQDeXv5aOsGNm/6LHDKa37zpsedvmTVy+5eyuYkkTLeN6UvBgMKjhcNh3DK2g6e27ePTqi/qOIa1pSXZX1WCxde4ykiQBFoIkqbEQJEmAhaBVZNvmTX1HkCaahaBVw9lEUr8sBEkSYCFIkhoLQZIEWAiSpMZCkCQBFoIkqbEQJEmAhSBJaiwESRJgIUiSGgtBkgR0UAhJPpPkqSQPLLE+ST6V5ECS+5O8atSZJElH6uIbwmeBC4+y/g3ApvbYCny6g0ySpAVGXghV9TXgmaMMuQS4qebcA5yRZMOoc0mSDrcajiGcCTw2b/nx9toRkmxNMkwynJ2d7SScJE2K1VAIx62qdlbVoKoGU1NTfceRpDVlNRTCE8DGectntdckSR1aDYVwG/D2NtvoNcBzVfVk36EkadKcOuoPSPI54AJgfZLHgSuBdQBVdT1wO/BG4ADwA+Ado84kSTrSyAuhqt56jPUFvHvUOSRJR7cadhlJklYBC0GSBFgIkqRmIgthx8z+viNI0qozkYVwzZ2P9B1BkladiSwESdKRLARJEmAhSJKakZ+Y1rcdM/sXPWYwvX3XYcvbNm/i8i1ndxVLkladzJ0oPH4Gg0ENh8Nl/ez09l08evVFK5xIkla/JLurarDYOncZSZIAC0GS1FgIkiRgQgth2+ZNfUeQpFVnIgvB2USSdKSJLARJ0pEsBEkSYCFIkpqxPTEtySzw3WX++HrgeysYp09uy+q0VrZlrWwHuC0/87KqmlpsxdgWwslIMlzqTL1x47asTmtlW9bKdoDbcjzcZSRJAiwESVIzqYWws+8AK8htWZ3Wyrasle0At+WYJvIYgiTpSJP6DUGStICFIEkCJrgQknw0yf1J9ib5SpJf7zvTciX5hyQPt+35tyRn9J1puZK8JcmDSX6aZOymCCa5MMm3kxxIsr3vPMuV5DNJnkryQN9ZTlaSjUnuSvJQ+7u1re9My5HkhUm+keS+th0fWfHPmNRjCEl+uaqeb8/fB/xOVb2z51jLkuSPga9W1aEkHweoqg/2HGtZkvw28FPgH4G/rarl3RavB0lOAfYDW4DHgW8Cb62qh3oNtgxJ/gA4CNxUVa/oO8/JSLIB2FBVe5K8GNgN/Om4/bkkCfCiqjqYZB1wN7Ctqu5Zqc+Y2G8IPyuD5kXA2DZjVX2lqg61xXuAs/rMczKqal9VfbvvHMt0PnCgqr5TVT8G/hW4pOdMy1JVXwOe6TvHSqiqJ6tqT3v+fWAfcGa/qU5czTnYFte1x4r+3prYQgBI8rEkjwF/AVzRd54V8lfAHX2HmFBnAo/NW36cMfzFs5YlmQbOA+7tOcqyJDklyV7gKWCmqlZ0O9Z0IST5zyQPLPK4BKCqPlxVG4Gbgff0m/bojrUtbcyHgUPMbc+qdTzbIq20JKcDtwDvX7CHYGxU1U+q6lzm9gKcn2RFd+edupJvttpU1R8d59CbgduBK0cY56Qca1uSXAa8Cdhcq/zA0An8uYybJ4CN85bPaq+pZ22f+y3AzVV1a995TlZVPZvkLuBCYMUO/K/pbwhHk2T+fTQvAR7uK8vJSnIh8HfAxVX1g77zTLBvApuS/EaSXwQuBW7rOdPEawdjbwD2VdUn+s6zXEmmfjaDMMlpzE1eWNHfW5M8y+gW4LeYm9HyXeCdVTWW/5tLcgB4AfB0e+meMZ4x9WfAtcAU8Cywt6r+pNdQJyDJG4FPAqcAn6mqj/WbaHmSfA64gLnLLP8PcGVV3dBrqGVK8vvA14FvMffvHeBDVXV7f6lOXJJXAjcy93frF4AvVNVVK/oZk1oIkqTDTewuI0nS4SwESRJgIUiSGgtBkgRYCJKkxkKQJAEWgiSpsRCkFdKuub+lPf/7JNf2nUk6EWv6WkZSx64Erkrya8xdUfPinvNIJ8QzlaUVlOS/gNOBC9q196Wx4S4jaYUk+V1gA/Bjy0DjyEKQVkC7TePNzF0592C7Aq00ViwE6SQl+SXgVuADVbUP+Cir+N4a0lI8hiBJAvyGIElqLARJEmAhSJIaC0GSBFgIkqTGQpAkARaCJKn5fy1ZQQrmcjEPAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "ynew = y + 2.0\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(X, ynew, '+', markersize=10)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "SPqTm57mjmr8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        },
        "outputId": "eeed1a0c-f10d-4b62-b7f5-453d4ad22e25"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.499]]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAa50lEQVR4nO3deZTU1bnu8e8LNDOIQDPTNCIqCALaIoLIJE4gRJSrYjzxmhM85mgwmihOMYooigooRsUhYsQYYzTmiBkQmklkaGaZBxERZBCQGXrY94/dnoumkQaqalfV7/msxVpd1W33U4r19G949zbnHCIiEl1lQgcQEZGwVAQiIhGnIhARiTgVgYhIxKkIREQirlzoAMejdu3aLjs7O3QMEZGUMnfu3G3OuczvP5+SRZCdnU1eXl7oGCIiKcXMPi/peZ0aEhGJOBWBiEjEqQhERCJORSAiKWvEhJWhI6QFFYGIpKxRE1eFjpAWVAQiIhGnIhARiTgVgYhIxKXkQJmIRM+ICStLvCaQPXj8dx4P6tGcX/Y8LVGx0oKl4sY0OTk5TpPFIpI9eDzrhvUKHSNlmNlc51zO95/XqSERkYhTEYiIRJyKQEQk4lQEIpKyBvVoHjpCWlARiEjK0t1BsaEiEBGJOBWBiEjEqQhERCJORSAiEnEqAhGRiFMRiIhEnIpARCTiVAQiIhGnIhARiTgVgYhIxKkIREQiTkUgIhJxwYvAzCqa2WwzW2hmS8zsodCZRESiJBn2LD4IdHfO7TGzDGC6mf3dOTczdDARkSgIXgTOb5q8p/hhRvGf1NtIWUQkRQU/NQRgZmXNbAGwBZjgnJtVwtcMNLM8M8vbunVrwjOKiKSrpCgC51yhc64t0Ahob2atSviaMc65HOdcTmZmZsIzioikq6Qogm8553YCucClgaOIiERG8CIws0wzq1H8cSWgJ7A8aCgRkQgJfrEYqA+MNbOy+GJ62zn3QeBMIiKREbwInHOLgHahc4ikuhETVmozdzkuwU8NiUhsjJq4KnQESVEqAhGRiFMRiIhEnIpARCTigl8sFpFjN2LCyhKvCWQPHv+dx4N6NNcFZDkq80v9pJacnByXl5cXOoZIUskePJ51w3qFjiFJzMzmOudyvv+8Tg2JiEScikBEJOJUBCIiqaKoMC7fVkUgkiYG9WgeOoLES2EBzB0Lo9rClmUx//a6a0gkTejuoDRUVARL34NJQ2H7GmiYA4WHYv5jVAQiIsnGOVg1ASY9DF8thswWcO2bcPrlYBbzH6ciEBFJJp/PgIkPw/pPoEYTuHIMtL4aypSN249UEYiIJINNC2HiEFg9AarWg15PQbv/gHLl4/6jVQQiIiFtWw25j8CS96BiDbjot9D+ZihfOWERVAQiIiF8swEmD4MFb0K5inDhr6HjbVDxpIRHURGIxJA2h5Gj2rsNpj0Fc172j9sPhM53QtXMYJFUBCIxNGriKhWBlOzALvhkNHzyHOTvg7YDoMvdUCMrdDIVgYhIXOXvh9kvwfSnYf8OaNkXut0PmcnzC4OKQEQkHgrzYf4bMOUJ2L0RTr0Iut8PDZJvi3YVgYhILBUVwad/gdyhsOMzaHweXPUSZF8QOtkRqQhEjpM2h5HvcA5W/sPPAmxZAnVbwYC3ofnFcZkGjiVtTCMSQ9ocJqI+m+angTfMhpqnQLf74Mx+UCa51vU80sY0wY8IzKwx8DpQF3DAGOfcqLCpRERKYeN8XwBrJkG1BtB7JLT7MZTNCJ3smAQvAqAAuNM5N8/MqgFzzWyCc25p6GAiIiXaugImPQLL/gaVasLFj8C5/wkZlUInOy7Bi8A5twnYVPzxbjNbBjQEVAQiklx2rofJj8PCNyGjMnQZDOf/N1SsHjrZCQleBIczs2ygHTCrhM8NBAYCZGWFH8AQKYk2h0lTe7b4aeC8VwGD826BzndAldqhk8VE0lwsNrOqwBRgqHPu3R/6Wl0sFpGE2L8TZjwDM5+HgoP+/H+Xu+GkhqGTHZekvVgMYGYZwF+AcUcrARGRuDu0D2a/CNNHwoGd0Ooq6Hov1D41dLK4CF4EZmbAK8Ay59zTofOISIQVHIJ5Y2HqcNiz2c8AdH8A6p8VOllcBS8CoBNwA7DYzBYUP3evc+7DcJFEJFKKCmHxnyH3Udj5OWR1hP5jocn5oZMlRPAicM5NB5J77E5E0pNzsHy8vxV06zKo1xquf8evC5Tk08CxFLwIRESCWDvZD4N9ORdqNYf+r0GLvkk3DZwIKgIRiZYNeb4APpsC1RtBn2ehzQAoG923w+i+chGJli3L/Cmg5R9A5VpwyWOQcxNkVAydLDgVgYiktx3rIPcxWPQnqFDNLwjX4Rb/sQAqAhFJV7s3+9tA574GZcpCx1vhgjugcs3QyZKOikBE0su+7cXTwC9AUT60uwG63AXVG4ROlrRUBCKSHg7ugVnPw8fPwsFd0Lo/dB0MtZqFTpb0VAQiktoKDvrTP1OHw96tcNplfm/geq1CJ0sZKgIRSU1FhbDwLZg8DL5ZD9md4do3oXH70MlSjopARFKLc35DmElDYdsKqN8WrhgJzbpHaho4llQEIpIanIO1uX4YbON8qH0a/J/XoUUfFcAJit4stUTCiAkrQ0eQWPpiNoy9Av5wJez9Gvr+Dm75BFr2VQnEgI4IJC2NmriKX/Y8LXQMOVFffeqngVf+HapkwmVPwDk3QrkKoZOlFRWBiCSf7Wv9ktCL34EK1f2eAB1ugfJVQidLSyoCEUkeuzbClCdg/h+gTAZccDt0/IWmgeNMRSAi4e3bDtOfhtkvQVGBP/1z4a+hWr3QySJBRSApb8SElYyauOrfns8ePP47jwf1aK7rBsnm4G6/MfyMZ/3Hba71m8PXbBo6WaSYcy50hmOWk5Pj8vLyQseQJJY9eDzrhvUKHUOOJP8A5L0K056CfdvgjN5+GrhOi9DJ0pqZzXXO5Xz/eR0RiEjiFBbAwjdh8uOwawM07QI9HoRG54ROFmkqAhGJv6IiWPpXyB0KX6+Ghjnwo9/BKV1CJxNUBCIST87B6o/8NPBXiyCzhV8P6PTLNQiWRFQEkpYG9WgeOoKsnwkfPQTrZ0CNJnDlGGh9td8kRpKKikDSku4OCmjTIpg0BFb9C6rWhcufhLN/AuXKh04mR5AURWBmrwK9gS3OOS0iLpKKtq321wCWvAsVa8BFv4X2N0P5yqGTyVEkRREArwGjgdcD5xCRY/XNBpjyOMwfB+UqQudfQcfboFKN0MmklJKiCJxzU80sO3QOETkGe7fBtKdhzsuAg/Y/g853QtU6oZPJMUqKIigNMxsIDATIysoKnEYkwg7sgk9GwyfPQf4+aDMAut4NNfT/ZapKmf0InHNjnHM5zrmczMzM0HFSitbml5jI3++XghjVxp8KOrUH/Hwm/Og5lUCKS5kjAjl+WptfTkhhPsx/w68Kunuj3xKyx2+gQbvQySRGVAQiUrKiIn8HUO5Qvz9Ao/bQbww07Rw6mcRYUhSBmf0R6ArUNrMNwIPOuVfCphKJKOdg5T/9LMDmT6FuK7juLTjtUk0Dp6mkKALn3HWhM4gIsG66Xw7ii1lwclO46hU4sx+USZnLiXIckqIIJHa0Nr8cl43zYeIQWDMRqtWH3iOh3Y+hbEboZJIA2o8gArQ2vxzR1pWQ+wgsfR8qnQwX3OHnATIqhU4mcaD9CETk/9u53u8JsPBNyKjsdwU7/1aoWD10MglARSASJXu2+l3B8l4BDM67BS74JVTVbE6UqQhEomD/Tj8MNvN5KDgA7a73RwEnNQqdTJKAiiACtDZ/hB3aB7NfhOkj4cBOfwdQt/ug9qmhk0kSOWoRmNkE4FfOuYUJyCNxoLuDIqjgEMx/HaYMhz1fQfOL/ebw9duETiZJqDRHBHcDI81sHXCvc25TfCOJyHErKoTF78DkR2HHOsg6H/r/Hpp0DJ1MkthRi8A5Nw/oZmZXAf8ws3eBJ5xz++OeTkRKxzlY8SFMegS2LIV6reH6d+DUizQNLEdVqnFBMzNgBfA8cBuwysxuiGcwESmltVPg5YvgrQFQcBCufhUGToXmPVUCUiqluUbwMdAUWALMBG4ElgODzKyzc25gXBOKSMk2zIWJD8FnU6B6Q+jzrN8boKzuAZFjU5q/MQOBpe7fR5BvM7NlccgkIj9kyzJ/Cmj5B1C5FlzyKOT8FDIqhk4mKao01wiW/MCntW6BSKLsWAeTh8HCt6B8Veh6L5z/c6hQLXQySXEndAzpnFsbqyAicgS7N8PU4TD3NShTFjre6tcEqlwzdDJJEzqZKJKs9u+Aj0fBzBeg8BCcfYOfBq7eIHQySTMqApFkc2gvzHrBl8CBb6DV1dDtXqjVLHQySVMqApFkUXDQn/6Z+iTs3eJ3BOt+v58JEIkjFYFIaEWFsOhPkPsYfLMemlwA17wBWeeFTiYRoSIQCcU5WPY//lbQbSugflu4YiQ0665BMEkoFYFIojkHa3P93sAb50Pt06D/WGjZVwUgQagIRBLpizl+GnjdNDipMfR9Ds66VtPAEpT+9okkwuYl/hTQig+hSiZc9gSccyOUqxA6mYiKQCSutq/1F4EX/xkqVPd3AZ13C1SoGjqZyP9KiiIws0uBUUBZ4GXn3LDAkUROzK5NMPUJmPc6lMmAToP8H00DSxIKXgRmVhZ4DugJbADmmNnfnHNLwyYTOQ77tsP0ETB7DBQV+NM/F/4aqtULnUzkiIIXAdAeWP3tukVm9hbQF1ARSOo4uMdvDD/jGTi4G866BroOhppNQycTOapkKIKGwBeHPd4A/NskjZkNxC+JTVZWVmKSiRxN/gGY+3s/DbxvG5zR218HqNMidDKRUkuGIigV59wYYAxATk7O9/dGEEmswgJY+Ee/LPSuDdC0C/T4DTTKCZ1M5JglQxF8CTQ+7HGj4udEkk9RESx7HyYNha9XQcNz4EfPwSldQycTOW7JUARzgOZm1hRfANcCA8JGEvke52D1RJj0MGxaCJkt4JpxcEYvTQNLygteBM65AjO7Ffgn/vbRV4+yK5pIYq2fCR89BOtnQI0mcOWL0Lq/3yRGJA0ELwIA59yHwIehc4h8x1eLYeIQWPVPqFoXLn8Szv4JlCsfOplITCVFEYgkla/XQO5Q+PQvUPEk6PEgnHczlK8SOplIXKgIRL71zZcw5XGY/4ZfA6jzndDxF1CpRuhkInGlIhDZ+zVMfxpmvwQ4aP8zXwJV64ROJpIQKgKJrgO74JPn4JPRkL8P2lznp4FraGBRokVFINGTvx/mvAzTnob926FFHz8NnHl66GQiQagIJDoK82HBOJj8OOze6LeE7P4ANDw7dDKRoFQEkv6KimDJu/5OoO1rodG50G8MNO0cOplIUlARSPpyDlb9y88CbF4Mdc6E696C0y7VNLDIYVQEkp7Wfew3h/9iJpzcFPq9BK2uhjJlQicTSToqAkkvGxf4AlgzEarVh94joN0NUDYjdDKRpKUikPSwdaW/BrD0r1DpZOg5xM8DZFQKnUwk6akIJLXt/AKmDIMFb0K5SnDhXdDxVr80hIiUiopAUtOerTDtKch7xT8+77/ggjugambYXCIpSEUgqeXANzBjtJ8ILtgPba+HLndDjcZH/2dFpEQqAkkNh/bB7DEwfQQc2AlnXgnd7oPazUMnE0l5KgJJbgWHYP7rMGU47PkKTu0JPR6A+m1CJxNJGyoCSU5FhX4/gNyhsGMdNO4A/X8PTTqGTiaSdlQEklycgxV/h0lDYMtSqNsaBvwZmvfUNLBInKgIJHl8NtUPg22YAzWbwdWvQssrNQ0sEmcqAgnvy7m+ANZOhmoN4IpR/m4gTQOLJISKQMLZstyfAlr+AVSuBZc8Cjk/hYyKoZOJRIqKQBJvx+cw+TFY9CfIqAJd74EOP4eK1UMnE4kkFYEkzu7NMO1JyPs9WBn/5n/BHVClVuhkIpEWtAjMrD/wW6AF0N45lxcyj8TJ/h3w8TMw6wUoOAhn3+DXBDqpYehkIkL4I4JPgX7Ai4FzSDwc2guzXoSPR/qlIVpd5aeBazULnUxEDhO0CJxzywBM94enl4JDMPc1mDoc9m6B5pf4aeB6rUMnE5EShD4iKDUzGwgMBMjKygqcRkpUVOgvAE9+DHauhyad4Jo/QFaH0MlE5AfEvQjM7COgXgmfus85935pv49zbgwwBiAnJ8fFKJ7EgnOw7H9g0iOwbYVfB6j3CGjWQ9PAIikg7kXgnLso3j9DAlqT64fBNs6DWs2h/1ho2VcFIJJCUubUkCSZDXkw8SG/LET1RtBnNLS5Dsrqr5RIqgl9++iVwLNAJjDezBY45y4JmUmOYvNSfwpoxXioXBsuHQY5N0G5CqGTichxCn3X0HvAeyEzSClt/6x4GvhtqFANut0PHW6BClVDJxORE6TjePlhuzb520DnjYUyGdDpF9DpdqhcM3QyEYkRFYGUbN92Pwg2awwU5cM5N0LnX0H1+qGTiUiMqQjkuw7ugZnPw4xn4OBuOOsa6DoYajYNnUxE4kRFIF7BQch7FaY+Cfu2wem9oPv9ULdl6GQiEmcqgqgrLIBFb8HkYfDNF9D0QujxIDTKCZ1MRBJERRBVzsHS9/3m8NtWQoOzoc+z0Kxb6GQikmAqgqhxDtZMhIlDYNMCyDwDrhkHZ/TSNLBIRKkIomT9LL8cxOfToUYWXPkitO4PZcqGTiYiAakIouCrxX4aeOU/oEoduPxJOPsnUK586GQikgRUBOns6zWQ+yh8+g5UPMlfBD7vZihfJXQyEUkiKoJ0tGsjTHkc5v3BrwHU+U7oeBtUOjl0MhFJQiqCdLL3a5j+NMx+CVwRnPtTPw1crW7oZCKSxFQE6eDALpj5O5gxGvL3wlnX+mngk5uETiYiKUBFkMry98OcV2DaU7B/O7S4wq8KWueM0MlEJIWoCFJRYT4sGAeTH4fdG+GUbtDjN9Dw7NDJRCQFqQhSSVERLH0PJg2F7Wug0bnQ70W/LISIyHFSEaQC52DVBJj0sJ8JqNMSrv0jnH6ZpoFF5ISpCJLd5zP8NPD6T+DkbOj3ErS6StPAIhIzKoJktXEBTBoCqz+CqvWg11PQ7j80DSwiMaciSDbbVvnlIJb+1Q+A9XwYzv0ZlK8cOpmIpCkVQbLY+QVMGQYL3oRyleDCu6DjrX5pCBGROFIRhLZ3m58DmPOyf9z+Zr8kRNXMsLlEJDJUBKEc+MZPAs/8HeTvg7YDoMtgqNE4dDIRiZigRWBmw4ErgEPAGuD/Oud2hswUd/n7YfYYmD4C9u+Alj+CbvdB5mmhk4lIRIU+IpgA3OOcKzCzx4F7gLsDZ4qPwnyY9zpMHQ67N8GpF0H3B6BB29DJRCTighaBc+5fhz2cCVwdKkvcFBX5/QByh8KOddC4A1z1CmR3Cp1MRAQIf0RwuJuAPx3pk2Y2EBgIkJWVlahMx885vyPYxCGwZQnUbQ0D3obmF2saWESSStyLwMw+AuqV8Kn7nHPvF3/NfUABMO5I38c5NwYYA5CTk+PiEDV2Ppvmp4E3zIaap/gjgDP7QZkyoZOJiPybuBeBc+6iH/q8md0I9AZ6OOeS+w3+aL6c5wtgbS5UawBXjIK210PZjNDJRESOKPRdQ5cCdwFdnHP7QmY5IVtX+GngZX+DSjXh4kfg3P+EjEqhk4mIHFXoawSjgQrABPPnzWc65/4rbKRjsONzvzfwwj9CRhXoeg90+DlUrB46mYhIqYW+a+jUkD//uO3ZAlOfhLxXwcr4N/8L7oAqtUInExE5ZqGPCFLL/p0w4xmY+TwUHIR2P4Yud8NJDUMnExE5biqC0ji0F2a9CB+P9EtDtLrKTwPXahY6mYjICVMR/JCCQzBvrJ8G3rPZzwB0fwDqnxU6mYhIzKgISlJUCIv/DLmPws7PIasj9B8LTc4PnUxEJOZUBIdzDpZ/4G8F3boc6p0F17/j1wXSNLCIpCkVwbfW5PphsI3zoFZz6P8atOiraWARSXsqgg15MPEh+GwqVG8EfUZDm+ugrP7ViEg0RPfdbvNSfwpoxXioXBsuHQY5N0G5CqGTiYgkVPSKYPtnMPkxWPQ2VKgG3e6HDrdAhaqhk4mIBBGtIpgy3G8QX6YcdPoFdLodKtcMnUpEJKhoFUGNLDj7J3Dhr6F6/dBpRESSQrSKoM01/o+IiPwv3RspIhJxKgIRkYhTEYiIRJyKQEQk4lQEIiIRpyIQEYk4FYGISMSpCEREIs6cc6EzHDMz2wp8HjrHcagNbAsdIoGi9npBrzkqUvU1N3HOZX7/yZQsglRlZnnOuZzQORIlaq8X9JqjIt1es04NiYhEnIpARCTiVASJNSZ0gASL2usFveaoSKvXrGsEIiIRpyMCEZGIUxGIiESciiAAM7vTzJyZ1Q6dJd7MbLiZLTezRWb2npnVCJ0pXszsUjNbYWarzWxw6DzxZmaNzSzXzJaa2RIzGxQ6UyKYWVkzm29mH4TOEisqggQzs8bAxcD60FkSZALQyjl3FrASuCdwnrgws7LAc8BlQEvgOjNrGTZV3BUAdzrnWgIdgP+OwGsGGAQsCx0illQEiTcCuAuIxFV659y/nHMFxQ9nAo1C5omj9sBq59xa59wh4C2gb+BMceWc2+Scm1f88W78m2PDsKniy8waAb2Al0NniSUVQQKZWV/gS+fcwtBZArkJ+HvoEHHSEPjisMcbSPM3xcOZWTbQDpgVOEq8jcT/IlcUOEdMRWvz+gQws4+AeiV86j7gXvxpobTyQ6/ZOfd+8dfchz+VMC6R2ST+zKwq8BfgdufcrtB54sXMegNbnHNzzaxr4DgxpSKIMefcRSU9b2atgabAQjMDf4pknpm1d859lcCIMXek1/wtM7sR6A30cOk7uPIl0Piwx42Kn0trZpaBL4Fxzrl3Q+eJs05AHzO7HKgIVDezN5xzPw6c64RpoCwQM1sH5DjnUnEFw1Izs0uBp4EuzrmtofPEi5mVw18M74EvgDnAAOfckqDB4sj8bzRjge3OudsDx0mo4iOCXznnegeOEhO6RiDxNhqoBkwwswVm9kLoQPFQfEH8VuCf+Iumb6dzCRTrBNwAdC/+b7ug+LdlSTE6IhARiTgdEYiIRJyKQEQk4lQEIiIRpyIQEYk4FYGISMSpCEREIk5FICIScSoCkRgoXpe/Z/HHj5jZs6EziZSW1hoSiY0HgYfNrA5+Fc4+gfOIlJomi0VixMymAFWBrsXr84ukBJ0aEomB4tVl6wOHVAKSalQEIifIzOrj91noC+wpXnFVJGWoCEROgJlVBt7F7927DBiCv14gkjJ0jUBEJOJ0RCAiEnEqAhGRiFMRiIhEnIpARCTiVAQiIhGnIhARiTgVgYhIxP0/W+ZrQSPu39IAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# get maximum likelihood estimate\n",
        "theta_ml = max_lik_estimate(X, ynew)\n",
        "print(theta_ml)\n",
        "\n",
        "# define a test set\n",
        "Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs\n",
        "\n",
        "# predict the function values at the test points using the maximum likelihood estimator\n",
        "ml_prediction = predict_with_estimate(Xtest, theta_ml)\n",
        "\n",
        "# plot\n",
        "plt.figure()\n",
        "plt.plot(X, ynew, '+', markersize=10)\n",
        "plt.plot(Xtest, ml_prediction)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u72apap4jmr8"
      },
      "source": [
        "#### Question:\n",
        "1. This maximum likelihood estimate doesn't look too good: The orange line is too far away from the observations although we just shifted them by 2. Why is this the case?\n",
        "2. How can we fix this problem?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9IJsENbOjmr-"
      },
      "source": [
        "Let us now define a linear regression model that is slightly more flexible:\n",
        "$$\n",
        "y = \\theta_0 + \\boldsymbol x^T \\boldsymbol\\theta_1 + \\epsilon\\,,\\quad \\epsilon\\sim\\mathcal N(0,\\sigma^2)\n",
        "$$\n",
        "Here, we added an offset (bias) parameter $\\theta_0$ to our original model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSv808spjmr-"
      },
      "source": [
        "#### Question:\n",
        "1. What is the effect of this bias parameter, i.e., what additional flexibility does it offer?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6Bco1A1jmr-"
      },
      "source": [
        "If we now define the inputs to be the augmented vector $\\boldsymbol x_{\\text{aug}} = \\begin{bmatrix}1\\\\\\boldsymbol x\\end{bmatrix}$, we can write the new linear regression model as \n",
        "$$\n",
        "y = \\boldsymbol x_{\\text{aug}}^T\\boldsymbol\\theta_{\\text{aug}} + \\epsilon\\,,\\quad \\boldsymbol\\theta_{\\text{aug}} = \\begin{bmatrix}\n",
        "\\theta_0\\\\\n",
        "\\boldsymbol\\theta_1\n",
        "\\end{bmatrix}\\,.\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "Tx9_4cUbjmr_"
      },
      "outputs": [],
      "source": [
        "N, D = X.shape\n",
        "a = np.ones((N,1))\n",
        "X_aug = np.column_stack((a,X))# EDIT HERE # augmented training inputs of size N x (D+1)\n",
        "theta_aug = np.zeros((D+1, 1)) # new theta vector of size (D+1) x 1"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "X_aug"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JGknyP8NE6KP",
        "outputId": "7b4bb04f-753a-44e5-b525-2b4ab0710e51"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 1., -3.],\n",
              "       [ 1., -1.],\n",
              "       [ 1.,  0.],\n",
              "       [ 1.,  1.],\n",
              "       [ 1.,  3.]])"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3q46Dvttjmr_"
      },
      "source": [
        "Let us now compute the maximum likelihood estimator for this setting.\n",
        "_Hint:_ If possible, re-use code that you have already written"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "onVTg70Pjmr_"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def max_lik_estimate_aug(X_aug, y):\n",
        "  XX=X_aug.T@X_aug\n",
        "  Xy=X_aug.T@y  \n",
        "  theta_aug_ml = scipy.linalg.solve(XX,Xy) ## <-- EDIT THIS LINE\n",
        "\n",
        "  \n",
        "  return theta_aug_ml"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "DB_LxK8ejmsA"
      },
      "outputs": [],
      "source": [
        "theta_aug_ml = max_lik_estimate_aug(X_aug, y)\n",
        "\n",
        "#@ test\n",
        "assert np.any(theta_aug_ml), \"Max_lik_estimate_aug not implemented\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JtKQSJEzjmsB"
      },
      "source": [
        "Now, we can make predictions again:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "rnB0c7nRjmsB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "d1ea43e9-062a-4395-e7f2-09e8a247a725"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbuUlEQVR4nO3dd5hUVZ7G8e+PliBBQBoEgaYRQQfEgC2goCJBQAkj6ogB4w7rLipgQIKYM6OAiqMYVl0Z06hLDg0qSQEbFCVIDoLkDELT4ewfF2cQQRu6qk5V3ffzPP1IV/XT9ZZov33vufd3zDmHiIiEVxHfAURExC8VgYhIyKkIRERCTkUgIhJyKgIRkZA7zneAY5GamurS09N9xxARSSizZ8/e7JyreOjjCVkE6enpZGVl+Y4hIpJQzGzV4R7XqSERkZBTEYiIhJyKQEQk5FQEIiIhpyIQEQk5FYGISMipCEREQk5FICKSCH7eCmN7w74dEf/WKgIRkXjmHMz/FIY0hK9fg1VfRvwlEvLOYhGRUNi5DsbcCz+MgipnQ5dPoXL9iL+MikBEJN44B9/8L4x/APKyodWj0LgbpETnR7aKQEQknmxdASO7w4rJUKMJdHgRKtSK6kuqCERE4kF+Hsx8FT57DCwF2g2EBjdDkegv5aoIRER827gQht8Ba7OgduugBMpWjdnLqwhERHzJ3Q/TB8HkZ6F4Gej0OtS/CsxiGkNFICLiw9rZMPxO2DgfzrgS2j4LpVK9RFERiIjE0v6f4Yun4KuXoHRluPZ9OK2t10gqAhGRWFkxFUbeBVuXQ4Ob4NLHoERZ36lUBCIiUbdvJ2Q+CLP/B8rXhJtGQs2LfKf6FxWBiEg0LRoHo3rC7vVw/h1wST8oVtJ3ql9REYiIRMOezTD2fpj3T6hUF655F6qd6zvVYakIREQiyTmY9zGM7RWcEmrWF5r2hOOK+U52RCoCEZFI2flTcBpo8Tioei50eAlOqus71R9SEYiIFJZzMOdtmNAf8nKg9ZPQ6HYokuI7WYGoCERECmPrchhxF6ycCukXQocX4MRTfKc6KioCEZFjkZ8HM16Gz56AlKLQ/gVocGPMx0NEgopARORobVgAw7vBT3OgTlto9zyccLLvVMdMRSAiUlC5+2Hqc8FHibJw1ZtQr1NCHgUcTEUgIlIQa7KCUdGbFkL9q6HNM1Cqgu9UEeG9CMysOvAOcBLggKHOucF+U4mIHLB/T7AOMOPl4PTPdR9Cnda+U0WU9yIAcoF7nHNzzKwMMNvMMp1zC3wHE5GQWz45GBK3bSVk3AotH4ESJ/hOFXHei8A5tw5Yd+DPu8xsIVAVUBGIiB97t0Nmf5jzTnAp6M2jIb2p71RR470IDmZm6cA5wMzDPNcV6AqQlpYW22AiEh4/jIHRd8PuDdCkOzTrA0WP950qquKmCMysNPAx0MM5t/PQ551zQ4GhABkZGS7G8UQk2e3eFMwHmv8JVKoHnf8BVRv4ThUTcVEEZlaUoASGOec+8Z1HRELEOfj+o2BS6P7dcMkDwZFAHA+JizTvRWBmBrwBLHTOPe87j4iEyI41MOpuWDIeqp0XDImrdLrvVDHnvQiAJkAX4Hsz+/bAY32dc2P8RRKRpJafH+wWlvkQuDxo/RQ0+s+EGRIXad6LwDk3DUjs2/JEJHFsWRYMiVs1DU5pBu0HQ/l036m88l4EIiIxkZcLM4bA509CSvHgNNA5NyT8eIhIUBGISPJb/30wHmLdt3B6O7jsb3BCFd+p4oaKQESSV242TBkA0wbC8eXh6reg7p91FHCIIr4DiIgcq4GZi4/85I+z4JULgyKofzV0mwX1rlAJHIaKQEQS1uBJS377YPZuGNsb3rgUcn6G6z+GK16BkifGPmCC0KkhEUkeyz4PhsRtXw3n/RVaPgTFy/hOFfdUBCKS+PZugwkPwDfvQoVT4ZaxUOMC36kShopARBLbwpEw+h7Ysxma3g0X3w9FS/hOlVBUBCKSEAZmLv7VmkBFtjOk6FvwwSwW5NfgvpxHmT+xJt3danq2quMvaAIy5xJvkGdGRobLysryHUNEfHAO5r4P43qTvXcPxVv2gQvugpSivpPFPTOb7ZzLOPRxHRGISOLYvhpG9oBlk6B6Yy5beiWTLuzqO1XC0+WjIhL/8vNh1mswpDGsngFtB8AtY1nmqvpOlhR0RCAi8W3zEhhxJ6z+Cmo1h3aDoHwN36mSiopAROJTXg58+SJ88XSwVeSf/w5nXfurO4O7t6jtMWDyUBGISPxZNzcYErf+O/hTe7jsOShz0m++TFcHRYaKQETiR84+mPIsTBsEJSvAX96Buh19p0p6KgIRiQ+rZwRHAVuWwFnXQesnNB8oRlQEIuJX9i6Y9GhwVVDZ6nDDJ3BqC9+pQkVFICL+LJ0Y3BewY02wZ3Dz/lC8tO9UoaMiEJHY+3krjO8Lc9+D1Dpw6zhIa+w7VWipCEQktub/H4y5NyiDC++Fi+7TkDjPVAQiEhu71gcFsHAkVDkrWAuocqbvVIKKQESizTn4dlhwKig3G1o+DOffCSn68RMv9DchItGzbRWM7A7LP4e0C6DDi5B6qu9UcggVgYhEXn5ecDnopEeDkRCXPwfn3gpFNOcyHqkIRCSyNi0KhsT9OBNObQXtBkK56r5Tye9QEYhIZOTlwPRBMPlZKFYKrhgKZ/7lV0PiJD6pCESk8H76BobfCRu+h3pXBPsFlK7oO5UUkIpARI5dzt5gTPSXL0KpinDNMPhTO9+p5CipCETk2Kz6MhgSt3UZNLgRWj0Gx5fznUqOgYpARI7Ovp0w8WHIegPK1YAbh8MpzXynkkKIiyIwszeBdsBG59wZvvOIyBEsyQyGxO1cC427QfN+wcKwJLR4uaj3LaCN7xAicgR7tsAnXWHYVcF00Nsyoc2TKoEkERdHBM65KWaW7juHiBzCOZj/KYy5D/Zth4t6wUX3wnHFfSeTCIqLIigIM+sKdAVIS0vznEYkBHaug9H3wKLRcPI50GE4VNaZ22SUMEXgnBsKDAXIyMhwnuOIJC/nYM47MKE/5GXDpY9Do//SkLgkpr9ZEfm3rStg5F2wYgrUaAodXoAKtXynkihTEYhIMCRu5qvw2WNgKcF8oAY3a0hcSMRFEZjZe0AzINXM1gAPOefe8JtKJCQ2LgxuDFubBbVbByVQtqrvVBJDcVEEzrlrfWcQCZ3c/TBtIEwZAMXLQKfXof5VGhIXQnFRBCISY2tnB0PiNs6HM66Ets9CqVTfqcQTFYFImOz/Gb54Er4aAqUrw7Xvw2ltfacSz7QSJJIkBmYu/v0vWDEVXmkSTAo9pwt0m6ESEEBFIJI0Bk9acvgn9u0I5gO93S64R+CmkcFloSXKxjSfxC+dGhJJZovGwaiesHs9nH8HXNIPipX0nUrijIpAJBnt2Qxj74d5/4RKdeGad6Haub5TSZxSEYgkE+dg3scwtlewb0CzPtD0bjiumO9kEsdUBCIJaGDm4t+sCVRmCxMfvISWKd/wTf6p9Mq5n8tyLqGnSkD+gDmXePPbMjIyXFZWlu8YIvEhPx/mvMWukX0pUxRo0R8a3Q5FUnwnkzhjZrOdcxmHPq4jApFEtmUZjOwOK6fyXX49mvz3MDixpu9UkmBUBCKJKD8vuCns8ycgpRi0H8z1H6WyUiUgx0BFIJJoNiyA4d3gpzlQpy20ex5OOBk+Gu07mSQoFYFIosjNhqnPw9TnoMQJcNWbUK/Tv4bEdW9R23NASVQqApFEsCYrGBW9aSGceQ20fgpKVfjVl/RsVcdTOEl0KgKReLZ/D3z2BMx4OTj9c92HUKe171SSZFQEIvFq+eRg28htKyHjNmj5cHBKSCTCVAQi8WbvdsjsH2wgf+IpcPNoSG/qO5UkMRWBSDz5YQyMvht2b4Am3YMREUWP951KkpyKQCQe7N4UzAea/wmcdAZ0/gdUbeA7lYSEikDEJ+fguw9h3P3BwnDzB6BJD0gp6juZhIiKQMSXHWuCvQKWTIBq50GHl6DS6b5TSQipCERiLT8fZr8JmQ+By4c2z0DDv2pInHijIhCJpS3LYMSdsGo6nNIM2g+G8um+U0nIac9ikQg64gbyebkwfTD8/QLYMA86DoEu/6cSkLigIhCJoMNuIL/+e3i9BWQ+CKe2hG6z4Jwb/jUjSMQ3nRoSiZbcbJgyAKYNhOPLw9VvQ92OKgCJOyoCkWj4cVYwJG7zIjizM7R5Ckqe6DuVyGGpCEQiqCT7YGxvmPkKlK0G138MtVv6jiXyu/6wCMwsE7jXOTc3BnlEEsahG8g3LfI944u9DjM38U5uK57Z0Jk9b2TTvcVijYiWuPaHm9ebWQPgOWAl0Nc5ty4GuX6XNq+XuLJ3G0x4AL55l2X5Vah125tQ4wLfqUR+40ib1//hVUPOuTnOuUuAUcA4M3vIzDQFSwRgwQgY0gi+fQ+a9uSy/U+pBCThFOjyUTMzYBHwd+BOYImZdYlUCDNrY2aLzGypmfWO1PcViZpdG+CDLvBhFyhdCf76GbR8mGyK+U4mctQKskYwHagJzAdmADcDPwDdzexC51zXwgQwsxRgCNAKWAN8bWYjnHMLCvN9RaLCOZj7HozrAzl7oXn/YFy0hsRJAivIVUNdgQXut4sJd5rZwghkaAgsdc4tBzCz94GOgIpA4sv21TCyByybBNUbQ4cXoeKvF4G1gbwkoj8sAufc/N95+vIIZKgK/HjQ52uARod+kZl1JSgl0tLSIvCyIgWUnw9fvwYTHwk+bzsAzvsPKPLbM6u6OkgSUaHuI/jlt/hYcM4NBYZCcNVQrF5XQm7T4mBI3I8zoFbzYEhcOf0iIsklHm4oWwtUP+jzagceE/EnLwe+fAG+eAaKloCOL8PZ12k8hCSleCiCr4HaZlaToAA6A9f5jSShtm5uMB5i/XfBbKC2A6DMSb5TiUSN9yJwzuWa2R3AeCAFePMP1iVEoiNnH0x+JhgXXSoV/vK/ULeD71QiUee9CACcc2OAMb5zSIit+ipYC9iyBM6+AVo/HkwMFQmBuCgCEW+ydwVXA339GpRNgy6fBovCIiGiIpDwWjIRRnaHnWuh0e3BzWHFS/tOJRJzKgIJn5+3wvi+wR3CqXXg1vGQ9ptbV0RCQ0Ug4eEcLBgOY+4NJoZedF/wcVxx38lEvFIRSDjsWg+j74EfRkGVs4O1gMr1facSiQsqAkluzsG3w4JTQbnZ0PIROP8OSNF/+iK/0P8Nkry2rQwWg5d/ATWaQPsXIPVU36lE4o6KQJJPfh7MGgqTHgVLgcufh3NvOeyQOBFREUiy2bQoGA+xZhac2graDwo2kReRI1IRSHLIy4Hpg2Dys1CsNHR6DepfrSFxIgWgIpDE99M3wVHAhnlQrxO0fRZKV/SdSiRhqAgkceXshS+egi9fhFKVoPM/4PRI7JUkEi4qAklMK6cHQ+K2LoMGN0Krx+D4cr5TiSQkFYEkln07YeLDkPUGlKsBNw6HU5r5TiWS0FQEkjgWT4BRPYMhcY27QfN+UKyU71QiCU9FIPFvzxYY3we++wAqng63ZUL183ynEkkausNG4pdzMO9jGNIw+OfF98N/TilQCQzMXByDgCLJQUUg8WnnOnj/OvjnrVCuelAAl/Qt8KTQwZOWRDmgSPLQqSGJL87BnHdgQn/Iy4ZLH4dG/6UhcSJRpP+7JH5sXR4MiVsxBWo0hQ4vQIVavlOJJD0VgfiXnwcz/g6fPQ5FjoN2g6DBTRoSJxIjKgLxa+PCYDzE2iyo3RraDYSyVY/qWwzMXHzYNYH03qN/9Xn3FrXp2apOoeKKJCNzzvnOcNQyMjJcVlaW7xhSGLn7YdpAmDIASpwQzAc648qIDYlL7z2alU9r3ITIwcxstnMu49DHdUQgsbd2dnAUsHFBMCG0zdNQKtV3KpHQUhFI7Oz/GT5/Ama8DKUrw7Xvw2ltfacSCT0VgcTGiqnBkLhtK4Ldwlo9AiXK+k4lIqgIJNr27YDMB2H2W1C+Jtw0CmpeGPWX7d6idtRfQyRZqAgkehaNC4bE7V4PF9wJzfpCsZIxeWldHSRScCoCibw9m2Hs/TDvn1CpLnR+F6qe6zuViByBikAi55chcWN7BfsGNOsLTXvCccV8JxOR3+H11k0zu9rM5ptZvpn95tpWSSA71sJ7neHj26B8Otw+FZrdrxIQSQC+jwjmAZ2AVz3nkGOVnw9z3g6GxOXnQusnodHtUCTFdzIRKSCvReCcWwhgEbqbVGJsy7JgSNzKqVDzImj/ApxY03cqETlKvo8ICszMugJdAdLS0jynCbm83OCmsM+fgJRiQQE0uDFi4yFEJLaiXgRmNhGofJin+jnnhhf0+zjnhgJDIZg1FKF4crQ2zA/GQ/w0B067DC5/Dk442XcqESmEqBeBc65ltF9DYiA3G6Y+D1OfC+4IvupNqNdJRwEiSUAD30Og0Pv3rsmCVy+GyU9DvSug26yITgoVEb98Xz56hZmtAc4HRpvZeJ95ktUx79+7fw+M6wuvt4TsnXDdR3Dla1CqQmQDiohXvq8a+hT41GcGOYLlk2HkXbBtJWTcBi0fDvYNEJGkkzBXDUmM7N0Omf2DDeRPrAU3j4H0Jr5TiUgUqQjk334YDaPuhj0boUl3aNYHih7vO5WIRJmKIMkc0/69uzcF84HmfwInnQHXvgdVG8QirojEAe1ZHAJH3L/XOfjuQxh3f7AwfHEvaNIDUorGPKOIRJ/2LJZf2/5jsFfA0kyo1hA6vgQVT/OdSkQ8UBGETX4+ZL0BEx8Glx9sHN+wq4bEiYSYiiBMNi8N9g1e/SWc0iyYEVS+hu9UIuKZiiAEejSvCdMGwudPQdES0OElOOcG3RksIoCKIPmt/54eK7rBurlwertgSFyZw80AFJGwUhEkq5x9MGUATB8Ex5eHq98K5gSJiBxCRZCMVs8M1gI2L4KzroPWT0DJE32nEpE4pSJIJtm74bPHYOarULYaXP8x1NYUcBH5fSqCZLF0EozsATt+hIZ/hRYPQvEyvlOJSAJQESS6vdtgfD/4dhhUqA23jIUa5/tOJSIJREWQyBaMgDH3wp7NcOE9cFGv4PJQEZGjoCJIRLs2BAWwcARUrg/XfwRVzvKdSkQSlIogkTgHc9+DcX0gZy+0eAguuFND4kSkUFQEiWLbKhjVA5Z9BtUbB0PiUmv7TiUiSUBFEO/y8+Hr14MhcWZw2d+CrSOLeN1uWkSSiIognm1aHNwY9uMMqNUC2g+Ccmm+U4lIklERxKO8HJg+GCY/A0VLwp9fgbM6a0iciESFiiDerJsLw7vB+u+hbkdoOwDKnOQ7lYgkMRVBvMjZFxwBTB8MpVLhmnfhT+19pxKREFARxINVX8GIO2DLUjj7Bmj9eDAxVEQkBlQEPmXvgkmPwqzXoFx16PIp1GruO5WIhIyKwJclE4P7AnasgUa3Q/MHoHhp36lEJIRUBLH281YY3ze4Qzj1NLhtAlRv6DuViISYiiBWnIMFw4MZQXu3wUX3BR/HFfedTERCTkUQC7vWw+h74IdRUOXsYC2gcn3fqUREABVBdDkH37wb7BeQlw2tHoXG3SBF/9pFJH7oJ1K0bFsJI7vD8i8g7QLo8CKknuo7lYjIb3gtAjMbALQH9gPLgFucc9t9Ziq0/DyYNTS4LNRS4PLn4dxbNCROROKW759OmcAZzrkzgcVAH895CmfjD/BmaxjXG2o0gW4z4DxNChWR+Ob1iMA5N+GgT2cAV/nKUii5+2H6IJgyAIqVhk6vQf2rNSRORBJCPK0R3Ap8cKQnzawr0BUgLS2ORjGvnROMit4wD+p1grbPQumKvlOJiBRY1IvAzCYClQ/zVD/n3PADX9MPyAWGHen7OOeGAkMBMjIyXBSiHp2cvfD5k/DVS1CqEnT+B5x+ue9UIiJHLepF4Jxr+XvPm9nNQDughXPO/w/4glg5LTgK2LocGtwIrR6D48v5TiUickx8XzXUBugFXOyc+9lnlgLZtxMmPgRZb0K5GnDjcDilme9UIiKF4nuN4CWgOJBpwcLqDOfc7X4jHcHiCTCqJ+z6Cc6/Ay7pC8VK+U4lIlJovq8aiv87rPZsgfF94LsPoOLp8JdMqJbhO5WISMT4PiKIX87B/E9gTC/YtwMu7g0X3q0hcSKSdFQEh7NzHYy+GxaNgZMbQMeX4KR6vlOJiESFiuBgzsGcd2BCf8jbD5c+Do3/G4qk+E4mIhI1KoJfbF0OI+6ClVMh/UJoPxgq1PKdSkQk6lQE+Xkw8xWY9BikFIV2g6DBTZoPJCKhEe4i2LAguDFsbRbUaRNMCi1b1XcqEZGYCmcR5O6Hac/DlL9BiRPgyjfgjCs1JE5EQil8RbBmNoy4AzYuCCaEtnkaSqX6TiUi4k24imDyAPjiSShdGa79AE5r4zuRiIh34SqCE2sGC8GtHoESZX2nERGJC+EqgvpXBR8iIvIvukZSRCTkVAQiIiGnIhARCTkVgYhIyKkIRERCTkUgIhJyKgIRkZBTEYiIhJw553xnOGpmtglY5TvHMUgFNvsOEUNhe7+g9xwWifqeazjnKh76YEIWQaIysyznXIbvHLEStvcLes9hkWzvWaeGRERCTkUgIhJyKoLYGuo7QIyF7f2C3nNYJNV71hqBiEjI6YhARCTkVAQiIiGnIvDAzO4xM2dmSb9ZspkNMLMfzOw7M/vUzMr5zhQtZtbGzBaZ2VIz6+07T7SZWXUz+9zMFpjZfDPr7jtTLJhZipl9Y2ajfGeJFBVBjJlZdeBSYLXvLDGSCZzhnDsTWAz08ZwnKswsBRgCtAXqAteaWV2/qaIuF7jHOVcXaAx0C8F7BugOLPQdIpJUBLE3EOgFhGKV3jk3wTmXe+DTGUA1n3miqCGw1Dm33Dm3H3gf6Og5U1Q559Y55+Yc+PMugh+OVf2mii4zqwZcDrzuO0skqQhiyMw6Amudc3N9Z/HkVmCs7xBRUhX48aDP15DkPxQPZmbpwDnATM9Rom0QwS9y+Z5zRFS4Nq+PATObCFQ+zFP9gL4Ep4WSyu+9Z+fc8ANf04/gVMKwWGaT6DOz0sDHQA/n3E7feaLFzNoBG51zs82smec4EaUiiDDnXMvDPW5m9YGawFwzg+AUyRwza+icWx/DiBF3pPf8CzO7GWgHtHDJe+PKWqD6QZ9XO/BYUjOzogQlMMw594nvPFHWBOhgZpcBJYATzOxd59wNnnMVmm4o88TMVgIZzrlEnGBYYGbWBngeuNg5t8l3nmgxs+MIFsNbEBTA18B1zrn5XoNFkQW/0bwNbHXO9fAcJ6YOHBHc65xr5zlKRGiNQKLtJaAMkGlm35rZK74DRcOBBfE7gPEEi6YfJnMJHNAE6AI0P/B3++2B35YlweiIQEQk5HREICIScioCEZGQUxGIiIScikBEJORUBCIiIaciEBEJORWBiEjIqQhEIuDAXP5WB/78uJm96DuTSEFp1pBIZDwEPGpmlQimcHbwnEekwHRnsUiEmNlkoDTQ7MB8fpGEoFNDIhFwYLpsFWC/SkASjYpApJDMrArBPgsdgd0HJq6KJAwVgUghmFlJ4BOCvXsXAo8RrBeIJAytEYiIhJyOCEREQk5FICIScioCEZGQUxGIiIScikBEJORUBCIiIaciEBEJuf8HVg2MNhAEcdEAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# define a test set (we also need to augment the test inputs with ones)\n",
        "N_test, D_test = Xtest.shape\n",
        "ones = np.ones([N_test, 1])\n",
        "Xtest_aug = np.concatenate((ones, Xtest), axis = 1) # EDIT HERE# 100 x (D + 1) vector of test inputs\n",
        "\n",
        "# predict the function values at the test points using the maximum likelihood estimator\n",
        "ml_prediction = predict_with_estimate(Xtest_aug, theta_aug_ml)\n",
        "\n",
        "# plot\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+', markersize=10)\n",
        "plt.plot(Xtest, ml_prediction)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8tACtvWjmsB"
      },
      "source": [
        "It seems this has solved our problem! \n",
        "#### Question:\n",
        "1. Play around with the first parameter of $\\boldsymbol\\theta_{\\text{aug}}$ and see how the fit of the function changes.\n",
        "2. Play around with the second parameter of $\\boldsymbol\\theta_{\\text{aug}}$ and see how the fit of the function changes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GeYfdhHVjmsB"
      },
      "source": [
        "### Nonlinear Features\n",
        "So far, we have looked at linear regression with linear features. This allowed us to fit straight lines. However, linear regression also allows us to fit functions that are nonlinear in the inputs $\\boldsymbol x$, as long as the parameters $\\boldsymbol\\theta$ appear linearly. This means, we can learn functions of the form\n",
        "$$\n",
        "f(\\boldsymbol x, \\boldsymbol\\theta) = \\sum_{k = 1}^K \\theta_k \\phi_k(\\boldsymbol x)\\,,\n",
        "$$\n",
        "where the features $\\phi_k(\\boldsymbol x)$ are (possibly nonlinear) transformations of the inputs $\\boldsymbol x$.\n",
        "\n",
        "Let us have a look at an example where the observations clearly do not lie on a straight line:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "z2Ta2WKRjmsC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "4fc44f78-2e97-4fcb-8ba2-e578ab350cd3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAN9ElEQVR4nO3dfYyld1mH8evLLggFFEzHpLZdhz+2xKa+oCf1BUMqy2qlhGqisUQJVZMNfwBLg4FREhtBkhoNS0MMcQNFDA1E2xoJW7FjaUWM1M7WKm2X7ja1pa1IF7WBqkmt3P4xp7I7bOnsvP1m5r4+yWTPeebMee4nu3uueZ7nvKSqkCT186zRA0iSxjAAktSUAZCkpgyAJDVlACSpqZ2jBzgdZ555Zs3Ozo4eQ5K2lMOHD3+lqmaWLt9SAZidnWVhYWH0GJK0pSR58FTLPQQkSU0ZAElqygBIUlMGQJKaMgCS1FSbAByYPzp6BEnaVNY9AEmuSfJokrtOWPadSeaTHJv++eL1nuPqm4+t9yokaUvZiD2APwIuXrJsDri5qnYDN0+vS5I20Lq/EKyqPpNkdsniS4GLppc/AtwKvGOt131g/uhJv/nPzh0CYP+e3Vyx97y1Xp0kbSnZiA+EmQbgk1V1wfT6Y1X1ounlAP/x1PVT/Ow+YB/Arl27fvjBB0/5grZnNDt3iAeuumRFPytJW1mSw1U1Wbp8+EngWizQ01aoqg5W1aSqJjMz3/RWFpKkFRoVgC8nOQtg+uej673C/Xt2r/cqJGlLGRWATwBvmF5+A/Dn671Cj/lL0sk24mmgHwP+DnhpkoeT/BpwFbA3yTHgVdPrkqQNtBHPAnrd03xrz3qvW5L09IafBJYkjWEAJKkpAyBJTRkASWrKAEhSUwZAkpoyAJLUlAGQpKYMgCQ1ZQAkqSkDIElNGQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlMGQJKaMgCS1JQBkKSmDIAkNWUAJKmpoQFIckWSu5PcleRjSZ47ch5J2owOzB9dl/sdFoAkZwNvASZVdQGwA7hs1DyStFldffOxdbnf0YeAdgLPS7ITOAP4l8HzSFIbO0etuKoeSfL7wBeB/wZuqqqblt4uyT5gH8CuXbs2dkhJGuTA/NGTfvOfnTsEwP49u7li73lrso5U1Zrc0WmvOHkxcD3wi8BjwJ8C11XVR5/uZyaTSS0sLGzMgJK0SczOHeKBqy5Z8c8nOVxVk6XLRx4CehXwz1V1vKr+B7gB+PGB80hSKyMD8EXgR5OckSTAHuDIwHkkaVPav2f3utzvsABU1W3AdcAdwOensxwcNY8kbVZrdcx/qWEngQGq6krgypEzSFJXo58GKkkaxABIUlMGQJKaMgCS1JQBkKSmDIAkNWUAJKkpAyBJTRkASWrKAEhSUwZAkpoyAJLUlAGQpKYMgCQ1ZQAkqSkDIElNGQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlNDA5DkRUmuS/KFJEeS/NjIeSSpk52D13818Kmq+vkkzwHOGDyPJLUxLABJvgN4BXA5QFU9ATwxah5J6mbkIaCXAMeBDyf5hyQfTPL8pTdKsi/JQpKF48ePb/yUkrRNjQzATuCHgA9U1cuA/wTmlt6oqg5W1aSqJjMzMxs9oyRtWyMD8DDwcFXdNr1+HYtBkCRtgGEBqKp/BR5K8tLpoj3APaPmkaRuRj8L6M3AtdNnAN0P/MrgeSSpjaEBqKo7gcnIGSSpK18JLElNGQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlMGQJKaMgCS1JQBkKSmDIAkNWUAJKkpAyBJTRkASWrKAEhSUwZAkpoyAJLU1DMGIMl8kh/YiGEkSRtnOXsA7wDel+TDSc5a74EkSRvjGQNQVXdU1U8CnwQ+leTKJM9b/9EkSetpWecAkgS4F/gA8GbgWJLXr+dgkqT1tZxzAH8LPAIcAM4GLgcuAi5McnA9h5MkrZ+dy7jNPuCeqqoly9+c5MhqB0iyA1gAHqmq16z2/iRJy7OccwB3n+LB/ymXrMEM+4FVh0SSdHpW9TqAqrp/NT+f5BwWI/LB1dyPJOn0jX4h2PuAtwNff7obJNmXZCHJwvHjxzdsMEna7oYFIMlrgEer6vC3ul1VHayqSVVNZmZmNmg6Sdr+Ru4BvBx4bZIHgI8Dr0zy0YHzSFIrwwJQVb9RVedU1SxwGfDpqvrlUfNIUjejzwFIkgZZzusA1l1V3QrcOngMSWrFPQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlMGQJKaMgCS1JQBkKSmDIAkNWUAJKkpAyBJTRkASWrKAEhSUwZAkpoyAJLUlAGQpKYMgCQ1ZQAkqSkDIElNGQBJasoASFJTwwKQ5NwktyS5J8ndSfaPmkWSOto5cN1PAm+rqjuSvBA4nGS+qu4ZOJMktTFsD6CqvlRVd0wvfw04Apw9ah5J6mZTnANIMgu8DLjtFN/bl2QhycLx48c3fDZJ2q6GByDJC4DrgbdW1VeXfr+qDlbVpKomMzMzGz+gJG1TQwOQ5NksPvhfW1U3jJxFkroZ+SygAB8CjlTVe0fNIUldjdwDeDnweuCVSe6cfr164DyS1Mqwp4FW1WeBjFq/JHU3/CSw+jowf3T0CFJrBkDDXH3zsdEjSK0ZAElqauRbQaihA/NHT/rNf3buEAD79+zmir3njRpLailVNXqGZZtMJrWwsDB6DK2R2blDPHDVJaPHkLa9JIerarJ0uYeAJKkpA6Bh9u/ZPXoEqTUDoGE85i+NZQAkqSkDIElNGQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlMGQJKaMgCS1JQBkKSmDIAkNWUAJKkpAyBJTQ0NQJKLk9yb5L4kcyNnkbTowPzR0SNogwwLQJIdwB8APwOcD7wuyfmj5pG06Oqbj40eQRtk5B7AhcB9VXV/VT0BfBy4dOA8ktTKzoHrPht46ITrDwM/svRGSfYB+wB27dq1MZNJzRyYP3rSb/6zc4cA2L9nN1fsPW/UWFpnIwOwLFV1EDgIMJlMavA40rZ0xd7z/v+BfnbuEA9cdcngibQRRh4CegQ494Tr50yXSZI2wMgA3A7sTvKSJM8BLgM+MXAeSSwe9lEPww4BVdWTSd4E/CWwA7imqu4eNY+kRR7z72PoOYCquhG4ceQMktSVrwSWpKYMgCQ1ZQAkqSkDIElNGQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIa+DA/NHRI0inzQBIa+DEz9OVtgoDIElNbfoPhZc2qwPzR0/6zX927hCw+JGKfqqWtoJU1egZlm0ymdTCwsLoMaRvMjt3iAeuumT0GNIpJTlcVZOlyz0EJElNGQBpDezfs3v0CNJpMwDSGvCYv7YiAyBJTRkASWrKAEhSUwZAkpoyAJLU1JZ6IViS48CDK/zxM4GvrOE4I22Xbdku2wFuy2a1XbZltdvxPVU1s3ThlgrAaiRZONUr4bai7bIt22U7wG3ZrLbLtqzXdngISJKaMgCS1FSnABwcPcAa2i7bsl22A9yWzWq7bMu6bEebcwCSpJN12gOQJJ3AAEhSU60CkOTdSf4pyZ1Jbkry3aNnWqkkv5fkC9Pt+bMkLxo900ok+YUkdyf5epIt+XS9JBcnuTfJfUnmRs+zUkmuSfJokrtGz7IaSc5NckuSe6b/tvaPnmmlkjw3yd8n+cfptvz2mt5/p3MASb69qr46vfwW4PyqeuPgsVYkyU8Bn66qJ5P8LkBVvWPwWKctyfcCXwf+EPj1qtpSH/mWZAdwFNgLPAzcDryuqu4ZOtgKJHkF8Djwx1V1weh5VirJWcBZVXVHkhcCh4Gf3aJ/JwGeX1WPJ3k28Flgf1V9bi3uv9UewFMP/lPPB7Zs/arqpqp6cnr1c8A5I+dZqao6UlX3jp5jFS4E7quq+6vqCeDjwKWDZ1qRqvoM8O+j51itqvpSVd0xvfw14Ahw9tipVqYWPT69+uzp15o9brUKAECS9yR5CPgl4LdGz7NGfhX4i9FDNHU28NAJ1x9miz7YbEdJZoGXAbcNHmXFkuxIcifwKDBfVWu2LdsuAEn+Ksldp/i6FKCq3llV5wLXAm8aO+239kzbMr3NO4EnWdyeTWk52yGttSQvAK4H3rpk739Lqar/raofZHEv/8Ika3Z4buda3dFmUVWvWuZNrwVuBK5cx3FW5Zm2JcnlwGuAPbWJT+acxt/JVvQIcO4J18+ZLtNA0+Pl1wPXVtUNo+dZC1X1WJJbgIuBNTlRv+32AL6VJCd+cvelwBdGzbJaSS4G3g68tqr+a/Q8jd0O7E7ykiTPAS4DPjF4ptamJ04/BBypqveOnmc1ksw89Qy/JM9j8ckGa/a41e1ZQNcDL2XxWScPAm+sqi3521qS+4BvA/5tuuhzW/EZTUl+Dng/MAM8BtxZVT89dKjTlOTVwPuAHcA1VfWesROtTJKPARex+NbDXwaurKoPDR1qBZL8BPA3wOdZ/L8O8JtVdeO4qVYmyfcDH2Hx39azgD+pqnet2f13CoAk6RtaHQKSJH2DAZCkpgyAJDVlACSpKQMgSU0ZAElqygBIUlMGQFqF6fvO751e/p0k7x89k7Rc2+69gKQNdiXwriTfxeK7Tr528DzSsvlKYGmVkvw18ALgoun7z0tbgoeApFVI8n3AWcATPvhrqzEA0gpNP3rwWhbfWfbx6Tu0SluGAZBWIMkZwA3A26rqCPBuNvFnS0in4jkASWrKPQBJasoASFJTBkCSmjIAktSUAZCkpgyAJDVlACSpqf8DiWdZ3ji7YVYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "y = np.array([10.05, 1.5, -1.234, 0.02, 8.03]).reshape(-1,1)\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DiKgTcMojmsC"
      },
      "source": [
        "#### Polynomial Regression\n",
        "One class of functions that is covered by linear regression is the family of polynomials because we can write a polynomial of degree $K$ as\n",
        "$$\n",
        "\\sum_{k=0}^K \\theta_k x^k = \\boldsymbol \\phi(x)^T\\boldsymbol\\theta\\,,\\quad\n",
        "\\boldsymbol\\phi(x)= \n",
        "\\begin{bmatrix}\n",
        "x^0\\\\\n",
        "x^1\\\\\n",
        "\\vdots\\\\\n",
        "x^K\n",
        "\\end{bmatrix}\\in\\mathbb{R}^{K+1}\\,.\n",
        "$$\n",
        "Here, $\\boldsymbol\\phi(x)$ is a nonlinear feature transformation of the inputs $x\\in\\mathbb{R}$.\n",
        "\n",
        "Similar to the earlier case we can define a matrix that collects all the feature transformations of the training inputs:\n",
        "$$\n",
        "\\boldsymbol\\Phi = \\begin{bmatrix}\n",
        "\\boldsymbol\\phi(x_1) & \\boldsymbol\\phi(x_2) & \\cdots & \\boldsymbol\\phi(x_n)\n",
        "\\end{bmatrix}^T \\in\\mathbb{R}^{N\\times K+1}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rd8U1F_sjmsD"
      },
      "source": [
        "Let us start by computing the feature matrix $\\boldsymbol \\Phi$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "GvttnFECjmsD"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def poly_features(X, K):\n",
        "    \n",
        "    # X: inputs of size N x 1\n",
        "    # K: degree of the polynomial\n",
        "    # computes the feature matrix Phi (N x (K+1))\n",
        "    \n",
        "    X = X.flatten()\n",
        "    N = X.shape[0]\n",
        "    \n",
        "    #initialize Phi\n",
        "    Phi = np.zeros((N, K+1))\n",
        "\n",
        "    for j i \n",
        "    # Compute the feature matrix in stages\n",
        "    # first=np.ones(len(X))\n",
        "    # second=X\n",
        "    # third=X**2\n",
        "\n",
        "\n",
        "    # Phi = np.column_stack((first,second,third)) ## <-- EDIT THIS LINE\n",
        "    return Phi"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "K=2\n",
        "poly_features(X, K)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sQLBYtqKNiSi",
        "outputId": "2eb2d294-904f-457d-8cf4-f2219e14692a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 1., -3.,  9.],\n",
              "       [ 1., -1.,  1.],\n",
              "       [ 1.,  0.,  0.],\n",
              "       [ 1.,  1.,  1.],\n",
              "       [ 1.,  3.,  9.]])"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKOdRXxsjmsE"
      },
      "source": [
        "With this feature matrix we get the maximum likelihood estimator as\n",
        "$$\n",
        "\\boldsymbol \\theta^\\text{ML} = (\\boldsymbol\\Phi^T\\boldsymbol\\Phi)^{-1}\\boldsymbol\\Phi^T\\boldsymbol y\n",
        "$$\n",
        "For reasons of numerical stability, we often add a small diagonal \"jitter\" $\\kappa>0$ to $\\boldsymbol\\Phi^T\\boldsymbol\\Phi$ so that we can invert the matrix without significant problems so that the maximum likelihood estimate becomes\n",
        "$$\n",
        "\\boldsymbol \\theta^\\text{ML} = (\\boldsymbol\\Phi^T\\boldsymbol\\Phi + \\kappa\\boldsymbol I)^{-1}\\boldsymbol\\Phi^T\\boldsymbol y\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "iYEpMgvQjmsE"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def nonlinear_features_maximum_likelihood(Phi, y):\n",
        "    # Phi: features matrix for training inputs. Size of N x D\n",
        "    # y: training targets. Size of N by 1\n",
        "    # returns: maximum likelihood estimator theta_ml. Size of D x 1\n",
        "    \n",
        "    jitter = 1e-08 # good for numerical stability\n",
        "    \n",
        "    K = Phi.shape[1]  \n",
        "    \n",
        "    \n",
        "    theta_ml = np.zeros((K,1)) ## <-- EDIT THIS LINE\n",
        "    \n",
        "    return theta_ml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HCQTRM2mjmsF"
      },
      "source": [
        "Now we have all the ingredients together: The computation of the feature matrix and the computation of the maximum likelihood estimator for polynomial regression. Let's see how this works.\n",
        "\n",
        "To make predictions at test inputs $\\boldsymbol X_{\\text{test}}\\in\\mathbb{R}$, we need to compute the features (nonlinear transformations) $\\boldsymbol\\Phi_{\\text{test}}= \\boldsymbol\\phi(\\boldsymbol X_{\\text{test}})$ of $\\boldsymbol X_{\\text{test}}$ to give us the predicted mean\n",
        "$$\n",
        "\\mathbb{E}[\\boldsymbol y_{\\text{test}}] = \\boldsymbol \\Phi_{\\text{test}}\\boldsymbol\\theta^{\\text{ML}}\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "85vx1pVqjmsF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "cb984e3c-b92c-433f-f0a8-ed6a2ca0cad8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAi2UlEQVR4nO3dd3zV1f3H8dcHwpKtBGSDsgRH1ABuUISiogiO4sBdtFZLHa2rP21rtVRrsbVWpYri3ggVVCIOnMhGNsiQIJJEkCEzyfn9cS42Yi4mkHvP9+a+n49HHjfJDff7bmryzvec8z1fc84hIiJSmiqhA4iISHSpJEREJC6VhIiIxKWSEBGRuFQSIiISV0boABWpUaNGrk2bNqFjiIiklGnTphU45zJLe65SlUSbNm2YOnVq6BgiIinFzFbEe07DTSIiEpdKQkRE4lJJiIhIXCoJERGJSyUhIiJxqSRERCQulYSIiMSlkhARSXXvDYOl7yfkpVUSIiKp7LsCXxIrJyfk5VUSIiKpbMlEwEG7kxPy8ioJEZFUtiQHamdC06yEvLxKQkQkVRUX+TOJA3tBlcT8OldJiIikqq9mwpa1CRtqApWEiEjqWpIDGBx4UsIOoZIQEUlVi3Og+ZFQe7+EHUIlISKSijavhVXToH3vhB5GJSEikoq+eIdELn3dSSUhIpKKFudArX2h2eEJPYxKQkQk1RQXwxcToV0vqFI1oYdSSYiIpJpV0+C7fGjfJ+GHCl4SZtbSzN41s3lmNtfMhsY+v6+Z5ZjZ4thjw9BZRUQiYcHrUCUj4ZPWEIGSAAqBG5xznYGjgF+ZWWfgZmCic649MDH2sYiILBwPbY6DWon/2zl4STjnVjvnpsfe3wjMB5oD/YFRsS8bBZwZJKCISJTkL4KCRdCpX1IOF7wkSjKzNsDhwGSgiXNudeypr4EmoXKJiETGwnH+seMpSTlcZErCzOoArwC/cc5tKPmcc84BLs6/G2JmU81san5+fhKSiogEtGCc3/G1foukHC4SJWFm1fAF8Yxz7tXYp9eYWdPY802BvNL+rXNuhHMu2zmXnZmZmZzAIiIhbPwacqckbagJIlASZmbAY8B859zfSzw1Frg49v7FwJhkZxMRiZSF4/1jp9OSdsiMpB0pvmOBwcDnZjYz9rlbgWHAi2Z2ObACODdMPBGRiFgwHhq2hcYHJe2QwUvCOfchYHGe7pXMLCIikbV1Ayx7H7oNAYv3K7PiBR9uEhGRMlg4Hoq2w0FnJPWwKgkRkVQw51Wo3xJadE3qYVUSIiJRt3mt39Cvy4CE3cs6HpWEiEjUzf8vFBfCwWcl/dAqCRGRqJvzCux7ADQ9LOmHVkmIiETZxjWw/AN/FpHEVU07qSRERKJs3hhwxUGGmkAlISISbXNfhcadk3oBXUkqCRGRqPp2JXz5CXQZGCyCSkJEJKpmP+8fDzk7WASVhIhIFDkHM5+F1sfBvm2DxVBJiIhE0ZefwtqlkHV+0BgqCRGRKJr5DFSrDZ37B42hkhARiZrt38Hc0dDlTKhRJ2gUlYSISNTM/y9s3wRZF4ROopIQEYmcGU9DwzbQ+pjQSVQSIiKRsm6F34Yj64Ig23DsSiUhIhIl054AqwKHnRc6CaCSEBGJjsJtMP1J6HAKNGgZOg0QkZIws5Fmlmdmc0p87g9mtsrMZsbeTg2ZUUQk4eaNhc0F0PXy0Em+F4mSAJ4A+pby+eHOuazY2/gkZxIRSa4p//H3jTjgxNBJvheJknDOTQLWhs4hIhLM6tmwcjJ0vSLptyjdnegkKd01ZjY7NhzVMHQYEZGEmfoYZNQKvg3HrqJcEg8BBwJZwGrgvtK+yMyGmNlUM5uan5+fxHgiIhVk63qY/SIcchbUitbfw5EtCefcGudckXOuGPgP0C3O141wzmU757IzMzOTG1JEpCJMfwp2bPZDTRET2ZIws6YlPhwAzIn3tSIiKatoB3z6kN8SvNnhodP8SEboAABm9hzQE2hkZrnAHUBPM8sCHLAcuDJUPhGRhJnzKmzIhX5/D52kVJEoCedcaZcWPpb0ICIiyeQcfPxPyOwE7XqHTlOqyA43iYhUel+8A2vmwDHXRmrZa0nRTCUikg4+/ifU2R8OOSd0krhUEiIiIayeDUvfg6OugowaodPEpZIQEQlh0j1Qox4ceWnoJLulkhARSbbVs/3d5476JdRqEDrNbqkkRESS7f2/Qo36cNTVoZP8JJWEiEgyrZ4FC16Ho6+O/FkEqCRERJLrvWFQs74fakoBKgkRkWT5agYsHA9HX+OLIgWoJEREksE5ePsPULMBdE+dXYZUEiIiybBkor8uosdNKXMWASoJEZHEKyqECb+Hhm0juR347kRigz8RkUpt5jOQPx/OfRIyqodOUy46kxARSaRtm+Ddu6BldzjojNBpyk1nEiIiifTR/bBpDfz8GTALnabcdCYhIpIoBUvgo3/4XV5bdg2dZo+oJEREEsE5GH8DZNSCPneFTrPHNNyUyrZvhq+mw6Y82LIOtm2E2o2gXnNo0MqvpIjojUxEKr05r/glr6f+Deo2CZ1mj6kkUs2Wb2HWc7A4B1Z8BIVb439t7Uw4sBe07w0dfgY16iYtpkha27oe3roVmmZB9mWh0+yVSJSEmY0E+gF5zrmDY5/bF3gBaAMsB851zq0LlTGU4TmLuK53B3+W8OnD8PEDsG09NOoI2ZfDAT2hQUuo1RCq14HNBbA+F9YuhWWTYPEEmP08VK8Lh18I3X4B+x0Y+n+WSOU28U5/hn/e81Claug0e8Wcc6EzYGYnAJuAJ0uUxD3AWufcMDO7GWjonLtpd6+TnZ3tpk6dmvjASdTm5nEsH7wDxl0Pm7+BjqfBibfA/oeU7QWKi2DlZzB1JMwdDcWF0GUAnHwHNGyT0OwiaWnZJBh1OnS/Ck75a+g0ZWJm05xz2aU+F4WSADCzNsDrJUpiIdDTObfazJoC7znnOu7uNSpdSRQVMuKOixiSMQ6aHwmn3Astjtzz19v4NXw2Aj75N7giv3/M8TemxHbFIilh20Z46BiokgFXfQTV9wmdqEx2VxJRntVs4pxbHXv/ayB1Z37KaXjOIo68+Vkm//FYhmSM44nCPrT/YijD5+/lnELd/aHX7fDr6XDIufDxv+DB7rDwjYoJLpLucm6Hb1fCmQ+lTEH8lCifSXzrnGtQ4vl1zrmGpfy7IcAQgFatWh25YsWK5AROpC3r4Il+8M0XDN18Kf+4+y+JOc5XM2DMNbBmDhw6CPr+BfbZNzHHEqnsvngHnhrgtwH/WWoteU3VM4k1sWEmYo95pX2Rc26Ecy7bOZedmZmZ1IAJsW0TPHMOFCyC855lTPFxiTtWs8PhF+/6XSnnvAwPH+/nL0SkfL4rgNeuhkYd4KTfh05ToaJcEmOBi2PvXwyMCZglOXZshefPh1XT4eyRcOBJDO3VPrHHzKgOJ94KV7wNVTPg8VPgkwf9hUAi8tOKi2H0VbB5LZz1GFSrFTpRhYpESZjZc8AnQEczyzWzy4FhQG8zWwycHPu4cht3Ayx7H/o/CAedDuCXvyZDs8NhyPvQoa9f3/3iRf6sRkR275MHYEkO9L0bmh4aOk2Fi8R1Es658+I81SupQUL6/GWY+TSc8FvIivftSLBaDeDnT/sziZz/g8eX+XXe9VuEySMSdSunwMQ/Qef+/rqlSigSZxJpb91yeP06aNENetwcNosZHHMNnP8SrFsBI070Pwgi8kOb8uClS/w2OKf/MyV3eC0LlURoRTvglSsAg7Me9fMCUdD+ZLg8xy/jG9UP5r8eOpFIdBRuhxcG+wtcf/5Upb7WSCUR2gf3Qe4UOH04NGwdOs0PNe4EV0yEJl3gxcEw5bHQiUTCcw7G3wgrP4UzH4Smh4VOlFAqiZDWrYAP/g5dBsLBZ4VOU7rajeDi/0K73n5rkHfv1sonSWsTn7obpo+C466P7s9tBVJJhDTh92BVoM+doZPsXvXaMOhZOHwwvP9XeOMmv+xPJN0sfJMeX/wN2v+s0l0PEU9EBsDT0LJJMH8snHhbaqweqpoBZzwANevDJ//ye9Sc8UB05lBEEi13Krx0CfNcaw49e2TK7+5aVvoJD6GoEN68Beq3gmOuDZ2m7Mygz599Ubx7F2zf5C8eyqgeOplIQj0+Noczpl3KJleXy7b/joI73gdgaK/2ybuWKRCVRAjTR/n9ks4ZlXpXZ5pBj9/5e1e8dYuf0D73ScioETqZSGKsz+XSpddD7ersd/mbFNy7gOXDTgudKmk0J5Fshdtg0t+g1dH+ApxUdfTVcNp9sOhNv5XIji2hE4lUvA2r/b0htnwLF7ycljfsUkkk26znYeNX/q/xVL/4pusVfl5iyUR49uf+ntsilcWmPF8Qm/Lgwleg+REAid9PLWJUEslUVAgfDvf7JB1wYug0FeOIi/ze+csmwXODVBRSOWz8GkadARtWwQUvQctu3z9V2ecgdqWSSKZ5r8G6ZX59daqfRZSUdR4MeFhFIZXDuhUwsi98+yWc/wK0PiZ0oqBUEsninL9wrlEH6NQvdJqKd9ggFYWkvvxFfrv8LWvhojHQ9oTQiYJTSSTLorcgb64/i6hSSb/tJYtCk9mSanKnwuN9/X5ql4yHll1DJ4qESvrbKoI+/qe/LuKQs0MnSazDBsGZ/4al78ELF/obKYlE3bwx8MRpUKMuXPYm7H9w6ESRoZJIhvxFsOIj6Ho5VK0WOk3iZZ0fW/X0tr+OonBb6EQipXMOPn4AXrwY9j/Ub2iZhstcd0clkQzTR0GVapB1QegkyXPEYOh3Pyye4H8AC7eHTiTyQzu2wphf+T3UOveHi8f6DS3lB1QSibZjK8x8FjqdBnUyQ6dJruxLYxfcvQEvX+rHekWiYH2un3+Y+Yy/0dfZj6fe7gdJom05Em3B636lxJGXhE4SRtcroLgI3vgdvHwZnD0yPYbcJLqWvudv9LVjq9/duFP6bLGxJyJfEma2HNgIFAGFzrnssInKadoT0KA1tO0ROkk43a/0RfHWLfDqL2BghO7AJ+mjqNBvdT/pXr8U/ZKnILNj6FSRlyo/qSc65wpChyi3giWw/APodXvlXfZaVkdfDa7of/fQGDBCRSHJsz4XXr0SVnwIWRfCqff4+6TIT/rJn1IzywFudM7NSkKeymX6E1Alw/9HKX5b9OIiePuOWFE8kjZ78ktAs1+CcTdAcaH/b+6wQaETpZSy/Cl3E3B/bNjnVufc6sRG+hEHTDAzBzzinBuR5OPvmeIimPUCdOgLdZuEThMdx/3Gn1FM/JMvijMfUlFIYmxe68th7qvQopu/0FPLW8vtJ0vCOTcdONHMzgLeNLNXgXucc8m6nPY459wqM2sM5JjZAufcpJ1PmtkQYAhAq1atkhSpDJZ/CN/lVf6L5/bE8Tf49env3OkfBzysopCKNfc1GH8jbFnnbzN67HUa3txDZRooNzMDFgIPAdcCi81scCKD7eScWxV7zANGA912eX6Ecy7bOZedmRmhJaZzR0O1ffy9cOXHTrjRz9V8/iKMvtJPKorsrY1r4IXB8NLFUK8ZDHkPTvitCmIvlGVO4iOgLTAX+BS4BFgADDWz451zQxIVzsxqA1Wccxtj7/cB/pSo41WYokJ//+oOfaH6PqHTRNfxN/ghp7f/4IfnBo7Q8ljZM8VFMHUkTLwTCrdCrzvgmF+rHCpAWb6DQ4B5zjm3y+evNbP5CchUUhNgtD+RIQN41jn3ZoKPufeWvQ+bv4GDB4ZOEn3HXQdWFXL+D4q2+4uadM9sKY+vZsDr18NX0+GAnnDqfdCoXehUlUZZ5iTm7ubphF6F4pxbChyWyGMkxNzRUL0utOsdOklqOPbXULU6vHmT3+vpnFFQrWboVBJ13xX4BRDTn4TamXDWY3DwWZXrXi0RsFfnYrFf4lJS4XaY/1/odKp+0ZXHUVf5oaZx1/v7UQx6RuvYpXSF22HKo/DeMNjxHRz9K3874Jr1QyerlDRgV9GWvgdbv4UuGmoqt66XQ0ZNGHsNPDUAzn8RajUInUqiwjlYMM4PTa5dCgeeBH2H6arpBFNJVLS5r0KN+nBgJbmHdbIdfoE/g3jlChjVDwa/pp05BVZOgZzb4cuPoVFHuOBlaHeyhpaSQCVRkYoKYeF4v2FYRo3QaVJXlzN9UbxwIYz8GQweDQ0idA2MJE/+Ipj4R79RZu3GflfhIy7RqqUkSvMNhSrYysmwdT107Bs6Sepr39ufRXyXD4/1gTXzQieSZFq3Al67Gv7d3Q/hnngb/HqG31VYBZFUKomKtHiCv7nQARpqqhCtj4ZL3/DvP94XVnwSNo8k3vpVfiuNB46Ez1+G7r+EX8/0E9M16oROl5ZUyRVp8QT/i61mvdBJKo8mXeCyt+DpgfBkfxj4CHQZEDqVVLQNq+HD4X5rfVcEh18IJ/wO6jcPnSztqSQqyrcrIW8e9Plz6CSVT8PWcNkEeP58eOkS/70+5lpNWlYG61fBR/fDtFF+l9as8/2WLQ3bhE4mMSqJirJ4gn/UXk2JUXs/uGiM3+cp5/9g3XI45a/axiNVrVvhy2HG0+CK/fbdx98A+x4QOpnsQiVRURZP8Hega9Q+dJLKq1pNv23HxNbw0T/gm8X+6ux99g2dTMoqfxF8+HeY/aLft+vwC+C46/3ZokSSSqIi7NgKS9+HIwZrCCTRqlSB3n+CzE7w36HwaC8473ldUBV1q6b5OYf5r/sLJrtfCUdfozmHFKDVTRVh+YdQuAXa9wmdJH1knQ8Xvw7bNsJ/esG8sd8/NTxnUcBg8j3nYMnbMOp0+M9JsGySn2+4bg70/YsKIkWoJCrC4rcgoxa0OS50kvTSqru/X0BmB78xYM7tUFTIPyYuDp0svRXt8HdlfPg4ePosKFjsF3RcN9ffAEhX0KcUDTdVhMUToO0JUK1W6CTpp34Lfy3Fm7f4eYpV02mM7mEcxNb1fgnr5EdgwyrIPAj6/xsOOUfbv6cwlcTeWrfcvx11degk6SujBsNrXMWX26tz17KRvFljBpfduox3io9gaK/2XNe7Q+iEldvapTB5BMx4CrZvgjbHQ7/hfqv8KhqsSHUqib217AP/2Ob4sDnS3HW9O0DvYVBwOcv+eRYjq/8Nul8FPU8KHa1ycg6Wf+DPGhaM8/co7zLQb9vdLCt0OqlAKom9tfxD2KcRND4odBIBaNSegdv/yMIen8Lkh2HJRBjwMLTIDp2scti+2d+XfPIj/uLRWvv66xu6XgH1moZOJwmgktgbO/+aanOclr5GyFW9ukDvAf4e42Ovhcd6+/sd97xZ80Z76psvYMpjMPNpP/fQ5BDo/6C/E5y+p5WaSmJvrF3qJ+jaXh86iZTw/RzEgSfCLz+Gt271V/fOe81vNd3u5JDxUkfRDlj4Bkx7HL54B6pkwEFn+LOG1sfoD6M0EfmSMLO+wD+AqsCjzrlhgSP9z/Kd8xEnhM0h8dWsB/3/5VfYjLveL8nsMhD63OlXRsmPrV3mt8uY8TRs+hrqNYeet8KRF0Pd/UOnkySLdEmYWVXgQaA3kAtMMbOxzrlo3Fxg2QdQp4m24kgFB/TwZxUf3g8f3Of/Qj7mWjh2aNpvQT08ZxHX9WjhJ6BnPAXL3vdbZrTrDdn3+0fdwyFtRf3/+W7AEufcUgAzex7oD4QvCc1HpJ6MGtDzJsg6D97+I0y6B6aPguNv9H8lp9vdBIuLYeVkmrz/V5g6DbZt8HcAPPH3/op2XREtRL8kmgMrS3ycC3QPlOWHChbDpjVa+pqKGrSCsx/zS2Rzboc3fuvnLI6/HrIu9BsJVmZr5sHnL/mb+qz/kjOr1oBOZ/liaH2srm2QH4h6SfwkMxsCDAFo1SqJ90FePsk/ttV8RMpq2RUuHe+HV969298R7b1h0PUX0PXyyrN9hHOQvwDmvgZzR0PBQoqpyqSig3mtqB8TirPZPLkmTN7A0F5LdPGh/EDUS2IV0LLExy1in/uec24EMAIgOzvbJS3Zsg+gbjPtf5/qzOCAntC2h9+A7pN/wXt3++2suwyAIy6CVken3pBiUSHkToGF4/xcw9qlgPnh0W6/oErn/vSs05ieQJubx7F82GmBA0tURb0kpgDtzawtvhwGAeeHjURsPuJDaNcr9X55SOnM/OT2AT0gf6G/EO/zl2HWc7BfOzjkXOhyZrS3JF+f67esX5Ljl6xuXe/vud72BL8td6fTtDpJyi3SJeGcKzSza4C38EtgRzrn5gaOBd8sgc0FfvxWKp/Mjn7voT5/hnlj/FLQ9/7izzAyD4KOp8CBJ0HL7uE2rnMO1i2DlZ/Bl5/4s6C1S/1zdZpAp37QvrfPWbP+bl9qaC+tzpP4zLnkjdAkWnZ2tps6dWriDzTjGRhzNVz9qbbjSBcbVsP8sb40Vk7292OuVhtadvNvLbpC08Ogdmbcs8vhOYv2bLy/aIe/diF/AXw9G1bPgq9mwnd5/vka9fzFbW17QNvjoXEXTT5LuZjZNOdcqXvXRPpMIrJyp0CN+tAowkMPUrHqNfV3U+t+JWzd4Jc/f/EOfDkZJt3r79MMfi+jzE5+rqp+c6jXzP9lX7M+b74zi+uOrOY3w7OqgIPCbbBjC2z/DjZ/49++y/NDR9+uhG9X+DOE4kL/+lbVv367Xr6YWh3lP65SNdi3Rio3nUnsiYeOhTqNYfDoxB9Lom/bJvhqOqyZ6//az1vgf7lv/BrYw5+vWvv6K8IbtPIXazbq6G+u1Liz9kqSCqcziYq0baPf/bJTv9BJJCpq1PGTw7suhy7awaPjP2bsx7Ooa5upyxb2YStVrZg+nRrRu3MTf0fDajX90NU++8I++/mlt9Vrh/nfIrILlUR5rZrmhxZadA2dRKKuajWuOL0HV5zeA9BSU0lNmt0qr5VT/GOLI8PmEBFJApVEeeV+5seHazUMnURSjJaaSipSSZSHc35lU0sNNUn5absLSUUqifL4ZglsWQctuoVOIiKSFCqJ8siNzUe0VEmISHpQSZTHys90EZ2IpBWVRHnkTvGrmrTlgYikCf22K6udF9FpPkJE0ohKoqxWz/YX0TU/InQSEZGkUUmU1epZ/rFpVtAYIiLJpJIoq9WzoM7+ULdJ6CQiIkmjkiir1bOgWVboFCIiSaWSKIvt30HBQn9TGRGRNKKSKIs1c/2ktUpCRNKMSqIsvp+0VkmISHqJbEmY2R/MbJWZzYy9nRoszOqZsE8jqNc8WAQRkRCiftOh4c65v4UOwVez/FlEnBvci4hUVpE9k4iMHVshf76GmkQkLUW9JK4xs9lmNtLMwtzlJ28eFBeqJEQkLQUtCTN728zmlPLWH3gIOBDIAlYD98V5jSFmNtXMpubn51d8SE1ai0gaCzon4Zw7uSxfZ2b/AV6P8xojgBEA2dnZruLSxayeCTXrQ8M2Ff7SIiJRF9nhJjNrWuLDAcCcIEFWa9JaRNJXlFc33WNmWYADlgNXJj1B0Q5/IV335B9aRCQKIlsSzrnBoTOQvwCKtsP+mo8QkfQU2eGmSFgzzz/uf3DYHCIigagkdidvHlSpBvu1C51ERCQIlcTu5M2DRh2garXQSUREglBJ7M6aedCkc+gUIiLBqCTi2boeNuRCY5WEiKQvlUQ8efP9o0pCRNKYSiKeNXP9o4abRCSNqSTiyZsP1etC/Zahk4iIBKOSiCdvHjQ+SNtxiEhaU0mUxjlfEhpqEpE0p5IozcavYcs6aNwldBIRkaBUEqXJi01aNz4obA4RkcBUEqXR8lcREUAlUbo186BOE6i9X+gkIiJBqSRKkzdXZxEiIqgkfqy4CPIXQhNNWouIqCR2tXYZFG7VpLWICCqJH8tf4B8zVRIiIiqJXRUs8o+N2ofNISISAUFLwszOMbO5ZlZsZtm7PHeLmS0xs4Vm9rOkhSpYDHWbQs16STukiEhUZQQ+/hxgIPBIyU+aWWdgENAFaAa8bWYdnHNFCU9UsEhnESIiMUHPJJxz851zC0t5qj/wvHNum3NuGbAE6JaEQP5MolGHhB9KRCQVRHVOojmwssTHubHPJdamPNi2XiUhIhKT8OEmM3sb2L+Up25zzo2pgNcfAgwBaNWq1d69mCatRUR+IOEl4Zw7eQ/+2Sqg5N1+WsQ+V9rrjwBGAGRnZ7s9ONb/fF8SOpMQEYHoDjeNBQaZWQ0zawu0Bz5L+FELFkO12lC3WcIPJSKSCkIvgR1gZrnA0cA4M3sLwDk3F3gRmAe8CfwqeSub2kGVqHaniEhyBV0C65wbDYyO89xdwF1JDVSwGFp1T+ohRUSiTH8y77R9M6xfqfkIEZESVBI7rf0CcFrZJCJSgkpiJ61sEhH5EZXETgWLAYN9DwydREQkMlQSOxUsgoatoVrN0ElERCJDJbFTwSINNYmI7EIlAVBcDAVLVBIiIrtQSQBsyIXCLVrZJCKyC5UEaGWTiEgcKgmA6nWg42kqCRGRXYS+M100tDrKv4mIyA/oTEJEROJSSYiISFwqCRERiUslISIicakkREQkLpWEiIjEpZIQEZG4VBIiIhKXOedCZ6gwZpYPrNjDf94IKKjAOBUlqrkgutmUq3yUq3wqY67WzrnM0p6oVCWxN8xsqnMuO3SOXUU1F0Q3m3KVj3KVT7rl0nCTiIjEpZIQEZG4VBL/MyJ0gDiimguim025yke5yietcmlOQkRE4tKZhIiIxKWSEBGRuFQSpTCzG8zMmVmj0FkAzOxOM5ttZjPNbIKZNQudCcDM7jWzBbFso82sQehMAGZ2jpnNNbNiMwu+VNHM+prZQjNbYmY3h86zk5mNNLM8M5sTOstOZtbSzN41s3mx/w+Hhs4EYGY1zewzM5sVy/XH0JlKMrOqZjbDzF6v6NdWSezCzFoCfYAvQ2cp4V7n3KHOuSzgdeD2wHl2ygEOds4dCiwCbgmcZ6c5wEBgUuggZlYVeBA4BegMnGdmncOm+t4TQN/QIXZRCNzgnOsMHAX8KiLfr23ASc65w4AsoK+ZRel2lkOB+Yl4YZXEjw0HfgdEZkbfObehxIe1iUg259wE51xh7MNPgRYh8+zknJvvnFsYOkdMN2CJc26pc2478DzQP3AmAJxzk4C1oXOU5Jxb7ZybHnt/I/4XX/OwqcB5m2IfVou9ReLn0MxaAKcBjybi9VUSJZhZf2CVc25W6Cy7MrO7zGwlcAHROZMo6TLgjdAhIqg5sLLEx7lE4JdeKjCzNsDhwOTAUYDvh3RmAnlAjnMuErmA+/F/2BYn4sUzEvGiUWZmbwP7l/LUbcCt+KGmpNtdLufcGOfcbcBtZnYLcA1wRxRyxb7mNvwwwTPJyFTWXJK6zKwO8Arwm13OpINxzhUBWbG5t9FmdrBzLuh8jpn1A/Kcc9PMrGcijpF2JeGcO7m0z5vZIUBbYJaZgR86mW5m3ZxzX4fKVYpngPEkqSR+KpeZXQL0A3q5JF50U47vV2irgJYlPm4R+5zEYWbV8AXxjHPu1dB5duWc+9bM3sXP54Se9D8WOMPMTgVqAvXM7Gnn3IUVdQANN8U45z53zjV2zrVxzrXBDwsckYyC+Clm1r7Eh/2BBaGylGRmffGnuWc45zaHzhNRU4D2ZtbWzKoDg4CxgTNFlvm/0B4D5jvn/h46z05mlrlz9Z6Z1QJ6E4GfQ+fcLc65FrHfWYOAdyqyIEAlkSqGmdkcM5uNHw6LxLJA4F9AXSAntjz34dCBAMxsgJnlAkcD48zsrVBZYhP71wBv4SdhX3TOzQ2VpyQzew74BOhoZrlmdnnoTPi/jAcDJ8X+m5oZ+ys5tKbAu7GfwSn4OYkKX24aRdqWQ0RE4tKZhIiIxKWSEBGRuFQSIiISl0pCRETiUkmIiEhcKgkREYlLJSEiInGpJEQSKHZvhN6x9/9sZg+EziRSHmm3d5NIkt0B/MnMGuN3ND0jcB6RctEV1yIJZmbvA3WAnrF7JIikDA03iSRQbHfhpsB2FYSkIpWESIKYWVP81u79gU2xXXNFUopKQiQBzGwf4FX8/ZrnA3eSpHuAiFQkzUmIiEhcOpMQEZG4VBIiIhKXSkJEROJSSYiISFwqCRERiUslISIicakkREQkrv8HgckYWP/u+5kAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "K = 5 # Define the degree of the polynomial we wish to fit\n",
        "Phi = poly_features(X, K) # N x (K+1) feature matrix\n",
        "\n",
        "#@ Test\n",
        "assert np.any(Phi), \"Poly_features not implemented\"\n",
        "\n",
        "theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator\n",
        "\n",
        "#@ Test\n",
        "assert np.any(theta_ml), \"nonlinear_features_maximum_likelihood\"\n",
        "\n",
        "# test inputs\n",
        "Xtest = np.linspace(-4,4,100).reshape(-1,1)\n",
        "\n",
        "# feature matrix for test inputs\n",
        "Phi_test = poly_features(Xtest, K)\n",
        "\n",
        "#@ Test\n",
        "assert np.any(theta_ml), \"nonlinear_features_maximum_likelihood\"\n",
        "\n",
        "y_pred = Phi_test @ theta_ml # predicted y-values\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.plot(Xtest, y_pred)\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LBB5jXSUjmsH"
      },
      "source": [
        "Experiment with different polynomial degrees in the code above.\n",
        "#### Questions:\n",
        "1. What do you observe?\n",
        "2. What is a good fit?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OqsfY87jmsH"
      },
      "source": [
        "## Evaluating the Quality of the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-tuGiLu1jmsH"
      },
      "source": [
        "Let us have a look at a more interesting data set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "ViO-tOIDjmsI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "outputId": "d1d190ef-24e5-49f8-e791-b69fa64f7fe3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEJCAYAAACHRBAhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQ3UlEQVR4nO3df2xd513H8c+HpN0GAzqIp2ZJioNwENHWdZsVdeo/Za5R2k4JG1RKBaOFTflnGSGaBC6RVtGBFDQJL0C1KerKClTrprFSUwcyNy1UAjbilLTkxxKbqCEJhXhtN5gGq7J++eMeh5uba/tx7o/n3HPfL8nKPfccH3+b9J6Pnx/nPI4IAQCwlB/IXQAAoDcQGACAJAQGACAJgQEASEJgAACSEBgAgCSlCAzbD9u+YPvoAvtt+w9tz9p+wfa7u10jAPS7UgSGpM9L2rzI/tslDRVf2yV9pgs1AQDqrMxdgCRFxLO2Bxc5ZKukP43aXYZfs32d7dUR8dJC37Bq1aoYHFzslACARocPH/5mRAw021eKwEiwRtLZuu1zxXsLBsbg4KCmp6c7XRcAVIrtMwvtK0uXVFvY3m572vb03Nxc7nIAoFJ6JTDOS1pXt722eO8yEbEvIoYjYnhgoGmLCgBwlXolMCYk/UoxW+pmSd9ebPwCANB+pRjDsP0FSbdKWmX7nKT7JV0jSRHxWUn7Jd0haVbSdyX9ap5KAaB/lSIwIuLuJfaHpI92qRwAQBO90iUF9JTxqVO5SwDajsAAOmDvwZncJQBtR2AAAJKUYgwDqILxqVOXtSwGxyYlSTtHhrRrdMOyz7Xc7wE6jcAA2mTX6IZLF/nBsUm9uOfOqz7X3oMzBAZKhy4pAEASWhhAB+wcGVr297SzS6tVdImhGdducaie4eHh4OGD6FWtdmn1+s9HPrYPR8Rws310SQEAktAlBZTQ1XRptapMXWIoJ7qkAFyBLqn+RZcUAKBlBAaAK+ToEkP5ERgArsCYBZohMAAASQgMAEASAgMAkITAAAAkITAAAEkIDABAEgIDAJCEwAAAJCEwgAoanzqVuwRUEIGBSur3C2b9U2eBdiEwUElcMIH2Yz0MoCJYzwKdxnoYqIzGC+a8frxgsp4FrtZi62HQwkBl7BrdcCkYuGAC7ccYBlBBrGeBTiAwUEn9fsHsty44dAeBgUriggm0H4EBAEhCYABN9PuNf0AzBAbQBDf+AVciMAAASbgPAyhwpzSwOO70Bprgxj/0q8Xu9KZLCgCQpBSBYXuz7ZO2Z22PNdl/r+0520eKr4/kqBP9o99v/AOayT6GYXuFpAcljUo6J+mQ7YmION5w6BcjYkfXC0RfYswCuFIZWhibJM1GxOmIeE3SY5K2Zq4JANCgDIGxRtLZuu1zxXuNfsH2C7a/bHtdd0oDAMwrQ2Ck+CtJgxFxo6QpSY80O8j2dtvTtqfn5ua6WiAAVF0ZAuO8pPoWw9rivUsi4uWI+F6x+ZCk9zQ7UUTsi4jhiBgeGBjoSLEA0K/KEBiHJA3ZXm/7WknbJE3UH2B7dd3mFkknulgfAEAlmCUVERdt75B0QNIKSQ9HxDHbD0iajogJSb9ue4uki5JekXRvtoIBoE9xpzcA4BLu9AYAtIzAAAAkITAAAEkIDABAEgIDAJCEwAAAJCEwAABJCAwAQBICAwCQhMAAACQhMAAASQgMAEASAgMAkITAAAAkITAAAEkIDABAEgIDAJCEwAAAJCEwAABJCAwAQBICA6U0PnUqdwkAGhAYKKW9B2dylwCgAYEBoO1oIVbTytwFAPPGp05d1rIYHJuUJO0cGdKu0Q25ysJV2Htwhn+zCiIwUBq7RjdcusgMjk3qxT13Zq4IQD0CA0Bb0EKsPgIDpbRzZCh3CVgmWojVx6A3SonfSIHyITAAtB0txGoiMAC0HS3EaiIwAABJCAwAQBICAwCQhMAAACQhMAAASQgMAEASAgMAkKQUgWF7s+2TtmdtjzXZ/wbbXyz2f932YIYyAaCvZQ8M2yskPSjpdkkbJd1te2PDYR+W9GpE/JSkcUm/390qAQDZA0PSJkmzEXE6Il6T9JikrQ3HbJX0SPH6y5JGbLuLNQJA3ytDYKyRdLZu+1zxXtNjIuKipG9L+vGuVAcAkFSOwGgb29ttT9uenpuby10OAFRKGQLjvKR1ddtri/eaHmN7paQflfRy44kiYl9EDEfE8MDAQIfKBYD+VIbAOCRpyPZ629dK2iZpouGYCUn3FK9/UdLTERFdrBEA+l72Ffci4qLtHZIOSFoh6eGIOGb7AUnTETEh6XOS/sz2rKRXVAsVAEAXZQ8MSYqI/ZL2N7z3ibrX/yvprm7XBQD4f2XokgIA9AACAwCQhMAAACQhMAAASQgMAEASAgMAkITAAAAkWTIwbE/Zfmc3igEAlFdKC+O3JH3a9p/YXt3pggAA5bRkYETEcxHxs5KelPQ3tu+3/abOlwYAKJOkMYxisaKTkj4j6WOSZmx/qJOFAQDKJWUM4+9Ve7z4uGoLGd0r6VZJm2zv62RxAIDySHn44HZJx5s8Tvxjtk90oCYAQAktGRgRcWyR3Xe2sRYAQIm1dB9GRJxuVyEAgHLjxj0AQBICAwCQhMAAACQhMAAASQgMAEASAgMAkITAAAAkITAAAEkIDABAEgIDHTE+dSp3CQDajMBAR+w9OJO7BABtRmAAAJKkPN4cSDI+deqylsXg2KQkaefIkHaNbshVFoA28ZXLXFTD8PBwTE9P5y6jbw2OTerFPTz9Hug1tg9HxHCzfXRJAQCSEBjoiJ0jQ7lLANBmBAY6gjELoHoIDABAEgIDAJCEwAAAJCEwAABJCAwAQBICAwCQJGtg2P4x21O2Z4o/37LAcd+3faT4muh2nQB6S6tPS+Zpy83lbmGMSToYEUOSDhbbzfxPRNxUfG3pXnkAelGrT0vmacvN5Q6MrZIeKV4/Iunn85UCAFhM1ocP2v5WRFxXvLakV+e3G467KOmIpIuS9kTEXy51bh4+CPSXxqclz0t9WnKr318Viz18sOOBYfspSdc32bVb0iP1AWH71Yi4YhzD9pqIOG/7JyU9LWkkIv61yXHbJW2XpBtuuOE9Z86cadN/BYBe0urTkvv5acuLBUbH18OIiNsW2mf7P22vjoiXbK+WdGGBc5wv/jxt+28lvUvSFYEREfsk7ZNqLYw2lA8AKOQew5iQdE/x+h5JTzQeYPsttt9QvF4l6RZJx7tWIYCe0+rTknnacnO5A2OPpFHbM5JuK7Zle9j2Q8UxPyNp2vbzkp5RbQyDwAAqrNVpra2OOfTTmMVyZF2iNSJeljTS5P1pSR8pXv+DpHd0uTQAGe09OMNFu4RytzAAAD0iawsDAOY1TmsdHJuU1H/TWsss630YncR9GEDv6udprbktNq2WLikAQBICA0DpMK21nAgMAKXDmEU5ERgAgCQEBgAgCYEBAEhCYAAAkhAYAIAkBAYAIAmBAQBIQmAAAJIQGACAJARGRbW6AA2A3tWpzz+BUVH1j4kG0F869fknMAAASVgPo0IaF6CZxwI0QPW16/O/2HoYBEZFsQANkM/41Kmsv6S18vlnASUA6KKqjiESGBXFAjRA/+rU558uKQBog6qMITKGgWXL3QcL9LJeHkNkDAPLVtU+WABXj8AAgDar6hgiXVId0otdOlXpgwVw9RbrklrZ7WL6xd6DMz13kd01uuFSzb3cBwugM+iSAgAkoYXRRo1dOoNjk5J6s0unqn2wAK4eYxgdQpcOgF7EtFoAQMsIjA6hSwdA1RAYHdJrYxYAsBQCAwCQhMAAACQhMAAASQgMAECSrIFh+y7bx2y/brvpvN/iuM22T9qetT3WzRoBADW5WxhHJX1Q0rMLHWB7haQHJd0uaaOku21v7E55AIB5WR8NEhEnJMn2YodtkjQbEaeLYx+TtFXS8Y4XCAC4JHcLI8UaSWfrts8V713B9nbb07an5+bmulIcAPSLjrcwbD8l6fomu3ZHxBPt/FkRsU/SPqn2LKl2nhsA+l3HAyMibmvxFOclravbXlu8BwDool7okjokacj2etvXStomaaLTP3R86lSnfwQA9JTc02o/YPucpPdKmrR9oHj/bbb3S1JEXJS0Q9IBSSckfSkijnW6tmZLlQJAP8s9S+pxSY83ef/fJd1Rt71f0v4ulgYAaMCKe3WqtGIeALQbK+4tgBXzAPQjVtwDALSMwFgAK+YBwOUIjAXkHrNgWi+AsiEwSoppvQDKhsAAACRhWm2JMK0XQJkxrbakmNYLIAem1QIAWkZglBTTegGUDYFRUoxZAP2rrNPqCQwAKJmyTqsnMAAASZhWCwAl0AvT6plWCwAlk3NaPdNqAQAtIzAAoGTKOq2ewACAkinLmEUjAgMAkITAAAAkITAAAEkIDABAEgIDAJCksjfu2Z6TdKaFU6yS9M02ldNO1LU81LU81LU8VazrJyJioNmOygZGq2xPL3S3Y07UtTzUtTzUtTz9VhddUgCAJAQGACAJgbGwfbkLWAB1LQ91LQ91LU9f1cUYBgAgCS0MAEASAmMJtj9uO2yvyl3LPNuftP2C7SO2v2r7bSWo6VO2v1HU9bjt63LXNM/2XbaP2X7ddtYZLbY32z5pe9b2WM5a6tl+2PYF20dz11LP9jrbz9g+Xvwb7sxdkyTZfqPtf7L9fFHX7+SuaZ7tFbb/2faT7T43gbEI2+sk/Zykf8tdS4NPRcSNEXGTpCclfSJzPZI0JentEXGjpFOS7stcT72jkj4o6dmcRdheIelBSbdL2ijpbtsbc9ZU5/OSNucuoomLkj4eERsl3SzpoyX5O/uepPdFxDsl3SRps+2b85Z0yU5JJzpxYgJjceOSflNSqQZ6IuK/6jZ/SCWoLyK+GhEXi82vSVqbs556EXEiIk7mrkPSJkmzEXE6Il6T9JikrZlrkiRFxLOSXsldR6OIeCkinite/7dqF8I1eauSouY7xeY1xVf2z6HttZLulPRQJ85PYCzA9lZJ5yPi+dy1NGP792yflfRLKkcLo96vSfrr3EWU0BpJZ+u2z6kEF79eYXtQ0rskfT1zKZIudf0ckXRB0lRElKGuT6v2S+7rnTj5yk6ctFfYfkrS9U127Zb026p1R2WxWG0R8URE7Ja02/Z9knZIuj93TcUxu1XrRni00/Ustzb0LttvlvQXkn6joYWdTUR8X9JNxXjd47bfHhHZxoBsv1/ShYg4bPvWTvyMvg6MiLit2fu23yFpvaTnbUu17pXnbG+KiP/IWVsTj0rary4ExlI12b5X0vsljUSX52sv4+8rp/OS1tVtry3ewyJsX6NaWDwaEV/JXU+jiPiW7WdUGwPKOWngFklbbN8h6Y2SfsT2n0fEL7frB9Al1URE/EtEvDUiBiNiULWug3d3KyyWYrt+wd+tkr6Rq5Z5tjer1hTeEhHfzV1PSR2SNGR7ve1rJW2TNJG5plJz7Te2z0k6ERF/kLueebYH5mcC2n6TpFFl/hxGxH0Rsba4Zm2T9HQ7w0IiMHrVHttHbb+gWrdZGaYa/rGkH5Y0VUz3/WzugubZ/oDtc5LeK2nS9oEcdRSTAnZIOqDa4O2XIuJYjloa2f6CpH+U9NO2z9n+cO6aCrdI+pCk9xX/Xx0pfoPObbWkZ4rP4CHVxjDaPo21bLjTGwCQhBYGACAJgQEASEJgAACSEBgAgCQEBgAgCYEBAEhCYAAAkhAYQBcVazuMFq9/1/Yf5a4JSNXXz5ICMrhf0gO236rak1e3ZK4HSMad3kCX2f47SW+WdGuxxgPQE+iSArqoeBLyakmvERboNQQG0CW2V6v2OPqtkr5TPOEX6BkEBtAFtn9Q0ldUW5/6hKRPqgtrmADtxBgGACAJLQwAQBICAwCQhMAAACQhMAAASQgMAEASAgMAkITAAAAkITAAAEn+DzVyavULb08lAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "def f(x):   \n",
        "    return np.cos(x) + 0.2*np.random.normal(size=(x.shape))\n",
        "\n",
        "X = np.linspace(-4,4,20).reshape(-1,1)\n",
        "y = f(X)\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bwQHT4r9jmsI"
      },
      "source": [
        "Now, let us use the work from above and fit polynomials to this dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "5rSGbiz7jmsJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "25092e41-ca8d-4cda-c8ca-7e9f1bef1f09"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAABZLklEQVR4nO3dd3hUVfrA8e9J74UUUiEB6VWq0gTpgqCgYl2wsWJ3RRdX/WFfLLv2hmsBe0cUlI4gPYTeQiAB0jvpdc7vj5sZUibJhGRyJ8n5PE8ekjt35r4TMvPOae8RUkoURVEUpSF2egegKIqitA4qYSiKoigWUQlDURRFsYhKGIqiKIpFVMJQFEVRLOKgdwDW5O/vLyMiIvQOQ1EUpdXYu3dvhpQywNxtbTphREREEBUVpXcYiqIorYYQ4kxdt6kuKUVRFMUiKmEoiqIoFlEJQ1EURbGIShiKoiiKRVTCUBRFUSyiEoaiKIpiEZUwFEVRFIuohKEoOigzlPFDzA+UGcr0DkVRLKYShqLoYOPZjTy741m2JmzVOxRFsZhKGIqig93JuwGIOx+ncySKYjmVMBRFB7tTVMJQWh+VMBSlhaUUpBCfGw9AXK5KGErroRKGorSwPSl7ABgQMIC483FIKXWOSFEsoxKGorSwXcm78Hb2ZnLEZPJK88gsztQ7JEWxiEoYitKCpJTsTtnNsKBhdPXuCqhxDKX1UAlDUZqRQRooKCuguLzY7O0JeQkkFyQzLGgYkd6RAKbxDEWxdW16AyVFaSm/xP7Ci7tepKi8CABPR09WXLOCQLfAauftStkFwLDgYXR074irg2utFsaf5/5kSNAQ3B3dWyZ4RbGQamEoSjNYf3Y97o7uLBiwgIcGPURxRTFvRb9V67zdybsJcA0g0isSO2FHZ6/O1RLGyeyT3L/xfj49/Gm915u/dj6v7Xmt2Z+HotTHJhKGEOITIUSaEOJwHbcLIcRbQohYIcRBIcSglo5RUepzNPMow4KGce/Ae7mr313c2utWVp5aydHMo6ZzpJTsStnFsOBhCCEAiPSKrJYwtiZqK7/XxK+pc/ZUSkEKO5J38MWxL9T4h9KibCJhAJ8BU+q5fSrQrfJrPvB+C8SkKBbJKMogrTCN3n69Tcfu7n83Ps4+vLrnVdMb//70/WQVZzE8aLjpvEjvSJLyk0xjHsZSIfG58ZzMOWn2etsStwEghODtfW9b5Tkpijk2kTCklFuArHpOmQksl5qdgI8QIrhlolOU+h3LPAZQLWF4Only38D7iEqNYk38Gt7f/z53rrkTH2cfRoeNNp0X6R2JRHIm9wx5pXnsS9vHrG6zsBN2rI1fa/Z625K2EegWyF397mLdmXUczjDbMFeUZmcTCcMCocC5Kj8nVB6rRQgxXwgRJYSISk9Pb5HglPbN2O3Uq0Ovasdnd59NV++uPLblMd478B4TOk/g55k/4+/qbzrHOFMqLjeOHUk7qJAVzOw6kyEdh7D2zNpa3VLlhnJ2Ju9kZMhI5vaei6+zL29Ev2HdJ6golVpLwrCYlHKplHKIlHJIQECA3uEo7cDRzKNEeEXg4eRR7biDnQNPX/40gwIH8d7493hlzCvVkgVAJ69OCATx5+PZmrgVTydP+gf0Z1LnScSdjyM2J7ba+YczDpNXmseI0BF4OHkwv/98diXvYkfSDqs/T0VpLQkjEQiv8nNY5TFF0d3RrKP08utl9rYtBz1ZNnVZtW6oqlwdXAl2D+Z0zmn+SvyLkSEjcbBzYHzn8dgJO9bEr6l2/rakbdgJOy4PvhyAG3rcQKhHKM/teI7MIrViXLGu1pIwVgJ/q5wtdRlwXkqZrHdQSvuQXZxNXmme2duyirNIKUihj18fs7e/ucH8wHVVkd6RbEncQkZRBgU5lwDg7+rP4I6Da3VLbU/cTl//vng7ewPgZO/EK2NeIaMog/s33E9hWWFjn56iWMwmEoYQ4mtgB9BDCJEghLhTCHGPEOKeylNWA6eBWOAj4F6dQlXakYS8BJ7b8Rzjvx/PPzb/w+w5dY1fNEakdyQFZQUA/L7b23S8ZrdUTnEOhzIOMSpkVLX79w/oz8tjXuZo1lEe3/I4OcU5fH70c6b9NI271t510XEpSk02sdJbSnlTA7dL4L4WCkdReH//+3x48EPshB1hnmFEpUZRVF6Eq4NrtfOMM6Sqdkm9vi6mWssiYtEqAB4a341HJnavdS3jwHcfvz7srPA0HZ/QeQJLdi/hqW1P8cqYVziWeQyJZEToiFqPcWWnK1k0bBEv7XqJsd+NpUJW4O3sTWJKIqUVpTjZO1U7//V1MWZjUZT62ETCUBRbcr7kPEsPLmV06GieuuwpTuacZMH6BexL28eIkOpv1kczj9LJsxOeThfe6B+Z2N30ZhyxaBXxS6bVe72ok9rLMPp4kOk+oCWY/4z9D/+37f+4/tfrCfcMx8vJi75+fc0+zk09b6KkvIT43Hhu6HEDZ/PO8tifj3H6/Gl6duhZ7dw3N5xUCUNpNJUwFKWG9WfWUy7LWTBwAR3dO+Lp5ImDcGB38m6zCaN/QP8mXe+5KVfhuvsA98y+h8tfiK6RYLrTx68PT/71JLtTdjOp8yTs7ezrfKx5feeZvnexdwG0ciM1E4aiXAyVMBSlht/jf6ezV2fTuISboxt9/fuaNj4yyinOIakgiRt73ljnYz00vluD13NxcOHZEc/WeXuQexBLJy7lj/g/GpWcOnl1wsnOiZjsGKDxXWWKUpNKGIpSRUZRBntS9nB3v7tN9Z4AhgYN5ZPDn5Bfmm9ab2Ec8K66wrumxr4R15Vg7O3smdal/q6tmhzsHOjq05WT2SdNsTSmq0xRarKJWVKKYivWxq/FIA1MjZxa7fjw4OFUyAqi06JNx45mVc6QqmMNxsVo7k/63Xy7mVoYitJUKmEoShV/xP9BN99udPXpWu34gIABONo5sjt5NwBlhjJ+PfUrl/hcgpeTlx6hWqS7b3fSi9LJLs6udtySrjJFqUklDKVdW7J7CcuOLKPcUE5KQQr70vYxNWJqrfNcHFwYGDiQ3Slawvjm+DecPn+ahwY91NIhN0o3Xy0xGLuljNSYhXIx1BiG0m7lFOfw5bEvAfg97nfTWMSUCPOV9ocGDeX9/e8Tdz6O9/a/x8iQkVwRdkWLxXsxuvtqiSEmO4ZhwcN0jkZp7VQLQ2mXXl8Xw5m8MwDc0P0GkguS+T7me/r69SXcK9zsfYYFDUMieWDjAxSXF/P4sMerDYzr5fV1dY9R+Ln40cGlQ517ayhKY6iEobRLb244yZlcLWHc1vs2fpn5C3N7z+WBQQ/UeZ/+/v1xsXfhTO4Zbu51M128u7RUuPWqr16VEIJuPt2IyVID30rTqYShtFvx5+OxF/aEeobi4+LDwqELay3Mq8rR3pEhQUPo4NKBewbcU+d5tqabbzdOnT9FhaFC71CUVk6NYSjtRs2Fa+9t24m9iw/vbIizeBD4+ZHPU1JRUq0UiB4aswivu293isqLSMhPoLNX5xaNU2lbVMJQ2o2aC9f6di4l0K03j0ywfMZQzQ2Q9NKYRXhVZ0p19uqMlJKi8iLcHN1aJFal7VBdUko7JTmbd7ZdfOLu6tMVgSAmO4aMogxuX3M7U3+aSklFid6hKa2MShhKu3TXWD+KyouI8Iqw+D71zUbSU0OL8FwdXOnk1YlN5zZxw683sDd1L1nFWWogXGk0lTCUdmnSAK3ia2dvy1sYluyepwdLxl+6+3bneNZxXBxceOfKdwA4nHnY2qEpbYwaw1DapfjceIBGtTBaszk95hDoFsi9A+/F09GTDi4dOJyhEobSODaRMIQQU4A3AXvgf1LKJTVunwe8CiRWHnpHSvk/a8RikAYWb1/MZcGXNbo6qNJ6nMk9g4u9C4FugfWe11ZKgg8PHs7w4OGmn/v49eFIxhEdI1JaI90ThhDCHngXmAgkAHuEECullEdrnPqtlPJ+a8djJ+z489yfONg5qITRhp3JPUMnr07Yifp7ZdtqSfC+/n35K/EvCsoKcHd01zscpZWwhTGMYUCslPK0lLIU+AaYqWdAwR7BJOcn6xmCYmVncs+0ixlSdenr3xeJNO3poSiW0L2FAYQC56r8nAAMN3PebCHEGCAGeERKec7MOQgh5gPzATp16nRRAYW4h3Dq/KmLuq9iG6SUbD63GQMGvJy8CHANIMI7AtBKkyfkJTCh84RGPaZFJcGlhNxEyDyl/Xs+AfKSoTATCrOgOAfKiqG8GMorp7UKO+3L0RWcPcDJA9w6gHsguAeAdxj4dALfzuAdDvVs0WqpPn59ADiScYShQUOb/HhK+2ALCcMSvwJfSylLhBB/B5YBV5o7UUq5FFgKMGTIEHkxFwv2COavxL+QUtpEcTml8Q5mHOTBTQ9WO/bsiGeZ1W0WSflJlMvyRrcwao1ZlBZCykFIPqB9pR6GjFgoK6h+npuf9uXaAbzCtMTg4AIOToAAaQBZAWVFUFoAJXmQcRLit0FRVvXHcnAB/+4Q2AuC+kPoIAgeAE6N61byc/UjxD1EzZRSGsUWEkYiULU8aBgXBrcBkFJmVvnxf8Ar1gwoxD2E4opiskuy6eDSwZqXUqwkNjsWgHfHv4uTvRNvR7/Nu/vfZXqX6aaig42eIVVaoL2Jx2+BszshaT8YyrTb3AMgqB8MGgH+3cCvq9Ya8AoFR5eLfyIVZVorJecMZJ+BjBhIOwZxW+Hgt9o5wk67dsRoiBgFnUeAi3eDD93Hv4+aKaU0ii0kjD1ANyFEJFqiuBG4ueoJQohgKaVxUGEGcMyaAQV7BAOQnJ+sEkYrFZ8bj7O9MyNDRmJvZw+D4O61d/PTyZ8orSgFsKyFkR0Px36Dk2vh7A6oKAV7JwgZBCPuh7BhEHIpeAaBNVqj9o7QIVL7qik/DRKjIXGvFtvuj2DHOyDsodPl0G0i9JgKAT3MPnRf/76sO7OOnOIcfFx8mj92pc3RPWFIKcuFEPcDa9Cm1X4ipTwihHgOiJJSrgQeFELMAMqBLGCeNWMKcQ8BIDE/kT7+fax5KcVK4s7H0dmrs5YsgOFBwxkUOIiPDn3EiJAReDl54ePsY/7OWXFw6Ac4+gukHtKOBfSCYfPhkvHam7Gja8s8kfp4BEKPKdoXaGMjCXvg1EYtwa1frH0F9ITe10DfWdWSR1+/vgAcyTzCyNCROjwBpbXRPWEASClXA6trHPu/Kt8/ATzRUvGEeGgJI7lAzZRqreLOx9GzQ0/Tz0IIFgxcwN1r7+a3U7/R26939fGp4lw4/AMc+BbO7dSOhV8Gk16AntPNf8K3NY4uEDla+5qwWOvKOr4ajq6AP1+GP5doLaOBN0Pf2drvAMHhjMMqYSgWsYmEYWu8nLxwc3AjKT9J71CUi1BaUUpCfgJTI6vvzW1sZUSnRV/ojkrcC1GfwuEfoaxQ+zQ+fjH0ux58zO+812p4h8Hw+dpXXor2HPd/DasXwpon8ehzLRFuQRzOOExGUQYfHPiAnck7+WzKZzZTlVexLSphmCGEIMQjhKQClTBao3N55zBIA5He1VsFVVsZEcWF8L+JkLAbHN2h33UweJ72CbwtzozzDILL79O+kg9C9HI48A19vZxYX5DMVT9so1QaqJAVrI1fy829bm74MZV2xxYW7tmkYHe1eK+1ijsfB1ArYVBWxPCzB3ktD67bsQwK0mDKy/DocZjxNoQObpvJoqbg/jDtNXj0OEN7XEuRgNG5OfySWUKkozcb49fqHaFio1QLow4hHiHsT9+vdxjKRTAmDNO02dJC2PM/2P42oiCNyeHD4YYl0H1KsyyCa7WcPZg54VVGFy7CP3EfbH+L8ekH+bQ0h5xNL+Bz+YPg4qV3lIoNUS2MOgS7B5NXmkd+ab7eoSiNFHc+jo5uHXET9rBrKbw1ENY9DR17w9zf4I410HNa+04WleyEHf7uAdB9Esz7jQkTXqNCCP7c+x680Q/+ekNbUKgoqIRRJ+NMKTWO0frEn48j0t4N3h4Cvz8Gft20JPG3X7QZRO2h2+ki9e59PUHuQWzoPRHChmrTct8aBHuXQUW53uEpOlMJow7B7hcW7ymth4zfTlzGESITDoKbL9z2M8z7DTpdpndorYIQgivDr2R71hEK5yyHeavAOxR+fRA+HAOn/9Q7REVHKmHUQbUwWo/X18VAbjL8eBcZn08jX0gies+GuzdD1ytVi6KRxncaT0lFCduStmmlRu5cBzcsh9I8WD4DvrlFWwFfw5/n/iSlIKXlA1ZajEoYdfB39cfRzlG1MGxdRTn5m9+Ed4bC0ZXEDb4NgMje14Gd+vO+GIM6DsLH2YcNZzdoB4SA3jN5q9dXMP7/4NQmePcy2PofKNfKrJRVlPHQpof46thXOkauWJt6RdXBTtgR7B6sWhi2LPkAfDSOpx2/0Lqc7t1BfNdRgJkptYrFHOwcGBs+li3ntlBWUWY6/t9NZ2H0o3D/bug2ATY8Bx+MgrM7SSlIoUJWkFGUoWPkbVNGUQY5xTn1npNemM7pnNNWj0UljHrU3EjpgQ0PcMvqW9iftl+/oBQoK2LPRw9Q/sFY0pLPsqD0ISIOzyPi1eP8eCgaVwdXOrp11DvKVm1S50nkleXx3M7nKDfUGOz2DoM5X8DN32kzqD6ZQuKmZwDIKsmq/WBKk9y6+lbGfT+Of2z+B1sSttT6/5BScu+Ge7l59c2kFaZZNRaVMOoR4n5htfe+tH1sTthMTFYMt/1+G09sfYL0wnSdI2yHEvfCh2MYmrgch0G3ELhoP78bhhO/ZDrxS6bh651DhFeE2sekiUaFjmLBgAWsiF1Bv/duJuKJnwFtm9qIRau0caPuk+HeHTBsPkknfwcg6/xZPcNuczKLMknMT6S3X2/2pu7lvg338fiWx6udsz1pO8ezjlNQVsBrUa9ZNR6VMOoR7BFMRlEGJRUlfHL4E3ycffhj9h/c3e9u1sav5eltT+sdYvtRUQYbX9TKeZQWaLOfZr4Drr7VTos7H6e6o5qBEIJ7B97LU8OfwtHzOCNH/4CT3yYW35rGknl5zB8bqp3o7AFXvULiEG3sKDsnDtY8eWE3QaVJTuacBOCBSx9g/XXruavfXaw7s46/Ev8ynfPx4Y8JdAvkzr538nvc7+xM3mm1eFTCqIexzPmOpB1sPreZm3rehJ+rHw8OepDpXadzIvuEvgG2F1mn4eNJsOUV6H8DLNiuzX6qZNw6tai8iOSCZJUwmtGcnnN47YrXiM2OxTlwDf/d+19e3PUiq06vqnZesr22CDLbwQm54x1YOg5Sj+gRcpsSkxUDQHff7jjaO7JgwAI6eXbi5d0vU1ZRxsH0g+xJ2cPfev+NewbcQ5hHGC/teqna2FNzUgmjHsapta9FvYazvTM39rzRdFuYRxgZRRkUlhXqFV77cOAb+GA0ZJ3SpnZe+wG4+lQ7xbh16pncM0ikae9upXlMipjEjpt3MDf4G3bevBM3BzdOn68+wJqYr22SWYKBwjmfQ0G6ljT2fKztc65clBPZJ/B39Tdt5OZk78Q/h/2T+Nx4vjr+FZ8c/gRPJ0+u634dLg4u/Gv4v4g7H8eyo8usEo9KGPUwLt47k3uGay65ptrue+GeWunrhPwEXWJr80oL4OcF8PPftb2r79kGvWfWebqUknf3vYuDnQP9/Pu1YKDtg52wY+GkPrg7uhPpHWmq12WUVJCEg51Wmi4r7FKtFRgxClb9A76fC0U5OkTd+p3MPkl33+p7yY8JG8Oo0FG8t/89Np7dyE09b8LdUdvTfXTYaMZ3Gs+yI8us8mFWJYx6dHTviJ2ww07YMbf33Gq3mRJGnkoYzS7jJHw0Hg58DVcs0lZqN7A3xedHP2dzwmYWDllIqEdoCwXaPtVMGGWGMtIK00xvbNnF2eARALf8ABOeheOrYOkVWll1xWLlhnJic2JrJQyAx4c+TmlFKU72Ttzcs3op+ieGPcFX077CzdGt2WOyiYQhhJgihDghhIgVQiwyc7uzEOLbytt3CSEiWiIuRztHIr0imRwxmXCv6m9YYZ5hgLb3gtKMjvwMS8dqpcdv+wnGPdFgkcBD6Yd4Pfp1xoWPq/XiUZpfpHckyQXJpk+wqQWpGKTBtOVrVnHl1Fo7Oxj1MNz+uzYI/vFEbQMnxSJncs9QZigzmzAivSN56rKneGLYE/i5+lW7raN7R9MH2uame8IQQtgD7wJTgd7ATUKI3jVOuxPIllJeArwOvNxS8S2/ajkvjHyh1nFvZ288nTxVwmguhgpY/wx8Pw8Ce8Pft1Yb2K5LTnEOj215jADXAJ4f+byaTtsCjJMKzuSeATDtTNnXX0sY2cXZ1e8QPgz+vkUrZrjiHvjtH9qsN6VeMdkXBrzNmd19NrO7z27JkPRPGMAwIFZKeVpKWQp8A9TsrJ4JGEdxfgDGixZ6Z/By8sLJ3snsbeGe4apLqjkU5cBXc+Cv17Vd74wF7xoQnRrN9b9dT2phKq+MeQVvZ2+rh6pApJeWMIzdUsYB7z7+fYAqLYyqPALhthUw4kGI+hiWXwMFalV4fWKyY3AQDnTx7qJ3KCa2kDBCgaof0xMqj5k9R0pZDpwH/DBDCDFfCBElhIhKT7fuwrpwz3A16N1Umafgf+Ph9GaY/jpc/SY4mE/QRhWGCj488CG3r7kdRztHvpj6BQMDB7ZIuAp08uqEnbAjLldLGEkFSdgJOyK9InF1cDWfMADsHWDS8zDrI0iM0mZRpRxuwchbl5jsGCJ9InG0d9Q7FBNbSBjNSkq5VEo5REo5JCAgwKrXCvMIIzE/kQpDhVWv02bF/6Uli8IsmLsShtxh0d2+PfEt7+x/hykRU/hu+nemT7ZKy3CydyLUI9TUwkjKTyLANQBHe0d8nX1rd0nV1P8GuH01GMrgk8kQo7aENScmO6bO7ii92ELCSASqjtCEVR4ze44QwgHwBjJbJLp6hHuGU24oJ6VQlXRutH1fat0S7gFw9wboPMLiu/4e9zs9fHuwZPQSPJw8rBejUqeqM6WS8pNMM9N8XXzrbmFUFToY7t4Efl3h6zmw+yNrhtvqnC85T0pBikoYZuwBugkhIoUQTsCNwMoa56wEjPNarwM2Sqn/aiDjTAQ18N0IUsLmJfDLvRAxUttroYPlfbSpBansT9/PxM4T1QC3jiK9IjmTe4YKQwVJ+UmmRa4dXDpYljAAvIJh3mroNhlWL4Q/ngCDwYpRtx4ns7WSICph1FA5JnE/sAY4BnwnpTwihHhOCDGj8rSPAT8hRCzwD6DW1Fs9GKfWqoFvC1WUazu3bf43DLhJm6dfY9V2Q4x7NEzsPNEKASqWivSOpKSihIT8BFILU00Jw9fFl+ySBrqkqnL2gBu/hGF/h53vwY93qDpUNDxDSi8OegcAIKVcDayucez/qnxfDFzf0nE1pKNbRxzsHFQLwxKlhdqU2ZNrYMxjMO7Ji9oJb/3Z9XT17koXH9uZOdIeGafW7kreRYWsMNVd6+DSgayiLKSUlrcA7exh6svazLh1/weFmTDnS3Dxslb4Ni8mOwYfZx8CXK07DttYurcwWjN7O3vCPMJUwmhIUQ58MYtnsvfw3Yh5cOVTDSaLnOIcntvxHPdvuJ+SCu0TZ2ZRJntT9zKh8wTrx6zUy5gwjFVTq3ZJlRpKKSxvZFkKIWDkQ3Dth3BmO3x2FeS33+0DjCVBbK3bVSWMJgr1DFVdUvXJT4PPpkNCFOt8AtjWQJvWIA38dPInrl5xNT+d/Ik/E/7kP1H/AWDjuY0YpEF1R9kAXxdffJx92JW8C6DaoDfUsRbDEgNuhJu+hYxY+HQqnG9/r63SilJO5tSuIWULVMJoonCPcM7lncMGxuBtT845bdpk1inKbvyS3IqiBvu334h+g8XbF9PFuwvfX/09t/W+ja+Pf82GMxtYf2Y9nTw72eQLqT2K9I6ksLwQgSDIPQjAVKDzohMGaNu/3vYz5KfCJ1O0tTrtSFRqFEXlRVwWfJneodSiEkYThXuGk1+Wz/mS83qHYluy4uDTq7TVvLet4Hz4YMBM2Yga1sWvY2ToSD6b8hndfLvxyKBH6OPXh6e3P83u5N1M6DzB5prp7ZWxWyrANcBUDcGYMBpci9GQzpfD3F+hrFBLGmnHmvZ4rcjWhK042zszLHiY3qHUohJGE9WcWptVnMWJrHa+sVJGrJYsSnK1BXmdhps2sa/vk2dKQQoJ+QmMDBlpSgqO9o68OuZVpJSUy3LVHWVDjCVCjOMXcKFLqskJAyBkoFa4UNhp3ZrtYFW4lJLN5zYzLGgYrg6ueodTi0oYTVS1am1BWQF3/HEH96y/R+eodJQeow1YVpRqNaFCLgUwdUXlluZSZjBfeC4qNQqAIR2HVDse7hXOy2Ne5tpLrqWPn1rVbSuMLYxqCcNZSxiZxebX1aYWpJJakGr5RQJ6aKvC7Z1g2dVtvkR6XG4cCfkJXBF2hd6hmKUSRhMZE8bZvLM8sfUJTp0/RUZRBuWGcp0j00HGSVg2HaRBSxZBfU03Vf3EWVf3XVRKFJ6OnmbHKMaEjeG5kc+p7igbYkwYVfcfcXN0w9XBtc4WxqKti1iwYUHjxvz8usLtq8DJvTJpHGhS3LZsy7ktgPb3botUwmgiVwdXAlwDWH50OZvObaJXh14A5JTk6BtYS8uI1boNDBUw9zcI7Fnt5qq/j7q6pfam7mVQx0HYN7D/hWIbQj1CmdVtFuM7j6923NfZfHkQKSUx2TGczD7J0cyjjbtYhy7ahxBnT1g+s812T21J3EI3324EewTrHYpZKmE0gzDPMPJK85jZdSZ39NUK6DVLH25rkXVaa1kYyrWByhrJAqr/Psz9btIL04nPjacwt7NVQ1Waj72dPc+OeLZWN2EHlw5m/4+zirPILc0FYEXsisZf0LezNibm4ArLZ7S5gfDc0lyiU6NttjsKVMJoFqNDRzMyZCRPX/508w76tQbnE2DZTCgv1l7MHWvufaWpOp3W3O/GOH6x+YCndeJUWkxdBQiNxQoDXQNZHbea0orSxj94hy7alr12jrBshtaybSO2J26nQlaohNHW3d3/bj6Y+AHO9s4XFi6VNH4eek5xDn9f93dSClpJ9du8VO1FW5yjzZvvWPeAdHZxtmmjenNrMaJSonB3dMdQHFLrNqV1qasAYXxuPAD3DLyH3NJcNp/bfHEX8OuqtWSlQeueymkblRa2JGzBx9mHfv799A6lTiphNDPjPHTjNNLGOJhxkO1J29mftr95g7KGwiz4/FrIS4ZbvjfNhqpLTkkOnb207qaaLYzX18Xw9aHNnM8OBeyJWLSKiEWreH1djJWCV6zJ2CVVc2A77nwcLvYuXHPJNQS6BvLLqV8u/iIB3bUPKSV5WtLIT2ti1PqqMFSwNXEro0JH2fQYnkoYzcy4TejFdEkZ90aua0qizSgt0LZUzTwJN34FnRpekZpdnI2/qz/ezt61Pn3+bZQf9s5pPDpmKgDxS6YRv2Qaj0xUK7pbI18XX7P1pOLOx9HZqzOOdo5c3fVqtiVuI72wCfWigvtrH1bykrUPL0Wttxv4QPoBckpybLo7ClTCaHaOdo54OXldVGmEpILKhFFkwwmjogy+m6ttsTn7f9B1XK1TDqYfJL80v9qx7JJsfJ19ze7Itjd1LwBDg4ZaL26lxZjKgxRVfw3EnY8zTcWdcckMKmQFq06vatrFOg3XyqOnn4Cvb4KyoqY9nk42nduEg50DI0NH6h1KvVTCsIIOLh0atydApZR8beyiSXV4rMlggBULIHadtv9275m1TskqzuK232/jq+NfVTueU5yDr4uv2d9NVGoUrg6u9PbrzUPju1n1KSjWZ24cr6SihMT8RFPC6OLdhf4B/fk+5vumr1nqeiXMWgpnd8IPd2j7rrQym89tZmjHoXg62fakD5UwrMDXxYJ9jc0wtTBstUtq3dNw6Hu48mkYPM/sKftS92GQBs7mnjUdKywrpLiiGB9nH7O/m2OZx+jj1wdHO0fVDdUGmKsndSb3DBJpShgAd/S9g7N5Z/n11K9Nv2jfWTD1FTixGlY9ou3s2ErEnY8jPjeecZ1qt9Ztja4JQwjRQQixTghxsvJf3zrOqxBC7K/8qrl9q83xcfa5qFZCcn4yYKMtjJ3vw453YNh8GP1onaftTdO6l5ILkk3HjIv2Orh0MDvlMiE/gU5enZo/ZkUX5irWGqfUVk0YV4ZfST//frx34D3TnicXM9HBdJ/h87XNuaKXa7s6thKbzm0CYGzYWH0DsYDeLYxFwAYpZTdgA3VvvVokpRxY+TWjjnNsRl0Ll+pTVlFGepE2AGhzYxhHVmj7LfecDlOW1Lv5UXRqNHBhAB8uTKP1cfbB19mX8yXnMUht7+ai8iIyijII8wizXvxKizK3J4YxYRhnygEIIXhw0IOkFKTw/YnvAXhzw8lGX6/afcY9CQNvhT9fhn1fXEz4LW7zuc306tDLZld3V6V3wpgJLKv8fhlwjX6hNB9fF19ySnIaVS8npSAFibzoAXOrObsTfpoP4cO0Qe56pvwVlhVyPOs4DnYOpBSmUGGoAC5MMTaOYVTICnJLtBW/iXmJwIWaXErr5+rgiquDa62EEeIeUqsC62XBlzE8aDgfHfqIgrKCpl9cCLj6DegyDn59CGI3NP0xrSizKJP9afsZGz5W71AsonfC6CilNPZdpAAd6zjPRQgRJYTYKYS4pr4HFELMrzw3Kj1dny0efZ19tTfFyjIIljB24fTx60NReRGFZY3c4tIask7DNzeDdxjc9A041l9ueX/6ftNK1XJDuanFZHzjMI5hwIUB0YR8bUc11cJoWy7xuYRtidtMH5ric+OrdUdVFVB+DVnFWVz6xtMAFq3DeX1djOm8Wvexd4QblkNAT21Gnw3XndqSsAWJZFy47Y9fQAskDCHEeiHEYTNf1abYSO0vq66P5J2llEOAm4E3hBBd67qelHKplHKIlHJIQIA+G6hfTHkQ44B3X3+twqvurYyibPjyBm017S3fg1uHBu8SnRqNnbBjSuQU4EISNI5h+Lr4mspfG383xu1tVQujbbmx542cPn+ax377HilltSm1Nf172nSuDL8Sn5C/sHNOsmgdziMTu5vOAzNrd1y84ObvtGKFX83RqhLYoE3nNhHkHkTPDrXrr9kiqycMKeUEKWVfM1+/AKlCiGCAyn/NLteUUiZW/nsa2AzUv6xYZ6ZZInVMrd1wZgM3/XYTeaV5pmPJ+ckIhKmQm64Jo7wUvr0NsuNhzpdaKQYLRKdF08O3B918tKmxxnGM7OJs7IU9nk6etZJpQn4C7o7u+Dj7NPvTUPQzJWIKfi5+rDr7HamFqRSVF9WZMACevvxpvJy8cA1f3nxjeN6hcNPXUJQF39jeGo3i8mJ2JO1gbNjYVlO2X+8uqZXA3Mrv5wK1agUIIXyFEM6V3/sDI4FG1kZuWeYG/Yy2J23nsS2PcTjzMPvS9pmOJxUkEeAaQEd3rVdOt4FvKWH1QojfCjPfhQjLFhKVVZRxMP0ggzsOJthdG7wztjCyS7LxdvbGTtjV+t0k5CUQ5hHWal4wimWc7J2Y03MODh4n2Hh2I0C9CcPf1Z+3rnwLR6dC/rH5H5RVmN9ky5x61+6EDIRZH0FitLaGyGCw+HGtLTotmuKKYkaHjdY7FIvpnTCWABOFECeBCZU/I4QYIoT4X+U5vYAoIcQBYBOwREpp0wmjrn2ND6Qf4OFND9PZqzN2wo7DGRf6VpPzkwnyCMLPxQ/QsYWx60OIXqZNnR0wx+K7Hck8QklFCYM6DsLN0Q1fZ19TCyOnOMf0OzHV2qrspkrIS1DdUW2McXzhle86IA0OvLTjDQDW7q//zbq3X2+WjH6R6LRoXtr9ksXXa3DtTq/pMPFZOPKzNnvKRuxK3oWDcKi1w6Qt0zVhSCkzpZTjpZTdKruusiqPR0kp76r8fruUsp+UckDlvx/rGbMljN0rNRcu3bv+Xvxd/flo0kd08e7CoYxDptuTC5IJcQ+pt3Vizp6UPdWmsDZJ7AZY8wT0mAbjnmrUXaPTtOm0lwZqvYXBHsEXuqRKsk2/Eyd7J9wd3U3F6RLyE6rt2Ka0fsbxhbgX51CWOxBhX4SHowf/mjyswftOiZzCrb1u5YeYH0grbMaCgiMehIG3wJ9L4GgTih5aqKSihDXxa+qdKbkzeSf9A/rj5uhm9Xiai94tjDbJxcFF26ayyhjGr6d+Jb8sn6UTl+Lv6k8//34czjiMlBKDNJBckEywRzAuDi64O7pbtNq7qLyIBesX8NGhj5oedEYs/HA7BPSCWR+CXeP+NKJTo4nwisDf1R+AEPcQ00B+dnG2KRHChR3ZMooyKKkoUS2MNqwsS+vSjPSOtLjbcXa32QAXX/7cHCG0cjZhQ+Hne6w+c+rHmB9Z+OdCDqSb3072fMl5jmUeY3jwcKvG0dxUwrCSmov3YnNi6eTZyfTm2Ne/LzklOSTkJ5BZlEmZoYwQd20vCD8Xv1qF28yJSomipKKE1IImzgApztUGBYW9Nkjo3Lh6NgZpYF/aPgZ1HGQ6FuwRTHJ+MlJKckpyqg1qG383akpt2/fAqDFc1/06pkRMsfg+XX26EuYRZloB3WwcnGHOF+Dirf29F1hvnHBb0jbgQsu7pj0pe5BILgtuuNKzLWkwYVSW7BjQEsG0JTWrssbmxNLN98LgnHGTlMMZh02fxEM8tIRR1wY0NW1P2g5ARlHGxQdqLCiYeQpuWKZtg9lIB9MPkluaW63abIh7CMUVxWQWZ5JTklO9heHiS3ZJtppS2w48MrE7iy9fzN/6/M3i+wghGNdpHLuSdzXPYr6qPIO06rZ5qfD9XKsUKiypKGFPyh5Aq61mzs7knbg6uNr0ZknmWNLC+Cfa2odPjVNglYZVrZlUXF7M2dyzXOJzien2S3wvwdnemUMZh0w1pILcgwAtYVjSJfVX4l8ApgVyF2Xrf+D4bzDpBYgcc1EP8dvp33C2d65WC8eY/GKyYjBIg2n9BVz43STkJSAQpnMVxWhc+DjKDGVsS9zW/A8eOlhbDR6/FTY80+wPH50aTVF5EUHuQexL32cqg1PVruRdDO44GEd7x2a/vjU1mDCklNFSynHAb8AfQojFQoj6l/wqpk/RoJVFkEi6+lxYz+Bo50ivDr04nHHYNP3U1CXl6tdgCyMxP5H43HjThkTGMhyNErMWNr0I/W6AyxY0eLq5lbdlhjLWxq9lbPhYPJw8TMeNSeBI5hEAfFx8TLcZK9aeyztHoFsgzvbOjY9dadMuDbwUb2fv5h3HqGrgzTD0btj+Nhz+sVkfelviNpzsnJjXZx7nS84Tfz6+2u0pBSnE58a3uu4osHAMQ2ijVSeA94EHgJNCiNusGVhrV3WbytgcbaN644I2o77+fTmWeYxzeefwdPI0veEa71tfEjB+8roq8ioM0lArwVQYKurfZyA7Hn66C4L6wtVv1ltQ0MhcYbgdSTvILslmWuS0aseNazGMCaNqC6ODcwfKDGWcyD6huqMUsxzsHBgTOoYtiVuavl9GXSa/BOHD4Zf7IfVIsz3stqRtDOo4iJEh2oB/zXGMXcm7ANpmwhBCbAMSgdeBUGAeMBYYJoRYas3gWjNfF19KKkooKi/iZM5JHO0cCfcKr3ZOP/9+FFcUszVxq6l1AVrCkMh6N2HanrSdYPdghgVpUxVrdkst/HMhT2x9wvydy4q1ldwAN3wOThc/re+307/h7ezNqNBR1Y57OXnh7ujO0UxtyUzNMQzQxnXUgLdSl3GdxnG+5Hy1Ba7N5fV1MeDgBNcv0yZ5fHubNvmjiVIKUojNiWVU6Cg6e3XGUO5eK/5dybvwdfatNqbZWljSwpgPhEopJ0opn5ZS/ialjJVSPgC0niWKLcz4iTqrOIvY7FgivSNxtKveX2kc8EopSKlW2tjPtf7Fe2WGMnYm72Rk6EjTNNaaA9+HMw+zJaGOT2erF0LKQbh2KXSoe/Ut1F/kraCsgE1nNzG58+RafbFCCILdg03dbTXHMECbXaVaGEpdRoaMxNHOsVGzpaSUrDuzzrS/Rl1MrWWvYLjuU63FvfL+Jm+8ZGz5P/ttBZFPrKaiqDMrjm0zvWaklOxK3sWw4GHYidY3SdWSMYwjsu7VJ9PqON7uGd8Uc0pyiM2JrTbgbRTmGYa3szdwoQsHLqyGrqs8yMH0gxSUFTAyZCQBblqBxaoJo8xQRlphGoXlhZzIPlH9ztGfw77PYfRC6NHwVMf6irxtPLuR4opipnUx/2dQdUFe1TEM4/MDNUNKqZuboxvDg4ez6ewmi7cKOJp5lH9s/gcrTq6w/EIRI2HCYm1B364PLi7YStuSttHRrSOnnr2d+CXTqCiMwM4pk6jFw3lkYncOpB8grSit1a2/MGpSiqssBqiYYUwYCXkJJBckm21+CiFM1Wmrdkk11MLYlrgNe2HP8ODhphZGeuGFLqm0wjTTzAzjhkaA1k+7eqE2G2rcv5rw7DSrTq8ixD2EgYEDzd6elqXNjTDuj2BUtXtKdUkp9RkbNpaE/ATO5p1t+GQwlduJSo2qdVu9JdFHPKhVOFj7FJzbXe81tidu54ujtTdnKjOUsSNpB6NCR5kWKVYURgCwL20fZYYynt/5PIGugY1al2JLWl+bqJXo4Kx9ijbOx+7qbb7iq7FbqlqXVD31pKSU/JX4FwMCBuDp5ImzvTNOwqPaGEbVUiGmhFGSr+0N4OINsz+udyOkulQt8pZRlMHO5J1c1eWqOpvW++K0F03NSrRVu6dUC0Opz7BgbYxud0r9b+JGxkkWUalRtVol9ZZEFwKueU/b++X726Gw9mvPIA18eOBD7ll/Dy/veZn1Z9ZXu/1g+kHyy/IZGXqhYOe9I8bgbO9MdGo0y44sIyY7hicvexJPp8YtjrUVKmFYifFTtPEP/RLf2l1SAMOChiEQ1bqsPJ08cRAOtbqkpJS8Hv06x7KOMaHzBNPxomK3al1SxnGDgQEDiU6LRhoMsOofkHVK2zXPI/CinlPVIm8rYldQISu4uuvVdZ4vy7TfQc2E4ebohou9Vj7FmBwVxZwIrwgCXAPYk7zHovOPZh5FIMgoyrC4VWLi6gPXfwb5qbDi3mrjGQVlBTy6+VHe2f8O07pMo2eHnry06yXTJmmFZYX8e9e/8XT0rDb7aeGkPvTz78emc5t4f//7TOw8kSs7Xdm4uGyIShhW4u7ojqOdI/G58bg6uNZZYG9o0FA23rCx2hoNO2FXa7W3QRr49+5/8+nhT5nTYw639LrFdJss96rWwkjM17Y9varLVWQVZxG38004+C1cseiiF+dVZZAGfoz5kcEdB9PFu0u126o2+w1lPgAcOlteaw2Hr4svoR6hqqy5Ui8hBEODhrI7ZXeD4xglFSWcyjllekPem7q3znPrLIkecqm2iDXmd9j5vunwa1GvsfHcRhYOWchLo17imRHPkFmcyRt738AgDTyx9QlO5pzklSteqdV6uDTwUhLzE3F2cOZfw5veFawnlTCsRAhhamV09e5a74wI4zhEVR1cLyQMKSXP7XiOr49/zdzec3ly+JO8uT7W9MYsyz3Zn3jW1B+bnJ+Mv6s/lwdfDkD0rje1RDFmYbM8t13Ju0jIT+D67tfXuq1qs9/YwpjZv3utEtRdfLqYNotSlPoMCxpGZnEmcefjTMdismO4ZdUt1cbuYrJiKJflXBV5FR1cOhCVUnscw6jekujD/66NZ6z7P0jUks6OpB2M7zSeuX3mIoS20dltvW7j+5jveXTzo2w8t5HHhjxWa3o5XOhWe2zIY2Zf662JShhWZJwNVFd3VEP3NXZJ7UnZw48nf+T2vrfz6JBHEUJUf2Mu98TZpYC4f1/FIxO7k1SQRIhHCJ1dO9JBCva6umqbyNQYt0jKT2L16dWNXiX+fcz3+Dj7VOsWM0dWeODp6EmgW+0usLfGvcXiyxc36rpK+2Rca7QrZZfp2LIjyziYcZC1Z9aajhnX/PT178vgjoPrbWFUdSTjiKlVDmjjGTPf0epO/XAHKVmxJOYnMihwULX73TvwXkI9Qll/dj2zu82u1uqvanjQcH655heu7XatRfHYMpUwrMjYd29uSm1D/FwulAdZfnQ5HVw6cN/A+8x24RjKPSkzlJn6U5Pztb01xPr/Y3BhAdE+Adoffw0fHvyQf279J3esucNUCLCm5PxknvrrKc7kngG0we5NZzcxo+uMBkt6PDS+O59N/Yw7+95Z6zYne6dWV0dH0UeYZxhB7kGmCSTZxdn8EfcHgGk3P9AGvH2cfQh2D2Zwx8EkFSQ1uFdMUXkRd6+9m8Xbanx4ceugTQ7JOUv0uscBqlVjBm0s7rUrXuOOvnfw5PAn6+xeFULU6rptrVTCsCJjl9TFJAxjAcLT50/zZ8Kf3NjjxjrfoKf01JrX6YXpF/bWKCmC3UsZHDSUpJJsUgpSat3vUMYhwjzCiMmOYfbK2fwSW3tjmS+OfcEvp37hltW3sCdlDytiV1Auy5ndfXaDz+GRid3p7tvdtNZEUS6GEIJhQcPYk7IHgzSwInYFpYZSrgy/kr2pe8kpzgG0FkZvv94IIUy72DXUylh/Zj15ZXnsSd1Tu+pzp+FwxSKi06Jxt3Oiu2/tbqy+/n15ZPAj7ebDj64JQwhxvRDiiBDCIISoc59CIcQUIcQJIUSsEGJRS8bYFKYuqYtJGK4dKKko4YMDH+Bkp+2PXJe/DdOm5qYXpZNRlKHtrXF8DQT1Z9AorTxIzRdOYVkhp3JOMa3LNH6c8SO9/Hrx1LanOH3+wtKaCkMFq+NWMyhwEH4ufsxfN59lR5YxpOOQNvOJSWkdhgYNJackh5jsGL478R2DOw5mfv/5VMgKtiRuMQ149/brDWivOU8nzwYTxorYFXg7e2OQBtadWVf7hNGPstfTlwGFBTjknLPGU2tV9G5hHAZmAVvqOkEIYQ+8C0wFegM3CSF6t0x4TTM6dDRTI6aa7cNviHG66e9xv3N116urrY6uKcD1wmrvpDytLzaktARmf0x3/754OHrUeuEczzqOQRro69+XEI8Q/nPFf3C0c+SrY1+ZztmVvIuMogxu7X0rn1/1OUM7ai9ac4PdimJNxnGMN6LfICE/gRt73Ehvv94EugWy8exG04C3MWHY29kzOLD+cYxzeefYnbKb23rdxiU+l5i6uao6X15ArChnUGkF/HiXVfbPaE303tP7mJTyRAOnDQNipZSnpZSlwDfATOtH13QjQ0fyyhWvXNTU0aoJoqHNZ4zlQdKL0kk+8DkAIcPvh4Du2NvZc2ngpab+XyPjiljjSnM/Vz+mRk5l5amVprGQX0//iqejJ2PCxuDl5MV7E95j2ZRlTI2c2ujnoyhNEeIRQqhHKNsSt+Hn4sf4TuMRQnBl+JVsS9xmqghrTBgAgzsOJj43vtpMqqpWxK5AIJh5yUwmR0xmX9q+WrtX7k/bD8CgIQsgMQq2vmadJ9hK6N3CsEQoULUtmFB5zCwhxHwhRJQQIio9vQkbC+nEuF6hg6uWMMaEjWmw+8fd0R1XB1cyMk6QdPQnAEKG32e6fWToSOJz4zmXd+HXeDjzMB3dOlab5ndrr1spKi/i55M/U1hWyIazG5gUMck0duJg58CgjoPU2glFF8ZWxqxus0xjBld2upLiimKWH12Ot7N3tRI7gzsOBsyXCakwVPBL7C+MCB1BkHsQUyKmIJHVZl0B7E3bi4OdA/2G3gv9b4Q/X4GEuqfrtnVWTxhCiPVCiMNmvqzSSpBSLpVSDpFSDgkICLDGJazKWEUz0iuSy4Mv576B9zVwD02Aqz8Zp9aS5OyKt5MXbk7upttGh2pFhY079IE2QGhsXRj18uvFoMBBfH38a9adWUdReVG9K7kVpSVN6DwBTyfPal2iQ4KG4OnoSVphGn38+lT7MNPLrxcBrgGsPLWy1mPtTN5JamEq116iTXWN8I6gZ4ee/BFfvVsqOjWavn59cXFwgateAa9Q+OlurdROO2T1hCGlnCCl7Gvmq/aUHPMSgaobSYRVHmvT3BzdWDppabUmdn38SwpJL8snKaQvITVWlXfy6kQnz05sTdgKQG5pLmdyz5hdOHdLr1tIzE/kv3v/S6hHKJcGXtr0J6MozWBM2Bi23bitWt01RztHxoRr1QtqvlYc7ByY1W0W2xK31Zo2/nPsz3g7ezMufJzp2OSIyRxMP2iailtcXsyRzCNc2rHyNeDiDdd+AFlxsKZ1r9i+WK2hS2oP0E0IESmEcAJuBGp/ZGjF6q2iaYkzOwjIOkeGuy/JsszsHtmjw0azO2W39iLI0Aq09fGvnTCu7HQlQe5BZBVnMa3LtFZZs19pu8x1h47vNB6Avn59a912XffrEELw48kL27DGZsey4cwGru5yNU72TqbjkyMmA9rYhpSSQxmHKDeUMzhw8IUHjBgJIx+E6GXaFsftjN7Taq8VQiQAlwOrhBBrKo+HCCFWA0gpy4H7gTXAMeA7KWXz7adoA+qtotmQknxYcQ8BDm6k2wmSCpKq7a1hNDp0NCUVJexJ2WOq6GmuheFg58CtvW7FTthxdRfVHaXYvvGdxvOfK/7DFeFX1LotyD2IMWFj+OnkT5RVlCGl5KXdL+Hm6Mb8/vOrnRvuGc7QoKG8f+B9rvnlGj488CECUbt8/7gnIbA3rHzAbFXbtkzvWVI/SynDpJTOUsqOUsrJlceTpJRXVTlvtZSyu5Syq5TyRf0itkFrn4LsM/j3vpaC8kKKyovMtjCGBA3Bxd6FrYlbOZxxGEOpX50L6m7rfRu/XvMrEd4RVg5eUZrOTtgxKWISDnYOZm+f02MOWcVZbDi3gd/jfmdPyh4eGvRQtX1ZjN6+8m2eHfEs3s7e7ErZRc8OPWu/Thycta6pwgxY/Zg1npLNMv8bVnRTZxVNc2I3wN5PYcQDBIQMgdM/A9U3YzJytndmWPAwtiZspcxQRkVR3ftQ2Ak7Onl1anTsimKLRoSMINQjlM+Pfk5yfjK9/Xozu5v5SgXuju7M6jaLWd1mcS73XLUuq2qCB8AV/4RNL0Kv6dCn9deJsoTqoLYxFnVDARSfh5UPgn93GPdUtemxVQcFqyrL60FCfgKphalUFIc1fqxEUVohO2HHdd2v42D6QdKL0nly+JPYW7CBWLhXOB3dO9Z9wqhHtHLov/0D8tOaMWLbpRJGa7XmSchLgmveB0cX02pvoM69N/5v/CzT94aisMaNlShKK3btJdfiYu/C7G6z6R/Q/6Ieo9YHK3tHuOYDKM3Xtj5uB1TCaI1Orod9n8OIByBMK8FlbGG4Objh5eRl9m5hnmF08e6CnbCjorjOtY+K0ub4ufqx8pqVPHnZkxf9GMY1UtUE9oSxT8DRX+DIz02IsHVQYxitTfF5+PVB8O8BYy/MBfdx9sHBzoEQj5B6V2Lf0usWDmccxjdYbV6ktC91ddU22YgH4dhKWLUQIkaDe+veJKk+KmG0NusWQ14y3LkOHF1Mh4UQBLgGmJ1SW9UNPW7ghh43WDtKRWkTXl8XU61lYVwr9dD4bhe6cu0dYOZ78OEYbdbU9Z/qEWqLUAmjNYnbqs2Kuvx+U1dUVY8PfbzVbwGpKLbkkYkXtheOWLTKtFaqlo69Yew/YeML0He2NnOqDVJjGK1FaaG2UMg3Uls4ZMaEzhNqLzJSFKVljHwYOvaDVY9CUY7e0ViFShitxaYXITsOZrwNTm56R6Mo7U6Da6TsHWHGW1CQBuv+r2WCamEqYbQGiXth53sweB5EjtY7GkVplyyafh46SOsyjl4GcXXuC9dqqYRh6yrKYOVD4B4IE5/TOxpFURoy9gmt63jlg1pXchuiEoat2/EupB6Cq17VyisrimLbnNy0rqnsONjyit7RNCuVMGxZ1mnYvAR6TofeM/SORlEUS0WOgYG3wPa3IbXtFNdWCcNWSanVqLFz0FoXiqK0LpNe0HoFfn0IDAa9o2kWKmHYqkPfw+lNMGExeNWuPqsoio1z6wCTX4KEPRD1sd7RNAuVMGxRUba2BWToEBhyp97RKIpysfrPgS5jYcNzkJusdzRNphKGLdrwHBRmwvTXwU79FylKqyWE9jouL2kT+4DrvUXr9UKII0IIgxCidq2LC+fFCyEOCSH2CyGiWjLGFnduD0R9CsMXQPDFlWFWFMWGdOgCox+FIz/BqY16R9Mken98PQzMAixZ4TJOSjlQSllnYmn1Ksrht0e0MYtxT+gdjaIozWXUw9Chq1Y2pKxY72gumt57eh+TUp7QMwabsnuptuZiyhJw9tQ7GkVRmouDM0z7jzZVftsbekdz0fRuYVhKAmuFEHuFEPPrO1EIMV8IESWEiEpPT2+h8JpBbjJsegkumQi9rtY7GkVRmlvXcVol263/hcxTekdzUayeMIQQ64UQh818zWzEw4ySUg4CpgL3CSHG1HWilHKplHKIlHJIQEBAXafZnnVPQ0UpXPWKNlCmKErbM/klsHeC3/+prbVqZay+H4aUckIzPEZi5b9pQoifgWFYNu7ROsRt0dZdXPFPbYBMUZS2yTNIG59c8y84sRp61rG/ho2y+S4pIYS7EMLT+D0wCW2wvG0oL9W2dvTpDKMe0TsaRVGsbdh8COgFfyyCsiK9o2kUvafVXiuESAAuB1YJIdZUHg8RQqyuPK0j8JcQ4gCwG1glpfxDn4itYNcHkHECpr4Mjq56R6MoirXZO8K01yDnLPz1ut7RNIquW7RKKX8GfjZzPAm4qvL708CAFg6tZeSlwJ8vQ7fJ0GOq3tEoitJSIkZBv+vhrzdgwI2tpiva5ruk2rT1z2gD3VP+rXckiqK0tInPa62NNea3XLZFKmHo5dxuOPA1XH4f+HXVOxpFUVqaVzCMeUwb/I5dr3c0FlEJQw+GClj9GHgGw+iFekejKIpeLlugdUf98YS2u6aNUwlDD/u+gOT9WpPU2UPvaBRF0YuDs1bZISNGq/Rg41TCaGnF52Hj8xB+GfS7Tu9oFEXRW/fJWoWHzUsgP03vaOqlEkZL2/IqFGTA1CVqRbeiKJopS7Q1GRue0zuSeqmE0ZIyT8HOD7S9fkMu1TsaRVFshf8lMPzvld3VB/WOpk66rsNod9Y+pfVZjv8/vSO5KGVlZSQkJFBc3HrLMytti4uLC2FhYTg6OuodStONeUybOfnHEzDvN5vsgVAJo6Wc2qRNnxu/GDw76h3NRUlISMDT05OIiAiEDf4xK+2LlJLMzEwSEhKIjIzUO5ymc/WBcU/Cqn/A8d9ssmq16pJqCYYKbXGOT2e47F69o7loxcXF+Pn5qWSh2AQhBH5+fm2rxTtoLgT21nojykv0jqYWlTBawr4vIO0ITHwWHF30jqZJVLJQbEmb+3u0d4DJL0J2vFZnzsaohGFtJXmw8QUIHw69r9E7GkVRbF3XK6H7FNjyHyjI1DuaalTCsLa/3oCCNG3jlLb2achCr6+LscrjPvPMM7z22mt13r5ixQqOHj1qlWsrilVNfA5K8+HPJXpHUo1KGNaUcw52vKNVpQwbonc0unlzw0ldrqsShtJqBfSAwfMg6hPI0Of1Y45KGNa08XltG8bxi/WOpM148cUX6d69O6NGjeLEiRMAfPTRRwwdOpQBAwYwe/ZsCgsL2b59OytXruSxxx5j4MCBnDp1yux5imKzxj4BDq6wznam4auEYS3JB+Dgt3D5veATrnc0Le71dTFELFpFxKJVAKbvm9I9tXfvXr755hv279/P6tWr2bNnDwCzZs1iz549HDhwgF69evHxxx8zYsQIZsyYwauvvsr+/fvp2rWr2fMUxWZ5BMDoR7Tp+HFb9Y4G0HkdhhDiVeBqoBQ4Bdwupcwxc94U4E3AHviflNK2OvZqkhLWPg2uHdrttquPTOzOIxO7A1qyiF/S9L2Lt27dyrXXXoubmxsAM2bMAODw4cM89dRT5OTkkJ+fz+TJk83e39LzFMVmXHYv7PkE1j0Nd20EO30/4+vdwlgH9JVS9gdigCdqniCEsAfeBaYCvYGbhBC9WzTKxordAHF/whX/BBdvvaNp8+bNm8c777zDoUOHWLx4cZ3z8i09T1FshqMrXPkUJO2DIz/pHY2+CUNKuVZKWV75404gzMxpw4BYKeVpKWUp8A0ws6VibDRDhfZpwDcShtyhdzQ24aHx3ZrlccaMGcOKFSsoKioiLy+PX3/9FYC8vDyCg4MpKyvjyy+/NJ3v6elJXl6e6ee6zlMUm9b/BujYTytMqPNiPr1bGFXdAfxu5ngocK7KzwmVx8wSQswXQkQJIaLS09ObOUQLHPga0o7ChMXg4NTy17dBxq6ppho0aBBz5sxhwIABTJ06laFDhwLw/PPPM3z4cEaOHEnPnj1N59944428+uqrXHrppZw6darO8xTFptnZa4t+c85os6Z0JKSU1r2AEOuBIDM3PSml/KXynCeBIcAsWSMgIcR1wBQp5V2VP98GDJdS3t/QtYcMGSKjoqKa+hQsV1YEbw8GzyC4a0ObW3dx7NgxevXqpXcYilJNu/m7XD5Tq2T74D6t7pSVCCH2SinNrgOw+qC3lHJCfbcLIeYB04HxNZNFpUSg6jSjsMpjtmf3UshNhFlL21yyUBRFZxOfgw/HwLY3YMIzuoSga5dU5eynx4EZUsq6JsXvAboJISKFEE7AjcDKlorRYkXZsPW/0G0SRIzSOxpFUdqa4AHaIuCdH0Busi4h6D2G8Q7gCawTQuwXQnwAIIQIEUKsBqgcFL8fWAMcA76TUh7RK+A6/fWGtv2qWqSnKIq1jHsSDOXw58u6XF7XdRhSykvqOJ4EXFXl59XA6paKq9HOJ2qVJfvPgaC+ekejKEpb1SEShtwOez6Gy+/XduprQXq3MNqGP5eANMC4f+kdiaIobd2Yx8DBRSs91MJUwmiqjFjY96W25sK3s97RKIrS1nkEwuX3wdEVkBjdopdWCaOpNr2gZfvRC/WORGmkzZs3M336dABWrlzJkiV1V5zJycnhvffeM/2clJTEddddZ/UYFcWsEQ9opYc2PNuil1UJoymS9sORn7UCgx4BekejVKqoqGj0fWbMmMGiRYvqvL1mwggJCeGHH364qPgUpclcvGD0o3B6M8RtabHL6jro3eptfAFcfbVs3978vghSDjXvYwb1g6n115WMj49nypQpDB48mOjoaPr06cPy5cvp3bs3c+bMYd26dTz++ON06NCBxYsXU1JSQteuXfn000/x8PDgjz/+4OGHH8bNzY1Roy5Mf/7ss8+IiorinXfeITU1lXvuuYfTp08D8P777/PWW29x6tQpBg4cyMSJE7nvvvuYPn06hw8fpri4mAULFhAVFYWDgwP//e9/GTduHJ999hkrV66ksLCQU6dOce211/LKK6807+9Mab+G3gk73oUNz8Oda1tk7ZdqYVysM9shdp1WjVYVGGxRJ06c4N577+XYsWN4eXmZPvn7+fkRHR3NhAkTeOGFF1i/fj3R0dEMGTKE//73vxQXF3P33Xfz66+/snfvXlJSUsw+/oMPPsgVV1zBgQMHTElpyZIldO3alf379/Pqq69WO//dd99FCMGhQ4f4+uuvmTt3rqmw4f79+/n22285dOgQ3377LefOnTN3SUVpPEdXuOJxSNgNMWta5JKqhXExpNQKgXkEwdC79Y5GHw20BKwpPDyckSNHAnDrrbfy1ltvATBnzhwAdu7cydGjR03nlJaWcvnll3P8+HEiIyPp1q2b6b5Lly6t9fgbN25k+fLlANjb2+Pt7U12dnad8fz111888IDWyuzZsyedO3cmJkbb92P8+PF4e2sfKHr37s2ZM2cID29/+6MoVnLprbDtTa23o9skq5c/VwnjYpzaAGd3wLT/gJOb3tG0O6JG09v4s7u7OwBSSiZOnMjXX39d7bz9+/e3SHxVOTs7m763t7envLy8nrMVpZHsHbXp/D/dDUd/hr6zrXo51SXVWFJq2dy7E1z6N72jaZfOnj3Ljh07APjqq6+qjUUAXHbZZWzbto3Y2FgACgoKiImJoWfPnsTHx3Pq1CmAWgnFaPz48bz//vuANoB+/vz5WqXSqxo9erSpXHpMTAxnz56lR48eTX+iimKJvrMhsDdsfBEqrPuBRCWMxjrxu7aZydh/qvLlOunRowfvvvsuvXr1Ijs7mwULFlS7PSAggM8++4ybbrqJ/v37m7qjXFxcWLp0KdOmTWPQoEEEBgaaffw333yTTZs20a9fPwYPHszRo0fx8/Nj5MiR9O3bl8cee6za+ffeey8Gg4F+/foxZ84cPvvss2otC0WxKjt7rZWRdUrbFtqKrF7eXE/NXt7cYIAPR2tlzO/bDfbtq0fPFspIx8fHm2YnKQrYxt+l7qSEpVdAUQ48sFfrqrpI9ZU3Vy2Mxji6AlIPw9gn2l2yUBTFhgmhFSbMOQP7rbebpEoYljJUwOZ/Q0Av6DtL72jarYiICNW6UBRzuk2C0CHw56tW28pVJQxLHf4RMmJg7CKtz1BRFMWWCAFXPgm5CRC93CqXUAnDEhWV9ec79oVeM/SORlEUxbwu46DTCNjymjbW2sxUR7wlDv8AmbFww+dWXxijKIpy0YSASS9AfqpWFLWZqYTREFProh/0nK53NIqiKPULG2y1h9Z7T+9XhRDHhRAHhRA/CyF86jgvXghxqHIb12acJ2uBQ99B1unKsQvVumjvnnnmGV577bVax1esWMHRo0cb/Xjx8fF89dVXpp8/++wz7r///ouOLyIigoyMjIu+f3NQpeDbLr3fAdcBfaWU/YEY4Il6zh0npRxY1/xgq6gohz9fgaD+0HNai11WaRo9ym/UlzDqi6dmwrA1F/O7VKXg2y699/ReW+XHnYBtfQw59B1kx8GNX7VI6eDW5OXdL3M863izPmbPDj3557B/1nvO888/zxdffEFAQADh4eEMHjyYhQsXMnbsWAYOHMhff/3FTTfdxMCBA1m4cCHl5eUMHTqU999/H2dnZyIiIoiKisLf35+oqCgWLlzI5s2beeaZZzh79iynT5/m7NmzPPzwwzz44IMAvPjiiyxbtozAwEDTNavavn07K1eu5M8//+SFF17gxx9/5M4776wWz6FDh5g+fbrpk7aHhwf5+fksWrSIY8eOMXDgQObOnYuvry9JSUlMmTKl3pLoGzZsMPv8AF555RV+//13XF1d+eqrr7jkkkv4/vvvefbZZ03FFLds2UJFRQWLFi1i8+bNlJSUcN999/H3v/+dzZs38/TTT+Pr68vx48eZNWsW4eHh3HfffYDWyvLw8OCee+5h5syZZGdnU1ZWxgsvvMDMmTNZtGhRs5WCr6io4M477yQqKgohBHfccQePPPJIk//WlItjS2MYdwB1rWuXwFohhAQ+lFLWLjFaSQgxH5gP0KlTp4uPpqJcm2kQ1A96XHXxj6M0mz179vDjjz9y4MABysrKGDRoULU379LSUqKioiguLqZbt25s2LCB7t2787e//Y3333+fhx9+uN7HP378OJs2bSIvL48ePXqwYMECDh48yDfffMP+/fspLy+vdU2AESNGMGPGjGoJoWo8APPmzTN7zSVLlvDaa6/x22+/AVqX1P79+9m3bx/Ozs706NGDBx54oFqF2+LiYubNm1fn8/P29ubQoUMsX76chx9+mN9++43nnnuONWvWEBoaSk5ODgAff/wx3t7e7Nmzh5KSEkaOHMmkSZMAiI6O5vDhw0RGRrJv3z4efvhhU8L47rvvWLNmDS4uLvz88894eXmRkZHBZZddxowZM1iyZAmHDx82FXuMj483xV61FPzx48eZNGmSqbKvueedlpZGYmKiae2NMXZFH1ZPGEKI9UCQmZuelFL+UnnOk0A5UNcSxVFSykQhRCCwTghxXEppdpupymSyFLTSIBcd+JGftNosc75QrQszGmoJWMO2bduYOXMmLi4uuLi4cPXVV1e73Vje/MSJE0RGRtK9e3cA5s6dy7vvvttgwpg2bRrOzs44OzsTGBhIamoqW7du5dprr8XNTatKPGOG5dOqjfE0VkMl0Rt6fjfddJPpX+On8ZEjRzJv3jxuuOEGZs3SFp6uXbuWgwcPmrqLzp8/z8mTJ3FycmLYsGFERkYCcOmll5KWlkZSUhLp6en4+voSHh5OWVkZ//rXv9iyZQt2dnYkJiaSmppa73NrbCn4Pn36cPr0aR544AGmTZtmSmiKPqyeMKSUE+q7XQgxD5gOjJd1FLaSUiZW/psmhPgZGAZYb19CQ4U2dhHYB3qosYvWwljevD4ODg4YDAYA0yZHRs1dirxqPFWvazAYKC0trfN+TY2javl34/cffPABu3btYtWqVQwePJi9e/cipeTtt99m8uTJ1e6/efPmWr/L66+/nh9++IGUlBRTIvzyyy9JT09n7969ODo6EhERUet32hjmnrevry8HDhxgzZo1fPDBB3z33Xd88sknF30NpWn0niU1BXgcmCGlLKzjHHchhKfxe2ASYN3aEEd+hsyT2m5WamaUzRg5ciS//vorxcXF5Ofnm7pxaurRowfx8fGm8uaff/45V1xxBaDNItq7dy8AP/74Y4PXHDNmDCtWrKCoqIi8vDx+/fVXs+fVV/685nVXrlxJWVmZRfczp77nB/Dtt9+a/r388ssBOHXqFMOHD+e5554jICCAc+fOMXnyZN5//31TLDExMRQUFJi95pw5c/jmm2/44YcfuP766wGtRRIYGIijoyObNm3izJkzDT6nxpaCz8jIwGAwMHv2bF544QWio6Mt/j0pzU/vMYx3AGe0biaAnVLKe4QQIcD/pJRXAR2BnytvdwC+klL+YbWIDAbY8qpWM0qt6rYpQ4cOZcaMGfTv35+OHTvSr18/UxdGVS4uLnz66adcf/31pkHhe+65B4DFixdz55138vTTTzN27NgGrzlo0CDmzJnDgAEDCAwMZOjQoWbPu/HGG7n77rt56623zM4Iuvvuu5k5cyYDBgxgypQppk/w/fv3x97engEDBjBv3jx8fX0bjKm+5weQnZ1N//79cXZ2Nu358dhjj3Hy5EmklIwfP54BAwbQv39/4uPjGTRoEFJKAgICWLFihdlr9unTh7y8PEJDQwkODgbglltu4eqrr6Zfv34MGTKEnj17AlQrBT916lTT2AdopeAXLFhAv379cHBwaLAUfGJiIrfffrupdfbvf/+7wd+PYj2qvHlNJXnwxxNwyQToc41V4mqtbKGMdH5+Ph4eHhQWFjJmzBiWLl3KoEGDdI1J0Zct/F22JfWVN9e7hWF7nD1h5jt6R6HUYf78+Rw9epTi4mLmzp2rkoWitCCVMJRWxZYXuSlKW6dGdJVGactdmErro/4eW5ZKGIrFXFxcyMzMVC9SxSZIKcnMzMTFpfmrsirmqS4pxWJhYWEkJCSQnp6udyiKAmgfYsLCwvQOo91QCUOxmKOjo2n1r6Io7Y/qklIURVEsohKGoiiKYhGVMBRFURSLtOmV3kKIdOCM3nE0kj+g75ZpLU895/ZBPefWobOUMsDcDW06YbRGQoioFt1V0Aao59w+qOfc+qkuKUVRFMUiKmEoiqIoFlEJw/bUuf1sG6aec/ugnnMrp8YwFEVRFIuoFoaiKIpiEZUwFEVRFIuohGHDhBCPCiGkEMJf71isTQjxqhDiuBDioBDiZyGEj94xWYMQYooQ4oQQIlYIsUjveKxNCBEuhNgkhDgqhDgihHhI75haihDCXgixTwhhfvP5VkglDBslhAgHJgFn9Y6lhawD+kop+wMxwBM6x9PshBD2wLvAVKA3cJMQore+UVldOfColLI3cBlwXzt4zkYPAcf0DqI5qYRhu14HHgfaxawEKeVaKWV55Y87gbZYs3oYECulPC2lLAW+AWbqHJNVSSmTpZTRld/nob2BhuoblfUJIcKAacD/9I6lOamEYYOEEDOBRCnlAb1j0ckdwO96B2EFocC5Kj8n0A7ePI2EEBHApcAunUNpCW+gfeAz6BxHs1L7YehECLEeCDJz05PAv9C6o9qU+p6zlPKXynOeROvG+LIlY1OsSwjhAfwIPCylzNU7HmsSQkwH0qSUe4UQY3UOp1mphKETKeUEc8eFEP2ASOCAEAK0rploIcQwKWVKC4bY7Op6zkZCiHnAdGC8bJsLhBKB8Co/h1Uea9OEEI5oyeJLKeVPesfTAkYCM4QQVwEugJcQ4gsp5a06x9VkauGejRNCxANDpJStreJlowghpgD/Ba6QUrbJPWCFEA5oA/rj0RLFHuBmKeURXQOzIqF96lkGZEkpH9Y5nBZX2cJYKKWcrnMozUKNYSi24h3AE1gnhNgvhPhA74CaW+Wg/v3AGrTB3+/acrKoNBK4Dbiy8v91f+Unb6UVUi0MRVEUxSKqhaEoiqJYRCUMRVEUxSIqYSiKoigWUQlDURRFsYhKGIqiKIpFVMJQFEVRLKIShqIoimIRlTAUpQVV7g0xsfL7F4QQb+sdk6JYStWSUpSWtRh4TggRiFa5dYbO8SiKxdRKb0VpYUKIPwEPYGzlHhGK0iqoLilFaUGV1YiDgVKVLJTWRiUMRWkhQohgtH0+ZgL5lRV6FaXVUAlDUVqAEMIN+Altf+tjwPNo4xmK0mqoMQxFURTFIqqFoSiKolhEJQxFURTFIiphKIqiKBZRCUNRFEWxiEoYiqIoikVUwlAURVEsohKGoiiKYpH/B/2mgygFqbv0AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "## EDIT THIS CELL\n",
        "K = 2 # Define the degree of the polynomial we wish to fit\n",
        "\n",
        "Phi = poly_features(X, K) # N x (K+1) feature matrix\n",
        "\n",
        "#@ Test\n",
        "assert np.any(Phi), \"Poly_features not implemented\"\n",
        "\n",
        "theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator\n",
        "\n",
        "#@ Test\n",
        "assert np.any(theta_ml), \"nonlinear_features_maximum_likelihood\"\n",
        "\n",
        "# test inputs\n",
        "Xtest = np.linspace(-5,5,100).reshape(-1,1)\n",
        "ytest = f(Xtest) # ground-truth y-values\n",
        "\n",
        "# feature matrix for test inputs\n",
        "Phi_test = poly_features(Xtest, K)\n",
        "\n",
        "#@ Test\n",
        "assert np.any(Phi), \"Poly_features not implemented\"\n",
        "\n",
        "\n",
        "y_pred = Xtest*0 # <-- EDIT THIS LINE\n",
        "\n",
        "#@ Test\n",
        "assert np.any(y_pred), \"Prediction not implemented\"\n",
        "\n",
        "# plot\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.plot(Xtest, y_pred)\n",
        "plt.plot(Xtest, ytest)\n",
        "plt.legend([\"data\", \"prediction\", \"ground truth observations\"])\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5SqfFQCjmsJ"
      },
      "source": [
        "#### Questions:\n",
        "1. Try out different degrees of polynomials. \n",
        "2. Based on visual inspection, what looks like the best fit?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KqXCX9OjmsK"
      },
      "source": [
        "Let us now look at a more systematic way to assess the quality of the polynomial that we are trying to fit. For this, we compute the root-mean-squared-error (RMSE) between the $y$-values predicted by our polynomial and the ground-truth $y$-values. The RMSE is then defined as\n",
        "$$\n",
        "\\text{RMSE} = \\sqrt{\\frac{1}{N}\\sum_{n=1}^N(y_n - y_n^\\text{pred})^2}\n",
        "$$\n",
        "Write a function that computes the RMSE."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "X5iylTkGjmsK"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def RMSE(y, ypred):\n",
        "\n",
        "    rmse = -1 ## <-- EDIT THIS LINE\n",
        "\n",
        "    return rmse"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xKEXpGYIjmsK"
      },
      "source": [
        "Now compute the RMSE for different degrees of the polynomial we want to fit."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "3k15zOZwjmsL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "eb001015-eed4-41e2-ddae-ca1fb4b0edf8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAm/0lEQVR4nO3deZRV5Znv8e9zTg3UABRQpwooQKAKNOWAGsSgRjEGghnUTAZNJ2Ykdsd00kn6XrOS5cq1b1Zu0ukkKx1vp21jx+TGaGKGJgk2EBWNiSLlhAICBTIUYwHFWBQ1PfePvQsOZc2cXafOqd9nrbPOHt69z1P7DE/t/b77fc3dERGR4SuW7gBERCS9lAhERIY5JQIRkWFOiUBEZJhTIhARGeZy0h1Af5WWlvrUqVPTHYaISEZ5/vnn97t7oqt1GZcIpk6dSk1NTbrDEBHJKGa2rbt1ujQkIjLMKRGIiAxzSgQiIsOcEoGIyDCnRCAiMswpEYiIDHNKBCIiw1zG3UcwUDVbD/LUpv3k58TIz4mRl/ScF4+/cVlOjPyc+KnyRfk5FOUPm8MlIsPIsPlle2F7Az94bNOAt4/HjCV3XMn5E0enMCoRkfQbNolg8dWVfPqt02lua+dkazvN4ePkqee2M5adbG0Pyra0cfB4M9989DVe2nFIiUBEss6wSQQAZhZe7on3a7v2duf7f9rE5n3HI4pMRCR9VFncB7GYMT1RRG39sXSHIiKSckoEfVSZKGbzPiUCEck+kSYCM1toZhvMrNbM7uxi/ffM7KXwsdHMDkUZz9moKitm56ETNDa3pjsUEZGUiqyOwMziwD3AfKAOWG1mS9x9XUcZd/+HpPKfAy6JKp6zVVVWDMCW+uNcUKEKYxHJHlGeEcwBat19i7s3Aw8BN/ZQ/hbgFxHGc1YqE0Ei2Kx6AhHJMlEmggpgR9J8XbjsDczsHGAa8HiE8ZyVqaWFxAxqVU8gIllmqFQWLwIecfe2rlaa2WIzqzGzmvr6+kEOLZCfE+eccUVKBCKSdaJMBDuByUnzk8JlXVlED5eF3P1ed5/t7rMTiS6H3BwUlYkiXRoSkawTZSJYDcwws2lmlkfwY7+kcyEzOw8YAzwTYSwpUVlWzOv7j9Pa1p7uUEREUiayRODurcAdwDJgPfBLd19rZneb2Q1JRRcBD7m7RxVLqlQmimlpc3Y0nEh3KCIiKRNpFxPuvhRY2mnZXZ3mvx5lDKnU0YS0dt8xppUWpTkaEZHUGCqVxRmhowmpKoxFJJsoEfTD6IJcEiPzVWEsIllFiaCfqhLFOiMQkayiRNBPlWVBE9IMqNsWEekTJYJ+qkoUc7SplfqjJ9MdiohISigR9FNV2UhAFcYikj2UCPqpsixoNqoKYxHJFkoE/TR+1AiK8uI6IxCRrKFE0E9mRmVZMZvrNX6xiGQHJYIBUBNSEckmSgQDUFlWzJ4jTRxtakl3KCIiZ02JYAA6uprYostDIpIFlAgGILnzORGRTKdEMADnjCskJ2ZqQioiWUGJYABy4zHOGVeoMwIRyQpKBANUVVZMrc4IRCQLKBEMUGWimO0HGmnRsJUikuGUCAaoqqyY1nZn2wG1HBKRzBZpIjCzhWa2wcxqzezObsrcbGbrzGytmT0YZTypdHq0MiUCEclskY1ZbGZx4B5gPlAHrDazJe6+LqnMDOArwJXu3mBmZVHFk2qVYRNStRwSkUwX5RnBHKDW3be4ezPwEHBjpzKfBu5x9wYAd98XYTwpVZyfw4TRI9RySEQyXpSJoALYkTRfFy5LNhOYaWZ/MbNnzWxhhPGkXGWiWGcEIpLx0l1ZnAPMAOYBtwD/YWYlnQuZ2WIzqzGzmvr6+sGNsAdVZcVs3qdhK0Uks0WZCHYCk5PmJ4XLktUBS9y9xd1fBzYSJIYzuPu97j7b3WcnEonIAu6vykQRx5vb2HOkKd2hiIgMWJSJYDUww8ymmVkesAhY0qnM7wjOBjCzUoJLRVsijCmlKtXnkIhkgcgSgbu3AncAy4D1wC/dfa2Z3W1mN4TFlgEHzGwd8ATwj+5+IKqYUk2dz4lINois+SiAuy8FlnZadlfStANfDB8ZJ1Gcz8gROaowFpGMlu7K4oxmZkGfQzojEJEMpkRwloImpLq7WEQylxLBWaoqK6b+6EkOn9CwlSKSmZQIzlJVQhXGIpLZlAjOkvocEpFMp0RwliaPKSAvHmOzzghEJEMpEZylnHiMqaWFOiMQkYylRJACakIqIplMiSAFqhLFbD/YSFNLW7pDERHpNyWCFKgsK6bdYduBxnSHIiLSb0oEKVCpJqQiksGUCFJgeqIIUBNSEclMSgQpUJiXQ0VJgc4IRCQjKRGkiFoOiUimUiJIkcpEMVv2H6O9XcNWikhmUSJIkaqyYppa2tl56ES6QxER6RclghSpVIWxiGQoJYIU0bCVIpKplAhSZFxxPmMKc3VGICIZJ9JEYGYLzWyDmdWa2Z1drP+YmdWb2Uvh41NRxhO1ykQxm/dptDIRySyRDV5vZnHgHmA+UAesNrMl7r6uU9GH3f2OqOIYTFVlxSxftzfdYYiI9EuUZwRzgFp33+LuzcBDwI0Rvl7aVZUVc/B4MwePN6c7FBGRPosyEVQAO5Lm68Jlnb3fzNaY2SNmNrmrHZnZYjOrMbOa+vr6KGJNiY4+h1RPICKZJN2Vxb8Hprr7RcAK4IGuCrn7ve4+291nJxKJQQ2wP9RySEQyUZSJYCeQ/B/+pHDZKe5+wN1PhrP3AW+OMJ7ITSwpID9Hw1aKSGaJMhGsBmaY2TQzywMWAUuSC5jZhKTZG4D1EcYTuXjMmJ4oplaXhkQkg0TWasjdW83sDmAZEAfud/e1ZnY3UOPuS4C/N7MbgFbgIPCxqOIZLFVlxby0oyHdYYiI9FlkiQDA3ZcCSzstuytp+ivAV6KMYbBVJor4w5pdNLW0MSI3nu5wRER6le7K4qxTVVaMu1oOiUjmUCJIsdNNSHWHsYhkBiWCFJtWWkTM1IRURDKHEkGKjciNM3lsoS4NiUjGUCKIQND5nBKBiGQGJYIIVJUVs2X/cdo0bKWIZAAlgghUJopobm2nrqEx3aGIiPRKiSAC6nNIRDKJEkEE1AupiGQSJYIIlBTmUVqcpzMCEckISgQRqUwUKxGISEZQIohIZVkxm+uP466WQyIytCkRRKQqUczhEy3sP6ZhK0VkaFMiiEhHyyFVGIvIUKdEEJFKNSEVkQyhRBCRCaNGUJgXVyIQkSFPiSAisZgxPVGkS0MiMuQpEUSoSp3PiUgGiDQRmNlCM9tgZrVmdmcP5d5vZm5ms6OMZ7BVlRWz63ATx0+2pjsUEZFu9ZgIzOxtSdPTOq17Xy/bxoF7gOuBauAWM6vuotxI4PPAqr6HnRk6uprYotHKRGQI6+2M4DtJ07/utO5rvWw7B6h19y3u3gw8BNzYRbl/Ar4FNPWyv4xzqvO5+qNpjkREpHu9JQLrZrqr+c4qgB1J83XhstM7MLsUmOzuf+wxCLPFZlZjZjX19fW9vOzQcc64IuIxY/M+nRGIyNDVWyLwbqa7mu8XM4sB3wW+1FtZd7/X3We7++xEInE2Lzuo8nJinDO2UE1IRWRIy+ll/XQzW0Lw33/HNOH8tO43A2AnMDlpflK4rMNI4AJgpZkBjAeWmNkN7l7Tx/iHvKDPISUCERm6eksEydf0v9NpXef5zlYDM8JK5p3AIuDWjpXufhgo7Zg3s5XAl7MpCUBQYbxywz5a29rJiau1rogMPT0mAnd/MnnezHIJ/ovf6e77etm21czuAJYBceB+d19rZncDNe6+pKfts0VVWTEtbc62g42nWhGJiAwlPSYCM/sR8K/hD/ho4BmgDRhrZl9291/0tL27LwWWdlp2Vzdl5/Un8ExRmSgC4PmtDYzIjQOna9nNwMI5CxeeqoE/NW+MK8ojFuutbl5EZGB6uzT0Vne/PZz+OLDR3W8ys/HAo0CPiUCCM4KYwf/49ZoB7+Omiyfy/UWXpDAqEZHTeksEyZ3pzwd+BeDue8z0H2pfjByRy88+eTk7G07gYUOrjrFqOppdudPtuhXr9vLfa/dwormNgrz44AUuIsNGb4ngkJm9m6Cy90rgkwBmlgMURBxb1riyqrT3Qt2YOq6QpzbW83TtfuZXl6cwKhGRQG/NWD4D3AH8J/AFd98TLr8O6PEmMEmNy6eNY2R+DivW7em9sIjIAPTWamgjsLCL5csIWgNJxPJyYlx7XhmPrd9HW7sTV6WxiKRYb62GftDTenf/+9SGI12ZX13Okpd38cL2Bi6bOjbd4YhIlumtjuB24FXgl8Aueu9fSCIw79wEuXFj+do9SgQiknK91RFMAO4F3gF8BMgF/svdH3D3B6IOTgIjR+RyRWUpy9ftxf2sungSEXmDHhOBux9w9x+5+7UE9xGUAOvM7CODEZycNr+6nG0HGtmkDuxEJMX61PlN2F3054G/IbiR7Pkog5I36mg6umLd3jRHIiLZprcRyu42s+eBLwJPArPd/ZPuvm5QopNTykeNYNbkEpavVTNSEUmt3s4IvkZwOWgW8E3gBTNbY2avmNnA+0yQAVlQXc7LdYfZczjrBnMTkTTqrdVQb2MOyCBaUF3OPy/bwIr1e/nIW85JdzgikiV6qyze1tWDYAjKqwYnROlQVVbMtNIiXR4SkZTqrY5glJl9xcx+aGYLLPA5YAtw8+CEKB3MjPnV5Ty75QBHmlrSHY6IZIne6gh+BpwLvAJ8CngC+ABwk7vf2NOGEo0F1eW0tDkrN9SnOxQRyRK9jlns7hcCmNl9wG5girurtjJNLpkyhtLiPFas28sNsyamOxwRyQK9nRGcuv7g7m1AnZJAesVjxnXnlbPytX00t7anOxwRyQK9JYJZZnYkfBwFLuqYNrMjve3czBaa2QYzqzWzO7tYf3vYFPUlM3vazKoH+ocMJwvOL+foyVae3XIg3aGISBbordVQ3N1HhY+R7p6TND2qp23NLA7cA1wPVAO3dPFD/6C7X+juFwPfBr478D9l+LiyqpSC3DjLNUaBiKRAn7qYGKA5QK27b3H3ZuAh4IwKZndPPqso4vQIjdKDEblxrpmZYMW6vbS365CJyNmJMhFUENxv0KEuXHYGM/usmW0mOCPocnwDM1tsZjVmVlNfr9YyEPQ9tPfISV7ZeTjdoYhIhosyEfSJu9/j7pXA/yTo0qKrMve6+2x3n51IJAY3wCHqbeeVEY+ZLg+JyFmLMhHsBCYnzU8Kl3XnIeCmCOPJKmOK8pgzdax6IxWRsxZlIlgNzDCzaWaWBywCliQXMLMZSbPvAjZFGE/WmV9dzsa9x9i6/3i6QxGRDBZZInD3VuAOgkHu1wO/dPe1YdfWN4TF7jCztWb2EkFX17dFFU820hgFIpIKvd1ZfFbcfSmwtNOyu5KmPx/l62e7yWMLedOEUSxft4dPXz093eGISIZKe2WxnJ0F1eXUbGtg/7GT6Q5FRDKUEkGGm19djjs8vn5fukMRkQylRJDhzp84ioqSAjUjFZEBUyLIcB1jFPx5034am1vTHY6IZCAlgiywoLqck63tPLVxf7pDEZEMpESQBS6bNpbRBblqRioiA6JEkAVy4zHedl4Zj722l9Y2jVEgIv2jRJAlFlSXc6ixhZptDekORUQyjBJBlrh6ZoK8nBjL1+rykIj0jxJBlijKz+GqqlKWr9uDu8YoEJG+UyLIIguqy6lrOMFre46mOxQRySBKBFnkujeVY4YuD4lIvygRZJHEyHwunTKGFet1l7GI9J0SQZaZX13OqzuPsPPQiXSHIiIZQokgyywIxyj4k24uE5E+UiLIMtMTxVQmitQJnYj0mRJBFlpw/nhWbTnI4caWdIciIhlAiSALza8up7XdeWKDxigQkd5FmgjMbKGZbTCzWjO7s4v1XzSzdWa2xsweM7NzooxnuLh4UgmJkfm6PCQifRJZIjCzOHAPcD1QDdxiZtWdir0IzHb3i4BHgG9HFc9wEosFYxQ8uaGeppa2dIcjIkNclGcEc4Bad9/i7s3AQ8CNyQXc/Ql3bwxnnwUmRRjPsDK/upzjzW08s/lAukMRkSEuJ8J9VwA7kubrgMt7KP9J4NGuVpjZYmAxwJQpU1IVX1a7onIcRXlxfv/yLmaOHwmAhevMTpezcGnHMju9gvycOKMLcgclXhFJnygTQZ+Z2d8As4Frulrv7vcC9wLMnj1bPar1QX5OnHnnlfGbF3fymxd3Dng/VWXFXFE5jrnTx/GW6eMYU5SXwihFZCiIMhHsBCYnzU8Kl53BzN4OfBW4xt1PRhjPsHPXu6uZNzOBA4Tp08MJPzXPGfPJZQ41tvDc6wd55Pk6fvrMNgDeNGEUc6ePY27lOOaEI6OJSGazqLosNrMcYCNwHUECWA3c6u5rk8pcQlBJvNDdN/Vlv7Nnz/aampoIIpbutLS1s6buEM9sPsAzWw5Qs7WBk63txAwuqBjN3PCM4bKpYynKHxInmSLSiZk97+6zu1wXZd/1ZvZO4PtAHLjf3b9hZncDNe6+xMz+BFwI7A432e7uN/S0TyWC9DvZ2saL28PEsPkAL+5ooKXNyYkZsyaXnLqMNLO8mMTIfCy5UkJE0iJtiSAKSgRDz4nmNp7f1sAzW/bz180HWFN3mLb24HOVnxNj0pgCpowtZPLYQiaPCZ/HFjB5bCGjRujSkshg6CkR6DxezlpBXpyrZpRy1YxSAI6dbOXF7Q1s3X+cHQ0n2H6gkR0NjdRsa+BoU+sZ25YU5obJoeCMRFFSkEteToz8nBh54SM/Jx7Mx2PEYjrLEEkVJQJJueL8HN46I8FbZyTesO5wYws7GhrZfrCRHQfD54YTvLb7KH9at4/mtvY+vUZu3MiLx8jPjZMXj52RNCoTxSy6bDJzK8fpspRIHygRyKAaXZjL6MLRXFAx+g3r2tudvUeb2HHwBMdOttDc2s7JpEcw30bzqekzl3WUe3JjPUte3sW00iJunTOF9795EmPV7FWkW6ojkKzT1NLG0ld28+Cq7dRsayAvHuP6C8dz65wpzJk2VmcJMiypsliGrQ17jvLgqm385sWdHG1qpaqsODhLuHQSowtVUS3DhxKBDHsnmtv4/ZpdPLhqOy/tOER+Tox3XTSBD19+DpdOKdFZgmQ9JQKRJGt3HebBVdv53Ys7Od7cxnnjR3Lr5VO46ZIKNWeVrKVEINKF4ydbWfLyLn6+ahuv7jxCQW6cd100gesvGM+VVaWMyI2nO0SRlFEiEOnFmrpDPLhqO39cs5ujJ1spzIsz79wEC6rHc+25ZapPkIynRCDSR82t7Tyz5QDL1+5hxbq97Dt6kpyY8Zbp41hwfjnzq8uZMLog3WGK9JsSgcgAtLc7L9cdYtnavSxfu4ct+48DMGvSaBacP54F1eVUlRWrolkyghKBSArU7jvG8nV7WL52Ly/tOATA9NIi5p9fzoLq8VwyuURdX8iQpUQgkmJ7DjexYn1wpvDM5gO0tjujC3IpHmA33LEYvPPCCdxxbRUj1XJJIqBEIBKhwydaWLlhH89uOUBL28C+T4cam/nT+n2UFufxpQXncvPsycR1diEppEQgkgHW1B3in/6wjtVbGzhv/Ejuenc1V1SVpjssyRI9JYLYYAcjIl27aFIJv/zMXO659VKONrVy632r+PRPa3g9rKQWiYoSgcgQYma866IJPPala/jHd5zLX2v3s+B7T/KNP67j8ImWdIcnWUqJQGQIGpEb57PXVvHEl+fx3ksquO/p17n2Oyv5f89uo7WPYzaI9FWkicDMFprZBjOrNbM7u1h/tZm9YGatZvaBKGMRyURlo0bw7Q/M4vd3XEVVWTFf+92rvOsHT/PnTfXpDk2ySGSJwMziwD3A9UA1cIuZVXcqth34GPBgVHGIZIMLKkbz8OK38G8fvpTGllY+8uPn+NQDq9lSfyzdoUkWiHKEsjlArbtvATCzh4AbgXUdBdx9a7hO57oivTAzrr9wAteeV8ZP/rqVHz5ey4LvPcVH507lhosnYoAZGEbHzc7J82dMh+vAKMqPUzZyhJqrDmNRJoIKYEfSfB1w+UB2ZGaLgcUAU6ZMOfvIRDLYiNw4t19TyfsvncR3V2zgP//6Ovf/5fWz2mdOzJhYUsCkMR2PQiaNKaCipIBJYwsZP0qJIptlxJjF7n4vcC8E9xGkORyRISExMp9vvu8iPvXW6Wzdfxx3cMDdw2cIlpC0Dhw/o+zRplZ2HjpBXcMJdjY0snJDPfuOnjzjtXJixoSSEUwqKTyVKCrGFDCttJCLJpWQG1e7k0wWZSLYCUxOmp8ULhORFKpMFFOZKE7pPpta2tjVkRwOnaCuoZG6hmD+qU317D1yOlEU5+dwZdU4rplZxjXnJqgoUe+smSbKRLAamGFm0wgSwCLg1ghfT0RSZERunOmJYqZ3k2BOtrax+1ATr+05wpMb9/Pkhn0sW7sXgBllxVwzM8E15ya4bOpYDfCTASLtYsLM3gl8H4gD97v7N8zsbqDG3ZeY2WXAb4ExQBOwx93P72mf6mJCZOhxdzbXH2Plhnqe3FjPqi0HaW5rpyA3ztzKcUFimJlgamlRukMdttTXkIgMqsbmVlZtOcjKDft4cmM9Ww80AnDOuMJTSWFu5TgK8zKimjIrKBGISFpt3X+cpzbVs3JDPc9sPsCJljby4jGuOTfBDbMmct2bypQUItZTItCRF5HITS0tYmppER+dO5WmljZqtjbw2Gt7+eOa3axYt5eC3Dhvry7nhlkTuXpmKfk5qlcYTDojEJG0aWt3Vm89yJKXd/HoK7tpaGxh5IgcFp4/nhsunsjc6ePIUdPUlNClIREZ8lra2vlL7X6WvLyL5Wv3cuxkK6XFebzzwgm8Z9ZE3jxljIYCPQtKBCKSUZpa2li5oZ7fv7yLP63fy8nWdiaOHsG7Z03kPRdN5IKKUZgpKfSHEoGIZKxjJ1t5bP1efv/yLp7cWE9LmzOttIjppUXkxI2cWOzUc27cTi+LGTnxYFk8ZuTGTy+7bOoYLppUku4/bVCpslhEMlZxfg43XlzBjRdXcKixmWVr9/Doq3vYe7SJ1janpa2d1nY/Nd3W3mlZezud/9/NiRnfev9FvP/Nk9LzRw0xSgQikjFKCvP40GVT+NBl/et8sr09SAitbc6xk638w8Mv8aVfvczOQyf43Nuqhv1lJlXHi0jWi8WM/Jw4Rfk5lI8awU8+Pof3XVLBd1ds5M5fv0LLMB/1TWcEIjLs5OXE+JebZzFpTAE/eLyW3UeauOfWSxg5IjfdoaWFzghEZFgyM7644Fz+z/su5C+1+7n5359l75GmdIeVFkoEIjKsLZozhR/fNpvtB47z3nv+woY9R9Md0qBTIhCRYW/euWU8/Jm5tLY7H/i3v/LX2v3pDmlQKRGIiAAXVIzmt5+9kgklI7jtP5/jNy/UpTukQaNEICISqigp4Fe3X8Hsc8byxV++zA8f30Sm3XQ7EEoEIiJJRhfk8sAn5vDeSyr4zvKNfOU32d+8VM1HRUQ6ycuJ8d2bZ1FRUsAPn6hl9+Em7vnwpRTnZ+dPps4IRES6YGZ8+R3n8s33XcjTtfu5+UfPZG3z0kgTgZktNLMNZlZrZnd2sT7fzB4O168ys6lRxiMi0l+3zJnCfbfNZmvYvPTVnYez7lJRZL2Pmlkc2AjMB+qA1cAt7r4uqczfARe5++1mtgh4r7t/qKf9qvdREUmHV3ce5uM/WU390ZMAjMzPYXRhLiWFuYwpzGN0QfBcUph7xnRJx3NBsDwes7T0bZSu3kfnALXuviUM4iHgRmBdUpkbga+H048APzQz8+FQTS8iGeWCitH84XNXsWztHhqOt9DQ2MzhEy0camymobGFuoYTHAqXtffyCxYziMeMmAVdZMfNiMUsaRnELUgY8VPL4Qtvn8l7Zk1M+d8WZSKoAHYkzdcBl3dXxt1bzewwMA44424OM1sMLAaYMqV/vQ6KiKRK+agRfHTu1B7LtLc7R5taaWhs5lCYKA41Bs9HmlppbXfa2502D5+Tp91pa+eM9e3utHmwrKQwmr6QMqIK3N3vBe6F4NJQmsMREelWLGaMLsxldEQ/2lGIsrJ4JzA5aX5SuKzLMmaWA4wGDkQYk4iIdBJlIlgNzDCzaWaWBywClnQqswS4LZz+APC46gdERAZXZJeGwmv+dwDLgDhwv7uvNbO7gRp3XwL8GPiZmdUCBwmShYiIDKJI6wjcfSmwtNOyu5Kmm4APRhmDiIj0THcWi4gMc0oEIiLDnBKBiMgwp0QgIjLMRdbXUFTMrB7YNsDNS+l01/IQobj6R3H131CNTXH1z9nEdY67J7pakXGJ4GyYWU13nS6lk+LqH8XVf0M1NsXVP1HFpUtDIiLDnBKBiMgwN9wSwb3pDqAbiqt/FFf/DdXYFFf/RBLXsKojEBGRNxpuZwQiItKJEoGIyDCXlYnAzBaa2QYzqzWzO7tYn29mD4frV5nZ1EGIabKZPWFm68xsrZl9vosy88zssJm9FD7u6mpfEcS21cxeCV/zDQNCW+AH4fFaY2aXDkJM5yYdh5fM7IiZfaFTmUE7XmZ2v5ntM7NXk5aNNbMVZrYpfB7Tzba3hWU2mdltXZVJYUz/bGavhe/Tb82spJtte3zPI4rt62a2M+n9emc32/b4/Y0groeTYtpqZi91s20kx6y734ZB/Xy5e1Y9CLq83gxMB/KAl4HqTmX+DvhROL0IeHgQ4poAXBpOjwQ2dhHXPOAPaThmW4HSHta/E3gUMOAtwKo0vKd7CG6IScvxAq4GLgVeTVr2beDOcPpO4FtdbDcW2BI+jwmnx0QY0wIgJ5z+Vlcx9eU9jyi2rwNf7sN73eP3N9VxdVr/L8Bdg3nMuvttGMzPVzaeEcwBat19i7s3Aw8BN3YqcyPwQDj9CHCdmVmUQbn7bnd/IZw+CqwnGLM5E9wI/NQDzwIlZjZhEF//OmCzuw/0jvKz5u5PEYyZkSz5c/QAcFMXm74DWOHuB929AVgBLIwqJndf7u6t4eyzBCMDDrpujldf9OX7G0lc4W/AzcAvUvV6fYypu9+GQft8ZWMiqAB2JM3X8cYf3FNlwi/NYWDcoEQHhJeiLgFWdbF6rpm9bGaPmtn5gxSSA8vN7HkzW9zF+r4c0ygtovsvZzqOV4dyd98dTu8Byrsok85j9wmCM7mu9PaeR+WO8LLV/d1c6kjn8XorsNfdN3WzPvJj1um3YdA+X9mYCIY0MysGfg18wd2PdFr9AsHlj1nAvwK/G6SwrnL3S4Hrgc+a2dWD9Lq9smCY0xuAX3WxOl3H6w08OE8fMm2xzeyrQCvw826KpOM9/zegErgY2E1wGWYouYWezwYiPWY9/TZE/fnKxkSwE5icND8pXNZlGTPLAUYDB6IOzMxyCd7on7v7bzqvd/cj7n4snF4K5JpZadRxufvO8Hkf8FuC0/NkfTmmUbkeeMHd93Zeka7jlWRvxyWy8HlfF2UG/diZ2ceAdwMfDn9A3qAP73nKufted29z93bgP7p5zbR81sLfgfcBD3dXJspj1s1vw6B9vrIxEawGZpjZtPC/yUXAkk5llgAdtesfAB7v7guTKuH1xx8D6939u92UGd9RV2Fmcwjen0gTlJkVmdnIjmmCysZXOxVbAnzUAm8BDiedskat2//S0nG8Okn+HN0G/FcXZZYBC8xsTHgpZEG4LBJmthD4H8AN7t7YTZm+vOdRxJZcr/Tebl6zL9/fKLwdeM3d67paGeUx6+G3YfA+X6muAR8KD4JWLhsJWh98NVx2N8GXA2AEwaWGWuA5YPogxHQVwandGuCl8PFO4Hbg9rDMHcBagpYSzwJXDEJc08PXezl87Y7jlRyXAfeEx/MVYPYgvY9FBD/so5OWpeV4ESSj3UALwXXYTxLUKz0GbAL+BIwNy84G7kva9hPhZ60W+HjEMdUSXDPu+Ix1tI6bCCzt6T0fhOP1s/Dzs4bgR25C59jC+Td8f6OMK1z+k47PVVLZQTlmPfw2DNrnS11MiIgMc9l4aUhERPpBiUBEZJhTIhARGeaUCEREhjklAhGRYU6JQIaMsHfKL6c7jp6Y2QfNbL2ZPZGCfW0d5BvgOl53opk90kuZqck9dEp2y0l3ACKpZmY5frrjtVT7JPBpd386ov1Hzt13EdxIKQLojEDSzMy+amYbzexp4Nyk5ZVm9t9hB19/NrPzkpY/G/YL/7/N7Fi4fF5YbgmwzsziFvTNvzrs5OwzSfv+x6Tl/6ubuG4JX+NVM/tWuOwugpt/fmxm/9yp/Dwze8rM/mhBX/o/MrNYd/vqtO3dljTWgpl9w8w+H+5zpZk9YsEYAz9PupP6OjN7Mdzv/WaWHy7fambftLDPfDO71MyWmdlmM7s9LHPqv/1w+s9m9kL4uKLfb6JkvlTfUaiHHn19AG8muNO0EBhFcGfkl8N1jwEzwunLCboBAfgDcEs4fTtwLJyeBxwHpoXzi4GvhdP5QA0wjeAW/HsJ7paOhfu7ulNcE4HtQILgrPlx4KZw3Uq6uLM6fP0mgjtQ4wTdAX+gl31tBUqBqQT9KRHGtJngrtJ5BD3jTgqXP0OQiEYQ3D08M9zmpwQdlXXs82/D6e8R3K06Mnz9veHyqYT98YfHfkQ4PQOo6VxGj+x/6NKQpNNbgd962CdO+N98Ry+MVwC/stPDROSHz3M53S/7g8B3kvb3nLu/Hk4vAC4ys45LIKMJfugWhI8Xw+XF4fKnkvZzGbDS3evDeH5OMKDJ73r5e55z9y3hNr8g+NFu6W1f7r7VzA6Y2SUEXQ2/6O4Hwr/9OQ/7v7Fg5KypwFHgdXffGO7iAeCzwPfD+Y6+eV4Bij3o4/6omZ20N45Ylgv80MwuBtqAmb38jZKFlAhkKIoBh9z94n5udzxp2oDPufsZHXCZ2TuAb7r7v59diF3q3F9Lf/pvuQ/4GDAeuD9p+cmk6Tb69p3t2Ka90/btXWz/D8BeYBbBcW/qc8SSNVRHIOn0FHCTmRWEPTu+B4LupYHXzeyDcGrM5FnhNs8C7w+nF/Ww72XA31rQvS9mNjPsNXIZ8InwrAMzqzCzsk7bPgdcY2alZhYn6AH1yT78PXPCXjNjwIeAp/uxr98SjCx1Gb33HrkBmGpmVeH8R/oYX1dGA7s96Br6IwSXtWSYUSKQtPFgeL6HCXp0fJSgC+IOHwY+aWYdvT12DFf4BeCLZrYGqCK4ht6V+4B1wAthxei/E4zlu5zgktIzZvYKwVClIzvFtZtgjNgnwtied/euugDubDXwQ4KhBl8nuOzVp315MCzjE8Av3b2tpxdx9ybg4wSXzl4h+E//R32Iryv/F7gtPM7nceZZlQwT6n1UMoqZFQIn3N3NbBFBxXHKxrQ9i7jmEVR0v3uA28cIRlz7oHc/VKJIJFRHIJnmzQSVmwYcIuiLPaOZWTVB66XfKglIOuiMQERkmFMdgYjIMKdEICIyzCkRiIgMc0oEIiLDnBKBiMgw9/8Bc+hcx512iqYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "## EDIT THIS CELL\n",
        "K_max = 20\n",
        "rmse_train = np.zeros((K_max+1,))\n",
        "\n",
        "for k in range(K_max+1):\n",
        "    \n",
        "     \n",
        "    # feature matrix\n",
        "    Phi = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # # maximum likelihood estimate\n",
        "    theta_ml = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # # predict y-values of training set\n",
        "    ypred_train = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # # RMSE on training set\n",
        "    rmse_train[k] = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    assert rmse_train[k] != -1, \"RMSE not implemented\"\n",
        "    \n",
        "    #@ end\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(rmse_train)\n",
        "plt.xlabel(\"degree of polynomial\")\n",
        "plt.ylabel(\"RMSE\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EnbLPIyNjmsL"
      },
      "source": [
        "#### Question: \n",
        "1. What do you observe?\n",
        "2. What is the best polynomial fit according to this plot?\n",
        "3. Write some code that plots the function that uses the best polynomial degree (use the test set for this plot). What do you observe now?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "G0RWggS7jmsL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "732b75de-ba3c-447b-fead-2379124642a8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzG0lEQVR4nO3deXiU1fXA8e8hYZWdAAIBEpTIIqsB2QUjioKouKFWpWr52bpQ6lLUulRrSysVt7qjuFHrBiLgEgQERIUgAURkNUAAWQIJIAQIub8/7iQGzDJJZua+M3M+z5NnJjPvvO8JJHPmbueKMQallFKqLFVcB6CUUio8aMJQSinlF00YSiml/KIJQymllF80YSillPJLrOsAgiUuLs4kJCS4DkMppcLK0qVLdxtjGhf3XMQmjISEBNLS0lyHoZRSYUVENpX0nHZJKaWU8osmDKWUUn7RhKGUUsovETuGoULv6NGjZGZmkpub6zoUFSFq1KhBfHw8VatWdR2KQhOGCqDMzEzq1KlDQkICIuI6HBXmjDFkZWWRmZlJYmKi63AU2iWlAig3N5dGjRppslABISI0atRIW6weoglDBZQmCxVI+vvkLZ5IGCLyiojsFJHvSnh+oIjkiEi67+uBUMeolFJhIf2/sHRyUE7tiYQBTAaGlHHMAmNMV9/XwyGISYW5hx56iAkTJpT4/LRp0/j+++9DGJFSIfDt67D8f0E5tScShjFmPrDHdRzKjYmpa51cVxOGikj7MqFei6Cc2hMJw0+9RWS5iHwsIh2LO0BERotImoik7dq1K9TxqQp68vN1ATvXo48+SlJSEv369WPNmjUAvPTSS/To0YMuXbpw6aWXcvDgQRYtWsT06dO566676Nq1Kxs2bCj2OKXCSn4+7NsOdaM7YXwLtDbGdAGeBqYVd5Ax5kVjTLIxJrlx42JrZ6kItnTpUt5++23S09OZNWsWS5YsAWDEiBEsWbKE5cuX0759eyZNmkSfPn0YPnw4jz32GOnp6ZxyyinFHqdUWPl5J+QfhXrxQTl9WKzDMMbsK3J/log8KyJxxpjdLuNSFTcxde1xLYuEcTMBGJPSlrGDkyp0zgULFnDJJZdQq1YtAIYPHw7Ad999x1/+8heys7M5cOAA5513XrGv9/c4pTwrZ6u9jeaEISInAzuMMUZEemJbRlmOw1KVMHZwUmFiSBg3k4zxQ4N2rVGjRjFt2jS6dOnC5MmTmTdvXqWOU8qzcrbY20jukhKR/wJfAaeJSKaI3CgiN4vIzb5DLgO+E5HlwFPASGOMcRWv8qYBAwYwbdo0Dh06xP79+/noo48A2L9/P82aNePo0aO89dZbhcfXqVOH/fv3F35f0nFKhY19UdDCMMZcVcbzzwDPhCgcFWJjUtoG5Dzdu3fnyiuvpEuXLjRp0oQePXoA8Mgjj3DmmWfSuHFjzjzzzMIkMXLkSH73u9/x1FNP8d5775V4nFJhI2crxNaEmg2CcnqJ1A/qycnJRjdQCq3Vq1fTvn1712GoCKO/V+XwznWwYxXctrTCpxCRpcaY5OKe80SXlFJKqQDIyQxadxRowlBKqciRsxXqasJQSilVmrwjcGBH0FZ5gyYMpZSKDPu3AyZoU2pBE4ZSSkWGwim1mjCUUkqVJifT3tZrGbRLaMJQqgzTp09n/Pjxzq4/cOBACqaIX3DBBWRnZ5ORkcHpp5/u9zmKlnp/4IEHmD17NgAJCQns3h34Cjslnffdd9+lffv2DBo0iLS0NG6//XYA5s2bx6JFiwIeR1QpSBhB7JLyxMI9pbxs+PDhhXWpXJs1axYA2dnZFT7Hww+7205m0qRJvPTSS/Tr1w+A5GQ73X/evHnUrl2bPn36OIst7O3bCjXqQfXaQbuEtjBUxMjIyKBdu3aMGjWKpKQkrrnmGmbPnk3fvn1p27YtixcvBmDx4sX07t2bbt260adPn8Iy6BMnTuSGG24AYOXKlZx++ukcPHiQyZMnc+uttwK23tTvf/97evXqRZs2bZg3bx433HAD7du3Z9SoUYWx1K79yx/te++9V/icv68vSXGf3Ddu3Ei3bt1YsmQJGzZsYMiQIZxxxhn079+fH3744VfnGDVqFO+9917h908//TTdu3enU6dOhcfv2bOHiy++mM6dO9OrVy9WrFhR6uNZWVmce+65dOzYkZtuuoniFgQ//PDDLFy4kBtvvJG77rqLefPmMWzYMDIyMnj++eeZOHEiXbt2ZcGCBWX+O6hiBHlKLWgLQwXLx+Pgp5WBPefJneD80ruG1q9fz7vvvssrr7xCjx49mDJlCgsXLmT69On8/e9/Z9q0abRr144FCxYQGxvL7Nmzuffee3n//fcZM2YMAwcOZOrUqTz66KO88MILhZVvi9q7dy9fffUV06dPZ/jw4Xz55Ze8/PLL9OjRg/T0dLp27VpqjJV9fVFr1qxh5MiRTJ48mS5dupCSksLzzz9P27Zt+eabb/jDH/7AnDlzSj1HXFwc3377Lc8++ywTJkzg5Zdf5sEHH6Rbt25MmzaNOXPmcN1115Genl7i43/961/p168fDzzwADNnziy2NPwDDzzAnDlzmDBhAsnJyYXFHRMSErj55pupXbs2d955p98/uzpBEDdOKqAJQ0WUxMREOnXqBEDHjh1JSUlBROjUqRMZGRkA5OTkcP3117Nu3TpEhKNHjwJQpUoVJk+eTOfOnfm///s/+vbtW+w1LrzwwsJzNm3a9LjrZWRklPmGX9nXF9i1axcXXXQRH3zwAR06dODAgQMsWrSIyy+/vPCYw4cPl3meESNGAHDGGWfwwQcfALBw4ULef/99AM4++2yysrLYt29fiY/Pnz+/8LVDhw6lQYPg1DJSpcjJhPgeQb2EJgwVHGW0BIKlevXqhferVKlS+H2VKlXIy8sD4P7772fQoEFMnTqVjIwMBg4cWPiadevWUbt2bbZt21bmNYqe/8RriEjh47m5ueV+vT/q1atHq1atWLhwIR06dCA/P5/69euTnp7u9zmKxhMTE1Ou6ysPOXIQDu0N6oA36BiGikI5OTm0aGH/sCZPnnzc47fffjvz588nKyvruH7+8mratCmrV68mPz+fqVOnVjbkYlWrVo2pU6fy+uuvM2XKFOrWrUtiYiLvvvsuAMYYli9fXqFz9+/fv7DE+7x584iLi6Nu3bolPj5gwACmTJkCwMcff8zevXvLdb0TS82rcgpyWfMCmjBU1Ln77ru555576Nat23GfqMeOHcstt9xCUlISkyZNYty4cezcubNC1xg/fjzDhg2jT58+NGvWLFCh/8pJJ53EjBkzmDhxItOnT+ett95i0qRJdOnShY4dO/Lhhx9W6LwPPfQQS5cupXPnzowbN47XXnut1McffPBB5s+fT8eOHfnggw9o1apVua534YUXMnXqVB30rqgQTKkFLW+uAkjLUKtg0N8rPyx7Ez68BW5fBg3bVOpUWt5cKaUiWYhaGJowTnR4P6S9Aju+dx2JUkr5JycTTmoCsdXLPrYSNGGc6NhRmDEWNs51HUlYitQuTuWG/j75ad/WoK/BAE0Yv1azAVSrA3s3uY4k7NSoUYOsrCz9I1cBYYwhKyuLGjVquA7F+4K8014BXYdxIhFo0BqyN7uOJOzEx8eTmZnJrl27XIeiIkSNGjWIjw/+G2FYM8a+X7U9N+iX0oRRnPqtYG+G6yjCTtWqVUlMTHQdhlLR5cBOyMuF+q2DfintkipOfV8LQ7tWlFJeV9AbUr98a18qQhNGceq3giMH4OAe15EopVTpsn3jrQ20heFGwT98tg58K6U8ruB9Kog77RXQhFGcgqadJgyllNdlb4ZajYK6cVIBTRjFKUwYOlNKKeVxezeFZMAbPJIwROQVEdkpIt+V8LyIyFMisl5EVohI96AGVKMe1KivazGUUt6XvTkkA97gkYQBTAaGlPL8+UBb39do4LmgR6RrMZRSXpefDzlbQjLgDR5JGMaY+UBpU5IuAl431tdAfREJXs1osBlbxzCUUl524Cc4diTqWhhlaQFsKfJ9pu+x44jIaBFJE5G0Sq821rUYSimvK1yDEUUtjEAxxrxojEk2xiQ3bty4cier39qunjxQsQ10lFIq6DRhFGsrUHSScbzvseApXIuh4xhKKY8q6DavH/w1GBA+CWM6cJ1vtlQvIMcYsz2oV9S1GKqSJqaudR2CinR7N9l9MKrWDMnlPJEwROS/wFfAaSKSKSI3isjNInKz75BZwEZgPfAS8IegB6UJQ1XSk5+vcx2CinTZm0M2Qwo8Uq3WGHNVGc8b4JYQhWNVOwlqxWmXlAqpialrGTs4yXUYKlxkb4YWwV2WVpQnWhie1aC1Lt5T5TIxdS0J42aSMG4mQOF9f7untFWi/JZ/zG6cFKIBb/BIC8Oz6reC7StcR6HCyNjBSYUthIRxM8kYP9RxRCpi7d8O+UdDtgYDtIVRuvqt7SrK/HzXkagIVtlWSUWvqcJcCPfBKKAtjNLUb2VXUR74Ceo2dx2NCjNjUtr6dZyLVsmTn6/TsZJwV9Bd3iAhZJfUhFGagtkHezdpwlDlVviGnHcYMtNg05ewLR0O7LALQnNzoHodqFnflqdu1oXzqsTAgR5Qu4nL0FU4yN4MCNQL3Z7nmjBKU7/I4r3Wvd3GosJLfj5kzIe0V2HtJ7ZqAAKNT4M6zaDRqbYq8uH9NnHs3w7fPM8L1Y7AhInQuh90uwY6XGRn7AXAxNS1xw2qF3SBjUlpq62NcJS92f4uxVYP2SU1YZSmYAervRlOw1Bh5FgeLHsDFj0NezZAzQbQ/TpoM8h+6KjZoOTXHs2Fn1bAxi9g+RSY9nuYdRf0HA19x9iWSCXogHyEyd4U0vEL0IRRuqo1oG4L2Puj60hUOFg3Gz77C+xaDS3OgEtetC2EqjX8e33VGtCyp/0acCds/hoWvwgLH4e0V6D/HTZ5+Hs+Fdn2boJWvUJ6SU0YZWnYBvZsdB2F8rKfd8NHY+CHGdAgEa54A9pfCCIVP6eIbZG07g39xsLnf4XU+yF9Cox4AZp1qVTI/g7IK4/KOwz7Mu37UwjptNqyNEyEPdrCUCVYNxue6wPrPoNzHoJbvoEOwyuXLE7UrDP85n24+l04tBdeOhu++Jft/qogHbMIc9mbweRrwvCchm3g5512cFKpAvnH4LP74a1LoWZD+N0c2xII5gBk0rnwh6+gw8Uw91F48xI4WNq+YypiFfR6aMLwmIL/EG1lqAJHfob//QYWPQXJN8LouXByp9Bcu1ZDuGwSXPSsHeN4OQV2azmRqJO1wd5qwvCYwoSh4xgK2LcdXj3fTpU9/zEY9njISksfp9s1cP0MyN0HL6XYmVUqeuzZCNXr2Q8QIaQJoywNEu2tJgy1dxO8cq79dHfV23DmaLfxtDrTtm7qtYC3LrfjKSo67Nlox1cDOVbmB00YZaleG2o31YQR7fZugteG2UV2138ESee5jsiq34rnEp60CwLfvgrWfOw6IhUKezaGvDsKNGH4p4HOlIo05Sq+VzRZXPdhSPcf8Mc/5++C66dD09Pt2MrqGa5DUsF07KidJaUJw6N0LUbE8XvfiQM74fXhvySL5t2CG1hF1WwA102z8b33W8hY6DoiFSzZm8Ec04ThWQ3bwP5tcOSg60hUKB35GaZcYZPGb6Z6KlkUWxL9oYU82/wftkX836thxyrHUaqgKOjtcJAwdKW3Pxr6Br73ZkDTDk5DURVXruJ7x/Lg3d/C9uUw8r8Qf0YoQy1TqXWhst+HSYPhzUvhxlSo39JRlCooHK3BAE0Y/ik6tVYTRtjyu/ieMTDrTlj3KQx9HE4bEsIoA6B+S7sy/JXzbQvpxlQ7eUNFhj0boVptJyXwtUvKHw11am1UWfqq/er7R+hxo+toylRsXaimHeGKybDrB/jwDzYJqsjgaEotaMLwT80GtvyDJoyIUWLxvS1LYNbdcOpgSHkgtEFVUIl1oU45G875K3z/oa14qyKDoym1oAnDfw3baJnzCFLsm+yBnfDOtXYh3IgXoUrMcU+H5T7YfW6D0y+Dzx+Bdamuo1GVlX/MjqVqwvA4nVob2Y7lwbuj4FA2XPlmsSUX/J6K6yUiMPxpu0bj/ZsgJ9N1RKoycrZA/lFNGJ7XsI39Y8s77DoSFQzzH7N7bl/4ZOgKCYZKtVpwxWuQnwcfjLafUlV4cjhDCjRh+K9hoq0/n73ZdSQq0DZ/DfP/BZ1HQpcrj3uq2PUO42aGX/dUo1Ng6L9tUpz/mOtoVEU5Thg6rdZfRafWxuluZRHjUDa8/zu7N/IFv34jjah9sLuMhA1z4Yt/QuIAaN3HdUSqvPb8CLE1ofbJTi7viRaGiAwRkTUisl5ExhXz/CgR2SUi6b6vm0IepJY5jzzGwMw/wb6tcOkkqFHXdUTBN3QCNEiwSTI3x3U0qrwKptRWcfPW7TxhiEgM8B/gfKADcJWIFLc67n/GmK6+r5dDGiRArUZQve4vG5eo8LdqKnz3Pgy6B+KTyzw8IvbBrl4HRrxkS918ep/raFR5OZxSCx5IGEBPYL0xZqMx5gjwNnCR45h+TcR2RWWF4UwZ9WsHdtnV3M27Q9+xfr0kYvbBjk+GvmNg2Rs61Tac5B+zXVIFC4kd8ELCaAFsKfJ9pu+xE10qIitE5D0RKbY4joiMFpE0EUnbtWtX4CONS9LtMCPFrDvtPu0XPwsxUTiUN/AeaNwOpt8Gh/a6jkb5I3sTHDts34cc8ULC8MdHQIIxpjOQCrxW3EHGmBeNMcnGmOTGjRsHPoq4tra/+/D+wJ9bhc6qafD9NBg4Dpq0dx2NG7HVbbI8sBM+udd1NMofBR9W405zFoIXEsZWoGiLId73WCFjTJYxpmABxMuAm9KhBZk9a72Ty6sAOLgHZt4BzbpCnzGuo3GrxRnQ74+wfApsmOM6GlWWXWvsrcNZml5IGEuAtiKSKCLVgJHA9KIHiEizIt8OB1aHML5fFCQM7ZYKX6n3Q2529HZFnWjA3dDwFJjxJzh6yHU0qjS718JJjYutQhAqzhOGMSYPuBX4FJsI3jHGrBKRh0VkuO+w20VklYgsB24HRjkJtkEiSIz9j1PhZ9MiWPYm9L7VVnNVULUGDJto66TNn+A6GlWa3eucjl+ARxbuGWNmAbNOeOyBIvfvAe4JdVy/ElvNzmHXhBF+8o7AjLFQrxWcdbfraLylzVnQ5Sr48gnodFn0jut4mTGwew10uNhpGM5bGGFHZ0qFp6+esXtDDJ0A1U5yHY33nPuoXWf00R8hP991NOpEB7PsbDbHLQxNGOUV19YOemsBt/CxdxN88S9ofyEknec6Gm86qRGc+whs+RpW/M91NOpEBQPejTVhhJe4JDh2xM6JVuHhs/vswssh/3Qdibd1uRrie0DqA5C7z3U0qqiCbnBtYYQZnSkVXjZ+Aas/gv5/shsjqZJVqQLn/wt+3mULFCrv2L0WqtaCuvFOw9CEUV4Fc6B14Nv7juXBJ+OgfmvofZvraMJDi+7Q/Vr45vlfukGUe7vXQqNTnRUdLKAJo7xqNYRacZowwkHaK7DzezjvUTt9VPkn5UE7MeDjP9vZOcq93WuhsbsV3gU0YVSEzpTyvp+zYO7fIPEsaDfMdTTh5aQ4GHQfbJwLaz9xHY06chCytzgfvwBNGBUT11ZbGF73xT/h8AE4/592wFuVT/IN0KgtfPYXOHbUdTTRLWs9YDyxcZsmjIqIS7Lzon/Och2JKs7u9ZA2Cc64XhehVVRMVTvNNms9pL3qOproVjhDSrukwlNhEULtlvKk2Q9CbA1bwltVXNIQu5XrvH/YrWyVG7vXglSx+7I7pgmjInSmlHdtWgQ/zLBVWGs3cR1NeBOxK8AP7YUFWmfKmV1rbEmi2OquI9GEUSH1W0FMdU0YXpOfb7cdrdMcet3iOprI0KwzdL0Gvn7e7vamQs8DRQcLaMKoiCoxtpWx8wfXkaiivp8K276FlPuhWi3X0USOs++zv/Nz/+46kuiTf8yOI3lgwBs0YVRck/aw0822HOp4E1PX2pk8c/4GTU+Hzle6Dimy1G0OvX4PK9+B7StcRxNd9my027I26eA6EkATRsU16QD7MnUw0AOe/HwdfPu6/eNKecB+GlaB1feP5MbWhc//6jqS6LLjO3vrkf1bNGFUVMF/4C7tlnKtBoftuotWvaHtua7DiUw16zPh0IWwfjb8ON91NNFjxyq7aZsHptSCJoyKK5jfv2OV2zii1MTUtSSMm0nCuJn8NuZTOLCDy9YNZuJsneocLG8cG2yL36U+qCVDQmXH97aGlEdK23hix72wVK+l3XBm5/euI4lKYwcnMXZwEhzaS874myBpCO9dfYfrsCLOxNS1tssPgGrcmTWUCfte4KO3X+DCq252GltU2PEdtDjDdRSFymxhiEiqiHQJRTBhRUQHvr3gyyepwyE4+37XkUSksYOTyBg/lIzxQwGY8Mg/IO40Lsx6RTcRC7bcfXbfHY+MX4B/XVJ/Bp4QkVdFpFmwAworTdrbLiltnrtxYCd88wJrG58LJ5/uOproUCUGBt1r95de+a7raCJbwYfRcEoYxphvjTGDgBnAJyLyoIjUDH5oYaBJR8jNhv3bXUcSnRY+AXm5tBv5qOtIosKYFN9agPbD4eTOtmSIFiYMnp2+8dFwShgAIiLAGuA54DZgnYhcG8zAwkJT39xoHccIvX3bbIHBziM9s6gp0o0d7FttXKWK7QLcmwHL3nAaU0TbscqOk9Zr6TqSQv6MYXwJbAUmAi2AUcBAoKeIvBjM4DyvYDHNDk0YIbfg35CfB2fd7TqS6NR2MMT3hC8eg6O5rqOJTDtW2fcYD5Xn96eFMRpoYYwZbIy53xgzwxiz3hhzG9A/yPF5W62GUPtkbWGEWvZmWPoadPsNNEx0HU10ErElWPZvszsbqsAyxn4Q9VB3FPg3hrHKmBJHdYcGOJ7w07SDJoxQmz/BvmENuMt1JNEtcQAk9IeFE+2ucCpwcjLhcE74JYzSGGM2BiqQsNWkgy0/rFMMQ2PvJkh/C7pfD/XiXUejBt0LP++040kqcHZ4b8AbdKV35TXpAHm5to6RCr4F/7abyfQb6zoSBdC6D7QZaGesHfnZdTSRo6CGlMd2jPREwhCRISKyRkTWi8i4Yp6vLiL/8z3/jYgkOAizeDpTKnSOa120cB2NKjDwXji4G5a87DqSyLFjld13p0Y915Ecx3nCEJEY4D/A+UAH4CoRObGW743AXmPMqdjZWv8MbZSliDsNEJ0pFQrauvCmVmfCKSnw5ZNw+IDraCLDjlW2VL/HOE8YQE9gvTFmozHmCPA2cNEJx1wEvOa7/x6Q4lsb4l61WtCwzS9NSBUcha2L67R14UWD7oWDWbA4umfaB8TRXLtpkkf2wCjKCwmjBbClyPeZvseKPcYYkwfkAI1OPJGIjBaRNBFJ27VrV5DCLcbJneCnlaG7XjRa+Li2LrwsPhlOHQyLntZWRmXtXAXmmCfL3XghYQSMMeZFY0yyMSa5cePGobtwsy62SNjBPaG7ZjTJ3gLL3rLrLnRmlHcNHAeH9uhYRmVtW2Zvm3d3G0cxvJAwtgJF177H+x4r9hgRiQXqAVkhic4fzbva2590+8qg+PJJwGjrwuvik+1YxqKndcZUZWxbBrUa2UFvj/FCwlgCtBWRRBGpBowEpp9wzHTget/9y4A5pSwmDL1mXe3ttnSXUUSm/T/Z7Ve7XOXJPyB1goHjfDOmdF1GhW1Lh+bdPFUSpIDzhOEbk7gV+BRYDbxjjFklIg+LyHDfYZOARiKyHvgT8Kupt07Vagj1WsH25a4jiTxfPmVrRvX/k+tIlD9a9oQ2g2yrUFsZ5XfkoC1r3ryb60iK5Ykd94wxs4BZJzz2QJH7ucDloY6rXJp11oQRaAd22TpFna+wM9FUeBg4Dl45z/7f9bnNdTTh5aeVdsDbownDeQsjYjTvCns2QG6O60gix1fP2FX0/XXr1bDSqhcknmVbh0cPuY4mvBQOeGvCiGwF4xg6vTYwDvpm25w+Qve7CEdn/dnWmFr6WtnHql9sWwa1m0Idb25uqgkjUJr5tj3Xge/A+OYFOHIA+t/pOhJVEQl9oXU/+PIJ3S+jPLYts9NpPTjgDZowAqd2E6jTXMcxAiF3H3zzHLQb9kutLhV+zrrLbl+c/qbrSMLD4f2we61nu6NAE0ZgNesC29NdRxH+0ibZsSAduwhviWdByzN9e68fcR2N921fARhNGFGjeVfYvU5LI1TGkYOw6Bm7AKyF91a6qnIQsVvo5myB5VNcR+N9hQPeXZ2GURpNGIHUrAtgtBBhZXz7ml34pbvpRYZTUqDFGbDgcTh21HU03rZtGdSNt93bHqUJI5B0xXfl5B22UzFb94XWvV1HowKhYCvd7E2w8l3X0Xjbtm+hhXe7o0ATRmDVORlOaqID3xWVPgX2b4MBOjMqoiQNsRWdF/xbtzIuyaFsu2unh8cvQBNGYInY/sdt37qOJPwcy7NTMJt3s6UlVOQQgQF32z0eVk11HY03bU2ztx6sUFuUJoxAi+8Ju36wnxiU/757H/Zm2HUXHp2Driqh3TC7IdD8xyA/33U03rP5a5AYiO/hOpJSacIItJa+//CCTwyqbPn5doOkxu3htAtcR6OCoUoV29W46wdYfWIxasXmr209uuq1XUdSKk0YgdbiDLsz3JbFriMJH2tm2jeS/nfYNxYVmTpcDHFJ2so4Ud4RyEyDVt6f6KF/nYFWvQ406agJw1/GwPwJ0CAROl7iOhoVTFVibJfjju9g7ceuo/GOn1ZA3iFbtNHjNGEEQ8ue9hODzggp24bP7er4fmMhxhPV9lUwnX6pLVX/xT/thwUFm7+yty01YUSnlj3hyH7bzaJKN//fULeF3VFPRb6YWNvK2L4c1n3mOhpv2Py1TaJ1mrqOpEyaMIKhZU97u+Ubt3F4XcaXsHkR9LkdYqsBMDF1reOgVNB1vgLqt9ZWBtiff/NXYTF+AZowgqNBItSKgy1LXEfibQsm2H+n7tcVPvTk5+scBqRCIqaq3XJ361LbJRnNstbDwaywGL8ATRjBIeIbx9CB7xJtXQob5kDvW6BaLdfRqFDrcjXUawnzoryVUTB+oS2MKNeyp/308HOW60i8acHjUKMe9LiJialrSRg3k4RxMwEK72v3VASLrWYnOmQuho1zXUfjzuavoVYjaHSq60j8otNSgiXeN46RuQROG+I2Fq/Z8T38MMNu41mjLmMH12Xs4CTAJouM8UMdB6hCottv7AeHeeNtOZhoXOFfMH4RJj+7tjCCpXk3qBKrA9/FWfBvqFYbzrzZdSTKpdjq0H+s/RvZOM91NKG3f4ctOBgm4xegCSN4qtWyFTp1Ad/xdq+HVR9AjxuhVsNfPT0mpa2DoJQz3a6106rnjY++sYyMBfa2VR+3cZSDJoxgatXb1pTKO+w6Eu9YOBFiqkPvW4t9uqBrSkWJ2Op2LGPL1/DjF66jCa0Nc6FmA0/vsHciTRjBlDgA8nLtOIaCvZtgxdtwxihP7yqmQqz7dVCnOcz9R/S0MoyxswTbDLQlU8KEJoxgat3HFiL8cb7rSLzhyyfsv0ef21xHorwktjoMuMO2MjbMcR1NaOz6wW4WdsrZriMpF00YwVSjnh381oQBOVth2ZvQ9Wqo18J1NMprul1r12XMfTQ6WhkFiTHMNgtzmjBEpKGIpIrIOt9tgxKOOyYi6b6v8CqmnzjAdkkd+dl1JG59+SSYfOj3J9eRKC+KrW73/t66tNw1psJyvc6GObbUe/2WriMpF9ctjHHA58aYtsDnvu+Lc8gY09X3NTx04QVA4gDIz4NNX7mOxJ1922HpZFtgsEFr19Eor+p6NZvym5S7lRF25WSO5to6amHWHQXuE8ZFwGu++68BF7sLJUha9oIqVaNvBkhRi56ySbO/ti5UKWKq8lTeCFvJ9ocZrqMJni1f2/0vwjBhuF7p3dQYs913/yegpPq+NUQkDcgDxhtjpoUiuICoVsuWCYnWcYz9OyDtFeh8pS3hrNQJJqauLWwlxNCXP+R/yNH/3sMn/U7lj+e2L/M1QGFZmTEpbb0/NXvDHPshsnVf15GUW9AThojMBk4u5qn7in5jjDEiUlI7tLUxZquItAHmiMhKY8yGYq41GhgN0KpVq0pGHkCJA+zCpEN77bzraLLoKTh2xO7nrFQxxg5OOq40zCmX/w3ev5F2J68Eik8YJ74mrMrJbJhjV3d7fP/u4gS9S8oYc44x5vRivj4EdohIMwDf7c4SzrHVd7sRmAd0K+G4F40xycaY5MaNGwfl56mQxAGAsf2W0eTALtu66HQ5NDrFdTQqXHQcYaskzH3U7ncdSQ7shJ9WwinhNTuqgOsxjOnA9b771wMfnniAiDQQkeq++3FAX+D7kEUYCC2SIbZm9HVLffmEXbg44C7XkShHyjuDaUxKW6hSBc5+APZmwLLX/XtNuFiXam/DcPwC3CeM8cBgEVkHnOP7HhFJFpGXfce0B9JEZDkwFzuGEV4JI7YatO4dXQPf+3+CJS/bsYu4MPqDVgFV3hlMheMPbQfb0jpf/KvMKemeH7MoavV0u96kWVfXkVSI04RhjMkyxqQYY9r6uq72+B5PM8bc5Lu/yBjTyRjTxXc7yWXMFdZmkF3dmb3ZdSShsfAJOHZUWxeqYkQg5UE4sAO+ecF1NIGRu8+OX7S/MGzKmZ/IdQsjepx2vr1d+6nbOEJh33Y7dtH1Kh27iEIB2xCrdW9oe67t2jy4J/CBhtq6z+wEkA4XuY6kwjRhhEpcW2h4Cqz52HUkwbfwcTDHtHURpcYOTiJj/NDCmUsF9yvUdXTOQ3B4P8yfENggXfh+GtQ++ZfN1cKQJoxQOu18WwP/8H7XkQRPTqZd1d31GmiQ4DoaFe6adrS/S4tfhD0/uo6m4o78DOtmQ/thdlA/TIVv5OEoaYhtkm6I4D2M5423t9q6UARoBtOg+yCmKnz+18qfy5X1s+3q7vbhVdnoRJowQqlVL6hRP3K7pXavg/S3IPnGsCuqpoIjIDOY6jazJfFXTYUtYbq3zPfToWbDsFzdXZQmjFCKqWqnC677FPKPuY4m8Ob8za436X+H60hUpOlzO5zUBD77S/iVPz+aC2s/sd1RMa6rMVWOJoxQSxoCB7MgM811JIG1Ld0O6vW+BWp7aJW9igzVa8PZ99nCfd+97zqa8tkwB44cgPbhOzuqgCaMUDv1HKgSC2sjrFtqziO2Tlaf4vfqVqrSul0LJ3eGz+4Pr/1l0t+CWnG+EkHhTRNGqNWsb1ewrvnEdSSB8+MCO6jXb6zdZVCpYKgSAxc8Zrc2XfC462j8s/8nO2bZ9Wpb8SHMacJwod1Q2LUadq1xHUnl5efbfuW68dBztOtoVKRr1Qs6XWGrIO/Z6DqasqW/Zdckdb++7GPDgCYMFzqOAKkCK95xHUm5FLtS97v3YXs6pNwPVWuGPCYVhQY/bPeT+ORe15GULj+f7C8nQet+EHeq62gCQhOGC3WaQpuBsPLdsJrx8atCckdz7dz4kzvbT31KhULdZnDWXXYc8IdZrqMpWcYC6uduhe7XuY4kYDRhuNLpCsjeBFsWu46k4ha/ADlb4Ny/hfXqVRWGet0CTTrAzDtsUT8v+vY1ckwt6BDei/WK0r9yV9oPs2sWVvzPdSSlKqmQ3HOzFsP8f9vicG3Ochylijqx1WD407B/u52hV07lLoRYznN3G/dfDq/8kA+O9Sfh/jkVK77oQeG9iiScVa8D7S6wq1fP/6dd1OdBJW6FOeNPdm754IcdRqeiWnyynWix+EW7q2NL/4v6Pfn5uqDtozF2cBJja30MqXm8fWxQeG0fWwZtYbjU6Qo4tAfWf+46kvLZvgKWvgo9boImxe+5rFRIpNwPdVvA9Nu9s53r0UOw6BlIPIs1ppXraAJKE4ZLp6bY+jIe75YqMCalrR2k//huu0hv0D2lHh8JTXDlcdXrwLDH7TT1eX8v9dCA7dNRlqWvwc874aw/h9f2sX7QLimXYqrC6SNg2ZuQm+P5RW9jByfBindh81dw4ZM2aZQimM1+pQolnWfXOSx8wlZSSOhX7GEldq8G0tFcu+FT636Q0JexCYG/hEvawnCt27WQl2uTRgWF7JP84QOQer/dj7jbtaG5plL+OO/v0DARpt4Mh7LdxbHsDTsQf1ZklvfXhOFa8662VMjiFytcwfZX6yOCZd4/7B/DBY/ZMg3FCFmzX6miqteGES/Bvm0wq+w366B0FeUdsa2clmdCYmTOHNQuKS8482Z493q733e7C1xHU7yt38LXz8IZvy11NkpImv1KFSc+Gc76sx3LSBwA3UtuBQelq3TZG7AvE4Y/CSKBP78HaMLwgnbDbC2mb57zO2FMTF17XMui4BP9mJS2gf9jOHbUzkI5qQkMDuNdz1Tk638HbF4EM/9kZ/DFJ4fmugd2wucPQ6s+cEpKaK7pgCYML4iJhZ43weyHYMf30LRDmS8J6Sf5r56BHSvhijfKNTAfaTNEVBiIiYXLXoUXB8L/fgOjv7CleILtk3Fw9KCdDBKhrQvQMQzv6H69Xfn9zfOuIzle1ga7T3e7YeUucaAzpJQTtRrCyCl25uE71wZ/fcbaz2wRzv53QuPI/p3XhOEVtRpClyvtmoyfd5frpUH7JH8sz846ialuB7qVChcnnw4X/Qe2fAPv32h/l4Ph8AHb/dW4nd0PJsJpwvCS3rfCsSMwf0K5Xha0T/ILH4fMxXZhVN3mwbmGUsFy+gg73Xb1dPjwFrt3SyAZAx//GXIy4cKnImKDpLJowvCSuLZ2fcOSl2FvhttYMtNsV1Sny6HTZW5jUaqiet8Cg/4CK962LYFAbicwfwKkvwkD7oRWZwbuvB6mCcNrBt5j9/ye8zd3MRw+AB/8zrYqLihfa0cpzxlwp+0uWvoqfDDa1nqqrPQpMPdv0HkkDLqv8ucLE04ThohcLiKrRCRfREqc/yYiQ0RkjYisF5FxoYwx5Oo2g95/sJsrbV8e+usbYz+J7fkRLnne7kGuVDgTgZQH4ey/wMp34NUL7AK/ilqXCtNvs4vzhj8d0bOiTuS6hfEdMAKYX9IBIhID/Ac4H+gAXCUiZc87DWd9x9g6TakPhv7aXz9rB94H3VtiTR6lwo4IDLjLzp7avRZeHATrZ5fvHPnHbDftlCugcXu48o2oGLcoymnCMMasNsasKeOwnsB6Y8xGY8wR4G3gouBH51CNevaXe+NcWP1R6K67YS589hdof6GdIqhUpGk3FG5MtaVE3rwU3rwMdv5Q9uuyt8Brw215nM5Xwg2feL5YaDCEw8K9FsCWIt9nAsWOMInIaGA0QKtWYV6Hvsfv7Cf96bdDfA+oc3Jwr7c3A977LcSdBhc/p1uuqsjVtAP8/itbv+2Lf8FzfexWA6cOtrf14iHvsP3aOM8OmG+YC7E14OLnoetVrn8CZ4KeMERkNlDcu919xpgPA3ktY8yLwIsAycnJAZwOEToTU9faabKx1WDEy/DCADsl8Jr3gtdXun8HvDECTD6MfMvuMaBUJIutBn1uhS5X2XLkP8yAdZ8Vf2zdeOj3R7u4tkHrUEbpOUFPGMaYcyp5iq1AyyLfx/sei0jH7SHROAnOfQRm3QmLX4IzRwf+ggf3wBsX2yq0106DRqcE/hpKedVJjezf2LmP2KoGG+bYFeKx1SGmGjTpAK37aovbJxy6pJYAbUUkEZsoRgJXuw0phHrcZKvYpt5vq8Q27xq4c+fmwBuX2D+Ua96JmrnkShWr0Sn6gakMrqfVXiIimUBvYKaIfOp7vLmIzAIwxuQBtwKfAquBd4wxq1zFHAyl7iEhYkscnNTYtgR+WhmYi+ZshdcuhB3f2dkebQYG5rxKqYglJpArHz0kOTnZpKWluQ6j3EqsPLtnI0weZnfnu36GXxVtS5SZBm9fDUd+hstesVtcKqUUICJLjTHFrovTjrlw0bANXP8RVKkKrw+HLUvKf478fFg6GV69gOy8WLhptiYLpZTfNGF4TKmVZxudAqNm2Ol9kwbDJ/fYVoI/MtNg0jnw0Rho2ZNB2ffbDWaUUspPmjA8pszKs3Ft4feLoMeNdlX2s73gq2dtl9WJcvfByvfsRjIvp9iqmhc/D9dNZy91g/MDKKUilo5hhLNNX8HHd8NPK+z3jdraWlTH8uxYx47vbLn0kxpDt9/wzNGLmPDFr2voBGVbV6VUWCptDEMTRiTY86Odert+NhzeDzFV7VeTDrbMR3wPqBJz3EuCvq2rUioslZYwwmEdhipLw0TodbP9UkqpINExjCgVtG1dlYpQE1PXug7BOU0YUUrHLJQqnyc/X+c6BOc0YSillPKLjmEopVQJJqauPa5lUVC+J1pnFuosKaWU8kO0zCzU0iBKKaUqTROGUkr5QWcWasJQSim/ROOYxYk0YSillPKLJgyllFJ+0YShlFLKL5owlFJK+UUThlJKKb9E7MI9EdkFbHIdRwXEAbtdBxFi+jNHB/2Zw0NrY0zj4p6I2IQRrkQkraRVlpFKf+booD9z+NMuKaWUUn7RhKGUUsovmjC850XXATigP3N00J85zOkYhlJKKb9oC0MppZRfNGEopZTyiyYMDxORO0TEiEic61iCTUQeE5EfRGSFiEwVkfquYwoGERkiImtEZL2IjHMdT7CJSEsRmSsi34vIKhEZ4zqmUBGRGBFZJiIzXMcSKJowPEpEWgLnAptdxxIiqcDpxpjOwFrgHsfxBJyIxAD/Ac4HOgBXiUgHt1EFXR5whzGmA9ALuCUKfuYCY4DVroMIJE0Y3jURuBuIilkJxpjPjDF5vm+/BuJdxhMkPYH1xpiNxpgjwNvARY5jCipjzHZjzLe++/uxb6At3EYVfCISDwwFXnYdSyBpwvAgEbkI2GqMWe46FkduAD52HUQQtAC2FPk+kyh48ywgIglAN+Abx6GEwhPYD3z5juMIqFjXAUQrEZkNnFzMU/cB92K7oyJKaT+zMeZD3zH3Ybsx3gplbCq4RKQ28D7wR2PMPtfxBJOIDAN2GmOWishAx+EElCYMR4wx5xT3uIh0AhKB5SICtmvmWxHpaYz5KYQhBlxJP3MBERkFDANSTGQuENoKtCzyfbzvsYgmIlWxyeItY8wHruMJgb7AcBG5AKgB1BWRN40xv3EcV6Xpwj2PE5EMINkYE24VL8tFRIYAjwNnGWN2uY4nGEQkFjugn4JNFEuAq40xq5wGFkRiP/W8BuwxxvzRcTgh52th3GmMGeY4lIDQMQzlFc8AdYBUEUkXkeddBxRovkH9W4FPsYO/70RysvDpC1wLnO37f033ffJWYUhbGEoppfyiLQyllFJ+0YShlFLKL5owlFJK+UUThlJKKb9owlBKKeUXTRhKKaX8oglDKaWUXzRhKBVCvr0hBvvu/01EnnYdk1L+0lpSSoXWg8DDItIEW7l1uON4lPKbrvRWKsRE5AugNjDQt0eEUmFBu6SUCiFfNeJmwBFNFircaMJQKkREpBl2n4+LgAO+Cr1KhQ1NGEqFgIjUAj7A7m+9GngEO56hVNjQMQyllFJ+0RaGUkopv2jCUEop5RdNGEoppfyiCUMppZRfNGEopZTyiyYMpZRSftGEoZRSyi//D23bwnoS2sVtAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# WRITE THE PLOTTING CODE HERE\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "#@ solution\n",
        "\n",
        "# feature matrix\n",
        "Phi = poly_features(X, 5)\n",
        "\n",
        "# maximum likelihood estimate\n",
        "theta_ml = nonlinear_features_maximum_likelihood(Phi, y)   \n",
        "\n",
        "# feature matrix for test inputs\n",
        "Phi_test = poly_features(Xtest, 5)\n",
        "\n",
        "\n",
        "ypred_test = Xtest*0 ## <--- EDIT THIS LINE \n",
        "\n",
        "#@ Test\n",
        "assert any(ypred_test), \"Prediction not implemented\"\n",
        "\n",
        "plt.plot(Xtest, ypred_test) \n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\")\n",
        "plt.legend([\"data\", \"maximum likelihood fit\"]);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvKIZI6RjmsM"
      },
      "source": [
        "The RMSE on the training data is somewhat misleading, because we are interested in the generalization performance of the model. Therefore, we are going to compute the RMSE on the test set and use this to choose a good polynomial degree."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "hHPCncvLjmsM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "940cf1fc-9bad-46cb-8749-04970b521679"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwzklEQVR4nO3deXhU5fnw8e89k30FEkBIwAREFNmUgIKouCECigooWFypiFutrVZ961L9tdVWq9atuFFrtYiILCoqKqAom+wgO4iBsIVANiAhyTzvH+cExpBMAsnMmczcn+uaa84259xzZjJ3znOeRYwxKKWUUjVxOR2AUkqp4KaJQimllE+aKJRSSvmkiUIppZRPmiiUUkr5FOF0AP6QmppqMjIynA5DKaUajSVLluw1xjSvbl1IJoqMjAwWL17sdBhKKdVoiMjPNa3ToiellFI+aaJQSinlkyYKpZRSPoXkPYrqlJWVsX37dkpKSpwOJSzExMSQnp5OZGSk06EopeopbBLF9u3bSUxMJCMjAxFxOpyQZowhLy+P7du3k5mZ6XQ4Sql6Cpuip5KSElJSUjRJBICIkJKSoldvSoWIsEkUgCaJANJzrVToCJuiJ6WUCkkVZZC7DnYsg4N50Pe+Bj9EWF1ROCk/P59XX331hF47cOBA8vPzfW7z2GOP8dVXX53Q/utj6tSprFmzJuDHVSoseSogdz2seB9m/AHevBSeSodxfWH6PbDgX+DxNPhh9YoiQCoTxZ133nnMuvLyciIiav4oZsyYUev+n3zyyXrFd6KmTp3K4MGD6dSpkyPHVypkGQP7f7KuFHKWwo7lsHM5HC621kfGQ6tukDUa0s6C1mdC00xwNfz//5ooAuShhx5i8+bNdO/enUsvvZRBgwbx6KOP0rRpU9atW8eGDRu46qqr2LZtGyUlJdx7772MGTMGONolSXFxMZdffjl9+/Zl3rx5pKWlMW3aNGJjY7n55psZPHgww4YNIyMjg5tuuomPP/6YsrIyJk2axGmnnUZubi7XX389O3bsoHfv3nz55ZcsWbKE1NTUI3FWVFQwevRoFi9ejIhw6623ct9997F582buuusucnNziYuL44033mDfvn1Mnz6db775hj//+c9MnjyZ9u3bO3WKlWqcyg9D0U4ozIGCHNizxkoOO5ZBSb61jTsaTuoC3UYeTQqpp4LLHZAQwzJRPPHxj6zZUdig++zUOonHrzijxvVPP/00q1evZvny5QDMmTOHpUuXsnr16iNVSMePH0+zZs04dOgQPXv2ZOjQoaSkpPxiPxs3bmTChAm88cYbXHvttUyePJlRo0Ydc7zU1FSWLl3Kq6++yrPPPsubb77JE088wUUXXcTDDz/M559/zltvvXXM65YvX05OTg6rV68GOFLkNWbMGMaNG0eHDh1YuHAhd955J7NmzeLKK688kqCUUlVUlEHhDvuRczQZVE4X7oDiPYDXkNSuCGjRCToNOZoUWnQCt3NtksIyUQSLXr16/aKdwYsvvsiUKVMA2LZtGxs3bjwmUWRmZtK9e3cAevTowdatW6vd9zXXXHNkm48++giA77777sj+BwwYQNOmTY95Xbt27diyZQv33HMPgwYNon///hQXFzNv3jyGDx9+ZLvS0tITe9NKhTJPBcx8FLLnW4mgahIAiE6CpNaQlAYtO0Ny+tH5pDRoejJExjoSfk3CMlH4+s8/kOLj449Mz5kzh6+++or58+cTFxdHv379qm2HEB0dfWTa7XZz6NChavdduZ3b7aa8vLzOMTVt2pQVK1bwxRdfMG7cOD744ANeeOEFmjRpcuRqSClVg9WTYcErcPK50KH/sUkgqTXEJDkd5XELy0ThhMTERIqKimpcX1BQQNOmTYmLi2PdunUsWLCgwWM499xz+eCDD3jwwQeZOXMm+/fvP2abvXv3EhUVxdChQ+nYsSOjRo0iKSmJzMxMJk2axPDhwzHGsHLlSrp161br+1IqbFSUwey/QssucNMnfrmp7JTQeSdBLiUlhXPPPZfOnTvzwAMPHLN+wIABlJeXc/rpp/PQQw9xzjnnNHgMjz/+ODNnzqRz585MmjSJk046icTExF9sk5OTQ79+/ejevTujRo3iqaeeAuC9997jrbfeolu3bpxxxhlMmzYNgBEjRvDMM89w5plnsnnz5gaPWalGY9m7Vi2lix8NqSQBIMaY2rdqZLKyskzVgYvWrl3L6aef7lBEwaG0tBS3201ERATz58/njjvu8Gtxkp5zFTbKSuDFM62iptEzoRH2TCAiS4wxWdWt06KnMJKdnc21116Lx+MhKiqKN954w+mQlAoNi9+Coh1wzeuNMknURhNFGOnQoQPLli1zOgylQktpEcz9B7S7EDLPczoavwitgjSllAq0Bf+y+li66FGnI/EbTRRKKXWiDu6DeS/BaYMhvYfT0fiNJgqllDpR3//TKnq68I9OR+JXmiiUUupEFO2Cha9Bl+HQMrQ7xdREESD16WYc4IUXXuDgwYP1jmPOnDnMmzev3vtRKuzN/Qd4yqDfQ05H4neaKAJEE4VSIWT/z7D433DmKEgJ/R6TNVEEiHc345Uts5955hl69uxJ165defzxxwE4cOAAgwYNolu3bnTu3JmJEyfy4osvsmPHDi688EIuvPDCavfdqVMnunbtyv333w9Abm4uQ4cOpWfPnvTs2ZPvv/+erVu3Mm7cOJ5//nm6d+/O3LlzA3cClAol3/wNxAXn/8HpSAIiPNtRfPYQ7FrVsPs8qQtc/nSNq6t2Mz5z5kw2btzIokWLMMZw5ZVX8u2335Kbm0vr1q359NNPAasPqOTkZJ577jlmz579i7EjAPLy8pgyZQrr1q1DRI50C37vvfdy33330bdvX7Kzs7nssstYu3YtY8eOJSEh4UhCUUodp9wNsGICnHMnJKc5HU1AhGeiCAIzZ85k5syZnHnmmQAUFxezceNGzjvvPH7/+9/z4IMPMnjwYM47z3cDnuTkZGJiYhg9ejSDBw9m8ODBAHz11Ve/GKK0sLCQ4uJi/70hpcLF7L9AZJxfxqYOVuGZKHz85x8oxhgefvhhbr/99mPWLV26lBkzZvDII49w8cUX89hjj9W4n4iICBYtWsTXX3/Nhx9+yMsvv8ysWbPweDwsWLCAmJgYf74NpcLLzhWwZqpV5BSfWuvmoULvUQRI1e64L7vsMsaPH3/kv/ycnBz27NnDjh07iIuLY9SoUTzwwAMsXbq02tdXKi4upqCggIEDB/L888+zYsUKAPr3789LL710ZLvKIi/tFlypepj1Z4hpAn3udjqSgArPKwoHeHczfvnll/PMM8+wdu1aevfuDUBCQgLvvvsumzZt4oEHHsDlchEZGcm//vUvwBqKdMCAAbRu3ZrZs2cf2W9RURFDhgyhpKQEYwzPPfccYI2Wd9ddd9G1a1fKy8s5//zzGTduHFdccQXDhg1j2rRpvPTSS7UWbSmlbD/Ph40z4ZI/QUyy09EElHYzrvxGz7kKGcbA24MgbxP8ZhlExdf+mkbGVzfjWvSklFK12TwLfv4ezrs/JJNEbTRRKKWUL8bA109CclvocZPT0TiiUSQKEYkXkcUiMrg++wnFYrZgpedahYy1H8PO5VZXHRHRTkfjCEcShYiMF5E9IrK6yvIBIrJeRDaJiHcHKg8CH9TnmDExMeTl5ekPWAAYY8jLy9Oquarx81RY7SZSOkDX65yOxjFO1Xp6G3gZeKdygYi4gVeAS4HtwA8iMh1IA9YA9frVSU9PZ/v27eTm5tZnN6qOYmJiSE9PdzoMpepn1STIXQfD3wZ3+FYSdeSdG2O+FZGMKot7AZuMMVsAROR9YAiQAMQDnYBDIjLDGOM53mNGRkaSmZlZv8CVUuGj/DDM/qvVPc/pQ5yOxlHBlCLTgG1e89uBs40xdwOIyM3A3pqShIiMAcYAtG3b1r+RKqVC37L/Qv7PcP0kcDWK27l+02jevTHmbWPMJz7Wv26MyTLGZDVv3jyQoSmlQk3ZIfjm79DmHOhwqdPROC6YrihygDZe8+n2MqWUCqxFb0DxLhj2Fog4HY3jgumK4gegg4hkikgUMAKY7nBMSqlwUn4YNn0N3z0P7S+CjL5ORxQUHLmiEJEJQD8gVUS2A48bY94SkbuBLwA3MN4Y86MT8SmlwsihfNj4Jaz/FDZ+BYeLIDrJ6tNJAc7VehpZw/IZwIwAh6OUCjf52bD+M1j3qdU1h6cc4pvDGVfBaYOgXT+IjHU6yqARTPcolFLKP4yxxpJY/5l15VA5wmXqqdD7bis5pGWFfe2mmmiiUEqFpvLD8PN3sG6GlSAKtwMCbc6GS5+EjoMg9RSno2wUNFEopUKLMbDodZj1FygtgIhY68b0hQ/DqQPCamS6hqKJQikVOspL4dPfwbJ3of3F0PPX1v2GqDinI2vUNFEopUJD0W744AbYttAa07rfw3rPoYFoolBKNX47lsH7v4JD+60O/M642umIQoomCqVU47bqQ5h2l1W99dYvoFVXpyMKOZoolFKNk8cDs/4PvnsO2vaBa9+BBO3nzR80USilGp+SQvjoNtjwOfS4GS5/BiKinI4qZGmiUEo1LnmbYcJIyNsEA5+1ajZpx31+pYlCKdV4bJ4Nk262EsMNU6DdBU5HFBa07phSKvgZAwvGwbtDIak13DZbk0QA6RWFUiq4eTeiO20wXD0OohOdjiqsaKJQSgUvbUQXFDRRKKWC047l8P712oguCGiiUEoFn5++tWo2xTbVRnRBQBOFUiq4rPsUJt0CzTKtmk1JrZ2OKOxpYZ9SKngsnwATb4CTOsMtn2mSCBKaKJRSwWHBv2DqWMjoCzdOh7hmTkekbFr0pJRyljEw52n45mmr+uuw8RAR7XRUyosmCqWUczwe+PwhWPQadB8FV/wT3PqzFGz0E1FKOaOizOoefOVE6H039P+z9tkUpDRRKKUCr+yQVbNpw2dw0SNw3v2aJIKYJgqlVGCVFFptJH7+3ur9tddtTkekaqGJQikVOAf2wrvXwO4fYeib0GWY0xGpOtBEoZQKjILt8M5VULANRvwPTr3M6YhUHWmiUEr5396NVpIoLbRaW5/cx+mI1HHQRKGU8q8dy61xJABu/gRadXM0HHX8tGW2Usp/fp4H/7kCImPtzv00STRGekWhlPKP7IXWlURyulXclJzudETqBGmiUEo1vF2r4H/DIfEkuOkTSGzpdESqHrToSSnVsPI2w3+vhqgEuHGaJokQoIlCKdVwCnLgnSFgPHDDVGjS1umIVAPQoielVMM4sBf+exWUFMBNH0PzU52OSDUQTRRKqforKbBaXOdnw6iPoHV3pyNSDUgThVKqfsoOWX037f4RRkyAjHOdjkg1ME0USqkTV1EGH9xktZcY+iac2t/piJQfaKJQSp0YTwVMuR02fgGDX9AO/kKY1npSSh0/Y+DT38PqyXDJnyDrFqcjUn6kiUIpdfy+fgKW/Bv63mc9VEgL+qInEbkKGAQkAW8ZY2Y6G5FSYe67561H1q1w8eNOR6MCwJErChEZLyJ7RGR1leUDRGS9iGwSkYcAjDFTjTG3AWOB65yIVyllWzwevvoTdB5mjU6nw5eGBaeKnt4GBngvEBE38ApwOdAJGCkinbw2ecRer5RywqoP4ZPfQYfL4Opx4HI7HZEKEEcShTHmW2BflcW9gE3GmC3GmMPA+8AQsfwN+MwYs7SmfYrIGBFZLCKLc3Nz/Re8UuFow0yrhtPJfeDa/4A70umIVAAF083sNGCb1/x2e9k9wCXAMBEZW9OLjTGvG2OyjDFZzZs392+kSoWTrd/DBzdAy84w8n1rbAkVVoL+ZrYx5kXgRafjUKrRKimEFRPg0H4oL4GyEig/dPS5vNRqXV1eYj+Xeq0vgcPFkHKK1TVHTJLT70Y5IJgSRQ7Qxms+3V6mlDpR5aXw/vWwda4174q0rggiYiAyBiJiISL66LLYptZ8ROzR9TFJ0OMWiE9x9r0oxwRTovgB6CAimVgJYgRwvbMhKdWIeTww9U4rSVw1Drpeqzeg1QlxqnrsBGA+0FFEtovIaGNMOXA38AWwFvjAGPOjE/EpFRK+fgJWf2i1deg+UpOEOmGOXFEYY0bWsHwGMCPA4SgVeha9Ad+/AFmjteW0qrdgqvWklGoIaz+BGQ9Ax4Ew8BltFKfqTROFUqFk2yKYPBrSesDQt7S4STUIn4lCRC7yms6ssu4afwWllDoBezfB/66DpNZw/USIinM6IhUiaruieNZrenKVdY80cCxKqRNVnAvvDbWKmX71IcSnOh2RCiG13cyWGqarm1dKOeHwAfjftVC0G27+BFLaOx2RCjG1JQpTw3R180qpQKsohw9vhZ3L4br3ID3L6YhUCKotUbQTkelYVw+V09jzmTW/TCnld8bAjN/Dhs9h0D/gtIFOR6RCVG2JYojX9LNV1lWdV0oF0tx/wJK3rXYSPX/tdDQqhPlMFMaYb7znRSQS6AzkGGP2+DMwpZQPyyfArP+DLtfCRY85HY0KcbVVjx0nImfY08nACuAdYJmIVNu6WinlZ5tnwfS7IfN8GPIKuLQ5lPKv2r5h53n1t3QLsMEY0wXoAfzBr5EppY61axVMvBFSO8J170JElNMRqTBQW6I47DV9KTAVwBizy18BKaVqkL8N3hsO0Ynwq0kQk+x0RCpM1HYzO19EBmN1+30uMBpARCIAHeZKqeOxbwvkbQbEboUkdj9MAuLymq767ALjgU/us9pM3Po5JKc5+EZUuKktUdyONbrcScBvva4kLgY+9WdgSoUEY+DneTD/ZVhfz46RXZEwajK0PKNhYlOqjmqr9bQBGFDN8i+wxo1QSlWnohzWTod5L8GOpRDbDC54EE65xFpvDGC8nj3VLLOXY6zmrU0zIPUUh96QCmc+E4WI+Byr2hjzm4YNR6lGrrQIlr0L81+Fgmxo1h4GPQfdRmonfarRqq3oaSywGvgA2IH276RU9Qp3wMLXYMm/oaQA2vaGAU9Bx8u1q2/V6NWWKFoBw4HrgHJgIvChMSbfz3Ep1Tjs/hHmvQyrJoGpgNOvgN73QJueTkemVIOp7R5FHjAOGCci6cAIYI2IPGiM+W8gAlQq6BgDW2Zb9x82z4LIOMi6Fc65A5ppF2gq9NRpzGwROQsYidWW4jNgiT+DUiporZkG3/wddq+G+BZw0aNWkohr5nRkSvlNbTeznwQGAWuB94GHjTHlgQhMqaCzYSZ8cCM0P83qOqPLcIiIdjoqpfyutiuKR4CfgG72468ilS2FMMaYrv4NT6kgUbgTpo6Flp3h119DZIzTESkVMLUlCi1wVcpTAVPGQNkhGDZek4QKO7XdzP65uuUi4sK6Z1HteqVCynfPwU/fwpUvQ/OOTkejVMDV1s14kog8LCIvi0h/sdwDbAGuDUyISjkoeyHMfgo6D4UzRzkdjVKOqK3o6b/AfmA+8Gvg/2Hdn7jKGLPcv6Ep5bBD+2HyaGjSBgY/b3fSp1T4qXXMbHv8CUTkTWAn0NYYU+L3yJRykjEw/R4o2gm3ztQuvVVYq208irLKCWNMBbBdk4QKC4vHw9qP4eLHIL2H09Eo5ajarii6iUihPS1ArD1fWT02ya/RKeWE3T/C5w9D+4ut7jiUCnO11XrS3sxUeDl8ECbdYhU1XT1Ox6NWijp24aFU2Pj8Idi7Hm6YAgktnI5GqaCg/y4pVWn1ZFj6H+h7H7S/yOlolAoamiiUAti/FT7+LaT3hAv/6HQ0SgUVTRRKVZTBh6MBgaFvgjvS6YiUCip6j0KpWX+GnMUw/G1rXGql1C/oFYUKb5tnwfcvwFk3wRlXOx2NUkFJE4UKX8V74KPbrfElBjztdDRKBS0telLhyeOBKbdDaSHcOA2i4pyOSKmgpYlChaf59njXg56Dlp2cjkapoBb0iUJE4oFXgcPAHGPMew6HpJzm8UBBNrgiICLWGkgoIrburai3L4avn4TTr7TGu1ZK+eRIohCR8cBgYI8xprPX8gHAPwE38KYx5mngGuBDY8zHIjIR0EQRroyB9TNg9l9h9+pj17ujvBJHDETGVv+8bSEktoIrX9Suw5WqA6euKN4GXgbeqVwgIm7gFeBSYDvwg4hMB9KBVfZmFYENUwUFY2DTVzD7L7BjGTRrBwP+ZiWEshIoP1TLcwmUFsGBXGs40+gkuPIliG3q9DtTqlFwJFEYY74VkYwqi3sBm4wxWwBE5H1gCFbSSAeW46OWloiMAcYAtG3btuGDbmjGWK2Bd62yHvnZvrf39Z9vdCL0/R0ktWrQEIPClm+sBLFtISS3tYYj7TYS3EFfaqpUyAimv7Y0YJvX/HbgbOBF4GURGQR8XNOLjTGvA68DZGVlGT/GefzKS2HP2qNJYdcqq+ik1O7BXdyQ1BqkpjxYy9sp2g1rpsF170KbXg0aumN+nm8liK1zIbG1ddP5zBsgIsrpyJQKO8GUKKpljDkA3OJ0HHV2cJ+VBLyTQu468JRb66MSoGVn6HodnNTFerQ43So/P1G718D7I+HfA2HQP6DHTQ3zXpywfYmVIDZ/DfEtrPYNPW6xipmUUo4IpkSRA7Txmk+3lwXMN4/3I8pTioj1H7xQ+cyRee/pqtukReTTwpN7dIeJraxEcOpldlLoCk0zG36Mg5ad4LbZ1vjOH/8Gdq6wfmAb03/fO1daN6k3fAaxzeDSJ6HnryEq3unIlAp7wZQofgA6iEgmVoIYAVwfyAAymsXgtm+XV6aEo6mhhuX2vQNjYPXBViwovYQ98afSp88FXNGnO7FRARr7Ka4Z/OpD+PoJ+P6fsGcNXPtO8I+psGctzHnKKjqLSYaLHoGzx1r3XZRSQUGMCXxxvohMAPoBqcBu4HFjzFsiMhB4Aat67HhjzF9OZP9ZWVlm8eLFDRRt3Xk8hlnr9vDqnE0szc4nJT6Km/tkcGPvDJLjAtgj6aoPYdrdVq2eEe9CWhCO+Zy7Ab79uxVrVDyccyf0vgtimzgdmVJhSUSWGGOyql3nRKLwN6cSRSVjDD9s3c+/5mxi9vpc4qPcXH92W0b3bcdJyQEqa9+5Et7/FRTvhiv+Cd1HBua4vhhj3Zye9zJs/AIi46DXbdDnXohPcTo6pcKaJgoHrd1ZyGvfbObjlTtxCVxzZjpjLmhH++YJ/j/4gTyYdJP143z2HdD//5wZa6GiDH6cAvNegl0rIS4Fet5m3YNIaB74eJRSx9BEEQS27TvIG3O3MPGHbRyu8HBZp5O4o197urVp4t8DV5TDl4/Cglch4zxrzIX4VP8es9KhfFjyNix8DYp2QOqpVvFS1+vqV8tLKdXgNFEEkb3Fpbz9/Vbemb+VwpJy+rRP4Y5+7el7Siriz+4kVrwP038DCS2t+xatuvnvWPu3woJxsOy/cLjYSlB97oFTLm34Gl9KqQahiSIIFZeWM2FhNm9+t4XdhaW0So4hNvLEakjFRLq55qw0RvRqS0K0j4psOUth4iirrceQl6HLsBOMvgbbfrB6ZV37sdV4sPNQ6wrCn0lJKdUgNFEEsdLyCqYuy+H7TXm1tb+uUc7+gyzNzicpJoIbe2dwU58MmidGV79x8R744CbIngd9fgOX/Alc9ajC66mAdZ/C/JetbjaikyHrZuh1OySnnfh+lVIBpYkiDCzN3s9r32xm5prdRLpdDO+Rzm3ntSMjtZoGa+WH4Yv/Bz+8Ae0uhDOuAuOxfvSNsaZNhdcyz7EPTwVUlFpXD/u3QpOTrSquZ46C6ADcqFdKNShNFGFkc24xb87dwuQlOZR7PFzeuRW3X9COrulNjt146Tvw6e+h4vBxHkWsoiVxQdpZVvHSaYPrd2WilHKUJoowtKewhH/P28q7C36mqKSc3u1SGNuvPed3qHLTvKTQuuEs7qM//i7X0enK5S6v9TqGg1IhRxNFGCsqKWPComze+u4ndheWcnqrJMZe0I5BXVoR4dYaSEopiyYKxeFyD1OX5/D6t1vYtKeYtCax3HZeJtf2bENcVDB1+aWUcoImCnWEx2P4et0eXvtmM4t/3k9ybCTpTX03fvNV0hQT4SY+OoKEmAgSoyNIiI4gPjqCxBhrOqHy2Ws6MTqS+Gi3XtEoFUR8JQr9VzLMuFzCpZ1acmmnlizeuo//Lcqm8FBZjdv7+j/CYFXvzT94mG37D1JcUk5xaTkHD9dtxNqoCBfRbhfRkS6i3C6iI932s+uXzxFua9sIl/3sJjJCcIngEnCJIICIIPa8S345L9jPAm6XkJIQTcvEaFokxdAyKVqvqpTyQf86wlhWRjOyMpo1+H4rPIYDh8uPJI6iknIOlFrTxSXlFNnPB8vKOVzuobTc4/VccWS+pMxD4aFySssrjt2uwoMxBmPAYwyeel4YJ0RH0CIpmpaJMdZzUgwtKhOJJhQV5vRbrxqc2yUkxUSSFBPYDgi9E4fBfjYcs6y8wpBXXMruwlL2FJWwu7CU3YUl5BZZz8uy89ldWEJpueeYYyRGR5CaGE1qQhSpCdGk2M9HH/Z8YjTxUW7/dsuiVIBoolAh40hRE7X/ODeLj6JDy5oHRzLGUHio/BeJZI+dSPYWl7K3uJSNe4pZsKWU/QerL7qLiXSREm8ljeYJUaTERxMX7UbqEF91EqLddGqdTOe0JNKaxGoSUgGjiUKpaogIyXGRJMdF+kwoAGUVHvYdOExuUSl5Bw6zt6j0SDLJKz5MbnEpOfklrNheQEkd799U52BZBRV2GVvTuEg6pyVbj9bJdElLpk0zTR7KPzRRKFVPkW4XLZNiaJnk30GpSsoqWLeriFU5BfyYU8CqnALenLuFsgoreSTFRNA5zUoaZ9jPJzeLw+XS5KHqRxOFUo1ETKSb7m2a0N1rDJPS8go27CpmVU4Bq3cUsDqngH9/v5XDFdb9lcToCDq1TqLjSYlEeVVH9r7w8L4K+UVKqRwWHiE20m1Xb7arQ3tVhfaejorQKs+hSBOFUo1YdISbLunJdElPPrKsrMLDht1FrM4pYHVOIatyCpiyLAePXWzlXUHMu/qz8Vrzi+WGI4mnNlFuFwkxEcRHu4mPstrTtEiMoU2zONo2i+PkFOu5VXKMtqNpRLTBnVKqVuUVHg6UVlB82KrqXFnl+Ui15yPTFb9YVlxSzu7CErbtP3ikiAysmnFpTWJp2yyOtnbyqHy0aRZHcqwDQ/aGOW1wp5Sqlwi3i+Q4F8lxJ/YDXuEx7CosITvvINv2HSR730F+tp8/X72LfQd+2YNxcmwkJ6fE0b55Al3SkunWpglntE4i5gQH91L1o4lCKeV3lVcQaU1i6d0+5Zj1RSVlbNt3iOx9B8iuTCR5B/l+016mLMsBIMIlnNoykW5tkumW3oSu6U04tWWCFmEFgCYKpZTjEmMi6dQ6kk6tk45Zt6ughBXb81m5PZ+V2wv4dOVOJizaBlhtVc5obSWObm2S6ZrehIyUOK0m3MD0HoVSqlExxrA17yArt+ezYlsBK7bn8+OOAkrKrBvuSTERdGvThA4tEomKcOF2gdvlwi1ydPrIMutqp3KZS4QIt9AkLooLOjQPq6rFeo9CKRUyRITM1HgyU+MZ0t0al728wsOG3cVW8rATyOKt2VR4DBXGHGmoeDweHHAad/Rr39DhN0qaKJRSjV6E20Wn1kl0ap3EiF5tj1lv7I4jKzzmaPKoOJpEKpd5PIZyj+GvM9by/FcbuOyMlrRrrmPA610gpVTIExHcLiEqwkVslJuE6AiS4yJpFh9F88RoTkqOIa1JLG2axZGZGs9frupMdISLhyavOtL+JJxpolBKqSpaJMXw6KBOLNq6j/cWZTsdjuM0USilVDWGZ6Vz7ikpPD1jLTn5h5wOx1GaKJRSqhoiwlNXd8Vj4I9TVhGKNUTrShOFUkrVoG1KHPdf1pE563OZtnyH0+E4RhOFUkr5cHOfDLq3acITH//I3uJSp8NxhCYKpZTywe0S/j6sK8Wl5Tzx8Rqnw3GEJgqllKrFqS0TufvCDny8YgdfrtntdDgBp4lCKaXq4I5+7enYMpFHpq6isKT6cdJDlSYKpZSqg6gIF38b1pXcolKemrHO6XACShOFUkrVUfc2TRjdN5MJi7KZt3mv0+EEjCYKpZQ6Dr+7tCMnp8Tx8EerOHS4wulwAkIThVJKHYfYKDdPXdOFn/MO8vxXG5wOJyCCPlGIyFUi8oaITBSR/k7Ho5RSfdqnMrJXW96cu4UV2/KdDsfv/JooRGS8iOwRkdVVlg8QkfUisklEHvK1D2PMVGPMbcBY4Dp/xquUUnX18MDTaJ4YzYOTV3K43ON0OH7l7yuKt4EB3gtExA28AlwOdAJGikgnEekiIp9UebTweukj9uuUUspxSTGR/OWqLqzbVcS4bzY7HY5f+XXgImPMtyKSUWVxL2CTMWYLgIi8DwwxxjwFDK66D7EGv30a+MwYs9Sf8Sql1PG4pFNLrujWmpdmbeTyzifRoWWi0yH5hRP3KNKAbV7z2+1lNbkHuAQYJiJja9pIRMaIyGIRWZybm9swkSqlVC0ev6ITCdER/GHyyhMacrUxCPqb2caYF40xPYwxY40x43xs97oxJssYk9W8efNAhqiUCmOpCdE8fsUZLMvO5z/ztjodjl84kShygDZe8+n2MqWUapSGdG/NhR2b88wX69m276DT4TQ4JxLFD0AHEckUkShgBDDdgTiUUqpBiAh/uboLLoGHPwq9QY78XT12AjAf6Cgi20VktDGmHLgb+AJYC3xgjPnRn3EopZS/tW4Sy0MDT+e7TXt5dub6kLpf4e9aTyNrWD4DmOHPYyulVKD9qldblmfn88rszSzLzueF67rTIinG6bDqLehvZiulVGPhcgnPDu/K34d1ZWn2fga+OJe5Gxt/LUxNFEop1YBEhGuz2jD97r40i4/ixvGLeOaLdZRXNN7W25oolFLKD05tmci0u/pyXVYbXpm9mRGvL2BH/iGnwzohmiiUUspPYqPcPD20K/8c0Z21OwsZ+OJcvmqEQ6lqolBKKT8b0j2NT35zHmlNYvn1O4v5v0/WNKqOBDVRKKVUAGSmxvPRnX24uU8Gb333E8PHzSM7r3E0ztNEoZRSARId4eZPV57BuFFn8dPeAwx6cS4zVu10OqxaaaJQSqkAG9C5FZ/+5jzat0jgzveW8sjUVZSUBe+wqpoolFLKAW2axTFpbG9uP78d7y7I5upX57E5t9jpsKqliUIppRwS6Xbx8MDT+ffNPdlVcIgrXvqOf361kSU/7wuqm90Sap1XAWRlZZnFixc7HYZSStXZzoJDPDBpJd9t2gtATKSLHic35ezMFM7ObEb3tk2IjnD77fgissQYk1XtOk0USikVPPKKS1n00z4W/rSPBVvyWL+7CGMgKsLFmW2acHa7FM7JbMZZJzclJrLhEocmCqWUaqTyDx4+kjgW/pTHmh2FeAxEuV10a5NsXXG0a0aPk5sSF3Xi/bxqolBKqRBRWFLG4q37WLhlHwt+2sfqnAIqPIYIl3Bm2ya8P6Y3bpcc9359JQq/djOulFKqYSXFRHLRaS256LSWABSXlluJ46d97Cs+fEJJojaaKJRSqhFLiI6gX8cW9OvYwm/H0OqxSimlfNJEoZRSyidNFEoppXzSRKGUUsonTRRKKaV80kShlFLKJ00USimlfNJEoZRSyqeQ7MJDRHKBn0/w5anA3gYMp6FoXMdH4zo+GtfxCcW4TjbGNK9uRUgmivoQkcU19XfiJI3r+Ghcx0fjOj7hFpcWPSmllPJJE4VSSimfNFEc63WnA6iBxnV8NK7jo3Edn7CKS+9RKKWU8kmvKJRSSvmkiUIppZRPYZsoRGSAiKwXkU0i8lA166NFZKK9fqGIZAQgpjYiMltE1ojIjyJybzXb9BORAhFZbj8e83dc9nG3isgq+5jHjDMrlhft87VSRM4KQEwdvc7DchEpFJHfVtkmIOdLRMaLyB4RWe21rJmIfCkiG+3npjW89iZ7m40iclMA4npGRNbZn9MUEWlSw2t9fuZ+iOtPIpLj9VkNrOG1Pv92/RDXRK+YtorI8hpe68/zVe1vQ8C+Y8aYsHsAbmAz0A6IAlYAnapscycwzp4eAUwMQFytgLPs6URgQzVx9QM+ceCcbQVSfawfCHwGCHAOsNCBz3QXVqOhgJ8v4HzgLGC117K/Aw/Z0w8Bf6vmdc2ALfZzU3u6qZ/j6g9E2NN/qy6uunzmfojrT8D9dficff7tNnRcVdb/A3jMgfNV7W9DoL5j4XpF0QvYZIzZYow5DLwPDKmyzRDgP/b0h8DFItLwg9F6McbsNMYstaeLgLVAmj+P2YCGAO8YywKgiYi0CuDxLwY2G2NOtEV+vRhjvgX2VVns/R36D3BVNS+9DPjSGLPPGLMf+BIY4M+4jDEzjTHl9uwCIL2hjlefuOqoLn+7fonL/vu/FpjQUMerKx+/DQH5joVrokgDtnnNb+fYH+Qj29h/VAVASkCiA+yirjOBhdWs7i0iK0TkMxE5I0AhGWCmiCwRkTHVrK/LOfWnEdT8B+zE+QJoaYzZaU/vAlpWs43T5+1WrCvB6tT2mfvD3XaR2PgailGcPF/nAbuNMRtrWB+Q81XltyEg37FwTRRBTUQSgMnAb40xhVVWL8UqXukGvARMDVBYfY0xZwGXA3eJyPkBOm6tRCQKuBKYVM1qp87XLxirDCCo6qKLyB+BcuC9GjYJ9Gf+L6A90B3YiVXME0xG4vtqwu/ny9dvgz+/Y+GaKHKANl7z6fayarcRkQggGcjzd2AiEon1RXjPGPNR1fXGmEJjTLE9PQOIFJFUf8dljMmxn/cAU7CKALzV5Zz6y+XAUmPM7qornDpftt2VxW/2855qtnHkvInIzcBg4Ff2D8wx6vCZNyhjzG5jTIUxxgO8UcPxnDpfEcA1wMSatvH3+arhtyEg37FwTRQ/AB1EJNP+b3QEML3KNtOBytoBw4BZNf1BNRS7DPQtYK0x5rkatjmp8l6JiPTC+gz9msBEJF5EEiunsW6Grq6y2XTgRrGcAxR4XRL7W43/6Tlxvrx4f4duAqZVs80XQH8RaWoXtfS3l/mNiAwA/gBcaYw5WMM2dfnMGzou73taV9dwvLr87frDJcA6Y8z26lb6+3z5+G0IzHfMH3foG8MDq5bOBqwaFH+0lz2J9ccDEINVlLEJWAS0C0BMfbEuHVcCy+3HQGAsMNbe5m7gR6zaHguAPgGIq519vBX2sSvPl3dcArxin89VQFaAPsd4rB/+ZK9lAT9fWIlqJ1CGVQY8Guue1tfARuAroJm9bRbwptdrb7W/Z5uAWwIQ1yasMuvK71hl7b7WwAxfn7mf4/qv/d1ZifUD2KpqXPb8MX+7/ozLXv525XfKa9tAnq+afhsC8h3TLjyUUkr5FK5FT0oppepIE4VSSimfNFEopZTySROFUkopnzRRKKWU8kkThWo07N5F73c6Dl9EZLiIrBWR2Q2wr60BbBzofdzWIvJhLdtkePewqkJbhNMBKBVoIhJhjnaK19BGA7cZY77z0/79zhizA6uRqVKAXlGoICcifxSRDSLyHdDRa3l7Efnc7oBtroic5rV8gT0uwJ9FpNhe3s/ebjqwRkTcYo3L8IPdCd3tXvt+wGv5EzXENdI+xmoR+Zu97DGshlFvicgzVbbvJyLfisinYo2lME5EXDXtq8prnxSvcTZE5C8icq+9zzki8qFY40u859UK/WIRWWbvd7yIRNvLt4rIU2KPmSAiZ4nIFyKyWUTG2tscuVqwp+eKyFL70ee4P0TV+DV0C1V96KOhHkAPrJa6cUASVqvS++11XwMd7OmzsbpYAfgEGGlPjwWK7el+wAEg054fAzxiT0cDi4FMrO4NXsdqae6y93d+lbhaA9lAc6yr8lnAVfa6OVTTKt0+fglWC143VlfPw2rZ11YgFcjA6ssKO6bNWC1y+2H1apxuL5+PlahisFpen2q/5h2sTuQq93mHPf08VkvfRPv4u+3lGdjjMdjnPsae7gAsrrqNPkL/oUVPKpidB0wxdn9E9tVAZQ+afYBJcnSIkGj7uTdH++T/H/Cs1/4WGWN+sqf7A11FpLKIJRnrh7C//VhmL0+wl3/rtZ+ewBxjTK4dz3tYA95MreX9LDLGbLFfMwHrR72stn0ZY7aKSJ6InInVjfQyY0ye/d4XGbv/IbFGXssAioCfjDEb7F38B7gLeMGer+wbaRWQYKzxDYpEpFSOHe0uEnhZRLoDFcCptbxHFYI0UajGyAXkG2O6H+frDnhNC3CPMeYXnaOJyGXAU8aY1+oXYrWq9pdzPP3nvAncDJwEjPdaXuo1XUHd/qYrX+Op8npPNa+/D9gNdMM67yV1jliFDL1HoYLZt8BVIhJr98x5BVhdhwM/ichwODJedzf7NQuAofb0CB/7/gK4Q6yumxGRU+1eP78AbrWvWhCRNBFpUeW1i4ALRCRVRNxYvdd+U4f308vu9dQFXAd8dxz7moI1KllPau/5cz2QISKn2PM31DG+6iQDO43V9fcNWMVmKsxoolBBy1hDP07E6pHzM6wupiv9ChgtIpW9dVYOh/lb4HcishI4BasMvzpvAmuApfaN29ewxpGeiVVkNV9EVmENg5tYJa6dWOMTz7ZjW2KMqa5756p+AF7GGsbyJ6xitTrty1jDfs4GPjDGVPg6iDGmBLgFq2huFdaVwrg6xFedV4Gb7PN8Gr+8KlNhQnuPVSFFROKAQ8YYIyIjsG5sN9iYyvWIqx/WjfjBJ/h6F9ZofcNNzUNxKuUXeo9ChZoeWDdfBcjH6oe/URORTli1r6ZoklBO0CsKpZRSPuk9CqWUUj5polBKKeWTJgqllFI+aaJQSinlkyYKpZRSPv1/LUGgBx8WIZIAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "## EDIT THIS CELL\n",
        "K_max = 20\n",
        "rmse_train = np.zeros((K_max+1,))\n",
        "rmse_test = np.zeros((K_max+1,))\n",
        "\n",
        "for k in range(K_max+1):\n",
        "    \n",
        "    # feature matrix\n",
        "    Phi = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # maximum likelihood estimate\n",
        "    theta_ml = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # predict y-values of training set\n",
        "    ypred_train = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # RMSE on training set\n",
        "    rmse_train[k] = 0 ## <--- EDIT THIS LINE\n",
        "            \n",
        "    # feature matrix for test inputs\n",
        "    Phi_test = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # prediction (test set)\n",
        "    ypred_test = 0 ## <--- EDIT THIS LINE\n",
        "    \n",
        "    # RMSE on test set\n",
        "    rmse_test[k] = -1 ## <--- EDIT THIS LINE\n",
        " \n",
        "    \n",
        "\n",
        "plt.figure()\n",
        "plt.semilogy(rmse_train) # this plots the RMSE on a logarithmic scale\n",
        "plt.semilogy(rmse_test) # this plots the RMSE on a logarithmic scale\n",
        "plt.xlabel(\"degree of polynomial\")\n",
        "plt.ylabel(\"RMSE\")\n",
        "plt.legend([\"training set\", \"test set\"]);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXSP-E6ijmsN"
      },
      "source": [
        "#### Questions:\n",
        "1. What do you observe now?\n",
        "2. Why does the RMSE for the test set not always go down?\n",
        "3. Which polynomial degree would you choose now?\n",
        "4. Plot the fit for the \"best\" polynomial degree."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "HgA4n5yFjmsN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "46ec5576-5a84-4bc7-eace-22b354accf02"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzG0lEQVR4nO3deXiU1fXA8e8hYZWdAAIBEpTIIqsB2QUjioKouKFWpWr52bpQ6lLUulRrSysVt7qjuFHrBiLgEgQERIUgAURkNUAAWQIJIAQIub8/7iQGzDJJZua+M3M+z5NnJjPvvO8JJHPmbueKMQallFKqLFVcB6CUUio8aMJQSinlF00YSiml/KIJQymllF80YSillPJLrOsAgiUuLs4kJCS4DkMppcLK0qVLdxtjGhf3XMQmjISEBNLS0lyHoZRSYUVENpX0nHZJKaWU8osmDKWUUn7RhKGUUsovETuGoULv6NGjZGZmkpub6zoUFSFq1KhBfHw8VatWdR2KQhOGCqDMzEzq1KlDQkICIuI6HBXmjDFkZWWRmZlJYmKi63AU2iWlAig3N5dGjRppslABISI0atRIW6weoglDBZQmCxVI+vvkLZ5IGCLyiojsFJHvSnh+oIjkiEi67+uBUMeolFJhIf2/sHRyUE7tiYQBTAaGlHHMAmNMV9/XwyGISYW5hx56iAkTJpT4/LRp0/j+++9DGJFSIfDt67D8f0E5tScShjFmPrDHdRzKjYmpa51cVxOGikj7MqFei6Cc2hMJw0+9RWS5iHwsIh2LO0BERotImoik7dq1K9TxqQp68vN1ATvXo48+SlJSEv369WPNmjUAvPTSS/To0YMuXbpw6aWXcvDgQRYtWsT06dO566676Nq1Kxs2bCj2OKXCSn4+7NsOdaM7YXwLtDbGdAGeBqYVd5Ax5kVjTLIxJrlx42JrZ6kItnTpUt5++23S09OZNWsWS5YsAWDEiBEsWbKE5cuX0759eyZNmkSfPn0YPnw4jz32GOnp6ZxyyinFHqdUWPl5J+QfhXrxQTl9WKzDMMbsK3J/log8KyJxxpjdLuNSFTcxde1xLYuEcTMBGJPSlrGDkyp0zgULFnDJJZdQq1YtAIYPHw7Ad999x1/+8heys7M5cOAA5513XrGv9/c4pTwrZ6u9jeaEISInAzuMMUZEemJbRlmOw1KVMHZwUmFiSBg3k4zxQ4N2rVGjRjFt2jS6dOnC5MmTmTdvXqWOU8qzcrbY20jukhKR/wJfAaeJSKaI3CgiN4vIzb5DLgO+E5HlwFPASGOMcRWv8qYBAwYwbdo0Dh06xP79+/noo48A2L9/P82aNePo0aO89dZbhcfXqVOH/fv3F35f0nFKhY19UdDCMMZcVcbzzwDPhCgcFWJjUtoG5Dzdu3fnyiuvpEuXLjRp0oQePXoA8Mgjj3DmmWfSuHFjzjzzzMIkMXLkSH73u9/x1FNP8d5775V4nFJhI2crxNaEmg2CcnqJ1A/qycnJRjdQCq3Vq1fTvn1712GoCKO/V+XwznWwYxXctrTCpxCRpcaY5OKe80SXlFJKqQDIyQxadxRowlBKqciRsxXqasJQSilVmrwjcGBH0FZ5gyYMpZSKDPu3AyZoU2pBE4ZSSkWGwim1mjCUUkqVJifT3tZrGbRLaMJQqgzTp09n/Pjxzq4/cOBACqaIX3DBBWRnZ5ORkcHpp5/u9zmKlnp/4IEHmD17NgAJCQns3h34Cjslnffdd9+lffv2DBo0iLS0NG6//XYA5s2bx6JFiwIeR1QpSBhB7JLyxMI9pbxs+PDhhXWpXJs1axYA2dnZFT7Hww+7205m0qRJvPTSS/Tr1w+A5GQ73X/evHnUrl2bPn36OIst7O3bCjXqQfXaQbuEtjBUxMjIyKBdu3aMGjWKpKQkrrnmGmbPnk3fvn1p27YtixcvBmDx4sX07t2bbt260adPn8Iy6BMnTuSGG24AYOXKlZx++ukcPHiQyZMnc+uttwK23tTvf/97evXqRZs2bZg3bx433HAD7du3Z9SoUYWx1K79yx/te++9V/icv68vSXGf3Ddu3Ei3bt1YsmQJGzZsYMiQIZxxxhn079+fH3744VfnGDVqFO+9917h908//TTdu3enU6dOhcfv2bOHiy++mM6dO9OrVy9WrFhR6uNZWVmce+65dOzYkZtuuoniFgQ//PDDLFy4kBtvvJG77rqLefPmMWzYMDIyMnj++eeZOHEiXbt2ZcGCBWX+O6hiBHlKLWgLQwXLx+Pgp5WBPefJneD80ruG1q9fz7vvvssrr7xCjx49mDJlCgsXLmT69On8/e9/Z9q0abRr144FCxYQGxvL7Nmzuffee3n//fcZM2YMAwcOZOrUqTz66KO88MILhZVvi9q7dy9fffUV06dPZ/jw4Xz55Ze8/PLL9OjRg/T0dLp27VpqjJV9fVFr1qxh5MiRTJ48mS5dupCSksLzzz9P27Zt+eabb/jDH/7AnDlzSj1HXFwc3377Lc8++ywTJkzg5Zdf5sEHH6Rbt25MmzaNOXPmcN1115Genl7i43/961/p168fDzzwADNnziy2NPwDDzzAnDlzmDBhAsnJyYXFHRMSErj55pupXbs2d955p98/uzpBEDdOKqAJQ0WUxMREOnXqBEDHjh1JSUlBROjUqRMZGRkA5OTkcP3117Nu3TpEhKNHjwJQpUoVJk+eTOfOnfm///s/+vbtW+w1LrzwwsJzNm3a9LjrZWRklPmGX9nXF9i1axcXXXQRH3zwAR06dODAgQMsWrSIyy+/vPCYw4cPl3meESNGAHDGGWfwwQcfALBw4ULef/99AM4++2yysrLYt29fiY/Pnz+/8LVDhw6lQYPg1DJSpcjJhPgeQb2EJgwVHGW0BIKlevXqhferVKlS+H2VKlXIy8sD4P7772fQoEFMnTqVjIwMBg4cWPiadevWUbt2bbZt21bmNYqe/8RriEjh47m5ueV+vT/q1atHq1atWLhwIR06dCA/P5/69euTnp7u9zmKxhMTE1Ou6ysPOXIQDu0N6oA36BiGikI5OTm0aGH/sCZPnnzc47fffjvz588nKyvruH7+8mratCmrV68mPz+fqVOnVjbkYlWrVo2pU6fy+uuvM2XKFOrWrUtiYiLvvvsuAMYYli9fXqFz9+/fv7DE+7x584iLi6Nu3bolPj5gwACmTJkCwMcff8zevXvLdb0TS82rcgpyWfMCmjBU1Ln77ru555576Nat23GfqMeOHcstt9xCUlISkyZNYty4cezcubNC1xg/fjzDhg2jT58+NGvWLFCh/8pJJ53EjBkzmDhxItOnT+ett95i0qRJdOnShY4dO/Lhhx9W6LwPPfQQS5cupXPnzowbN47XXnut1McffPBB5s+fT8eOHfnggw9o1apVua534YUXMnXqVB30rqgQTKkFLW+uAkjLUKtg0N8rPyx7Ez68BW5fBg3bVOpUWt5cKaUiWYhaGJowTnR4P6S9Aju+dx2JUkr5JycTTmoCsdXLPrYSNGGc6NhRmDEWNs51HUlYitQuTuWG/j75ad/WoK/BAE0Yv1azAVSrA3s3uY4k7NSoUYOsrCz9I1cBYYwhKyuLGjVquA7F+4K8014BXYdxIhFo0BqyN7uOJOzEx8eTmZnJrl27XIeiIkSNGjWIjw/+G2FYM8a+X7U9N+iX0oRRnPqtYG+G6yjCTtWqVUlMTHQdhlLR5cBOyMuF+q2DfintkipOfV8LQ7tWlFJeV9AbUr98a18qQhNGceq3giMH4OAe15EopVTpsn3jrQ20heFGwT98tg58K6U8ruB9Kog77RXQhFGcgqadJgyllNdlb4ZajYK6cVIBTRjFKUwYOlNKKeVxezeFZMAbPJIwROQVEdkpIt+V8LyIyFMisl5EVohI96AGVKMe1KivazGUUt6XvTkkA97gkYQBTAaGlPL8+UBb39do4LmgR6RrMZRSXpefDzlbQjLgDR5JGMaY+UBpU5IuAl431tdAfREJXs1osBlbxzCUUl524Cc4diTqWhhlaQFsKfJ9pu+x44jIaBFJE5G0Sq821rUYSimvK1yDEUUtjEAxxrxojEk2xiQ3bty4cier39qunjxQsQ10lFIq6DRhFGsrUHSScbzvseApXIuh4xhKKY8q6DavH/w1GBA+CWM6cJ1vtlQvIMcYsz2oV9S1GKqSJqaudR2CinR7N9l9MKrWDMnlPJEwROS/wFfAaSKSKSI3isjNInKz75BZwEZgPfAS8IegB6UJQ1XSk5+vcx2CinTZm0M2Qwo8Uq3WGHNVGc8b4JYQhWNVOwlqxWmXlAqpialrGTs4yXUYKlxkb4YWwV2WVpQnWhie1aC1Lt5T5TIxdS0J42aSMG4mQOF9f7untFWi/JZ/zG6cFKIBb/BIC8Oz6reC7StcR6HCyNjBSYUthIRxM8kYP9RxRCpi7d8O+UdDtgYDtIVRuvqt7SrK/HzXkagIVtlWSUWvqcJcCPfBKKAtjNLUb2VXUR74Ceo2dx2NCjNjUtr6dZyLVsmTn6/TsZJwV9Bd3iAhZJfUhFGagtkHezdpwlDlVviGnHcYMtNg05ewLR0O7LALQnNzoHodqFnflqdu1oXzqsTAgR5Qu4nL0FU4yN4MCNQL3Z7nmjBKU7/I4r3Wvd3GosJLfj5kzIe0V2HtJ7ZqAAKNT4M6zaDRqbYq8uH9NnHs3w7fPM8L1Y7AhInQuh90uwY6XGRn7AXAxNS1xw2qF3SBjUlpq62NcJS92f4uxVYP2SU1YZSmYAervRlOw1Bh5FgeLHsDFj0NezZAzQbQ/TpoM8h+6KjZoOTXHs2Fn1bAxi9g+RSY9nuYdRf0HA19x9iWSCXogHyEyd4U0vEL0IRRuqo1oG4L2Puj60hUOFg3Gz77C+xaDS3OgEtetC2EqjX8e33VGtCyp/0acCds/hoWvwgLH4e0V6D/HTZ5+Hs+Fdn2boJWvUJ6SU0YZWnYBvZsdB2F8rKfd8NHY+CHGdAgEa54A9pfCCIVP6eIbZG07g39xsLnf4XU+yF9Cox4AZp1qVTI/g7IK4/KOwz7Mu37UwjptNqyNEyEPdrCUCVYNxue6wPrPoNzHoJbvoEOwyuXLE7UrDP85n24+l04tBdeOhu++Jft/qogHbMIc9mbweRrwvCchm3g5512cFKpAvnH4LP74a1LoWZD+N0c2xII5gBk0rnwh6+gw8Uw91F48xI4WNq+YypiFfR6aMLwmIL/EG1lqAJHfob//QYWPQXJN8LouXByp9Bcu1ZDuGwSXPSsHeN4OQV2azmRqJO1wd5qwvCYwoSh4xgK2LcdXj3fTpU9/zEY9njISksfp9s1cP0MyN0HL6XYmVUqeuzZCNXr2Q8QIaQJoywNEu2tJgy1dxO8cq79dHfV23DmaLfxtDrTtm7qtYC3LrfjKSo67Nlox1cDOVbmB00YZaleG2o31YQR7fZugteG2UV2138ESee5jsiq34rnEp60CwLfvgrWfOw6IhUKezaGvDsKNGH4p4HOlIo05Sq+VzRZXPdhSPcf8Mc/5++C66dD09Pt2MrqGa5DUsF07KidJaUJw6N0LUbE8XvfiQM74fXhvySL5t2CG1hF1WwA102z8b33W8hY6DoiFSzZm8Ec04ThWQ3bwP5tcOSg60hUKB35GaZcYZPGb6Z6KlkUWxL9oYU82/wftkX836thxyrHUaqgKOjtcJAwdKW3Pxr6Br73ZkDTDk5DURVXruJ7x/Lg3d/C9uUw8r8Qf0YoQy1TqXWhst+HSYPhzUvhxlSo39JRlCooHK3BAE0Y/ik6tVYTRtjyu/ieMTDrTlj3KQx9HE4bEsIoA6B+S7sy/JXzbQvpxlQ7eUNFhj0boVptJyXwtUvKHw11am1UWfqq/er7R+hxo+toylRsXaimHeGKybDrB/jwDzYJqsjgaEotaMLwT80GtvyDJoyIUWLxvS1LYNbdcOpgSHkgtEFVUIl1oU45G875K3z/oa14qyKDoym1oAnDfw3baJnzCFLsm+yBnfDOtXYh3IgXoUrMcU+H5T7YfW6D0y+Dzx+Bdamuo1GVlX/MjqVqwvA4nVob2Y7lwbuj4FA2XPlmsSUX/J6K6yUiMPxpu0bj/ZsgJ9N1RKoycrZA/lFNGJ7XsI39Y8s77DoSFQzzH7N7bl/4ZOgKCYZKtVpwxWuQnwcfjLafUlV4cjhDCjRh+K9hoq0/n73ZdSQq0DZ/DfP/BZ1HQpcrj3uq2PUO42aGX/dUo1Ng6L9tUpz/mOtoVEU5Thg6rdZfRafWxuluZRHjUDa8/zu7N/IFv34jjah9sLuMhA1z4Yt/QuIAaN3HdUSqvPb8CLE1ofbJTi7viRaGiAwRkTUisl5ExhXz/CgR2SUi6b6vm0IepJY5jzzGwMw/wb6tcOkkqFHXdUTBN3QCNEiwSTI3x3U0qrwKptRWcfPW7TxhiEgM8B/gfKADcJWIFLc67n/GmK6+r5dDGiRArUZQve4vG5eo8LdqKnz3Pgy6B+KTyzw8IvbBrl4HRrxkS918ep/raFR5OZxSCx5IGEBPYL0xZqMx5gjwNnCR45h+TcR2RWWF4UwZ9WsHdtnV3M27Q9+xfr0kYvbBjk+GvmNg2Rs61Tac5B+zXVIFC4kd8ELCaAFsKfJ9pu+xE10qIitE5D0RKbY4joiMFpE0EUnbtWtX4CONS9LtMCPFrDvtPu0XPwsxUTiUN/AeaNwOpt8Gh/a6jkb5I3sTHDts34cc8ULC8MdHQIIxpjOQCrxW3EHGmBeNMcnGmOTGjRsHPoq4tra/+/D+wJ9bhc6qafD9NBg4Dpq0dx2NG7HVbbI8sBM+udd1NMofBR9W405zFoIXEsZWoGiLId73WCFjTJYxpmABxMuAm9KhBZk9a72Ty6sAOLgHZt4BzbpCnzGuo3GrxRnQ74+wfApsmOM6GlWWXWvsrcNZml5IGEuAtiKSKCLVgJHA9KIHiEizIt8OB1aHML5fFCQM7ZYKX6n3Q2529HZFnWjA3dDwFJjxJzh6yHU0qjS718JJjYutQhAqzhOGMSYPuBX4FJsI3jHGrBKRh0VkuO+w20VklYgsB24HRjkJtkEiSIz9j1PhZ9MiWPYm9L7VVnNVULUGDJto66TNn+A6GlWa3eucjl+ARxbuGWNmAbNOeOyBIvfvAe4JdVy/ElvNzmHXhBF+8o7AjLFQrxWcdbfraLylzVnQ5Sr48gnodFn0jut4mTGwew10uNhpGM5bGGFHZ0qFp6+esXtDDJ0A1U5yHY33nPuoXWf00R8hP991NOpEB7PsbDbHLQxNGOUV19YOemsBt/CxdxN88S9ofyEknec6Gm86qRGc+whs+RpW/M91NOpEBQPejTVhhJe4JDh2xM6JVuHhs/vswssh/3Qdibd1uRrie0DqA5C7z3U0qqiCbnBtYYQZnSkVXjZ+Aas/gv5/shsjqZJVqQLn/wt+3mULFCrv2L0WqtaCuvFOw9CEUV4Fc6B14Nv7juXBJ+OgfmvofZvraMJDi+7Q/Vr45vlfukGUe7vXQqNTnRUdLKAJo7xqNYRacZowwkHaK7DzezjvUTt9VPkn5UE7MeDjP9vZOcq93WuhsbsV3gU0YVSEzpTyvp+zYO7fIPEsaDfMdTTh5aQ4GHQfbJwLaz9xHY06chCytzgfvwBNGBUT11ZbGF73xT/h8AE4/592wFuVT/IN0KgtfPYXOHbUdTTRLWs9YDyxcZsmjIqIS7Lzon/Och2JKs7u9ZA2Cc64XhehVVRMVTvNNms9pL3qOproVjhDSrukwlNhEULtlvKk2Q9CbA1bwltVXNIQu5XrvH/YrWyVG7vXglSx+7I7pgmjInSmlHdtWgQ/zLBVWGs3cR1NeBOxK8AP7YUFWmfKmV1rbEmi2OquI9GEUSH1W0FMdU0YXpOfb7cdrdMcet3iOprI0KwzdL0Gvn7e7vamQs8DRQcLaMKoiCoxtpWx8wfXkaiivp8K276FlPuhWi3X0USOs++zv/Nz/+46kuiTf8yOI3lgwBs0YVRck/aw0822HOp4E1PX2pk8c/4GTU+Hzle6Dimy1G0OvX4PK9+B7StcRxNd9my027I26eA6EkATRsU16QD7MnUw0AOe/HwdfPu6/eNKecB+GlaB1feP5MbWhc//6jqS6LLjO3vrkf1bNGFUVMF/4C7tlnKtBoftuotWvaHtua7DiUw16zPh0IWwfjb8ON91NNFjxyq7aZsHptSCJoyKK5jfv2OV2zii1MTUtSSMm0nCuJn8NuZTOLCDy9YNZuJsneocLG8cG2yL36U+qCVDQmXH97aGlEdK23hix72wVK+l3XBm5/euI4lKYwcnMXZwEhzaS874myBpCO9dfYfrsCLOxNS1tssPgGrcmTWUCfte4KO3X+DCq252GltU2PEdtDjDdRSFymxhiEiqiHQJRTBhRUQHvr3gyyepwyE4+37XkUSksYOTyBg/lIzxQwGY8Mg/IO40Lsx6RTcRC7bcfXbfHY+MX4B/XVJ/Bp4QkVdFpFmwAworTdrbLiltnrtxYCd88wJrG58LJ5/uOproUCUGBt1r95de+a7raCJbwYfRcEoYxphvjTGDgBnAJyLyoIjUDH5oYaBJR8jNhv3bXUcSnRY+AXm5tBv5qOtIosKYFN9agPbD4eTOtmSIFiYMnp2+8dFwShgAIiLAGuA54DZgnYhcG8zAwkJT39xoHccIvX3bbIHBziM9s6gp0o0d7FttXKWK7QLcmwHL3nAaU0TbscqOk9Zr6TqSQv6MYXwJbAUmAi2AUcBAoKeIvBjM4DyvYDHNDk0YIbfg35CfB2fd7TqS6NR2MMT3hC8eg6O5rqOJTDtW2fcYD5Xn96eFMRpoYYwZbIy53xgzwxiz3hhzG9A/yPF5W62GUPtkbWGEWvZmWPoadPsNNEx0HU10ErElWPZvszsbqsAyxn4Q9VB3FPg3hrHKmBJHdYcGOJ7w07SDJoxQmz/BvmENuMt1JNEtcQAk9IeFE+2ucCpwcjLhcE74JYzSGGM2BiqQsNWkgy0/rFMMQ2PvJkh/C7pfD/XiXUejBt0LP++040kqcHZ4b8AbdKV35TXpAHm5to6RCr4F/7abyfQb6zoSBdC6D7QZaGesHfnZdTSRo6CGlMd2jPREwhCRISKyRkTWi8i4Yp6vLiL/8z3/jYgkOAizeDpTKnSOa120cB2NKjDwXji4G5a87DqSyLFjld13p0Y915Ecx3nCEJEY4D/A+UAH4CoRObGW743AXmPMqdjZWv8MbZSliDsNEJ0pFQrauvCmVmfCKSnw5ZNw+IDraCLDjlW2VL/HOE8YQE9gvTFmozHmCPA2cNEJx1wEvOa7/x6Q4lsb4l61WtCwzS9NSBUcha2L67R14UWD7oWDWbA4umfaB8TRXLtpkkf2wCjKCwmjBbClyPeZvseKPcYYkwfkAI1OPJGIjBaRNBFJ27VrV5DCLcbJneCnlaG7XjRa+Li2LrwsPhlOHQyLntZWRmXtXAXmmCfL3XghYQSMMeZFY0yyMSa5cePGobtwsy62SNjBPaG7ZjTJ3gLL3rLrLnRmlHcNHAeH9uhYRmVtW2Zvm3d3G0cxvJAwtgJF177H+x4r9hgRiQXqAVkhic4fzbva2590+8qg+PJJwGjrwuvik+1YxqKndcZUZWxbBrUa2UFvj/FCwlgCtBWRRBGpBowEpp9wzHTget/9y4A5pSwmDL1mXe3ttnSXUUSm/T/Z7Ve7XOXJPyB1goHjfDOmdF1GhW1Lh+bdPFUSpIDzhOEbk7gV+BRYDbxjjFklIg+LyHDfYZOARiKyHvgT8Kupt07Vagj1WsH25a4jiTxfPmVrRvX/k+tIlD9a9oQ2g2yrUFsZ5XfkoC1r3ryb60iK5Ykd94wxs4BZJzz2QJH7ucDloY6rXJp11oQRaAd22TpFna+wM9FUeBg4Dl45z/7f9bnNdTTh5aeVdsDbownDeQsjYjTvCns2QG6O60gix1fP2FX0/XXr1bDSqhcknmVbh0cPuY4mvBQOeGvCiGwF4xg6vTYwDvpm25w+Qve7CEdn/dnWmFr6WtnHql9sWwa1m0Idb25uqgkjUJr5tj3Xge/A+OYFOHIA+t/pOhJVEQl9oXU/+PIJ3S+jPLYts9NpPTjgDZowAqd2E6jTXMcxAiF3H3zzHLQb9kutLhV+zrrLbl+c/qbrSMLD4f2we61nu6NAE0ZgNesC29NdRxH+0ibZsSAduwhviWdByzN9e68fcR2N921fARhNGFGjeVfYvU5LI1TGkYOw6Bm7AKyF91a6qnIQsVvo5myB5VNcR+N9hQPeXZ2GURpNGIHUrAtgtBBhZXz7ml34pbvpRYZTUqDFGbDgcTh21HU03rZtGdSNt93bHqUJI5B0xXfl5B22UzFb94XWvV1HowKhYCvd7E2w8l3X0Xjbtm+hhXe7o0ATRmDVORlOaqID3xWVPgX2b4MBOjMqoiQNsRWdF/xbtzIuyaFsu2unh8cvQBNGYInY/sdt37qOJPwcy7NTMJt3s6UlVOQQgQF32z0eVk11HY03bU2ztx6sUFuUJoxAi+8Ju36wnxiU/757H/Zm2HUXHp2Driqh3TC7IdD8xyA/33U03rP5a5AYiO/hOpJSacIItJa+//CCTwyqbPn5doOkxu3htAtcR6OCoUoV29W46wdYfWIxasXmr209uuq1XUdSKk0YgdbiDLsz3JbFriMJH2tm2jeS/nfYNxYVmTpcDHFJ2so4Ud4RyEyDVt6f6KF/nYFWvQ406agJw1/GwPwJ0CAROl7iOhoVTFVibJfjju9g7ceuo/GOn1ZA3iFbtNHjNGEEQ8ue9hODzggp24bP7er4fmMhxhPV9lUwnX6pLVX/xT/thwUFm7+yty01YUSnlj3hyH7bzaJKN//fULeF3VFPRb6YWNvK2L4c1n3mOhpv2Py1TaJ1mrqOpEyaMIKhZU97u+Ubt3F4XcaXsHkR9LkdYqsBMDF1reOgVNB1vgLqt9ZWBtiff/NXYTF+AZowgqNBItSKgy1LXEfibQsm2H+n7tcVPvTk5+scBqRCIqaq3XJ361LbJRnNstbDwaywGL8ATRjBIeIbx9CB7xJtXQob5kDvW6BaLdfRqFDrcjXUawnzoryVUTB+oS2MKNeyp/308HOW60i8acHjUKMe9LiJialrSRg3k4RxMwEK72v3VASLrWYnOmQuho1zXUfjzuavoVYjaHSq60j8otNSgiXeN46RuQROG+I2Fq/Z8T38MMNu41mjLmMH12Xs4CTAJouM8UMdB6hCottv7AeHeeNtOZhoXOFfMH4RJj+7tjCCpXk3qBKrA9/FWfBvqFYbzrzZdSTKpdjq0H+s/RvZOM91NKG3f4ctOBgm4xegCSN4qtWyFTp1Ad/xdq+HVR9AjxuhVsNfPT0mpa2DoJQz3a6106rnjY++sYyMBfa2VR+3cZSDJoxgatXb1pTKO+w6Eu9YOBFiqkPvW4t9uqBrSkWJ2Op2LGPL1/DjF66jCa0Nc6FmA0/vsHciTRjBlDgA8nLtOIaCvZtgxdtwxihP7yqmQqz7dVCnOcz9R/S0MoyxswTbDLQlU8KEJoxgat3HFiL8cb7rSLzhyyfsv0ef21xHorwktjoMuMO2MjbMcR1NaOz6wW4WdsrZriMpF00YwVSjnh381oQBOVth2ZvQ9Wqo18J1NMprul1r12XMfTQ6WhkFiTHMNgtzmjBEpKGIpIrIOt9tgxKOOyYi6b6v8CqmnzjAdkkd+dl1JG59+SSYfOj3J9eRKC+KrW73/t66tNw1psJyvc6GObbUe/2WriMpF9ctjHHA58aYtsDnvu+Lc8gY09X3NTx04QVA4gDIz4NNX7mOxJ1922HpZFtgsEFr19Eor+p6NZvym5S7lRF25WSO5to6amHWHQXuE8ZFwGu++68BF7sLJUha9oIqVaNvBkhRi56ySbO/ti5UKWKq8lTeCFvJ9ocZrqMJni1f2/0vwjBhuF7p3dQYs913/yegpPq+NUQkDcgDxhtjpoUiuICoVsuWCYnWcYz9OyDtFeh8pS3hrNQJJqauLWwlxNCXP+R/yNH/3sMn/U7lj+e2L/M1QGFZmTEpbb0/NXvDHPshsnVf15GUW9AThojMBk4u5qn7in5jjDEiUlI7tLUxZquItAHmiMhKY8yGYq41GhgN0KpVq0pGHkCJA+zCpEN77bzraLLoKTh2xO7nrFQxxg5OOq40zCmX/w3ev5F2J68Eik8YJ74mrMrJbJhjV3d7fP/u4gS9S8oYc44x5vRivj4EdohIMwDf7c4SzrHVd7sRmAd0K+G4F40xycaY5MaNGwfl56mQxAGAsf2W0eTALtu66HQ5NDrFdTQqXHQcYaskzH3U7ncdSQ7shJ9WwinhNTuqgOsxjOnA9b771wMfnniAiDQQkeq++3FAX+D7kEUYCC2SIbZm9HVLffmEXbg44C7XkShHyjuDaUxKW6hSBc5+APZmwLLX/XtNuFiXam/DcPwC3CeM8cBgEVkHnOP7HhFJFpGXfce0B9JEZDkwFzuGEV4JI7YatO4dXQPf+3+CJS/bsYu4MPqDVgFV3hlMheMPbQfb0jpf/KvMKemeH7MoavV0u96kWVfXkVSI04RhjMkyxqQYY9r6uq72+B5PM8bc5Lu/yBjTyRjTxXc7yWXMFdZmkF3dmb3ZdSShsfAJOHZUWxeqYkQg5UE4sAO+ecF1NIGRu8+OX7S/MGzKmZ/IdQsjepx2vr1d+6nbOEJh33Y7dtH1Kh27iEIB2xCrdW9oe67t2jy4J/CBhtq6z+wEkA4XuY6kwjRhhEpcW2h4Cqz52HUkwbfwcTDHtHURpcYOTiJj/NDCmUsF9yvUdXTOQ3B4P8yfENggXfh+GtQ++ZfN1cKQJoxQOu18WwP/8H7XkQRPTqZd1d31GmiQ4DoaFe6adrS/S4tfhD0/uo6m4o78DOtmQ/thdlA/TIVv5OEoaYhtkm6I4D2M5423t9q6UARoBtOg+yCmKnz+18qfy5X1s+3q7vbhVdnoRJowQqlVL6hRP3K7pXavg/S3IPnGsCuqpoIjIDOY6jazJfFXTYUtYbq3zPfToWbDsFzdXZQmjFCKqWqnC677FPKPuY4m8Ob8za436X+H60hUpOlzO5zUBD77S/iVPz+aC2s/sd1RMa6rMVWOJoxQSxoCB7MgM811JIG1Ld0O6vW+BWp7aJW9igzVa8PZ99nCfd+97zqa8tkwB44cgPbhOzuqgCaMUDv1HKgSC2sjrFtqziO2Tlaf4vfqVqrSul0LJ3eGz+4Pr/1l0t+CWnG+EkHhTRNGqNWsb1ewrvnEdSSB8+MCO6jXb6zdZVCpYKgSAxc8Zrc2XfC462j8s/8nO2bZ9Wpb8SHMacJwod1Q2LUadq1xHUnl5efbfuW68dBztOtoVKRr1Qs6XWGrIO/Z6DqasqW/Zdckdb++7GPDgCYMFzqOAKkCK95xHUm5FLtS97v3YXs6pNwPVWuGPCYVhQY/bPeT+ORe15GULj+f7C8nQet+EHeq62gCQhOGC3WaQpuBsPLdsJrx8atCckdz7dz4kzvbT31KhULdZnDWXXYc8IdZrqMpWcYC6uduhe7XuY4kYDRhuNLpCsjeBFsWu46k4ha/ADlb4Ny/hfXqVRWGet0CTTrAzDtsUT8v+vY1ckwt6BDei/WK0r9yV9oPs2sWVvzPdSSlKqmQ3HOzFsP8f9vicG3Ochylijqx1WD407B/u52hV07lLoRYznN3G/dfDq/8kA+O9Sfh/jkVK77oQeG9iiScVa8D7S6wq1fP/6dd1OdBJW6FOeNPdm754IcdRqeiWnyynWix+EW7q2NL/4v6Pfn5uqDtozF2cBJja30MqXm8fWxQeG0fWwZtYbjU6Qo4tAfWf+46kvLZvgKWvgo9boImxe+5rFRIpNwPdVvA9Nu9s53r0UOw6BlIPIs1ppXraAJKE4ZLp6bY+jIe75YqMCalrR2k//huu0hv0D2lHh8JTXDlcdXrwLDH7TT1eX8v9dCA7dNRlqWvwc874aw/h9f2sX7QLimXYqrC6SNg2ZuQm+P5RW9jByfBindh81dw4ZM2aZQimM1+pQolnWfXOSx8wlZSSOhX7GEldq8G0tFcu+FT636Q0JexCYG/hEvawnCt27WQl2uTRgWF7JP84QOQer/dj7jbtaG5plL+OO/v0DARpt4Mh7LdxbHsDTsQf1ZklvfXhOFa8662VMjiFytcwfZX6yOCZd4/7B/DBY/ZMg3FCFmzX6miqteGES/Bvm0wq+w366B0FeUdsa2clmdCYmTOHNQuKS8482Z493q733e7C1xHU7yt38LXz8IZvy11NkpImv1KFSc+Gc76sx3LSBwA3UtuBQelq3TZG7AvE4Y/CSKBP78HaMLwgnbDbC2mb57zO2FMTF17XMui4BP9mJS2gf9jOHbUzkI5qQkMDuNdz1Tk638HbF4EM/9kZ/DFJ4fmugd2wucPQ6s+cEpKaK7pgCYML4iJhZ43weyHYMf30LRDmS8J6Sf5r56BHSvhijfKNTAfaTNEVBiIiYXLXoUXB8L/fgOjv7CleILtk3Fw9KCdDBKhrQvQMQzv6H69Xfn9zfOuIzle1ga7T3e7YeUucaAzpJQTtRrCyCl25uE71wZ/fcbaz2wRzv53QuPI/p3XhOEVtRpClyvtmoyfd5frpUH7JH8sz846ialuB7qVChcnnw4X/Qe2fAPv32h/l4Ph8AHb/dW4nd0PJsJpwvCS3rfCsSMwf0K5Xha0T/ILH4fMxXZhVN3mwbmGUsFy+gg73Xb1dPjwFrt3SyAZAx//GXIy4cKnImKDpLJowvCSuLZ2fcOSl2FvhttYMtNsV1Sny6HTZW5jUaqiet8Cg/4CK962LYFAbicwfwKkvwkD7oRWZwbuvB6mCcNrBt5j9/ye8zd3MRw+AB/8zrYqLihfa0cpzxlwp+0uWvoqfDDa1nqqrPQpMPdv0HkkDLqv8ucLE04ThohcLiKrRCRfREqc/yYiQ0RkjYisF5FxoYwx5Oo2g95/sJsrbV8e+usbYz+J7fkRLnne7kGuVDgTgZQH4ey/wMp34NUL7AK/ilqXCtNvs4vzhj8d0bOiTuS6hfEdMAKYX9IBIhID/Ac4H+gAXCUiZc87DWd9x9g6TakPhv7aXz9rB94H3VtiTR6lwo4IDLjLzp7avRZeHATrZ5fvHPnHbDftlCugcXu48o2oGLcoymnCMMasNsasKeOwnsB6Y8xGY8wR4G3gouBH51CNevaXe+NcWP1R6K67YS589hdof6GdIqhUpGk3FG5MtaVE3rwU3rwMdv5Q9uuyt8Brw215nM5Xwg2feL5YaDCEw8K9FsCWIt9nAsWOMInIaGA0QKtWYV6Hvsfv7Cf96bdDfA+oc3Jwr7c3A977LcSdBhc/p1uuqsjVtAP8/itbv+2Lf8FzfexWA6cOtrf14iHvsP3aOM8OmG+YC7E14OLnoetVrn8CZ4KeMERkNlDcu919xpgPA3ktY8yLwIsAycnJAZwOEToTU9faabKx1WDEy/DCADsl8Jr3gtdXun8HvDECTD6MfMvuMaBUJIutBn1uhS5X2XLkP8yAdZ8Vf2zdeOj3R7u4tkHrUEbpOUFPGMaYcyp5iq1AyyLfx/sei0jH7SHROAnOfQRm3QmLX4IzRwf+ggf3wBsX2yq0106DRqcE/hpKedVJjezf2LmP2KoGG+bYFeKx1SGmGjTpAK37aovbJxy6pJYAbUUkEZsoRgJXuw0phHrcZKvYpt5vq8Q27xq4c+fmwBuX2D+Ua96JmrnkShWr0Sn6gakMrqfVXiIimUBvYKaIfOp7vLmIzAIwxuQBtwKfAquBd4wxq1zFHAyl7iEhYkscnNTYtgR+WhmYi+ZshdcuhB3f2dkebQYG5rxKqYglJpArHz0kOTnZpKWluQ6j3EqsPLtnI0weZnfnu36GXxVtS5SZBm9fDUd+hstesVtcKqUUICJLjTHFrovTjrlw0bANXP8RVKkKrw+HLUvKf478fFg6GV69gOy8WLhptiYLpZTfNGF4TKmVZxudAqNm2Ol9kwbDJ/fYVoI/MtNg0jnw0Rho2ZNB2ffbDWaUUspPmjA8pszKs3Ft4feLoMeNdlX2s73gq2dtl9WJcvfByvfsRjIvp9iqmhc/D9dNZy91g/MDKKUilo5hhLNNX8HHd8NPK+z3jdraWlTH8uxYx47vbLn0kxpDt9/wzNGLmPDFr2voBGVbV6VUWCptDEMTRiTY86Odert+NhzeDzFV7VeTDrbMR3wPqBJz3EuCvq2rUioslZYwwmEdhipLw0TodbP9UkqpINExjCgVtG1dlYpQE1PXug7BOU0YUUrHLJQqnyc/X+c6BOc0YSillPKLjmEopVQJJqauPa5lUVC+J1pnFuosKaWU8kO0zCzU0iBKKaUqTROGUkr5QWcWasJQSim/ROOYxYk0YSillPKLJgyllFJ+0YShlFLKL5owlFJK+UUThlJKKb9E7MI9EdkFbHIdRwXEAbtdBxFi+jNHB/2Zw0NrY0zj4p6I2IQRrkQkraRVlpFKf+booD9z+NMuKaWUUn7RhKGUUsovmjC850XXATigP3N00J85zOkYhlJKKb9oC0MppZRfNGEopZTyiyYMDxORO0TEiEic61iCTUQeE5EfRGSFiEwVkfquYwoGERkiImtEZL2IjHMdT7CJSEsRmSsi34vIKhEZ4zqmUBGRGBFZJiIzXMcSKJowPEpEWgLnAptdxxIiqcDpxpjOwFrgHsfxBJyIxAD/Ac4HOgBXiUgHt1EFXR5whzGmA9ALuCUKfuYCY4DVroMIJE0Y3jURuBuIilkJxpjPjDF5vm+/BuJdxhMkPYH1xpiNxpgjwNvARY5jCipjzHZjzLe++/uxb6At3EYVfCISDwwFXnYdSyBpwvAgEbkI2GqMWe46FkduAD52HUQQtAC2FPk+kyh48ywgIglAN+Abx6GEwhPYD3z5juMIqFjXAUQrEZkNnFzMU/cB92K7oyJKaT+zMeZD3zH3Ybsx3gplbCq4RKQ28D7wR2PMPtfxBJOIDAN2GmOWishAx+EElCYMR4wx5xT3uIh0AhKB5SICtmvmWxHpaYz5KYQhBlxJP3MBERkFDANSTGQuENoKtCzyfbzvsYgmIlWxyeItY8wHruMJgb7AcBG5AKgB1BWRN40xv3EcV6Xpwj2PE5EMINkYE24VL8tFRIYAjwNnGWN2uY4nGEQkFjugn4JNFEuAq40xq5wGFkRiP/W8BuwxxvzRcTgh52th3GmMGeY4lIDQMQzlFc8AdYBUEUkXkeddBxRovkH9W4FPsYO/70RysvDpC1wLnO37f033ffJWYUhbGEoppfyiLQyllFJ+0YShlFLKL5owlFJK+UUThlJKKb9owlBKKeUXTRhKKaX8oglDKaWUXzRhKBVCvr0hBvvu/01EnnYdk1L+0lpSSoXWg8DDItIEW7l1uON4lPKbrvRWKsRE5AugNjDQt0eEUmFBu6SUCiFfNeJmwBFNFircaMJQKkREpBl2n4+LgAO+Cr1KhQ1NGEqFgIjUAj7A7m+9GngEO56hVNjQMQyllFJ+0RaGUkopv2jCUEop5RdNGEoppfyiCUMppZRfNGEopZTyiyYMpZRSftGEoZRSyi//D23bwnoS2sVtAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# WRITE THE PLOTTING CODE HERE\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "#@ solution\n",
        "k = 5\n",
        "# feature matrix\n",
        "Phi = poly_features(X, k)\n",
        "\n",
        "# maximum likelihood estimate\n",
        "theta_ml = nonlinear_features_maximum_likelihood(Phi, y)   \n",
        "\n",
        "# feature matrix for test inputs\n",
        "Phi_test = poly_features(Xtest, k)\n",
        "\n",
        "ypred_test = Xtest*0 ## <--- EDIT THIS LINE (hint: you may require a few lines to do the computation)\n",
        "\n",
        "#@ Test\n",
        "assert any(ypred_test), \"Prediction not implemented\"\n",
        "\n",
        "plt.plot(Xtest, ypred_test) \n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\")\n",
        "plt.legend([\"data\", \"maximum likelihood fit\"]);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59ohZghTjmsO"
      },
      "source": [
        "#### Question\n",
        "If you did not have a designated test set, what could you do to estimate the generalization error (purely using the training set)?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kLiBNTDBjmsO"
      },
      "source": [
        "## 2. Maximum A Posteriori Estimation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8ZSyllXjmsO"
      },
      "source": [
        "We are still considering the model\n",
        "$$\n",
        "y = \\boldsymbol\\phi(\\boldsymbol x)^T\\boldsymbol\\theta + \\epsilon\\,,\\quad \\epsilon\\sim\\mathcal N(0,\\sigma^2)\\,.\n",
        "$$\n",
        "We assume that the noise variance $\\sigma^2$ is known."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHvGyqI1jmsP"
      },
      "source": [
        "Instead of maximizing the likelihood, we can look at the maximum of the posterior distribution on the parameters $\\boldsymbol\\theta$, which is given as\n",
        "$$\n",
        "p(\\boldsymbol\\theta|\\mathcal X, \\mathcal Y) = \\frac{\\overbrace{p(\\mathcal Y|\\mathcal X, \\boldsymbol\\theta)}^{\\text{likelihood}}\\overbrace{p(\\boldsymbol\\theta)}^{\\text{prior}}}{\\underbrace{p(\\mathcal Y|\\mathcal X)}_{\\text{evidence}}}\n",
        "$$\n",
        "The purpose of the parameter prior $p(\\boldsymbol\\theta)$ is to discourage the parameters to attain extreme values, a sign that the model overfits. The prior allows us to specify a \"reasonable\" range of parameter values. Typically, we choose a Gaussian prior $\\mathcal N(\\boldsymbol 0, \\alpha^2\\boldsymbol I)$, centered at $\\boldsymbol 0$ with variance $\\alpha^2$ along each parameter dimension."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kWSQ9xSujmsQ"
      },
      "source": [
        "The MAP estimate of the parameters is\n",
        "$$\n",
        "\\boldsymbol\\theta^{\\text{MAP}} = (\\boldsymbol\\Phi^T\\boldsymbol\\Phi + \\frac{\\sigma^2}{\\alpha^2}\\boldsymbol I)^{-1}\\boldsymbol\\Phi^T\\boldsymbol y\n",
        "$$\n",
        "where $\\sigma^2$ is the variance of the noise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "NRTqC7_NjmsQ"
      },
      "outputs": [],
      "source": [
        "## EDIT THIS FUNCTION\n",
        "def map_estimate_poly(Phi, y, sigma, alpha):\n",
        "    # Phi: training inputs, Size of N x (K+1)\n",
        "    # y: training targets, Size of D x 1\n",
        "    # sigma: standard deviation of the noise \n",
        "    # alpha: standard deviation of the prior on the parameters\n",
        "    # returns: MAP estimate theta_map, Size of (K+1) x 1\n",
        "    \n",
        "    \n",
        "    theta_map = np.zeros((K+1,1)) ## <-- EDIT THIS LINE\n",
        "    \n",
        "    return theta_map"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "8Zg-ywutjmsR"
      },
      "outputs": [],
      "source": [
        "# define the function we wish to estimate later\n",
        "def g(x, sigma):\n",
        "    p = np.hstack([x**0, x**1, np.sin(x)])\n",
        "    w = np.array([-1.0, 0.1, 1.0]).reshape(-1,1)\n",
        "    return p @ w + sigma*np.random.normal(size=x.shape) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "WH8UmUOujmsR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "040779ad-0cbd-4ee5-b445-d08fc1b0d6fd"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAS3ElEQVR4nO3df4xl5V3H8fdHsFVrtVXWQtmlg3YxJYhVb1ZNjbYu4BaQba01xahgNZsmomujqVM3sYbaBNPEtdFGu6kaohjaqATs0sKAv6IJymyltZQyrBsqu1U7LVVr/EG2/frHXHBY7syeZeae596571dyM/ec8+y93zuE+dxznvM8T6oKSZJO50taFyBJmg4GhiSpEwNDktSJgSFJ6sTAkCR1cnbrAsbpnHPOqbm5udZlSNLUOHLkyGeqatuoY1s6MObm5lhcXGxdhiRNjSSfXOuYl6QkSZ0YGJKkTgwMSVInBoYkqRMDQ5LUiYExIw4uLLUuQdKUMzBmxLvufaR1CZKmnIEhSepkIgIjyZ4kDyc5mmR+xPHrkywneWD4+MkWdU6bgwtLzM0fZm7+MMBTz708JenZSOsFlJKcBSwBlwPHgfuBa6vq46vaXA8MquqGM3ntwWBQjvReMTd/mEdvuqp1GZImXJIjVTUYdWwSzjB2AUer6lhVPQHcCuxtXJMk6RSTEBjnA4+t2j4+3Heq1yX5aJI/SrJjrRdLsi/JYpLF5eXlza51au3fvbN1CZKm3CQERhd/CsxV1aXAAnDzWg2r6lBVDapqsG3byAkXZ9KbL7+odQmSptwkBMYJYPUZw/bhvqdU1Wer6n+Hm+8Fvq2n2qaCndiS+jAJgXE/sDPJhUmeA7wBuGN1gyTnrdq8Bniox/omnmMsJPWh+XoYVXUyyQ3AXcBZwO9W1YNJbgQWq+oO4GeSXAOcBB4Hrm9WsCTNqOa31Y7TVr6t9uDC0sgzi/27d9pfIelZW++2WgNjC3CMhaTNMunjMCRJU8DA2AIcYyGpDwbGFmCfhaQ+GBiSpE4MDElSJwaGJKkTA0NSc05vMx0MDEnNOb3NdDAwJEmdNJ9LStJsOnV6myeXEnZ6m8nl1CCSmnN6m8nh1CCSpA0zMCQ15/Q208HAkNScfRbTwcCQJHViYEiSOpmIwEiyJ8nDSY4mmR9x/LlJ3jc8/rdJ5hqUKUkzrXlgJDkLeDfwauBi4NokF5/S7CeAz1XVS4GDwK/2W6UkqXlgALuAo1V1rKqeAG4F9p7SZi9w8/D5HwG7k6THGiVp5k1CYJwPPLZq+/hw38g2VXUS+Hfga0e9WJJ9SRaTLC4vL4+hXEmaTZMQGJuqqg5V1aCqBtu2bWtdjiRtGZMQGCeAHau2tw/3jWyT5Gzgq4HP9lKdJAmYjMC4H9iZ5MIkzwHeANxxSps7gOuGz38Q+LPaypNgSdIEaj5bbVWdTHIDcBdwFvC7VfVgkhuBxaq6A/gd4PeTHAUeZyVUJEk9ah4YAFV1J3DnKft+adXz/wFe33ddkqT/NwmXpCRJU8DAkCR1YmBIkjoxMCRJnRgYkqRODAxJUicGhiSpEwNDktSJgSFJ6sTAkCR1YmBIkjoxMCRJnRgYm+TgwlLrEiRprAyMTfKuex9pXYIkjZWBIUnqZCLWw5hWBxeWnnZmMTd/GID9u3fy5ssvalWWJI1FWq50muRrgPcBc8CjwA9V1edGtPsC8A/DzX+qqmu6vP5gMKjFxcXNKfY05uYP8+hNV/XyXpI0LkmOVNVg1LHWl6TmgXuraidw73B7lP+uqpcPH53CQpK0uVoHxl7g5uHzm4HXtCtlY/bv3tm6BEkaq9aB8aKq+ufh838BXrRGuy9LspjkviSv6ae0M2OfhaStbuyd3knuAc4dcejA6o2qqiRrdai8pKpOJPl64M+S/ENV/eMa77cP2AdwwQUXbKBySdJqYz/DqKrLquqSEY/bgX9Nch7A8Oen13iNE8Ofx4C/AL5lnfc7VFWDqhps27Zt0z+PtJU44FRnovUlqTuA64bPrwNuP7VBkhcmee7w+TnAK4CP91ahtIU54FRnonVg3ARcnuQR4LLhNkkGSd47bPMyYDHJR4A/B26qKgNDknrWdBzGuPU5DkOaFqcOOH2SA04F64/DMDCkGeaAU51qkgfuSZKmhIEhzTAHnOpMGBjSDLPPQmfCwJAkdWJgSJI6MTAkSZ0YGJKkTgwMSVInBoYkqRMDQ5LUiYEhdeA04JKBIXXiNOCSgSFJ6mjsS7RK0+rUacDn5g8DTgPewsGFJX/nE8DAkNbw5ssveuqPlNOAt/Wuex8xMCaAl6QkSZ00DYwkr0/yYJIvJhm5YMew3Z4kDyc5mmS+zxolcBrwFg4uLDE3f/ipS4FPPveOtXaarriX5GXAF4H3AD9fVc9YHi/JWcAScDlwHLgfuLbLut6uuCdtDV4S7M96K+417cOoqocAkqzXbBdwtKqODdveCuwFThsYkqTNMw19GOcDj63aPj7cN1KSfUkWkywuLy+PvThJ4+clwckw9jOMJPcA5444dKCqbt/s96uqQ8AhWLkktdmvL6l/3iE1GcYeGFV12QZf4gSwY9X29uE+SVKPpuGS1P3AziQXJnkO8AbgjsY1SdLMaX1b7WuTHAe+Ezic5K7h/hcnuROgqk4CNwB3AQ8B76+qB1vVLEmzqmlgVNVtVbW9qp5bVS+qqu8b7v9UVV25qt2dVXVRVX1DVb2jXcWSNPnGNVZlGi5JSZLOwLhmVzYwJEmdOPmgJE2p1bP49jG7soEhSVNq9Sy+fcyu7CUpSVInnmFsAS4uI82OLpeexjWVStPZasdtVmardSZPaTaN4//99War9ZKUJKkTL0lNKdebltT3LL5ektoCvCQlabN4SUqStGEGxhbg4jKS+nDawEiykOSb+yhGz459FpL60OUM4xeAX0/ye0nOG3dBkqTJdNrAqKoPV9WrgA8AH0rytiRfPv7SJGkyjGu68GnTqQ8jSYCHgd8Cfhp4JMmPjrMwSZoU45oufNp06cP4G1bW0D4InA9cD7wS2JXk0DiLkzQ+fmvWmeoycG8f8PF65oCNn07y0EbePMnrgV8GXgbsqqqRgyaSPAp8HvgCcHKte4Qldbd6plM9k4Njn+m0gXGa9bM3OlrsY8APAO/p0PZVVfWZDb6fJHXSx3Th02ZDU4NU1bEN/vuHAFa6SCSNm9+atRHTMpdUAXcnKeA9VbVm30mSfaxcRuOCCy7oqTxpOvit+dlxcOyKsQdGknuAc0ccOlBVt3d8me+qqhNJvg5YSPKJqvqrUQ2HYXIIVuaSelZFS9Iqnn2tGHtgVNVlm/AaJ4Y/P53kNmAXMDIwJHXjt2adqYmfSyrJ85I8/8nnwBWsdJZL2gC/NetMNQ2MJK9Nchz4TuBwkruG+1+c5M5hsxcBf53kI8DfAYer6kPjrs171CXp6Zp2elfVbcBtI/Z/Crhy+PwY0Pvkh96jLklPN/GXpCRJk2FabqvthfeoS9LaXKJ1Dd6jLmkWuUSrJGnDDIw1eI+6JD2dgbEG+ywk6ekMDGlCORZIk8bAkCaUq7xp0hgYeorfaCWtx3EYeoqj29tzLJAmmYExAw4uLPnHZkq4XoUmmYExA9Y7c/AbraSuDIwZ5zfayeVYIE0aA2OL8sxh+vnfSZPGwNiins2Zg99oJa3H22r1FL/RSlpP6xX33pnkE0k+muS2JC9Yo92eJA8nOZpkvucyp55nDpI2Q+szjAXgkqq6FFgC3npqgyRnAe8GXg1cDFyb5OJeq5xynjlI2gxNA6Oq7q6qk8PN+4DtI5rtAo5W1bGqegK4FdjbV42SpBWtzzBWeyPwwRH7zwceW7V9fLhvpCT7kiwmWVxeXt7kEiVpdo39Lqkk9wDnjjh0oKpuH7Y5AJwEbtno+1XVIeAQrKy4t9HXkyStGHtgVNVl6x1Pcj1wNbC7Rq8XewLYsWp7+3CfJKlHre+S2gO8Bbimqv5rjWb3AzuTXJjkOcAbgDv6qlGStKJ1H8ZvAs8HFpI8kOS3AZK8OMmdAMNO8RuAu4CHgPdX1YOtCtbmc1p1aTo0HeldVS9dY/+ngCtXbd8J3NlXXeqX06pL06H1GYYkaUo4l5SacHJEafpk9I1JW8NgMKjFxcXWZeg0nFZdmhxJjlTVYNQxL0lJkjoxMNSckyNK08HAUHP2WUjTwcCQJHViYEiSOjEwJEmdGBiSpE4MDElSJwaGJKkTA0OS1ImBIUnqxMCQJHViYEiSOjEwJEmdNF0PI8k7ge8HngD+Efjxqvq3Ee0eBT4PfAE4udbUu5Kk8Wl9hrEAXFJVlwJLwFvXafuqqnq5YSFJbTQNjKq6u6pODjfvA7a3rEeStLbWZxirvRH44BrHCrg7yZEk+9Z7kST7kiwmWVxeXt70IiVpPQcXllqXMDZjD4wk9yT52IjH3lVtDgAngVvWeJnvqqpvBV4N/FSS717r/arqUFUNqmqwbdu2Tf0sknQ6q9eq32rG3uldVZetdzzJ9cDVwO5aY4Hxqjox/PnpJLcBu4C/2uRSJUnraH2X1B7gLcD3VNV/rdHmecCXVNXnh8+vAG7ssUxJWtfBhaWnnVnMzR8GVpYf3korSmaNL/X9vHlyFHgu8Nnhrvuq6k1JXgy8t6quTPL1wG3D42cDf1hV7+jy+oPBoBYXFze9bklay9z8YR696arWZTxrSY6sdTdq0zOMqnrpGvs/BVw5fH4M+OY+65IkPdMk3SUlSVNv/+6drUsYGwNDkjbRVuqzOJWBIUnqxMCQJHViYEiSOjEwJEmdGBiSpE4MDElSJwaGJKkTA0OS1ImBIUnqxMCQJHViYEiSOjEwJEmdGBiSpE4MDElSJ80DI8nbk3w0yQNJ7h6utjeq3XVJHhk+ruu7Tkmadc0DA3hnVV1aVS8HPgD80qkNknwN8Dbg24FdwNuSvLDXKiVpxjUPjKr6j1WbzwNGLTL+fcBCVT1eVZ8DFoA9fdQnSVrRdE3vJyV5B/BjwL8DrxrR5HzgsVXbx4f7JEk96eUMI8k9ST424rEXoKoOVNUO4Bbghg2+174ki0kWl5eXN6N8SRI9BUZVXVZVl4x43H5K01uA1414iRPAjlXb24f7Rr3XoaoaVNVg27Ztm/MBNJUOLiy1LkHaUpr3YSTZuWpzL/CJEc3uAq5I8sJhZ/cVw33Smt517yOtS5C2lEnow7gpyTcCXwQ+CbwJIMkAeFNV/WRVPZ7k7cD9w39zY1U93qZcSZpNqRp1U9LWMBgManFxsXUZ6tHBhaWRZxb7d+/kzZdf1KAiabokOVJVg5HHDAxtVXPzh3n0pqtalyFNlfUCo3kfhiRpOhgY2rL27955+kaSOjMwtGXZZyFtLgNDktSJgSFJ6sTAkCR1YmCod07ZIU0nA0O9c8oOaToZGJKkTiZhLinNgFOn7JibPww4ZYc0TZwaRL1zyg5pcjk1iCRpwwwM9c4pO6TpZGCod/ZZSNPJwJAkdWJgSJI6MTAkSZ0YGJKkTgwMSVInW3rgXpJl4JOt61jHOcBnWhfRkJ9/tj8/+DuYxM//kqraNurAlg6MSZdkca0RlbPAzz/bnx/8HUzb5/eSlCSpEwNDktSJgdHWodYFNObn16z/Dqbq89uHIUnqxDMMSVInBoYkqRMDY0Ik+bkkleSc1rX0Kck7k3wiyUeT3JbkBa1r6kOSPUkeTnI0yXzrevqUZEeSP0/y8SQPJtnfuqYWkpyV5O+TfKB1LV0ZGBMgyQ7gCuCfWtfSwAJwSVVdCiwBb21cz9glOQt4N/Bq4GLg2iQXt62qVyeBn6uqi4HvAH5qxj7/k/YDD7Uu4kwYGJPhIPAWYObuQKiqu6vq5HDzPmB7y3p6sgs4WlXHquoJ4FZgb+OaelNV/1xVHx4+/zwrfzTPb1tVv5JsB64C3tu6ljNhYDSWZC9woqo+0rqWCfBG4IOti+jB+cBjq7aPM2N/MJ+UZA74FuBvG5fSt19n5UviFxvXcUbObl3ALEhyD3DuiEMHgF9k5XLUlrXe56+q24dtDrByqeKWPmtTO0m+Evhj4Ger6j9a19OXJFcDn66qI0le2bicM2Jg9KCqLhu1P8k3ARcCH0kCK5djPpxkV1X9S48ljtVan/9JSa4HrgZ212wMDDoB7Fi1vX24b2Yk+VJWwuKWqvqT1vX07BXANUmuBL4M+Kokf1BVP9K4rtNy4N4ESfIoMKiqSZu9cmyS7AF+DfieqlpuXU8fkpzNSgf/blaC4n7gh6vqwaaF9SQr345uBh6vqp9tXE5TwzOMn6+qqxuX0ol9GGrtN4HnAwtJHkjy260LGrdhJ/8NwF2sdPi+f1bCYugVwI8C3zv8b/7A8Nu2JpxnGJKkTjzDkCR1YmBIkjoxMCRJnRgYkqRODAxJUicGhiSpEwNDktSJgSH1aLgOxOXD57+S5Dda1yR15VxSUr/eBtyY5OtYmaX1msb1SJ050lvqWZK/BL4SeOVwPQhpKnhJSurRcIbi84AnDAtNGwND6kmS81hZ72Mv8J/DmXqlqWFgSD1I8hXAn7CylvVDwNtZ6c+QpoZ9GJKkTjzDkCR1YmBIkjoxMCRJnRgYkqRODAxJUicGhiSpEwNDktTJ/wGlAEiL9GCC2wAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Generate some data\n",
        "sigma = 1.0 # noise standard deviation\n",
        "alpha = 1.0 # standard deviation of the parameter prior\n",
        "N = 20\n",
        "\n",
        "np.random.seed(42)\n",
        "\n",
        "X = (np.random.rand(N)*10.0 - 5.0).reshape(-1,1)\n",
        "y = g(X, sigma) # training targets\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.xlabel(\"$x$\")\n",
        "plt.ylabel(\"$y$\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "bF1SRHhljmsR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "619e60ab-8f66-44d3-e640-203ab09d4780"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAABTeUlEQVR4nO3dd1jV1R/A8fe5Ay4bRMQBipqKA3GglSNXrjRHwzQtbVk2HJVpU3M1rMy0Mlfm1nJkprnS1NQchSM3ioiTPWTdcX5/oPw0FVDu4MJ5PQ/PE9zv95zPVfp47vme8zlCSomiKIrivDSODkBRFEUpGpXIFUVRnJxK5IqiKE5OJXJFURQnpxK5oiiKk9M5otOyZcvKkJAQR3StKIritPbt2xcvpQz4788dkshDQkLYu3evI7pWFEVxWkKIM7f6uZpaURRFcXIqkSuKojg5lcgVRVGcnEPmyG/FaDQSGxtLVlaWo0NRnIjBYCAoKAi9Xu/oUBTFYYpNIo+NjcXLy4uQkBCEEI4OR3ECUkoSEhKIjY2latWqjg5HURym2EytZGVl4e/vr5K4UmhCCPz9/dWnOKXUKzaJHFBJXLlj6ndGUYpZIlcURSmpjJcucfnLL8k+fdrqbatEfhujR4/ms88+u+3rK1eu5PDhw3aMSFEUZ2aMiSFh2neYLlywettOn8gnbTjukH5VIlcU5U6Y09IA0Hj7WL1tp0/kkzedsFpb48ePp2bNmrRo0YJjx44BMGPGDJo0aUJ4eDiPPvooGRkZ7Nixg1WrVjF8+HAaNGhAVFTULa9TFEW5xpySCoDWx9vqbTt9IreWffv2sXjxYiIjI1mzZg179uwB4JFHHmHPnj3s37+f2rVrM2vWLJo1a0a3bt2YOHEikZGRVK9e/ZbXKYqiXGNJTQFA6239RF5s1pHfiUkbjt8wEg8Z+SsAQ9rVYFj7mnfV5rZt2+jZsyfu7u4AdOvWDYBDhw7x3nvvkZycTHp6Oh07drzl/YW9TlGU0unaiFzj6Wn1tp0ykQ9rXzMvYYeM/JXoj7vYrK8BAwawcuVKwsPDmTNnDlu2bCnSdYqilE7m1FQ0Xl4IrdbqbauplaseeOABVq5cSWZmJmlpafzyyy8ApKWlUaFCBYxGIwsWLMi73svLi7SrDy/yu05RFAXAnJpik2kVKAGJfEi7GlZpp1GjRjzxxBOEh4fTuXNnmjRpAsDYsWO59957ad68OaGhoXnX9+7dm4kTJ9KwYUOioqJue52iKAqAJTUNjQ0edAIIKaVNGs5PRESE/O/BEkeOHKF27dp2j0Vxfup3R3EG0U/2Rbi4UGXO93fdhhBin5Qy4r8/d/oRuaIoijMwp6ag9fKySdsqkSuKotiBJSXVZlMrKpEriqLYgTk1Fa0NdnWCSuSKoig2Z8nORmZnF99VK0KIYCHEZiHEYSHEv0KIIdYITFEUpaSwpNpuez5YZ0OQCXhDSvm3EMIL2CeE2CClVBWlFEVRyJ1WAdAU1xG5lPKClPLvq/+dBhwBKhW1XaVg0dHR1KtXD4C9e/cyePDgfK+fMGHCDd83a9bMZrEpivJ/eQWzimsiv54QIgRoCPx1i9cGCiH2CiH2xsXFWbPbEsdkMt3xPREREXz11Vf5XvPfRL5jx4477kdRlDtntmHBLLBiIhdCeALLgKFSytT/vi6lnC6ljJBSRgQEBFirW6uJjo4mNDSUAQMGULNmTfr27cvGjRtp3rw5NWrUYPfu3QDs3r2b+++/n4YNG9KsWbO8crdz5syhe/futG7dmho1avDhhx/esh9PT0+GDRtG3bp1adeuHdf+UWvdujVDhw4lIiKCyZMns2/fPlq1akXjxo3p2LEjF64Wo9+3bx/h4eGEh4fz9ddf57W7ZcsWunbtCkB6ejrPPPMMYWFh1K9fn2XLljFy5EgyMzNp0KABffv2zYsFcg8xHj58OPXq1SMsLIwlS5bktdm6dWsee+wxQkND6du3L47YQKYozu7aHLlm8zsQu8/q7VulaJYQQk9uEl8gpVxe5AbXjoSLB4vczA3Kh0Hnj/O95OTJk/z444/Mnj2bJk2asHDhQrZv386qVauYMGECK1euJDQ0lG3btqHT6di4cSPvvPMOy5YtA3KT/KFDh3B3d6dJkyZ06dKFiIgbN2FduXKFiIgIJk2axJgxY/jwww+ZOnUqADk5Oezduxej0UirVq34+eefCQgIYMmSJbz77rvMnj2bZ555hqlTp/LAAw8wfPjwW76PsWPH4uPjw8GDuX+GSUlJPProo0ydOpXIyMibrl++fDmRkZHs37+f+Ph4mjRpwgMPPADAP//8w7///kvFihVp3rw5f/75Jy1atLijP3pFKe3Mqbl1mbQXd4Ap0+rtFzmRi9zTb2cBR6SUXxQ9JMepWrUqYWFhAHkjZiEEYWFhREdHA5CSkkL//v05ceIEQgiMRmPe/e3bt8ff3x/IrWO+ffv2mxK5RqPhiSeeAKBfv3488sgjea9d+/mxY8c4dOgQ7du3B8BsNlOhQgWSk5NJTk7OS7JPPfUUa9euvel9bNy4kcWLF+d97+fnl+/73r59O3369EGr1RIYGEirVq3Ys2cP3t7eNG3alKCgIAAaNGhAdHS0SuSKcofyplZcLODub/X2rTEibw48BRwUQkRe/dk7Uso1d91iASNnW3F1dc37b41Gk/e9RqPJm7d+//33adOmDStWrCA6OprWrVvn3fPfE90Lc8L79dd4eHgAuVMddevWZefOnTdcm5ycfEfvxxqu/zPRarV3NX+vKKWdJSUV4apHaLBJIrfGqpXtUkohpawvpWxw9evuk3gxl5KSQqVKuYty5syZc8NrGzZsIDExkczMTFauXEnz5s1vut9isfDTTz8BsHDhwluObmvVqkVcXFxeIjcajfz777/4+vri6+vL9u3bAW5bLrd9+/Y3zJ8nJSUBoNfrb/gEcU3Lli1ZsmQJZrOZuLg4tm7dStOmTQv6o1AUpZDMqalo3fW537jl/wn5bqidnXforbfe4u2336Zhw4Y3jU6bNm3Ko48+Sv369Xn00UdvmlaB3FH37t27qVevHr///jsffPDBTde4uLjw008/MWLECMLDw2nQoEHeCpPvv/+eV155hQYNGtz2weN7771HUlIS9erVIzw8nM2bNwMwcOBA6tevn/ew85qePXtSv359wsPDadu2LZ9++inly5e/qz8fRVFuZk5NRWvQgsEHtHqrt6/K2FrJnDlz2Lt3b96Dy9vx9PQkPT3dTlGVDs7+u6OUfGeeehrijlHloRwY/M9dt6PK2CqKojiIOTUVjY0edIKTntlZHA0YMIABAwYUeJ0ajStK6WNOTcXgZ7RZIlcjckVRFBuzpKSg1eaoEbmiKIozkkYjlowMNJpMm6xYATUiVxRFsSnz1elUW47IVSJXFEWxIUuKbXd1gkrkxcro0aP57LPPbvr5ypUrOXz4zsu7R0dHs3Dhwrzv58yZw6uvvlrgfdu2baNu3bo0aNCAzEzr1YVQZXSV0iivFrleJfJiwxFb1PNL5PnF899EXlgLFizg7bffJjIyEjc3tzu+/3ZUGV2lNMqrRe4iVSK3h7Fjx1KrVi1atGhBnz598kbH/y0xu2nTJho2bEhYWBjPPvss2dnZAISEhBAfHw/kHvRwrQ7L6NGjefbZZ2ndujXVqlW7oW74+PHjqVmzJi1atMgriXu9HTt2sGrVKoYPH06DBg2Iioq6KZ4BAwbkbfuH/5enHTlyJNu2baNBgwZMmjQJgPPnz9OpUydq1KjBW2+9dVN/M2fOZOnSpbz//vv07dv3hvK4AK+++mpeaYKQkBBGjRpFo0aNCAsL4+jRo4Aqo6so17N1wSwopqtWPtn9CUcTj1q1zdAyoYxoOuK2r+/Zs4dly5axf/9+jEYjjRo1onHjxnmvXysxm5WVRY0aNdi0aRM1a9bk6aef5ttvv2Xo0KH59n/06FE2b95MWloatWrVYtCgQRw4cIDFixcTGRmJyWS6qU/InX7o1q0bXbt25bHHHrspHuC269c//vhjPvvsM1avXg3kTq1ERkbyzz//4OrqSq1atXjttdcIDg7Ou+f5559n+/btef1t2bIl3/dVtmxZ/v77b7755hs+++wzZs6cqcroKsp18mqRqzly2/vzzz/p3r07BoMBLy8vHn744Rtev77EbNWqValZsyYA/fv3Z+vWrQW236VLF1xdXSlbtizlypXj0qVLbNu2jZ49e+Lu7o63tzfdunUrdLzX4rlT7dq1w8fHB4PBQJ06dThz5sxdtXPNtTK8jRs3ziv1u3HjRl555ZW8a+62jC6QV0ZXo9HkldFVFGeSV4vcRYKbr036KJYj8vxGzo5yrcRsfnQ6HRaLBYCsrKwbXrN2Odjr47m+X4vFQk5Ozm3vu9M4rm8bbv++bFXiVpXRVZydOTUFodOg8fQDjdYmfagR+VXNmzfnl19+ISsri/T09LzpiP+qVasW0dHRnDx5EoB58+bRqlUrIHfOeN++3GOcrp0alJ8HHniAlStXkpmZSVpaGr/88sstr/Py8iItLe227Vzf76pVq/JK1RZ0X2FUqVKFw4cPk52dTXJyMps2bSrwHlVGV1H+z5KaisagBfcyNutDJfKrmjRpQrdu3ahfvz6dO3cmLCwMHx+fm64zGAx8//33PP7444SFhaHRaHjppZcAGDVqFEOGDCEiIgKttuB/eRs1asQTTzxBeHg4nTt3pkmTJre8rnfv3kycOJGGDRsSFRV10+svvPACf/zxB+Hh4ezcuTNvtF6/fn20Wi3h4eF5DzvvVHBwML169aJevXr06tWLhg0bFniPKqOrKP9nTklF6ypsNj8OqoztDdLT0/H09CQjI4MHHniA6dOn06hRI4fGpBSsOPzuKMrtnBnwDPLMXkIG1oM+i4rU1u3K2BbLOXJHGThwIIcPHyYrK4v+/furJK4oSpFZUlPR6Uw2nVpRifw6d7N5RlEUJT/m1FRcdNk2nVpRc+SKoig2ZE5JQau3XS1yUIlcURTFZqTFgiU93abb80ElckVRFJuxpKWBlDbd1QkqkSuKotjMtcqHWhtWPgSVyG1u1apVfPzxxw7rv3Xr1nk1WR566CGSk5OJjo6mXr16hW7j+vK6H3zwARs3bgRuLBJmTbZqV1HsLS+Ru0hwU6tWnFa3bt3uqIaKLa1ZswaA5OTku25jzJgxVopGUUq+GwtmFfOdnUKI2UKIy0KIQ9ZozxGio6MJDQ1lwIAB1KxZk759+7Jx40aaN29OjRo12L17NwC7d+/m/vvvp2HDhjRr1iyv9OykSZN49tlnATh48CD16tUjIyPjhsMcBgwYwKBBg7jvvvuoVq0aW7Zs4dlnn6V27do3VDC8Vt4V4Keffsp7rbD3386tRrqnTp2iYcOG7Nmzh6ioKDp16kTjxo1p2bJlXlna6/23ZO6UKVNuKmObmJhIjx49qF+/Pvfddx8HDhzI9+cJCQl06NCBunXr8vzzz6tStUqJYb46aNK6AgZfm/VjrRH5HGAqMNcajV2cMIHsI9YtY+taO5Ty77yT7zUnT57kxx9/ZPbs2TRp0oSFCxeyfft2Vq1axYQJE1i5ciWhoaFs27YNnU7Hxo0beeedd1i2bBlDhgyhdevWrFixgvHjx/Pdd9/h7u5+Ux9JSUns3LmTVatW0a1bN/78809mzpxJkyZNiIyMpEGDBvnGWNT7r3fs2DF69+7NnDlzCA8Pp127dkybNo0aNWrw119/8fLLL/P777/n28atytiOGjWKhg0bsnLlSn7//XeefvppIiMjb/vzDz/8kBYtWvDBBx/w66+/MmvWrEK/B0UpzkwJiQDofL1BY7uZbKskcinlViFEiDXacqSqVasSFhYGQN26dWnXrh1CCMLCwvLKp6akpNC/f39OnDiBECKvCJRGo2HOnDnUr1+fF198kebNm9+yj4cffjivzcDAwBv6i46OLjARF/X+a+Li4ujevTvLly+nTp06pKens2PHDh5//PG8a64dmJGf68vYLl++HMgtS3utaFjbtm1JSEggNTX1tj/funVr3r1dunQpsOytojgLc2IiCND62W5aBew4Ry6EGAgMBKhcuXK+1xY0craV60umajSavO81Gk1e+dT333+fNm3asGLFCqKjo/NOAQI4ceIEnp6enD9/vsA+rm//v30IIfJ+fruysfndXxg+Pj5UrlyZ7du3U6dOHSwWC76+vrc8+CE/ti5jqyjOzJSYgNagQXiWtWk/dlu1IqWcLqWMkFJGBAQE2Ktbq0tJSaFSpUoAeUeeXfv54MGD2bp1KwkJCTfMI9+pwMBAjhw5gsViYcWKFUUN+ZZcXFxYsWIFc+fOZeHChXh7e1O1alV+/PFHIPf4tf37999V2y1btmTBggVA7nFtZcuWxdvb+7Y/f+CBB/LKI6xduzav7K2iODtzYhJaAzZ90Alq1code+utt+jfvz/jxo2jS5cueT8fNmwYr7zyCjVr1mTWrFm0adMm77iyO/Xxxx/TtWtXAgICiIiIID093Vrh38DDw4PVq1fTvn17PD09WbBgAYMGDWLcuHEYjUZ69+5NeHj4Hbd77YzS+vXr4+7uzg8//JDvz0eNGkWfPn2oW7cuzZo1K/ATm6I4C3NiIjpXs03XkIMVy9henSNfLaUscIFycS1jqzgn9bujFFdRD3XB1XiEoBH94cFRRW7vdmVsrbX8cBGwE6glhIgVQjxnjXYVRVGcmTkxAa2ryeYjcmutWuljjXYURVFKCmkyYU5OQVfJtpuBoJht0VcbQZQ7pX5nlOIqbzOQwfZz5MUmkRsMBhISEtT/mEqhSSlJSEjAYDA4OhRFuUneZiBX2xbMgmK0aiUoKIjY2Fji4uIcHYriRAwGA0FBQY4OQ1FuYk7KTeRaVwt4lrNpX8Umkev1eqpWreroMBRFUazCnHh1RG6wgGegTfsqNlMriqIoJcm1qRWtrxfoXAu4umhUIlcURbGBvKmVMrYdjYNK5IqiKDZhSkhE66ZB+JS3eV8qkRczkzYcd3QIiqJYgTkxEa3BAp4qkZc6kzedcHQIiqJYgSkxEZ1LDnipqRVFURSnZE6IQ+titsuIvNgsPyzNJm04fsNIPGTkrwAMaVeDYe1rOiosRVGKwJyYiK6cxS4jcpXIi4Fh7WvmJeyQkb8S/XGXAu5QFKU4k0Yj5tR0tJXtMyJXUyt2ph5mKkrJ9/9Dly3gpRJ5iVPQw8wh7WrYKRJFUWzFZMddnaASebGj5sQVxfld256v9XAFV0+b96fmyO1APcxUlNIlb0RexrZ1yK9RidwO1MNMRSldzNfqrATYfloF1NSKoiiK1ZmSEkGAtmwFu/SnErmdqYeZilLymRMS0bpaED4qkZdIak5cKc7U8ljrMMdfRudqtsuKFVBz5LclLRbS1q/HeOEiwkWPcHHBvVEjXKtXd3RoimIzkzedUIMNKzDFX7bbGnJQifyWMiMjuTj6fbKOnrzxBQG+3btQ9vW30Jez7dFNiqI4L3NCAq6u9llDDiqR30CazVwc9T7JP61A52amYksTns2aIM1mLBlZJG06SOKq1aSs+Y1yQwdT5tmBjg5ZUQBIzErkRNIJLmVcIi0njbScNMzSjIfOA3e9O/4Gf6r6ViXYKxi9Rn/DvWp5rPWZklPxqGhWI3JHiP/yM5J/WoFfzSsE9OuKtstY8Pj/6deB/U7jt2w0Fxdu5dKnk7CkJVN2yFsOjFgprc6ln2Nb7Da2n9vOofhDJGQlFOo+nUbHPb73cG/5e2lWsRmNAhup5bFWJo1GLFcyrx66rEbkdpW+ZTPxM+bgUy2bwK8WI0Ka3XxRmaq4vPADwR12c35QX+K+/R6BBf8hI+0fsFLqpGSnsPrUapafWM7xpNyHkkGeQbQMakkN3xrU8KtBJc9KeLt44+HigVZoyTRlkp6TTnxmPKdSThGVHMXB+IMsPLqQHw7/gJvOjfZV2tPjnh40Dmzs4HdYMpiSkgDQuWnAzc8ufapEDuTEnuP860Nx9TVSftToWyfx64gqTak4/Sd44XEuf/sDCPAfrJK5YhsxqTHMODiDtafXkm3Opq5/XYZHDOeBoAeo4l0FIcRt7/XQe+Ch9yDQI5C6Zevm/TzDmMHfl/9m45mNrItex6qoVQR5BtHhvs5kmbIw6Az2eGslUt72fF9vyOfvxpqsksiFEJ2AyYAWmCml/Nga7dqDNJk499IApDGLoMFt0dz7dKHuE0HhVJy9EvnsI1z+dg6G8MZ4tGpv42iV0iQ2LZbpB6azKmoVOo2ObtW78VjNx6jjX6fIbbvr3WlRqQUtKrVgRNMRbIrZxKKji9gZN4NOy5bzdN2n6RPaBzedmxXeSelitvP2fLDCOnIhhBb4GugM1AH6CCGK/ptmJ6krlpJ1MpbyHcrg0m/KHd0rKtSh4tfzcPG0cP7N1/PqKyhKUWSZspjyzxS6rezGr6d+pU9oH9Y+spYP7v/AKkn8v9x0bnSt1pX5neczu+NsavrVZNK+SXRb2Y01p9YgpbR6nyWZ6dr2/LIBduvTGhuCmgInpZSnpJQ5wGKguxXatTlpsRD/9WRcfYx4v/U96FzvuA1NtSZUfOsZTFeMXHipt/qlV4pkW+w2ev7ck+kHptMxpCNrHlnDiKYjCHC3fVIQQtCkfBOmd5jO9x2/x8/VjxHbRvDU2qc4lnjM5v2XFKbLlwHQVQiyW5/WSOSVgLPXfR979Wc3EEIMFELsFULsjYuLs0K3RZe2dhU5F1Pxb18bERh61+24PTaSwE7VSD9wlqTJY6wYoVJaZBgzGL1jNC9vehm9Vs+sDrP4qOVHBHrYZ9XDf0WUj2Bx18WMbT6Ws2ln6b26N99EfoPRbHRIPM7EdOE8Gp0FTVnnSuSFIqWcLqWMkFJGBATY7yNHPvGQMPkz9B4mvF8aV7TGhMBvwmI8qmi4PGMxOVFHrROkUir8G/8vT6x+guUnlvNcvef46eGfaFqhqaPDQiM09LinBz93/5lOVTvx7f5veeLXJ9TovADGc2fQuZsR3vZZQw7WSeTngODrvg+6+rNi7cqWjWTFJODfpioiKLzI7QmDNxU+mgRYuDTy5aIHqJR4UkqWHF1CvzX9yDRlMqvjLIY2HoqL1sXRod3A1+DLRy0/YmrbqSRnJfPkr0+y+OhiNY14G6aLF9C7WexyVuc11kjke4AaQoiqQggXoDewygrt2lTCpI/QuZnxecV6UyH6Rh0o2zGU9IMXSF8xx2rtKiVPjjmH0TtHM+6vcTSr1Ixl3ZbRpHwTR4eVr1bBrfjx4R9pWqEp4/8az+tbXic1J9XRYRU7xsvx6NzN4GW/abEiJ3IppQl4FVgHHAGWSin/LWq7tpR1MJKM4xfwb1ERTdX7rNq2/+gZuPhILn7yGZbMDKu2rZQM8ZnxPLPuGZafWM4LYS8wpe0UfFx9HB1Wofi7+fN1u695M+JNtpzdwpO/PklUcpSjwyo2pNmMKSkVnZsZvG96VGgzVpkjl1KukVLWlFJWl1KOt0abtpS6cBoIifczb1i9beFdjsBXn8aYbCZx3GCrt684t+iUaPqt6ceJpBN83upzBjcajEY4VzVpjdDQv25/ZnWcRXpOOn3X9OX3mN8dHVaxYIpPAItE76UDd/+Cb7AS5/oNsgIpJalbduFREXQNHrJJH5793sarlgfxK7djPKMeDCm5DsQd4Om1T5NhzGBWh1l0COng6JCKpFFgIxZ3XUyIdwhDNg9hxoEZpX7e3HTpIgC6AH+77eqEUpjIsyL3YEzKxrt5OGi0tulECMqN+gRpgfgxw2zTh+JU/jz3J8+tew5PF0/mPzSfsIAwR4dkFeU9yvND5x/oUq0LX/3zFaN3jsZoKb1LFI0XcxO5vkJFu/Zb6hJ52qLpICRevV6waT8ujdrhd28lknecIufgLpv2pRRvm2M289rvr1HVpyrzOs+jsndlR4dkVa5aVz5q8REv1n+R5SeW88rGV0jLSXN0WA5hungJAF1QVbv2W6oSuZSS1K178AgSaOu0tXl//u99jtBA3LgRNu9LKZ7WR6/n9S2vE1omlBkdZuDvZr95U3sSQvBqw1cZ02wMey7u4dl1zxKfGe/osOzOeO4MQiPRVrLvSWKlKpFn7d2BMTkH75aNQWP7t66/pwFl2oaSuv8yWdtX27w/pXj57fRvDN86nLCAMKa3n+40K1OKomeNnkxtN5UzqWd4eu3TnE07W/BNJYgpNhqdmxnhV8Wu/ZaqRJ66eAZoJF69BtrtkFn/d79E4yK5/LHaul+abDqziZHbRtKwXEOmPTgNTxdPR4dkN80rNWdGhxmk5qTy9NqnS9VOUNOFC+jdzeBr3+mzUpPIpZSkbtuHZ7AGba2WNxxtZUva8iH4d23KlZNpZK5fbJc+FcfaFruNN7e+Sd2ydfm63de4690dHZLdhQeE80OnH9AIDc+ue5ZD8YccHZJdGOOubgbyVSNym8jauwNTqgmvlk3tuiwIwO/NiWhdJfFTPrdrv4r97b6wm2FbhlHDtwbfPvgtHnoPR4fkMNV9qzO381y8XLx4fv3z/H3pb0eHZFNSSkyJ6eg9teBuv1rkUIpOCLqydgkAgxJqs/Pq4bL2OmRWWyaQMp0iiPt5H1mbl2Fo86jN+lIc50jCEQZvHkyQZxDftf8Obxdv23ZosUDcUbh8GBJPQUIUpF+C7FTISgWLCXQG0BvA1Tv3475vFfCvBkFNwCfY5oOaSp6V+KHTDzy//nle2vgSk9tM5v6K99u0T0cxJyUhzRZ0/r52Hyw6fSK3SAtHE48SeTmS0ymniU6N5nz6eTJNmWSZszBZTHjqPRmyK44y/lC3Vw6P+mfy1sJ4jn/wPDqNff4I/N78lIQ1bYmf/ClBKpGXODGpMby08SW8XLyY1n4afgYbndUYdxyO/QrR2+HsHshO+f9r3kHgXQEMvrlJW6MDUzaYsiAjEU6sz03013iWh8r3QWgXqNkRDLZ5GBvoEcj3nb5n4IaBvPb7a0xuM5nmlZrbpC9HMl1dQ64LLGf3vp0ykWebs/k95nc2ntnI7ou7Sc5OBnLPJwzxDqGOfx089B4YdAZ0Qkd6ZgpVz/7E/jA9q0//ypLjS/GoCq2WzKN5xea0DGpJm+A2Nn0gpQ2oiF+HhiT8Gkn2nz/j2twpzt5QCiE+M54XN7yIRVr4rv13lPewctW7xFPwzwI4/DMkXH22ExAK9XpC8H1QoT74VQWXQszFGzMh7hjE7oGzu+H0Vji8ErQuUL0tNH4GarS3+ma5sm5lmdVhFgM3DGTw74P5ss2XtAxqadU+HM14KfcfSX1F+9Uhv0Y4YkttRESE3Lt37x3fdzjhMMtPLGfN6TWk5aQR4BbA/RXv574K99GkfBMC3QNveRBtxh9rOfPi61R6tQuer3zKufRzfLp5Iz5loth2bhuJWYm46dzoXLUzj9V4jHpl6+V7oO3dMl08S9SD7fEM9aPSTzut3r5ifxnGDAb8NoDo1GhmdphJ/YD6d93WpA3H/z/FZzbBkVWw7/vcZCs0ENISaj8MtTqDj5WShcUC5/bm9nVwGaSdB78QaPI8RDwLLtad40/JTuGF9S9wMvkkk1pPolVwK6u270hJc2dzccJE7vm8P/outjmMXQixT0oZ8d+fO9WIfOXJlaw8uZIHqzxIj3t60LR800IVHMrY9DMA7h0eRyM0BHsFM6XbM0Du1MyBuAOsOLmCtafXsvzEcsIDwhkUPohmFZtZNaHrygfj17Y+CRsOELB3PS4Rzl1ro7QzWUwM3zqcY0nHmNJ2SpGSOMDkTScY1royRC6AHV9BUnTuFEmb96BhX/C2wbZvjQaCm+Z+tRsFR1fDX9Nh/Xvw51fQ6i1o1B901qmR7uPqw4wOM3hxw4sM2zKML9t8yQNBD1ilbUczxpwEIdFVrmX3vp1qRB6fGY+r1hUvF687ui+m6/2YEpKotuNIvg8h0nPS+eXUL8w+NJuLVy5Sv2x9hjQaYtXTWkxnozjZsQs+jctTYd4Wq7Wr2JeUkvF/jWfJsSW8f9/79KrVq2gNmk28/cFbfOT3K6RfhEqNocXrUOuhO968JnNyyDpxgpxTp8g5exbj2VhMly9jTk/DkpaOzMoCnQ6h06ExGNAFBKALDERfsSKG2qEY6tZFlxEFmz6EM3/mjtA7f5o7j24lKdkpDNwwkBNJJ0pMMj8/qB9Xdu2mxq9LoGJDm/RxuxG5UyXyu2HJzuZ4o3B8m1Sg/JzNhbonx5zDypMrmXlwJheuXKB9lfYMjxhOBc8KVonpwnNdSNkZxT0/z0dX46a/E8UJzDk0h8/3fc4z9Z7h9cav33U7k9Yf49CWJYzULaaG5hx7LDX5wvQ4TVt3Z1iHwo3szKmpZOzZw5Wdu8iMjCT72DGk8f+Fq3TlyqELDETr7Y3GywuNqyvSbEaaTFgyrmC6HJeb6BMT/39P+fJ4tmiBZ2gZPOIXoUk+BvUehU4fg6d1HuZdP80yuc1kp58zP/NIe+SlU4Rs2GOz5YelNpFnbFrJmVfeJmhoT7xemnBH92aZspjz7xxmHZwFwIvhLzKg7oAir3TJPrSHU489hX/b6pT75tcitaXY36aYTQzbPIz2VdozsdXEu68nnnQG1gyHE+vA/x5evPAw340fVailazlnz5K2YSNpGzaQuX8/WCwINzfc6tfHUK8ubnXr4lqrFvpKldAYDIUKx5x+hewjh8n8918y/4nkyvbtWK5cQRgMeDcMws97L4byLojuX0Ed6zysv5bMo5Kj+KrtV069miXqgSa4uiUR9NsJmy0/LLWJPO6d54hf/ic11y5GW7XBXbVxPv08E/dMZGPMRsLKhjGu+Tiq+VYrUlyxj7fmyrEL3LN+LdryRWtLsZ8jCUfo/1t/7vG9h9kdZ2PQFS5J3sBsgl3fwJaPcr9v8w7c+xIh764n+uMut73NlJRE6upfSVm5kqx/cw/hcq1TG6/WrfG4/37cwsMRLtY771Pm5JCxbx+pv60j9ZdfsGRk4FpOR9l7LuP1yFOITuNB51rkfpKzknl+/fNEp0Yzpe0Up1xnLqXkWP16+NXVE7g40mb9lNpEfqZTU8xpaVTbfrjI/0r+dvo3xv01jkxjJkMaDeGpOk/d9cPQzO1riX7+dcr1DMf/I7V13xnEZcTR59c+ACzqsogA94A7byQhCla8BLG7c+e/O38Kvrlnl9+wauUqKSWZe/eSuGAhaZs2gdGIoU4dvLt2xatDe1yC7LPUzZx+hdTVv5D4w1xyTp/G4JdDQOsAPIYvQpQpesnWpKwknlv/HGdTzzK13VTurXCvFaK2H3NaGsebNKVc+wr4T7HdaUm3S+RIKe3+1bhxY2kP5owMeaROLXnx2Qet1mZcRpx8ddOrst6cevLVTa/K5Kzku24rulNTebxhTWlJuWy1+BTbyDRmyt6/9JZN5jeRRxKO3HkDFouUe2ZLOa6ClB8FS3ngx/wvz86WScuWy6gePeXhWqHyWNN75cUJE2Tm0aN3+Q6sw2IyyaQVK+SJls3k4VqhMubBmjLn79+s0nZCZoLssbKHjJgXIXdf2G2VNu0l69gxebhWqEwe96RN+wH2ylvk1BJdayXzj1VIs8C9mfXm3cq6leWrNl8xoskItp/bTq9fenEg7sBdteX/0iuYMjSkTPvAavEp1ielZPTO0RxKOMRHLT8itEzonTWQlQpLn4bVQyEoAgbthLDHbnmpJSuLxPkLONmxExfeeQfMZsqPHcM9WzYT+PbbGGrZf2nb9YRWi2+PHlTbuJlyLz/DlQsaTj09mKQv3ynyMW9lDGWY0WEGFT0r8sqmV9h70T6f2q3BeDb3AGp9JfsWy7qmRCfyrF25q1Tc2vSwartCCPrV6cfcTnMRQjDgtwGsPLnyjtvx6PYUrgEuJK7YjDRmWTVGxXpmH5rNr6d+5bWGr9Gucrs7u/niIZjeGo7+Cg9+CE+tBJ+bT1eXOTkkLljAyfbtuTRuHPqKFQmeMYOqP6/E7/HH0bi5WeW9WIvGxQX/wW9R7adFGMq7cXHaCmL7dMGcmlqkdsu6lWVWx1mU9yjPy5tedppCW6bo3FK9uhDb1WzKT8lO5EeOovOwoKsabpP2wwLCWNJ1CY0CG/H+n+/z6Z5PMVlMhb5fCEGZvr3IThJcWfipTWIs6WxdV/6Ps38w+e/JdArpxAthd3g84IEfYWY7yLkC/X+BFkNvWhMuLRZSfvmFqIe6cGnsOFxDqlJl3lxCFszHs2ULm+wwtiaX0AZUXr2DwE5BpO8/xemu7ck6VrS/k2vb+QPdAxm0cZBTJPOc0ydASPRV6zmk/xKdyDPPxOEW5G3TSmQ+rj5Me3AafWv3Zd7heby66VXSc9ILfb/3M2+i8xAkzl+au11auSO2rCt/KvkUI7aNILRMKGOajyl8UrVYYNMYWP587sael7ZByM3Te5kHDhDduw/nh7+FxtOT4BnTqTz3B9ybNLHyO7Et4epOmc/XUuX5esjUJKIfe4TU334rUpsB7gHM7jibcu7leGnjS+y7tM9K0dpGzpkz6D3MiADHrEArsYncnHgZY7IFQ03b/8HqNDpGNh3J6PtHs+vCLgb8NoBLVy4VfCOgcXXFr3s7rpw1k7Vhjm0DVQotNSeVwZsH46p15au2X+GmK+TURnY6LH0Ktn0OjZ7OnUr5zwYaU0IC599+h+heT2C8cJ4KH31E1eXL8GzZsliPwPP99KPV4T5sCVVfb4HBJ4Nzw4aRuHBhkfq7lszLe5Rn0MZB7Lm4p0jt2VLOucu4+GjsXof8mhKbyLN25I4IDA3sN7p5tOajfN3ua86mnaXf2n6cTDpZqPv8Xh2N0EHi9G9tHGHJMGnDcUJG/ppXT/7af1trmsVsMfPW1rc4l36OSa0nFb6aYfplmNMFjq3J3QH58Fc31CiRUpK8bBlRD3UhZfVq/F94nuprf8O3Zw+EHc6QLaoCP/1otOj6zaTykPZ4Vsji0pixxE39ukgPQa9P5q9seoVdF3bddVu2IqXEGJeOS3kblS4ubBB3+wU8DvwLWICIwt5nj+WH8aMGysO1QqXx9EGb9/Vfh+MPyzZL2sj7F9wv/770d6HuufDqE/JwaC2Z8/c6G0dXslQZsdrqbX6+93NZb049ufTY0sLfFH9Syi/rSzk2UMqja296OTs6Wkb3e0oerhUqT/ftK7NOnrRixPZR6D9rk1Fa5veR5zpVlodrhcqLEyZIi8VSpL7jMuJkj5U9ZKO5jeTWs1uL1Ja1GS9dkodrhcqEEY/YvC9stPzwEPAIsLWI7Vhd7oNOia5KXbv3Xdu/NvMfmo+/mz8D1w9k+7ntBd5TZtiHACRO/cjW4Sn5WHt6Ld8f+p5eNXvxeM3HC3fTuX0wqwNkp8GA1VCrU95L0mIhcf4CTnXvQdbRo5Qf8yFV5s7FtXp1G70D67qrTz9aHaLXbCr0qodfzSsk/jCXuMmTixRHWbeyzO44m+q+1Rm8eTCbYjYVqT1ryjnyDwAu99zhslRrulV2v9MvYAvFbER+8t468my3pjbvJz/xGfHy8VWPywY/NJBrTq0p8PrYPu3l0Xo1pSnmkB2iKxm+WH/Mam0djj8sI+ZFyKfXPC1zTDmFu+n0dinHV5RyUpiUcSdueCnn/HkZ3X+APFwrVJ55/gWZc/Gi1WJ1hDv+9JOZIi3ftpDnu1STh2uFyrjp04scQ0p2inxy9ZMy/IdwuerkqiK3Zw1JX4+Vh2uFyuw/f7J5Xzh6Q5AQYqAQYq8QYm9cXJxN+zInXCIn2YKhZtG3DheFv5s/szrOIrxcOCO2jmD5ieX5Xl9m8NtYjBqSp6gNQoVlrbNWE7MSGbJ5CD6uPnze+nP0Wn3BN53cCPMfza0T/uxvUPaevJdS163PHYUfOED5MR8SPP079IGBVonVaRi8EU8upXwrF7zvgbjPvyBp0aIiNent4s30DtNpHNiYd7a/w8IjRXugag05J4+CRqKv47gaMQUmciHERiHEoVt83VH5MynldCllhJQyIiDgLmpU3IGsP9cCYGhovTrid8vLxYtpD06jWaVmjNoxikVHb/+L7HZfG9yr+pC46RAy5bIdoyzdjBYjb/7xJolZiUxuM5mybmULvunIaljYOzd5P7M279AHS2YmF0aN5tyQIbhUqULVFcvx69WrWK9GKawh7Wrc+U3eFRD9FlPx3mQ8q7lycew40rdtK1IcHnoPvnnwG9oEt+Gj3R/x7f5vi7yrtChyYs7i4iURPtYpc303CkzkUsoHpZT1bvH1sz0CvBtZ+/4EwNC8s4MjyWXQGfiqzVe0CW7DhL8m8MO/P9z22jIvDMJ0RUPqDDUqt5eJeyay5+IeRt0/irplC/FM5cgv8GN/qNgA+q8Gj9zEnxMdTXTvPiQvWYL/888RsmA+LlUcs2XbFu7600+FcMRjM6jU8AyugW6cGzqMrONFW2HkqnXli9Zf0L16d76J/IZxu8bd0WY8a8q5lIRLWQ8mbbTdnoaCFP81T3ch68ixqw86azs6lDwuWhc+b/05HUM68tnez5h5cOYtr/Ps8RQu/i4krtiCzMm0c5Slz/ITy1l0dBH96/Tn4eoPF3zD0V/hxwG5J8D0Ww5uvgCkbtjA6ccex3TxIsEzplPuzTetWlLW6dXuiqbjuwQ3iULoJLGDXsaUkFCkJnUaHWOaj+G5es+x9PhShm4eSoYxw0oBF440m8lJzMGlUjmbbk4rSJESuRCipxAiFrgf+FUIsc46YRVN1pl4DEE+jg7jJnqNno9bfsxDVR9i8t+TmXFgxk3XCI2GMk8+TlaCIGPxJw6IsvSIvBzJ2F1jub/C/QxtPLTgG46ugaX9oUID6LcMDN5Is5nLn3/BudcG41K1at7GHuUWWryOvmEngu+LxRR3mdjBQ244yehuaISGoY2H8t6977Ht3DaeXfcscRm2fQYH/98cZTp9GGkWuIQ49kyBIiVyKeUKKWWQlNJVShkopbTeoX53yRx/gZwUiVut4rm8S6fRMb7FeLpU68JX/3zFd/u/u+kan2ffQOsuSJj/I1jMDoiy5Lt45SLDtgyjgkcFJraaWPCpT1G/506nVKgPTy0Hgw/mtDTOvvwyCTNm4NurF1UWzEdf6eaCWMpVGg30+Ba3ahWo0DyHzH37uPzll1Zp+onQJ5jcZjKnUk7R+9fe/Bv/r1XavZ1ro++Ny3JnmD87lftw3Nqb0wqrxE2tZP25BgBDw+JbmF6n0TG++XgervYwUyOnMv3A9Bte17i5UaZ7O67EWMhaN8tBUZZcmaZMhmweQoYxg6/afIWPawGf3s7sgEVPQtlaV0fiPmSfPk10rye48ucOyo8eRYUxH6JRUykFc/OFJ+bjUykZ34beJM6aTdrv1jmIoXVwa+Z2notWaOn/W3/Wnl5rlXbzc5+8CMDot54BIPrjLkR/3MVqq6kKq8Ql8uwDufUYDPe1d3Akt3btX2qtRsvY5mN5uNrDTPlnyk1z5nnb9mdMAwc+kS9ppJSM+nMURxKO8MkDn3CP3z3533Dub1jQC3yC4KkV4ObHlb92E927D+bkZCrPnoVf7972Cb6kKF8PunxG4D1HMVT25/zIt8mJjbVK06FlQlnUZRF1/evy1ta3mLhnIkZz0aZvrrnV5qjIP/9BaCW66mFW6eNulbxEHhWFxkWireLAXVb5uP6ByLVkfm3OfPah2f9/zd8f37YNSTmSgXGvOqDZWmYenMna6LUMbjSY1sGt87847njuOnF3P3j6Z/AMIHnlSmKefx6dvz8hPy7Fo6njl7g6pQZ90YQ/RqXwY2A2cm7Y60WeL7/G382fmR1m0ie0D3MPz2XAbwM4n36+yO0Oa18zb8QNuaPve2QWLn4uCK327pZnWkmJS+Q5sXG4BhicoggR5Cbz8S3G0zmkM5P2TbphaaL/62MBSPxaPfS0hk1nNjHlnyk8VPUhnqv3XP4Xp8TCvJ6g0cFTK5HeFYmbMpULI9/GvXFjQhYvstt5mSWSEND1C1wqVaBCs2yyDh4k/ttpVmter9Xzzr3v8Fmrz4hKieLxXx5nffR6q7V/TU58Oi7lfQHrbU67G86R7QpLSrLjsnCtVIgNHXZUUL0KnUbHhJYT8pYmzjs8DwB9SHW8m1Qnee9lzCd3Oyz+kuBwwmHe3v42YWXD+LDZh/lv0MlIhHmPQHYq9PsJ6VOFix98QPzXX+PTsyeVp3+H1tvbfsGXVAYfeGw23gEX8WlQlvjvviMzMtKqXXQM6ciPXX8k2CuYN/54g7f+eIvkrOQitzukXQ1k6iWMqQKX4OCiB1pUt9q3b+svW9VaMcYcl4drhcr40S/YpH1ryK9eRY45Rw7bPEzWm1NPzj88X0opZeY/u3LrVLzayV4hljgX0y/Ktkvaygd/fFDGZcTlf3H2FSlntJNyTICUp7ZKc0aGjHnxJXm4Vqi89OWXRa7ip9zC1s+k6W0feaLFffJEhw7SnJ5u9S5yzDlyWuQ02WBuA9lqcSv52+nfivx3mb1zuTxcK1QmTv3QSlEWDEfXWrGHnH9yizC6hjr2wcPd0mv0fPLAJ7Sr3I6Pd3/MoqOLMDS4F4/agSRuPYXlwlFHh+h0MowZDN48mHRjOlPbTs1/+73ZBMueg9i98OhMzGXqE/Psc6T/8QflR31AuSFDSsRW+2Kn+VC01ZtSodEFjDFnufSJ9Y891Gv0vBj+Iou7LCbAPYA3/3iTF9a/wImku9/Ek3M499Qi17oR1grzrpWoRJ59OBIA14YtHBtIPgp6IKLX6Jn4wERaB7dmwl8TWHJ0CWWHvY05W0Py5HfsFGXJYLaYGbFtBEcTj/LJA59Qq0w+J9BLCWuH5x4K8dBETAH3c+bp/mQdOkSlL7/Er08f+wVe2mi00HMaHuVyKNPUh+SlS7myY8cNl1hrXXatMrVY1GUR7937HkeTjvL4L48zZueYu3oYmnM8d626vrZK5FaVHXUSoZPo7rHNYcvWUJgHInqtni9afUHroNaM+2scqwNTcAvxJWH9IWSidZZplXRSSj7d8ylbzm5hRJMRBa9Q2fY57J0NzYeSU/Ehovv1I+fsWYK/m4Z3xw42j9feG0iKnTLVoON4AoIP4xLoy4X3P8By5Urey9bc/q7T6Hgi9AlW91jN4zUfZ8XJFXRZ0YXRO0YTkxpT6Hayo06hcdWgK1f4IoBZpiybFPgqUYk85+wlXP1dnGbFSn70Wj2ft/6cVkGtGLtrLId7tcSUoSXl67cdHZpTmHd4HguPLqR/nf48WfvJ/C8+sBR+Hwthvciu3p8zfftiTkqmyuxZeDRrZpd4HVmno9hoPABN7Y5UCI/BeO4cl4t4GEVBfA2+vHvfu6x9ZC2P1XiMVVGr6LKiCy+sf4F10evyX3+elULWuXQMIeUKnG6TUhJ5OZIPd35Im6VtOBh/0MrvBArYl+xcsi9n4BHquFKS1uaideGL1l8wbMswRpxdw5zyLiT88hc+w+IRnsVrZU5xsi56HZ/t/Yz2VdrzesTr+V8cvR1+fgVCWpJVZygxT/cHi4Uq8+ZiqJXPVIxifULAw5NxP3svfg09SZw3n2dO+3DEPwQgb9XXkHY1rLrUr7xHed69711eqP8CK06sYNmJZbz5x5t46j25v+L9tKzUkvsr3k+ge2Be0pand5GdrMevdb1btplhzODvy3+z6/wu/oj9g+jUaNx0brSv0h4PvYfVYr9G2GKYX5CIiAi5d+9eq7ZpvhTD8VYdCXj0PsqO/96qbTtajjmHN7a8QdaG33l9pYVKz7fC+03rrbktSXZd2MWgjYOoX7Y+37X/DoPOcPuL447DrPbgWY6s5lOIGTQM4eJC5Tnf41rN9kWQJm04fsuRuLUTldOJXIT5x0Gc2lITjU8AVVeuoNoHG/I24tia2WJm54WdbDyzkW2x27icmXs2gI+rD6F+oYT4hBC8bw9Npxwn9vWexLWsR5Y5i+TsZM6knCE6NZqYtBhMFhN6jZ5G5RrxULWH6BjSschJXAixT0p506R8iRmR5/zzB+C8K1byc21kPly+wbmtG8hYvoVGLych3B14ancx9G/Cvwz5fQhVfaoypd2U/JP4lXhY8Bho9WQ2Gk/Mi0PQeHpQZc4cXCpXtku8w9rXzEvYISN/tVuiKvbCe6M9vJIKiX9ydnMaCTNnAvar667VaGlRqQUtKrVASsnRxKNExkVyLPEYxxKP8Vv0b0ScTKIp8EnGKi7t/gXInXsP9gomxDuEtpXb0iSwCQ0DG+Kmc7N5zCUmkf+zcQt+gGu4feY07U2v1TOx7edM3/UIbeefZMXn/XjkfbV1/5rolGhe3vgyfgY/pj04DW+XfDbsGLNgUR9Iv0TmfVOIGfIeWm9vqsz9QVUvLA6EgK5f4hlzL1613EmY9h1vj/zKQaEIavvXprb/dWcbWCycX1iTVDc35j63DoPegEFrwKAzoBGOeT7n/E8Fr8o8fhyhkehrN3Z0KDaj1+gZOOInUvy1eKw/xeRdnzj0iKviIjYtlufXPw/AtAenUc693O0vtlhg5SCI3U1m3XeJGTkRrY8PVebNdWgSd2SdjmLJuwJ0+oTA0FMILXTeOLf4/K7HHSU7TuJWPZiKXhUpYyiDu97dYUkcSlAiL5uWgksZPUJfiENznZhe70roSy9SOQ4OrZzLR7s/wiItjg7LYS5eucjz658ny5zF9PbTCfEJyf+GLRPg3+VkVnuZmPHz0Pr55SbxihXtEu/tlOo58dsJ740+rC0B9ZK5smMHqWvWODoiACynd5CVosctvJGjQ8nj1In8+homhtRsLhsMDinqbm8+T76MS1lXXthqZvGRhYzcNtJqpTqdSVxGHC+sf4GU7BS+e/C7/Df8AEQugq0TyQzoTsyUTblJfO4P6CuUnJVOJcrVKRa/WmYMFVy59NFHmFNTHR0V2Xv/AIvAENHc0aHkcepEfq2s5KkRjbFcEVRvFOqQou72JrRayr74Al6JgjFnQ1h7ei2v/f6a3c8rdKSLVy7yzLpnuJRxiW8f/LbgQ5Oj/4RVr5Hp1pSYH47mTqf8MEcl8eLONxjR8UPKh8ViTkgg7qspjo6IrIMHADCEFZ+FFU6dyK/JidwKCFxDC3ECegnh/eRLuAYYCPvlBKMbvMHOCzt5bt1zxGfGOzo0mzuXfo4Bvw0gITOB6e2n06Bcg/xvSIiCJX3JMgUTsywFradXbhK/i+mUkv5pr1hq/AxuDe/Fr2YOSQsXknX4sONiyUgk62wSGnfXYvVgvEQk8uxDuWvSXerf5+BI7EdotQQMG0pOqoa2y9czqfUkTiafpN+afpxKPuXo8GwmOiWaAb8NIDUnlRkdZhScxDMSYWEvspI0xKxzQePhSeUirE5ROzAdQKOBbl8RUD8DrUFw8cMxSIuDngvF7iEzSY9brWrFqoBaiUjkOaeiAIlL3dJxWsu1UaFnz6dxq+JN/Kp9tHatwvedvifTlEm/tf3Yc3GPg6O0voNxB3l67dNkm7KZ3XE29creelddHlMOLH2a7DPniNkSgHA1UGXO9+pACGfkXx3tQ+9RLiyezP37SVm+3CFhWE7tIDtZj6FR8ToTuEQkcuP5i+g8NWjc3B0dSpEV5qP7tVGhEIJyb4/ClKkh8aOh1Ctbj4VdFhLgFsDA9QNZdHRR8VmyVUR/nP2D59Y/h4feg3kPzSO0TAFH+UkJq4eRc3AHMX9WBp0LlefMwaXKnW8sKehgEMVO7n0Jnxb1cCtn5vLEzzAlJdk9hOyd60AKDPUb2r3v/JSIRJ5zORUXf9vvnrKHO/3o7t76ITzrBpKw6TimqH1U8qzE/Ifm06JSCyb8NYFRO0aRY86xUbS2J6VkydElDNmcu2Nz3kPzqOJdiGS8/Qtyti/izI6qSKGnyvezca1W9a5iuNVZjaXhoXqxo9EienxD+Yg0zGkpxE360r79J54m6+QZANzqFa/ncc6fyKUkJzkHffmSXUQqv1FhwPufYDEJEsbmFojycvFictvJvFj/RVacXEH/tf05m3bWkeHflRxzDh/u/JBxf42jeaXmfN/x+/wPhrjm3xUYV40nZntlpEVP5e9n41pDbbgpEQJqYug5nDL3pJP8449kHjhgv76PriYj3gWtrzc6B+87+C+n36JviT+LOVODSzF6gnyn/ls86VZV3gqqy+HbvBaJfx7Dd+tSXB/ohUZoeLXhq9QuU5v3/3yfXr/0YlSzUXQK6WSnd1U0lzMu8/qW19kft58Xwl7glQavoNVoC74x5i+M8wdxZlslzGY9lefMsmoVQ7UDsxhoNpiyHVaRevYiFz94n5BlyxHaQvxuFJH8dxVXLnvg8WCrYvWgEyjamZ3AROAocABYAfgW5j5rntmZ+ccyebhWqEyeMcFqbTpSfmd65neN8UKsPBoWKs+0ryctJuMNr8Wmxconf31S1ptTT76//X2Zmp1qtXht4fczv8uWi1rKJvObyHWn1xX+xoQoaRxVVZ68t7Y82qiRzIiMtF2QimNdPiaTnwuSh2uFyoT5C2zfX+oFeeUV/9xcs7rg/0dtBRud2bkBqCelrA8cB+x+6oHxeO5HK5ea9e3dtcPcalSoK1+JgL5duBJjIn3mBze8VsmzEnM6zeG5es/xc9TP9Py5J1tjt9or3ELLNGUydudYBm8eTKBHIIu7LKZDSCFP58lIxDTjMc6s1WPMdiV4xgzcwovvSVFKEQXUxPuZt3Avl03c559iirfx/omjq0k/bwCtBs8Wxe8oySIlcinleiml6eq3uwC7r+vKiY4CQF/H8efmWUNhPrrf7iGb37AJuPjruTRzOZbkSze8ptfoGdp4KPM6z8NT78krm15h5LaRXM64bJW4i2r7ue08uupRlh5fyoC6A1jw0AKq+RayJrgxC9OsXsSsSMOYZSD4u+9wb1R86mAotiHuf5nyD1dFZmVxacwHBd9QFEd+If2SN+6NGqP18bFtX3fBmg87nwXW3u5FIcRAIcReIcTeuLg4q3VqjD2HxgW0/vlUvHMiRVkJIVxcCBw5HGOaIHHMoFteUz+gPksfXsrA+gNZH72eriu68t3+78gyZd11v0Vx8cpFXt/yOoM2DkIrtMzqMIs3It7ARetSuAYsZkzznyFmQTQ5GQaCp03Do2np2E9Q6mm0uD43E/96OaSu30z6li226ScjEeOhP8lOkHi2bm2bPoqowEQuhNgohDh0i6/u113zLmACFtyuHSnldCllhJQyIiCg8IeVFiTnUhJ6P5fi9/DBQTwffgqvsEDi1x0m+69bV4tz0brwWsPX+Ln7zzSv2JypkVPpuqIrC44sINOUaZc44zPj+WT3J3RZ3oWtsVt5reFrLOu2jKYV7iAJS4l5+ZvEfLeHnHQDQd9Mw+P++20XtFL8lKmK/4hxuHgZufjucCwZNqg3dPw30s/lVlX1bNPa+u1bQZGPehNCDABeBNpJKQv1p2jNo96i7g3FtXIgQT/+YZX2SgJjzElOde2KIUBP5d/2IfT5j273XNzD1H+m8vflvyljKEPf2n3pcU+PvLrekzYct9qa6VPJp1h6fCnLji/DaDHSrXo3Xgx/kUqed77qyLzuE86Mnk5OmoGgb7/Ds2Xxm7tU7CPji96cmb6fMo91JHDcl9Zt/IeHiZl/khwRTPV16xw6aLzdUW9FmloRQnQC3gK6FTaJW5PMSMaYBi4Vy9u762JNX/keAl94jIxzJpI/fa3A65uUb8IPnX9gTqc51PavzZR/ptD+p/a8vPFlfjv9G5M3F22tbkJmAitPrqT/2v50/7k7S44toUNIB1b1WMWY5mPuKomb/pjOmVEzyEl1Jejrb1QSL+XcX5mBT20ticvWkblvp/UajtmF5cQ2Mi5q8Wzduth+8i/SiFwIcRJwBRKu/miXlPKlgu6z1ojcuH8zJ594mfIv9sBv2EdFbq8kkRYLMV3uJSs2jWo/zUNfq0mh741OiWZV1Cp+jvqZyxmXkVLQoFx97q1wL6FlQqnmU43KXpXRa28+xMNkMXEm9Qwnkk5wLOkYO8/v5N+EfwEI9grmsZqP0b16d/zd/O/6vZl2LSFmyLvkXHEh6Ouv8WzV5q7bUkoO89HtnOrzHFoPV0LW70Dj7ln0Ruf1JO2vg8Ru1FN59iw8mjn2KEmbHL4spbynKPcXVc7RSABcahSv7bLFgdBoqPD5N5zq9RQXhr1E8Kq/ELrC/XWH+IRgTuhE1L6qaN3OoPU4wd9ZJ4i8PAMhcv/h1wgNHnoPPPQeuOncyDJlkZ6TTroxHUnuNVqhJaxsGK81fI0WlVoQWia0yMdhmf5ew5kh72G8oif4m6l4PKCSuJJLG9qCCkOf4uxHC4gb1ovA74p4otDZPRD1O2lZbdC4n8U9oviujHPqnZ3G08cASvQ5nUXhUqcJ5fp14NKcDSSOfRH/D2cV+t7/7iQ9/uZkMowZRKdGE5UcxZnUM6TlpHHFeIUMUwZuOje8XbzxcvEi2CuYGn41qOpTFVetq9XeT86eNcS8PAxTlo7gb6bg8UBbq7WtlAye/d/Dd9suEv84iefcD/B4eszdN/bHxxjxJ3X3aXy6d0O4FHIllQM4dSLPiYkBIdFXUdumb8fvrS/J2NWMyz/+iXvr5bi1eeSu23LXu1PHvw51/OtYMcLCyd71KzEvv47FpKHKtKm4NW9n9xgU5xD45SKudGjBhcmLqRrWCG3DHnfeSOw+OLmRxMROSNMh/J9/3upxWpNTF80yXohD76Mr9JRBaSQ0GipMW4zOXXBuxHuYEy4VfNN/OLq+SOa2Xzjz4htIqaHKrGkqiSv50nh6UWnytxgztJwf/jry0Io7a8CUDevewST8SNp2Au+HHrqr8sf25NSJPCc+HZeyXo4Oo9jTlg+h0ujhGNMsXHip1x3XKHdkudb0n+cSM2g4Gj2EzJmOoUkrh8WiOA+3Js0IfHMo6bGuxI8aDAd+LNyNFguseBHO7iIppxMyMxP/gS/YNlgrcN5EbsrBmGJBX8F6m4tKMveuz1KuewPSDl4m/u3nHB1OoSTPmsTZkRPQ+wiqLJyPS3hLR4ekOBG/Zwfi0+Nh4v/1JPXL12DLJ7mnRt2OlLD+Xfh3Bebm75K44R88H2yHoWbxrzvvtIncHHsEc44Gl8qVHR2K0ygzfj4+DfyIX7mTlJmfOjqc25IWC3HjRnBh4nQ8KmmosnQ5+prFd8WAUjwJISg/ZhxuDcI5v8efzGUTYVoLOHOLdeZXEmDjKNj1Ddz7EklR3lhSUyn7YoGrqYsFp51czjnyNwD6atarNV3SCa2OCjN+wdijFee/mI2+aiju7bo5OqwbWK5c4fwrA0jbdQifUC0VZq5GlA1xdFiKk9K4uBA0dSrRT/TmzDYdQboreH7fCSo0gPL1oFwdiN0LR1eDOQfq9ybNpQNxU4fi2bYtbmEFnAtbTDjtiNwYdRgAl9DidXZecSe8/AmauQAXT8nZ10eQuXubo0PKkxN7jugenUj76yDlWvtQYf4WlcSVItOVLUuVRQtxrXYPZ9cLkvSPg8EHjq2Fde9A1O8Q8SwM2kF64NOcGzoMQ+3aVPz0E0eHXmhOm8hzzpwGQF+rgWMDcULaquFUnvIpWr2ZmOcHkrHT8XVq0tav4/TDnTFevEzwk9Xx/+p3hGfJPr5PsR99uXJUmTcXj2bNuDjvT2L/qc6VVouRbxyHN45hafMhaQcvEPvKq7hUr07lGdPRelphZ6idFLlo1t2wxhb9C0+1Iu3AZWruP2KlqEof465lnHntbcxZWoK/+Rr3lvbfYGPJyeHy+LEkLfkJg18OlV7qgEu/KaB12lk/pRiTRiNxU78mafFiLCkpuFSvjsbdnawjR8BkwqV6darM/QGd/92XkLAlmxTNciRTQgo6X+vtGiyN9Pc9SpWpH6FzMxHz0iskL5xj1/4zDx4kukdXkpb8RJnQTKpMfh+X/t+qJK7ctUkbjuf7utDrKTdsKDX+2EKFCRPQ+vmiMRjwf+YZgr75mqpLlxTbJJ4fp/0/xpichT5QLT0sKv29j1Lla8G5N0dwYcwnZEbuJXDcF2hsuB3ZkpFB3JdfkjhvPjqDiaBOOryGL4dK6lQfpWgmbzpRqH0PGoMB30d64vtITztEZXvOOSI35WBMl+gDnO9fzuJI1+QRKi/4iTLhguRVmzjT86Hcj5pWJk0mkpctI6pTBxLnzsO3ejrVXr8XrwnbVRJXlCJwyhG55VIUlhwNuooVHR1KiSGCwgmcvRW3MY9w8bcYTj/yCL6PdCPgzZHo/PyK1LY0mUjbuIm4yV+SczoaQxkjlTpbcH9uEtR71ErvQCmtJm04zuRNJ/K+Dxn5K5BbWsKRu5LtySkTuTHqEAD64BDHBlLSeJTFe/xGPFpPJG7aDJKW/0zq2nX4PPIofk/0xrXGndVcMV68SPJPy0heugTT5ThcvC1UapGC18OPI9q9D54l45xVxbH+W6kz+uMuDo7I/pwzkV8rXxuiNgNZnc4Fbed3Kd+0N77fv0L8uqMkLVxA0vyFuIXVwb1ZS9zC62OoWw+dn29eaU9LRgbG8+fJiTlLxu7dXNmxg+zjx0GARwUj5Vum4tmyBaLjOAi0f/VERSnJnDORx54BQF/dOXZdOSX/6hjeWEtQz82YNk0l5fddpJ7ZT8L0w3D9ilWtFo2rC5aM/x/aLHQCtwAzAeHpeIdIXJo/Bk1fgPJh9n8fSqni6EqdjuKUidx04QIg0VVSdVZsSgio3hZd9bb4PxqF//5FWE5sJevQQbISBBajwGISSLNAa7Cg9zCj9zBhqOSHpmZrqN4GanYC9zKOfidKKVFa5sT/yykTufFyAjpPLUJ/85mRio34V4e276FpC+7GTNwvHoSMBMhKgew0cPMD70rgXRF8K+f+I6Aoil04ZyJPTEPv6+boMEovvRsEN3V0FIqiXOWU68hNKUZ0/j6ODkNRFKVYcLpELjOTMWYItatTURTlKqdL5OaYI0izQF8pyNGhKIqiFAtOl8iNp/4FQFe5uoMjURRFKR6cLpGbzpwEQF+1toMjURRFKR6cLpEbY2MA0FdTuwMVRVGgiIlcCDFWCHFACBEphFgvhLB5FSvjpcsILWjLqtNjFEVRoOgj8olSyvpSygbAauCDooeUP1NcMjovHULjdB8mFEVRbKJI2VBKmXrdtx7cWIXDJoxJV9D7edi6G0VRFKdR5GGtEGK8EOIs0Jd8RuRCiIFCiL1CiL1xcXF315nFjDHNjL5c0epjK45X0JFciqIUXoGJXAixUQhx6BZf3QGklO9KKYOBBcCrt2tHSjldShkhpYwICLi7zTwy6RymTA268uXv6n6l+Lj+IABFUYqmwForUsoHC9nWAmANMKpIEeXDdPoQSIG+UhVbdaEoiuJ0ilQ0SwhRQ0p5bWjVHTha9JBuzxid27y+yj227EaxEXUkl6LYRlGrH34shKgFWIAzwEtFD+n2jDGnAdBVr2vLbhQbUUdyKYptFCmRSyntenKu6fw5APQhpfMUEEVRlFtxqsXYRm0QGjdXtJ6ejg5FKaLSeiSXotiCUyVy7179CXzvfUeHUeLZY2mgmhNXFOtxqkTu3rgxvo/adTanVFJLAxXFuThVIlcURVFu5pRndirWp5YGKorzElLavDzKTSIiIuTevXvt3q9SOGppoKIUT0KIfVLKiP/+XE2tKIqiODmVyJWbqKWBiuJcVCJXbqLmxBXFuahEriiK4uRUIlcURXFyKpEriqI4OZXIFUVRnJxK5IqiKE7OIRuChBBx5NYvdzZlgXhHB2FHpe39gnrPpYWzvucqUsqbzsp0SCJ3VkKIvbfaVVVSlbb3C+o9lxYl7T2rqRVFURQnpxK5oiiKk1OJ/M5Md3QAdlba3i+o91xalKj3rObIFUVRnJwakSuKojg5lcgVRVGcnErkd0EI8YYQQgohyjo6FlsTQkwUQhwVQhwQQqwQQvg6OiZbEUJ0EkIcE0KcFEKMdHQ8tiaECBZCbBZCHBZC/CuEGOLomOxBCKEVQvwjhFjt6FisRSXyOySECAY6ADGOjsVONgD1pJT1gePA2w6OxyaEEFrga6AzUAfoI4So49iobM4EvCGlrAPcB7xSCt4zwBDgiKODsCaVyO/cJOAtoFQ8JZZSrpdSmq5+uwsIcmQ8NtQUOCmlPCWlzAEWA90dHJNNSSkvSCn/vvrfaeQmt0qOjcq2hBBBQBdgpqNjsSaVyO+AEKI7cE5Kud/RsTjIs8BaRwdhI5WAs9d9H0sJT2rXE0KEAA2Bvxwciq19Se5AzOLgOKxK5+gAihshxEag/C1eehd4h9xplRIlv/cspfz56jXvkvtRfIE9Y1NsTwjhCSwDhkopUx0dj60IIboCl6WU+4QQrR0cjlWpRP4fUsoHb/VzIUQYUBXYL4SA3CmGv4UQTaWUF+0YotXd7j1fI4QYAHQF2smSu/HgHBB83fdBV39Wogkh9OQm8QVSyuWOjsfGmgPdhBAPAQbAWwgxX0rZz8FxFZnaEHSXhBDRQISU0hkrqBWaEKIT8AXQSkoZ5+h4bEUIoSP3YW47chP4HuBJKeW/Dg3MhkTuiOQHIFFKOdTB4djV1RH5m1LKrg4OxSrUHLlSkKmAF7BBCBEphJjm6IBs4eoD3VeBdeQ+9FtakpP4Vc2Bp4C2V/9uI6+OVhUno0bkiqIoTk6NyBVFUZycSuSKoihOTiVyRVEUJ6cSuaIoipNTiVxRFMXJqUSuKIri5FQiVxRFcXL/A9rV/mnzQWPHAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# get the MAP estimate\n",
        "K = 8 # polynomial degree   \n",
        "\n",
        "# feature matrix\n",
        "Phi = poly_features(X, K)\n",
        "\n",
        "theta_map = map_estimate_poly(Phi, y, sigma, alpha)\n",
        "\n",
        "#@ Test\n",
        "assert any(theta_map), \"map_estimate_poly not implemented\"\n",
        "\n",
        "# maximum likelihood estimate\n",
        "theta_ml = nonlinear_features_maximum_likelihood(Phi, y)\n",
        "\n",
        "Xtest = np.linspace(-5,5,100).reshape(-1,1)\n",
        "ytest = g(Xtest, sigma)\n",
        "\n",
        "Phi_test = poly_features(Xtest, K)\n",
        "y_pred_map = Phi_test @ theta_map\n",
        "\n",
        "y_pred_mle = Phi_test @ theta_ml\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(X, y, '+')\n",
        "plt.plot(Xtest, y_pred_map)\n",
        "plt.plot(Xtest, g(Xtest, 0))\n",
        "plt.plot(Xtest, y_pred_mle)\n",
        "\n",
        "plt.legend([\"data\", \"map prediction\", \"ground truth function\", \"maximum likelihood\"]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "D0xjFzhqjmsR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b52964b2-ae5c-4ffd-b1d4-77a0e7fd33f2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-1.49712990e+00 -1.39320006e+00]\n",
            " [ 8.56868912e-01  1.00079003e+00]\n",
            " [-1.28335730e-01 -1.32425656e-01]\n",
            " [-7.75319509e-02 -1.17294894e-01]\n",
            " [ 3.56425467e-02  3.13966316e-02]\n",
            " [-4.11626749e-03 -9.30590932e-04]\n",
            " [-2.48817783e-03 -2.03609310e-03]\n",
            " [ 2.70146690e-04  1.93616129e-04]\n",
            " [ 5.35996050e-05  4.16800598e-05]]\n"
          ]
        }
      ],
      "source": [
        "print(np.hstack([theta_ml, theta_map]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBiBGHpJjmsS"
      },
      "source": [
        "Now, let us compute the RMSE for different polynomial degrees and see whether the MAP estimate addresses the overfitting issue we encountered with the maximum likelihood estimate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "_xuBNr4AjmsS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 355
        },
        "outputId": "9bc10773-6562-4ae8-ba44-bad092949371"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-118-fd7ecd4ba848>:13: LinAlgWarning: Ill-conditioned matrix (rcond=3.34036e-18): result may not be accurate.\n",
            "  theta_map = scipy.linalg.solve(PP, Phi.T @ y)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f7275ec8cd0>"
            ]
          },
          "metadata": {},
          "execution_count": 123
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEGCAYAAAC6i5gfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAABBa0lEQVR4nO3dd3xW9dn48c+VHQKEvRL23suBA8VZREDqpnXiaJ3V5+nTR22rtrWtv+pT2zpQRItWxQkWFFdVZIgKQTYGSEggrJAJSci8r98f5yTchOzcI+N6v1736z73Oec+3+9h5Mp3nOsrqooxxhgTSCHBroAxxpjWx4KPMcaYgLPgY4wxJuAs+BhjjAk4Cz7GGGMCLizYFWguunTpov369Qt2NYwxptlISEjIUNWuVR2z4FNH/fr1Y926dcGuhjHGNBsiklrdMet2M8YYE3AWfIwxxgScBR9jjDEBZ2M+jVBSUkJaWhqFhYXBroppYqKiooiPjyc8PDzYVTGmSbLg0whpaWm0a9eOfv36ISLBro5pIlSVzMxM0tLS6N+/f7CrY0yTZN1ujVBYWEjnzp0t8JgTiAidO3e2FrExNbDg00gWeExV7N+FMTWz4GOMMaZqOz+DlX8Fj8fnl7bg08yJCNddd13F59LSUrp27cr06dMbdL0lS5bw+OOP+6p69TZlypSKh3mnTZtGTk4OKSkpjBo1qs7XePTRR3nyyScBePjhh/nPf/4DOA8KZ2Rk+LzO/rquMUG37X34Zi6E+D5U2ISDZi4mJoYtW7Zw7NgxoqOj+eyzz4iLi2vw9WbOnMnMmTN9WMOGW7ZsGQA5OTkNvsbvf/97H9XGmFbo8A7oOtQvl7aWTwswbdo0PvzwQwAWLlzI7NmzK4599913nHHGGYwfP54zzzyTxMREAJ566inmzJkDwObNmxk1ahQFBQUsWLCAu+++G4CbbrqJO+64g0mTJjFgwACWL1/OnDlzGD58ODfddFNFGW3btq3YfvfddyuO1fX71amqRZGcnMz48eNZu3YtSUlJTJ06lYkTJzJ58mR++OGHk65x00038e6771Z8fvrpp5kwYQKjR4+uOD8rK4tZs2YxZswYJk2axKZNm2rcn5mZycUXX8zIkSO59dZbsdWATYukChk7oMtgv1y+Vbd8RCQGeA4oBpar6usNvdbvlm5l2/4jPqsbwIhe7Xlkxshaz7v22mv5/e9/z/Tp09m0aRNz5sxh5cqVAAwbNoyVK1cSFhbGf/7zHx566CHee+89fvGLXzBlyhQWL17MH//4R1544QXatGlz0rWzs7NZs2YNS5YsYebMmaxevZr58+dz6qmnsmHDBsaNG1dj3Rr7fW+JiYlce+21LFiwgLFjx3LBBRfw/PPPM3jwYL799lvuvPNOvvjiixqv0aVLF9avX89zzz3Hk08+yfz583nkkUcYP34877//Pl988QU33HADGzZsqHb/7373O84++2wefvhhPvzwQ1566aU634MxzUb+YSjMgS7+afn4PfiIyMvAdCBdVUdVOjYUeMtr1wDgYVX9my/LEpGpwN+BUGC+qpYPalwOvKuqS0XkLaDBwSeYxowZQ0pKCgsXLmTatGknHMvNzeXGG29k586diAglJSUAhISEsGDBAsaMGcPPfvYzzjrrrCqvPWPGDESE0aNH0717d0aPHg3AyJEjSUlJqTV4NPb75Q4fPsxll13GokWLGDFiBHl5eXz99ddcddVVFecUFRXVep3LL78cgIkTJ7Jo0SIAVq1axXvvvQfA+eefT2ZmJkeOHKl2/4oVKyq+e+mll9KxY8c63YMxzcphp5ekObd8FgDPAK9WPqCqicA4ABEJBfYBiyufJyLdgGOqetRr3yBV3VVbWe51nwUuAtKAtSKyRFW3AfHAZvfUsgbdnasuLRR/mjlzJr/85S9Zvnw5mZmZFft/+9vfct5557F48WJSUlKYMmVKxbGdO3fStm1b9u/fX+11IyMjASdYlW+Xfy4tLQVOnFZc+dmWuny/LmJjY+nTpw+rVq1ixIgReDweOnTowIYNG+p8De/6hIaG1qt8Y1qdjB3Oe3Md81HVFUBWHU69AEhS1apScJ8LvC8ikQAichvwdB3LOg3YparJqloMvAlc5h5LwwlAUM2fhYjMEJF5ubm5dbiF4JkzZw6PPPJIRcuiXG5ubsUEhAULFpyw/95772XFihVkZmaeMC5SX927d2f79u14PB4WLz7pdwefiIiIYPHixbz66qu88cYbtG/fnv79+/POO+8ATlaBjRs3NujakydP5vXXnUbv8uXL6dKlC+3bt692/znnnMMbb7wBwEcffUR2drYP7tCYJiZjB4THQPuGT2CqSVOacHAtsLCqA6r6DvAJ8JaI/BSYA1xV1blViAP2en1Oc/cBLAKuEJG5wNJqyl6qqrfHxsbWsbjgiI+P59577z1p/69+9SsefPBBxo8ff8Jv+vfffz933XUXQ4YM4aWXXuKBBx4gPT29QWU//vjjTJ8+nTPPPJOePXs2+B5qExMTwwcffMBTTz3FkiVLeP3113nppZcYO3YsI0eO5N///neDrvvoo4+SkJDAmDFjeOCBB3jllVdq3P/II4+wYsUKRo4cyaJFi+jTp4/P7tGYJqN8soGfHpiWQMzUEZF+wAeVx3y8jkcA+4GRqnqohuu8CUwDBqrq4bqUJSJXAlNV9Vb38/XA6ap6d33u4ZRTTtHKi8lt376d4cOH1+cyphWxfx+mWfvrSOh3Flw+r8GXEJEEVT2lqmNNpeVzCbC+lsAzGRiFMyb0SD2uvQ/o7fU53t1njDGmKkV5cCQNugzxWxFNJfjMppouNwARGQ/MwxmruRnoLCKP1fHaa4HBItLfbWFdCyxpZH2NMablytzpvDfn4CMiC4E1wFARSRORW9z9y0Skl/uszUU44y/VaQNcrapJquoBbgBOmphQVVmqWgrcjTNmtB14W1W3+vIejTGmRTns35luEICp1qo6u5r93g+kdK7lGqsrfS4BXqxHWcuAZbVW1hhjDGQkgoRCR/+tR9VUut2MMcY0FRk7oNMACIvwWxEWfIwxxpzIjwlFy1nwaebquqTCrFmzmDRp0gn7Hn30UeLi4hg3bhyjRo1iyRKbh2FMq1dWAllJfkurU86CTzPnvaQCUOWSCjk5OSQkJJCbm0tycvIJx+6//342bNjAO++8w5w5c/D4YdEoY0wzkp0CnlK/JRQtZ8GnBahpSQWARYsWMWPGDK699lrefPPNKq8xfPhwwsLCbFE0Y1q78oSiXf03zRpa+ZIKPvXRA3Bwc+3n1UeP0XBJ7auK1rSkAjgB6eGHH6Z79+5cccUVPPTQQydd49tvvyUkJISuXbv69BaMMc1MeULRzv7tdrPg0wLUtKTCoUOH2LlzJ2effTYiQnh4OFu2bKlYlvqpp57itddeo127drz11lsnZKg2xrRCGTugXS+Iau/XYiz4+EodWij+VN2SCm+//TbZ2dn07+/M1z9y5AgLFy7kj3/8I+CM+fzyl78MSp2NMU3Q4US/d7mBjfm0GNUtqbBw4UI+/vhjUlJSSElJISEhodpxH2NMK6cKGTv9mlannAWfFqKqJRVSUlJITU09YYp1//79iY2N5dtvvw10FY0xTd3RA1B8NCDBx7rdmrm8vLyT9k2ZMqVixdJ9+05O4L1+/XoATj/9dL/WzRjTzFQsnW0tH2OMMYGS4Waz9nN2A7DgY4wxplxGIkTGQtvufi/Kgk8jBWIlWNP82L8L0ywdTvTr0tneLPg0QlRUFJmZmfaDxpxAVcnMzCQqKirYVTGmfjJ2BqTLDVr5hAN3IbvngGJguaq+Xp/vx8fHk5aWxuHDh/1SP9N8RUVFER8fH+xqGFN3hbmQdzAgkw3Az8FHRF4GpgPpqjqqmnM6APOBUYACc1R1jS/LE5GpwN+BUGC+qpY/EXo58K6qLhWRt4B6BZ/w8PCKhzeNMaZZy/D/0tne/N3ttgCYWss5fwc+VtVhwFicpa4riEg3EWlXad+gupYnIqHAs8AlwAhgtoiMcA/HA3vd7bJa6mmMMS1XRULRwHS7+TX4qOoKIKu64yISC5wDvOSeX6yqOZVOOxd4X0Qi3e/cBjxdj/JOA3aparKqFgNvApe5x9JwAhBU82chIjNEZF5ubm51t2GMMc1fRiKERkCHvgEpLtgTDvoDh4F/isj3IjLfHYepoKrvAJ8Ab4nIT4E5wFX1KCOO460bcAJO+YI3i4ArRGQusLSqL6vqUlW9PTY2th5FGmNMM5OxEzoNhNDATAUIdvAJAyYAc1V1PJAPPFD5JFX9C1AIzAVmqurJj/U3gKrmq+rNqnpHfScbGGNMixKghKLlgh180oA0VS1PNPYuTjA6gYhMxpmQsBh4pJ5l7AN6e32Od/cZY4wBKC2C7N0Bm2wAQQ4+qnoQ2Csi5SNcFwDbvM8RkfHAPJxxmpuBziLyWD2KWQsMFpH+IhIBXAssaXTljTGmpchKBvX4felsb34NPiKyEFgDDBWRNBG5xd2/TER6uafdA7wuIpuAccCfKl2mDXC1qiapqge4AUita3mqWgrcjTNutB14W1W3+vRGjTGmOQvQ0tne/DqypKqzq9k/zWt7A3BKDddYXelzCfBiPctbBiyrvcbGGNMKlT/j07m6p1h8L9hjPsYYY4ItIxFi+0DECZONST9aSFZ+sV+KtOBjjDGtXTUz3eZ9lcyZj39OSZnH50Va8DHGmNbM44HMXVXOdFuXms2YuA6Eh/o+VFjwMcaY1uxIGpQUnBR8CkvK2Lo/l4n9OvqlWAs+xhjTmh3e4bxXCj4b9+ZQUqac0teCjzHGGF/LcINPpYSi61KzAZhowccYY4zPZSRCdCeI6XLC7oTUbAZ1a0uHNhF+KdaCjzHGtGaHd5zU5ebxKAmp2X7rcgMLPsYY07pl7DhpmnXS4Txyj5UwwYKPMcYYnyvIgoKMk3K6lY/3WMvHGGOM72VUPdNtXUo2nWMi6N8lpoov+YYFH2OMaa2qSSiakJrFhL4dERG/FW3BxxhjWquMHRAW5eR1cx0+WkRKZoFfu9zAgo8xxrReGTug82AIOR4KEsrHe/yU2aCcBR9jjGmtqkgompCaRURYCKPiYv1atAUfY4xpjUqOQc6ekycbpGYzJi6WyLBQvxbfqoOPiMSIyCsi8qKI/DTY9THGmIDJ3AXoCcGnsKSMLfv8l0zUm7+X0X5ZRNJFZEsN56SIyGYR2SAi6/xRnohMFZFEEdklIg94HboceFdVbwNmNqZsY4xpVipmuh1/xmdTWq6bTLST34v3d8tnATC1Duedp6rjVPWk5bRFpJuItKu0r7q1Xk8qT0RCgWeBS4ARwGwRGeEejgf2uttldainMca0DBk7QUKg08CKXetSswD/JRP15tfgo6orgKxGXuZc4H0RiQQQkduAp+tR3mnALlVNVtVi4E3gMvdYGk4Agmr+LERkhojMy83NbdxdGGNMU5KRCB36QnhUxa6ElGwGdI2hU4x/kol6awpjPgp8KiIJInL7SQdV3wE+Ad5yx2XmAFfV4/pxHG/dgBNw4tztRcAVIjIXWFpl5VSXqurtsbH+nflhjDEBdXjHCV1uHo+SsMe/yUS9hQWklJqdrar7RKQb8JmI/OC2YCqo6l9E5E1gLjBQVfN8UbCq5gM3++JaxhjTbHjKnAkHg86v2JWckUdOQUlAxnugCbR8VHWf+54OLMbpJjuBiEwGRrnHH6lnEfuA3l6f4919xhjTOuWkQlnRCQlF16W4i8cFYKYbBDn4uFOd25VvAxcDlWeqjQfm4YzT3Ax0FpHH6lHMWmCwiPQXkQjgWmCJL+pvjDHNUhVLZ69LzaZTTAQD/JhM1Ju/p1ovBNYAQ0UkTURucfcvE5FeQHdglYhsBL4DPlTVjytdpg1wtaomqaoHuAFIrWt5qloK3I0zbrQdeFtVt/r+bo0xppmoWDr7ePBJSM1mQh//JhP15tcxH1WdXc3+aV4fx9ZyjdWVPpcAL9azvGXAshora4wxrUVGIsR0g2iniy0jr4jdGflcc2rvWr7oO0Ef8zHGGBNglZbOTgjA4nGVWfAxxpjWRPWkpbMTUrOJCPV/MlFvFnyMMaY1yT8MhTmVZrplMTo+lqhw/yYT9WbBxxhjWpOKpbMHA+XJRI8EtMsNLPgYY0zrUimh6OZ9uRSXeQKSz82bBR9jjGlNMnZAeAy0d7KMVTxcasHHGGOM32TscLrc3Od5ElKzGNAlhs5tIwNaDQs+xhjTmnglFFVVElKzA97qAQs+xhjTehTlwZG0iskGSYfzyS4o4ZQA5XPzZsHHGGNai8ydzrs7zTqhYvG4wGSy9mbBxxhjWovyhKJut9u6lGw6tglnYNfAJBP1ZsHHGGNai4xEkFDo2B+gYrwnUMlEvVnwMcaY1iJjB3QaAGERZOYVkZyRH5QuN7DgY4wxrYfXTLeKZKJBmGwAFnyMMaZ1KCuBrOSKmW7lyURHBzCZqDcLPsYY0xpkp4CnpGKm27rUbEbFtQ9oMlFvFnyMMaY1qMjpNoTCkjI2p+VySr/gjPeABR9jjGkdyrNZdx7MliAlE/Xm12W0mzoRiQGeA4qB5ar6epCrZIwx/pGxA9r1gqj2rEtNAgKfTNSbX1s+IvKyiKSLyJZazgsVke9F5AN/lCciU0UkUUR2icgDXocuB95V1duAmY0p2xhjmrTDiRWTDdalZNO/SwxdApxM1Ju/u90WAFPrcN4vgO1VHRCRbiLSrtK+QXUtT0RCgWeBS4ARwGwRGeEejgf2uttldainMcY0P6qQsRO6DkVVWb8nOMlEvfk1+KjqCiCrpnNEJB64FJhfzSnnAu+LSKR7/m3A0/Uo7zRgl6omq2ox8CZwmXssDScAQTV/FiIyQ0Tm5ebm1nQbxhjTdB09AMVHocsQkjPyycovDvjKpZU1hQkHfwN+BXiqOqiq7wCfAG+JyE+BOcBV9bh+HMdbN+AEnDh3exFwhYjMBZZWU/5SVb09NjY4c+GNMabRKpbOHkJCSnAfLi0X1AkHIjIdSFfVBBGZUt15qvoXEXkTmAsMVNU8X5SvqvnAzb64ljHGNFleCUXXrT9EhzbhDOjSNqhVqrHlIyLne233r3Tsch+UfxYwU0RScLrDzheR16qox2RgFLAYeKSeZewDent9jnf3GWNM65CRCJGx0LY761KzmdinIyEhgU8m6q22brcnvbbfq3TsN40tXFUfVNV4Ve0HXAt8oarXeZ8jIuOBeTjjNDcDnUXksXoUsxYYLCL9RSTCLWdJY+tujDHNhrt0dlZBCcmH85kY5C43qD34SDXbVX0++csiC4E1wFARSRORW9z9y0SkVx3r2Aa4WlWTVNUD3ACk1rU8VS0F7sYZN9oOvK2qW+tYtjHGNH9uQtGKZKJBymTtrbYxH61mu6rPJ39ZdXY1+6dVsW85sLyK/asrfS4BXqxnecuAZbXV1xhjWpzCXMg7CF0Gsy41i/BQYUx88CdQ1RZ8BojIEpxWTvk27uf+1X/NGGNMk5BxfOnshM3ZjIqLDVoyUW+1BZ/LvLafrHSs8mdjjDFNjZtQtLjTIDbtS+LGM/oGuUKOGoOPqn7l/VlEwnFmne1T1XR/VswYY4wPZCRCaASb8ztQXOoJ2sqlldU21fp5ERnpbscCG4FXge9FpMrxFWOMMU1Ixk7oNJB1e44CwU0m6q222W6TvWaG3QzsUNXRwEScrATGGGOassOJ0HUI61Kz6de5DV3bBS+ZqLfagk+x1/ZFwPsAqnrQXxUyxhjjI6VFkJ2Cdh7M+tTsJtPlBrUHnxwRme4+6HkW8DGAiIQB0f6unDHGmEbISgYtIz2qH5n5xUHP5+atttluPwP+AfQA7vNq8VwAfOjPihljjGkkd6bbxsLuQEnQM1l7q2222w6qWI9HVT/ByRhgjDGmqXKf8VmRGUtsdC4DuwY3mai3GoOPiPyjpuOqeq9vq2OMMcZnMhIhtg9r9h5jYt/gJxP1Vlu328+BLcDbwH7qkM/NGGNME3E4kZKOA0n6IZ/LJ8TXfn4A1RZ8euIs3HYNUAq8Bbyrqjl+rpcxxpjG8HggcxcH+k0AaFLjPVDLbDdVzVTV51X1PJznfDoA20Tk+kBUzhhjTAMdSYOSAraW9iA8VBjbu0Owa3SCOq1kKiITgNk4z/p8BCT4s1LGGGMayV29dE1uJ0b2ahrJRL3VNuHg98ClOOvgvAk86K6PY4wxpinLcILPJ+kdmDGpaXW5Qe0tn98Au4Gx7utPIgLOxANV1TH+rZ4xxpgGyUikNLIjhwrbNqmHS8vVFnxszR5jjGmOMnaSEdUXcmlSaXXK1faQaXXLVYfgjAFVedwYY0yQHU5kZ+hp9G1CyUS91bakQnsReVBEnhGRi8VxD5AMXB2YKhpjjKmXgiwoyGBtXtcms4RCZbV1u/0LyAbWALcCD+GM98xS1Q3+rZoxxpgGcScbbCrqxsVNsMsNag8+A9z1exCR+cABoI+qFvq9ZgEgIjHAczhLRyxX1deDXCVjjGk8N6HoLo3joSY42QBqX1KhpHxDVcuAtPoEHhF5WUTSRWRLNcejROQ7EdkoIltF5Hd1vXZ9yhORqSKSKCK7ROQBr0OX42RsuA2Y2ZiyjTGmycjYQYlEkBfZg0FNKJmot9qCz1gROeK+jgJjyrdF5Egdrr+AKrJieykCzlfVscA4YKqITPI+QUS6iUi7SvsG1bU8EQkFngUuAUYAs0VkhHs4HtjrbpfVci/GGNM8ZOwgVeIY37dzk0om6q229DqhqtrefbVT1TCv7fa1XVxVVwBZNRxXVc1zP4a7L6102rnA+yISCSAitwFP16O804BdqpqsqsU4D8te5h5LwwlAUM2fhYjMEJF5ubm51d2GMcY0KWXpiWwr6cEp/ZrmeA/U3vLxOxEJFZENQDrwmap+631cVd/BWTvoLRH5KTAHJ9lpXcVxvHUDTsCJc7cXAVeIyFxgaVVfVtWlqnp7bGxsPYo0xpggKTlGSO4ekjy9muxMN6hjbjd/cseSxolIB2CxiIxS1S2VzvmLiLwJzAUGerWWGlt2Pk7CVGOMaRkydyEouyWOn8d3CHZtqhX0lk85d5mGL6lijEhEJgOjgMXAI/W89D6gt9fneHefMca0PO5MN7oOJTqiaSUT9RbU4CMiXd0WDyISjZM1+4dK54wH5uGM09wMdBaRx+pRzFpgsIj0F5EI4FpgiQ+qb4wxTU5Z+g7KVOjRf2Swq1IjvwYfEVmI84DqUBFJE5Fb3P3LRKQXzmJ1X4rIJpwg8ZmqflDpMm2Aq1U1SVU9wA1Uk9anqvLcLNx344wbbQfeVtWtvr9bY4wJviNpW9ir3Rjfv0ewq1Ijv475qOrsavZPczf3A+NrucbqSp9LgBfrWd4yYFlt9TXGmObOk76DJG3akw2gCY35GGOMaSRPGe0LUkiP7EO39lHBrk2NLPgYY0wLodmphGsJdB0a7KrUyoKPMca0EOnJmwDo2GdUkGtSOws+xhjTQhzavRmAASNqHEpvEiz4GGNMC1F0YDsZxDKod+/aTw4yCz7GGNNCtDmSxOHIvk02mag3Cz7GGNMC5OQXEVe6l9KOg4NdlTqx4GOMMS3A5p276CD5xMSPqP3kJsCCjzHGtABpOzYC0GvQ2CDXpG4s+BhjTAuQl7YNgKiew4Nck7qx4GOMMc1ccamHyJydFIdEQ/u42r/QBFjwMcaYZm7r/lz66T4K2g8Aafoz3cCCjzHGNHsJqdkMDNlPZDPpcgMLPsaYJmBX+lFeXJHMt8mZwa5Ks7Q5eT9xkkl0z2HBrkqdBX0ZbWNM65Samc8Hmw6wdON+fjh4tGL/mQM7818XDeGUfp2CWLuG83iUA0cK6RUbhQSgC0xVydzjLlHWpeknFC1nwccYEzD7c47x4aYDLN20n01puQBM7NuRR2eM4ILh3fl02yHmLt/Flc+vYfLgLtx/0RAm9Gna69KUKywp498b9vHiyt3sSs9jbO8O3DVlIBcO7+7XjAN7sgrociwFImgW2azLWfAxxvhV+tFCPtp8kKUb97MuNRuAMfGxPDRtGJeO6UVch+iKc285uz+zT+vNa9+k8sJXyVz+3NdMGdqV+y8cwtjeHYJ0BzXLKSjmtW9SWfB1Khl5RYzo2Z77LhzMovX7uP1fCQzt3o47zxvIpaN7Ehbq+5GOdSnOeI9KKNKxv8+v7y+iqsGuQ7Nwyimn6Lp164JdDWOahez8Yj7acpAPNu3nm+RMPArDerRj+pieTB/Ti35dYmq9Rn5RKa+uSeWFFUnkFJRwwbBu3H/REEbFxQbgDmq3J7OAl1fv5q21ezlWUsa5Q7rys3MGcMbAzogIpWUePth0gGe/3MXO9Dz6dm7Dz88dyOUT4ogMC/VZPR5ctJnzN/4XF3bJRu5pWj+jRCRBVU+p8pgFn7qx4GNMzY4UlvDp1kMs3bif1bsyKPUoA7rEMH1sL2aM6cng7u0adN28olJe+TqFeSuSyT1WwsUjunPfhUMY0au9j++gbjbszeHFFcl8tOUAoSHCZePiuG3yAIb2qPr+PB7ls+2HePbLXWxKy6VH+yhuP2cAs0/rQ3RE44PQxU99xT/z7yZu4GiY/Uajr+dLFnx8wIKPMScrKC7lP9vTWbpxP18lHqa4zENch2hmjO3F9DE9Gdmrvc8G3Y8UlrBgdQovrkzmaGEpl4zqwX0XDqn2h74veTzKFz+kM29FMt+lZNEuKozrJvXlpjP70b2Oy1WrKit3ZvDMl7v4bncWnWIiuOXs/lx/Rl/aR4U3qF65BSVM/P0yEqPnEHrWPXDhow26jr/UFHxszMcYUy+FJWUsT0xn6aYDfL79EIUlHrq3j+S6SX2ZMbYn43p38Mssr/ZR4dx7wWBuPLMfL63azcurdvPx1oNMG92T+y4Y3OCWVU0KS8pY/P0+XlyZTPLhfOI6RPPw9BFcfWpv2kbW78eniHDOkK6cM6Qra1OyePbLXTzxSSLPf5XEjWf04+az+tG5bWS9rrl+TzZ9JJ1QLW1WM93AWj51Zi0f05oVl3pYteswSzce4LNth8grKqVzTASXjO7BjDG9OLVfp4CvIZNTUMz8lbv55+rdFJSUMXNsL+69YDADu7Zt9LWz8p1JBK+uSSEjr5hRce25/ZyBTBvVw6eTBrbsy+W55bv4aMtBosJCmX1aH247pz89Y6Nr/zLwxCc/kLTiLZ4P/yvc+gXET/RZ3XzBut18oLUGnzKPsjYli2WbD/Dd7iz+MGsUpzbT5y9M/ZV5lH+u3s0zX+4ip6CE9lFhXDKqJ9PH9uSMAZ39MnurvrLyi3lxZTILVqdQVFrGrHFx3HPBYPrXYVJDZamZ+cxfuZt3EvZSWOLh/GHduG3yACYN6OTXZ3Z2pR9l7vJk3t+wjxCBKyfG8/NzB9K3c833cM0La5iWu5AbC16BB/ZCVHDGwapjwacaIhIDPAcUA8tV9fXqzm1NwafMo3y32wk4H289yOGjRUSGhRATGUaICB/ee3ad+7lN87Xj0FF+9e4mNuzN4dwhXbnhjL5MHtyViLDgB5yqZOQVMW9FMq+uSaGkTPnx+DjuPX8wfTq3qfW76/dk8+KKZD7eepDwkBBmje/FbZMH+KUrryZ7swp4YUUSb69Lo7TMw8yxvbjzvEEMqaIeJWUeRj/6CW92fYVxJRvhv7cHtK51EdTgIyIvA9OBdFUdVcXx3sCrQHdAgXmq+ndflyUiU4G/A6HAfFV9XESuB3JUdamIvKWq11R37ZYefMo8yre7M52As+UQGXlFRIWHcN7Qbkwb3ZPzh3VjX84xZj27mmE92vHm7Wc02R9CpnGKSz3MXZ7EM1/upF1UOI/MGMHMsb3895v/kf2wbz0c2go9x8CgiyC04cPR6UcLeeGrZF77JpUyj3LFhHjuPn8QvTudGITKZ6G9uCKZdanZxEaHc92kPtx4Rj+6BfmXq/QjhcxftZvXvkmloLiMi0d05+7zBzEmvkPFORv25jDr2dV83/NxOnboCDcuCV6FqxHs4HMOkAe8Wk3w6Qn0VNX1ItIOSABmqeo2r3O6AcdU9ajXvkGquqsuZYlIKLADuAhIA9YCs4HLgI9UdYOIvKGqP6nuPlpi8Ckt8/Dd7iw+3HyAT7YeJCOvmKjwEM4fdjzgtIk48YfAh5sOcNcb67l+Ul/+MOukv07TzG3cm8P/vreJHw4eZebYXjwyY0S9B8FrlJ8J+9fD/u+dgLP/e8g7eOI57eNg/PUw4XqIjW9wUelHCnlueRJvfLcHj0e5+tTe3HXeIDrHRPDe+jTmr9zN7ox84jtGc+vZ/bnqlN7E1HMSgb9l5xez4OsUFnydQu6xEiYP7sJd5w3i9P6deGnVbh77cBvJ7X9GyLifwLQngl3dkwS9201E+gEfVBV8qjj338AzqvqZ176rgJ8D01S1SERuAy5X1UvqUpaInAE8qqo/cj8/6B5KA7JV9QMReVNVr63iejOAGYMGDbpt586ddb/pJqq0zMO35QFny0Ey84uJDg/l/OHdmDaqJ+cN63pSwKnsz8u288KKZJ64cgxXndI7QDU3/nSsuIyn/rOD+SuT6dYuisdmjeLCEd0bd9HCXNi/wQkw+9fDvu8hd497UKDLYOg1AXqNh7gJ0HUYJC+HhAWQ9IWzNMDgi2HizTD4Ighp2DMxB3KP8dyXSby51im7bWQY2QUljI2P5fZzBvKjkd2bxNhVTfKKSnntm1Tmr9xNRl4Rp/TtSHGZBzl6gH8X3QrTnoTTbgt2NU/SbIKPe94KYJSqHql07FfAmcA7wN3ARaqaV5eyRORKYKqq3up+vh44Hfhf4BmgEFjVUsd8Sss8fJN8vIWTlV9Mm4hQzh/WjUtH92TK0G71etittMzDDS9/x7rUbN77+ZmMjm8aT5ybhlmTlMkDizaRmlnAT07vwwOXDKv/cyfFBXBw0/HWzP71kOnVMdGhrxNgyoNNz7E1D45np8D6V+H71yDvkNMamnCD0yKKbdhiaftyjvH88iSyCoq5YVJfTuvv30kE/lBYUsbb6/bywlfJ7Ms5xv8MPsBde/8bblgCA84NdvVO0iyCj4i0Bb4C/qiqi6o5501gGjBQVQ/Xtazqgo+q3l3Xe2huwae0zMOaZGcM55OthyoCzgXDu3Pp6B6cO6R+AaeyzLwiZjy9ChFh6T1n0ykmwoe1N4FwpLCEPy/7gYXf7aFv5zb8+fLRnDmwS+1fLC2GQ1u8us++h8PbQT3O8Xa93NbMeOe91wRo08AZkmUlkPgRJPzTbQ2FwOAfwSk3w6ALG9waau6KSz188cMhJmW8R4flv4b/ToR2PYJdrZM0+YdMRSQceA94vYbAMxkYBSwGHsFp/dTVPsC7fyje3deilJR5+Dopk4/cFk52QQkxbsCZNronU4Z2JSrcN/9ZO7eN5PnrJ3Ll82u4Z+F6Xrn5tCbfdWGO+3z7IX69eAvpRwu5/ZwB3H/hkOp/GSnIcgLAvgQn4BzaCmXFzrHoTk6LZtilbqAZD+17+q6ioeEwYqbzytp9vDW04yNoH++2hq5rcGuouYoIC2HqqJ7w4W6IbA9tG9lFGgRBb/mI0+59BchS1fuq+f544A2cmWy7gdeBJFX9TV3KEpEwnAkHF+AEnbXAT1R1a13voam1fErLPBzILWRvVgF7swtISM3m022HyCkooW1kGBcMdyYNnDvEdwGnKm+v3cuv3tvEz88dyAOXNJ+FrFqrzLwifrd0G0s27mdo93b8vyvHMK66bNGHd8C3z8PGhVBS4PyQ6znW7T5zWzQd+gR+2eayEkhcBuv+CclfOq2hIVOdsaFBFwS/NaQKxXkQGYBp2q/McLo8b/vc/2U1QFBbPiKyEJgCdBGRNOARVX1JRJYBtwIDgOuBzSKywf3aQ6q6zOsybYCrVTXJveYNwE11LUtVS0XkbuATnKnWL9cn8ASDx6Mczitib1YBadnHKoLM3qxj7M0u4EBuIWWe4784tI0M40I34Jzj54Dj7epTe7MhLYfnv0pibHwsl4z24W+9xmdUlSUb9/O7pds4WljC/RcO4Y4pA0+eLq/q/EBf8xzs+gxCI2HMVXDqbdBjDIQ0gdZtaDiMuMx5Ze2G9a84raHEZRDb+3hrqH0v/9VBFfIzICsJMpOc8a2sJMhMdt5LCqDTQOh3NvSb7Lz7skVY7vAOJ+A2Q636IdP68HXLR1XJPVZSEUwqB5d92ccoKvWc8J1u7SKJ7xhN705t6N2xDb07RbvvbegZGxW0bq+i0jKueeEbdh46yvt3nRXwB/NMzQ7kHuM3i7fw+Q/pjO3dgb9cMebkZJwlx2DT2/DNXGf8JqYbnHornDIH2nYNTsXro7TYCT4JC9zWUKjbGrqpca2hYzleASbJ3d7lBJmi3OPnhYRBx35OwOk8yBnj2pcAKauPn9d50InBqLFjNIW58HgfJ5no2fc37lp+EvQJBy1BQ4KPqrLjUN5JgSUt+xhpWQUcLSo94fzY6PATAkrvjtHEu4EmvmN0wFozDXEg9xgznl5F+6hw3r/7rAZn6TW+4/EoC9fu4c/LfqDU4+GXFw/l5rP6E+qdg+3oQVg7H9a9DAWZ0H00nHEnjLoCwnz4fE8gZSVDwiuw4XXIP+zVGrq+6tZHcb7zncxdlYJMEhRkeJ0o0KH38QDTeaC7PdCZzVfVg7GeMji4GVJWOa/Ur72C0WDo7waivmdDu3qO26Stg/kXwLULYdi0+n03QCz4+EBDWz6jH/2Eo4VOkIkOD/VquTjv8eUtmE5tms4PbFXnP23OXsgtf6U5XQnn/m+1D/59k5zJT+d/y/nDuvHCdRMDnmjSHJeSkc8DizbxTXIWZw7szOOXjzkxzcz+DU4rZ8t74CmFodNg0h3OD8JmNv24WqXFkPih2xpa7rSGhl4C8ac43XXlAefogRO/167n8aBSEWAGOS2b8EZmPvCUOVPSd688HoyK3Wfnuwx1W0Zu66i2Fuf3r8O/74R71jv1bIIs+PhAQ4PPl4npdIgOJ75jG7q0jWgazxWUFsORNCeg5LiBJXeP13YalBWd+J2Ids6+uIlw04fVdmO8vGo3v/9gG//zo6Hcdd6gANyM8VZa5uHl1bv5v093EBEawq8vHc41p/Z2/t15ypyuqW/mQupqiGjrjI2cdnuT/eHlM5lJx2fKFWRAm85eLZgBx7c7DYDIxmfFrrOyUji40QlEu1fCnjXOZAVwHrr17qaLqTQN/rNH4Jvn4KEDjUpH5E8WfHygqc12q9GxHDeIuMEkZ4/X9l7noT0q/b237eG0aDr0dt5j+3h97g1RsbDpLVj8sxr7mFWV+97awJKN+1lw82mcO6QZjBe0ENsPHOF/39vEprRcLhzencdmjaJHbBQUHnF+6H77POSkOn+3p//MSV8T1coeEC4rcbrZojsEuyZVKyuFAxsgpbxltAZK8p1jXYef2E235G6nBXfXN0Gtck0s+PhAQIJPabHzD624wOniKs53XuXbJ7wXOL8hlW8fyz4eYIqOnHjd0Ag3oLiBpCLAuNvt4+rWv68K79wIPyxzpnb2HFvlaQXFpVz+3NccyC3kg3vOPimho/GtotIynv0yiee+3EVsdDiPzhzJ9DE9kewU+G4erP+X07XT5wyna23opU32N2VTSVmJ00WassIJRnu+cf7Pg9ONOHw6XP1qUKtYEws+PtDg4PP5H5zAUG0wKXADTr7T914fYdEQEQMRbZzfYE8KLm7rJaar76bIFmTBc2c4vzne/lW1feCpmfnMeHoV8R3b8N4dZ/pkrfqWzONRiss8FJV6KC71UFLmvBdXfvc+XuahoLiMl1ftZmd6Hj8eH8dvLx1Op4x1TnfMD2736MjLnaATNyHYt2kaq7TYySqRshL2fudkehh6UorLJsOCjw80OPg8NcqZxhrRBsLdQBHexgka4W2czxFtj29XnOO+R8RU/b3wNsF75mLXf+C1K2DSnTD1z9We9mViOnMWrOXH4+L4v6vHNo3xrgDZk1nAk58mkn600A0WWn0wKfOc8MxWffWMjeJPM4dwXulqJ+gc2AjRHZ1p0qfe5p/nS4ypgyafXqdFu39LsGvge4MudAapv3kOhvwIBkyp8rTzhnbj/guH8NfPdjAmPpabzuof2HoGgaqy+Pt9PPzvrQgwrGc7oiNCiQ0NISIshHD3PTIshIjQ458rXqEnvlccDxWipJRISoikmAgpdd49RXQ5sJywj+5wxvK6DoMZf4fRVzu/sBjTRFnwMQ1z4e8g6Ut4/064Y7Xzm3YV7j5vEJvScnjsw+2M6BXLaf1b7hLcucdK+M37W1i6cT+n9e3I3IlpdC5JhNIiKC10X0XOrMHSIihyP1e8Kh0v/1xaeDyXWnUGXeR0rQ08v+VMlTYtmnW71VGzmu0WKPvWw0sXwYhZcOVL1Z52pLCEy55ZzdHC0ha7BPd3u7O4/60NHDxSyP0XDubO6P8Q8skDx0+QEAiLciZ2eL+HRlSxP/LkfaHV7A+LdGZBdbFp7abpsTEfH7DgU42vnoAvH4MrXoLRV1Z7WuLBo/z4uZa3BHdJmYd/fL6TZ7/cRe9ObfjbNeMYH5IML//I6Z684kVnYojNLjOtUE3Bp2X8BDDBc/b9EH8qfPhfzjTvagzt0Y6/XDmG9Xty+MMH26o9rzlJycjnyufX8PQXu7hiQjwf3juZ8V0U3rnJeUr+x3OdzMYWeIw5iQUf0zihYfDjF5yH496/Ezyeak+dPqYXt58zgH99k8o76/YGsJK+paq8s24vl/5jJbsP5/HsTybwxFVjaRsR6vwZHD0AVy2odhzMGGPBx/hC54Ew9U+w+yvnKfoa/OpHQzljQGd+/f4WtuzLrfHcpii3oIS73/ie/3l3E6PiYvn4vnO4dIw7lfnrp51Fzi5+DOInBreixjRxFnyMb0y4EYZcAv95FNK3V3taWGgIz/xkPF1iIvjZvxLIyq9lFlcTsiYpk6l/X8EnWw/yq6lDeeO2SfTqEO0c3POtc+/DZzqpa4wxNbLgY3xDBGb+wxnjWHSb8yR2NTq3jWTudRM5fLSIexd+36gHLAOhuNTD//v4B34y/xuiwkNZdOeZ3Dll0PGlCfIz4d2bncwSlz1jU52NqQMLPsZ32naDmU8765cs/1ONp47t3YE/zBrJql0ZPPlpYoAqWH/Jh/O48vmvmbs8iWtO6c0H95zNmPgOx0/weJxkq/mH4apXWl+iTmMayKbhGN8aNs1ZuGvV32DwxdD3zGpPvebUPmzYm8vc5UmMiWtaS3CrKm+t3cvvlm4jMjyE56+bwNRRVdRv9VPOctOX/h/0GhfwehrTXFnLx/jej/4EHfs6LYLCIzWe+ujMEYzr3YFfvrORXelHA1TBmmXnF3PHa+t5YNFmxvfpwMe/OKfqwJOyGr54zEncecotga+oMc2YBR/je5Ht4MfznOd+Pn6w5lPDQpl73QSiI0K5/V8JHC0sCVAlq7Z6VwZT/76Cz384xEPThvHaLac7a+JUlncY3p0DHfs7udRsnMeYerHgY/yjz+lw9n/Bhtdg25IaT+0ZG80zP5lAamYB//32RjxBmIBQXOrhz8u2c91L3xITGcbiO8/i9nMGVr0UuKcMFt0KhTlw9SsQ1T7g9TWmubMxH+M/Ux5wll9Y+gvofTq0617tqZMGdOahacP5wwfbuP7lbxnSvR3xHdsQ1yGa+I7RxHWIpkObcL8sy7ArPY9fvPk9W/cf4aen9+E3l46oef2hFU9C8nKY8Q/oMdrn9TGmNbDgY/wnNBwunwcvnAP/vgt++k6N3VNzzupHZl4Rn207xPd7cigoLjvheExEKHFuIIrv2MZrO5q4jtF0bRtZr+Ckqrzx3R7+8ME2osNDmXf9RC4e2aPmLyV/Bcv/DGOucSZWGGMaxBKL1pElFm2Eb+fBR/8Dl/4VTq3bwLyqklNQQlr2MfblFLjvx5x3dzv32InjQxFhISe0lOI6OEGpPFB1bxdJWKjT05yVX8z/vreJz7YdYvLgLvzfVWPpVlu27aMH4fnJTtqc276AyLYN+uMwprWwxeRMcJ16q5N25tPfQP9z65T+X0ToGBNBx5gIRsdX/ezM0cIS9uUcD0blgSkt5xjbt6eTkVd0wvmhIULP2CjiOkSTnJFPbkEJv7l0OHPO6l/12I63slJ471YozoMbl1jgMaaRWnXwEZEY4DmgGFiuqq8HuUotU0gIXPYsPHeGk/3glk+dLrlGahcVzrAe4QzrUfWAf2FJWaXgVFCxPbBrDA9PH8mIXnWcLPDV45CyEmbNhW7DG113Y1o7vwYfEXkZmA6kq+qohp7T2PJEZCrwdyAUmK+qj7uHLgfeVdWlIvIWYMHHX9r3ghl/c5YbWPEknFfzFGxfiAoPZWDXtgzs2shWyq7PnTqPvw7G/cQ3lTOmlfP3VOsFwNTGnCMi3USkXaV91fXbnHQtEQkFngUuAUYAs0VkhHs4HijP7X/i6LbxvZE/dgbqVzwBac1k/OzIfqe11m04XPJEsGtjTIvh1+CjqiuArEaecy7wvohEAojIbcDT9bjWacAuVU1W1WLgTeAy91gaTgCCav4sRGSGiMzLzW1+6f+bpGlPOK2gRbdDcX6wa1OzslLnQdKSQidvW0SbYNfImBajyT9kqqrvAJ8Ab4nIT4E5wFX1uEQcx1s34AScOHd7EXCFiMwFllZT/lJVvT021hJG+kRUrDNukpXsTEBoyr74A+xZ42Qw6Dok2LUxpkVpFhMOVPUvIvImMBcYqKp5PrpuPnCzL65l6qH/ZDjzbmfxtSGXwJCLg12jk+34BFb/DSbeDGPq87uOMaYumnzLB0BEJgOjgMXAI/X8+j6gt9fneHefCabzfwvdRjoPn+ZnBLs2J8rZ6yRF7TEapj5e+/nGmHpr8sFHRMYD83DGaW4GOovIY/W4xFpgsIj0F5EI4Fqg5mRjxv/CIp3sB4U5TvqdpvKwc2mxszBcWakzzhNey4OnxpgG8WvwEZGFwBpgqIikicgt7v5lItKrpnO8tAGuVtUkVfUANwCpdS1PVUuBu3HGjbYDb6vqVt/fram3HqOcFtAPH8CGJjLL/fPfQdpauOxp6Dww2LUxpsWy9Dp1ZOl1/MRTBq/MhAMb4I7V0LFf8Oryw4fw5k/gtNudWXnGmEapKb1Ok+92My1cSCj8eC5ICCz6mROMgiE7BRbfAb3Gw8X16dU1xjSEBR8TfB36OC2Nvd/A6r8HvvzSIifzAsBVC5zxKGOMX1nwMU3DmGtgxCz48k+w8zMoLghc2Z/+FvZ/D7OeC263nzGtSLN4zse0AiIw/SnY+x28fiUg0Kk/dBsB3Ucef+80wOmq85Wti+G7F2DSXTB8uu+ua4ypkQUf03S06QQ/XwWpqyF9Gxza6rwnLgP1OOeERUHXoc4zQt1HHA9KbbvXuFBdlTKT4N/3QPypcOGjPr8dY0z1LPiYpiWmM4yY6bzKlRyDwz/AoW3Hg1LS57DxjePnRHfyaiGNcIJTt+HVr7tTUgjv3AihYXDlPyEswr/3ZYw5gQUf0/SFRzuz0HqNP3F/foYbjLZB+lbn/fvXoMQrYWmHvicHpc6D4JMH4eBm+Mnb0KE3xpjAsuBjmq+YLtD/HOdVzuOBnNTjXXbl7zs+Pt51FxoBZcVw1n0w5EdBqboxrZ0FH9OyhIQ4ExU69T9xAkFJIWQkHm8lqTrZFYwxQWHBx7QO4VHQc6zzMsYEnT3nY4wxJuAs+BhjjAk4Cz7GGGMCzoKPMcaYgLPgY4wxJuAs+BhjjAk4Cz7GGGMCzoKPMcaYgLNltOtIRA4DqQ38ehcgw4fVCaaWci8t5T7A7qUpain3AY27l76q2rWqAxZ8AkBE1lW3jnlz01LupaXcB9i9NEUt5T7Af/di3W7GGGMCzoKPMcaYgLPgExjzgl0BH2op99JS7gPsXpqilnIf4Kd7sTEfY4wxAWctH2OMMQFnwccYY0zAWfDxIxGZKiKJIrJLRB4Idn0aSkR6i8iXIrJNRLaKyC+CXafGEpFQEfleRD4Idl0aQ0Q6iMi7IvKDiGwXkTOCXaeGEJH73X9bW0RkoYhEBbtOdSUiL4tIuohs8drXSUQ+E5Gd7nvHYNaxrqq5lyfcf1+bRGSxiHTwRVkWfPxEREKBZ4FLgBHAbBEZEdxaNVgp8N+qOgKYBNzVjO+l3C+A7cGuhA/8HfhYVYcBY2mG9yQiccC9wCmqOgoIBa4Nbq3qZQEwtdK+B4DPVXUw8Ln7uTlYwMn38hkwSlXHADuAB31RkAUf/zkN2KWqyapaDLwJXBbkOjWIqh5Q1fXu9lGcH3Bxwa1Vw4lIPHApMD/YdWkMEYkFzgFeAlDVYlXNCWqlGi4MiBaRMKANsD/I9akzVV0BZFXafRnwirv9CjArkHVqqKruRVU/VdVS9+M3QLwvyrLg4z9xwF6vz2k04x/Y5USkHzAe+DbIVWmMvwG/AjxBrkdj9QcOA/90uxDni0hMsCtVX6q6D3gS2AMcAHJV9dPg1qrRuqvqAXf7INA9mJXxoTnAR764kAUfU2ci0hZ4D7hPVY8Euz4NISLTgXRVTQh2XXwgDJgAzFXV8UA+zad7p4I7HnIZTjDtBcSIyHXBrZXvqPM8S7N/pkVEfo3TBf+6L65nwcd/9gG9vT7Hu/uaJREJxwk8r6vqomDXpxHOAmaKSApOV+j5IvJacKvUYGlAmqqWt0LfxQlGzc2FwG5VPayqJcAi4Mwg16mxDolITwD3PT3I9WkUEbkJmA78VH30cKgFH/9ZCwwWkf4iEoEzgLokyHVqEBERnHGF7ar612DXpzFU9UFVjVfVfjh/J1+oarP8LVtVDwJ7RWSou+sCYFsQq9RQe4BJItLG/bd2Ac1w4kQlS4Ab3e0bgX8HsS6NIiJTcbqpZ6pqga+ua8HHT9wBuruBT3D+I72tqluDW6sGOwu4HqeVsMF9TQt2pQwA9wCvi8gmYBzwp+BWp/7cltu7wHpgM87PpWaTnkZEFgJrgKEikiYitwCPAxeJyE6clt3jwaxjXVVzL88A7YDP3P/7z/ukLEuvY4wxJtCs5WOMMSbgLPgYY4wJOAs+xhhjAs6CjzHGmICz4GOMMSbgLPiYVk1EHhWRXwa7HjURkavcjNVf+uBaKSLSxRf1qme5vUTk3VrO6eedTdm0bGHBroAxLYGIhHklX/S1W4DbVHWVn67vd6q6H7gy2PUwTYe1fEyrIyK/FpEdIrIKGOq1f6CIfCwiCSKyUkSGee3/RkQ2i8hjIpLn7p/inrcE2OauEfSEiKx11z75mde1/8dr/++qqddst4wtIvL/3H0PA2cDL4nIE5XOnyIiK0TkQ3HWjXpeREKqu1al7/5eRO7z+vxHEfmFe83lcnyNoNfdrAOIyAVuAtPN7rovke7+FBH5s/sA4joRmSAin4hIkoj83D2nolXjbq8UkfXuq7mn0jENoar2slereQETcZ6ibwO0B3YBv3SPfQ4MdrdPx0m9A/ABMNvd/jmQ525PwUnm2d/9fDvwG3c7EliHkyzzYpwn9gXnF74PgHMq1asXTpqZrjg9El8As9xjy3HWuql8L1OAQmAAzho4n+G0Lmq6VgrQBegHrHf3hQBJQGf3mrk4uQhDcJ52PxuIwsnSPsT9zqs4CWbLr3mHu/0UsAnnifiuwCF3fz9gi7vdBohytwcD6yqfY6+W/7JuN9PaTAYWq5ujym21lGfsPhN4x/1FH5wAAnAGx9djeQMn/X+571R1t7t9MTBGRMq7l2Jxfrhe7L6+d/e3dfev8LrOqcByVT3s1ud1nLV63q/lfr5T1WT3OwtxAkVJbddS1RQRyRSR8Tjp/r9X1Uz33r9T1TT3uxtwgsJRnOSfO9xLvALchbM8BRzPW7gZaKvOuk9HRaRITl75Mhx4RkTGAWXAkFru0bRAFnyMcYQAOao6rp7fy/faFuAeVf3E+wQR+RHwZ1V9oXFVrFLl/Fj1yZc1H7gJ6AG87LW/yGu7jLr9nCj/jqfS9z1VfP9+4BDOyqshOK0308rYmI9pbVYAs0QkWkTaATMA1FmfaLeIXAVOJm8RGet+5xvgCne7puWdPwHuEGf5CURkiDiLu30CzHFbV4hInIh0q/Td74BzRaSLOEuwzwa+qsP9nCZO5vQQ4BpgVT2utRhnyeRT3TrWJBHoJyKD3M/X17F+VYkFDqiqx71OaAOvY5oxCz6mVVFnOfC3gI04KzKu9Tr8U+AWEdkIbOX4suf3Af8lTuboQThjIlWZj7OkwXp3cP0FIEydVTnfANaIyGacDM7tKtXrAM5CcF+6dUtQ1bqk4V+Lk3V4O7Abp0uxTtdSZ3n3L3EyrpfVVIiqFgI343RLbsZp0TQ0u/FzwI3un/MwTmw9mlbCslobUwsRaQMcU1UVkWtxJh9cVtv3AlCvKTiTJaY38PshOMsYXKWqO31YNWNqZWM+xtRuIs4AuQA5OOvYN2siMgJn1t1iCzwmGKzlY4wxJuBszMcYY0zAWfAxxhgTcBZ8jDHGBJwFH2OMMQFnwccYY0zA/X9FUFpu/jkrhgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "## EDIT THIS CELL\n",
        "\n",
        "K_max = 12 # this is the maximum degree of polynomial we will consider\n",
        "assert(K_max < N) # this is the latest point when we'll run into numerical problems\n",
        "\n",
        "rmse_mle = np.zeros((K_max+1,))\n",
        "rmse_map = np.zeros((K_max+1,))\n",
        "\n",
        "for k in range(K_max+1):\n",
        "    \n",
        "    # feature matrix\n",
        "    Phi = poly_features(X, k)\n",
        "\n",
        "    # predict the function values at the test input locations (maximum likelihood)\n",
        "    y_pred_test = 0*Xtest ## <--- EDIT THIS LINE\n",
        "      \n",
        "    ####################### SOLUTION\n",
        "    # feature matrix for test inputs\n",
        "    Phi_test = poly_features(Xtest, k)\n",
        "    \n",
        "    \n",
        "    rmse_mle[k] = -1 ## Compute the maximum likelihood estimator, compute the test-set predictions, compute the RMSE\n",
        "    rmse_map[k] = -1 ## Compute the MAP estimator, compute the test-set predicitons, compute the RMSE\n",
        "\n",
        "    \n",
        "    #@ Test\n",
        "    assert rmse_mle[k] != -1, \"mle prediction not implemented\"\n",
        "    assert rmse_map[k] != -1, \"map prediction not implemented\"\n",
        "\n",
        "plt.figure()\n",
        "plt.semilogy(rmse_mle) # this plots the RMSE on a logarithmic scale\n",
        "plt.semilogy(rmse_map) # this plots the RMSE on a logarithmic scale\n",
        "plt.xlabel(\"degree of polynomial\")\n",
        "plt.ylabel(\"RMSE\")\n",
        "plt.legend([\"Maximum likelihood\", \"MAP\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "khy32Ob8jmsS"
      },
      "source": [
        "#### Questions:\n",
        "1. What do you observe?\n",
        "2. What is the influence of the prior variance on the parameters ($\\alpha^2$)? Change the parameter and describe what happens."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}