7 - Naive Bayes/Gaussian Naive Bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d5894972",
   "metadata": {},
   "source": [
    "# Gaussian Naive Bayes\n",
    "\n",
    "## Overview\n",
    "- [1. Naive Bayes Rule](#1)\n",
    "- [2. Gaussian Naive Bayes](#2)\n",
    "- [3. Gaussian Naive Bayes Model from Scratch](#3)\n",
    "- [4. Gaussian Naive Bayes Model in `Sklearn`](#4)\n",
    "- [5. References](#5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2147e04",
   "metadata": {},
   "source": [
    "<a name='1' ></a>\n",
    "## 1. Naive Bayes Rule\n",
    "Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable. Bayes’ theorem states the following relationship, given class variable $y = c$  and dependent feature vector $\\mathbf{x} = \\begin{bmatrix} x_1 & x_2 & \\cdots & x_n \\end{bmatrix}$\n",
    "\n",
    "$$P(y = c \\mid \\mathbf{x}) = \\frac{P(y = c) P(\\mathbf{x} \\mid y = c)}{P(\\mathbf{x})}$$\n",
    "\n",
    "where\n",
    "\n",
    "- $P(y = c \\mid \\mathbf{x})$ is *the posterior probability*\n",
    "- $P(y)$ is *the prior probability*\n",
    "- $P(\\mathbf{x} \\mid y = c)$ is *the likelihood*\n",
    "- $P(\\mathbf{x})$ is *the evidence*\n",
    "\n",
    "To make the calculation easier, we will now use $P(y)$ instead of $P(y = c)$. In addition, we have:\n",
    "\n",
    "\\begin{align*}\n",
    "    P(y) P(\\mathbf{x} \\mid y) &= P(\\mathbf{x}, y) \\\\\n",
    "    &= P(x_1, x_2, ..., x_n, y) \\\\\n",
    "    &= P(x_1 \\mid x_2, ..., x_n, y) P(x_2, ..., x_n, y) \\\\\n",
    "    &= P(x_1 \\mid x_2, ..., x_n, y) \\underbrace{P(x_2 \\mid x_3 ..., x_n, y) P(x_3, ..., x_n, y)}_\\text{$P(x_2, ..., x_n, y)$}  \\\\\n",
    "    &= \\cdots \\\\\n",
    "    &= P(x_1 \\mid x_2, ..., x_n, y) P(x_2 \\mid x_3, ..., x_n, y) \\cdots P(x_{n-1} \\mid x_n, y) P(x_n \\mid y) P(y)\n",
    "\\end{align*}\n",
    "\n",
    "Using the *naive* conditional independence assumption that\n",
    "\n",
    "$$P(x_i | x_{i+1}, \\dots, x_{n-1}, x_n) = P(x_i | y),$$\n",
    "\n",
    "So  \n",
    "\n",
    "$$P(y) P(\\mathbf{x} \\mid y) = P(y) \\prod_{i=1}^{n} P(x_i \\mid y) \\\\\n",
    "\\Rightarrow P(y \\mid x_1, \\dots, x_n) = \\frac{P(y) \\prod_{i=1}^{n} P(x_i \\mid y)}{P(x_1, \\dots, x_n)}\n",
    "$$\n",
    "\n",
    "Since $P(\\mathbf{x}) = P(x_1, \\dots, x_n)$ is constant given the input, we can use the following classification rule:\n",
    "\n",
    "\\begin{align}\\begin{aligned}P(y \\mid x_1, \\dots, x_n) \\propto P(y) \\prod_{i=1}^{n} P(x_i \\mid y)\\\\\n",
    "\\Downarrow\\\\\\hat{y} = \\arg\\max_y P(y) \\prod_{i=1}^{n} P(x_i \\mid y),\\end{aligned}\\end{align}\n",
    "\n",
    "and we can use **Maximum A Posteriori (MAP)** estimation to estimate $P(y)$ and $P(x_i | y)$; the former is then the relative frequency of class  in the training set.\n",
    "\n",
    "The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of $P(x_i | y)$.\n",
    "\n",
    "In spite of their apparently over-simplified assumptions, naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering. They require a small amount of training data to estimate the necessary parameters.\n",
    "\n",
    "Naive Bayes learners and classifiers can be extremely fast compared to more sophisticated methods. The decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one dimensional distribution. This in turn helps to alleviate problems stemming from the curse of dimensionality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f894e787",
   "metadata": {},
   "source": [
    "<a name='2' ></a>\n",
    "## 2. Gaussian Naive Bayes\n",
    "**Gaussian Naive Bayes** implements the Gaussian Naive Bayes algorithm for classification. The likelihood of the features is assumed to be Gaussian:\n",
    "\n",
    "$$P(x_i \\mid y) = \\frac{1}{\\sqrt{2\\pi\\sigma^2_y}} \\exp\\left(-\\frac{(x_i - \\mu_y)^2}{2\\sigma^2_y}\\right)$$\n",
    "\n",
    "Use **Maximum Likelihood Estimation (MLE)** for $m$ examples and $x_i^{(j)}$ is $i^{th}$ feature at the $j^{th}$ observation *(Note: $L(\\mu_y, \\sigma_y)$ is $L(\\mu_{iy}, \\sigma_{iy})$ below)*\n",
    "\n",
    "$$L(\\mu_y, \\sigma_y) = \\prod_{j=1}^{m}P(x_i^{(j)}|\\mu_y, \\sigma_y^2) = \\prod_{i=1}^{m}  \\frac{1}{\\sqrt{2\\pi}\\sigma_y} e^{-\\frac{(x_i^{(j)} - \\mu_y)^2}{2\\sigma_y^2}}$$\n",
    "\n",
    "Take logarithm of $L$\n",
    "\n",
    "\\begin{align*}\n",
    "    l(\\mu_y, \\sigma_y) &= \\log L(\\mu_y, \\sigma_y) \\\\\n",
    "    &= \\log \\left( \\prod_{j=1}^{m}  \\frac{1}{\\sqrt{2\\pi}\\sigma_y} e^{-\\frac{(x_i^{(j)} - \\mu_y)^2}{2\\sigma_y^2}} \\right)\\\\\n",
    "    &= \\sum_{j=1}^{m} \\left[ -\\log \\sigma_y - \\frac12 \\log 2\\pi - \\frac{1}{2\\sigma_y^2}(x_i^{(j)} - \\mu_y)^2  \\right] \\\\\n",
    "    &= -m\\log \\sigma_y - \\frac{m}{2} \\log 2\\pi - \\frac{1}{2\\sigma_y^2} \\sum_{j=1}^{m}(x_i^{(j)} - \\mu_y)^2\n",
    "\\end{align*}\n",
    "\n",
    "Deriative\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{∂l}{∂\\mu_y} &= \\frac{1}{\\sigma_y^2} \\sum_{j=1}^m (x_i^{(j)} - \\mu_y) = 0 \\\\\n",
    "\\frac{∂l}{∂\\sigma_y} &= -\\frac{m}{\\sigma_y} +  \\frac{1}{\\sigma_y^3} \\sum_{j=1}^m (x_i^{(j)} - \\mu_y)^2 = 0\n",
    "\\end{align*}\n",
    "\n",
    "Solve the two above equations\n",
    "\n",
    "\\begin{align*}\n",
    "\\hat{\\mu_y} &= \\bar x_i = EX_{iy} \\\\\n",
    "\\hat{\\sigma_y} &= \\sqrt{\\frac{1}{m}\\sum_{j=1}^m (x_i^{(j)} - \\bar x)^2} = \\sqrt{DX_{iy}}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93bca5f8",
   "metadata": {},
   "source": [
    "<a name='3' ></a>\n",
    "## 3. Gaussian Naive Bayes Model from Scratch\n",
    "\n",
    "### Import package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f48b831d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145da35b",
   "metadata": {},
   "source": [
    "### Overview dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b5ee1e43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
       " 0                  5.1               3.5                1.4               0.2\n",
       " 1                  4.9               3.0                1.4               0.2\n",
       " 2                  4.7               3.2                1.3               0.2\n",
       " 3                  4.6               3.1                1.5               0.2\n",
       " 4                  5.0               3.6                1.4               0.2\n",
       " ..                 ...               ...                ...               ...\n",
       " 145                6.7               3.0                5.2               2.3\n",
       " 146                6.3               2.5                5.0               1.9\n",
       " 147                6.5               3.0                5.2               2.0\n",
       " 148                6.2               3.4                5.4               2.3\n",
       " 149                5.9               3.0                5.1               1.8\n",
       " \n",
       " [150 rows x 4 columns],\n",
       " 0      0\n",
       " 1      0\n",
       " 2      0\n",
       " 3      0\n",
       " 4      0\n",
       "       ..\n",
       " 145    2\n",
       " 146    2\n",
       " 147    2\n",
       " 148    2\n",
       " 149    2\n",
       " Name: target, Length: 150, dtype: int64)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overview dataset\n",
    "X, y = datasets.load_iris(return_X_y=True, as_frame=True)\n",
    "X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed3d0a92",
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = np.array(X), np.array(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ef8aa87",
   "metadata": {},
   "source": [
    "### Build model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0178445a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "\n",
    "class GaussianNaiveBayes():\n",
    "    \"\"\"Build GNB from scratch.\"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        self.mean_list = []\n",
    "        self.std_list = []\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        \"\"\"\n",
    "        We have to compute mean/standard deviation belong to each class.\n",
    "        \n",
    "        Args:\n",
    "            X: training examples\n",
    "            y: labels correspond to X\n",
    "        \"\"\"\n",
    "\n",
    "        for label in np.unique(y):\n",
    "            indices = np.where(y == label)[0]\n",
    "            self.mean_list.append(X[indices, :].mean(axis=0))\n",
    "            self.std_list.append(X[indices, :].std(axis=0))\n",
    "            \n",
    "            \n",
    "    def predict(self, x):\n",
    "        \"\"\"Predict label from input x.\"\"\"\n",
    "        probabilities = []\n",
    "        classes = len(self.mean_list)\n",
    "        for c in range(classes):\n",
    "            score = stats.multivariate_normal.pdf(x, mean=self.mean_list[c], cov=self.std_list[c])\n",
    "            probabilities.append(score)\n",
    "        return np.argmax(probabilities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d75958b7",
   "metadata": {},
   "source": [
    "### Training and Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "956dbccf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training model\n",
    "gnb = GaussianNaiveBayes()\n",
    "gnb.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "faf90bac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: [2 0]\n"
     ]
    }
   ],
   "source": [
    "# Testing\n",
    "x1 = [6.4, 3.1, 5.5, 1.8]\n",
    "x2 = [4.9, 3. , 1.4, 0.2]\n",
    "X_test = [x1, x2]\n",
    "\n",
    "y_test = []\n",
    "for x in X_test:\n",
    "    y_test.append(gnb.predict(x))\n",
    "    \n",
    "print(f'Label: {np.array(y_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "429fefee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([array([0.34894699, 0.37525458, 0.17191859, 0.10432641]),\n",
       "  array([0.51098337, 0.31064449, 0.46518813, 0.19576517]),\n",
       "  array([0.62948868, 0.31925538, 0.54634787, 0.27188968])],\n",
       " [array([5.006, 3.428, 1.462, 0.246]),\n",
       "  array([5.936, 2.77 , 4.26 , 1.326]),\n",
       "  array([6.588, 2.974, 5.552, 2.026])])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gnb.std_list, gnb.mean_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4c3f9f",
   "metadata": {},
   "source": [
    "<a name='4' ></a>\n",
    "## 4. Gaussian Naive Bayes Model in `sklearn`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d012b6aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: [2 0]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "# Training model\n",
    "clf = GaussianNB()\n",
    "clf.fit(X, y)\n",
    "\n",
    "# Testing\n",
    "X_test = [x1, x2]\n",
    "print(f'Label: {clf.predict(X_test)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2c929ade",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0.34894699, 0.37525458, 0.17191859, 0.10432643],\n",
       "        [0.51098337, 0.3106445 , 0.46518814, 0.19576517],\n",
       "        [0.62948868, 0.31925539, 0.54634788, 0.27188969]]),\n",
       " array([[5.006, 3.428, 1.462, 0.246],\n",
       "        [5.936, 2.77 , 4.26 , 1.326],\n",
       "        [6.588, 2.974, 5.552, 2.026]]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(clf.var_), clf.theta_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d4c980",
   "metadata": {},
   "source": [
    "**Nice! It's easy to see that our model and sklearn's model share the common result!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6fe4ab7",
   "metadata": {},
   "source": [
    "<a name='5' ></a>\n",
    "## 5. References\n",
    "- [https://en.wikipedia.org/wiki/Naive_Bayes_classifier](https://en.wikipedia.org/wiki/Naive_Bayes_classifier)\n",
    "- [https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case](https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case)\n",
    "- [https://machinelearningcoban.com/2017/08/08/nbc/](https://machinelearningcoban.com/2017/08/08/nbc/)\n",
    "- [https://scikit-learn.org/stable/modules/naive_bayes.html#](https://scikit-learn.org/stable/modules/naive_bayes.html#)\n",
    "- [https://www.python-engineer.com/courses/mlfromscratch/05_naivebayes/](https://www.python-engineer.com/courses/mlfromscratch/05_naivebayes/)\n",
    "- [https://phamdinhkhanh.github.io/deepai-book/ch_ml/NaiveBayes.html](https://phamdinhkhanh.github.io/deepai-book/ch_ml/NaiveBayes.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}