Day5.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz\n",
    "\n",
    "# Would Anyone Like to Give Their Simple Explanation of Gradient Descent?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review: Machine Learning Model Training Workflow\n",
    "\n",
    "1. Collect (and label, if necessary) data\n",
    "2. Define objective function; select performance metrics\n",
    "3. Select model architecture(s)\n",
    "4. Select optimization algorithm for each model architecture\n",
    "5. Split data into train/dev/test (*what are our options for how to do this?*)\n",
    "6. For each selection of hyperparameters, train on train; evaluate on dev\n",
    "7. For best selection of hyperparameters, evaluate on test\n",
    "\n",
    "# Generalization\n",
    "\n",
    "We split our data into train/dev/test (or training and validation) because we want to see if a model trained (on training data) can **generalize** to the test data. In other words, we want low loss on the training data *and* low loss on the test data. \n",
    "\n",
    "*What if you aren't getting low loss on the training data?*\n",
    "\n",
    "*What's one easy way to get loss of 0 on the training data? Would a model like that generalize?*\n",
    "\n",
    "## Assumptions Will Be Our Ruin\n",
    "\n",
    "We assume that the data we collected is a \"good sample\". What's a good sample?\n",
    "\n",
    "Furthermore, we assume that our split into train/dev/test is a \"good split\". What's a good split?\n",
    "\n",
    "We do cross-validation *because a single train/dev/test split may be pathological*:\n",
    "* https://direct.mit.edu/neco/article-abstract/10/7/1895/6224/Approximate-Statistical-Tests-for-Comparing\n",
    "\n",
    "But for a competition generally there is a *single* train/dev/test split. Bad things can happen! Examples:\n",
    "* https://aclanthology.org/P19-1267.pdf\n",
    "* https://aclanthology.org/2021.emnlp-main.368/\n",
    "* https://arxiv.org/pdf/2005.00636.pdf\n",
    "\n",
    "## The Nature of the Task, the Size of the Data and the Complexity of the Model All Interact\n",
    "\n",
    "All machine learning tasks are about fitting (approximating) a function characterized by the training data.\n",
    "\n",
    "If your function is super easy (e.g. label is a multiple of one variable) then a simple model architecture can fit it well.\n",
    "\n",
    "If your function is complex then you need more data and/or a more complex model architecture to fit it well.\n",
    "\n",
    "A complex model architecture and not enough data means your model will probably not generalize well.\n",
    "\n",
    "You can tell if a model is not fitting well by comparing the training and validation losses:\n",
    "* If training loss is low, and validation loss is high, the model is **overfitting**. Possible causes:\n",
    "  * Mismatch between training and validation data.\n",
    "  * Model architecture too complex.\n",
    "  * Not enough data.\n",
    "  * Other data pathologies (e.g. data sparsity, unbalanced split).\n",
    "* If training loss and validation loss are both high, the model is **underfitting**. Possible causes:\n",
    "  * Model architecture too simple.\n",
    "  * Not enough data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's Take a Look (With Weights and Biases!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch torchvision\n",
    "!pip install d2l==1.0.0b0\n",
    "!pip install wandb"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the terminal, type \"wandb login\". Paste in your API key from https://wandb.ai.\n",
    "\n",
    "Now import required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "import wandb\n",
    "import random\n",
    "\n",
    "torch.manual_seed(8)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the \"compact\" implementation of linear regression as a neural network in pytorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearRegression(d2l.Module):  #@save\n",
    "    \"\"\"The linear regression model implemented with high-level APIs.\"\"\"\n",
    "    def __init__(self, in_features, out_features, lr, weight_decay = 0):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        # a linear architecture; see https://pytorch.org/docs/stable/generated/torch.nn.LazyLinear.html\n",
    "        self.net = nn.Linear(in_features, out_features)\n",
    "        # initialize weights to random numbers\n",
    "        self.net.weight.data.normal_(0, 0.01)\n",
    "        # initialize bias to 0\n",
    "        self.net.bias.data.fill_(0)\n",
    "        self.wd = weight_decay # coming back to this later\n",
    "        self.last_epoch = 0\n",
    "\n",
    "    def forward(self, X):\n",
    "        # pass X through the network\n",
    "        return self.net(X)\n",
    "\n",
    "    def loss(self, y_hat, y):\n",
    "        # mean sum of squared error loss\n",
    "        fn = nn.MSELoss()\n",
    "        return fn(y_hat, y)\n",
    "\n",
    "    # we modify plot to log so we can log to wandb\n",
    "    def plot(self, key, value, train):\n",
    "        \"\"\"Plot a point in animation.\"\"\"\n",
    "        assert hasattr(self, 'trainer'), 'Trainer is not inited'\n",
    "        self.board.xlabel = 'epoch'\n",
    "        if train:\n",
    "            x = self.trainer.train_batch_idx / \\\n",
    "                self.trainer.num_train_batches\n",
    "            n = self.trainer.num_train_batches / \\\n",
    "                self.plot_train_per_epoch\n",
    "        else:\n",
    "            x = self.trainer.epoch + 1\n",
    "            n = self.trainer.num_val_batches / \\\n",
    "                self.plot_valid_per_epoch\n",
    "        if value:\n",
    "            l = value.to(d2l.cpu()).detach().numpy()\n",
    "            if train:\n",
    "                self.last_train_loss = l\n",
    "            else:\n",
    "                self.last_test_loss = l\n",
    "            self.board.draw(x, l,\n",
    "                            ('train_' if train else 'val_') + key,\n",
    "                            every_n=int(n))\n",
    " \n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        # stochastic gradient descent\n",
    "        return torch.optim.SGD(self.parameters(), self.lr, weight_decay=self.wd)\n",
    "\n",
    "# We use the trainer from d2l, modified for minibatch SGD\n",
    "@d2l.add_to_class(d2l.Trainer)  #@save\n",
    "def prepare_batch(self, batch):\n",
    "    return batch\n",
    "\n",
    "@d2l.add_to_class(d2l.Trainer)  #@save\n",
    "def fit_epoch(self):\n",
    "    self.model.train()\n",
    "    for batch in self.train_dataloader:\n",
    "        loss = self.model.training_step(self.prepare_batch(batch))\n",
    "        self.optim.zero_grad()\n",
    "        with torch.no_grad():\n",
    "            loss.backward()\n",
    "            if self.gradient_clip_val > 0:  # To be discussed later\n",
    "                self.clip_gradients(self.gradient_clip_val, self.model)\n",
    "            self.optim.step()\n",
    "        self.train_batch_idx += 1\n",
    "    if self.val_dataloader is None:\n",
    "        return\n",
    "    self.model.eval()\n",
    "    for batch in self.val_dataloader:\n",
    "        with torch.no_grad():\n",
    "            self.model.validation_step(self.prepare_batch(batch))\n",
    "        self.val_batch_idx += 1\n",
    "    # at end of epoch, log training and validation loss to wandb\n",
    "    wandb.log({\"epoch\":self.epoch, \"test_loss\":self.model.last_test_loss, \"train_loss\": self.model.last_train_loss})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use our reader for CSV data from Tuesday."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "class CsvData(d2l.DataModule):  #@save\n",
    "    def __init__(self, labelColIndex, path, batch_size=32, shuffle=True, split=0.2):\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        # read the data\n",
    "        df = pd.read_csv(path)\n",
    "        # drop any non-numeric columns\n",
    "        df = df._get_numeric_data()\n",
    "        # drop the label column from the features\n",
    "        colIndices = list(range(df.shape[1]))\n",
    "        colIndices.pop(labelColIndex)\n",
    "        features = df.iloc[:, colIndices]\n",
    "        # keep it in the label, obviously :)\n",
    "        labels = df.iloc[:, labelColIndex]\n",
    "        # split the dataset\n",
    "        self.train, self.val, self.train_y, self.val_y = train_test_split(features, labels, test_size=split, shuffle=shuffle)\n",
    "        print(\"shuffle\", shuffle, \"batch_size\", batch_size, \"split\", split)\n",
    "        print(self.get_feature_count(), self.get_train_data_size(), self.get_test_data_size())\n",
    "        \n",
    "    def get_feature_count(self):\n",
    "        return self.train.shape[1]\n",
    "\n",
    "    def get_train_data_size(self):\n",
    "        return self.train.shape[0]\n",
    "\n",
    "    def get_test_data_size(self):\n",
    "        return self.val.shape[0]\n",
    "        \n",
    "    def get_dataloader(self, train):\n",
    "        features = self.train if train else self.val\n",
    "        labels = self.train_y if train else self.val_y\n",
    "        get_tensor = lambda x: torch.tensor(x.values, dtype=torch.float32)\n",
    "        tensors = (get_tensor(features), get_tensor(labels))\n",
    "        return self.get_tensorloader(tensors, train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set our hyperparameters, initialize weights and biases to log, fit a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = 1\n",
    "lr = 0.01\n",
    "epochs = 10\n",
    "split = 0.5\n",
    "batch_size = 12\n",
    "shuffle = False\n",
    "\n",
    "# start a new wandb run to track this script\n",
    "wandb.init(\n",
    "    # set the wandb project where this run will be logged\n",
    "    project=\"iris-neural-regression\",\n",
    "    name=f\"experiment_{run}\",\n",
    "    # track hyperparameters and run metadata\n",
    "    config={\n",
    "    \"architecture\": \"Linear\",\n",
    "    \"dataset\": \"iris\",\n",
    "    \"learning_rate\": lr,\n",
    "    \"epochs\": epochs,\n",
    "    \"split\": split,\n",
    "    \"batch_size\": batch_size,\n",
    "    \"shuffle\": shuffle\n",
    "    }\n",
    ")\n",
    "\n",
    "# You can get the iris data from https://archive-beta.ics.uci.edu/dataset/53/iris\n",
    "data = CsvData(3,\"data/iris.data\", batch_size=batch_size, shuffle=shuffle, split=split)\n",
    "\n",
    "model = LinearRegression(data.get_feature_count(), 1, lr)\n",
    "\n",
    "trainer = d2l.Trainer(max_epochs=epochs)\n",
    "trainer.fit(model, data)\n",
    "\n",
    "model.trainer.epoch = 10\n",
    "model.plot(None, None, None)\n",
    "\n",
    "wandb.finish()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*What is going on here?* Underfitting or overfitting?\n",
    "\n",
    "What are some possible explanations?\n",
    "\n",
    "How would we fix it?\n",
    "\n",
    "Let's do it again, but change some things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = 2\n",
    "lr = 0.01\n",
    "epochs = 10\n",
    "split = 0.8\n",
    "batch_size = 12\n",
    "shuffle = True\n",
    "\n",
    "# start a new wandb run to track this script\n",
    "wandb.init(\n",
    "    # set the wandb project where this run will be logged\n",
    "    project=\"iris-neural-regression\",\n",
    "    name=f\"experiment_{run}\",\n",
    "    # track hyperparameters and run metadata\n",
    "    config={\n",
    "    \"architecture\": \"Linear\",\n",
    "    \"dataset\": \"iris\",\n",
    "    \"learning_rate\": lr,\n",
    "    \"epochs\": epochs,\n",
    "    \"split\": split,\n",
    "    \"batch_size\": batch_size,\n",
    "    \"shuffle\": shuffle\n",
    "    }\n",
    ")\n",
    "\n",
    "# You can get the iris data from https://archive-beta.ics.uci.edu/dataset/53/iris\n",
    "data = CsvData(3,\"data/iris.data\", batch_size=batch_size, shuffle=shuffle, split=split)\n",
    "\n",
    "model = LinearRegression(data.get_feature_count(), 1, lr)\n",
    "\n",
    "trainer = d2l.Trainer(max_epochs=epochs)\n",
    "trainer.fit(model, data)\n",
    "\n",
    "model.trainer.epoch = 10\n",
    "model.plot(None, None, None)\n",
    "\n",
    "wandb.finish()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Is this better?* \n",
    "\n",
    "*Are we underfitting? Are we overfitting?*\n",
    "\n",
    "One more time, adding **regularization**! Regularization can be used to \"tamp down\" a model architecture without changing the model complexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = 3\n",
    "lr = 0.01\n",
    "epochs = 10\n",
    "split = 0.9\n",
    "batch_size = 12\n",
    "shuffle = True\n",
    "weight_decay = 2\n",
    "\n",
    "# start a new wandb run to track this script\n",
    "wandb.init(\n",
    "    # set the wandb project where this run will be logged\n",
    "    project=\"iris-neural-regression\",\n",
    "    name=f\"experiment_{run}\",\n",
    "    # track hyperparameters and run metadata\n",
    "    config={\n",
    "    \"architecture\": \"Linear\",\n",
    "    \"dataset\": \"iris\",\n",
    "    \"learning_rate\": lr,\n",
    "    \"epochs\": epochs,\n",
    "    \"split\": split,\n",
    "    \"batch_size\": batch_size,\n",
    "    \"shuffle\": shuffle,\n",
    "    \"weight_decay\": weight_decay\n",
    "    }\n",
    ")\n",
    "\n",
    "model = LinearRegression(data.get_feature_count(), 1, lr, weight_decay=weight_decay)\n",
    "\n",
    "trainer = d2l.Trainer(max_epochs=epochs)\n",
    "trainer.fit(model, data)\n",
    "\n",
    "model.trainer.epoch = 10\n",
    "model.plot(None, None, None)\n",
    "\n",
    "wandb.finish()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Better or worse?*\n",
    "\n",
    "*What are some other things we could try?*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another nice explication w/ illustration of overfitting: https://thedatafrog.com/en/articles/overfitting-illustrated/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "3ad933181bd8a04b432d3370b9dc3b0662ad032c4dfaa4e4f1596c548f763858"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}