transformers_multiclass_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine Tuning Transformer for MultiClass Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "In this tutorial we will be fine tuning a transformer model for the **Multiclass text classification** problem. \n",
    "This is one of the most common business problems where a given piece of text/sentence/document needs to be classified into one of the categories out of the given list.\n",
    "\n",
    "#### Flow of the notebook\n",
    "\n",
    "The notebook will be divided into seperate sections to provide a organized walk through for the process used. This process can be modified for individual use cases. The sections are:\n",
    "\n",
    "1. [Importing Python Libraries and preparing the environment](#section01)\n",
    "2. [Importing and Pre-Processing the domain data](#section02)\n",
    "3. [Preparing the Dataset and Dataloader](#section03)\n",
    "4. [Creating the Neural Network for Fine Tuning](#section04)\n",
    "5. [Fine Tuning the Model](#section05)\n",
    "6. [Validating the Model Performance](#section06)\n",
    "7. [Saving the model and artifacts for Inference in Future](#section07)\n",
    "\n",
    "#### Technical Details\n",
    "\n",
    "This script leverages on multiple tools designed by other teams. Details of the tools used below. Please ensure that these elements are present in your setup to successfully implement this script.\n",
    "\n",
    " - Data: \n",
    "\t - We are using the News aggregator dataset available at by [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/News+Aggregator)\n",
    "\t - We are referring only to the first csv file from the data dump: `newsCorpora.csv`\n",
    "\t - There are `422937` rows of data.  Where each row has the following data-point: \n",
    "\t\t - ID Numeric ID  \n",
    "\t\t - TITLE News title  \n",
    "\t\t - URL Url  \n",
    "\t\t - PUBLISHER Publisher name  \n",
    "\t\t - CATEGORY News category (b = business, t = science and technology, e = entertainment, m = health)  \n",
    "\t\t - STORY Alphanumeric ID of the cluster that includes news about the same story  \n",
    "\t\t - HOSTNAME Url hostname  \n",
    "\t\t - TIMESTAMP Approximate time the news was published, as the number of milliseconds since the epoch 00:00:00 GMT, January 1, 1970\n",
    "\n",
    "\n",
    " - Language Model Used:\n",
    "\t - DistilBERT this is a smaller transformer model as compared to BERT or Roberta. It is created by process of distillation applied to Bert. \n",
    "\t - [Blog-Post](https://medium.com/huggingface/distilbert-8cf3380435b5)\n",
    "\t - [Research Paper](https://arxiv.org/abs/1910.01108)\n",
    "     - [Documentation for python](https://huggingface.co/transformers/model_doc/distilbert.html)\n",
    "\n",
    "\n",
    " - Hardware Requirements:\n",
    "\t - Python 3.6 and above\n",
    "\t - Pytorch, Transformers and All the stock Python ML Libraries\n",
    "\t - GPU enabled setup \n",
    "\n",
    "\n",
    " - Script Objective:\n",
    "\t - The objective of this script is to fine tune DistilBERT to be able to classify a news headline into the following categories:\n",
    "\t\t - Business\n",
    "\t\t - Technology\n",
    "\t\t - Health\n",
    "\t\t - Entertainment \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section01'></a>\n",
    "### Importing Python Libraries and preparing the environment\n",
    "\n",
    "At this step we will be importing the libraries and modules needed to run our script. Libraries are:\n",
    "* Pandas\n",
    "* Pytorch\n",
    "* Pytorch Utils for Dataset and Dataloader\n",
    "* Transformers\n",
    "* DistilBERT Model and Tokenizer\n",
    "\n",
    "Followed by that we will preapre the device for CUDA execeution. This configuration is needed if you want to leverage on onboard GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 114,
     "referenced_widgets": [
      "7532a60d077248ca963f514988d41acc",
      "e39121739a4c4bffb16155e9a61a58cc",
      "ca0092edb8f5442a988c87c89930a10e",
      "5a975bc922a14c1b82245195b7c7f659",
      "f9dd503454b2450e957a96bff807c8d9",
      "222af74a298140ff9ec6e6f1f286a52b",
      "3516912b49bf4e3e98c9ce6c7b1b1469",
      "8335299626724853865234b86745e2a5",
      "fd6f40d5854c49f79056cebd16f86626",
      "cde06f09879848a3bfc5892390a51d36",
      "b13c341a69c14c059edc40e6b438ee80",
      "dd7699e7813d4b9f9e80990498a39539",
      "dd589fc95cd94cd190bc2640f9ef618c",
      "4641ff326eea446c88903e70b85c90d7",
      "c1db6876e9b04293b4077f13006b4a19",
      "b8728a6fcdc54b72a10c8bfdcf794fad"
     ]
    },
    "colab_type": "code",
    "id": "wuMlXT80GAMK",
    "outputId": "074dad6a-a18e-45bd-8c9c-29e318962dcd"
   },
   "outputs": [],
   "source": [
    "# Importing the libraries needed\n",
    "import pandas as pd\n",
    "import torch\n",
    "import transformers\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from transformers import DistilBertModel, DistilBertTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xQMKTZ4ARk12"
   },
   "outputs": [],
   "source": [
    "# Setting up the device for GPU usage\n",
    "\n",
    "from torch import cuda\n",
    "device = 'cuda' if cuda.is_available() else 'cpu'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section02'></a>\n",
    "### Importing and Pre-Processing the domain data\n",
    "\n",
    "We will be working with the data and preparing for fine tuning purposes. \n",
    "*Assuming that the `newCorpora.csv` is already downloaded in your `data` folder*\n",
    "\n",
    "Import the file in a dataframe and give it the headers as per the documentation.\n",
    "Cleaning the file to remove the unwanted columns and create an additional column for training.\n",
    "The final Dataframe will be something like this:\n",
    "\n",
    "|TITLE|CATEGORY|ENCODED_CAT|\n",
    "|--|--|--|\n",
    "|  title_1|Entertainment | 1 |\n",
    "|  title_2|Entertainment | 1 |\n",
    "|  title_3|Business| 2 |\n",
    "|  title_4|Science| 3 |\n",
    "|  title_5|Science| 3 |\n",
    "|  title_6|Health| 4 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iNCaZ2epNcSO"
   },
   "outputs": [],
   "source": [
    "# Import the csv into pandas dataframe and add the headers\n",
    "df = pd.read_csv('./data/newsCorpora.csv', sep='\\t', names=['ID','TITLE', 'URL', 'PUBLISHER', 'CATEGORY', 'STORY', 'HOSTNAME', 'TIMESTAMP'])\n",
    "# df.head()\n",
    "# # Removing unwanted columns and only leaving title of news and the category which will be the target\n",
    "df = df[['TITLE','CATEGORY']]\n",
    "# df.head()\n",
    "\n",
    "# # Converting the codes to appropriate categories using a dictionary\n",
    "my_dict = {\n",
    "    'e':'Entertainment',\n",
    "    'b':'Business',\n",
    "    't':'Science',\n",
    "    'm':'Health'\n",
    "}\n",
    "\n",
    "def update_cat(x):\n",
    "    return my_dict[x]\n",
    "\n",
    "df['CATEGORY'] = df['CATEGORY'].apply(lambda x: update_cat(x))\n",
    "\n",
    "encode_dict = {}\n",
    "\n",
    "def encode_cat(x):\n",
    "    if x not in encode_dict.keys():\n",
    "        encode_dict[x]=len(encode_dict)\n",
    "    return encode_dict[x]\n",
    "\n",
    "df['ENCODE_CAT'] = df['CATEGORY'].apply(lambda x: encode_cat(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section03'></a>\n",
    "### Preparing the Dataset and Dataloader\n",
    "\n",
    "We will start with defining few key variables that will be used later during the training/fine tuning stage.\n",
    "Followed by creation of Dataset class - This defines how the text is pre-processed before sending it to the neural network. We will also define the Dataloader that will feed  the data in batches to the neural network for suitable training and processing. \n",
    "Dataset and Dataloader are constructs of the PyTorch library for defining and controlling the data pre-processing and its passage to neural network. For further reading into Dataset and Dataloader read the [docs at PyTorch](https://pytorch.org/docs/stable/data.html)\n",
    "\n",
    "#### *Triage* Dataset Class\n",
    "- This class is defined to accept the Dataframe as input and generate tokenized output that is used by the DistilBERT model for training. \n",
    "- We are using the DistilBERT tokenizer to tokenize the data in the `TITLE` column of the dataframe. \n",
    "- The tokenizer uses the `encode_plus` method to perform tokenization and generate the necessary outputs, namely: `ids`, `attention_mask`\n",
    "- To read further into the tokenizer, [refer to this document](https://huggingface.co/transformers/model_doc/distilbert.html#distilberttokenizer)\n",
    "- `target` is the encoded category on the news headline. \n",
    "- The *Triage* class is used to create 2 datasets, for training and for validation.\n",
    "- *Training Dataset* is used to fine tune the model: **80% of the original data**\n",
    "- *Validation Dataset* is used to evaluate the performance of the model. The model has not seen this data during training. \n",
    "\n",
    "#### Dataloader\n",
    "- Dataloader is used to for creating training and validation dataloader that load data to the neural network in a defined manner. This is needed because all the data from the dataset cannot be loaded to the memory at once, hence the amount of dataloaded to the memory and then passed to the neural network needs to be controlled.\n",
    "- This control is achieved using the parameters such as `batch_size` and `max_len`.\n",
    "- Training and Validation dataloaders are used in the training and validation part of the flow respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JrBr2YesGdO_"
   },
   "outputs": [],
   "source": [
    "# Defining some key variables that will be used later on in the training\n",
    "MAX_LEN = 512\n",
    "TRAIN_BATCH_SIZE = 4\n",
    "VALID_BATCH_SIZE = 2\n",
    "EPOCHS = 1\n",
    "LEARNING_RATE = 1e-05\n",
    "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2vX7kzaAHu39"
   },
   "outputs": [],
   "source": [
    "class Triage(Dataset):\n",
    "    def __init__(self, dataframe, tokenizer, max_len):\n",
    "        self.len = len(dataframe)\n",
    "        self.data = dataframe\n",
    "        self.tokenizer = tokenizer\n",
    "        self.max_len = max_len\n",
    "        \n",
    "    def __getitem__(self, index):\n",
    "        title = str(self.data.TITLE[index])\n",
    "        title = \" \".join(title.split())\n",
    "        inputs = self.tokenizer.encode_plus(\n",
    "            title,\n",
    "            None,\n",
    "            add_special_tokens=True,\n",
    "            max_length=self.max_len,\n",
    "            pad_to_max_length=True,\n",
    "            return_token_type_ids=True,\n",
    "            truncation=True\n",
    "        )\n",
    "        ids = inputs['input_ids']\n",
    "        mask = inputs['attention_mask']\n",
    "\n",
    "        return {\n",
    "            'ids': torch.tensor(ids, dtype=torch.long),\n",
    "            'mask': torch.tensor(mask, dtype=torch.long),\n",
    "            'targets': torch.tensor(self.data.ENCODE_CAT[index], dtype=torch.long)\n",
    "        } \n",
    "    \n",
    "    def __len__(self):\n",
    "        return self.len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Zcwq13c0NE9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FULL Dataset: (422419, 3)\n",
      "TRAIN Dataset: (337935, 3)\n",
      "TEST Dataset: (84484, 3)\n"
     ]
    }
   ],
   "source": [
    "# Creating the dataset and dataloader for the neural network\n",
    "\n",
    "train_size = 0.8\n",
    "train_dataset=df.sample(frac=train_size,random_state=200)\n",
    "test_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
    "train_dataset = train_dataset.reset_index(drop=True)\n",
    "\n",
    "\n",
    "print(\"FULL Dataset: {}\".format(df.shape))\n",
    "print(\"TRAIN Dataset: {}\".format(train_dataset.shape))\n",
    "print(\"TEST Dataset: {}\".format(test_dataset.shape))\n",
    "\n",
    "training_set = Triage(train_dataset, tokenizer, MAX_LEN)\n",
    "testing_set = Triage(test_dataset, tokenizer, MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l1BgA1CkQSYa"
   },
   "outputs": [],
   "source": [
    "train_params = {'batch_size': TRAIN_BATCH_SIZE,\n",
    "                'shuffle': True,\n",
    "                'num_workers': 0\n",
    "                }\n",
    "\n",
    "test_params = {'batch_size': VALID_BATCH_SIZE,\n",
    "                'shuffle': True,\n",
    "                'num_workers': 0\n",
    "                }\n",
    "\n",
    "training_loader = DataLoader(training_set, **train_params)\n",
    "testing_loader = DataLoader(testing_set, **test_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section04'></a>\n",
    "### Creating the Neural Network for Fine Tuning\n",
    "\n",
    "#### Neural Network\n",
    " - We will be creating a neural network with the `DistillBERTClass`. \n",
    " - This network will have the DistilBERT Language model followed by a `dropout` and finally a `Linear` layer to obtain the final outputs. \n",
    " - The data will be fed to the DistilBERT Language model as defined in the dataset. \n",
    " - Final layer outputs is what will be compared to the `encoded category` to determine the accuracy of models prediction. \n",
    " - We will initiate an instance of the network called `model`. This instance will be used for training and then to save the final trained model for future inference. \n",
    " \n",
    "#### Loss Function and Optimizer\n",
    " - `Loss Function` and `Optimizer` and defined in the next cell.\n",
    " - The `Loss Function` is used the calculate the difference in the output created by the model and the actual output. \n",
    " - `Optimizer` is used to update the weights of the neural network to improve its performance.\n",
    " \n",
    "#### Further Reading\n",
    "- You can refer to my [Pytorch Tutorials](https://github.com/abhimishra91/pytorch-tutorials) to get an intuition of Loss Function and Optimizer.\n",
    "- [Pytorch Documentation for Loss Function](https://pytorch.org/docs/stable/nn.html#loss-functions)\n",
    "- [Pytorch Documentation for Optimizer](https://pytorch.org/docs/stable/optim.html)\n",
    "- Refer to the links provided on the top of the notebook to read more about DistiBERT. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the customized model, by adding a drop out and a dense layer on top of distil bert to get the final output for the model. \n",
    "\n",
    "class DistillBERTClass(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(DistillBERTClass, self).__init__()\n",
    "        self.l1 = DistilBertModel.from_pretrained(\"distilbert-base-uncased\")\n",
    "        self.pre_classifier = torch.nn.Linear(768, 768)\n",
    "        self.dropout = torch.nn.Dropout(0.3)\n",
    "        self.classifier = torch.nn.Linear(768, 4)\n",
    "\n",
    "    def forward(self, input_ids, attention_mask):\n",
    "        output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)\n",
    "        hidden_state = output_1[0]\n",
    "        pooler = hidden_state[:, 0]\n",
    "        pooler = self.pre_classifier(pooler)\n",
    "        pooler = torch.nn.ReLU()(pooler)\n",
    "        pooler = self.dropout(pooler)\n",
    "        output = self.classifier(pooler)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DistillBERTClass(\n",
       "  (l1): DistilBertModel(\n",
       "    (embeddings): Embeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (transformer): Transformer(\n",
       "      (layer): ModuleList(\n",
       "        (0): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "        (1): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "        (2): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "        (3): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "        (4): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "        (5): TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (l2): Dropout(p=0.3, inplace=False)\n",
       "  (l3): Linear(in_features=768, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = DistillBERTClass()\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the loss function and optimizer\n",
    "loss_function = torch.nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(params =  model.parameters(), lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section05'></a>\n",
    "### Fine Tuning the Model\n",
    "\n",
    "After all the effort of loading and preparing the data and datasets, creating the model and defining its loss and optimizer. This is probably the easier steps in the process. \n",
    "\n",
    "Here we define a training function that trains the model on the training dataset created above, specified number of times (EPOCH), An epoch defines how many times the complete data will be passed through the network. \n",
    "\n",
    "Following events happen in this function to fine tune the neural network:\n",
    "- The dataloader passes data to the model based on the batch size. \n",
    "- Subsequent output from the model and the actual category are compared to calculate the loss. \n",
    "- Loss value is used to optimize the weights of the neurons in the network.\n",
    "- After every 5000 steps the loss value is printed in the console.\n",
    "\n",
    "As you can see just in 1 epoch by the final step the model was working with a miniscule loss of 0.0002485 i.e. the output is extremely close to the actual output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to calcuate the accuracy of the model\n",
    "\n",
    "def calcuate_accu(big_idx, targets):\n",
    "    n_correct = (big_idx==targets).sum().item()\n",
    "    return n_correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the training function on the 80% of the dataset for tuning the distilbert model\n",
    "\n",
    "def train(epoch):\n",
    "    tr_loss = 0\n",
    "    n_correct = 0\n",
    "    nb_tr_steps = 0\n",
    "    nb_tr_examples = 0\n",
    "    model.train()\n",
    "    for _,data in enumerate(training_loader, 0):\n",
    "        ids = data['ids'].to(device, dtype = torch.long)\n",
    "        mask = data['mask'].to(device, dtype = torch.long)\n",
    "        targets = data['targets'].to(device, dtype = torch.long)\n",
    "\n",
    "        outputs = model(ids, mask)\n",
    "        loss = loss_function(outputs, targets)\n",
    "        tr_loss += loss.item()\n",
    "        big_val, big_idx = torch.max(outputs.data, dim=1)\n",
    "        n_correct += calcuate_accu(big_idx, targets)\n",
    "\n",
    "        nb_tr_steps += 1\n",
    "        nb_tr_examples+=targets.size(0)\n",
    "        \n",
    "        if _%5000==0:\n",
    "            loss_step = tr_loss/nb_tr_steps\n",
    "            accu_step = (n_correct*100)/nb_tr_examples \n",
    "            print(f\"Training Loss per 5000 steps: {loss_step}\")\n",
    "            print(f\"Training Accuracy per 5000 steps: {accu_step}\")\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        # # When using GPU\n",
    "        optimizer.step()\n",
    "\n",
    "    print(f'The Total Accuracy for Epoch {epoch}: {(n_correct*100)/nb_tr_examples}')\n",
    "    epoch_loss = tr_loss/nb_tr_steps\n",
    "    epoch_accu = (n_correct*100)/nb_tr_examples\n",
    "    print(f\"Training Loss Epoch: {epoch_loss}\")\n",
    "    print(f\"Training Accuracy Epoch: {epoch_accu}\")\n",
    "\n",
    "    return "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Loss:  6.332988739013672\n",
      "Epoch: 0, Loss:  0.0013066530227661133\n",
      "Epoch: 0, Loss:  0.0029534101486206055\n",
      "Epoch: 0, Loss:  0.005258679389953613\n",
      "Epoch: 0, Loss:  0.0020235776901245117\n",
      "Epoch: 0, Loss:  0.0023298263549804688\n",
      "Epoch: 0, Loss:  0.0034378767013549805\n",
      "Epoch: 0, Loss:  0.004993081092834473\n",
      "Epoch: 0, Loss:  0.008559942245483398\n",
      "Epoch: 0, Loss:  0.0014510154724121094\n",
      "Epoch: 0, Loss:  0.0028634071350097656\n",
      "Epoch: 0, Loss:  0.0006411075592041016\n",
      "Epoch: 0, Loss:  0.0012137889862060547\n",
      "Epoch: 0, Loss:  0.002307891845703125\n",
      "Epoch: 0, Loss:  0.00028586387634277344\n",
      "Epoch: 0, Loss:  0.0029143095016479492\n",
      "Epoch: 0, Loss:  0.0002485513687133789\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCHS):\n",
    "    train(epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section06'></a>\n",
    "### Validating the Model\n",
    "\n",
    "During the validation stage we pass the unseen data(Testing Dataset) to the model. This step determines how good the model performs on the unseen data. \n",
    "\n",
    "This unseen data is the 20% of `newscorpora.csv` which was seperated during the Dataset creation stage. \n",
    "During the validation stage the weights of the model are not updated. Only the final output is compared to the actual value. This comparison is then used to calcuate the accuracy of the model. \n",
    "\n",
    "As you can see the model is predicting the correct category of a given headline to a 99.9% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def valid(model, testing_loader):\n",
    "    model.eval()\n",
    "    n_correct = 0; n_wrong = 0; total = 0\n",
    "    with torch.no_grad():\n",
    "        for _, data in enumerate(testing_loader, 0):\n",
    "            ids = data['ids'].to(device, dtype = torch.long)\n",
    "            mask = data['mask'].to(device, dtype = torch.long)\n",
    "            targets = data['targets'].to(device, dtype = torch.long)\n",
    "            outputs = model(ids, mask).squeeze()\n",
    "            loss = loss_function(outputs, targets)\n",
    "            tr_loss += loss.item()\n",
    "            big_val, big_idx = torch.max(outputs.data, dim=1)\n",
    "            n_correct += calcuate_accu(big_idx, targets)\n",
    "\n",
    "            nb_tr_steps += 1\n",
    "            nb_tr_examples+=targets.size(0)\n",
    "            \n",
    "            if _%5000==0:\n",
    "                loss_step = tr_loss/nb_tr_steps\n",
    "                accu_step = (n_correct*100)/nb_tr_examples\n",
    "                print(f\"Validation Loss per 100 steps: {loss_step}\")\n",
    "                print(f\"Validation Accuracy per 100 steps: {accu_step}\")\n",
    "    epoch_loss = tr_loss/nb_tr_steps\n",
    "    epoch_accu = (n_correct*100)/nb_tr_examples\n",
    "    print(f\"Validation Loss Epoch: {epoch_loss}\")\n",
    "    print(f\"Validation Accuracy Epoch: {epoch_accu}\")\n",
    "    \n",
    "    return epoch_accu\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the validation section to print the accuracy and see how it performs\n",
      "Here we are leveraging on the dataloader crearted for the validation dataset, the approcah is using more of pytorch\n",
      "Accuracy on test data = 99.99%\n"
     ]
    }
   ],
   "source": [
    "print('This is the validation section to print the accuracy and see how it performs')\n",
    "print('Here we are leveraging on the dataloader crearted for the validation dataset, the approcah is using more of pytorch')\n",
    "\n",
    "acc = valid(model, testing_loader)\n",
    "print(\"Accuracy on test data = %0.2f%%\" % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section07'></a>\n",
    "### Saving the Trained Model Artifacts for inference\n",
    "\n",
    "This is the final step in the process of fine tuning the model. \n",
    "\n",
    "The model and its vocabulary are saved locally. These files are then used in the future to make inference on new inputs of news headlines.\n",
    "\n",
    "Please remember that a trained neural network is only useful when used in actual inference after its training. \n",
    "\n",
    "In the lifecycle of an ML projects this is only half the job done. We will leave the inference of these models for some other day. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All files saved\n",
      "This tutorial is completed\n"
     ]
    }
   ],
   "source": [
    "# Saving the files for re-use\n",
    "\n",
    "output_model_file = './models/pytorch_distilbert_news.bin'\n",
    "output_vocab_file = './models/vocab_distilbert_news.bin'\n",
    "\n",
    "model_to_save = model\n",
    "torch.save(model_to_save, output_model_file)\n",
    "tokenizer.save_vocabulary(output_vocab_file)\n",
    "\n",
    "print('All files saved')\n",
    "print('This tutorial is completed')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "01_transformers_multiclass_classification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "222af74a298140ff9ec6e6f1f286a52b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3516912b49bf4e3e98c9ce6c7b1b1469": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4641ff326eea446c88903e70b85c90d7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a975bc922a14c1b82245195b7c7f659": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8335299626724853865234b86745e2a5",
      "placeholder": "​",
      "style": "IPY_MODEL_3516912b49bf4e3e98c9ce6c7b1b1469",
      "value": " 442/442 [00:01&lt;00:00, 276B/s]"
     }
    },
    "7532a60d077248ca963f514988d41acc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ca0092edb8f5442a988c87c89930a10e",
       "IPY_MODEL_5a975bc922a14c1b82245195b7c7f659"
      ],
      "layout": "IPY_MODEL_e39121739a4c4bffb16155e9a61a58cc"
     }
    },
    "8335299626724853865234b86745e2a5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b13c341a69c14c059edc40e6b438ee80": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4641ff326eea446c88903e70b85c90d7",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dd589fc95cd94cd190bc2640f9ef618c",
      "value": 231508
     }
    },
    "b8728a6fcdc54b72a10c8bfdcf794fad": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c1db6876e9b04293b4077f13006b4a19": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ca0092edb8f5442a988c87c89930a10e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_222af74a298140ff9ec6e6f1f286a52b",
      "max": 442,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f9dd503454b2450e957a96bff807c8d9",
      "value": 442
     }
    },
    "cde06f09879848a3bfc5892390a51d36": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dd589fc95cd94cd190bc2640f9ef618c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "dd7699e7813d4b9f9e80990498a39539": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b8728a6fcdc54b72a10c8bfdcf794fad",
      "placeholder": "​",
      "style": "IPY_MODEL_c1db6876e9b04293b4077f13006b4a19",
      "value": " 232k/232k [00:00&lt;00:00, 584kB/s]"
     }
    },
    "e39121739a4c4bffb16155e9a61a58cc": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f9dd503454b2450e957a96bff807c8d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "fd6f40d5854c49f79056cebd16f86626": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b13c341a69c14c059edc40e6b438ee80",
       "IPY_MODEL_dd7699e7813d4b9f9e80990498a39539"
      ],
      "layout": "IPY_MODEL_cde06f09879848a3bfc5892390a51d36"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}