fasthugs_seq_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastHugs\n",
    "This notebook gives a full run through to fine-tune a text classification model with **HuggingFace 🤗 transformers** and the new **fastai-v2** library.\n",
    "\n",
    "## Things You Might Like (❤️ ?)\n",
    "**FastHugsTokenizer:** A tokenizer wrapper than can be used with fastai-v2's tokenizer.\n",
    "\n",
    "**FastHugsModel:** A model wrapper over the HF models, more or less the same to the wrapper's from HF fastai-v1 articles mentioned below\n",
    "\n",
    "**Padding:** Padding settings for the padding token index and on whether the transformer prefers left or right padding\n",
    "\n",
    "**Model Splitters:** Functions to split the classification head from the model backbone in line with fastai-v2's new definition of `Learner` (in `splitters.py`\n",
    "\n",
    "## Housekeeping\n",
    "### Pretrained Transformers only for now 😐\n",
    "Initially, this notebook will only deal with finetuning HuggingFace's pretrained models. It covers BERT, DistilBERT, RoBERTa and ALBERT pretrained classification models only. These are the core transformer model architectures where HuggingFace have added a classification head. HuggingFace also has other versions of these model architectures such as the core model architecture and language model model architectures.\n",
    "\n",
    "If you'd like to try train a model from scratch HuggingFace just recently published an article on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train). Its well worth reading to see how their `tokenizers` library can be used, independent of their pretrained transformer models.\n",
    "\n",
    "### Read these first 👇\n",
    "This notebooks heavily borrows from [this notebook](https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers) , which in turn is based off of this [tutorial](https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta) and accompanying [article](https://towardsdatascience.com/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2). Huge thanks to  Melissa Rajaram and Maximilien Roberti for these great resources, if you're not familiar with the HuggingFace library please given them a read first as they are quite comprehensive.\n",
    "\n",
    "### fastai-v2  ✌️2️⃣\n",
    "[This paper](https://www.fast.ai/2020/02/13/fastai-A-Layered-API-for-Deep-Learning/) introduces the v2 version of the fastai library and you can follow and contribute to v2's progress [on the forums](https://forums.fast.ai/). This notebook uses the small IMDB dataset and is based off the [fastai-v2 ULMFiT tutorial](http://dev.fast.ai/tutorial.ulmfit). Huge thanks to Jeremy, Sylvain, Rachel and the fastai community for making this library what it is. I'm super excited about the additinal flexibility v2 brings. 🎉\n",
    "\n",
    "### Dependencies 📥\n",
    "If you haven't already, install HuggingFace's `transformers` library with: `pip install transformers`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "# CUDA ERROR DEBUGGING\n",
    "# https://lernapparat.de/debug-device-assert/\n",
    "import os\n",
    "os.environ['CUDA_LAUNCH_BLOCKING'] = \"1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from fastai2.basics import *\n",
    "from fastai2.text.all import *\n",
    "from fastai2.callback.all import *\n",
    "\n",
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoConfig\n",
    "from splitters import *\n",
    "\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "model_path = Path('models')\n",
    "df = pd.read_csv(path/'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastHugs Tokenizer\n",
    "This tokenizer wrapper is initialised with the pretrained HF tokenizer, you can also specify the max_seq_len if you want longer/shorter sequences. Given text it returns tokens and adds separator tokens depending on the model type being used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FastHugsTokenizer():\n",
    "    \"\"\" \n",
    "        transformer_tokenizer : takes the tokenizer that has been loaded from the tokenizer class\n",
    "        model_name : model type set by the user\n",
    "        max_seq_len : override default sequence length, typically 512 for bert-like models.\n",
    "                           `transformer_tokenizer.max_len_single_sentence` and `transformer_tokenizer.max_len_sentences_pair` \n",
    "                           both account for the need to add additional special tokens, i.e. for RoBERTa-base \n",
    "                           max_len_single_sentence==510, leaving space for the 2 additional special tokens \n",
    "                           to be added for the model's default 512 positional embeddings\n",
    "        pair : whether a single sentence (sequence) or pair of sentences are used\n",
    "        \n",
    "        NOTES:\n",
    "            - `init` will have to be modified to enable sequence lengths larger than the tokenizer default\n",
    "            - need to add case when pretrained==False\n",
    "            - Pretrained==True will cut the sequence at the max length\n",
    "            - Good functions in `tokenization_utils.py`\n",
    "            - tokenizer.encode_plus or tokenizer.batch_encode_plus are great, but don't play nice with fastai multiprocessiing\n",
    "            - https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode_plus\n",
    "            - encoded_dict=tokenizer.encode_plus(text=o, return_tensors=\"pt\", max_length=tokenizer.max_len, pad_to_max_length=True)\n",
    "        Returns:\n",
    "            - Tokenized text, up to the max sequence length set by the user or the tokenzier default\n",
    "    \"\"\"\n",
    "    def __init__(self, transformer_tokenizer=None, model_name='roberta', max_seq_len=None, \n",
    "                 pretrained=True, pair=False, **kwargs): \n",
    "        self.model_name, self.tok, self.max_seq_len=model_name, transformer_tokenizer, max_seq_len\n",
    "        if pretrained:\n",
    "            if self.max_seq_len:\n",
    "                if pair: assert self.max_seq_len<=self.tok.max_len_sentences_pair, 'WARNING: max_seq_len needs to be less than or equal to transformer_tokenizer.max_len_sentences_pair'\n",
    "                else: assert self.max_seq_len<=self.tok.max_len_single_sentence, 'WARNING: max_seq_len needs to be less than or equal to transformer_tokenizer.max_len_single_sentence'\n",
    "            else:\n",
    "                if pair: self.max_seq_len=ifnone(max_seq_len, self.tok.max_len_sentences_pair) \n",
    "                else: self.max_seq_len=ifnone(max_seq_len, self.tok.max_len_single_sentence)\n",
    "\n",
    "    def do_tokenize(self, o:str):\n",
    "        \"\"\"Returns tokenized text, adds prefix space if needed, limits the maximum sequence length\"\"\"\n",
    "        if 'roberta' in model_name: tokens=self.tok.tokenize(o, add_prefix_space=True)[:self.max_seq_len-2]\n",
    "        else: tokens = self.tok.tokenize(o)[:self.max_seq_len-2]\n",
    "        return tokens\n",
    "    def __call__(self, items): \n",
    "        for o in items: yield self.do_tokenize(o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastHugs Model\n",
    "This `nn.module` wraps the pretrained transformer model and initialises it with its config file. \n",
    "\n",
    "The `forward` of this module is taken straight from Melissa's notebook above and its purpose is to create the attention mask and grab only the logits from the output of the model (as the HappyFace transformer models also output the loss)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FastHugsModel(nn.Module):\n",
    "    'Inspired by https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers/data'\n",
    "    def __init__(self, transformer_cls, tokenizer, config_dict, n_class, pretrained=True):\n",
    "        super(FastHugsModel, self).__init__()\n",
    "        self.tok, self.config, self.config._num_labels = tokenizer, config_dict, n_class\n",
    "        # load model\n",
    "        if pretrained: self.transformer = transformer_cls.from_pretrained(model_name, config=self.config)\n",
    "        else: self.transformer = transformer_cls.from_config(config=self.config)\n",
    "        \n",
    "    def forward(self, input_ids, attention_mask=None):\n",
    "        attention_mask = (input_ids!=self.tok.pad_token_id).type(input_ids.type())\n",
    "        logits = self.transformer(input_ids, attention_mask = attention_mask)[0] \n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The HuggingFace bit\n",
    "\n",
    "### Define HuggingFace Model + Config\n",
    "- `AutoModelForSequenceClassification` will define our model. When this is padded to the `FastHugsModel` class below then model will be instantiated and the weights downloaded (if you are using a pretrained model)\n",
    "- `AutoConfig` will define the model architecture and settings\n",
    "- `model_name` is the model architecture (and optionally model weights) you'd like to use.\n",
    "    - Models tested: `bert-base-uncased`, `roberta-base`, `distilbert-base-cased`, `albert-base-v2`\n",
    "    - You can find all of HuggingFace's models at [https://huggingface.co/models](https://huggingface.co/models), although not all of them are supported by `AutoModel`,`AutoConfig` and `AutoTokenizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_name = 'roberta-base' \n",
    "model_class = AutoModelForSequenceClassification\n",
    "config_dict = AutoConfig.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HuggingFace Config changes\n",
    "Some config settings can be changed even when using pretrained weights. For example in the `FastHugsModel` class below `_num_labels` is set when the model (pretrained or not) is instantiated, depending on how many classes you have in your dataloader.\n",
    "\n",
    "When creating a **non-pretrained** model you can load a config with:\n",
    "\n",
    "> `config_dict = AutoConfig.for_model(model_name)`\n",
    "\n",
    "Alternatively you could load a pretrained config and modify that. For example if your are **not** using a pretrained model you can change the size of your input embeddings by changing `config_dict.max_position_embeddings = 1024`. (This won't work when using pretrained models as the pre-trained weights need the default `max_position_embeddings` size)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HuggingFace Tokenizer & Vocab\n",
    "- `AutoTokenizer` will load our tokenizer and enable us grab our vocab\n",
    "\n",
    "fastai expects `vocab` to be a list, however HuggingFace's `get_vocab` returns a `token : index` dict. We need to convert this dict to a list to be able to use it in fastai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50265"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "tokenizer_vocab=tokenizer.get_vocab() \n",
    "tokenizer_vocab_ls = [k for k, v in sorted(tokenizer_vocab.items(), key=lambda item: item[1])]\n",
    "len(tokenizer_vocab_ls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Special Tokens\n",
    "Its always good to know what special tokens your tokenizer needs, lets have a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bos_token': '<s>',\n",
       " 'eos_token': '</s>',\n",
       " 'unk_token': '<unk>',\n",
       " 'sep_token': '</s>',\n",
       " 'pad_token': '<pad>',\n",
       " 'cls_token': '<s>',\n",
       " 'mask_token': '<mask>'}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.special_tokens_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Fastai bit\n",
    "### Get fastai model splitter function\n",
    "In order to be able to fine-tune our classifier head we need to first split the HuggingFace model's classifier head from the body. These functions are dependent on the specific architecture and can be found in `splitter.py` of this repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitter_nm = model_name.split('-')[0] + '_cls_splitter'\n",
    "model_splitter = splitters[splitter_nm]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `fasthugstok` and our `tok_fn`\n",
    "Lets incorporate the `tokenizer` from HuggingFace into fastai-v2's framework by specifying a function called `fasthugstok` that we can then pass on to `Tokenizer.from_df`. (Note `.from_df` is the only method I have tested)\n",
    "\n",
    "#### Max Seqence Length\n",
    "`max_seq_len` is the longest sequece our tokenizer will output. We can also the max sequence length for the tokenizer by changing `max_seq_len`. It uses the tokenizer's default, typically `512`. `1024` or even `2048` can also be used depending on your GPU memory. Note when using pretrained models you won't be able to use a `max_seq_len` larger than the default. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "max_seq_len = 512  \n",
    "sentence_pair=False\n",
    "\n",
    "fasthugstok = partial(FastHugsTokenizer, transformer_tokenizer=tokenizer, model_name=model_name, \n",
    "                      max_seq_len=max_seq_len, sentence_pair=sentence_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up fastai's `Tokenizer.from_df`, we pass `rules=[]` to override fastai's default text processing rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastai_tokenizer = Tokenizer.from_df(text_cols='text', res_col_name='text', tok_func=fasthugstok, \n",
    "                                     rules=[], post_rules=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fastai_tokenizer.rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Special Tokens\n",
    "BERT-like transformers require special tokens to be added to the sequence, depending on the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SpecialClsTokens(Transform):\n",
    "    \"Add special token_ids to the numericalized tokens for Sequence Classification\"\n",
    "    def __init__(self, tokenizer):\n",
    "        self.tok=tokenizer\n",
    "    def encodes(self, o):\n",
    "        return(TensorText(self.tok.build_inputs_with_special_tokens(list(o))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt=[\"If you want the real version of this over blown American clown act, watch William Wylers' 1944 version - the true story of the 'Memphis Belle'. It's amazing what Hollywood will do to distort history and mock its' veterans, all for a buck. Well it must be the American way! Younger viewers will be beguiled by the nonsense, however older viewers with some sense of history will recognize this movie for what it is worth. Don't waste your time! However, if you don't want the truth, then put your mind in neutral and watch this movie.\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fht=FastHugsTokenizer(transformer_tokenizer=tokenizer, model_name='roberta', max_seq_len=max_seq_len, \n",
    "                 pretrained=True, pair=False)\n",
    "tokenized_text = next(fht(txt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test Numericalize working the same as HuggingFace\n",
    "test_eq(Numericalize(vocab=tokenizer_vocab_ls)(tokenized_text),\n",
    "        TensorText(tokenizer.convert_tokens_to_ids(tokenized_text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pre_special length: 118, with_special length: 120\n"
     ]
    }
   ],
   "source": [
    "# Check that additional tokens are being added\n",
    "pre_special=Numericalize(vocab=tokenizer_vocab_ls)(tokenized_text)\n",
    "with_special=SpecialClsTokens(tokenizer)(pre_special)\n",
    "print(f'pre_special length: {len(pre_special)}, with_special length: {len(with_special)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Dataloaders\n",
    "### Create Dataset\n",
    "Lets add our custom tokenizer function (`tok_fn`) and `transformer_vocab` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# class DumTfm(Transform):\n",
    "#     \"Add special token_ids to the numericalized tokens for Sequence Classification\"\n",
    "#     def __init__(self, tokenizer):\n",
    "#         self.tok=tokenizer\n",
    "#     def encodes(self, o):\n",
    "#         return o\n",
    "#     def decodes(self, o):\n",
    "#         return self.tok.decode(o.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "splits = ColSplitter()(df)\n",
    "x_tfms = [attrgetter(\"text\"), fastai_tokenizer, Numericalize(vocab=tokenizer_vocab_ls), SpecialClsTokens(tokenizer)]\n",
    "\n",
    "#x_tfms = [attrgetter(\"text\"), fastai_tokenizer, DumTfm(tokenizer)]\n",
    "dsets = Datasets(df, splits=splits, tfms=[x_tfms, [attrgetter(\"label\"), Categorize()]], dl_type=SortedDL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max len fixing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://github.com/huggingface/transformers/blob/a21d4fa410dc3b4c62f93aa0e6bbe4b75a101ee9/src/transformers/tokenization_utils.py#L1775\n",
    "\n",
    "# add_special_tokens=True\n",
    "    \n",
    "    \n",
    "# pair = bool(pair_ids is not None)\n",
    "# len_ids = len(ids)\n",
    "# len_pair_ids = len(pair_ids) if pair else 0\n",
    "\n",
    "# # Load from model defaults\n",
    "# if return_token_type_ids is None:\n",
    "#     return_token_type_ids = \"token_type_ids\" in self.model_input_names\n",
    "# if return_attention_mask is None:\n",
    "#     return_attention_mask = \"attention_mask\" in self.model_input_names\n",
    "\n",
    "# encoded_inputs = {}\n",
    "\n",
    "# # Truncation: Handle max sequence length\n",
    "# total_len = len_ids + len_pair_ids + (self.num_special_tokens_to_add(pair=pair) if add_special_tokens else 0)\n",
    "\n",
    "# if max_length and total_len > max_length:\n",
    "#     ids, pair_ids, overflowing_tokens = self.truncate_sequences(\n",
    "#         ids,\n",
    "#         pair_ids=pair_ids,\n",
    "#         num_tokens_to_remove=total_len - max_length,\n",
    "#         truncation_strategy=truncation_strategy,\n",
    "#         stride=stride,\n",
    "#     )\n",
    "#     if return_overflowing_tokens:\n",
    "#         encoded_inputs[\"overflowing_tokens\"] = overflowing_tokens\n",
    "#         encoded_inputs[\"num_truncated_tokens\"] = total_len - max_length\n",
    "\n",
    "# # Add special tokens\n",
    "# if add_special_tokens:\n",
    "#     sequence = self.build_inputs_with_special_tokens(ids, pair_ids)\n",
    "#     token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)\n",
    "# else:\n",
    "#     sequence = ids + pair_ids if pair else ids\n",
    "#     token_type_ids = [0] * len(ids) + ([1] * len(pair_ids) if pair else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What to do with `token_type_ids`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Padding\n",
    "We need to make sure our padding is done correctly as some transformer models prefer padding on the left while others prefer it on the right. `tokenizer.padding_side` will tell us which side is correct. e.g., BERT, Roberta prefers padding to the right, so we set `pad_first=False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transformer_padding(tokenizer=None, max_seq_len=None, sentence_pair=False): \n",
    "    if tokenizer.padding_side == 'right': pad_first=False\n",
    "    else: pad_first=True\n",
    "    max_seq_len = ifnone(max_seq_len, tokenizer.max_len) \n",
    "    return partial(pad_input_chunk, pad_first=pad_first, pad_idx=tokenizer.pad_token_id, seq_len=max_seq_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 4\n",
    "padding=transformer_padding(tokenizer, max_seq_len)\n",
    "dls = dsets.dataloaders(bs=bs, before_batch=[padding])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check our batch\n",
    "We can see our special RoBERTa tokens (`'<s>'`, `'</s>'`), which translate to `0, 2` in its vocab, have been added to the start and end of each sequence in the batch. Your can look at these indices in `tokenizer.get_vocab()` to confirm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 512]),\n",
       " torch.Size([4]),\n",
       " TensorText([[    0,    38,    21,  ...,    54,     5,     2],\n",
       "         [    0,   286,   167,  ..., 49069,  3809,     2],\n",
       "         [    0,    38,  3805,  ...,    10,   356,     2],\n",
       "         [    0,  4913,  3748,  ...,     7,    28,     2]], device='cuda:0'))"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "o=dls.one_batch(); o[0].size(), o[1].size(), o[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;s&gt; ĠI Ġwas Ġfortunate Ġenough Ġto Ġmeet ĠGeorge ĠPal Ġ( and Ġstill Ġhave Ġmy ĠDS : TM OB Ġposter Ġaut ographed Ġby Ġhim ) Ġat Ġa Ġconvention Ġshortly Ġafter Ġthe Ġrelease , Ġand Ġasked Ġhim Ġwhy Ġhe Ġchose Ġto Ġdo Ġthe Ġfilm Ġ\" camp \". ĠBefore Ġhe Ġcould Ġanswer , Ġtwo Ġstudio Ġfl acks Ġintercepted Ġand Ġlect ured Ġme Ġon</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;s&gt; ĠI Ġdo Ġmiss Ġthe Ġcompany ĠVest ron , Ġthey Ġsure Ġhad Ġtheir Ġfinger Ġon Ġthe Ġpulse Ġof Ġunique Ġand Ġunusual Ġcinema Ġback Ġin Ġthe Ġ1980 s . ĠThis Ġis Ġvery Ġapparent Ġwith Ġthe Ġastonishing ĠPaper house , Ġa Ġfilm Ġthat Ġtouches Ġme Ġdeeply Ġeach Ġand Ġevery Ġtime ĠI Ġwatch Ġit .&lt; br Ġ/ &gt;&lt; br Ġ/&gt; The Ġidea Ġof</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;s&gt; ĠAfter Ġthe Ġfailure Ġof Ġ\" The ĠCrus ades \" Ġat Ġthe Ġbox Ġoffice , ĠCecil ĠB . ĠDe M ille Ġstopped Ġdoing Ġfilms Ġabout Ġnon - American Ġhistory . ĠHis Ġfilms Ġfor Ġthe Ġnext Ġthirteen Ġyears Ġwere Ġabout Ġour Ġhistory Ġfrom ĠJean ĠLaf itte Ġto ĠWorld ĠWar ĠII Ġ( Dr . ĠWas sell ). ĠThe Ġfirst Ġin Ġorder Ġof</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=3, trunc_at=60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [NOT WORKING WITH EXTERNAL ADDITION OF SPECIAL TOKENS]\n",
    "### (Alternatively) Factory dataloader\n",
    "Here we set:\n",
    "- `tok_tfm=tok_fn` to use our HF tokenizer\n",
    "- `text_vocab=transformer_vocab` to load our pretrained vocab\n",
    "- `before_batch=transformer_padding(transformer_tokenizer)` to use our custom padding function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cls=SpecialClsTokens(tokenizer)\n",
    "\n",
    "# # Factory\n",
    "# fct_dls = TextDataLoaders.from_df(df, text_col=\"text\", tok_tfm=fastai_tokenizer, text_vocab=tokenizer_vocab_ls,\n",
    "#                               before_batch=[padding],\n",
    "#                               #item_tfms=[cls],\n",
    "#                               label_col='label', valid_col='is_valid', bs=bs)\n",
    "# fct_dls.show_batch(max_n=3, trunc_at=60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "#o=fct_dls.one_batch(); o[0].size(), o[1].size(), o[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create our learner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "opt_func = partial(Adam, decouple_wd=True)\n",
    "loss = LabelSmoothingCrossEntropy()\n",
    "\n",
    "fasthugs_model = FastHugsModel(transformer_cls=model_class, tokenizer=tokenizer, \n",
    "                               config_dict=config_dict, n_class=dls.c, pretrained=True)\n",
    "\n",
    "learn = Learner(dls, fasthugs_model, opt_func=opt_func, splitter=model_splitter, \n",
    "                loss_func=loss, metrics=[accuracy]).to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stage 1 training\n",
    "Lets freeze the model backbone and only train the classifier head. `freeze_to(1)` means that only the classifier head is trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "learn.freeze_to(1)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets find a learning rate to train our classifier head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=6.918309736647643e-07, lr_steep=0.03981071710586548)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3hc5Zn38e+tXizJllUwlm3Z2LhgikGYvpiWAEvoGyAJgUDCJpuyyb5kA2/eDYQNCdkkm2z6EmogQCgh1CwklFCMF4tg3G2Mq1xULFm9ztzvHzM2sizJkq2jGWl+n+uay3PKnHM/ljU/n3Oe8xxzd0REJHElxboAERGJLQWBiEiCUxCIiCQ4BYGISIJTEIiIJDgFgYhIgkuJdQGDVVBQ4KWlpbEuQ0RkRHnnnXdq3L2wt2UjLghKS0spLy+PdRkiIiOKmW3qa5lODYmIJDgFgYhIglMQiIgkOAWBiEiCUxCIiCS4wILAzO4xsyozW97H8ovMbKmZLTGzcjM7NahaRESkb0EeEdwHnNvP8peAo939GOA64K4AaxERGdH+Z/kOKupaAtl2YEHg7q8Btf0sb/IPH4aQDejBCCIivWhq7+IrD7/L/Qs3BrL9mF4jMLNLzGw18ByRowIREenhjfdr6AiFOXNWcSDbj2kQuPuT7j4LuBj4977WM7MbotcRyqurq4evQBGROPDy6kpyM1IoKx0XyPbjotdQ9DTSYWZW0MfyO929zN3LCgt7HSpDRGRUCoedl1dXc/rMIlKTg/nKjlkQmNl0M7Po+2OBNGBnrOoREYlHy7bWU9PUzlmzigLbR2CDzpnZw8ACoMDMKoBbgFQAd/81cBnwaTPrBFqBK7pdPBYREeClVZUkGZx+eHBnQwILAne/aj/Lvw98P6j9i4iMBi+truK4KeMYl50W2D7i4hqBiIjsa0d9Gyu2NQTWW2g3BYGISJx6eXUVAGfNDu76ACgIRETi1surKykZl8mMojGB7kdBICISh9o6Q7yxroazZxcT7WAZGAWBiEgceuuDnbR1hjkzwG6juykIRETi0F9WVZKVlswJ0/ID35eCQEQkDr2yuorTZhSQnpIc+L4UBCIicWZXSwfb6tsomxL80QAoCERE4s6GmmYASguyh2V/CgIRkTizcWckCKYWZA3L/hQEIiJxZkNNC0kGk/IVBCIiCWljTTOHjs0clgvFoCAQEYk7G2qamTpM1wdAQSAiElfcnY0KAhGRxLWzuYPG9i5KxysIREQS0saa3T2GFAQiIglpuO8hAAWBiEhc2bizmeQko2Rc5rDtU0EgIhJHNta0MGlcJqnJw/f1HNiezOweM6sys+V9LP+kmS2Nvhaa2dFB1SIiMlJsqGke1tNCEOwRwX3Auf0s3wCc7u5HAf8O3BlgLSIicc/d2bizeVh7DAGkBLVhd3/NzEr7Wb6w2+QioCSoWkRERoKqxnZaOkLD2mMI4ucawfXAn/paaGY3mFm5mZVXV1cPY1kiIsNnQwy6jkIcBIGZnUEkCL7R1zrufqe7l7l7WWFh4fAVJyIyjGJxDwEEeGpoIMzsKOAu4Dx33xnLWkREYm3DzmbSkpM4dOzwdR2FGB4RmNlk4A/A1e6+NlZ1iIjEi401zUzKzyQ5yYZ1v4EdEZjZw8ACoMDMKoBbgFQAd/818C1gPPBLMwPocveyoOoREYl3G2tahv20EATba+iq/Sz/LPDZoPYvIjKShMORrqOnzSgY9n3H/GKxiIjA9oY22rvCw34zGSgIRETiQqx6DIGCQEQkLsRi1NHdFAQiInFgY00z6SlJTMjNGPZ9KwhEROLA7jGGkoa56ygoCERE4kJk1NGsmOxbQSAiEmN1zR1srm1hWuGYmOxfQSAiEmMPvb2ZzpBz8TETY7J/BYGISAx1hsI88NYmTp1ewMxDcmJSg4JARCSG/rR8Bzsa2vjMKaUxq0FBICISQ/e8sYHS8VmcMbMoZjUoCEREYuRvm+tYsmUX155cGpNuo7spCEREYuTeNzeSk57C5WWTYlqHgkBEJAa217fy/LLtXHH8JMakx/QZYQoCEZFYeOCtTbg715xcGutSFAQiIsMtHHYefnszZ88uZlJ+bO4m7k5BICIyzLbUtVDX0smZs2LXU6g7BYGIyDBbvaMRIGY3kPUUWBCY2T1mVmVmy/tYPsvM3jKzdjO7Mag6RETizertkSA4vHiUBwFwH3BuP8trga8APwywBhGRuLOmsoHJ+Vlkx7i30G6BBYG7v0bky76v5VXuvhjoDKoGEZF4tHpHI7Pi5LQQ6BqBiMiwausMsbGmWUEwWGZ2g5mVm1l5dXV1rMsRETlg66qaCDvMPCQ31qXsMSKCwN3vdPcydy8rLCyMdTkiIgds1fYGIH56DMEICQIRkdFizY5G0lKSKB0f+xvJdgvskrWZPQwsAArMrAK4BUgFcPdfm9khQDmQC4TN7KvAHHdvCKomEZFYW1PZyOHFY0hJjp//hwcWBO5+1X6W7wBKgtq/iEg8Wr2jkb+bEV+nuOMnkkRERrmdTe1UN7bHVY8hUBCIiAybNXE2tMRuCgIRkWGye4yhWRMUBCIiCWnNjkbys9MoHJMe61L2oiAQERkmqysbmVmcg1nsnk/cGwWBiMgwCIedtTsa4+76ACgIRESGxebaFlo7Q3HXYwgUBCIiw+LDC8XxM8bQbgoCEZFhsGZHI2ZwePGYWJeyDwWBiMgw2P0wmqy0+HgYTXcKAhGRYbB6R6THUDxSEIiIBCwUdjbWNDMjDk8LgYJARCRw9a2dhJ24u5FsNwWBiEjAaps7ABiXnRbjSnqnIBARCVhdSyQI8hUEIiKJac8RQZaCQEQkIdU164hARCSh1bboiEBEJKHVNnWQmZpMZlpyrEvpVWBBYGb3mFmVmS3vY7mZ2U/NbJ2ZLTWzY4OqRUQklmpbOuL2tBAEe0RwH3BuP8vPA2ZEXzcAvwqwFhGRmKlrTtAgcPfXgNp+VrkI+K1HLALGmtmEoOoREYmV2pbOuL2HAGJ7jWAisKXbdEV03j7M7AYzKzez8urq6mEpTkRkqNQ1d5CflRrrMvoUyyDo7Vlt3tuK7n6nu5e5e1lhYWHAZYmIDK265g4dEfShApjUbboE2BajWkREAtHRFaaxvYv8OO06CrENgqeBT0d7D50I1Lv79hjWIyIy5Ha1xPc4QwCBPSHBzB4GFgAFZlYB3AKkArj7r4HngfOBdUAL8JmgahERiZWdcX5XMQQYBO5+1X6WO/DFoPYvIhIP6uJ8nCHQncUiIoHaPbzE+DEjPAjM7DAzS4++X2BmXzGzscGWJiIy8o2mI4IngJCZTQfuBqYCDwVWlYjIKFHb3AnA2FFwH0HY3buAS4CfuPvXAN0FLCKyH3UtHeRmpJCaHL9n4gdaWaeZXQVcAzwbnRe/8SYiEidq43ycIRh4EHwGOAm43d03mNlU4MHgyhIRGR3qWuL7rmIYYPdRd18JfAXAzMYBOe5+R5CFiYiMBjubOpiQlxHrMvo10F5Dr5pZrpnlA+8B95rZfwZbmojIyDcSjggGemooz90bgEuBe939OODs4MoSERn53J3a5g7Gj5IgSIk+K+DjfHixWERE+tHaGaK9KzxqjghuA14APnD3xWY2DXg/uLJEREa+2t3jDMXxzWQw8IvFjwGPdZteD1wWVFEiIqNBXfRmslFxRGBmJWb2ZPRh9JVm9oSZlQRdnIjISLZ7nKH87Pi+7Wqgp4buJfL8gEOJPE7ymeg8ERHpw0gYZwgGHgSF7n6vu3dFX/cBemakiEg/akfAswhg4EFQY2afMrPk6OtTwM4gCxMRGelqmztIMsjNGB2nhq4j0nV0B7AduBw9UUxEpF+1LR2My0ojKcliXUq/BhQE7r7Z3S9090J3L3L3i4ncXCYiIn2oGwEDzsHBPaHsX/a3gpmda2ZrzGydmd3Uy/IpZvaSmS2NDmOhnkgiMmrUNsf/8BJwcEHQ77GOmSUDvwDOA+YAV5nZnB6r/RD4rbsfReSmte8dRD0iInGlrqUj7m8mg4MLAt/P8vnAOndf7+4dwCPART3WmQO8FH3/Si/LRURGrNrmzpF/RGBmjWbW0Murkcg9Bf2ZCGzpNl0Rndfde3x4h/IlQI6ZjR9E/SIiccndI0cEcX4zGewnCNw9x91ze3nluPv+hqfo7dRRz6OIG4HTzexd4HRgK9C1z4bMbjCzcjMrr66u3s9uRURir6Gti1DY4/5mMji4U0P7UwFM6jZdAmzrvoK7b3P3S919HvDN6Lz6nhty9zvdvczdywoLdR+biMS/kXIzGQQbBIuBGWY21czSgCuJDFOxh5kVmNnuGm4G7gmwHhGRYbM7CEb8NYKD4e5dwJeIDF+9CnjU3VeY2W1mdmF0tQXAGjNbCxQDtwdVj4jIcNo9zlC8P5QGBjgM9YFy9+eB53vM+1a3948DjwdZg4hILOweeTTRrxGIiCSsOl0jEBFJLG2dob2ma1s6SEtJIistOUYVDZyCQETkINW3dHLKHS9z42Pv4R7pJV/XHLmr2Cy+B5wDBYGIyEF76O3N7Gzu4PF3Kvjlqx8AI+euYgj4YrGIyGjX0RXmvoUbOGX6eArGpPODF9YwrSCb2ub2EXFXMSgIREQOynPLtlHZ0M4dlx3FSdPGs7m2ha89uoT0lGROm1EQ6/IGRKeGREQOkLtz1+sbmF40htNnFJKRmsydV5cxPjud+tbOEXEPASgIREQO2KL1tazY1sD1p07d8xSywpx07rqmjJz0FEoLsmNc4cDo1JCIyAG6+4315Genccm8vQdWnj0hl//95llkpsZ/11HQEYGIyAFZX93EX1ZV8akTp5DRyxd+VlrKiOg6CgoCEZEDcs+bG0hLSeLqE6fEupSDpiAQERmktzfU8mh5BRcfcyiFOemxLuegKQhERAbhvS27uO6+xUwal8k3zp0V63KGhIJARGSAVu9o4NP3vM247FR+99kTGT9m5B8NgIJARGRA1lc38am73iYzNZmHPnsih+RlxLqkIaMgEBHZj9aOEFff/TbuzoOfPYFJ+VmxLmlI6T4CEZH9eGTxZrbuauWhz53A9KIxsS5nyOmIQESkH+1dIf77r+uZPzWfkw8bGWMHDZaCQESkH4+/U8GOhja+cuaMWJcSmECDwMzONbM1ZrbOzG7qZflkM3vFzN41s6Vmdn6Q9YiIDEZnKMyvXv2AYyaN5ZTp42NdTmACCwIzSwZ+AZwHzAGuMrM5PVb7f8Cj7j4PuBL4ZVD1iIgM1lNLtlFR18pXzpo+YoaLOBBBHhHMB9a5+3p37wAeAS7qsY4DudH3ecC2AOsRERmwUNj55SvrmDMhlzNmFsW6nEAFGQQTgS3dpiui87q7FfiUmVUAzwNf7m1DZnaDmZWbWXl1dXUQtYqI7OW5ZdtZX9PMl88c3UcDEGwQ9PY35z2mrwLuc/cS4HzgATPbpyZ3v9Pdy9y9rLCwMIBSRUQ+FA47v3h5HTOKxvDRIw6JdTmBCzIIKoBJ3aZL2PfUz/XAowDu/haQAYzO/lkiMmL89f1q1lQ28oUFh+154MxoFmQQLAZmmNlUM0sjcjH46R7rbAbOAjCz2USCQOd+RCSm7nljA0U56Vxw1KGxLmVYBBYE7t4FfAl4AVhFpHfQCjO7zcwujK72f4DPmdl7wMPAte7e8/SRiMiwWbOjkdffr+Gak0tJS0mMW60CHWLC3Z8nchG4+7xvdXu/EjglyBpERAbjnjc2kJGaxCfmT451KcMmMeJORGQAapraeXLJVi49toRx2WmxLmfYKAhERKJ+t2gzHV1hrjtlaqxLGVYKAhFJSDvq22jvCu2ZbusM8cCijSyYWTgqRxjtj4ahFpGEsb66ieeWbue5ZdtZvaORcVmpXDKvhCuOn8R7Fbuoaerg+lMT62gAFAQikgDer2zk648vZcmWXQAcXzqOm86bxbKKeh5YtJF73txAekoSM4tzOHV64t3KpCAQkVHL3XmsvIJvPb2cMekp/NsFczj/yEOYkJe5Z52dTe08+e5Wnl+2nS8sGP3DSfTGRlq3/bKyMi8vL491GSIS55rau/jmk8t4ask2Tpk+nh9fcQxFOaPnOcODZWbvuHtZb8t0RCAio86ulg4u+9VCNtQ0c+NHDucLC6aTnABDRRwoBYGIjCruzo2PLWVzbQsPXH8CpyTgOf/BUvdRERlV7n5jA39ZVcnN581WCAyQgkBERo2/ba7jjj+t5iNzivnMKaWxLmfEUBD0wd3ZWNMc6zJEZIB2tXTw5Yfe5ZC8DH5w+dEJ2fvnQOkaQR8eXLSJf3tqBT++4mgumVcS63JEEp67s6ulk407m9lc28LmnS20dYUIO4TdKd9YR1VjG499/mTyslJjXe6IoiDoRVtniJ+/sg6AW55awUnTCjgkL3G7nYkMt+rGdp5buo0PqpvZtquVrbta2VrXSmN7117rpSQZSWaYQVpyEt++cC7HTBobo6pHLgVBLx4t30JlQzvfveRIbnt2BTf9YSn3Xnu8DjVFhkhXKMzCD3ZiBhPyMijOzSArLYXX1lbzyOLNvLSqiq6wk5uRwsRxWZSMy+SEqflMys+idHw2U8ZnMSk/i4zU5Fg3ZVRQEPTQ3hXil698wPzSfK6aP4mOrhC3PrOS3y/ewpUJND65yEA1tnXyzqY63ttST0qyMS4rjfzsNMaPSWNKfhaFOel7/hNV39rJ7xdv5v6Fm9i6q3Wv7aQmG50hZ3x2GtedOpWPl5UwvSgnFk1KOAqCHh5dvIUdDW386OORi02fPqmUF1ZU8p3nVnHqjAJKxmXFukSRwO3ufXPitPF87rSp5GTsfc59S20LDy7axJsf1LByWwPhfgYoyMlIYXrRGCbkZfDXNdU0d4Q4YWo+/3bBbMZmpVHZ0MaO+jaqG9s5bso4zppdnDBPBosXGmKim/auEAt+8CoTx2by2OdP2vO/mC21LZz7k9c4etJYHrz+hIR4mLUkps5QmJ+9vI6fv/w+uZmp7GrpJD87jS+dMZ1PnjiZD6qa+e/XPuDZpdtJMiibks/xU/M5YWo+8yaPJcmMupYOdjZ1UNPUzsaaZtZVN/FBVeQC7wlT87nu1KnMnZgX66YmnJgNMWFm5wL/BSQDd7n7HT2W/xg4IzqZBRS5e8yu9DxaXsH2+rZ9up5Nys/i/10wh5v/sIzvv7Cam8+bHasSZQQKh52lW+tZtH4nORkplETPeRfmpLN2RyPlm+oo31jHym31FOdlMHtCLnMm5HJ4cQ5tnSGqGtupbGijpqmdjq4wYXdCYacr7HSGnPbOEB2hMB1dYZo7QrS0d9HSEaKtM8RRJXmcPaeYs2YV77fDw/rqJr72+yW8V1HPpcdO5NYLj2BDdTPf/5/V3PbsSn728vvUtXSSnZbM9adO5bpTpva6zQl5mR8O6jYziL9RGWqBHRGYWTKwFjgHqAAWA1dFn1Pc2/pfBua5+3X9bTeoI4L2rhBn/OBVJozN5PFuRwO7uTvfemoFDyzaxK0fm8O1AT/BqL6lk8fe2UJ6ajJFOekU5qRTMjaTotyB9V5q7QhRvqmWmcU5A/6M9K2hrZNfvLyOHQ1ttHaEaO0M0dEVZkbxGI6dPI55k8dROj4Ld6hpbqeyvp1Ntc38dU01r6ypoqapo9/tTy3I5siJeVQ2tLFqewMNbV37rJOdlkxGajJJSUayGclJRlpKEmnJSaSnRv7MSk8hOy2ZrLQUkpNg0fpaNte2AHDEobkU5qSTZMbug9pdLZ3UNLVT09RBU3sXeZmpfPeSI/n7oybste/X36/mt29t4uiSPK4+sVTdM0egWB0RzAfWufv6aBGPABcBvQYBcBVwS4D19OulVVVsq2/ju5ce2WvvIDPj1guPoLKhjW8/u5Li3AzOO3JCL1s6eJUNbXz67rdZU9m4z7JrTprCTefNJjOt994SVY1tPPDWJh5ctIm6lk4ApheN4aRp4zllegHnzCnW4FuDVNPUzjX3vM3qHY2UjMskMzWZzLRkksz447vbeHDRZgBy0lNo7QzR1e2EeU5GCgtmFnHWrCJOm1FARyhMRV0rFXUt7KhvZ1phNsdNGUfBmPQ9n3F3tu5q5f2qJrLTUijKSacoN52stMH/uro766qa+POqSl5fW0Ndc8eefvdhh7zMFOZOzKMwJ52inAwumTex1//lnzajkNNmFB7A356MBEEGwURgS7fpCuCE3lY0synAVODlAOvp15Itu0hLTuLkw/oemyQ5yfjpVfP4xG8W8c+/X0JBTjrHl+Yf8D7bu0Kkp+z9hb6+uomr736bXS0dPHD9fGYW51DV2E51Yzuvrqni/rc28fr7NfznFcfs6S/d1hli0fqdPLt0O08v2UZnOMw5s4u5/LgSNu5sZuEHO3nibxU8sGgTRxyay3cunsu8yeMOuO5EsnVXK1ff9b9sq2/l7mvKWDCzaK/loXDki/bdzXWs3N7AmPQUJuRlcEheJhPyMph5SA6pyXtf+JyQl9nvvxszi54+OviOCWbGjOIcZhTn8E8Lph/09mR0CvLU0D8AH3X3z0anrwbmu/uXe1n3G0BJb8uiy28AbgCYPHnycZs2bRryeq+88y1aO0I89aVT97tuXXMHl/16ITWN7Xz30iP5+yMnDPoeg9++tZFbn17B3Il5nD27mHPmFNMVcq69920A7vvMfI4s2feC2sJ1Ndz42HtUNrbzyRMms21XG2+uq6G1M0RmajKXH1fCdadOZWpB9l6f6wyF+dPyHXzn2ZVUN7Vz1fzJ/OtHZzI2K21QdSeS9dVNfOqu/6WxrYt7PnP8QYW+SKz1d2ooyCA4CbjV3T8anb4ZwN2/18u67wJfdPeF+9tuENcIwmHnqG+/yCXzJvLvF88d0Ge21Lbwjw+8w8rtDZw2o4DbLpq7z5dvXx5dvIV/fWIp80vz6QyHWbJlF+5gBofmZfLA9fOZVtj3w7PrWzv59jMr+MPftjJxbCZnzS7ijFlFnDRt/H5vsGls6+Qnf3mf+xZuJC8zlTsuPZKPHHHIgOrera0zxEurqnhnUx1XzZ/EjOL46uvdFQqzbVcb2+tbmTI+m+Lc9EEFdV1zB48s3sJvXl+PAfdfN1+9XGTEi1UQpBC5WHwWsJXIxeJPuPuKHuvNBF4ApvoAigkiCNZVNXH2f/6V/7j8KD5eNmnAn+sKhXlw0SZ+9OJa2rvC/OPp07j25FLGdzvf29PT723jnx95l1OnF3DXNWWkpyRT1djGy6uqWFfVxGdPmzbg4SzqWzvJzUg5oDueV25r4F+feI/lWxu4+sQpfPPvZ+8VIhV1Lfx1bTWpSUmMyUhhTHoKobDzp+Xb+dOyHTS2d+25rf//nj+bT5805aDuvK5t7mDltgber2pkXVUT71c1UVHbQlZ6CvnRG5TyMlNpbO9kZ1MHtc0d1Ld2kp6aRHZapL60lCS217expbZlr/P0BWPSmTsxl6Mm5nHu3AnMOTR3n/27Oyu3N/DbhZv445KttHeFOWnaeL5zyVwO6yeURUaKmARBdMfnAz8h0n30Hne/3cxuA8rd/enoOrcCGe5+00C2GUQQPPluBV/7/Xv8z1dPY9Yh+35J7E9VQxu3P7+Kp5ZsIyXJOHNWEf9QNokFMwv3Oj/84oodfOF3f+O4KeO4/zPz+7zgO1zau0L88IU1/Ob1DcwszuGOy45kXVUTT/ytgkXra3v9zJj0FM6dewiXzJvI9KIx3PTEUl5ZU82CmYX8x+VH9fsowHVVTVQ2tFHX0kFdSyfVje2s2t7Aym0Ne91lmpeZyoyiMUwen0VbZ2ivL/6cjBTGZ6fvCYaOUJim9i6a27to7QwxIS+D0vHZlI7Ppig3nY01zSzf1sDyrfW8X9VEKOwccWgu/3BcCecfOYE1lY28tKqKl1dXsbm2hYzUJC6ZV8I1J085oH8LIvEqZkEQhCCC4NvPrODhtzez/NaPkpJ84Hc0rq1s5LHyLTz57lZqmjrIyUghOy0FJ/J3vLOpgyMm5vG7z57AmPT4uan71TVV3PjYe3u6OE4Zn8Vlx5ZwwVETSEtJoqm9i8a2Ljq7wsybPG6vAHN3Hli0idufW0V2egpfO3sGVxw/ea87QyvqWrjtmZW8uLJyr/2aRbpNzj00j7kTczni0DwOL86hYExaIOM67Wrp4Kkl23jsnS0s39qwZ356ShKnTC/gzFlFXHDUBF03kVFJQbAfl/8qcmni8S+cPCTb6wyFeWV1Fa+urSYUivz9mkFuZir/tOCwuPyiqW5s58l3Kzh28jiOmzJu0F/E66oaufkPy1i8sY6ScZn881kz+NjRh3LPmxv46UvvYxhfPOMwykrzGZeVxrisVMZmpcVsKIFV2xt4eXUVsyfkcNK0gpgfnYkETUHQj65QmCNvfZEr50/ilo8dMWTbTUTuzl/XVvOjF9eybGs96SlJtHeF+egRxXzrY0cwcWxmrEsUSVgxG2JiJFhX3URr9FZ8OThmxoKZRZx+eCEvrKjkuWXbuWTeoZw5qzjWpYlIPxI+CJZW1ANwVIkeZjFUzIxz5x7CuXMH1y1VRGIj4cd6XVZRz5j0FKaOH9g9ACIio03CB8HSrfXMnZiroaVFJGEldBB0dIVZtb1Bp4VEJKEldBCsrWykoyusC8UiktASOgj2XCieqCMCEUlcCR0Ey7buIi8zlUn56t8uIokroYNgaUU9R5XkBTKcgYjISJGwQdDWGWLNjkZdHxCRhJewQbBqewNdYedIXR8QkQSXsEGw8IOdABw7WUEgIoktYYPgxZWVHF2SR1HuwB4CIyIyWiVkEFQ2tPHell2cM0eDoYmIJGQQ/GVV5AEp58zRoGgiIgkZBH9eWcnk/CwOL9azaEVEEi4Imtq7WLhuJ+fMKdb9AyIiBBwEZnauma0xs3Vm1uvD6c3s42a20sxWmNlDQdYD8NraajpCYV0fEBGJCuzBNGaWDPwCOAeoABab2dPuvrLbOjOAm4FT3L3OzIqCqme3P6+sZGxWKmVTxgW9KxGRESHII4L5wDp3X+/uHcAjwEU91vkc8At3rwNw96oA66EzFObl1VWcOauIlOSEOysmItKrIL8NJwJbuk1XROd1dzhwuJm9aWaLzOzc3jZkZjeYWbmZlVdXV3pMBpQAAAgmSURBVB9wQYs31lLf2slHdFpIRGSPIIOgtyux3mM6BZgBLACuAu4ys31u9XX3O929zN3LCgsLD7igP6+sJC0lidNmHPg2RERGmyCDoAKY1G26BNjWyzpPuXunu28A1hAJhiHn7vx5ZSWnTi8gOz2wSyMiIiNOkEGwGJhhZlPNLA24Eni6xzp/BM4AMLMCIqeK1gdRzOodjVTUtaq3kIhID4EFgbt3AV8CXgBWAY+6+wozu83MLoyu9gKw08xWAq8AX3f3nUHUU1HXSsGYNM6aHXjHJBGREcXce562j29lZWVeXl5+QJ8Nh52kJN1EJiKJx8zecfey3pYlVB9KhYCIyL4SKghERGRfCgIRkQSnIBARSXAKAhGRBKcgEBFJcAoCEZEEpyAQEUlwI+6GMjOrB97vNisPqO9jevf73X8WADUHuOue+xnsOv3Vub/poWzH/urc3/KhbAcE+zMZTDt6zhst7eg5Hct29LeO2hF8O6a4e+8jbrr7iHoBdw50evf7bn+WD9V+B7vOYOoOsh0DactwtSPon8lg2tFXrSO9Hf21a7jb0d86akds2rH7NRJPDT0ziOln+lhnKPY72HUGU3fP6aFsx0C2k4jt6DlvtLSj53Qs29HfOmpHbNoBjMBTQwfDzMq9j7E2RpLR0g4YPW1RO+KL2jE4I/GI4GDcGesChshoaQeMnraoHfFF7RiEhDoiEBGRfSXaEYGIiPSgIBARSXAKAhGRBKcgiDKz08zs12Z2l5ktjHU9B8rMkszsdjP7mZldE+t6DpSZLTCz16M/kwWxrudgmFm2mb1jZhfEupYDZWazoz+Lx83sC7Gu52CY2cVm9hsze8rMPhLreg6UmU0zs7vN7PGD3daoCAIzu8fMqsxseY/555rZGjNbZ2Y39bcNd3/d3T8PPAvcH2S9fRmKdgAXAROBTqAiqFr7M0TtcKAJyGBktwPgG8CjwVS5f0P0+7Eq+vvxcSBm3TKHqC1/dPfPAdcCVwRYbp+GqB3r3f36ISnoQO9ai6cX8HfAscDybvOSgQ+AaUAa8B4wBziSyJd991dRt889CuSO1HYANwH/GP3s4yO4HUnRzxUDvxvB7TgbuJLIl84FI7Ud0c9cCCwEPhGLdgxlW6Kf+xFw7Chox0H/nqcwCrj7a2ZW2mP2fGCdu68HMLNHgIvc/XtAr4foZjYZqHf3hgDL7dNQtMPMKoCO6GQouGr7NlQ/j6g6ID2IOvdniH4eZwDZRH6hW83seXcPB1p4D0P183D3p4Gnzew54KHgKu7bEP1MDLgD+JO7/y3Yins3xL8jB21UBEEfJgJbuk1XACfs5zPXA/cGVtGBGWw7/gD8zMxOA14LsrBBGlQ7zOxS4KPAWODnwZY2KINqh7t/E8DMrgVqhjsE+jHYn8cC4FIiofx8oJUN3mB/R75M5Egtz8ymu/uvgyxuEAb7MxkP3A7MM7Obo4FxQEZzEFgv8/q9e87dbwmoloMxqHa4ewuRQIs3g23HH4iEWrwZ9L8rAHe/b+hLOSiD/Xm8CrwaVDEHabBt+Snw0+DKOWCDbcdO4PNDseNRcbG4DxXApG7TJcC2GNVyMNSO+KJ2xJ/R0paYtWM0B8FiYIaZTTWzNCIX7J6OcU0HQu2IL2pH/BktbYldO2J19X+Ir8A/DGznwy6T10fnnw+sJXIl/puxrlPtUDvUDrUlHtuhQedERBLcaD41JCIiA6AgEBFJcAoCEZEEpyAQEUlwCgIRkQSnIBARSXAKAhkVzKxpmPd3l5nNGaJthcxsiZktN7NnzGzsftYfa2b/NBT7FgE9vF5GCTNrcvcxQ7i9FHfvGqrt7Wdfe2o3s/uBte5+ez/rlwLPuvvc4ahPRj8dEcioZWaFZvaEmS2Ovk6Jzp9vZgvN7N3onzOj8681s8fM7BngRYs8Je1VizyVa7WZ/S46hDHR+WXR900WeSrce2a2yMyKo/MPi04vNrPbBnjU8haRUSgxszFm9pKZ/c3MlpnZRdF17gAOix5F/CC67tej+1lqZt8ewr9GSQAKAhnN/gv4sbsfD1wG3BWdvxr4O3efB3wL+G63z5wEXOPuZ0an5wFfJfI8gWnAKb3sJxtY5O5HExn6+3Pd9v9f0f3vd/AwM0sGzuLD8WXagEvc/VjgDOBH0SC6CfjA3Y9x969b5HGLM4iMZ38McJyZ/d3+9iey22gehlrkbGBO9D/xALlmlgPkAfeb2Qwiw/ymdvvMn929ttv02+5eAWBmS4BS4I0e++kg8tQogHeAc6LvTwIujr5/CPhhH3Vmdtv2O8Cfo/MN+G70Sz1M5EihuJfPfyT6ejc6PYZIMMTT8ygkjikIZDRLAk5y99buM83sZ8Ar7n5J9Hz7q90WN/fYRnu39yF6/53p9A8vtvW1Tn9a3f0YM8sjEihfJDJe/ieBQuA4d+80s41EnuHckwHfc/f/HuR+RQCdGpLR7UXgS7snzOyY6Ns8YGv0/bUB7n8RkVNSEBlSuF/uXg98BbjRzFKJ1FkVDYEzgCnRVRuBnG4ffQG4zsx2X3CeaGZFQ9QGSQAKAhktssysotvrX4h8qZZFL6Cu5MOnOf0H8D0ze5PIA8OD8lXgX8zsbWACUL+/D7j7u0QeWn4l8Dsi9ZcTOTpYHV1nJ/BmtLvpD9z9RSKnnt4ys2XA4+wdFCL9UvdRkYCYWRaR0z5uZlcCV7n7Rfv7nMhw0zUCkeAcB/w82tNnF3BdjOsR6ZWOCEREEpyuEYiIJDgFgYhIglMQiIgkOAWBiEiCUxCIiCQ4BYGISIL7//qwNvIqf+KAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find(suggestions=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.LineCollection at 0x7f7e946e59d0>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXyddZn//9eVfU+zd0nbtDR0oSyFUJaCFhAFhwEBR8ERQUAcZ9DvOD+dwZ8z4vAdHZwZxxkdl2EAURAQcEPAQaUgSKk0SOne0r3pkqVJs+/n+v5xTkuaJmmS5s45yXk/H4/z4NzLue/r05Dzzn3fn/tzm7sjIiLxKyHaBYiISHQpCERE4pyCQEQkzikIRETinIJARCTOKQhEROJcUrQLGKnCwkIvKyuLdhkiIhPKG2+8UefuRQMtm3BBUFZWRmVlZbTLEBGZUMxs92DLdGpIRCTOKQhEROKcgkBEJM4pCERE4pyCQEQkzgUWBGb2oJnVmNn6QZZfY2ZrzWyNmVWa2UVB1SIiIoML8ojgIeCKIZa/AJzp7mcBtwL3B1iLiMiE9r/rD1LV0BbItgMLAnd/GagfYnmLv/MwhExAD0YQERlAS2cPn3nsTX6wclcg24/qNQIzu9bMNgPPEj4qEBGRfn7/dh1dvSEuXVASyPajGgTu/jN3XwB8APi/g61nZndEriNU1tbWjl+BIiIxYMXmanLSkqgoywtk+zHRayhyGukUMyscZPl97l7h7hVFRQMOlSEiMimFQs6KzbW8e34xyYnBfGVHLQjMbJ6ZWeT92UAKcCha9YiIxKJ1+xqpa+nksgXFge0jsEHnzOwxYDlQaGZVwN1AMoC7fw+4HviYmXUD7cCH+1w8FhER4IVN1SQYvPvU4M6GBBYE7n7jCZZ/DfhaUPsXEZkMXthcwzmz88jLTAlsHzFxjUBERI53sLGDDfubAustdISCQEQkRq3YXAPAZQuDuz4ACgIRkZi1YnM1pXnplBdnBbofBYGISAzq6O7l99vqeM/CEiIdLAOjIBARiUGvbT9ER3eISwPsNnqEgkBEJAb9dlM1GSmJnDc3P/B9KQhERGLQi5truLi8kNSkxMD3pSAQEYkxh9u62N/YQcXs4I8GQEEgIhJzdta1AlBWmDku+1MQiIjEmF2HwkEwpzBjXPanIBARiTE769pIMJiZryAQEYlLu+pamT4lfVwuFIOCQEQk5uysa2XOOF0fAAWBiEhMcXd2KQhEROLXodYumjt7KCtQEIiIxKVddUd6DCkIRETi0njfQwAKAhGRmLLrUCuJCUZpXvq47VNBICISQ3bVtTEzL53kxPH7eg5sT2b2oJnVmNn6QZb/uZmtjbxWmtmZQdUiIjJR7KxrHdfTQhDsEcFDwBVDLN8JvNvdzwD+L3BfgLWIiMQ8d2fXodZx7TEEkBTUht39ZTMrG2L5yj6Tq4DSoGoREZkIapo7aevqHdceQxA71whuA3412EIzu8PMKs2ssra2dhzLEhEZPzuj0HUUYiAIzOwSwkHwd4Ot4+73uXuFu1cUFRWNX3EiIuMoGvcQQICnhobDzM4A7geudPdD0axFRCTadh5qJSUxgelTxq/rKETxiMDMZgE/BW5y963RqkNEJFbsqmtlZn46iQk2rvsN7IjAzB4DlgOFZlYF3A0kA7j794AvAQXAd8wMoMfdK4KqR0Qk1u2qaxv300IQbK+hG0+w/Hbg9qD2LyIykYRC4a6jF5cXjvu+o36xWERE4EBTB509oXG/mQwUBCIiMSFaPYZAQSAiEhOiMeroEQoCEZEYsKuuldSkBKblpI37vhUEIiIx4MgYQwnj3HUUFAQiIjEhPOpoRlT2rSAQEYmyhtYu9tS3MbcoKyr7VxCIiETZo6/vobvX+cBZM6KyfwWBiEgUdfeGePi13Vw0r5D5U7OjUoOCQEQkin61/iAHmzr4+LKyqNWgIBARiaIHf7+TsoIMLplfHLUaFAQiIlHyxz0NrNl7mFsuLItKt9EjFAQiIlHy/Vd3kZ2axAcrZka1DgWBiEgUHGhs57l1B/jwuTPJSo3qM8IUBCIi0fDwa7txd26+sCzapSgIRETGWyjkPPb6Ht6zsISZ+dG5m7gvBYGIyDjb29BGQ1s3ly6IXk+hvhQEIiLjbPPBZoCo3UDWX2BBYGYPmlmNma0fZPkCM3vNzDrN7HNB1SEiEms2HwgHwaklkzwIgIeAK4ZYXg98Bvi3AGsQEYk5W6qbmJWfQWaUewsdEVgQuPvLhL/sB1te4+6rge6gahARiUWbDzazIEZOC4GuEYiIjKuO7l521bUqCEbKzO4ws0ozq6ytrY12OVG1fPlyli9fHu0yRGSUttW0EHKYPzVnyPXG83d9QgSBu9/n7hXuXlFUVBTtckRERm3TgSYgdnoMwQQJAhGRyWLLwWZSkhIoK4j+jWRHBHbJ2sweA5YDhWZWBdwNJAO4+/fMbCpQCeQAITP7a2CRuzcFVZOISLRtqW7m1JIskhJj5+/wwILA3W88wfKDQGlQ+xcRiUWbDzbzrvLYOsUdO5EkIjLJHWrppLa5M6Z6DIGCQERk3GyJsaEljlAQiIiMkyNjDC2YpiAQEYlLWw42k5+ZQlFWarRLOYaCQERknGyubmZ+STZm0Xs+8UAUBCIi4yAUcrYebI656wOgIBARGRd76tto7+6NuR5DoCAQERkX71woHnqMoWhQEIiIjIMtB5sxg1NLsqJdynEUBCIi4+DIw2gyUmLjYTR9KQhERMbB5oPhHkOxSEEgIhKw3pCzq66V8hg8LQQKAhGRwDW2dxNyYu5GsiMUBCIiAatv7QIgLzMlypUMTEEgIhKwhrZwEOQrCERE4tPRI4IMBYGISFxqaNURgYhIXKtv0xGBiEhcq2/pIj05kfSUxGiXMqDAgsDMHjSzGjNbP8hyM7Nvmtk2M1trZmcHVYuISDTVt3XF7GkhCPaI4CHgiiGWXwmUR153AN8NsBYRkahpaI3TIHD3l4H6IVa5Bvihh60CppjZtKDqERGJlvq27pi9hwCie41gBrC3z3RVZN5xzOwOM6s0s8ra2tpxKU5EZKw0tHaRn5Ec7TIGFc0gGOhZbT7Qiu5+n7tXuHtFUVFRwGWJiIythtYuHREMogqY2We6FNgfpVpERALR1ROiubOH/BjtOgrRDYKngY9Feg+dDzS6+4Eo1iMiMuYOt8X2OEMAgT0hwcweA5YDhWZWBdwNJAO4+/eA54D3A9uANuDjQdUiIhIth2L8rmIIMAjc/cYTLHfgr4Lav4hILGiI8XGGQHcWi4gE6sjwEgVZEzwIzOwUM0uNvF9uZp8xsynBliYiMvFNpiOCnwC9ZjYPeACYAzwaWFUiIpNEfWs3AFMmwX0EIXfvAa4F/sPdPwvoLmARkRNoaOsiJy2J5MTYPRM/3Mq6zexG4Gbgmci82I03EZEYUR/j4wzB8IPg48AFwFfcfaeZzQEeCa4sEZHJoaEttu8qhmF2H3X3jcBnAMwsD8h293uDLExEZDI41NLFtNy0aJcxpOH2GnrJzHLMLB94C/i+mf17sKWJiEx8E+GIYLinhnLdvQm4Dvi+u58DvCe4skREJj53p761i4JJEgRJkWcFfIh3LhaLiMgQ2rt76ewJTZojgnuA54Ht7r7azOYCbwdXlojIxFd/ZJyhGL6ZDIZ/sfhJ4Mk+0zuA64MqSkRkMmiI3Ew2KY4IzKzUzH4WeRh9tZn9xMxKgy5ORGQiOzLOUH5mbN92NdxTQ98n/PyA6YQfJ/nLyDwRERnERBhnCIYfBEXu/n1374m8HgL0zEgRkSHUT4BnEcDwg6DOzD5qZomR10eBQ0EWJiIy0dW3dpFgkJM2OU4N3Uq46+hB4ADwQfREMRGRIdW3dZGXkUJCgkW7lCENKwjcfY+7X+3uRe5e7O4fIHxzmYiIDKJhAgw4Byf3hLK/OdEKZnaFmW0xs21mdtcAy2eb2QtmtjYyjIV6IonIpFHfGvvDS8DJBcGQxzpmlgh8G7gSWATcaGaL+q32b8AP3f0Mwjet/fNJ1CMiElMa2rpi/mYyOLkg8BMsXwpsc/cd7t4FPA5c02+dRcALkfcvDrBcRGTCqm/tnvhHBGbWbGZNA7yaCd9TMJQZwN4+01WReX29xTt3KF8LZJtZwQjqFxGJSe4ePiKI8ZvJ4ARB4O7Z7p4zwCvb3U80PMVAp476H0V8Dni3mb0JvBvYB/QctyGzO8ys0swqa2trT7BbEZHoa+rooTfkMX8zGZzcqaETqQJm9pkuBfb3XcHd97v7de6+BPhiZF5j/w25+33uXuHuFUVFuo9NRGLfRLmZDIINgtVAuZnNMbMU4AbCw1QcZWaFZnakhi8ADwZYj4jIuDkSBBP+GsHJcPce4E7Cw1dvAp5w9w1mdo+ZXR1ZbTmwxcy2AiXAV4KqR0RkPB0ZZyjWH0oDwxyGerTc/TnguX7zvtTn/VPAU0HWICISDUdGHo33awQiInGrQdcIRETiS0d37zHT9W1dpCQlkJGSGKWKhk9BICJykhrbull27wo+9+RbuId7yTe0hu8qNovtAedAQSAictIefX0Ph1q7eOqNKr7z0nZg4txVDAFfLBYRmey6ekI8tHIny+YVUJiVyr8+v4W5hZnUt3ZOiLuKQUEgInJSnl23n+qmTu69/gwumFvAnvo2PvvEGlKTErm4vDDa5Q2LTg2JiIySu3P/KzuZV5zFu8uLSEtO5L6bKijITKWxvXtC3EMACgIRkVFbtaOeDfubuO2iOUefQlaUncr9N1eQnZpEWWFmlCscHp0aEhEZpQd+v4P8zBSuXXLswMoLp+Xwhy9eRnpy7HcdBR0RiIiMyo7aFn67qYaPnj+btAG+8DNSkiZE11FQEIiIjMqDr+4kJSmBm86fHe1STpqCQERkhF7fWc8TlVV84KzpFGWnRruck6YgEBEZgbf2HubWh1YzMy+dv7tiQbTLGRMKAhGRYdp8sImPPfg6eZnJ/Oj28ynImvhHA6AgEBEZlh21LXz0/tdJT07k0dvPZ2puWrRLGjMKAhGRE2jv6uWmB17H3Xnk9vOYmZ8R7ZLGlO4jEBE5gcdX72Hf4XYe/cR5zCvOinY5Y05HBCIiQ+js6eW/f7eDpXPyufCUiTF20EgpCEREhvDUG1UcbOrgM5eWR7uUwAQaBGZ2hZltMbNtZnbXAMtnmdmLZvamma01s/cHWY+IyEh094b47kvbOWvmFJbNK4h2OYEJLAjMLBH4NnAlsAi40cwW9Vvt74En3H0JcAPwnaDqEREZqV+s2U9VQzufuWzehBkuYjSCPCJYCmxz9x3u3gU8DlzTbx0HciLvc4H9AdYjIjJsvSHnOy9uY9G0HC6ZXxztcgIVZBDMAPb2ma6KzOvry8BHzawKeA749EAbMrM7zKzSzCpra2uDqFVE5BjPrjvAjrpWPn3p5D4agGCDYKB/Oe83fSPwkLuXAu8HHjaz42py9/vcvcLdK4qKigIoVUTkHaGQ8+0V2ygvzuJ9p02NdjmBCzIIqoCZfaZLOf7Uz23AEwDu/hqQBkzO/lkiMmH87u1atlQ386nlpxx94MxkFmQQrAbKzWyOmaUQvhj8dL919gCXAZjZQsJBoHM/IhJVD/5+J8XZqVx1xvRolzIuAgsCd+8B7gSeBzYR7h20wczuMbOrI6v9f8AnzOwt4DHgFnfvf/pIRGTcbDnYzCtv13HzhWWkJMXHrVaBDjHh7s8Rvgjcd96X+rzfCCwLsgYRkZF48Pc7SUtO4CNLZ0W7lHETH3EnIjIMdS2d/GzNPq47u5S8zJRolzNuFAQiIhE/WrWHrp4Qty6bE+1SxpWCQETi0sHGDjp7eo9Od3T38vCqXSyfXzQpRxgdioahFpG4saO2hWfXHuDZdQfYfLCZvIxkrl1SyofPnclbVYepa+nitovi62gAFAQiEgferm7m80+tZc3ewwCcW5bHXVcuYF1VIw+v2sWDr+4kNSmB+SXZXDQv/m5lUhCIyKTl7jxZWcWXnl5PVmoS/3DVIt5/+lSm5aYfXedQSyc/e3Mfz607wKeWT/7hJAaiIBCRSamls4cv/mwdv1izn2XzCvjGh8+iOPv45wwXZKVy+8Vzuf3iuVGoMjYoCERk0jnc1sX1313JzrpWPvfeU/nU8nkkxsFQEaOlIBCRScXd+dyTa9lT38bDt53Hsjg85z9S6j4qIpPKA7/fyW83VfOFKxcqBIZJQSAik8Yf9zRw7682895FJXx8WVm0y5kwFASDcHd21bVGuwwRGabDbV18+tE3mZqbxr9+8My47P0zWrpGMIhHVu3mH36xgW98+EyuXVIa7XJE4p67c7itm12HWtlT38aeQ2109PQScgi5U7mrgZrmDp78iwvJzUiOdrkTioJgAB3dvfzXi9sAuPsXG7hgbiFTc4/vdiYiwaht7uTZtfvZXtvK/sPt7Dvczr6Gdpo7e45ZLynBSDDDDFISE/jHqxdz1swpUap64lIQDOCJyr1UN3Xy1WtP555nNnDXT9fy/VvO1aGmyBjp6Q2xcvshzGBabholOWlkpCTx8tZaHl+9hxc21dATcnLSkpiRl0FpXjrnzclnZn4GZQWZzC7IYGZ+BmnJidFuyqSgIOins6eX77y4naVl+dy4dCZdPb18+Zcb+fHqvdwQR+OTiwxXc0c3b+xu4K29jSQlGnkZKeRnplCQlcLs/AyKslOP/hHV2N7Nj1fv4Qcrd7PvcPsx20lONLp7nYLMFG69aA4fqihlXnF2NJoUdxQE/Tyxei8Hmzr4+ofCF5s+dkEZz2+o5p+e3cRF5YWU5mVEu0SRwB3pfXP+3AI+cfEcstOOPee+t76NR1bt5tXtdWzc30RoiOcKZqclMa84i2m5afxuSy2tXb2cNyeff7hqIVMyUqhu6uBgYwe1zZ2cMzuPyxaWxM2TwWKFgqCPzp5evvPSdipm53HhKQUAJCQY//LBM7jiP17mb59ayyO3nRcXD7OW+NTdG+JbK7bxXyveJic9mdd31vPIqt3ceck8/vz8WWyvaeW/X97OM2sPkGBQMTufOy8t57w5+SyZNYUEMxraujjU0kVdSye76lrZVtvC9ppW3trbyPtOm8qtF81h8YzcaDdV+gg0CMzsCuA/gUTgfne/t9/ybwCXRCYzgGJ3j9qVnicqqzjQ2HFc17OZ+Rn8/VWL+MJP1/G15zfzhSsXRqtEmYBCIWftvkZW7ThEdloSpZFz3kXZqWw92Ezl7gYqdzWwcX8jJblpLJyWw6JpOZxakk1Hdy81zZ1UN3VQ19JJV0+IkDu9Iacn5HT3Op3dvXT1hujqCdHa1UtbZw9tXb10dPdyRmku71lUwmULSk7Y4WFHbQuf/fEa3qpq5LqzZ/Dlq09jZ20rX/vfzdzzzEa+teJtGtq6yUxJ5LaL5nDrsjkDbnNabvo7g7rND+JfVMZaYEFgZonAt4HLgSpgtZk9HXlOMQDu/tk+638aWBJUPSfS2dPLd1/cxjmz81g2r+C45TecO5ON+5v479/tYFpOGrcE/ASjxrZunnxjL6nJiRRnp1KUnUrplPQTfzCivauXyt31zC/JpjhHPZ5OVlNHN99esY2DTR20d/XS3t1LV0+I8pIszp6Vx5JZeZQVZOAOda2dVDd2sru+ld9tqeXFLTXUtXQNuf05hZlUlOVT3dTBM2/t59E/7DluncyURNKSE0lIMBLNSEwwUpISSElMIDU5/N/c9GSm54YvvCYmwKod9bz4s/V8kfWcNj2HouxUEsw4clB7uK2bupZO6lq6aOnsITc9mW9/5Gz+5IxpAJw5cwqPfuJ8Xnm7lh++tpszS3O56fwydc+cZII8IlgKbHP3HQBm9jhwDbBxkPVvBO4OsJ4hvbCphv2NHXz1utMH7B1kZnz56tOoburgH5/ZSElOGleePi2QWqqbOvjYA6+zpbr5uGXZZZeRt+d3g362prmDh1/bzSOrdtPQ1g3AvOIsLphbwLJ5hVy+qESDb41QXUsnNz/4OpsPNlOal056ciLpKYkkmPHzN/fzyKrwl3Z2ahLt3b309Dlhnp2WxPL5xVy2oJiLywvp6g1R1dBOVUMbBxs7mVuUyTmz8yjMSj36GXdn3+F23q5pITMlieLsVIpzUslIGfmvq7uzraaF32yq5pWtdTS0dh3tdx9yyE1PYvGMXIqyUynOTuPaJTMG/Cv/4vIiLi4vGsW/nkwEQQbBDGBvn+kq4LyBVjSz2cAcYEWA9Qxpzd7DpCQmcOEpg49NkphgfPPGJXzkf1bxf368hsLsVM4tyx/1Pjt7eklNOrb7247aFm564HUOt3Xx8G1LmV+STU1zJ7XNnby0pYYfvAbtuWWs2Xv4aH/pju5eVu04xDNrD/D0mv10h0JcvrCED55Tyq5Drazcfoif/LGKh1ft5rTpOfzTBxazZFbeqOuOJ/sOt3PT/X9gf2M7D9xcwfL5xccs7w2Fv2jf3NPAxgNNZKUmMS03jam56UzLTWP+1GySE4+98DktN33I/2/MLHL66OQ7JpgZ5SXZlJdk85fL55309mRyCjIIBvqzc7C+BTcAT7l770ALzewO4A6AWbOC6cK5tuowC6dln7C3QlpyIg/cfC7Xf28ltz20mq9edzp/cvq0Ed9j8MPXdvHlpzeweEYu71lYwuWLSujpdW75/usAPH7HBZxeGr6gduTUziULivnfB/6FulOu5PrvruTPz5vF/sMdvLqtjvbuXtKTE/nwuTO59aI5zCnMPLqvO951Ct29IX61/iD/9MxGrvvuSm5cOou/fd98pmSkjKjueLKjtoWP3v8Hmjt6ePi28wb88k5MMOZPzWb+VHVzlIkryCCoAmb2mS4F9g+y7g3AXw22IXe/D7gPoKKiYoiOaqMTCjnr9zVx7ZIZw1o/LzOFH3x8KZ98+A3ufPRNfly+l3uuWXzMl+9Qnli9ly/9YgNLy/LpDoX4xm+38u+/2YoZTM9N5+HbljK3aOCHZ6c37WH62oc495Nf44ev7WbGlHT+rKKUSxYUc8HcgkFvsElOTODqM6dzyfwi/uO3b/PQyl387/qD3Hvd6bz3tKnDqvuIju5eXthUwxu7G7hx6UzKS2LrS7CnN8T+wx0caGxndkEmJTmpIwrqhtYuHl+9l/95ZQcGPHbH+erlIpNakEGwGig3sznAPsJf9h/pv5KZzQfygNcCrGVIO+paaensOfoX+HDMzM/g6TuX8ciq3Xz911t53zde5pPvnsstF5ZR0Od8b39Pv7Wfv/vpWi4uL+T+mytITUqkprmDFZtq2FbTwu0Xzz1h747E3k7+/UNncfefnkZOWtKIvuSy05L5h6sWcf3ZpfztT97ijoff4KbzZ/PFP1l4TIhUNbTxu621JCckkJWWRFZqEr0h51frD/CrdQdp7uzBDH70h938/+9fyMcumH1Sd17Xt3axcX8Tb9c0s62mhbdrWqiqbyMjNYn8yA1KuenJNHd2c6ili/rWLhrbu0lNTiAzJVxfSlICBxo72Fvfdsx5+sKsVBbPyOGMGblcsXgai6bnHLd/d2fjgSZ+uHI3P1+zj86eEBfMLeCfrl3MKYOEsshkEVgQuHuPmd0JPE+4++iD7r7BzO4BKt396ciqNwKPu/uY/6U/XOv2hR9ofcYIggAgKTGBW5bN4f2nT+Mrz23iWyu28d2XtnPpgmL+rGImy+cXHXN++NcbDvLZH6/h3LJ87rup4uj1geLstFHdtZybPvqeG4um5/CTT13Ivz2/hf95ZSev76zn3utPZ1tNCz/5YxWrdtQP+Lms1CSuWDyVa5fMYF5xFnf9ZC13P72BF7fU8C8fPGPARwEesa2mheqmDhraumho66a2uZNNB5rYuL/pmLtMc9OTKS/O4vxTCujo7uVQSxfba1tobO8mOy2JgsxUTinKIjc9ma7eEC2dPbR29tDe3cvCadlcuXgqZQWZFOeksquulfX7m1i/r5FX3q7jmyu2cdr0HP7snFLef/o0tlQ388KmGlZsrmFPfRtpyQlcd3YpN184mwVTjw8MkcnIovj9OyoVFRVeWVk5ptv8x19u4LHX97D+y+8jKXH0dzRurW7mycq9/OzNfdS1dJGdlkRmShIeuTRyqKWL02bk8qPbzyMrdXQZvHz5cgBeeumlUdfZ30tbavjck28d7eI4uyCD688u5aozppGSlEBLZw/NHT1094RYMiuP9JR3jhzcnYdX7eYrz24iMzWJz76nnA+fO+uYay1VDW3c88uN/Hpj9TH7NQt3m1w8PZfFM3I4bXoup5ZkU5iVEsi4TofbuvjFmv08+cZe1u9rOjo/NSmBZfMKuXRBMVedMU3XTSQmjPXvupm94e4VAy5TEMAHv7sSgKc+deGYbK+7N8SLm2t4aWstvb3hf18zyElP5i+Xn3JSXzRBBAGER3v82ZtVnD0rj3Nm5434i3hbTTNf+Ok6Vu9qoDQvnf9zWTl/euZ0Hnx1J9984W0M468uOYWKsnzyMlLIy0hmSkZK1IYS2HSgiRWba1g4LZsL5hYeE24isWA8gyDuh5jo6Q2xYX8TNyydeeKVhyk5MYH3njZ1xBdho6koO5U73nXKqD8/rzibJz55Ab/bWsvXf72Vzz+1lr//+Xo6e0K877QSvvSnpzFjBDfEBW3htBwWTtOpHxFQELCttoX2yK34cnLMjOXzi3n3qUU8v6GaZ9cd4Nol07l0QUm0SxORIcR9EKytagTgjFI9zGKsmBlXLJ7KFYsnzhGRSDyL+7Fe11U1kpWaxJyC4d0DICIy2cR9EKzd18jiGTkaWlpE4lZcB0FXT4hNB5p0WkhE4lpcB8HW6ma6ekK6UCwicS2uLxYfvVA8Y+IcEYz1/QMiEpvG83c9ro8I1u07TG56MjPzY6d/u4jIeIvrIFhb1cgZpbmBDGcgIjJRxG0QdHT3suVgs64PiEjci9sg2HSgiZ6Qc/oEuj4gIhKEuA2CldsPAXD2LAWBiMS3uA2CX2+s5szS3KOPgRQRiVdxGQTVTR28tfcwly/SYGgiInEZBL/dFH5AyuWLNCiaiEhcBsFvNlYzKz+DU0v0LFoRkbgLgpbOHlZuO8Tli0p0/4CICAEHgZldYWZbzGybmd01yDofMoqKnrEAAAjJSURBVLONZrbBzB4Nsh6Al7fW0tUb0vUBEZGIwMYaMrNE4NvA5UAVsNrMnnb3jX3WKQe+ACxz9wYzKw6qniN+s7GaKRnJVMzOC3pXIiITQpBHBEuBbe6+w927gMeBa/qt8wng2+7eAODuNQHWQ3dviBWba7h0QTFJiXF3VkxEZEBBfhvOAPb2ma6KzOvrVOBUM3vVzFaZ2RUDbcjM7jCzSjOrrK2tHXVBq3fV09jezXt1WkhE5Kggg2CgK7HebzoJKAeWAzcC95vZcbf6uvt97l7h7hVFRUWjLug3G6tJSUrg4vLRb0NEZLIJMgiqgJl9pkuB/QOs8wt373b3ncAWwsEw5tyd32ys5qJ5hWSmxvVjGEREjhFkEKwGys1sjpmlADcAT/db5+fAJQBmVkj4VNGOIIrZfLCZqoZ29RYSEeknsCBw9x7gTuB5YBPwhLtvMLN7zOzqyGrPA4fMbCPwIvB5dz8URD1VDe0UZqVw2cLAOyaJiEwo5t7/tH1sq6io8MrKylF9NhRyEhJ0E5mIxB8ze8PdKwZaFld9KBUCIiLHi6sgEBGR4ykIRETinIJARCTOKQhEROKcgkBEJM4pCERE4pyCQEQkzk24G8rMrBF4u8+sXKBxkOkj74/8txCoG+Wu++9npOsMVeeJpseyHSeq80TLx7IdEOzPZCTt6D9vsrSj/3Q02zHUOmpH8O2Y7e4Dj7jp7hPqBdw33Okj7/v8t3Ks9jvSdUZSd5DtGE5bxqsdQf9MRtKOwWqd6O0Yql3j3Y6h1lE7otOOI6+JeGrolyOY/uUg64zFfke6zkjq7j89lu0YznbisR39502WdvSfjmY7hlpH7YhOO4AJeGroZJhZpQ8y1sZEMlnaAZOnLWpHbFE7RmYiHhGcjPuiXcAYmSztgMnTFrUjtqgdIxBXRwQiInK8eDsiEBGRfhQEIiJxTkEgIhLnFAQRZnaxmX3PzO43s5XRrme0zCzBzL5iZt8ys5ujXc9omdlyM3sl8jNZHu16ToaZZZrZG2Z2VbRrGS0zWxj5WTxlZp+Kdj0nw8w+YGb/Y2a/MLP3Rrue0TKzuWb2gJk9dbLbmhRBYGYPmlmNma3vN/8KM9tiZtvM7K6htuHur7j7XwDPAD8Ist7BjEU7gGuAGUA3UBVUrUMZo3Y40AKkMbHbAfB3wBPBVHliY/T7sSny+/EhIGrdMseoLT93908AtwAfDrDcQY1RO3a4+21jUtBo71qLpRfwLuBsYH2feYnAdmAukAK8BSwCTif8Zd/3Vdznc08AORO1HcBdwCcjn31qArcjIfK5EuBHE7gd7wFuIPylc9VEbUfkM1cDK4GPRKMdY9mWyOe+Dpw9Cdpx0r/nSUwC7v6ymZX1m70U2ObuOwDM7HHgGnf/Z2DAQ3QzmwU0untTgOUOaizaYWZVQFdksje4agc3Vj+PiAYgNYg6T2SMfh6XAJmEf6Hbzew5dw8FWng/Y/XzcPengafN7Fng0eAqHtwY/UwMuBf4lbv/MdiKBzbGvyMnbVIEwSBmAHv7TFcB553gM7cB3w+sotEZaTt+CnzLzC4GXg6ysBEaUTvM7DrgfcAU4L+CLW1ERtQOd/8igJndAtSNdwgMYaQ/j+XAdYRD+blAKxu5kf6OfJrwkVqumc1z9+8FWdwIjPRnUgB8BVhiZl+IBMaoTOYgsAHmDXn3nLvfHVAtJ2NE7XD3NsKBFmtG2o6fEg61WDPi/68A3P2hsS/lpIz05/ES8FJQxZykkbblm8A3gytn1EbajkPAX4zFjifFxeJBVAEz+0yXAvujVMvJUDtii9oReyZLW6LWjskcBKuBcjObY2YphC/YPR3lmkZD7YgtakfsmSxtiV47onX1f4yvwD8GHOCdLpO3Rea/H9hK+Er8F6Ndp9qhdqgdaksstkODzomIxLnJfGpIRESGQUEgIhLnFAQiInFOQSAiEucUBCIicU5BICIS5xQEMimYWcs47+9+M1s0RtvqNbM1ZrbezH5pZlNOsP4UM/vLsdi3COjh9TJJmFmLu2eN4faS3L1nrLZ3gn0drd3MfgBsdfevDLF+GfCMuy8ej/pk8tMRgUxaZlZkZj8xs9WR17LI/KVmttLM3oz8d35k/i1m9qSZ/RL4tYWfkvaShZ/KtdnMfhQZwpjI/IrI+xYLPxXuLTNbZWYlkfmnRKZXm9k9wzxqeY3wKJSYWZaZvWBmfzSzdWZ2TWSde4FTIkcR/xpZ9/OR/aw1s38cw39GiQMKApnM/hP4hrufC1wP3B+Zvxl4l7svAb4EfLXPZy4Abnb3SyPTS4C/Jvw8gbnAsgH2kwmscvczCQ/9/Yk++//PyP5POHiYmSUCl/HO+DIdwLXufjZwCfD1SBDdBWx397Pc/fMWftxiOeHx7M8CzjGzd51ofyJHTOZhqEXeAyyK/BEPkGNm2UAu8AMzKyc8zG9yn8/8xt3r+0y/7u5VAGa2BigDft9vP12EnxoF8AZweeT9BcAHIu8fBf5tkDrT+2z7DeA3kfkGfDXypR4ifKRQMsDn3xt5vRmZziIcDLH0PAqJYQoCmcwSgAvcvb3vTDP7FvCiu18bOd/+Up/Frf220dnnfS8D/850+zsX2wZbZyjt7n6WmeUSDpS/Ijxe/p8DRcA57t5tZrsIP8O5PwP+2d3/e4T7FQF0akgmt18Ddx6ZMLOzIm9zgX2R97cEuP9VhE9JQXhI4SG5eyPwGeBzZpZMuM6aSAhcAsyOrNoMZPf56PPArWZ25ILzDDMrHqM2SBxQEMhkkWFmVX1ef0P4S7UicgF1I+88zelfgH82s1cJPzA8KH8N/I2ZvQ5MAxpP9AF3f5PwQ8tvAH5EuP5KwkcHmyPrHAJejXQ3/Vd3/zXhU0+vmdk64CmODQqRIan7qEhAzCyD8GkfN7MbgBvd/ZoTfU5kvOkagUhwzgH+K9LT5zBwa5TrERmQjghEROKcrhGIiMQ5BYGISJxTEIiIxDkFgYhInFMQiIjEOQWBiEic+3/vxuPgOLq73AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_lr_find()\n",
    "plt.vlines(9.999e-7, 0.65, 1.1)\n",
    "plt.vlines(0.10, 0.65, 1.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.723234</td>\n",
       "      <td>0.706531</td>\n",
       "      <td>0.535000</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.608959</td>\n",
       "      <td>0.579816</td>\n",
       "      <td>0.780000</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.536796</td>\n",
       "      <td>0.553176</td>\n",
       "      <td>0.835000</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(3, lr_max=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('roberta-fasthugs-stg1-1e-3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3xb1fn48c+RLO8943jEzt57OARCAgTCXgESRoEfBUrhS6HQAm1pGaWlQMtogZZVKAQCCWEHQgIJK9OB7D0cx3YS7z1ln98fV5LllSixZNny8369/LJ157mO8+jouec+R2mtEUII4btM3m6AEEIIz5JAL4QQPk4CvRBC+DgJ9EII4eMk0AshhI/z83YDWouNjdVpaWneboYQQvQoGzZsKNRax7W3rtsF+rS0NDIzM73dDCGE6FGUUgc7WiepGyGE8HES6IUQwsdJoBdCCB/X7XL0QghxohoaGsjJyaG2ttbbTfG4wMBAkpOTsVgsLu8jgV4I0ePl5OQQFhZGWloaSilvN8djtNYUFRWRk5NDenq6y/tJ6kYI0ePV1tYSExPj00EeQClFTEzMCX9ykUAvhPAJvh7k7U7mOn0u0BdW1vFTdom3m9GtLN12hKPlvp+7FEK0z+cC/ZwXV3HpC6u83Yxuo97axG1vbeC/P2R5uylC+KzS0lJeeOGFE97vvPPOo7S01AMtasnnAn1WUTUAtQ2NXm5J91Be20CThkMl1d5uihA+q6NA39h47Di0ZMkSIiMjPdUsB58L9HYVtVZvN6FbKK9pACC3pMYtx9Na849lu9lfUOmW4wnhC+6//3727dvH2LFjmTRpEjNnzuTqq69m1KhRAFxyySVMmDCBESNG8NJLLzn2S0tLo7CwkKysLIYNG8bNN9/MiBEjOPvss6mpcc//WXBxeKVSajbwLGAGXtFaP95q/dPATNvLYCBeax1pW9cIbLGty9ZaX+SOhnfEbFI0NmkqahuICwvw5Kl6hDJboM9xCvRaa4qq6okNPf7v57s9BXy/t5A7zxhESIAfR8vreO6rPXyyKY8V987wVLOFOGkPf7KN7Xnlbj3m8L7h/OnCER2uf/zxx9m6dSsbN25k5cqVnH/++WzdutUxBPK1114jOjqampoaJk2axOWXX05MTEyLY+zZs4d33nmHl19+mSuvvJL333+fa6+91i3tP26gV0qZgeeBWUAOsF4p9bHWert9G6313U7b/x8wzukQNVrrsW5prQuCLGYq66yUS48ewPF7KKyso7ahkUCLmT9+tI031xxk1f1n0Dcy6Jj7v7hyH6v2FREW4McdZwyipLrecTwhRPsmT57cYpz7c889xwcffADAoUOH2LNnT5tAn56eztixRqicMGECWVlZbmuPKz36ycBerfV+AKXUAuBiYHsH288D/uSe5p24QIuJyrrmlEVv5/x7yCmpYWB8KG+uMYrcfbXjKNdNTTvm/vkVRkD/dPNh7jhjEEWVRqCvqLXyU3YJ41KjPNNwIU7SsXreXSUkJMTx88qVK1m+fDmrV68mODiYGTNmtDsOPiCg+RO22Wx2a+rGlRx9EnDI6XWObVkbSql+QDrwtdPiQKVUplJqjVLqkg72u8W2TWZBQYGLTW9foMUMGDche7t9BZW8virL8TqnpJrGJo19GO6yHfnH3F9rTW5JDWaTYueRCvbmV7Toyf920WZPNFuIHicsLIyKiop215WVlREVFUVwcDA7d+5kzZo1Xdw613r07Y3O1x1sOxdYpLV2vtWcqrXOU0r1B75WSm3RWu9rcTCtXwJeApg4cWJHx3aJI9DXSOrmshdWOXL0YPToK2ob0Nq4l7FmXxFVdVZCAtr/MyiuqqemoZFfnD6Al77dxwc/5RIV7O9Yb23SFFXWEeNCrl8IXxYTE8O0adMYOXIkQUFBJCQkONbNnj2bf//734wePZohQ4aQkZHR5e1zJdDnAClOr5OBvA62nQvc7rxAa51n+75fKbUSI3+/r+2u7hFkC/RlvTx1o7Vu8TuwmBU5JTWOZReN6csHP+Wy4WAJ0we3OykN2cXGkMzxqZFMHxzHhz/lceGYvgAM7RPGziMVzHxqJT8+OAs/s88O4BLCJW+//Xa7ywMCAvj888/bXWfPw8fGxrJ161bH8nvvvdetbXPlf+d6YJBSKl0p5Y8RzD9uvZFSaggQBax2WhallAqw/RwLTKPj3L5bmE3GB5AiL90srKyz8sp3+7E2Nnnl/M7tsJs3OZWkyCBySqo5/cmVAMwYEodJwfqs4g6Psb+gCoD+caHMGp5AbmkN//5mH33CA3lu3jjCAv0or7VSXFXv0WsRQnTOcQO91toK3AEsBXYA72mttymlHlFKOQ+VnAcs0Fo7p16GAZlKqU3ACuBx59E6ntBgC7DeGhXy6ncH+PNnO1i0Iccr57draDT+Ge48cxCPXTKS5Khgth9uHnKWHBXEpLRoFmbmUGdt/6GO/YWVmE2K1Ohgpg9q7vXHhPozOCGMJ+eMBqBARuAI0a259Hlba71Eaz1Yaz1Aa/2YbdkftdYfO23zkNb6/lb7rdJaj9Jaj7F9f9W9zW+rOdB7p5cZEmCkjjbleP6x5mOxNhm/h/iwAEwmRXJUkKOHDhAR5M/Np/XnSHkta/a336vPKqwmJSoIfz8TKdHBXJuRCoDFlqaxP6dQUCGBXojuzOcSq/VW7/bo7ZyDqjdYbT16i9lIZaVEB7dYHxlsYdrAWPz9THy7u/2RTsWtHqrK6G+M+7W/mcaFBgIS6IXo7nwu0NtTFt4K9FV1Rhqkqt67o37sgd7PZPwT24M0wLu3ZBAbGkCQv5kRfcM7fIqwrKaBiKDmWWz6hBuB3R7oY8OMETiSuhGie/O5QF9n69GXVDfQ8nZB17AH+NJq7476abClbvxsPfqxKZGkx4Zw11mDmOIU9NNjQjhY1P6nj9aB3j6M0t7LD/b3IzrEn+wiKZgmRHfmc4He3ttsbNKO3n1Xso92Kays88objV3rHr3ZpFhx7wzuOmtwi+36xYSQV1bbptpnZZ2V3NIawp0CfVpMMI9eMpJn5jZXtBjaJ4x1B4odv3chxPGFhoYCkJeXx5w5c9rdZsaMGWRmZrrlfD4V6P+xbDdlNQ3YRlhS48FSxZsOlXLvwk2OewJ21bZAX9vQRFW990ol2wOvvUffkbRYI3dvHzNvd+bfVwK06NErpbguox/xYYGOZYMTwthfWMW/V+5jb34lpdUy1FIIV/Xt25dFixZ5/Dw+Feif+2oPAAF+xsgXT9Wk/3Z3AY9/vpNFG3JYuOFQi3WVdc3nLPTiTUprU8ubsR2x592PlLWsvXG03Gh7aAdPzdpdN7UfAKv3F3HWP77hyv+sPub2Qvii++67r0U9+oceeoiHH36YM888k/HjxzNq1Cg++uijNvtlZWUxcuRIAGpqapg7dy6jR4/mqquu6voyxT2NvSfviUBfb23iZ6+tc7z+YusRrpnSz/G6qs6KUqA1bMsrJy02pL3DeJz9gS176qYjfSKMQG+fanBzTil3vP2TY33RcR6GGhAXyoVj+vLJJuNh6d1HpU698LLP74cjW46/3YnoMwrOfbzD1XPnzuWuu+7il7/8JQDvvfceX3zxBXfffTfh4eEUFhaSkZHBRRdd1OGcry+++CLBwcFs3ryZzZs3M378eLc136d69K25O3WjtebhT7a1WLZ2fzE19Y2ON5WymgZOHRhLTIg/y7Yfcev5T4T9/sTxUjcJth69vUrlX5fsbJHGOW1Q7HHPNSwxrMVrb96bEMIbxo0bR35+Pnl5eWzatImoqCgSExP53e9+x+jRoznrrLPIzc3l6NGjHR7j22+/ddSfHz16NKNHj3Zb+3yyR29X4+Yc+dHyOuavzW6xrL6xib99sZPXV2UxY0gc2w+Xc82UVPzNJnYeab+aXVewPzBlOU4NmkCLmYggC0dsN2Q3HCzhyonJRIX4c9vpA4h0KmLWkRtOSaO4sp5Xvj8AGL8n+ycFIbrcMXrenjRnzhwWLVrEkSNHmDt3LvPnz6egoIANGzZgsVhIS0trtzyxs456+53lMz36KttN0MEJofzramPeE3f36HceaTnePN72ZKi9FPDKXcaDR7+eNZi02BCyiqq81rttHnVz/D+chPAAjpbX8lN2KfWNTZw9vA8PnDvMpSAPxjDLP1wwnHduNqry7cn33hucEN4yd+5cFixYwKJFi5gzZw5lZWXEx8djsVhYsWIFBw8ePOb+06dPZ/78+QBs3bqVzZvdVwbcZwJ9nbWJc0f24XfnDSMlyhhJ4u4cfeseemOTZnRyRItlgxNCiQkNIC0mmNqGJsdNza7WfDP2+P/ECeGB7DpawbyXjTrZk9KjT+qcgxKMIWOSpxe90YgRI6ioqCApKYnExESuueYaMjMzmThxIvPnz2fo0KHH3P+2226jsrKS0aNH88QTTzB58mS3tc1nUjfRIf68eO0EAHYfNQLygUL3Pshjv2FpV9vQyMd3nEpZdQNPfrmTt9ZkMyrJmNE9PdYIevcs3Mir109y1MnvKlYXh1eCEei/21PoeO08pPJExIYGEB3iz6OfbichPIALRvc9qeMI0VNt2dJ8Ezg2NpbVq9sfhVZZaXSG0tLSHOWJg4KCWLBggUfa5TM9emf2mvSPfrqdpib3pU6q6qyYFLx6/UR+OWMAb/w/4x03Itji+BRx4ZhEwJhMGOCHvUUs/jHXbW1wVUPTiaVu7Bb+YmqnznvxWCO437VgY6eOI4RwH5/p0TsLsDS/f7383X5umd7fLTc5Kuus9I8L5cxhCZw5LKHFuhumpTEuNYrJtrRHdEhzfru4quvTN64Or4TmsfT+fiYmpZ1c2sbugXOHkVNSw7LtRzlSVis3ZYXoBny6Rw/w1893snRbx0OaTkRFrbXDB4gC/MyOIG/3pwuHA7DPC5UsrS4Or4TmypZjkyM7fV5/PxM3npIGGPXshegqvWVY78lcp88HeoAVO9ufBPuhj7fx1NJdLh+3ss5KWKDrH4JunJbOmUPjWbO/qMtnnGpwcXglwPRBcbx7Swb/tI1W6iz7G0dOsfue7BPiWAIDAykqKvL5YK+1pqioiMDAE/uk7JOpGz+zid/OHsITXxhBfF9B+z1L+7DIe88Z4tJxK2utJISd2C/4ionJ/OKtH/lqZz7njOhzQvt2xokMrzSZVIuKlp3VJyIQk4KcEqlqKbpGcnIyOTk5FBS0P7eCLwkMDCQ5OfmE9vHJQA/Q36n0QEeB3q62odGlUTFVdVZCT6BHD3DWsAQSIwJ5Z112lwb65qJmXf+hzWI2kRgRRE6J9OhF17BYLKSnp3u7Gd2WT6ZuAAKcAndJdQMlrWq2OI/Gmf3Mt9z//vEfTqio6zhH3xE/s4lzRvRh7f6uLeXralEzT4kNC6DQ6Xe+Ymc+H23s+tFHQggfDvSt8/S5pS17l5VOM0BlFVWzYP2hY+b3tNYnnKO3m5gWRU1DI1tzy05435N1IqNuPCEq2NLizfXG19fzqwUbycxqf35aIYTn+Gygb52KaZ1GKGtnBqickhrWHSjmcFnblMO6A8VoDTEhrpUFcDa1fwxKwTcdzM3qCdYTGEfvCdHB/pTYatM71+zvyt+BEMLgs4H+eD36spq2gf7hT7Zx5X9Wc+0ra9usezfzENEh/lwxMeWE2xITGsCE1CiWbXfPME9XWBs1JmXcaPWGqBB/R4/+QGHz8NL10qMXosv5bKAPdHpoKtBiIu84gT4pMojlO4xhmPsKqvjjR1tbrN+SU8a4lEhCTjBHb3fW8AS25ZWTV1pDTkm1x4eBNTQ1eeVGrF1UsIUqW/nmPNsnpJToIPJKj129Twjhfj4c6Jt79NHB/m0m624d6O1110fYShf8b/VB6qxGUbSa+kb2FlQyMqllAbMTccbQeABOefxrTv3bCn45/0ePzYAFRo/e4qXePBg9ejAmSa+2zbrVPzaU/Ipanx/rLER30ysCfUSwP2U1LUfd2AP/rOEJ3Dq9PzOGGIH45tP6c/NpxjCtjdmlAOSWVqM1pHditqiBcaEtXn++9QjP2qY+9IQ6a2OLkUddzf68QW5ptaOEdHpsCLUNTZTXWo+1qxDCzXw40DdfWmSQpcMe/bNzx/LAecOYPbIPn915KheP7cttMwYCsDnHGCVjTzckdqJui3Ou/N1bjLrtn2zKc2vRNWe1DU1t7lN0Jfunn805ZVTZRjj1jzPeKAsqJH0jRFfy2UDvb8tPXzY+ichgS5tUTVlNA/5mU4tgOKJvBEopokP8SQgPYMdhY6IR+8TZfSODOtWmz391GgtuyWBK/xieuWosOSU1zHhqJY98sr1Tx21PTUNji+JuXa1PRCDxYQFsySlr0aMHOOsf37I9r/xYuwsh3MhnA71Sii0Pnc0Tl48mIshCaZtAX094kKXDqpbDEsPZbgv0eWU1KNU8v+rJGpYYToat1MA5I/owOT2a7OJqXvvhAPlu7uXW1jd6tUcPkBwVRH5FHVX1jfibTYzs23yP46kvXa8xJIToHJ8N9ABhgRb8zCYigi2UVTe0uAlYVtNARFDHI2iGJYazr6CSemsTh0triQ0NwN/Pfb+uIH8z7906lY/vmAbA6n1Fbjs2QK3VtbIOnhQZ7E9pTT1VdVaCA8yOG7QAX+/M79IHyITozXw60NtFBvlT39jUYg5ZI9B3PJPSsMRwGho1e/IryCuroa+H6qqP6BtBn/BA3ss81GL5N7sLOFR88kXBarpBj95+b6SqrpEQf+NN9YFzh3LNlFQsZsUnm/K82j4heoveEeiDjYDunKcvrW445uTXwxONYZbbcss5XFZLYkTn8vMdMZsUN52azg97i/h6Z/MDVde/to7Tnlhx0setbWhqcUPaG+yfpKrrrYQEGG86t54+gMcuHcUpA2L54KdcKmrbPrgmhHCv3hHobT1355E3x+vR948NITrEnzUHijhcWkNipOdmSrphWhoh/ma+3W3M2+o8vv5ExpxbG5v4/QdbyCqscrkip8dY68ko/5KKugZKqxsI9m+ZJrvrrEEUVNbx9y93e6mBQvQevSLQR5xEoDeZFFMHxLD4x1yq6hs7NbTyeCxmE/1iQsgqMkoF2GvEAI4bwq7IKqpi/tpsrnllrfcD/aZ3OGfPQ/yf+QNW7y8ivNXvelxqFHMnpfD2umzKpVcvhEf1jkDfKnXT2KSpqLUeM9AD/P68YY6fB8aHHmPLzkuPDSHLVhOm2Knq4/nPfc8Di7d0tFsLtQ1G8bDc0hpqGrycox93Hdkpl3CPZRHXmJfzs4x+bTa5alIq9dYmvnTTVI9CiPb1ikBvz8Xbn44ttwX84wV653Hzw2w5e0/pFxPMoZIampo0JVUte7jvrMt26RjON5tLqhsI8vdioDeZyJ3+OMsbx/Go33+Zbv2+zSZjkiOIDwvocKpHIYR7uBTolVKzlVK7lFJ7lVL3t7P+aaXURtvXbqVUqdO665VSe2xf17uz8a5qnaMvdTHQQ3OA79PJMfTHExsaQGOTpry2gWJb6mZivyjHeldG4NgfTLILdONw0JMxICGKOxruJFMPxv/DW2Ffy5vLSilmDonn2z0FXTopixC9zXEjgVLKDDwPnAsMB+YppYY7b6O1vltrPVZrPRb4J7DYtm808CdgCjAZ+JNSKoouFuxvxs+kHAHensKxj8Y5lndvzeD7+2Z2+GCVu0SFGG1Zd6CYP9kqZz47b5zj3sBpT6zg5W/303iMkgnV9S2LpAV6s0cPxIUFUEsAP6+/B2IHw7vXQu6PLbaZOTSOilorGw6WeKmVQvg+V7p8k4G9Wuv9Wut6YAFw8TG2nwe8Y/v5HGCZ1rpYa10CLANmd6bBJ0MpRWxoAPnldUBzoHelRx8eaCE5Ktij7QOIsqWXbnlzAyW2Tx59IwKZNjDWsc1jS3bw1Y6O89n2Hn1ylJFyij7G8NGuoJQiLiyAwf1S4Nr3ITga5s+Bwr2ObaYNjEUp4w1OCOEZrhRXTwKcn+bJweiht6GU6gekA18fY9+kdva7BbgFIDU11YUmnbi02GDHqJaCCiPgR53EbFGeEt1OW5RSbVIa+50m8WjN3qN/9OKRbD9cziXj2vyqu9zaB85EKUApuO5DeO0cePMSuOlLCO9LWKCFgXGhbDxUetxjCSFOjis9+vZyFh3lD+YCi7TW9hyCS/tqrV/SWk/UWk+Mi4tzoUknLj021DHT0dbcMoIsZvpFe76n7qqoDnrf9587tMXrTccIiPYqkVMHxHD7zIFeL4EAxjBVR9orZgBcswhqSuHNy6Da6MWPSYlk06FSqVMvhIe4EuhzAOf585KBjp5dn0tz2uZE9/Wo9NhgiqvqKatp4KdDpYxKivDqDEyttf50cZmtN54YEcQNp6Q5lq/aV9Rhnr66rhGTggAv34Q9pr5jYd7bULwP3r4K6qsZkxJJUVV9m3l9hRDu4UpEWA8MUkqlK6X8MYL5x603UkoNAaKA1U6LlwJnK6WibDdhz7Yt63JxYQEAbDxUyqZDpZw6KPY4e3StEH8zIbabpzOGxPHEnNGOdWGBRobtkrF9KatpYNeRinaPUV1v1JTx9I3jTkufDpe/CrmZsPB6xvY1yhdL+kYIzzhuoNdaW4E7MAL0DuA9rfU2pdQjSqmLnDadByzQTp+/tdbFwKMYbxbrgUdsy7pcZJDRY16y+TAAF4xO9EYzOqSUIt02MUdKVHCLTxu3zxzIv64ex3VT0wA4Wt5+SePqeqt3x86fiOEXwfn/gD1fMmL9AwT6wU/ZEuiF8ASXZrrWWi8BlrRa9sdWrx/qYN/XgNdOsn1uYx9Kueuo0RvuipE0J6rJdt+19ZSFgRYzF4zu6xhLX1BZ1+7+NQ2NPSfQA0y8EaoLMX39Z54Kb+COH+Zwy/T+9PFguQkheqNunMx1L/vTsbuPVhAd4u/W2vLuMjHNeMTggjHtf9qIDTXST/ZRQ63VNTR17/x8e067F6bcxgXVH3Cb+RNeXLm3w01rGxr5asdRKuusvLMu+5jPFAghmrnUo/cF9qdjq+sbSe1Go22c/e68Ydx0ajrxYe33aIP8zYQG+FHYQY++ztpIgF8P6tGDMezynL9AdSH3bVnAY5ujKD7rEXYdqWDqgJgWm/5j2W5e+nY/MSH+FFXV88DiLSz/9eker0MkRE/Xw7p/J8+5emK8h8sZnKxAi5l+MSHH3CY21J/Cyvp219VZe2CPHsBkgotf4GD0Kdxv/TevvvxP5r28ps0bWm6pMSqnyKno29PLpMyxEMfTA6PCyTGbFOG20SsJthE4PVFSVBAHi9p/aKre2uTVCcE7xc+fjRnPskkP4M6Sv5Jh2s7EPy/njrd/5PMtxg308MDmN+uB8aFEBVtYs79Ixt8LcRw9NCqcHPvTpyndNHXjipF9I9h5uIJ6a9siYHXWJvy70bMBJyouJpob639LNgm8bPk7I1QWn24+zG3zjfo4dU7VOX915iDuP3coRVX1/OvrjvP6QoheFujtN2BToj0zLWBXGJUcQX1jE1vamVi7R+bonfQJD6SMUK6ru49acyiv+z9OP3UEgOdX7G0x2qhfTDBnD+9DclQQL6zcR0lV++ksIUQvC/R+JuNyPTX/a1c4bVAcgRYTN7y2jkc+2d5iXV1PTt1g/Lv4mRRHiOHlfk8REWDi44h/EEcJTy7dxXd7CpmcHs3frxjDqKQIokL8+fe1E6hvbOKehZu83Xwhuq2eGxVOwq2n9wdgUA8epRERZOH35w0jOtSfDzfmtshP1/fUm7E2Qf5mfmZ7KGw/yfj/bDERjSX8kPQ8UxKN6xqdFMHlE5IdT/+OTIrgttMHsGJXPofLpISCEO3puVHhJFw8Nomsx88nJrTn3owFuG5qGrdM709xVT3vZR6iyTaevM7a1C2fDzgRd88axLDEcOZMSIbkCTD3LfxL9vBu2HNs+f1pPOA0vaPdRWP7ojV8v6fQCy0Wovvr2VGhFxuXYjxcdd/7W3g306gEXdfQs3P0AGGBFj7/1WnMHtnHWDDgDLjsJcheTdint2LWjW32GRgXSligHz9JrRwh2iWBvocanNCcftp5uBytdc8dR388Iy+D856EXUvgk19Bq+GUJpNibEok62XyEiHa5YNRoXfwM5sw2YpUvrH6ICt3F2Bt0j2+R9+hyTfD6ffDxrdg+UNtVk8fFMee/EpySo4/t64QvY0E+h5sze/OdPy8Pa8coMfn6I9pxv0w6efwwzOw6p8tVtnLJWw61HbYaVfacLCYm15fT521bYpJCG/x4ajg++LDAvn3teMBqKg1ZpfyydSNnVJw7hMw4lL48g+w8W3HKvs8uUc6KOHcVb7emc9XO/PbvOEUVtbx63c3yiTowit8OCr0DueM6IO/2eSoUd+Tx9G7xGSGS/8D/WfAR3fAri8AY9hpgJ+JI7YhlrmlNV4pjZBrmyVr7f6iFst/2FvI4p9yufzFVSzddqTL2yV6Nx+PCr5PKUVksIUjZUag78klEFzmFwBXvQWJo2Hh9XBwNUopEiMCOVxWS1ZhFdMe/5r/fLu/y5tmL7y2ttWNYedCdLe+uYFl248CUFpdz5l/X3nMuYCF6KxeEBV8X3SIv6PQmXOVTp8WEGZMNB6RAu9cBUe3kRAeyJGyWjbbykMstA077Ur2Hn3mweIW9YgKK+swKfj3teOJDwvgjVVZAOwrqGRfQRVLth7u8raK3kMCvQ+IDLaQZ+vRx4b6H2drHxISC9ctBksIvHkZY0JLySqqZqst0NvvW3SVOmsjR8prSY4KorahqcUIoMKKOuLCApg9MpHzRiWy4WAJDY1Njly+DA0VniSB3gekOdWwj+3hT/2esMhUI9hba7k95zfoyny+2VUAQHFVveOp4a5wsKiaJm3UIwLIKWkuyVBUVe/4t5nQL4qahkYWrMvmkU+NekWbc8qoqZeROsIzJND7gDvPHOT4uaeXdzgp8cPgmoWE1hfwuv/fyD1q5L+tTZri6q6rarm/wEifTR8UCzTn68FI3TgHeoC/fbHLsd7apPkpW0bkCM+QQO8D+kY2V+MM6UmTg7tTymTqL3+DoaZDvGT5B7OHRAJwtLzW9tSw53vL2cVGoM/ob4zpf2DxFt7fkMPGQ6UUVtQRY8Byg9AAACAASURBVEur2f+9KuuaU0tKtb2BK4S7SKD3Mfaqjr1R0PDZVM5+jlPM23lMP4uJJt5YlUX6A0sY8/CXHh9uWVrdgJ/JGAVlf8O9Z+EmLnn+Bwor64lz+rT1K6dPYQDDE8NZnyWBXnhGr5kc3Nd9fMe0DicN702iMq4FKoj54n7+7NfA7zJvAhS1DU0cKa/16FwE5bUNhAdZUErxw/1nMPaRZY519Y1NLe6f3D1rMHfPGsyh4mqq6q28u/4Q76zLpt4HKpCK7kf+onzE6ORIzhia4O1mdA8Zt6FPvYer/b7m134LmTspBYDdRysByC6q5qVv97m9h19eY3XMSxwZ3Hb0U2xY22Up0cEM7RPO5LRoahvanzlMiM6SQC98kjrzQZYGnMOdfh/y+9hvABwPJT3y6Xb+smQnn21x79j1itoGwpwmML/p1PQW6481ImpCmnGDVm7ICk+QQC98k1Jk3PkG1QPOJWzFH7inzyYWrMumqan5xuxdCzayOcd9T6SW11oJD2rOhv7+vGEtcvHHCvTxYYHEhvqz60iF29ojhJ0EeuGzIkKCCJ77OqSdxu1lf2dQxVo25pRyoLCKqf1jiA0N4NfvbcLaaDzBujW3jKzCKsf++RWuFUjTWlPb0Eh5TQPhTj16k0kxfXCs47Xz6Kj2DO0Tzk4J9MIDJNAL32YJhLlvo+OG8aLlGd5atIickhpOGRDDfecOYW9+pSMvfsE/v2fGUysBo9zw5Me+4s53fqLxOA9dPbl0F6MeWsrR8toWgR5aBveI45SnmJwezZbcMtIf+IyPNuaexMUK0T4J9ML3BYZj/tliik1RPFj2EANVDoP7hDFtgNHbzswq4b31LevirDtg5Mo/3pR3zGqTTU2aF1buo6FRU15rJSyw5UC2+LBAl5t5/SlpxIT4ozX8asFGVuzMd3lfIY5FAr3oHULj2XfOmzTgx//8H2dYcBnx4YEMiAvhq51H+e37mx2bVtZZ2X64nGDbWPjV+4o6Oir1jU0tXo9Mimjx2mxy/bmGiCALX91zOg9dOByAG2UCE+EmEuhFr3F6xmT2n/M/Ikx1pHx6DVQVceGYvqzZ3/JBpZF/Wsonm/KYOTSe0wbFknmMyULqrC0D/RnD4tts8/5tp7D816e71MbIYH/mTUl1vN5jGxIqRGdIoBe9SsYppxNywyJU2SGYP4cz+jcXhHv04hEttp0+KJbhieHsy6903LBtrcG2/KELh/Pjg7Pa5OjBqG0zMD60zfKOBPiZ+foe443hh72FLu8nREck0Ivep98pcMXrcHgTI7+/HQtGzZmU6GA+u/NUx2YzhsQzKCGM+sYmfv6/zHYPZa85H2gxEx3ivhLR/WJCGBgfynNf7WlR116IkyGBXvROQ86Fi/6Jaf8KXgl/GUUTSZFBjOgbQVSwhegQfxLCAxmbYuTcV+4qYMfh8jaHsffo3V22wGxS3D5zAFX1jY5JZYQ4WRLoRe817hqY9Qin13/H8qGfMTDOSON889uZfPvbmQAMjA/jN+cMAeDcZ7+jrLqhxSHsvW2LB6ZwHBQfBjSXbhDiZLn016mUmq2U2qWU2quUur+Dba5USm1XSm1TSr3ttLxRKbXR9vWxuxouhFtM+xWccicDst5BffskAOGBFkIDmodJjkuNdPy8fMfRFrvXe6hHDzAwPhSzSUn9G9Fpx/3rVEqZgeeBc4HhwDyl1PBW2wwCHgCmaa1HAHc5ra7RWo+1fV3kvqYL4SazHoGx18DKv8D6V9qsnto/hscvGwVAdnF1i3X2Hr0nJmUPtJjJ6B/Nl9uOeLzEsvBtrvx1Tgb2aq33a63rgQXAxa22uRl4XmtdAqC1lic9RM+hFFz4HAyeDZ/dC9s+aLVaMXdyKokRgRwqaRnoGxqNAOyJ1A3AxWOS2F9YxQ97Ox7LL8TxuPLXmQQ4PzaYY1vmbDAwWCn1g1JqjVJqttO6QKVUpm35Je2dQCl1i22bzIKCghO6ACHcwuwHc/4LqRnw/s2wb0WbTVKigskprmmxzNGj91AN+YvH9SUswI8lW91baVP0Lq78dbb3aF/rz5F+wCBgBjAPeEUpZU9spmqtJwJXA88opQa0OZjWL2mtJ2qtJ8bFxbnceCHcyj8Y5i2A2MGw4BrI3dBidVpsMHsLKnn00+0Me/ALoHnUjcXsmZm9AvzMjEyKYFtuGVprVu8roqBCJpgRJ8aVQJ8DpDi9Tgby2tnmI611g9b6ALALI/Cjtc6zfd8PrATGdbLNQnhOUCRctxhCYmD+FVC4x7FqfGoUxVX1vPr9AWoaGimqrHM8GevJWaFGJoWzLa+cp77cxbyX1/DEFzs9di7hm1z561wPDFJKpSul/IG5QOvRMx8CMwGUUrEYqZz9SqkopVSA0/JpwHZ3NV4IjwjrA9d9CMoEb14K5Ua/ZqJtchC7zTllzePoPZSjB7hyYgqBFjPPr9gHwPs/5si0keKEHPevU2ttBe4AlgI7gPe01tuUUo8opeyjaJYCRUqp7cAK4Dda6yJgGJCplNpkW/641loCvej+YgbANYugphTevAyqi0mPDSXQ0vxf5l8r9no8Rw8wKCGMT//vVK6cmExKdBBNGn751o8eO5/wPS79dWqtl2itB2utB2itH7Mt+6PW+mPbz1pr/Wut9XCt9Sit9QLb8lW212Ns31/13KUI4WZ9x8K8t6F4H7x9FWZrteMhJoANB0scPWtPjbqxS4sN4Yk5Y3h2rpH5XH+wWIZcCpfJk7FCHEv6dLj8VcjNhPeuZ3iCMZFIWkwwAJtsUxF6skfvbHxqFH++ZCRaQ05JzfF3EAIJ9EIc3/CL4IKnYe8yfl78dxRNDO0TDsAqW616T/fonQ1LND5VrNxdQHFVfZedV/RcEuiFcMWEG+CMBxl05DP+4DefemsjE/pFUWqrfRPQRT16gIFxRqB/8MOtXPjP77vsvKLnkkAvhKtOu4fq8Tdzk9/n/CZkCReN6etY1ZU9+ojg5pr3uaU1kqsXxyWBXghXKUXwBU/AqCsZtv1pLtNfOVadyJSB7vDs3LGOn/cVSBljcWwS6IU4ESYTXPICDJxF2PJ7WXh6IVc7Tf3XVS4em8Ta351JoMXEs1/tOf4OoleTQC/EiTJb4Mo3IGkCkzJ/w1/GlnqlGQnhgdx8Wn8+2ZTH7qMVXmmD6Bkk0AtxMvxD4Or3IDod3pkHhzd5pRnXn5KGScGnm1pXJRGimQR6IU5WcDRcu9ioj/PW5VC0r8ubEBsawKS0aL7cfvT4G4teSwK9EJ0RkQTXfQC6yaiLU3Gky5swc2g8O49UcLS8tsvPLXoGCfRCdFbsILhmIVQVGj37mq7N2Z8+2Cjt/c1umctBtE8CvRDukDQB5r4FBbuMnH1D15UnGNonjITwAH67aDMPfbyty84reg4J9EK4y4Az4LKXIHs1LPp/0GjtktMqpZjQzyih/PqqLKy20slC2EmgF8KdRl4G5z0Ju5bAJ3dCFz21mhQZ5Ph5S25Zl5xT9BwS6IVwt8k3w4wHYON8WP6nLjlln4jmQL/9cHmXnFP0HH7eboAQPun0+6CqAH54FoJjYdqdHj3ddRn98Pcz8ein29lztNKj5xI9j/TohfAEpeDcJ2DEpbDsQdj4tkdP5+9n4rqMfgxLDOf1VVl8t0dG4IhmEuiF8BSTGS79D/SfAR/dAbs+9/gp75s9hPiwAB77bIfHzyV6Dgn0QniSXwBc9RYkjoGFN8DB1R493SkDYrl4bF8OFFbR1CTli4VBAr0QnhYQZjxQFZECb18FR7Z69HRpsSHUWZuY+9Iaj55H9BwS6IXoCiGxcN1ioxjaW5dDSZbHTpUSZcxnuy6rWMoiCEACvRBdJzLVCPbWWqMuTmW+R04zpX80Zw2LB2Dptq6vvSO6Hwn0QnSl+GFGGqfiiNGzr3X/mPcAPzOvXD+JgfGhfL5FAr2QQC9E10uZDFf+D/K3w4KrocEz6ZVTB8ayOadU5pQVEuiF8IpBs+CSFyHrO1j8c2hqdP8pEkKpqm8kt7TrCqyJ7kkCvRDeMvpKmP047PgEPr3b7XVxBieEAbA9T0oi9HYS6IXwpozb4LR74Mc34Os/u/XQI/tGEBsawBurs9x6XNHzSK0bIbztjAeNSUu+e8oYhplxm1sOG+Rv5sIxiSxYdwitNUoptxxX9DwS6IXwNqXggqehphi+uB+CY4y0jhukRgdT09BIUVU9saEB1DY0Emgxu+XYoueQ1I0Q3YHJDJe9AmmnwYe3wZ7lbjms/eGpQ8XV1DY0MvTBL2QWql5IAr0Q3YUlEOa+DfHD4b3r4NC6Th8yJdoI9FlFVRRU1AHGLFRbcmRykt5EAr0Q3UlgOFz7PoT1gflXQH7nqlD2jwshKtjC1zsLKKqqdyz/14o9bbbdmlvG6IeW8uzytutEzyaBXojuJjQervvAqHz55mVQeuikD2Uxmzh/dCJfbjvC7qMVAExJj2b5jnzynergWBubuOCf31Nea+Xp5bs7fQmie5FAL0R3FJUG1y6G+iqjLk5V0Ukf6mdT06izNvHm6oMA3D5zII1Nmo835Tm22eSUygn2N8vTtD5GAr0Q3VWfkXD1Aig7BPPnQF3FSR1mUHwo0SH+jknDJ6ZF0Sc8sMXcspsOlQLwf2cMpLq+kSNS9dKnuBTolVKzlVK7lFJ7lVL3d7DNlUqp7UqpbUqpt52WX6+U2mP7ut5dDReiV+h3ClzxOhzeBO9eC9a6Ez6EUoq+kYGA0VsP9vdjQHwIS7ce4aVv96G1JqekhmB/MzOGGFUvN2aXuvMqhJcdN9ArpczA88C5wHBgnlJqeKttBgEPANO01iOAu2zLo4E/AVOAycCflFJRbr0CIXzdkHPh4n/B/pXwwS9Oqi7O3EmpjEwK59m54wDoH2vUwfnLkp3kldWSU1JNclQQo5IiCLSYWJ9V4uaLEN7kSo9+MrBXa71fa10PLAAubrXNzcDzWusSAK21vdD2OcAyrXWxbd0yYLZ7mi5ELzL2apj1KGxbDJ/fd8J1ca7N6Men/3cas4YnAHDp+CRSbUMvfzxYwvd7C0mKDMLfz8TYlEgyDxa7/RKE97gS6JMA59v+ObZlzgYDg5VSPyil1iilZp/AviilblFKZSqlMgsKZPZ6Ido17U445U5Y/zJ880SnDjU+NYov756O2aS4//3NVNc3OoqgTUqLZlteOVV1Vne0WnQDrgT69gpktO5O+AGDgBnAPOAVpVSki/uitX5Jaz1Raz0xLi7OhSYJ0UvNegTGXgMr/wLrX+nUoQItZib0i6KqvhF/s4nfzh4KwMS0aBqbND9Jnt5nuBLoc4AUp9fJQF4723yktW7QWh8AdmEEflf2FUK4Sim48DkYch58di9sXdypw80e0QcAfz8TZpPRLxufGolJwfosSd/4ClcC/XpgkFIqXSnlD8wFPm61zYfATAClVCxGKmc/sBQ4WykVZbsJe7ZtmRDiZJn9YM5rkJoBi2+BfStO+lBXTTL6YTdOS3MsCwu0MCwxXAK9DzluoNdaW4E7MAL0DuA9rfU2pdQjSqmLbJstBYqUUtuBFcBvtNZFWuti4FGMN4v1wCO2ZUKIzrAEwbwFEDsYFlwDuRtO6jAhAX7s/vO5/HrW4BbLJ6VFs2Z/EXmlNfxlyQ5e+nafO1otvER1tyfgJk6cqDMzM73dDCF6hooj8Oos4wnaG7+AuMHH38cFK3blc+N/17dYtu8v5znSO6L7UUpt0FpPbG+dPBkrRE8W1geu+xCUCd66DMpy3XLYmUPimZwW3WLZnH+vcsuxRdeTQC9ETxczwKh4WVNqBPtq92RHfzGjPwDv3pJBclQQP2WXcrhMJhrviSTQC+ELEsfAvHegeD+8fZWRyumkM4YmsOWhs5nSP4Y3b5oCwCvfHej0cUXXk0AvhK9IPw0ufxVyM+G966GxodOHDAu0GIeODWFUUgSvfn+AL7Ye6fRxRdeSQC+ELxl+kTH/7N5l8OEvoanJbYd+bt44TAr+tzrLbccUXUMmBxfC10y4AaqL4KtHjInGZ//VeNCqk9JjQ7h6Siqfbj7c+TaKLiWBXghfdOqvoaoQ1rwAoXFw2j1uOWxaTAil1Q1kFVZhUorUGKMwWkFFHdEh/jL8spuS1I0QvkgpOPsxGHWl0bPf8IZbDtsvJgSAGU+t5PSnjCdy9+ZXMOmx5Ty/Yq9bziHcTwK9EL7KZIJLXoCBs+DTu2B768olJy49Ntjxs9bwzPLdvLUmG4B/SaDvtiTQC+HLzBa48g1Imgjv3wQHvu3U4ZKjglu8fmb5HtYdMMbt11ubKK2u79TxhWdIoBfC1/mHwNXvQnR/eOdqY1rCkxRoMTt+HhgfCsD2w+WE+BvLDxZVd66twiMk0AvRGwRHw7WLISgS3roctizqdLmEZXdPx99shJDThxjzSBwsNgK91pry2s6P4xfuIYFeiN4iIgmu+wBMfkYa5+nh8PRIWHQTrHsZDm92aT7a/1w3gT9eMBylFGm2nP3/m5aOv9nEog05VNVZefm7/Yx+6Eue+2qPp69KuECqVwrR2zQ2wJEtcGgtZK8xvlfYxsb7h0LyREjJgNQpkDwJAsI6PNTe/Er2FVRyzog+vLEqiz99vI3kqCBySppr4pw9PIHn5o1rkfYR7nes6pUS6IXo7bSG0mw4tA4OrYHstXB0K6CNqpjxI4ygbw/+ESkdPoB143/XsWKXMe/zY5eO5PcfbAXgn/PGceGYvl11Rb3SsQK9PDAlRG+nFET1M75GX2Esqy2HnPVGb//QWti0oHmO2rC+tsBv++oz2pj1CjhvVCIrdhUQGWxh3qRUvt6Rz1c78/loY26LQF9W3cCba7K46dT+BPlLT9/TJNALIdoKDIeBZxpfAI1WyN9m9Pbtvf5tHxjrLMGQNAFSMzg3fgKPUkVG+gBMJsWrN0zisc+2898fsiipqicqxJ/iqnrOf+47DpfVkhgRxOUTkr13nb2EBHohxPGZ/YxSyIljYMotxrKyHFue3xb8v/sHobqRTYGKpvKh8EkGpGYwp/9wXv6uic+2HObajH68sy6bw2W1ACzddkQCfReQQC+EODkRycbXyMuN13WVkJuJOrQOc/Ya2Po+bPgvQ4ANQVFkfTuCsuqzWPe9P1P7jWRC/z78a8Vedh4pZ2ifcK9eiq+TQC+EcI+AUOg/w/gCY6hm/g44tIaD339BQtkmIr79njeApsIAGi3jCbHEs+WrPIZeerkx1l94hIy6EUJ43Dvrsnlg8RbiKeHhcRWcG5EN2WtozNuEGdvY/dghTqN7Mowned1QXrm3kFE3QgivOnNoPAD5RDH7ymscATw7r4D7//kG948sZRy7YPtH8OP/ALAGxqBSp2Dul2EE/75jwS/Aa9fQk0mgF0J4XHx4IPfMGkxqTDDKqZeelhjLzsDRvBvYh3GXj4amJpryd7Lk84+o3b+KGVmbid29xNjYHAB9x0HKZKPHnzKFDYVm/vDhVt64cRLx4YFeurruTwK9EKJL/N+Zg9osU0oxOjmCBesPceO0dIb0CeOHijju2DUaGE1sgD+Z945pHs+fvRbWvAirngMgWidyY+Ng1r1/ChdccCnEDpZ0Tzsk0AshvOrqyal8t6eQa19dy/rfn8VXO/Id6ypqrViDYvEbdiEMuxCAkrJyQou28N4Hi4gv3chZ5g1EH/wGnv8rBEU1P8iVmmF8ArAEeevSug0J9EIIr5o9sg+zhiewbPtRjpbXsvFQqWNdnbWJJ5fu4pczBxIRZKGgoo5Jf/2OKyYks7DgLK6Z8v/4qrGJbVt/5OOLzCh7r3/3F8YBTBZj7L8t1UNqBoTGe+lKvUdG3QghvG5rbhkX/PN7xqVG8lN2KTdOS+MP5w/nqv+sJvNgCQD7/nIec19azfqsEsd+r984ieziav740TbmTkrh8ctHGyuqiiBnXXPRttwfobHOWBeVbgv6thE+cUON2bh6OBl1I4To1kYmRXDRmL58vCkPgIz+MZhNiisnpTgC/fIdR1sEeYDhfcMZ0TeCP360jU825fHwxSMI8DOjg6P5KSiDETPPps7aRLhfExzeRN2BVejstQTu+wo2LzAOEhgByZOb6/ckTTAma/EhEuiFEN3CxWONQH/m0HjOHp4AQIrT1IVPLd2FScHl45NZuCGHQIuJuNAAlFK8ev1Ebnojkw1ZJZwyMJa1B4qZ+9IaAIb2CeOLu6ZDymRuXQ4rdw1i04NvEFF7yKlU8zr4eplxIpMf9BnVXK0zJQPCE7v89+FOEuiFEN3C6YPjuG/2UK6alOIYgpkS3XwjdU9+JReP7cvUATEs3JBDbUOTY7tJ6dEoBettgb6i1urYb+eRCnJLa8jMKmalrYTyh5vyuP6UARAzAMZebWxYUwKH1jcXbdvwOqx90VgXkdrc40/NgPjhYOo5VTcl0AshugU/s4nbZgxosaxvRBAXj+3LRxuNlM4pA2KYlNa2VEJ4oIWhfcJZn2VMVF5ZZ0xjOLFfFJkHS/j70l0s/ql56sTNOWVtGxAUBYPPNr7ANkHL5uaibQe+gy0LjXUB4bYJWmzBP3niMSdo8TYJ9EKIbstkUjw7d5wj0KfFhJAcFURCeABXT+7XYtvJaVEs3JCDtbGJsmoj0L/0s4n8dtFmR5D/19XjWJiZQ+bBYhqbNGbTMcbcmy1Gvj5pAkz9pW2CloO2wG/7Wvk4jglaEkY2j+5JmQKRKR75nZwMCfRCiB4jLTYEpRRrf3dWm3UT06J5Y/VBth8up6zGSN2EBfrxh/OHsXzHUQCmDYilrKaB33+wlUc/3c7PT0sn2ek+wDEpBVFpxteYq4xltWXGBC32Xv9P82HdS8a68KTmVE/KFOONwOydkCuBXgjR7b1wzXhe/yGL+LCOa91MTjdSOmv3F/Pu+myCLGYsZhNpsc0jaCKDLVwzpR8bs0t5fVUWr6/KYtvD5xAScJKhMDACBp5lfIExQcvRrS3n49222FhnCYHkCS3n4w2MOLnzniAJ9EKIbu+8UYmcN+rYI18SwgNJjQ7msSU72qz797UTyC6ucty8vXl6fxZuyAHguz0FzB7pplE1Zj+j+FrfsTDlVmNZ6SGnEg5r4LunQDcBChJGGLV77ME/sp9HSji4FOiVUrOBZwEz8IrW+vFW628AngTsdzv+pbV+xbauEdhiW56ttb7IDe0WQog2zhwWz39/yGqzfPbIPi1eD04I48VrxnPb/B/5cvtR9wX69kSmGF+j5hiv6yogd0NzumfzQsh8zViXMgVu+tLtTThuoFdKmYHngVlADrBeKfWx1np7q03f1Vrf0c4harTWYzvfVCGEOLYrJ6Y4Ar19LH5Hzh2VyGXjkvh6Zz61DY0EWozhkmU1Dbz87X72FVTy5BVjCD3ZtE5HAsLamaBlu9HbV555QteVK5gM7NVa7wdQSi0ALgZaB3ohhPCqYYnh/HLGAEIC/Lh95sDjbn/Z+GQW/5TLJ5vyuGKiMUrm6WW7eX1VFgC7jlbw+a9Oo7FJY1LK8WbgViaz8YBWn1HuP7aNK4E+CTjk9DoHmNLOdpcrpaYDu4G7tdb2fQKVUpmAFXhca/1hZxoshBDH8tvZQ13edtrAGJIig/h4Ux6VdVbey8whzKkHv7+giie+2MVHG3MJC7Sw6BdTiQltviGsteaNVVl8tTOf358/rNvOfevK54T27gy0roT2CZCmtR4NLAfecFqXaiu0czXwjFJqQKt9UUrdopTKVEplFhQUuNh0IYToHKUUUwfE8N2eQh7+ZDs7DpezzvbQ1Ts3Z5ASHcSr3x+gsLKeA4VV/GPZbse+ZTUNpD+whIc+2c53ewr53eIt7M2voKGxyVuX0yFXAn0O4DzyPxnIc95Aa12ktbaVhuNlYILTujzb9/3ASmBc6xNorV/SWk/UWk+Mi4s7oQsQQojOuOnUdK6f2q/N8qkDYjhnuHET96krxjBvciqLNuRQUlUPwPa8cse2g+JD+TG7lLP+8S23vbWhaxp+AlwJ9OuBQUqpdKWUPzAX+Nh5A6WU8y3ri4AdtuVRSqkA28+xwDQkty+E6EaGJYbz8MUj+fLu6XzzmxmcNiiWv1xq5Mt/ffZg/nrZKC4Z25c5E5KoszY5yiwUVhp926n9Y7jjjOb7Act35PPZ5sNdfyHHcNwcvdbaqpS6A1iKMbzyNa31NqXUI0Cm1vpj4E6l1EUYefhi4Abb7sOA/yilmjDeVB5vZ7SOEEJ43eAEo1bNmzc134IM9vdj3uRUAAbZ1u8vrALgaHktYIzRP1xe49gnPiyATzblcf7o7lPx0qVxQ1rrJcCSVsv+6PTzA8AD7ey3CvDcrWQhhOgi4YEWYkMD2JdfSZ21kf2FVQT4mQgP8iM0MIxbp/fnwjF9eXP1QT7dnMfhshoSI7rHNIY9f1oVIYToIoPiQ1m4IYchf/iCt9dmkxQVhFIKs0nxwHnDGJkUwa2n98fapHlhxT5vN9dBAr0QQrjovFHNT9iOTYnkqSvGtNmmf1woZ4/ow8INh9ia2045ZGD30QpOe+Jr8m3pH0+TQC+EEC66fEIyvzh9AP+5bgIf3j6N8alR7W53z6zBAPzxo628sy6bg0VV/O2LnVhtQy/nrznIoeIaPnCqke9JUtRMCCFcFOzvx/3nHv+BrLTYEM4YGs+SLUf4MbuUpMggcktreHHlPrY/cg6JkUbuPquoytNNBqRHL4QQHnHtlH5MSjN6/LmlzaNydhwup7ahEYD3MnPYeaSc//5wgP9847mcvvTohRDCA04ZGMspA2O5/MVVbDhY4li++2gl5baJUcID/Xjyi118tTMfgKsmpRAZ7O/2tkiPXgghPGh0sjG5yGTbXLcPLN7Caz8cIDEikCsnpTiCPMBPh0o90gYJ9EII4UFnD28eqfOL05tLfR0uq2XO+GTH6/duncrMIfEeaYOkboQQwoOmDojhmavGMqJvOIMSwth0qJTV+4sAHGfAawAABUZJREFU42nbBy8Yjlk1T4XoCRLohRDCwy4Zl+T4+e2bp5D+QHOhgZtOTff4+SXQCyFEF1JK8dQVY0gI73iic3eTQC+EEF1szoTk42/kRnIzVgghfJwEeiGE8HES6IUQwsdJoBdCCB8ngV4IIXycBHohhPBxEuiFEMLHSaAXQggfp7TW3m5DC0qpAuDgCe4WCxR6oDneINfSffnS9ci1dE+duZZ+Wuu49lZ0u0B/MpRSmVrrid5uhzvItXRfvnQ9ci3dk6euRVI3Qgjh4yTQCyGEj/OVQP+StxvgRnIt3ZcvXY9cS/fkkWvxiRy9EEKIjvlKj14IIUQHJNALIYSP6/GBXik1Wym1Sym1Vyl1v7fbczxKqdeUUvlKqa1Oy6KVUsuUUnts36Nsy5VS6jnbtW1WSo33XsvbUkqlKKVWKKV2KKW2KaV+ZVve465HKRWolFqnlNpku5aHbcvTlVJrbdfyrlLK37Y8wPZ6r219mjfb3x6llFkp9ZNS6lPb6x55LUqpLKXUFqXURqVUpm1Zj/sbA1BKRSqlFimldtr+30ztimvp0YFeKWUGngfOBYYD85RSw73bquN6HZjdatn9wFda60HAV7bXYFzXINvXLcCLXdRGV1mBe7TWw4AM4Hbb778nXk8dcIbWegwwFpitlMoA/gY8bbuWEuAm2/Y3ASVa64HA07btuptfATucXvfka5mptR7rNMa8J/6NATwLfKG1HgqMwfj38fy1aK177BcwFVjq9PoB4AFvt8uFdqcBW51e7wISbT8nArtsP/8HmNfedt3xC/gImNXTrwcIBn4EpmA8pejX+u8NWApMtf3sZ9tOebvtTteQbAsaZwCfAqoHX0sWENtqWY/7GwPCgQOtf7ddcS09ukcPJAGHnF7n2Jb1NAla68MAtu/xtuU95vpsH/fHAWvpoddjS3VsBPKBZcA+oFRrbbVt4txex7XY1pcBMV3b4mN6Bvgt0GR7HUPPvRYNfKmU2qCUusW2rCf+jfUHCoD/2lJqryilQuiCa+npgV61s8yXxov2iOtTSoUC7wN3aa3Lj7VpO8u6zfVorRu11mMxesOTgWHtbWb73m2vRSl1AZCvtd7gvLidTbv9tdhM01qPx0hl3K6Umn6MbbvztfgB44EXtdbjgCqa0zTtcdu19PRAnwOkOL1OBvK81JbOOKqUSgSwfc+3Le/216eUsmAE+fla68W2xT32egC01qXASoz7DpFKKT/bKuf2Oq7Ftj4CKO7alnZoGnCRUioLWICRvnmGnnktaK3zbN/zgQ8w3oR74t9YDpCjtV5re70II/B7/Fp6eqBfz/9v5+5VGgiiMAy/p/EHEVSws5CA2ImFhYWFoFikTiEIpvAqRPASvANrCxsJtsZeBP+igsbKQltri2MxZ0ECoohkneV7YMlkdov5YPaEnVkCM/E2wQCwDrRKHtNvtIBmtJukte6ifzN23xeBt+IR7z8wMwP2gXt33/t0Krs8ZjZpZmPRHgZWSRtlp0AjLuvNUmRsAG2PhdSyufu2u0+5+zTpnmi7+wYZZjGzETMbLdrAGtAhwznm7q/As5nNRtcKcEc/spS9QfEHGxx14IG0nrpT9nh+MN4D4AV4J/1ib5HWQ0+Ax/iciGuN9FbRE3ADLJQ9/p4sS6RHyWvgMo56jnmAOeAisnSA3eivAWdAFzgEBqN/KL5343yt7Axf5FoGjnPNEmO+iuO2uMdznGMxvnngPObZETDejyz6CwQRkYrLfelGRES+oUIvIlJxKvQiIhWnQi8iUnEq9CIiFadCLyJScSr0IiIV9wGWH4h0jkDB3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stage 2 - Unfreeze and train some more\n",
    "And now lets train the full model with differential learning rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.03019951581954956, lr_steep=0.25118863582611084)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEMCAYAAADJQLEhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxcdb3/8dcna5u9TdM2TdqmTUsXWpY2QAsXLCBScEERvYCoKFpQBPXen6IXFfVet+vVq6jALcgm+yZWRVkULFAqpHRLV9I2aZMuWZqtSbPNfH9/zLSkaZJmmZPJzLyfj8c8Oss5M28Oybxzzvcs5pxDRERiV1y4A4iISHipCEREYpyKQEQkxqkIRERinIpARCTGqQhERGKcZ0VgZpPN7GUz22Jmm8zsyz1MY2Z2u5mVmtkGM1vgVR4REelZgofv3Qn8u3PubTNLB9aY2YvOuc1dprkEmBm8nQXcGfxXRESGiWdF4JzbB+wL3m8ysy1AHtC1CC4DHnSBo9pWm1mWmeUG5+3RuHHjXEFBgVexRUSi0po1a2qcczk9veblGsFRZlYAnA78s9tLecCeLo8rgs/1WgQFBQUUFxeHOKGISHQzs/LeXvN8sNjM0oCnga845xq7v9zDLMed88LMlplZsZkVV1dXexFTRCRmeVoEZpZIoAQeds4908MkFcDkLo/zgb3dJ3LOLXfOFTnninJyelyzERGRQfJyryEDfgtscc79vJfJVgCfCu49tAho6Gt8QEREQs/LMYJzgE8CG81sXfC5/wCmADjn7gKeAy4FSoEW4DMe5hERkR54udfQa/Q8BtB1Ggfc6FUGERE5MR1ZLCIS41QEIiJh0OHzs2Vf9x0pw0NFICIyzPx+x5cfW8slv3yVJ4v3nHgG4PXSGg61dXqSR0UgIjLMfvrCNp7buJ9JmaO49dkSNlTU9zl9VWMrn73/LX703BZP8qgIRESG0WNv7ubOV3Zw9VlT+NPN55KTlswNv1tD7aG2Xue545UddPody86b7kkmFYGIyDB5vbSGbz1bwrkzx/G9D53M2NQk/u+TC6ltbudLj6yl0+c/bp699Yd55J+7+djCfKZmp3qSS0UgIjIMqppaueGhNRTmpPGbTywgMT7w9TsvL5MffmQ+b+ys5YfPbSWwV/27fvX3UgBuunCmZ9mG5aRzIiKx7rkN+2hq7eT2G04nY1TiMa99dGE+JXsbuPf1XYxKjONrF8/CzNhd28KTxXu4+qwp5GWN9iybikBEZBg8V7KfWRPSmTUxvcfXv/3+ubR1+rnjlR20dfr51vvncPvf3yE+zrjx/BmeZlMRiIh4rLqpjbfKDnLzBb1v3omLM37w4Xkkxcfx29d2UdXUxp837OUz50xjQsYoT/OpCEREPPbC5v04B5fMn9jndGbGbR+cS1JCHMtX7mR0YjxfWFLoeT4VgYiIx/6ycT/TxqUya0LPm4W6MjO+ecls8seMJisliXFpyZ7nUxGIiPShrdNHckL8oOeva27njZ21XH/edAJn5z8xM+NTiwsG/ZkDpd1HRUR6UVx2kNO//yJ/XH/c9bL67cUtB/D5HZfMyw1hstBSEYiI9MA5x38/v42Wdh/f+UMJNX0c+duXv2zcR/6Y0czLywhxwtBREYiI9OD10lre3HWQTy+eSnObj9tWbBrwezS2dvBaaQ1LT57Y781C4aAiEBHpxjnH/7ywjUmZo/iP98/hy++dyZ837OOvJfsH9D5/31JFh89xyfyRu1kIVAQiIsd5eVsV6/bUc9OFM0lOiGfZedOZm5vBt/9QQn1Le7/f5y8l+5iQkczpk7M8TDt0KgIRkS6cc/zshe1MGZvCFQvzAUiMj+O/rziFg83tfP9Pm/v1PjWH2nhlWzVLT55IXNzI3SwEHhaBmd1rZlVmVtLL65lm9kczW29mm8xMF64XkbB7ftN+Nu1t5MsXzjx6YjgInBzuC+8p5Jm3K/nmMxto6/T1+T53r9xJh8/Pp84u8Djx0Hl5HMH9wK+BB3t5/UZgs3Pug2aWA2wzs4edc/1f7xIRCZGqxlZe3lbFr18upTAnlQ+fnnfcNF+96CT8znHHKzvYsq+JO69ZQG7m8SeDqz3UxoNvlPPBUydRmJM2HPGHxLMicM6tNLOCviYB0i0wlJ4GHAS8uQ6biEjQ5r2NPLWmAp/fj885fH7HxsoGSioD1w/OzRzFTy6fR3wPm3Pi44yvL53N/LxM/t+T6/ngr17jN1cv4Kzp2cdMd89ru2jt9HHTBd6eLC5Uwnlk8a+BFcBeIB34V+fc8VdlAMxsGbAMYMqUKcMWUESiS2uHj88/WEx1Uxujk+KJjzPizCjITuFrF8/igtnjmT0x/YS7el4yP5cZ49O4/ndr+OS9b7L8kwtZMms8EDiS+MFVZXzglEnMGH/iU0qMBOEsgouBdcAFQCHwopm96pxr7D6hc245sBygqKjIdX9dRKQ/7l65k8r6wzz6+UUsLsw+8Qx9mDkhnWe+eDafuOefLHtwDf/3qYWcP2s897y2k5YOHzdHyNoAhHevoc8Az7iAUmAXMDuMeUQkih1obOWOV3aw9OSJQy6BI7JSknj4c2dx0sQ0rn9wDb9fW8EDq8q5dH4uM/txgrmRIpxFsBu4EMDMJgCzgJ1hzCMiUewnf92Kz+/4j0vnhPR9s1KSeOi6QBl89fH1HGrr7PO6AyORZ5uGzOxRYAkwzswqgNuARADn3F3AfwL3m9lGwIBbnHM1XuURkdi1fk89z7xdyQ3vKWRKdkrI3/9IGSz7XeCaxL1dhWyk8nKvoatO8Ppe4H1efb6ICAQOEPv+nzYzLi2ZG8/37iIvWSlJPHH9Ys/e30s6slhEotpLW6pYU17H1y4+ifRuF42XABWBiES1t3fXkRhvfHRBfrijjFgqAhGJauW1zUwek0JCvL7ueqMlIyJRraymhakeDBBHExWBiEQt5xzltc1MzU4Nd5QRTUUgIlGr5lA7ze0+po1TEfRFRSAiUausthlAm4ZOQEUgIlGrrCZQBAXaNNQnFYGIRK3y2hbi44y8McdfM0DepSIQkahVVttM/pjRx1xpTI6npSMiUau8tkV7DPWDikBEopJzjrLaZqZpoPiEVAQiEpUONrfT1NqpNYJ+UBGISFQqq20BoGCc1ghOREUgIlGp/OgxBFojOBEVgYhEpbLaFuIM8rXr6AmpCEQkKpXXNjMpazTJCfHhjjLiqQhEJCqV1bboiOJ+UhGISFQqq2nWQHE/eVYEZnavmVWZWUkf0ywxs3VmtsnM/uFVFhGJLfUt7TQc7tAaQT95uUZwP7C0txfNLAu4A/iQc+5k4GMeZhGRGHJk11HtMdQ/nhWBc24lcLCPSa4GnnHO7Q5OX+VVFhGJLUd2HS3QUcX9Es4xgpOAMWb2ipmtMbNP9TahmS0zs2IzK66urh7GiCISicpqWjCDyWNVBP0RziJIABYC7wcuBr5tZif1NKFzbrlzrsg5V5STkzOcGUUkApXXNpObMYpRidp1tD8SwvjZFUCNc64ZaDazlcCpwPYwZhKRKFBW20yBLk/Zb+FcI/gDcK6ZJZhZCnAWsCWMeUQkSpTp9NMD4tkagZk9CiwBxplZBXAbkAjgnLvLObfFzP4KbAD8wD3OuV53NRUR6Y+Gwx0cbG7XQPEAeFYEzrmr+jHNT4GfepVBRGLPrhpdsH6gdGSxiESVldurMYMFU8eEO0rEUBGISFR5YfN+Tp+cxfj0UeGOEjFUBCISNSrrD1NS2cj7Tp4Y7igRRUUgIlHjpc0HAHjf3AlhThJZVAQiEjVe2LyfwpxUpuekhTtKRFERiEhUaGjpYPXOg9osNAgqAhGJCi9vq8Lnd9osNAgqAhGJCi9s3s/49GROzc8Kd5SIoyIQkYjX2uHjlW3VvHfuBOLiLNxxIo6KQEQi3qodNbS0+7hIm4UGRUUgIhHvxc0HSE2K5+zC7HBHiUgqAhGJaH6/48XNVSyZPZ7kBF1/YDBUBCIS0fbUtVBzqI1/mTEu3FEilopARCLa1v1NAMzJzQhzksilIhCRiLZ1XxNmcNIEHU08WCoCEYloW/c3UpCdSkpSOK+8G9lUBCIS0bbub2LWhPRwx4hoKgIRiVgt7Z2U1TYzO1dFMBSeFYGZ3WtmVWbW53WIzewMM/OZ2RVeZRGR6PTOgUM4B7MnaqB4KLxcI7gfWNrXBGYWD/wEeN7DHCISpbbubwRgjtYIhsSzInDOrQQOnmCym4CngSqvcohI9Nqyr4mUpHgmj9GF6ocibGMEZpYHfAS4K1wZRCSybd3fyKyJ6TrR3BCFc7D4F8AtzjnfiSY0s2VmVmxmxdXV1cMQTURGOuccW/c3MXuiNgsNVTh3vC0CHjMzgHHApWbW6Zx7tvuEzrnlwHKAoqIiN6wpRWREOtDYRn1LhwaKQyBsReCcm3bkvpndD/yppxIQEenJkYFirREMnWdFYGaPAkuAcWZWAdwGJAI45zQuICJDcuQcQ1ojGDrPisA5d9UApr3WqxwiEp227mtkUuYoMlMSwx0l4unIYhGJSFv3NzFLm4VCQkUgIhGnvdNPadUhZuvU0yGhIhCRiLOj+hCdfqeB4hBREYhIxNmmi9GElIpARCLOlv2NJMXHMW1carijRAVdyUFERry2Th9b9zVxsKWduuZ2Vm6vYcb4NBLj9bdsKKgIRGTE++6KzTz65u5jnvv8udN6mVoGSkUgIiPeOweaOHlSBt+/bB5jU5MYm5JExmh9fYWKlqSIjHgVdYc5Z8Y4Fk4dE+4oUUkb2ERkRGvv9HOgqZW8MaPDHSVq9asIzKzQzJKD95eY2c1mluVtNBER2N/QinOQryLwTH/XCJ4GfGY2A/gtMA14xLNUIiJBFfUtAORnqQi80t8i8DvnOglcUewXzrmvArnexRIRCaioOwygTUMe6m8RdJjZVcCngT8Fn9Mp/0TEc5V1hzGD3EwVgVf6WwSfARYDP3DO7TKzacBD3sUSEQmorD/M+PRkkhK0b4tX+rX7qHNuM3AzgJmNAdKdcz/2MpiICATWCPLHpIQ7RlTr715Dr5hZhpmNBdYD95nZz72NJiISWCPI00Cxp/q7rpXpnGsELgfuc84tBN7rXSwREfD5HXvrD2ug2GP9LYIEM8sFPs67g8UiIp6qamql0++0RuCx/hbB94HngR3OubfMbDrwTl8zmNm9ZlZlZiW9vP4JM9sQvK0ys1MHFl1Eol1lcNdRHUzmrX4VgXPuSefcKc65LwQf73TOffQEs90PLO3j9V3Ae5xzpwD/CSzvTxYRiR2V9SqC4dDfweJ8M/t98C/8A2b2tJnl9zWPc24lcLCP11c55+qCD1cDfb6fiMSeIweTTdKmIU/1d9PQfcAKYBKQB/wx+FyoXAf8pbcXzWyZmRWbWXF1dXUIP1ZERrKKusOMTU0iJUknSvZSf4sgxzl3n3OuM3i7H8gJRQAzO59AEdzS2zTOueXOuSLnXFFOTkg+VkQigHYdHR79LYIaM7vGzOKDt2uA2qF+uJmdAtwDXOacG/L7iUh0qaxr0fjAMOhvEXyWwK6j+4F9wBUETjsxaGY2BXgG+KRzbvtQ3ktEoo9zTmsEw6S/p5jYDXyo63Nm9hXgF73NY2aPAkuAcWZWAdxG8ER1zrm7gO8A2cAdZgbQ6ZwrGvh/gohEo9rmdlo7/DqYbBgMZQTm3+ijCJxzV/U1s3Puc8DnhvD5IhLFjhxDoDUC7w3ldH4WshQiIt0cOYZAawTeG0oRuJClEBHp5t2jinXmUa/1uWnIzJro+QvfANW0iHimoq6F9OQEMkfrGlhe67MInHPpwxVERKJHh8/PA6vKONDYyjWLpjI1O3XA71Gps44OGx2uJyIhVVLZwNef2sDmfY3Exxm/fW0Xl8zP5YbzCpmfn9nv96mo066jw0VFICIh0dbp4xcvvcPylTsZm5rEXdcsYMGUMdy3qoyHVpfz5w37OLswm+vfU8h5M8cR3G28V5X1hzlr2thhSh/bVAQiEhIPrCrjzld28PGifG69dC6ZKYFt+7csnc0XlxTy6Ju7ufe1Mj5975vMnpjODe8p5AOn5JIQf/w+Kw2HO2hq7dSmoWGiq0GLSEi8VVZHYU4q/33FqUdL4Ij0UYksO6+QlV8/n59ecQo+v+Mrj69j6S9f5W9bDuDcsfukvHsMgfYYGg4qAhEJiY0VDczP63sMICkhjo8VTeb5r5zHXdcswO93XPdAMVfdvZq3d9fR2NpBW6dPxxAMM20aEpEhq2pqZX9jK/Pzs/o1fVycsXReLhfOmcBjb+7mFy+9w+V3rDpuOp1wbnioCERkyEoqGwBOuEbQXWJ8HJ9cXMCHT8/jLxv3B9cI/LR1+MjJGMW4tGQv4ko3KgIRGbINFQ2YwcmTMgY1f/qoRD5+xuQQp5L+0hiBiAxZSWUDM3LSSE3W35aRSEUgIkO2oR8DxTJyqQhEZEgONLZS1dQ2oKOGZWRREYjIkGysCAwUn6IiiFgqAhEZkg2VDcQZzM1VEUQqFYGIDElJZQMzx6czOik+3FFkkDwrAjO718yqzKykl9fNzG43s1Iz22BmC7zKIiLecM4FBoq1WSiieblGcD+wtI/XLwFmBm/LgDs9zCIiHtjf2ErNoTbtMRThPCsC59xK4GAfk1wGPOgCVgNZZpbrVR4RCb0jA8VaI4hs4RwjyAP2dHlcEXxOREaoqsZWfP53zxS6sbKB+Dhjbu7gjiiWkSGcRdDTVSl6uj4yZrbMzIrNrLi6utrjWCLSk4dWl7PoR3/j6rtXs7+hFQgcSDZzfBqjEjVQHMnCWQQVQNeTi+QDe3ua0Dm33DlX5JwrysnJGZZwIhLgnON/nt/Gt54t4fQpY9hY2cClt7/Ky9uqKKls0PEDUSCcRbAC+FRw76FFQINzbl8Y84hINx0+P7c8vYFfv1zKvxZN5vFli1jxpX9hfHoyn7nvLWqb2zVQHAU8O0OUmT0KLAHGmVkFcBuQCOCcuwt4DrgUKAVagM94lUVEBsY5x2ulNfzq76W8uesgN184k6++dyZmxozxaTx74zn815838/hbe1g0PTvccWWIrPsl4ka6oqIiV1xcHO4YIlHpcLuPp96u4IFVZZRWHSI7NYlbls7u9RTR7Z1+khJ0XGokMLM1zrminl7TOWNF5KgbHlrDP7ZXMz8vk5997FQ+cGouyQm9DwSrBKKDikBEAFi9s5Z/bK/maxfP4otLCjHracc+iUaqcxHBOcfPX9xOTnoy1/3LNJVAjFERiAhv7KjlzV0HuXFJoY4JiEEqApEYd2RtYGLGKK48c0q440gYqAhEYtxrpTUUl9dx4wUztDYQo1QEIjHsyNpAXtZoPl6UH+44EiYqApEY9tKWKtburudLF8zoczdRiW4qApEYVV7bzNeeWs/M8WlcsVBrA7FMRSASgxpbO7jugcAR+nd/qojEeH0VxDL93xeJMT6/4+ZH11JW08wdn1hAwbjUcEeSMNORxSIx5sd/2cIr26r5wUfmcXbhuHDHkRFARSASI5xz/OblUu5+dRefXjyVT5w1NdyRZIRQEYjEAJ/f8d0Vm/jd6nI+fNokvv2BueGOJCOIikAkyrV2+Pjq4+v4S8l+lp03nW8snU1cnM4lJO9SEYhEsYPN7dzw0Bre3HWQb71/Dp87d3q4I8kIpCIQiVKb9zay7HfFVDW28csrT+Oy0/LCHUlGKBWBSBRasX4vX39qPVmjk3jihsWcNjkr3JFkBFMRiESZX7y0nV+89A5nFIzhjk8sJCc9OdyRZITz9IAyM1tqZtvMrNTMvtHD61PM7GUzW2tmG8zsUi/ziES7+pZ2fvX3Ut5/Si4Pf26RSkD6xbMiMLN44DfAJcBc4Coz677P2reAJ5xzpwNXAnd4lUckFvx9axU+v2PZudN1PWHpNy9/Us4ESp1zO51z7cBjwGXdpnFARvB+JrDXwzwiUe+FTQeYkJHM/LzMcEeRCOJlEeQBe7o8rgg+19V3gWvMrAJ4DrjJqzDltc3c9/oumts6vfoIkbBq7fCx8p1qLpo7QccJyIB4WQQ9/SS6bo+vAu53zuUDlwK/M7PjMpnZMjMrNrPi6urqQYXZsq+R7/1xM7tqmgc1v8hI93ppDS3tPi6aOzHcUSTCeFkEFcDkLo/zOX7Tz3XAEwDOuTeAUcBxZ8Fyzi13zhU554pycnIGFWZqduAMi2W1KgKJTi9uPkBacgKLpo8NdxSJMF4WwVvATDObZmZJBAaDV3SbZjdwIYCZzSFQBIP7k/8EpmanAFBe2+LF24uElc/veGnLAd4zK0dXGpMB86wInHOdwJeA54EtBPYO2mRm3zezDwUn+3fg82a2HngUuNY5133zUUikJCUwISNZm4YkKq3bU0fNoXbeN3dCuKNIBPL0gDLn3HMEBoG7PvedLvc3A+d4maGrqdmplGvTkEShFzYfICHOWDJrfLijSASKqR2NC7JTKNOmIYlCL24+wKLp2WSOTgx3FIlAMVUEU7NTqW5q0y6kElVKqw6xs7qZ952szUIyODFVBNOC12bVgLFEkxc3HwDgvXNUBDI4MVUE7+45pHECiXydPj93vrKD/31pOwumZDEpa3S4I0mEiqmzjx45lmCXikAijHMOs3eP0dyyr5GvP7WBjZUNXHzyBP7zw/PCmE4iXUwVQVpyAuPSkimv0aYhiRxryg9y1d3/JN6MlKR4UpLj2VffSuboRH5z9QIunT/xmJIQGaiYKgI4sueQ1ggkcjy/6QA4uGbxFFrafbS0+8iak8hNF8xkbGpSuONJFIi9IhiXymvv1IQ7hki/rdpRw4KpWdz6/u5ncRcJjZgaLIbAGsH+xlYOt/vCHUXkhOpb2tm0t5GzC487BZdIyMRcERwZMC4/qM1DMvKt3lmLc3DOjOxwR5EoFnNFUHDkLKQaMJYIsGpHLSlJ8ZySr4vPi3dirgimjtOxBBI5Vu2o5cxpY0mMj7lfVRlGMffTlTEqkezUJJ1zSEa8qsZWSqsOcXahNguJt2KuCCBwhLHWCGSkW7WjFkADxeK5mCyCguxUynRdAhnhVu2oIXN0InNzM8IdRaJcTBbB1OxU9ja00tqhXUhl5Fq1o5bF07N1IXrxXEwWQUFwwHjPQY0TyMi052ALFXWHOVu7jcowiM0iOHohexWBjEyvlwaOftf4gAyHmDvFBLxbBBowlnCpbmrjjZ217DnYwu7aFirqW5g5Pp1/PWMyc3IzWLWjlvHpyRTmpIY7qsQAT4vAzJYCvwTigXuccz/uYZqPA98FHLDeOXe1l5kAMlMSyUpJ1IXsJSz2HGzho3euoqqpDYCc9GRyM0fxyD93c/+qMk7Jz6S8toXzZ+XorKIyLDwrAjOLB34DXARUAG+Z2YrgBeuPTDMT+CZwjnOuzsyG7crbgQvZa9OQDK/aQ218+t43ae3w8diyRZySn0lKUuDXsK65nWfXVfL4W3toONzB+bN1IXoZHl6uEZwJlDrndgKY2WPAZcDmLtN8HviNc64OwDlX5WGeYxRkp7B6Zy21h9rITksero+VGNbc1sln73+LyvrDPPS5szijYOwxr49JTeIz50zj2rML2NfQSm7mqDAllVjj5WBxHrCny+OK4HNdnQScZGavm9nq4Kak45jZMjMrNrPi6urqkIQ7b2YOBxrbWPSjv/HFh9fwj+3V+PwuJO8tscc5x5ryOg61dfb4enunnxseWsPGygZ+ffWC40qgKzNjUtZobRaSYePlGkFPP8Xdv2kTgJnAEiAfeNXM5jnn6o+ZybnlwHKAoqKikHxbf3RhPvPzM3n8rT0883YFz23cz4IpWTx43VmkJcfkGLoMwe1/K+V/X9pO+qgErj5rCteeXUBu5mgONLby9NsVPFlcwa6aZn7y0flcNFcXmZeRxctvvApgcpfH+cDeHqZZ7ZzrAHaZ2TYCxfCWh7mOOmlCOt/+wFy+vnQWv3+7klufLWHZg8Xce+0ZjEqMPzpdWU0zt//tHT65eCqnTxkT0gx+v2N9RT1zcjOO+UyJHPe8upP/fWk775+fi8Nx98qd/PbVXZySn8m6PfX4HZw5bSxfv3gWl8zPDXdckeN4WQRvATPNbBpQCVwJdN8j6FngKuB+MxtHYFPRTg8z9Sg5IZ4rz5xCcmIcX318PTc/upY7PrGAhPg4/rCuklt/X8Khtk5e3HKAx5Yt4uRJmSH5XL/fccvTG3hyTQWjE+NZMiuHi0+eyJzcDDZWNrCm/CBryuuoOdROSlI8qUkJpCTHMyc3g6UnT2TR9GySEqLnUJDmtk4272tkU2UDB5raaGnr5FCbj9YOH/PyMvngqbnkj0kJd8xjPPbmbv7rz1u4ZN5EfnnlaSTEx7HnYAv3vV7Gqh01XP+eQj5eNJlp47QbqIxc5px328XN7FLgFwR2H73XOfcDM/s+UOycW2GBjaA/A5YCPuAHzrnH+nrPoqIiV1xc7FnmB1aVcduKTVx+eh4J8cYTxRUsnDqGb1wymy8/upa2Tj+PX7+YGePThvQ5XUvg2rML6PT7eX7TAaqDuxQCZIxKYMHUMeSPGU1Lm4/m9k6aWjtZt6eelnYfGaMSuHDOBBZPz+a0KVkU5qQRHzwdQXunn8r6wzS3dTJjfNqIXdvo8PlZvnInT78d2HRy5McxMd5ITU4gNSmBxHg7evDfGQVjuHR+LonxcTS2dtB4uBOf38/8/CzOKBhDbuboYcv+x/V7ufmxtZw3M4e7P1UUVaUs0cfM1jjninp8zcsi8ILXRQBw+9/e4ecvbscMblwyg6+8dyYJ8XHsqmnmY3e9QUKc8eQNi5k8dnB/nfr9jm88s4Eniiv48oUz+epFJx19fu2eenbVNHNqfiaFOWk9nmemtcPHq+/U8NeS/fxt6wHqWzoASEtOYMb4NGoOtbG3/jD+Ll+qc3IzOG1yFqfkZ3Ha5Eymj+v5vYfTln2NfO2p9ZRUNvIvM8ZxRsFY5uVlMC8vk/HpyccMlu6ubWHF+kqeXbeX0qpDR59Pio8DCxQfQF7WaArHp+H3Ozr9fjp9DjNIjI87ejt9ShaXnTZpSGsXGysa+OidqzhtchYPfPZMRieNzKIVOUJFMEDOOZ4o3sPU7FQWTT/2XC9b9jVy5fLVpMtVQj0AAAtNSURBVI9K4Pr3FHLx3AmMz+j/bn6dPj+3/r6Ex4v3cPMFM/jqRScNae8Qv9+xs6aZdXvqWbenjtKqQ0zIGMXUsSlMyU5ldGI8GysbWL+nng0V9TQHr9WclpzAvLwM5udlMndSBnNzM5kePIq15lAb1U1tNLV2Mjc3gzGpSYPOB4G/+tfvqafd5yfOjPg4Y1VpLb9++R0yRyfyXx+ex9J5/dt27pyjsv4wifFxZI5OJDkhDp/fsWVfE8XlBykuq6OiroX4OCMhPo6EOMM56PT7afc5Wto6eSdYJGdOG8tHTs/jstMmHd2Xvz8aWzv4wO2v0enz8+ebzx3y8hEZDiqCEFu3p55/f2IdO6qbMYMFU8Zw+YI8rj5zSp9f6lVNrdz0yFr+uesgN10wg38bYgkMlM/v2FF9KFgKDWyoqGfL/qajf00nxBmdPexCOyc3g8XTs1k0fSxFBWMZ288vvrKaZp4o3sOTayqO2eR1xAdPncT3PnRyv98vVPYcbOEP6yp5Zm0lO6ubGZOSyLVnT+PTZ08lK6XvLM45vvTIWv66aT9PXL+IhVN73w1UZCRREXjAOUdp1SH+WrKf50r2s2VfI9efN51vXDK7xy/3N3cd5MZH3qaptYMffmQ+ly/ID0Pq43X6/OysaWbz3ka2HWgiOSGOnPRkctKSSUlKYO3uOt7YWcua8jragoUxPSeVhVPGMCc3g+y0JLJSkhiTkkjD4Q5Kqw5RWnWILfsaeXt3PXEGF8wez+UL8hmbmoTf7/A5R9boJObnh2bQfbCO7Pt/1z928NKWKlKT4rliYT4n52UydWwKU7NTGZ+efMwmtN+tLufbz5Zwy9LZfGFJYRjTiwyMisBjzjm+/YcSHlq9my8sKeTrF886WgatHT5++9oufv7idqaMTeHOaxYwe2LkXWiktcPHxsoGisvqWFN+kOLyuqNjE91ljAqMVVwwezxXLJzMxAg4Qnbr/kb+7x87+dOGvXT43v2dSEtO4JT8TE6bnMWUsSl8Z8UmFk/P5r5rzwj7GIvIQKgIhoHf77j12RIefXM3N18wgy8smcHD/yznrn/spOZQG5fMm8hPrjiFjFGJ4Y4aEs456lo6qGtpp665nbqWjqOD1ePSkiL2qNgOn5+99Ycpr22h/GAL2/Y3sn5PA1v2NdLpd0zISOa5m8/VaUkk4qgIhonf7/jmMxt5vHgP6ckJNLV1cnZhNl++cCZnTdcFRiJZa4ePTXsbmZg5irys4dtFVSRU+ioCnUshhOLijB9dPp/RSfGU1zbzxfNn9HlOGYkcoxLjWTg1tEeVi4wUKoIQi4szvvuhk8MdQ0Sk33QopIhIjFMRiIjEOBWBiEiMUxGIiMQ4FYGISIxTEYiIxDgVgYhIjFMRiIjEuIg7xYSZVQP1QEOXpzO7PO7p/pF/xwE1g/zoru87kNd7er63vF0f9zTNYPOfKHtf0/SWr6fHfS178C7/YJd998da9v3PdqLXB7vsu94fSb+3A83e9f5IWfZZzrmcHt/JORdxN2B5b497ut/l3+JQfWZ/X+/p+d7y9pQ5FPlPlH0g+Qe77L3MP9hl38/MWvbDuOx7yj8Sfm8Hmn04fnaGsuy73yJ109Af+3jc0/3u04fiM/v7ek/P95a36+O+phmo/szf3/zRtOy7P9ayP3GG/r4+2GXf9f5Iyj/Q7P357BPxctkfI+I2DQ2FmRW7Xs6+FwmUP3wiOTtEdv5Izg6RkT9S1wgGa3m4AwyR8odPJGeHyM4fydkhAvLH1BqBiIgcL9bWCEREpBsVgYhIjFMRiIjEOBVBkJmda2Z3mdk9ZrYq3HkGyszizOwHZvYrM/t0uPMMhJktMbNXg8t/SbjzDIaZpZrZGjP7QLizDISZzQku96fM7AvhzjNQZvZhM7vbzP5gZu8Ld56BMrPpZvZbM3sqnDmiogjM7F4zqzKzkm7PLzWzbWZWambf6Os9nHOvOuduAP4EPOBl3u5CkR+4DMgDOoAKr7J2F6LsDjgEjGIYs0PI8gPcAjzhTcqehejnfkvw5/7jwLDu4hii/M865z4PXAv8q4dxjxOi/Dudc9d5m7QfBnvE3ki6AecBC4CSLs/FAzuA6UASsB6YC8wn8GXf9Ta+y3xPABmRlh/4BnB9cN6nIix7XHC+CcDDEbjs3wtcSeDL6AORlD04z4eAVcDVkbbsu8z3M2BBBOcftt/Znm5RcfF659xKMyvo9vSZQKlzbieAmT0GXOac+xHQ4+q7mU0BGpxzjR7GPU4o8ptZBdAefOjzLu2xQrXsg+qAZC9y9iZEy/58IJXAL/xhM3vOOef3NDihW/bOuRXACjP7M/CId4mP+9xQLHsDfgz8xTn3treJjxXin/2wiooi6EUesKfL4wrgrBPMcx1wn2eJBmag+Z8BfmVm5wIrvQzWDwPKbmaXAxcDWcCvvY3WLwPK75y7FcDMrgVqhqME+jDQZb8EuJxAAT/nabL+GejP/U0E1sgyzWyGc+4uL8P1w0CXfzbwA+B0M/tmsDCGXTQXgfXwXJ9HzznnbvMoy2AMKL9zroVAkY0EA83+DIEiGykG/LMD4Jy7P/RRBmygy/4V4BWvwgzCQPPfDtzuXZwBG2j+WuAG7+L0T1QMFveiApjc5XE+sDdMWQYjkvNHcnaI7PyRnB2UPyyiuQjeAmaa2TQzSyIwmLcizJkGIpLzR3J2iOz8kZwdlD88wjlSHcLR+0eBfby76+R1wecvBbYTGMW/Ndw5ozF/JGeP9PyRnF35R9ZNJ50TEYlx0bxpSERE+kFFICIS41QEIiIxTkUgIhLjVAQiIjFORSAiEuNUBBIVzOzQMH/ePWY2N0Tv5TOzdWZWYmZ/NLOsE0yfZWZfDMVni4AuXi9RwswOOefSQvh+Cc65zlC93wk+62h2M3sA2O6c+0Ef0xcAf3LOzRuOfBL9tEYgUcvMcszsaTN7K3g7J/j8mWa2yszWBv+dFXz+WjN70sz+CLxggSunvWKBq3dtNbOHg6c9Jvh8UfD+IQtcHW69ma02swnB5wuDj98ys+/3c63lDQJnsMTM0szsb2b2tpltNLPLgtP8GCgMrkX8NDjt14Kfs8HMvhfCxSgxQEUg0eyXwP86584APgrcE3x+K3Cec+504DvAD7vMsxj4tHPuguDj04GvELjWwHTgnB4+JxVY7Zw7lcApwD/f5fN/Gfz8E554zMzigQt599w0rcBHnHMLgPOBnwWL6BvADufcac65r1ngEo0zCZwL/zRgoZmdd6LPEzkimk9DLfJeYG7wj3iADDNLBzKBB8xsJoFTBCd2medF59zBLo/fdM5VAJjZOqAAeK3b57QTuOIUwBrgouD9xcCHg/cfAf6nl5yju7z3GuDF4PMG/DD4pe4nsKYwoYf53xe8rQ0+TiNQDOG+LoVECBWBRLM4YLFz7nDXJ83sV8DLzrmPBLe3v9Ll5eZu79HW5b6Pnn9nOty7g229TdOXw86508wsk0Ch3EjgHPufAHKAhc65DjMrI3Bd5+4M+JFz7v8G+LkigDYNSXR7AfjSkQdmdlrwbiZQGbx/rYefv5rAJikInI64T865BuBm4P+ZWSKBnFXBEjgfmBqctAlI7zLr88BnzezIgHOemY0P0X+DxAAVgUSLFDOr6HL7NwJfqkXBAdTNvHslqP8GfmRmrxO42LhXvgL8m5m9CeQCDSeawTm3lsAFz68EHiaQv5jA2sHW4DS1wOvB3U1/6px7gcCmpzfMbCPwFMcWhUiftPuoiEfMLIXAZh9nZlcCVznnLjvRfCLDTWMEIt5ZCPw6uKdPPfDZMOcR6ZHWCEREYpzGCEREYpyKQEQkxqkIRERinIpARCTGqQhERGKcikBEJMb9fwd06iuoq7lkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find(suggestions=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.LineCollection at 0x7f7e94dd6b50>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEMCAYAAADJQLEhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU9Z3/8dcndxJyIRcQAgl3FEFAIqhUBbVqXe9tvfSqtVWrttttf9267bbuutvWdmvXtq611KrVemmtulXXeqmKWBQlgModIUBIQkhISMj9MvP9/TEDBsiVzMlkZt7Px2MezpxzZuadMcw753zPxZxziIhI7IoLdwAREQkvFYGISIxTEYiIxDgVgYhIjFMRiIjEOBWBiEiM86wIzGyCmb1uZpvMbIOZ/WM3y5iZ/dLMtpnZB2Z2sld5RESkewkevnYn8C3n3BozSwdWm9krzrmNXZb5BDAteFsI/Dr4XxERGSKeFYFzbg+wJ3i/wcw2AflA1yK4FHjYBY5qW2lmWWY2NvjcbuXm5rqJEyd6FVtEJCqtXr16n3Mur7t5Xq4RHGJmE4F5wDtHzMoHdnd5XBac1mMRTJw4keLi4hAnFBGJbma2q6d5ng8Wm9lI4CngG865A0fO7uYpR53zwsxuMLNiMyuurq72IqaISMzytAjMLJFACTzqnHu6m0XKgAldHo8HKo5cyDm31DlX5Jwrysvrds1GRESOkZd7DRnwO2CTc+7nPSz2LPCF4N5DpwL1vY0PiIhI6Hk5RrAI+DywzszeC077LlAA4Jy7D3gBuBDYBjQD13mYR0REuuHlXkN/p/sxgK7LOOAWrzKIiEjfdGSxiEiMUxGIiITJ1r0NtHb4wh1DRSAiEg4vrt/Def+9nJsfXYPf3/eVIlds20djW6cnWVQEIiJDbENFPf/0x/fJS0/mtc1V/PqN7b0uX9XQypceWsWPX9jkSR4VgYjIEKpqaOUrvy8mKzWR//vax7h4zjjuenkLK7bt6/E5S98oocPn58tnTPYkk4pARGSItHb4uPGR1dQ2t/PbLxQxOiOFO6+YzeS8kXz98bVU1rce9Zzqhjb+8M4uLpubz6TcNE9yqQhERIbInX/dzNrSOn5+5Vxm5WcCkJacwH2fO5mWDh+3PLaGts7DB49/+2YJ7Z1+bj17qme5VAQiIkOgvdPPU2vKuHxePhfOHnvYvKmj0/npp05i9a793PTI6kNlsK+xjYff3smlc/OZnDfSs2wqAhGRIbCypIaG1s6jSuCgi04ax48un83rW6r56h8Cawa/Xe792gAM0WmoRURi3csbK0lNiueMabk9LvOZhQU4HN97Zj1feXg1q3bUcvGccUzxcG0AVAQiIp7z+x0vb9jLWdPzSEmM73XZzy4sBOB7z6zHDL529jTP86kIREQ89l5ZHVUNbZx/4nH9Wv6zCwvJSEmkrrmdqaO9XRsAFYGISJ/aOn0kJ/T+l3xvXtpQSUKcsWTG6H4/5+I54475/QZKg8UiIr1YX17PyXe8wlOry47p+c4FNgudNiWHzNTEEKcLDRWBiEgv7v7bVpraffz7cxuoajj6gK++bKtqZMe+Js7r52ahcFARiIj0YH15PX/bVMUnTx5Pa4efO57bOODXeGlDJQDnzRwT6nghoyIQEenBL179kIyUBG6/ZCa3nj2V5z/Yw2ub9w7oNV7asJd5BVmMyUjxKOXgqQhERLqxvryeVzbu5ctnTCYjJZGbzprCtNEj+ddn1tPUz9NBl9e1sK68nvNmDt/NQqAiEBHp1i+DawPXLpoIQFJCHHd+cjYV9a387OUt/XqNv7xXDsD5Jw7fzULgYRGY2QNmVmVm63uYn2lmz5nZ+2a2wcx04XoRGRY2VNTz8sa9XP+xwNrAQfMLs/n8qYU8uGInP31xc68XlGnt8PHA33dyxrRcT88TFApeHkfwEHAP8HAP828BNjrnLjazPGCLmT3qnGv3MJOISI8aWjt488N9/OaN7aR3WRvo6vsXzaTT7+feZdvZUtnA3VfPJT3l6N1C/7hqN/sa27hlybwhSD44nhWBc265mU3sbREg3cwMGAnUAt5ch01EpIuWdh/3v1lCTVM7Pr/D5xy7a5tZWVJDh8+RlZrIDy6aSeaIo7/gkxLi+NHlszlhbAb//txGLr/3Le7/QhETu1wroL3Tz2/e2E5R4SgWTsoeyh/tmITzyOJ7gGeBCiAduMo55+9uQTO7AbgBoKCgYMgCikh0unfZNn712jYyUhKIjzPi4+LITkvkS4smcc4JYzi5IIuE+J63nJsZXzhtIlPzRnLzY2u48jdv88cbTzt04Zj/XVtORX0rP7xiNoG/dYc3c67viyYf84sH1gied87N6mbep4BFwDeBKcArwBzn3IHeXrOoqMgVFxeHPqyIxIQ99S0s+dkyzpt5HL+8ZvCbbT7c28BVS1eSnBDHn248jXFZIzj352+QlhzPc7d+bNgUgZmtds4VdTcvnHsNXQc87QK2ATuA48OYR0RiwF0vb8Xvh2+fPyMkrzdtTDp/uH4hze0+rl66kvvfLGHHviZuWTx12JRAX8JZBKXAOQBmNgaYAZSEMY+IRLkNFfU8taaM6xZNZEJ2ashed+a4DP5w/UIOtHTw479uZurokf0+0+hw4OXuo48DbwMzzKzMzK43s5vM7KbgIv8BnG5m64BXge845/Z5lUdEYptzjh+9sInMEYncvCT0V/yaPT6T31+/gPysEfzz+TOIi4uMtQHwdq+ha/qYXwGc59X7i4h0tWxrNSu21XD7xd3vDRQKJxeMYsVtZ3vy2l7SkcUiEhN+9tIWJuakHroCmHxERSAiUa+1w8fGPQe4bF4+SQn62juSPhERiXpl+5txDgpzQjdAHE1UBCIS9XbVNANQmJPWx5KxSUUgIlFv58EiCOEuo9FERSAiUa+0pomRyQlkpyWFO8qwpCIQkai3q7aZguzUiDnSd6ipCEQk6pXWNGuguBcqAhGJaj6/Y/f+ZgpUBD1SEYhIVNtT30KHz1GYrT2GeqIiEJGoVnpo11GtEfRERSAiUW1XbaAICrTraI9UBCIS1XbVNJMYb4zLGhHuKMOWikBEolppbRPjR6USH0GnhR5qKgIRiWq7apq1WagPKgIRiVrOOR1D0A8qAhGJWvubO2ho69QaQR9UBCIStXbVNAEwUWcd7ZWKQESiVmmtjiHoDy8vXv+AmVWZ2fpelllsZu+Z2QYze8OrLCISmw5eh2CCNg31yss1goeAC3qaaWZZwL3AJc65E4FPe5hFRGLQrppmjstIISUxPtxRhjXPisA5txyo7WWRzwBPO+dKg8tXeZVFRGJTaW2TTjbXD+EcI5gOjDKzZWa22sy+0NOCZnaDmRWbWXF1dfUQRhSRSLarpllXJeuHcBZBAjAf+AfgfOD7Zja9uwWdc0udc0XOuaK8vLyhzCgiEaql3UdVQ5sGivshIYzvXQbsc841AU1mthyYA2wNYyYRiRIH9xgq0K6jfQrnGsFfgDPMLMHMUoGFwKYw5hGRKHLwGAJtGuqbZ2sEZvY4sBjINbMy4HYgEcA5d59zbpOZvQh8APiB+51zPe5qKiIyEDqGoP88KwLn3DX9WOa/gP/yKoOIxK6SfU1kpCSQlZoU7ijDno4sFpGo45zj7x/uY37hqHBHiQgqAhGJOtuqGimtbeacE8aEO0pEUBGISNT526bA8annnDA6zEkig4pARKLO3zbtZVZ+BmMzdXnK/lARiEhUqWlsY03pfs45XpuF+ktFICJR5fUt1TgH52p8oN9UBCISVV7dtJcxGcnMys8Id5SIoSIQkajR1ulj+dZqzj5+DGYW7jgRQ0UgIlFjZUktTe0+Pj5TewsNhIpARKLGq5v2kpIYx+lTcsMdJaKoCEQkKjjneHVTFR+bmqcrkg2QikBEosLmygbK61o4VweRDZiKQESiwtrSOgAWTdVmoYFSEYhIVNi6t4G0pHjys3Q08UCpCEQkKmypbGD6cenExWm30YFSEYhIxHPOsWVvAzPGpIc7SkRSEYhIxNvX2E5tUzvTVQTHREUgIhFvS2UDAMcfpyI4Fp4VgZk9YGZVZtbrdYjN7BQz85nZp7zKIiLRbcveQBFMVxEcEy/XCB4CLuhtATOLB34CvORhDhGJclsqD5A7MonckcnhjhKRPCsC59xyoLaPxb4GPAVUeZVDRKLflr2NGh8YhLCNEZhZPnA5cF+4MohI5PP7HR/ubVARDEI4B4vvBr7jnPP1taCZ3WBmxWZWXF1dPQTRRCRSlO1vobndp4HiQUgI43sXAU8EzxmeC1xoZp3Ouf89ckHn3FJgKUBRUZEb0pQiMqxpoHjwwlYEzrlJB++b2UPA892VgIhIb7ZUHgDQpqFB8KwIzOxxYDGQa2ZlwO1AIoBzTuMCIhISW/Y2Mn7UCEYmh3MDR2Tz7JNzzl0zgGWv9SqHiES3LZUHdGqJQdKRxSISsdo7/ZRUNzFD4wODoiIQkYi1Y18TnX6nIhgkFYGIRKzNwYFiFcHgqAhEJGJt3dtAQpwxOXdkuKNENBWBiESsLZUNTMpNIylBX2WDof2tRCRitHb42FzZwP6mdmqa2nm/rJ6Fk7LDHSviqQhEJGJ89+l1PL22/LBpp0/RxeoHS0UgIhHjw6pG5k7I4vaLZ5KTlkz2yCQdSBYC+gRFJGJU1LVw3onHMa9gVLijRBWNsIhIRGhp91HT1M74USPCHSXq9KsIzGyKmSUH7y82s6+bWZa30UREPlJR3wLAuKyUMCeJPv1dI3gK8JnZVOB3wCTgMc9SiYgcoaIuWASZWiMItf4Wgd8510ngimJ3O+f+CRjrXSwRkcMdKoIsFUGo9bcIOszsGuCLwPPBaYneRBIROVr5/hbiDI7L1KahUOtvEVwHnAb80Dm3w8wmAX/wLpaIyOHK61oZk5FCYrz2cQm1fu0+6pzbCHwdwMxGAenOuTu9DCYi0lVFXYs2C3mkv3sNLTOzDDPLBt4HHjSzn3sbTUTkIxX1KgKv9HcdK9M5dwC4AnjQOTcfONe7WCIiH/H7HXvqWslXEXiiv0WQYGZjgSv5aLBYRGRI7Gtso93nJ1/HEHiiv0VwB/ASsN05t8rMJgMf9vYEM3vAzKrMbH0P8z9rZh8Eb2+Z2ZyBRReRWFGuXUc91a8icM496Zw7yTn31eDjEufcJ/t42kPABb3M3wGc5Zw7CfgPYGl/sohI7KmoawUgX6eX8ER/B4vHm9kzwb/w95rZU2Y2vrfnOOeWA7W9zH/LObc/+HAl0OvriUjsKq9rBrRG4JX+bhp6EHgWGAfkA88Fp4XK9cBfe5ppZjeYWbGZFVdXVx/TGyxevJjFixcfYzwRCaeKulbSkxPISPH+ONZY/K7obxHkOecedM51Bm8PAXmhCGBmSwgUwXd6WsY5t9Q5V+ScK8rLC8nbikgEKdcxBJ7qbxHsM7PPmVl88PY5oGawb25mJwH3A5c65wb9eiISnSrqWjQ+4KH+FsGXCOw6WgnsAT5F4LQTx8zMCoCngc8757YO5rVEJLoFjirWrqNe6e8pJkqBS7pOM7NvAHf39BwzexxYDOSaWRlwO8ET1Tnn7gN+AOQA95oZQKdzrmjgP4KIRLPm9k72N3do05CHBnOpym/SSxE4567p7cnOuS8DXx7E+4tIDDh4+mkdVeydwZzGz0KWQkSkB+UHjyFQEXhmMEXgQpZCRKQHuiCN93rdNGRmDXT/hW+A/q+IiOfK97cQH2eMTk8Od5So1WsROOfShyqIiESnnfua+N3fdzB3QhaXzB034AvLVNS1cFxGCgm6II1nBjNYLCLSI7/f8dBbO/npS5vp8DkeWbmLn7+ylZvOmsyniyaQkhjfr9cpr2vR+IDHVLEiEnKlNc1cvXQldzy/kUVTclnxnbN54NoixmQk8/2/bOBjP3mdXy/bTkNrR5+vFbggjY4h8JLWCEQk5G59fA079jXxs0/P4ZMn52NmHJeZwpIZo3lnRy33LtvOT17czK+XbeOLp0/kukWTyE5LOup1fMEL0ow7SWsEXlIRiEhItbT7WF9ez61LpvKp+YefVNjMOHVyDqdOzuGDsjrufX07v3ptGw+t2MnXzpnKF0+fSHLCR5uMqhva6PQ77THkMW0aEpGQ2lx5AL+DE/Mze13upPFZ3Pf5+bz8T2dSNHEUP3phM+f993JeXF9JfXMHrR2+Q6ef1nmGvKU1AhEJqfUVBwCY1UcRHDR9TDoPXreAN7ZW85/Pb+SmP6w+ahkNFntLRSAiIbWhvJ5RqYmMyxzYAO9Z0/NY9I9n8OKGSvYeaKO1w0dbp5/05ASmjR7pUVoBFYGIhNj6inpm5WcSPJnkgCTEx3HRSeM8SCW90RiBiIRMe6efLZUNnDiuf5uFZHhQEYhIyGzd20CHzzG7n+MDMjyoCEQkZDZU1AMwKz8jzElkIFQEIhIy68rrSU9JoCA7NdxRZABUBCISMuvLD3DiuIxjGiiW8FERiEhIdPr8bNpzgFkaKI44nhWBmT1gZlVmtr6H+WZmvzSzbWb2gZmd7FUWEfHe9uom2jr9/T6QTIYPL9cIHgIu6GX+J4BpwdsNwK89zCIiHltfroHiSOVZETjnlgO1vSxyKfCwC1gJZJnZWK/yiIi31lfUMyIxnkm5Ogo40oRzjCAf2N3lcVlwmohEgL0HWnHuoyvZbig/wMxxGcTHaaA40oSzCLr7benu+siY2Q1mVmxmxdXV1R7HEpHeOOf41asfsvBHr3Lzo2toaO3A73dsqKhn1jhtFopE4SyCMmBCl8fjgYruFnTOLXXOFTnnivLy8oYknIgcze933PH8Ru56ZSunTBzFyxv3csk9K3hxQyVN7b4+Tz0tw1M4i+BZ4AvBvYdOBeqdc3vCmEdEetHh8/OtJ9/nwRU7+dKiSfzxhtN4/Cun0tTWyc2PrgHQrqMRyrOzj5rZ48BiINfMyoDbgUQA59x9wAvAhcA2oBm4zqssInLsnHOsLKnlF69uZWVJLd8+fwY3L56CmbFgUjb/9/Uz+MYf17KtqpFpYzRQHIk8KwLn3DV9zHfALV69v4gMTofPz3PvV3D/mzvYuOcA2WlJ/OSTs7nqlILDlstLT+bRL59Ke6efxHgdoxqJdD0CEenWvz27gUffKWXa6JHcecVsLpuXT0pifI/LJyWoBCKVikBEjrK7tpk/rtrNNQsK+NHls3TuoCinCheRo/z6je3EmfH1c6aqBGKAikBEDlNR18KTxbv5dNF4xmbqovGxQEUgIoe5743tANy8ZGqYk8hQURGIyCGV9a088e5uPjV/PPlZWhuIFSoCETnkN8u343OOmxdrbSCWqAhEBIA99S089k4pV8zLZ4IuNRlTVAQiQlunj5sfXUOcGbeerbWBWKPjCERinHOO2/+ygbWlddz72ZMpzEkLdyQZYlojEIlxj75TyhOrdnPLkilcOFvXhopFKgKRGPbujlr+7dkNLJmRxzc/PiPccSRMVAQiMWp9eT03/WE1E7JTufvqebqyWAxTEYjEoJUlNVy9dCUjEuN58NpTyByRGO5IEkYaLBaJMX/buJdbHlvDhOxUHrl+gU4jISoCkVjyZPFubnt6HbPGZfDgdQvITksKdyQZBlQEIjHA53f85MXNLF1ewsem5nLf5+czMln//CVAvwkiUa6htYOvP76W17dU84XTCvn+RTN1JTE5jIpAJIrta2zjmqUrKdnXxH9cNovPn1oY7kgyDHn6Z4GZXWBmW8xsm5nd1s38AjN73czWmtkHZnahl3lEYs2TxWV8WNXI769boBKQHnlWBGYWD/wP8AlgJnCNmc08YrF/Bf7knJsHXA3c61UekVj06qa9nDgug49Nyw13FBnGvFwjWABsc86VOOfagSeAS49YxgEZwfuZQIWHeURiSm1TO2tK93POCWPCHUWGOS+LIB/Y3eVxWXBaV/8GfM7MyoAXgK95mKdPjW2dPPZOKVsqG8IZQyQkXt9chd/BuSeMDncUGea8LILujld3Rzy+BnjIOTceuBB4xMyOymRmN5hZsZkVV1dXexA1oNPn57vPrGP5Vu/eQ2SovLp5L6PTk5k1LjPcUWSY87IIyoAJXR6P5+hNP9cDfwJwzr0NpABHbcx0zi11zhU554ry8vI8igtZqUlkpCSwq7bJs/cQGQrtnX6Wb93HOSeMJk7nEJI+eFkEq4BpZjbJzJIIDAY/e8QypcA5AGZ2AoEiCOuf44U5aeyqaQ5nBJFBe3dHLY1tnZxzvMYHpG+eFYFzrhO4FXgJ2ERg76ANZnaHmV0SXOxbwFfM7H3gceBa59yRm4+GVEFOKqW1KgKJbH/btJfkhDgWTdXeQtI3Tw8oc869QGAQuOu0H3S5vxFY5GWGgZqYk8qL6yvp8Pl19KVEJOccr27ey6KpuYxIig93HIkA+qY7QmF2Gj6/o6KuJdxRRI7Jh1WN7K5t4RztLST9pCI4QkFOKoDGCSRi/W3TXgCND0i/qQiOUHiwCDROIBHq1U1VzMrP4LjMlHBHkQihIjjCmPQUkhLiKK3RLqQSWeqa2/nHJ9ayetd+PjFLF6GX/tPZR48QF2cUZKdq05AMa845zD46PuC1zXu57al11Da180/nTueGMyeHMZ1EGhVBNwqztQupDF+vb6nixodXkxBvpCYlMCIpjt21LRx/XDoPXHsKs/J1JLEMjIqgGwU5qbxdUnPUX10iw8FL6ytJTojjqlMm0NTuo7m9k6uKJvCVMyeTnKDdRWXgVATdKMxOpbndR3VjG6PTNeAmw8vKkhoWTs7hXy868qzuIsdGg8XdKMxNA6BU4wQyzOypb2FnTTOnTs4OdxSJIiqCbhRm61gCGZ5WltQAcNqUnDAnkWiiIujG+FGpxJmOJZDhZ+X2WjJHJHLCcRl9LyzSTyqCbiQlxDE2c4SOJZBhZ+WOGhZMytappSWkVAQ9KMxJZac2DckwUlHXwq6aZk6drM1CEloqgh4U6nTUMsy8syMwPqCBYgk1FUEPCrLTqG1qp6G1I9xRRAB4e3uNxgfEEyqCHhTqLKQyzKwsqWWhxgfEAyqCHhQEdyHV5iEZDsrrWiit1fiAeENF0AOtEchw8k7JwfEBFYGEnk4x0YP0lERy0pIordUupOK99k4/y7dWs7OmiV01zeyqbWZ0ejJXnzKB+YWjeHt7DVmpiRx/XHq4o0oU8rQIzOwC4BdAPHC/c+7Obpa5Evg3wAHvO+c+42WmgSjI0emoxXudPj83PlLM61uqAUhPSaAgO5U1u/bz59VlTB8zkn2N7RofEM94VgRmFg/8D/BxoAxYZWbPBi9Yf3CZacC/AIucc/vNbFhdZLUwO5VVO/eHO4ZEMecc3//Lel7fUs33L5rJFfPyyUpNxMxoauvkufcrePzdUmqb2lk8Y1j985Ao4uUawQJgm3OuBMDMngAuBTZ2WeYrwP845/YDOOeqPMwzYAU5aTz7fgXldS3kZ40IdxyJQve8to3H393NrUumcv3HJh02Ly05gasXFHD1ggL21LcwRmfCFY94OVicD+zu8rgsOK2r6cB0M1thZiuDm5KOYmY3mFmxmRVXV1cfU5hly5axbNmyAT1n0ZQcHHDGT17j+odW8eqmvfj87pjeX2JPQ2sH7+2uw7nuf2eeWl3GXa9s5Yp5+XzrvOm9vtbYzBHaLDREjuW7ItJ5uUbQ3W/tkf8iEoBpwGJgPPCmmc1yztUd9iTnlgJLAYqKiobsm3jh5ByWf3sJT6wq5Y+rynh1czFzJmTx6JcXMjJZ4+zSs4bWDq757UrWlx9gVn4GN545hU/MOg4zY9mWKh5/dzevbd7Loqk53PnJk3QBJAkrL7/NyoAJXR6PByq6WWalc64D2GFmWwgUwyoPcw3IhOxUvn3+8Xzj3Ok8s7acf3l6HTc9spoHrj2FpISPVqje3VHLE++W8p1PHM+YjNCtwjvnWFdez5S8kaSpfCJCa4ePGx5ezaY9Ddy8eAovrq/ka4+vZUL2CDp9jj31reSOTOams6bw1cVTDvs9EgkHL79ZVgHTzGwSUA5cDRy5R9D/AtcAD5lZLoFNRSUeZjpmifFxXFk0AQO+/ecP+H9Pvs/dV80F4DfLS/jZy1vw+R0bKg7wpxtPIzM1MSTve9fLW7nn9W0kJ8Rx1vQ8Lpw9luPHpvP+7jpW7dxP8c5aGlo7SUtOIDUpnowRiZw6KZsLTxrLjDHpUfGXZku7j417DrC+vJ7KA600tXXS2NZJW6efBROzuXjOOLLTksIdEwCf3/GNJ97j7ZIa/vuqOVw+bzzfOm8Gr2zcy0Nv7SAlMZ7bLz6Rc04YTWK8CkCGB+tp+2VIXtzsQuBuAruPPuCc+6GZ3QEUO+eetcC31F3ABYAP+KFz7oneXrOoqMgVFxd7lrk/fr1sOz95cTOfO7WAirpWXttcxT+cNJZL5ozja4+t5aTxmTxy/UJGJA3u+rH3vPYhP3t5K5fNHUdWahIvrq+k8kDrofnZaUkUFY4iLz05+OXoY19jGx+U1eF3MDk3jfNOPI5TJo5iXsGoQ1+Wzjlqm9op299C/qgR5I5MHlROLzjn+FPxbh5csZMPqxoPjc0kxhtpyQmkJSVgBmX7W0iIMxbPyOPs48fQ4fNT39JBfUsHmSMSKZo4inkTRg36/0V/M3/3mfU8/m4p379o5lGDvyLhZGarnXNF3c7zsgi8MByKwDnHHc9v5MEVO0mKj+P7F53A504txMx4Yd0ebnlsDUtmjOY3n59/zH/13f9mCf/5f5u4Yl4+P/v0HOLiDL/fsXZ3HTv3NTFnQhZT8tK6/Yu/uqGNlzdW8sK6PawsqT30JToxJ5WRKQns2tdMQ1vnoeULslOZV5DFvAlZzC0YxQlj08N6EfSqA63c9vQ6XttcxZwJWZw5LZdZ+ZmcND6T4zJSDvuZN1ce4Jk15TyztpyqhrZD01OT4mnp8OFcoDxOHJdJxohEOn1+On0Ov3MkxBuJ8XEkxccxftQILpuXz9wJWce8FvXw2zv5wV82cPPiKfzzBccP9mMQCSkVgQf8fscjK3cxv3AUs/IzD5v36Du7+N4z6zlreh5fPL2QRVNzB/TFevAL5R9mj+UXV88lYRCbEFrafXxQVsfa3Zp6OaUAAAq8SURBVHWsLd1Pa4efSblpFOakMi5rBLtqmlhbWsea0v3sPRD4Ik2Kj+OEcRnMGZ/JrHGZzByXwfQx6ZjBvsY2qg600dTeyaz8TDJSBrcJrKS6kcr6VsyM+DhjV00TP3xhEy3tPm77xPF88bSJ/dpbxud3lO1vJi05gYyURJIS4qhv6WDNrv28u7OWNbv209bpJyHOSIg34szo9Ds6fH7aO/1sq2qkrdPP5Lw0PnnyeK4smkBeev/XlDZXHuCSe1Zw+pQcHrz2lKjYJCfRRUUQBr/7+w7u/ttWGlo7SU9O4NyZY7h58RSmjen5FAEdPj//+fxGfv/2Lj4+cwz3fvbkId2OvKe+hfdK63hvd6A4NpTX09TuAyAhzvA5R9dflziDk8ZncfqUHE6fksu8gqx+DWi3dvj46/o9PPZOabcH7M0Zn8ldV85l6uiRIfvZ+tLQ2sEL6/bw1Jpy3t1RS3JCHNcsKODGsyYzNrP3Y0ha2n1ccs/f2d/cwYvfOGNYbmoTURGESXunnxXb9/HCB3t4cUMlifFx/OnGU5k6+ugyqGpo5dZH1/LuzlpuOHMy/3z+jEGtCYSC3+/YWdPEhooDbK48QEJcHKMzkhmdnkJSQhzFO2tZsW0f75fV4/M74uOMWeMyOGViNpPy0shOTWJUWhKpSfHsqmnmw6pGtlU18Nb2GuqaO5iYk8pnFhZw0vgs/M7h90N8nFE0cVRYB1JLqhu5743tPL2mnDgzLps3jvmFoyjMSWNSbhqj05MP+4v/e8+s49F3Snn4Sws4c3pe2HKL9EZFMAyUVDdy5W9WEmfw5E2nUZiTBgTGG978cB/f/vP71Ld08NNPzeGSOePCnHZgGlo7WL1rP6t21rJqx37eK6ujvdN/1HJxBoU5aczOz+SqUyZw2uScYX2QVNn+Zu57YztPFpfR1uXnSU9JYO6ELOYVjCI1KZ47/7qZG86czHcvPCGMaUV6pyIYJrZUNnD10rdJTUrgyZtOo7S2mZ+/spV3d9RSmJPKrz87n5njIv/qUx0+P7VN7dQ2tbO/qZ3Gtk4mZKcyKTeNlMTwDUIfq06fnz31reysaWLnviY2VTawtrSOLZUH8DuYnZ/JU189XccDyLCmIhhG1pXV85nfrsQBjW2djE5P5pYlU7l6wYSw7qkjA9fY1smG8nqmjUkfNscxiPSktyLQoapDbPb4TB760inc8fwmLj5pLJ87tTAi/0oWGJmcwEJdKEaigIogDOYXZvOXWxaFO4aICKBLVYqIxDwVgYhIjFMRiIjEOBWBiEiMUxGIiMQ4FYGISIxTEYiIxDgVgYhIjIu4U0yYWTVQB9QfMSuzj2lHzj/4uLtlcoF9xxCvuwwDydjTtO4yhjt3XzmPfBzpubveV+6BzVfu8OY+OC3LOdf96XGdcxF3A5YOdNqR8w8+7m4ZApfSDEmuwebuJWNYc/eVM9pyd/czKLdyR0Lu/rxWpG4aeu4Yph05/7l+LDNQfT3/WHIf+Xi45O5uejTn7npfuQc2X7mPTahy9/laEbdpaCiYWbHr4Sx9w5lyDy3lHlrK7Z1IXSPw2tJwBzhGyj20lHtoKbdHtEYgIhLjtEYgIhLjVAQiIjFORSAiEuNUBANgZmeY2X1mdr+ZvRXuPP1lZnFm9kMz+5WZfTHceQbCzBab2ZvBz31xuPMMhJmlmdlqM7so3Fn6y8xOCH7Wfzazr4Y7T3+Z2WVm9lsz+4uZnRfuPP1lZpPN7Hdm9udw5oiZIjCzB8ysyszWHzH9AjPbYmbbzOy23l7DOfemc+4m4Hng917m7ZJv0LmBS4F8oAMo8yrrkUKU3QGNQApDlD1EuQG+A/zJm5RHC9Hv+Kbg7/iVwJDs8hii3P/rnPsKcC1wlYdxu+YLRe4S59z13ibth2M54i0Sb8CZwMnA+i7T4oHtwGQgCXgfmAnMJvBl3/U2usvz/gRkREpu4DbgxuBz/xxJnzkQF3zeGODRCMp9LnA1gS+miyIld/A5lwBvAZ+JpNzB590FnByBuYfs32V3t5i5eL1zbrmZTTxi8gJgm3OuBMDMngAudc79GOh2dd7MCoB659wBD+MeEorcZlYGtAcf+rxLe7hQfeZB+4FkL3IeKUSf+RIgjcCXQIuZveCc8w/33MHXeRZ41sz+D3jMu8SH3i8Un7cBdwJ/dc6t8TZxQIh/v8MqZoqgB/nA7i6Py4CFfTzneuBBzxL1z0BzPw38yszOAJZ7GawfBpTdzK4AzgeygHu8jdarAeV2zn0PwMyuBfZ5XQK9GOjnvRi4gkDpvuBpst4N9Hf8awTWwjLNbKpz7j4vw/VioJ93DvBDYJ6Z/UuwMIZcrBeBdTOt1yPsnHO3e5RlIAaU2znXTKDAhoOBZn+aQJGF24B/VwCccw+FPsqADPTzXgYs8yrMAAw09y+BX3oXp98GmrsGuMm7OP0TM4PFPSgDJnR5PB6oCFOWgYjU3BC52ZV7aCn3EIr1IlgFTDOzSWaWRGBw79kwZ+qPSM0NkZtduYeWcg+lcI5UD+UNeBzYw0e7UF4fnH4hsJXASP/3wp0zWnJHcnblVu5ozt3dTSedExGJcbG+aUhEJOapCEREYpyKQEQkxqkIRERinIpARCTGqQhERGKcikCigpk1DvH73W9mM0P0Wj4ze8/M1pvZc2aW1cfyWWZ2cyjeWwR08XqJEmbW6JwbGcLXS3DOdYbq9fp4r0PZzez3wFbn3A97WX4i8LxzbtZQ5JPopzUCiVpmlmdmT5nZquBtUXD6AjN7y8zWBv87Izj9WjN70syeA162wNXRllngal2bzezR4OmOCU4vCt5vtMAV4N43s5VmNiY4fUrw8Sozu6Ofay1vEziDJWY20sxeNbM1ZrbOzC4NLnMnMCW4FvFfwWW/HXyfD8zs30P4MUoMUBFINPsF8N/OuVOATwL3B6dvBs50zs0DfgD8qMtzTgO+6Jw7O/h4HvANAtcVmAws6uZ90oCVzrk5BE7z/ZUu7/+L4Pv3eeIxM4sHzuGjc9O0Apc7504GlgB3BYvoNmC7c26uc+7bFrg04zQC58KfC8w3szP7ej+Rg2L9NNQS3c4FZgb/iAfIMLN0IBP4vZlNI3CK4MQuz3nFOVfb5fG7zrkyADN7D5gI/P2I92kncMUpgNXAx4P3TwMuC95/DPhZDzlHdHnt1cArwekG/Cj4pe4nsKYwppvnnxe8rQ0+HkmgGMJ97QmJECoCiWZxwGnOuZauE83sV8DrzrnLg9vbl3WZ3XTEa7R1ue+j+38zHe6jwbaelulNi3NurpllEiiUWwicW/+zQB4w3znXYWY7CVy7+UgG/Ng595sBvq8IoE1DEt1eBm49+MDM5gbvZgLlwfvXevj+KwlskoLA6Yh75ZyrB74O/D8zSySQsypYAkuAwuCiDUB6l6e+BHzJzA4OOOeb2egQ/QwSA1QEEi1Szaysy+2bBL5Ui4IDqBv56EpQPwV+bGYrCFxs3CvfAL5pZu8CY4H6vp7gnFtL4ILnVwOPEshfTGDtYHNwmRpgRXB30/9yzr1MYNPT22a2DvgzhxeFSK+0+6iIR8wslcBmH2dmVwPXOOcu7et5IkNNYwQi3pkP3BPc06cO+FKY84h0S2sEIiIxTmMEIiIxTkUgIhLjVAQiIjFORSAiEuNUBCIiMU5FICIS4/4/Ks3XMJmmFJcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_lr_find()\n",
    "plt.vlines(6.30e-8, 0.6, 1.2)\n",
    "plt.vlines(0.039, 0.6, 1.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.480574</td>\n",
       "      <td>0.409219</td>\n",
       "      <td>0.890000</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.340219</td>\n",
       "      <td>0.364467</td>\n",
       "      <td>0.890000</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.285851</td>\n",
       "      <td>0.371014</td>\n",
       "      <td>0.915000</td>\n",
       "      <td>00:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(3, lr_max=slice(1e-5, 1e-4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('roberta-fasthugs-stg2-3e-5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1fn48c8zk32HLIQkQMIOYQkQFkUWFwStilZa0S6u9avV2lq1xV+rbbW2Lv222tblq9alrYqKtlWLgiIIsihh3yFAgBAgIYEkhOw5vz/uzWSSTMgEEiYZnvfrNa/MnHvunXNCeObMuWcRYwxKKaX8l8PXBVBKKdWxNNArpZSf00CvlFJ+TgO9Ukr5OQ30Sinl5wJ8XYCm4uLiTGpqqq+LoZRSXcqaNWuOGmPiPR3rdIE+NTWVrKwsXxdDKaW6FBHZ19Ix7bpRSik/p4FeKaX8nAZ6pZTyc52uj14ppdqqurqa3NxcKioqfF2UDhcSEkJKSgqBgYFen6OBXinV5eXm5hIZGUlqaioi4uvidBhjDIWFheTm5pKWlub1edp1o5Tq8ioqKoiNjfXrIA8gIsTGxrb5m4sGeqWUX/D3IF/vdOrpV4F+X2EZn2094utiKKVUp+JXgf6uN9dy29+zyC/x/xsySqnO4/jx4zz33HNtPu/yyy/n+PHjHVCixvwq0IcFWveWF2w57OOSKKXOJS0F+tra2lOeN3/+fGJiYjqqWC5+FehTuocCkFN40sclUUqdS+bMmcPu3bvJyMhg7NixXHjhhdxwww0MHz4cgKuvvpoxY8aQnp7Oiy++6DovNTWVo0ePkpOTw5AhQ/jBD35Aeno6l156KeXl5e1WPr8aXllTa22LePBYy7+giupa5m86RFFZFbdMTMPhODdu4Ch1rvjNh1vYmlfSrtccmhTFr65Mb/H4448/zubNm1m/fj1LlizhG9/4Bps3b3YNgXzllVfo3r075eXljB07lmuvvZbY2NhG19i1axdvvfUWL730Et/+9rd57733+O53v9su5ferQF9dWwdAXnHLgf7387fx+kpr7Z/BiVFcMCDurJRNKXXuGDduXKNx7n/+85/517/+BcCBAwfYtWtXs0CflpZGRkYGAGPGjCEnJ6fdyuOXgX5f4UmufX4FN52fypUjkxrl2XGktFl+pZT/OFXL+2wJDw93PV+yZAmfffYZK1euJCwsjKlTp3ocBx8cHOx67nQ627Xrxq/66Kvsrpvi8mrW7DvGj95a1yyPMQ3Pj5dXna2iKaX8WGRkJKWlpR6PFRcX061bN8LCwti+fTurVq06y6XzMtCLyAwR2SEi2SIyp4U83xaRrSKyRUTedEu/UUR22Y8b26vgntR4aKEfK2sczMuqalzPC09ooFdKnbnY2FgmTpzIsGHDeOCBBxodmzFjBjU1NYwYMYKHHnqICRMmnPXytdp1IyJO4FlgGpALrBaRD4wxW93yDAAeBCYaY46JSIKd3h34FZAJGGCNfe6x9q+K1RUzODGS7YcbPlm3HiphYv+GfvgjJZWu50c10Cul2smbb77pMT04OJiPP/7Y47H6fvi4uDg2b97sSr///vvbtWzetOjHAdnGmD3GmCpgLjCzSZ4fAM/WB3BjTL6dPh341BhTZB/7FJjRPkVvrqrW0CMqhNdvGcei+6YANLr7boyh+GQ1d0zpR8/oEPJLK/hiZ4HHbwJKKeUvvAn0ycABt9e5dpq7gcBAEVkuIqtEZEYbzkVEbheRLBHJKigo8L70TVTX1BHodDBlYDz94iNIjAph6yEr0OcdL6eorIqq2jpiwgLp1S2M99ce5MZXvua3/9122u+plFKdnTeB3tNAc9PkdQAwAJgKXA+8LCIxXp6LMeZFY0ymMSYzPt7j3rZeqa6tIyig4S3Tk6LYklfMsl0FnP/45/z8vY0AxIQGMjatmyvf2v0NPUknKhv68JVSyh94E+hzgV5ur1OAPA95/mOMqTbG7AV2YAV+b85tN9W1Vou+3tCkKHYXlPHfjYcA+Gyb1aMUHRrIjPSeJERaw5nyjltDnTYcOM6wXy3g8+26MJpSyn94E+hXAwNEJE1EgoDZwAdN8vwbuBBAROKwunL2AAuAS0Wkm4h0Ay610zpEda0hwNFQpfSkKGrrDP9ad7BRvuiwQIanRPP1Ly7h3ksGcvREJU9+sp2Zzy4H4PPt+SillL9oddSNMaZGRO7GCtBO4BVjzBYReQTIMsZ8QENA3wrUAg8YYwoBRORRrA8LgEeMMUUdURGAqiZdN0N7RgNQWVNHfGQwBaXWiJuY0CBXnp7RIQA8t2S3K+3YyeqOKqJSSp11Xo2jN8bMN8YMNMb0M8Y8Zqc9bAd5jOWnxpihxpjhxpi5bue+Yozpbz9e7ZhqWJp23fTqHkqg0wr8gxMjXekxYQ17LQ7u2ZA+Pq07AQ5hd/6JjiymUuocFxERAUBeXh6zZs3ymGfq1KlkZWW1y/v51czYmlrTKNCLCE570bK0uIYpyfV98wAjUmJY//A0HrtmGP+4dTzfyuzlavkrpVRHSkpKYt68eR3+Pn611k1VkxY9gMPedquvW6APaJInJiyI74zvA0B8RBBFJ6uoqa1rlk8ppTz5+c9/Tp8+ffjhD38IwK9//WtEhKVLl3Ls2DGqq6v57W9/y8yZjacg5eTkcMUVV7B582bKy8u5+eab2bp1K0OGDNFlij0xxljDK52NR3TWB/resWFeXScuMhhjoOhkFQmRIe1eTqVUB/t4Dhze1L7XTBwOlz3e4uHZs2fzk5/8xBXo33nnHT755BPuvfdeoqKiOHr0KBMmTOCqq65qcc/X559/nrCwMDZu3MjGjRsZPXp0uxXfbwJ9bZ3BGJq16Ot/p93Dre6agFbWn4+LsPIdLdVAr5TyzqhRo8jPzycvL4+CggK6detGz549uffee1m6dCkOh4ODBw9y5MgREhMTPV5j6dKl3HPPPQCMGDGCESNGtFv5/CbQV9srVwYGeO66iQkN5P0fnk9i1KmDtyvQn9B+eqW6pFO0vDvSrFmzmDdvHocPH2b27Nm88cYbFBQUsGbNGgIDA0lNTfW4PLG7llr7Z8pvOqGr7PVqmrbo4yKsoZSRIQGM7t2NpJjQU16nPr8GeqVUW8yePZu5c+cyb948Zs2aRXFxMQkJCQQGBrJ48WL27dt3yvMnT57MG2+8AcDmzZvZuHFju5XNj1r09YG+8SfiazePY8GWw8RGBHs6rZk4e0SOLmGslGqL9PR0SktLSU5OpmfPnnznO9/hyiuvJDMzk4yMDAYPHnzK8++8805uvvlmRowYQUZGBuPGjWu3svlNoI8JDWTRfVOIDQ9qlN6rexi3Terr9XUigwMICnBoi14p1WabNjXcBI6Li2PlypUe8504Yc3VSU1NdS1PHBoayty5cz3mP1N+E+gDnA76xUec8XVEhPiIYAo00Cul/ITf9NG3p7iIIHYdOcG+wjJfF0Uppc6YBnoP4iKC2XSwmClPLfF1UZRSXjKm2Qroful06qmB3oOpgxNczyuqa31YEqWUN0JCQigsLPT7YG+MobCwkJCQts3x8Zs++vZ0/dhefLghj6/3FrG/6CQDe0S2fpJSymdSUlLIzc3lTHao6ypCQkJISUlp0zka6D0IcDr4xeVDmPnscj7ckMd9lw7ydZGUUqcQGBhIWlqar4vRaWnXTQvSk6LoGxfOW18faD2zUkp1YhroWxDgdHDFyCSKyiqprfPvfj+llH/TQH8K8ZHB1BkoLNMx9UqprksD/SnE28sm6EYkSqmuTAP9KcRHaqBXSnV9GuhPoX7LwSMlp15aVCmlOjMN9KfQMzqEoAAHuwt0KQSlVNflVaAXkRkiskNEskVkjofjN4lIgYistx+3uR2rdUv/oD0L39ECnA76x0ew80ipr4uilFKnrdUJUyLiBJ4FpgG5wGoR+cAYs7VJ1reNMXd7uES5MSbjzIvqG4MTI/liZwHVHjYeV0qprsCbyDUOyDbG7DHGVAFzgZmtnOM3Lh/ek8KyKh777zY+3JDn6+IopVSbeRPokwH36aG5dlpT14rIRhGZJyK93NJDRCRLRFaJyNWe3kBEbrfzZHW2tSouHJzAxP6xvLYihx+9tY7SimpfF0kppdrEm0DvabfaplNFPwRSjTEjgM+A192O9TbGZAI3AE+LSL9mFzPmRWNMpjEmMz4+3suinx1Oh/DQFUNdr9fsOwZAfmkFH2zI8/vV8pRSXZ83gT4XcG+hpwCN+jCMMYXGmPrB5i8BY9yO5dk/9wBLgFFnUF6fGJwYxd9vGYdD4OVlewF49KNt3PPWOm55bTVb80p8XEKllGqZN4F+NTBARNJEJAiYDTQaPSMiPd1eXgVss9O7iUiw/TwOmAg0vYnbJUweGM/PZwzmy+yjZOeXstZu2S/eUcCc99tvt3allGpvrQZ6Y0wNcDewACuAv2OM2SIij4jIVXa2e0Rki4hsAO4BbrLThwBZdvpi4HEPo3W6jCtHJgFw7fMrOXi8nJ9OGwhAUVkVm3KLyTte7sviKaWUR9LZ+pgzMzNNVlaWr4vRoj8s2MFfF2dzXWYvHr92OM8t2c1TC3YAEBLoYPujl/m4hEqpc5GIrLHvhzajG4+00f3TB3H3Rf0JCXQC1vDL+kBfUV3ny6IppZRHOgPoNNQHeYC0uHCCAxp+jXW6dr1SqpPRQN8OrrL77gEOHDvpw5IopVRzGujbwW+vGcbD9lj79QeO+7g0SinVmAb6dhAc4OT75/UhLMjpmlCllFKdhQb6dhLgdDA2tTtLdxawcMthVuw+6usiKaUUoIG+XV08JIGcwpPc/o813PDSV+SX6oYlSinf00DfjtKTohq9Xr9f++uVUr6ngb4d9YkNb/R6V/4JH5VEKaUaaKBvR7HhQa7nSdEh7DisO1MppXxPZ8a2IxFh0oA4+sVHUFZZwwcb8sgvrWDzwWImDYjXHaqUUj6hgb6d/ePW8QCs23+Md9fk8n9f7OFvX+7l6owknp7d5VZoVkr5AW1idpCBPSIBWLG7EIB/r8+julbXwlFKnX0a6DtIeHAAyTGhbDvUsClJZ59MdaDoJAWlla1nVEp1KRroO9Cw5MbDLXce6bw3Z7PzS5n05GJmvbBCv3ko5Wc00HegaUMTXc/Dg5zsKSjzYWlObdsh60NoX+FJ/nfhTh+XRinVnjTQd6BLhiS4nqfGhbPnaOcN9CUV1YA16evjzYf417pcqmq0Za+UP9BA34FiwoLo1T2UESnRjE3tzvLso4367DuT4nIr0F/QP459hSe59+0N/OXzXT4ulVKqPejwyg72+X1TcYpQWlHDm1/v5+nPdvL8d8bgcIivi9ZIcXk1QQEOhiVHu9K+3lvkwxIppdqLtug7WKDTgcMhRIcFMj09kQVbjvDhxjxfF6uZkvJqokICGdOnmytta14Jh4p1w3OlujqvAr2IzBCRHSKSLSJzPBy/SUQKRGS9/bjN7diNIrLLftzYnoXvav7wrREEBzhYvD3f10VppqS8hujQAJJiQkmIDAagtLKG837/Oe9mHfBx6ZRSZ6LVQC8iTuBZ4DJgKHC9iAz1kPVtY0yG/XjZPrc78CtgPDAO+JWIdPNw7jkhOMDJ9PREVu3pfF0ixeXVRIcGArD0ZxeS9ctLXMc+2njIV8VSSrUDb1r044BsY8weY0wVMBeY6eX1pwOfGmOKjDHHgE+BGadXVP8wLDmKwyUVfLL5sK+L0khxeTVRdqAPCXQSFxHsOnb8ZJWviqWUagfeBPpkwP27e66d1tS1IrJRROaJSK+2nCsit4tIlohkFRQUeFn0rmlIT2sS1R3/XNOpAmhJRUOLvl79apwbcotZuKVzfTAppbznTaD3NDzENHn9IZBqjBkBfAa83oZzMca8aIzJNMZkxsfHe1GkriujVwwRwdZgpw87UZeIe9dNvc/vm8qqBy8mOSaUf6za56OSKaXOlDeBPhfo5fY6BWg0bMQYU2iMqV8k5SVgjLfnnmsiQwLZ9OtLSekWyhc7OsdN2bo6Q4mHQB8dFkhidAgzM5JYubvQNalKKdW1eBPoVwMDRCRNRIKA2cAH7hlEpKfby6uAbfbzBcClItLNvgl7qZ12ThMRpg3twaLt+ay0V7f0pRNVNdQZiAoJ9Hh88sB4auoMK7Ktsm4+WEx5Ve3ZLKJS6gy0GuiNMTXA3VgBehvwjjFmi4g8IiJX2dnuEZEtIrIBuAe4yT63CHgU68NiNfCInXbOe2D6ICKDA3h3je+HLpbYs2Kbtujrje7djfAgJ0t3FbDtUAlX/OVLpjy1mPwS3fxcqa7Aq5mxxpj5wPwmaQ+7PX8QeLCFc18BXjmDMvqlsKAALhycwNKdBRhjEPHdTNn65Q+iWgj0QQEOzusXx+fb8l3r3+SXVjLn/U28ctPYs1ZOpdTp0ZmxPjQyJYajJ6o4eqJjRt+crKrBmGb3vptpCPQtf+5/d0JvDpdUMG9Nrittf9HJMy+kUqrDaaD3ocGJ1i5UHbFOfUV1LUMfXsCTC3a0mrekvAZouesGYMrAhtFQf/jWSKYN7cH+opNUVGtfvVKdnQZ6HxpoB/odh9s/0B+zx+g/v2R3q3lb66MH6wbyTeenEh7kZGZGEt8Z35uqmjq+shc+K62opsaLDUuMMSzcclg3N1HqLNJA70NxEcF0Dw/i8Y+3c8S+sbl4Rz7T/7SUrXlntpxxfSvdG8VeBHqAh68YyrqHLyXQ6WB8WixBAQ6W7izgs61HGP7rhTz+8XYAyqtq+XjTIY/dRh9vPszt/1jDa8tzvK+MUuqMaKD3sYrqWqpq67j37fXsKyzj5ldXs+NI6RmvBe8+5v2Bdzecsq++uLwah0B40KnvzTscQlCA9ScTGuRkfFp3vthZwNc5Vqt+7uoDVFTX8rcv93DnG2v5YEPzKRP1H2CHinXEjlJniwZ6H7t4SA8AsnKOcevrWa70pTsLqKw5/f7v+u6Y9KQo3l2Ty9r9njcmn/H0Uv66OJuo0MA2r5E/ZWA82fkneHHpHgBOVNawOqeI3GPW0sb/Wnew2Tn1G68cPaGbkCt1tmig97Enrh3On64bSVVtHdn5J1zpZVW1fHUGq1zWt+ifuHYEQQGOFhdR227fH2it28aTqYMabtD2iAomyOngr59ns8O+ubwiu5ATlQ1dSLuOlLLIXqLZva5KqY6lgd7HwoICuDojmf4JEa60Z2ZnEGz3f5+u+j76ntEhDO0ZxYYDxafMH+Rs+59Cv/gILugfB1grXv5gchpf7S1i3f7j9E+IoKq2jic+3k52/glOVtUw7U9LXefuyi/VETtKnSUa6DsBEeGaUdaino9ePYyZGcn0jY9gd8Hpt3rru24iQwLJ6BXDpoPFzUbF1NY19NvvOo0Wtojwz9vG8+jMdF747hguH96wEsaM9ES6hwfxj1X7uP0fWazff9x1bNKAOKprjevmrVKqY2mg7yTunNKPrF9ewvcm9AEgLS6MnMLTm5BkjGHB1sOEBzkJCnAwIiWa8upaspt8cLh3q/z44gGnXfbvnZfKkJ5R9Itv+FbSIzqEF747hvSkKPYUlLHpoPWN4paJafzhWyMBeDfrgFcTupRSZ0YDfSfhcEijzT5SY8M5UHTytMabHz1RxeaDJVyVkQTAyF4xAGxs0n1T6urHH8690waebtFdQgKdrufxEcGMS+vOX64fBcAn9nr2N09MpUdUCL+7ZjhlVbUcKNI9aZXqaBroO6nhydHU1BmWZx9t87n1fd9j+nQHIC02nMjgANbnHm+Ur75FH9nCqpWno36zkvhI62daXDjxkcGss7tuIkOsIZzDkq0NWFbuaXv9lFJto4G+k7p4SA+iQwOZv6ntm5OctJcQDrVb2A6HMKJXNBubBPrSCivQ12+E0h7qv0XER4QAVj/++LTuruP17zUsKZr0pCj+ujibujrtvlGqI2mg76SCAhwM6Rl5WsMQy+0WfWhQwz/viJQYth9qPNKlvuumvpXdHn5x+RA++tEF9I4Nc6W5B/oAe3SPwyHcMaUfB4rKeWKB3pRVqiNpoO/E+sVHsLugrMUblqUt7PhUvymIe5/5yBSrK6h+wtKBopPszi8DTm8MfUsCnA6GJUc3ShvVu5vHvNPTExnaM4pXv8xxLX+slGp/Gug7sf4JERSXV1NQ2nwW6eqcIob/eiHLdjUfa1/fag91C/TDU6wbspsOFlNSUc2kJxfz2PxtJEQGkxob3kE1sLjPEXAXFODgrgv7U1Vbx/bDZ7a2j1KqZRroO7ERKVbLeP2Bxn3rB4pO8q0XVgKweHvzQN/QddMQ6JOiQ4gND2JTbnGjWbKXDUts89IHbeX+zaKpUb2tD6C3V/t+py2l/JUG+k4sPSmaIKeDNU3WqXFft6aorHlrv77rJiywoe9dRBieEs2mg8WuLQDT4sK568L+HVH0Zp6aNYL/tcfPu0uKCeWG8b15e/UB3XxcqQ6igb4TCwl0Miw5irX7Ggd69y6ZrYead3nUt+hDghr/845IjmbnkVL2Hj1JeJCTxfdPJSEqpANK3ty3Mntx7ZgUj8euzkimps4w8jcLz0pZlDrXaKDv5Eb37saG3OJGNyur7ElUM9IT2V1QRkV1baMbtuVVzfvoAS5NT6TOwHtrc+keEXQWSu+d0b1jCA5wYAyMe+wz/rvxEP9YmcOuDth5S6lzkVeBXkRmiMgOEckWkTmnyDdLRIyIZNqvU0WkXETW248X2qvg54oxfbpRVVPHlryGWa2V1VagH9U7hto6w8X/+wUXPLHYddzVom8S6IclRzNpgLUIWXBAy/3mZ1uA08GrN1ubjOeXVnLXm2t56D9buOyZZT4umVL+odVALyJO4FngMmAocL2IDPWQLxK4B/iqyaHdxpgM+3FHO5T5nDK6jzU0ca3bomD1Lfr6YYsHj5dz8Hi5a6ZreXUtgU4h0MOKlLdN6gvAvsKyDi13Ww1JjGqWVlNndMtBpdqBNy36cUC2MWaPMaYKmAvM9JDvUeBJQLcOakc9okJIjgllndsN2Eq7xd4vPrzRrNbVOUW8tnwvzy/Z3WKLvX6US3Vt55qN2i3cc1eSN3veKqVOzZtAnwy4j33LtdNcRGQU0MsY85GH89NEZJ2IfCEikzy9gYjcLiJZIpJVUHD6a7D7q6FJUa4NQqChRR8c6KRHVMNCaH9btpelu6y1Y26emOrxWlEhgdx7yUD+fsu4jitwOxGBL3fpWjhKnSlvAr2nQdau5qCIOIA/Afd5yHcI6G2MGQX8FHhTRJp9RzfGvGiMyTTGZMbHxze7yLluSGIkewpOuCZC1ffRBwc4+OO3M5g2tAfXj+vFl9lH2XywmMkD47nv0kEtXu/Hlwxg8sDO93u+6fzURq+vGZXMxoPHdYMSpc6QN4E+F+jl9joFcN/1ORIYBiwRkRxgAvCBiGQaYyqNMYUAxpg1wG7gzNfDPccMTYqmzsBrK3LIL62gqrYOEQhwCCN7xfDS9zO59YI0wLqZ2SMyuJUrdk6/unIoe39/Oc/eMJqpg+K5ZlQyFdV1LNjieRtEpZR3vFnNajUwQETSgIPAbOCG+oPGmGIgrv61iCwB7jfGZIlIPFBkjKkVkb7AAGBPO5b/nFC/KNjjH2/n671FDEiIIDjAgUjDl61+8RFEBgdQWllDj7M0Nr691dfnGyN68o0RPamrM/SMDmH+pkPMzEhu5WylVEtabdEbY2qAu4EFwDbgHWPMFhF5RESuauX0ycBGEdkAzAPuMMac/o7X56hu4UGujbhX5xRRXl3bbI9XEaGfvaaMe799V+ZwCFMHJbA8u1BH3yh1Brxan9YYMx+Y3yTt4RbyTnV7/h7w3hmUT9leu3kc89bkcv+7G1i5u5AgD6NqHpg+iEXb8hvt3drVTewfy1tf72drXolrpyylVNvozNguZHp6DxIig9mVf4LggOb/dBP7x/HwlUOJjfCPFj1YM4OBRsNLlVJto4G+C4kMCXSNlvEU6P1RUkwoiVEhjSaMKaXa5tyIFn5kaE9rdOq5tNLj6D4xjVbsVEq1jQb6LqZ+96ajJ6p8XJKzZ3TvbuQeK+fdrAO88MVu3Y1KqTbSQN/F1G9Gci65cqS14fgD8zby+Mfb+cPCHT4ukVJdiwb6LiYk0ElcRBCje587I1B6RIVw/6UN8+z+te4gtXWda60epTozDfRd0Ff/7xLeu/N8XxfjrLpzasNOWAWllazYrWvgKOUtDfRdkNMhjWbFngucDuH/vjeG9+48n8jgAD7ckNf6SUopwMsJU0p1BtPTEwGYNDCOL3cdxRhzzn3gKXU6tEWvupzz+saSV1zB/qKTvi6KUl2CBnrV5ZzXLxaAlbsLfVwSpboGDfSqy+kXH0F8ZDAr92igV8obGuhVlyMiZPbpxvoDDcsiLNmRz0cb9QatUp5ooFdd0sheMewrPElRmTVD+KZXV3P3m+swRsfXK9WUBnrVJU3sZ+118+ZX+xqlZ+ef8EVxlOrUdHil6pKGp0RzQf84XvhiD8uzG/rqV+0pZECPSB+WTKnOR1v0qsuaOiieE5U1jW7KLtt1lMoa3UxcKXca6FXHKNoDi38POcuhprJD3uL8fnHN0hZuPcL0Py1l3f5jvL82t0PeV6muRrtuVMc4uBa+eAK+eBwCQqH3eEibDKmTIWkUOM/8T29wYkMXjUPgd9cMZ877m8gpPMk1z60AYMfhUm4Y35vE6BBKymuIj/Sf3beU8pZ0tlEKmZmZJisry9fFUO2h/BjsWwF7l8HepZC/xUoPioQ+50PaJCv49xgOjtP7cvna8r04nQ6+O743IsJ7a3K5790NjfI4HcIVI3ryn/V5bH1kOmFB2r5R/kdE1hhjMj0e8ybQi8gM4BnACbxsjHm8hXyzgHeBscaYLDvtQeBWoBa4xxiz4FTvpYHej5UdhRw76O9dBoW7rPSQGEi9ANKmWME/fjCc5ho2NbV1TPj9ohY3ZnnuO6P9avN0peqdKtC32rQRESfwLDANyAVWi8gHxpitTfJFAvcAX7mlDQVmA+lAEvCZiAw0xujdsnNReBykX2M9AEryrICfsxT2LIXtH9n5EqyAn2q3+Lv39TrwBzgdzBiWyD9X7fd4/OPNhzXQq3OON9B0N4AAAB2bSURBVN9hxwHZxpg9ACIyF5gJbG2S71HgSeB+t7SZwFxjTCWwV0Sy7eutPNOCKz8QlQQjr7MeAMdyGrp59i6Fze/Z+VIaunlSJ0FMr1Ne9ooRSfxz1X5+cfkQbpuUxntrD3L/uxtwCHy+7QilFdVEhgR2bN2U6kS8CfTJwAG317nAePcMIjIK6GWM+UhE7m9y7qom5yafZlmVv+uWaj1Gfw+MgcJs2PuFFfR3LYQNb9n50qygXx/4I3s0usyEvrF89KMLGJQYiYgwa0wKV4zoydp9x7jh5a8Y/uuFvH7LOKYMjHeds/1wCdGhgfSMDj179VXqLPEm0Hv6zuzq2BcRB/An4Ka2nut2jduB2wF69+7tRZGU3xOBuAHWY+xtUFcH+Vsb+vi3/BvWvm7ljR/c0M2TegGEdXdtol4vJNDJhL6xXDkyiQ835PGr/2zmP3ddQHSY1bKf8fQyAHIe/8ZZraZSZ4M3gT4XcP+unAK4rx4VCQwDltibQCQCH4jIVV6cC4Ax5kXgRbBuxrah/Opc4XBA4jDrMeFOqKuFQxusoJ+zDNa/CatfAsTKkzbFCv59zoeQKPsSwl+uH8W1o5O59fUsfvPhFv54XUajt9lfeJLesWE+qKBSHafVUTciEgDsBC4GDgKrgRuMMVtayL8EuN8YkyUi6cCbWP3yScAiYMCpbsbqqBt1WmqqIG+t3cf/BRz4GmorQZzWuP36Pv5eEyAojF/9ZzNvfr2f5XMuIj4imLQH5wMwMyOJp6/L0J2rVJdzRqNujDE1InI3sABreOUrxpgtIvIIkGWM+eAU524RkXewbtzWAHfpiBvVIQKCoPcE6zHlAaiugNyvG27srvgLfPkncARCylh+0mMC2+vCeXdVH26ZMth1mf+sz+M74/swLq27DyujVPvSCVPq3FB5AvavsoZy7l1qdfuYOioJQnpP4OndiRxLmMA7h+K5++LB3DttoK9LrFSbnPGEqbNJA706K8qPs2D++xxct4ArorJJOJkNwEkJZYNjKOMuvBpn38mQOBwcTh8XVqnWnVHXjVJ+KTSGKVfdyDdzB/DIoRK6U8LrF1USc+QrEnYtxvnZQ1Y+16xdezjnGczaVcpXNNCrc1ZIoJPrx/fmoX9vpogoKgaeR8q0G7n2+RVUHcvj/ctrCdpvD+d0zdqNbxjK2cZZu0r5igZ6dU67fFgiD/17MwCRIQGICD+bMZjZLx7npeODuGvmbCtj/axd1zj+9630qOSGiVtpk1udtauUL2igV+e02IhgfjZjEIu25dOrmzV+fkLfWC4Z0oOnFuxgysB4a/KVx1m7S1uYtTupYRx/k1m7SvmC3oxVyoOC0krGPvYZ900byI8uHnDqzHV1ULCtIfDnLIfKYutY3CC7m8depC1Mh22qjqE3Y5Vqo/jIYAYnRvJl9tHWA73DAT3SrYf7rN36bp6ms3ZT7f59t1m7SnUkDfRKtWDa0B78dXE2+SUVJESFeH+iwwnJo63HxB9DbbW149bepdas3dUvw6pn7Vm7GQ19/L0nQFB4x1VInbO060apFmTnl3LJH5dyy8Q0Hr5yaPtd2DVr127xH8yCuhrXrF3Xcg0pYyFAtz5U3tEJU0qdpsufWcbWQyU8MH0QV45I6pgFzypPwIFVDX389qxdAkKg1/iGoZxJo8Cp6+grzzTQK3Wadhec4Hsvf0VecQUAe39/eccveFZ+3Nprt76P/4g1/JOgCKtfv34op87a9Q+1NVBTDtXl1oiu0xyppTdjlTpN/eIjuOfiAcx5fxNgBf7+CZEd+6ahMTD4cusB9l67XzYezgk6a7cj1dVBTYX1qD5pdbdVn7SCcX1Qdk+vqWghzT7nVGl11Q3vmzIWbvus3aujgV6pVszMSGbxjnwWbDnCFzuPdnygbyo8DtKvhvSrqaiuJaQ8327tf2H18zebtWuP4/e3WbvGWDe2mwVcT0HYLRi7gnDTtPqg7CGtpvz0yugIhMDQhkeA2/Ow7s3TAkMhMMzqpgsMtbbX7ADadaOUly7+3yUkdwvj77eM65DrHyur4tjJKvrGR3g8Pn/TIX74xlquzkjiiVkjCA6wu22O7Wvo5tm7FEoPWelRyW7LNUyCmA7ava2u9hQtWfvhMa1py9iLNFPX9vKJw0Nw9RRwTxWEwyAwpIW0sIZznb5rO2vXjVLtYPLAeN78ar/Vqg5s/77xZxbt4t2sA3z584uICg3kqr9+yU3np/KtTGtZhe2HSwH49/o8vthZwJL7L7S2QuzWx3qM+q49a3e31drPWQbZn8LGudYbdEttGMoZHNXGlvEp0mqrTq/C9a3YALegWh9MgxObpLm1epulhTUJ1k3SnEH+9c3mNGigV8pLUwbG8+ryHJbsKGDGsMR2v35RWRVlVbW8/OUerh/Xmy15JTwwb6Mr0FfV1BHoFGaP7c0/Vu3jky2HuG5sk1a6CMT1tx5jb3WbtVu/Rs9/YO3fWy6EI7BxwHUPwqHdrK6FZoG5LUHYLRg7HO3+O1SeaaBXykvn9Yulb1w4v/pgM1k5RYQHB3DD+N70aMtkqlMor7Y2X3t28W5C3b4xLNhymOnpiZRWVBMZEsgjM9NZnn2UP366k4sG98DpELqHB3m+aKNZu3dY3Sz526xWuKcg7MOuB9Vx9CNVKS8FBzh5ctYIjpRU8vKXe3lm0S4mPbGY2/+exfoDx8/4+hXVtcRFWBOk/rBwpyv9n6v2AXCisoaIYGuFzb/eMJpjZdVMf3opox/9lJ/MXUdJRbXH6zbicELiMEzSKEgYbHX5RCRAcKQGeT+mgV6pNshMbViU7JIhCVTV1rFw6xF+N39bi+dUVHu3TXJFdS0De0TwutvN3v+Z3Jfl2UcpKK2ktKKGyBArGA9NimJWZgpFZVb/+L/X53HnP9d49T5vr97PRf/7BZU1un3zuUIDvVJt9JfrR3HrBWn89YbRvH7LOIIDHGw7VEJZZU2zvIt35DP4oU9Yu/9Yq9ctr64lNNDJlIHxfPKTSTx9XQbfHJ1CnYEPN+TZXTcNre7bLkhj8sB4/nZjJv3iw1meXcjxk63fGN18sIS9R8tYuvNo2yquuiwN9Eq10ZUjk3joiqGE2EH5rdsnUFpRw2srcprl3ZpXAsAf3bpiWlJeVUtIkNU3PzgxiqtHJTMoMZIxfbrxyEdbWZ1zjIjghiUQ+sZH8PdbxnHxkB786boMwPpAaE2R/WHgTV7lH7wK9CIyQ0R2iEi2iMzxcPwOEdkkIutF5EsRGWqnp4pIuZ2+XkReaO8KKOVro3t345IhCTy1YAeLth1pdMxhD+v7Mvson221ju08Usqv/rPZ1e1Sr6K6rtFN2HqPzhxGSrdQAArLKj2WYXhyNCNTonltRQ51dQ1zY2pq66itazxX5pj9vp9uPcLJqubfQpT/aTXQi4gTeBa4DBgKXF8fyN28aYwZbozJAJ4E/uh2bLcxJsN+3NFeBVeqM3lg+mAAbn09y9WKByi1b5Amx4Tym4+2YIzhwfc38frKfbzwxe5G16jvumlqaFIU//rhRABSunleVE1EuGliKrsLyli0Pd+V/tSCHfT7f/PJzi91pRWVVREbHkR5dS2fbcv3dDnlZ7xp0Y8Dso0xe4wxVcBcYKZ7BmNMidvLcKBzTbdVqoMNSozklZusSYmvrdjrSi+pqKZbWCA/nTaQA0Xl/GHhDvYVngTg1eV7eTfrgOtmbXlVLaFBnidixUcGs/SBC/ndNcNaLMMVI5LoGxfO/e9u4LJnlrElr5hPthwG4PUV+1z5jp2sYuqgBHpEBWv3zTnCm0CfDBxwe51rpzUiIneJyG6sFv09bofSRGSdiHwhIpM8vYGI3C4iWSKSVVBQ0IbiK9V5XDS4B9eMSmbBliNU1VhT9UsraogKDeSKkT1Jjgnl2cW7OXqikpvOT6VX9zAemLeRi/6whKycIspbmXHbOzaMyJCWlykOdDp4YPogisur2XaohN98uJXjJ61vFO9kHWBj7nGMMRwrqyYuMogrRiTxxY4CisubD8usrTN8taeQ9loi5aWle1hof+ios8+bQO9p7nCzf31jzLPGmH7Az4Ff2smHgN7GmFHAT4E3RaTZ3mnGmBeNMZnGmMz4+HjvS69UJ3PFiJ4Ul1ezPNsa0VJSXk1USCDBAU7e/p8JJMdYfe39EyKYf88kvjehD3nFFcx6YSWAx66btpgxLJFEewLX13uLKC6v5vbJfYkKDeTRj7ZSUl5DVW0dseFBXDkyiaraOuZvOsS+wrJG11myI5/rXlzFkp3t0/B6bP42bv/Hmmb3C9TZ4U2gzwV6ub1OAU71fW8ucDWAMabSGFNoP18D7AYGnl5Rler8LhgQR3RoIO+usb4Eu499T+kWxm+vtrpehiVHExLo5M6p/Qhz664JdJ7ZmiwiwqL7pvDO/5yHCAQHOPh2Zi/uuXgAq3OO8X9LrfsCSTGhjEyJpk9sGA++v4kpTy1hv92lBLC/yHo+9+v9Z1QeoNGw06/3Fp3x9VTbeRPoVwMDRCRNRIKA2cAH7hlExH335G8Au+z0ePtmLiLSFxgA7GmPgivVGQUHOLlhfG8+2XyYfYVlVteNW3fLhYMT2PKb6WT0igGsgLv1kRk8OjMdgOjQM99BKjw4gHFp3fn03sl89KML6J8QwXWZvRjZK4bnljQEehHhqpENy+J+tKmh/Xa4xNpoZdG2fPJLK067LCeranh/ba7r9Vtf72+37iDlvVYDvTGmBrgbWABsA94xxmwRkUdE5Co7290iskVE1mN10dxop08GNorIBmAecIcxRj/SlV+7+fxUAhwOnvhkOwUnKuke0XgdmvDg5ksNfO+8VObfM4mZGc1uf522/gmRDOhhrZ0fFODgyWtHuI4lRVtdSLdekObq6nlj1X5qaq17C0eKKwgLclJTZ3h/7cHTLsMrX+7lof9sAWBsajc+2JDHkwt2sCm3mG2HSlo5W7UXrxa3MMbMB+Y3SXvY7fmPWzjvPeC9MymgUl1NQlQId1/Unz9+ak2Squ+Xb83QpGa3r9rVoMRInA6hts4QH2mtqRMTFsSKORexcOth7vjnWr7MPsrUQQkcLqlgaM8onA7h1eV7+f55fQgLavtaOEdPNMwV+MO3RjLlqSU8v2Q3z9vfLO66sB+BTgc/uUR7dDuSzoxVqgN8/7w+ruc9o9tndcv2sHLORbxx23icjoZ7AQ6HcOHgBGLCAvnL59k8/vF2Vu0pon9CBPdPH8SRkkr+74u297juKyzjtRU5JEWHsHzORfSJDeel7zfeF+PZxbt5+rNdZ9Q9pFqngV6pDhATFuQKpj2jvWvRnw0JUSFM7B/XLD04wMkjM4exZt8x10SuWy5IY2xqd64cmcRfPt/V5hU67/jnWgCOllW5vtVMG9qDhfdO5oO7JzJlYDw9oqxvFi8v29viddSZ00CvVAe571KrOyI1zvNs1s7mqpFJroD8h2+NZKDdv//7bw4nMiSQ55dkt3qNoycqycopwhjDjsNWH3zv7o3rP7BHJCNSYnjlprGsevBivjk6mddX5DQb4qnajwZ6pTrInVP6kfXLSzpVi741d0ztB0Bmn26utIjgAL43oQ8Ltx5h79GWg3FFdS3f+PMyZr2wkueW7KbOwPT0Hi3uset0CCLCvZcMJCjAwQ/+ntVonR7VfjTQK9VBRMS1kUhX8b0JfVj30DRS48IbpX///D4EOhw89t+trpE5TS3bdZQjJdaia08t2AHAzIxkklq5Gd2rexiPzExn55ETfNFOE7RUYxrolVKNdPOwLWFCZAi3TUrjs235vLX6gIez4Ou9hQQHOPjg7omuNG9vRF8xIome0SH89r9bycrREdjtTQO9UsorD0wfxODESN5pIdDnHa8guVsoI1JiXBO/WmvN1wt0OvjdNcMpq6zl5ldXu1b9VO1DA71SyisiwnVje7HpYDHffG4533xueaNZrgePl7tu5n5w90TmXDaYhEjvu64uHJzAS9/PpLSyhuG/XsjHmw5hjKG6to4dh0t54pPtHnfxUq3TQK+U8to3R6eQHBPK2v3HWbv/OAvcVqTMO17umnHbJzacO6b0Q6Rta/cMT4nmV1da2138bN5Gfv/xdkb+ZiE/nruO55fsZs77m9qvMucQDfRKKa9Fhwby4vfHEGDPEfjlv7dQXVtHRXUtBScq6Rlz5pPDbp6Yxod3X0BpZQ0vLt3Dyapath+2Nk75cEMem3KLz/g9zjUa6JVSbZKeFM3O317GS9/P5OiJSpbuLGD74VKMgcGJke3yHsNTorlv2kD6xIa5FoAbnBhJZHBAs525VOvavniFUuqc53AIUwfFExsexHtrcxF724phydHt9h4/ungAP7p4AIeLK5jw+0VcMaInJypreXHpbnKOljUbAqpapi16pdRpCXQ6mDUmhfmbDvPfTYeIiwj2egG3tkiMDmH9w9O4c2p/bplorQz64jJd7bwtNNArpU7bPRc3bEXx3p3ntfnmq7fq1w5KiArh2jHJzFuTqwuhtYEGeqXUaQsPDuDp6zL45qhk+sSena6UH0zqS3VtHS8t1Va9tzTQK6XOyNWjkvnjdRln7f36xkcwa3QKryzPYdG2I2ftfbsyDfRKqS7noSuHMrBHJL/412bKq2p9XZxOTwO9UqrLiQoJ5NdXDuVwSQWvLN/L1rySRpubq8Y00CuluqTxfWOZNrQHTy3YweV/XsbkpxZTWdNy6z7naBkvL9vDsbKqFvP4Kx1Hr5Tqsh6dOYzdBSfYU2Ctk//Exzv42YxBhAQ6XXn+smgX89bmss9u8e89WsZj1wz3SXl9xasWvYjMEJEdIpItInM8HL9DRDaJyHoR+VJEhrode9A+b4eITG/Pwiulzm2J0SHMv2cS7//wfK7OSOKV5XuZ+Pjn7Ck44crz+socV5AHeOOr/Xy4Ic8HpfWdVgO9iDiBZ4HLgKHA9e6B3PamMWa4MSYDeBL4o33uUGA2kA7MAJ6zr6eUUu0iJNDJ6N7dePzaETz+zeGUV9fy+MfbAViz7xg1dYbLhiVydUYS8++ZxKjeMfzorXUs23XubHLiTYt+HJBtjNljjKkC5gIz3TMYY0rcXoYD9WuXzgTmGmMqjTF7gWz7ekop1a5CAp3MHtebH07tx8KtR3h52R6ufX4Fx09Wk9Erhqdnj2JoUhS/vjIdgO/97Ws25rZtw/OuyptAnwy47zSQa6c1IiJ3ichurBb9PW0893YRyRKRrIKCc+dTVinV/m69oC89ooJ54pPtrrThKQ1r8IzsFcOrN48lOMDBza+uPid2tPIm0Hua09xsB19jzLPGmH7Az4FftvHcF40xmcaYzPj4eC+KpJRSnoUGObn3koFU1xqCAxysf3ga5/eLa5TnwkEJfPzjSUSFBnLza6vZdqikhav5B28CfS7Qy+11CnCqOxlzgatP81yllDpjs8akMLBHBIN7RhET1nwPXLBm2P7ztvGEBwVw2TPLeOTDrY12zPIn3gyvXA0MEJE04CDWzdUb3DOIyABjzC775TeA+ucfAG+KyB+BJGAA8HV7FFwppVoS4HTwxm0TqKmrO2W+5JhQ5t4+gcfmb+OV5XtxCNw6KY2e0e2/CqcvtdqiN8bUAHcDC4BtwDvGmC0i8oiIXGVnu1tEtojIeuCnwI32uVuAd4CtwCfAXcYYna+slOpw8ZHBXgXs1Lhwnr1hNOPSuvPyl3u54InFLNmRfxZKePZIZ/uqkpmZabKysnxdDKXUOWjhlsPc/o81hAc5+fBHF9A3PqJdr3/8ZBVPLdjB5IHxTE9PbNdri8gaY0ymp2O6BIJSStkuTU9kxZyLCAxwcO/b66mt89wQXrargPN+v4i3vt7v9bX3FZYx4feLeOOr/dz3zoazOrRTl0BQSik3STGh/OaqdH48dz3vr83lW5m9Gh3fdaSU7/3NutX44PubOFFRw3XjehHgEJwOITjA85zQm15dTUW1dc8gKiSAq/66nOjQQB6+Yii5x8opKqvk11eld8jmLRrolVKqiatGJvHK8hx++99tBDiFhMgQQoOsGbg7j1jLKzz3ndH88I21PDZ/G4/N3wZAQmQwL34/kxHJ0TgcDQG7sqaWnMIyhvaM4s/XZxAXEczrK/bx8eZD3D9vA/U96OP7xnL58J7tXh/tulFKqSZEhGeuyyDQ6eDetzfwnZe/4pvPreDf6w7y3tpcACYPjOft2ydw+fBEZo/txbczUygqq+LqZ5fzzKJdja538Fg5xsCtF6TRPyGSmLAgfnzJAObePoHe3cMAGJvard377V310ZuxSinlWUV1Lc8s2sV7a3LJL61sdCzn8W80y78lr5hrnl0BwEs3ZpIcE8rK3Ud58pMdlFbW8M7/nMe4tO6Nzik8Uckzi3Zx36WDiA4NPO2ynupmrAZ6pZTywoYDx1m8I5+nP9vFmD7deO/O8z3mO36yilkvrCQ7/0SzY1/9v4vpERXSIeXTQK+UUu2kvKqWOmMID275Fmd+aQXPfp7NoeIKZo/rxdjU7qw/cJxJAzpuiRcN9Eop5ed0HL1SSp3DNNArpZSf00CvlFJ+TgO9Ukr5OQ30Sinl5zTQK6WUn9NAr5RSfk4DvVJK+blON2FKRAqAfW08LQ442gHF8QWtS+flT/XRunROZ1KXPsYYj1NvO12gPx0iktXSjLCuRuvSeflTfbQunVNH1UW7bpRSys9poFdKKT/nL4H+RV8XoB1pXTovf6qP1qVz6pC6+EUfvVJKqZb5S4teKaVUCzTQK6WUn+vygV5EZojIDhHJFpE5vi5Pa0TkFRHJF5HNbmndReRTEdll/+xmp4uI/Nmu20YRGe27kjcnIr1EZLGIbBORLSLyYzu9y9VHREJE5GsR2WDX5Td2epqIfGXX5W0RCbLTg+3X2fbxVF+W3xMRcYrIOhH5yH7dJesiIjkisklE1otIlp3W5f7GAEQkRkTmich2+//NeWejLl060IuIE3gWuAwYClwvIkN9W6pWvQbMaJI2B1hkjBkALLJfg1WvAfbjduD5s1RGb9UA9xljhgATgLvs339XrE8lcJExZiSQAcwQkQnAE8Cf7LocA261898KHDPG9Af+ZOfrbH4MbHN73ZXrcqExJsNtjHlX/BsDeAb4xBgzGBiJ9e/T8XUxxnTZB3AesMDt9YPAg74ulxflTgU2u73eAfS0n/cEdtjP/w+43lO+zvgA/gNM6+r1AcKAtcB4rFmKAU3/3oAFwHn28wA7n/i67G51SLGDxkXAR4B04brkAHFN0rrc3xgQBext+rs9G3Xp0i16IBk44PY6107ranoYYw4B2D8T7PQuUz/76/4o4Cu6aH3sro71QD7wKbAbOG6MqbGzuJfXVRf7eDEQe3ZLfEpPAz8D6uzXsXTduhhgoYisEZHb7bSu+DfWFygAXrW71F4WkXDOQl26eqAXD2n+NF60S9RPRCKA94CfGGNKTpXVQ1qnqY8xptYYk4HVGh4HDPGUzf7ZaesiIlcA+caYNe7JHrJ2+rrYJhpjRmN1ZdwlIpNPkbcz1yUAGA08b4wZBZTR0E3jSbvVpasH+lygl9vrFCDPR2U5E0dEpCeA/TPfTu/09RORQKwg/4Yx5n07ucvWB8AYcxxYgnXfIUZEAuxD7uV11cU+Hg0Und2StmgicJWI5ABzsbpvnqZr1gVjTJ79Mx/4F9aHcFf8G8sFco0xX9mv52EF/g6vS1cP9KuBAfZogiBgNvCBj8t0Oj4AbrSf34jV112f/n377vsEoLj+K15nICIC/A3YZoz5o9uhLlcfEYkXkRj7eShwCdaNssXALDtb07rU13EW8LmxO1J9zRjzoDEmxRiTivV/4nNjzHfognURkXARiax/DlwKbKYL/o0ZYw4DB0RkkJ10MbCVs1EXX9+gaIcbHJcDO7H6U3/h6/J4Ud63gENANdYn9q1Y/aGLgF32z+52XsEaVbQb2ARk+rr8TepyAdZXyY3AevtxeVesDzACWGfXZTPwsJ3eF/gayAbeBYLt9BD7dbZ9vK+v69BCvaYCH3XVuthl3mA/ttT/H++Kf2N2+TKALPvv7N9At7NRF10CQSml/FxX77pRSinVCg30Sinl5zTQK6WUn9NAr5RSfk4DvVJK+TkN9Eop5ec00CullJ/7/5/8b+B/m/hIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lets Look at the model's predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "('positive', tensor(1), tensor([0.0587, 0.9413]))"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"This was a really good movie, i loved it\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from fastai2.interpret import *\n",
    "#interp = Interpretation.from_learner(learn)\n",
    "interp = ClassificationInterpretation.from_learner(learn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>target</th>\n",
       "      <th>predicted</th>\n",
       "      <th>probability</th>\n",
       "      <th>loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;s&gt; ĠIn Ġ17 th ĠCentury ĠJapan , Ġthere Ġlived Ġa Ġsamurai Ġwho Ġwould Ġset Ġthe Ġstandard Ġfor Ġthe Ġages . ĠHis Ġname Ġwas ĠMay eda . ĠHe Ġis Ġsent Ġon Ġan Ġepic Ġjourney Ġacross Ġthe Ġworld Ġto Ġacquire Ġ5 , 000 Ġmus cats Ġfrom Ġthe ĠKing Ġof ĠSpain . ĠWhilst Ġat Ġsea Ġa Ġviolent Ġstorm Ġswall ows Ġtheir Ġprecious Ġgold Ġintended Ġto Ġbuy Ġthe Ġweapons Ġand Ġalmost Ġtakes Ġtheir Ġlives . ĠMay eda Ġmust Ġbattle Ġall Ġodds Ġto Ġsurvive Ġand Ġthe Ġsecure Ġthe Ġfate Ġof Ġhis Ġbeloved ĠJapan . ĠShogun ĠMay eda Ġis Ġa Ġmulti Ġmillion Ġdollar Ġaction Ġadventure Ġepic Ġset Ġacross Ġthree Ġcontinents .&lt; br Ġ/ &gt;&lt; br Ġ/&gt; Star ring Ġcinema Ġlegends ĠSho ĠKos ugi Ġ( T ench u : ĠStealth ĠAssassins ), ĠChristopher ĠLee Ġ( Star ĠWars , ĠLord Ġof Ġthe ĠRings ĠTrilogy ), ĠJohn ĠRh ys ĠDavies Ġ( Lord Ġof Ġthe ĠRings ĠTrilogy , ĠIndiana ĠJones</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.9492946267127991</td>\n",
       "      <td>2.8352389335632324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;s&gt; ĠIn Ġorder Ġto Ġhold Ġthe Ġpublic 's Ġattention Ġfor Ġthree Ġhours , Ġwe Ġwere Ġtreated Ġnot Ġso Ġmuch Ġto Ġa Ġfamily 's Ġr omp Ġthrough Ġfour Ġgenerations Ġand Ġ120 Ġyears Ġof ĠHungarian Ġhistory , Ġas Ġto Ġsexual Ġlia isons Ġwith Ġa Ġsister , Ġa Ġsister - in - law Ġand Ġother Ġadul ter ies . ĠOh Ġyes , Ġthere Ġwas Ġalso Ġa Ġtotally Ġgrat uitous Ġrape . ĠHaving Ġsaid Ġall Ġthis , Ġthe Ġfirst Ġstory Ġof Ġthe Ġrelationship Ġamong Ġthe Ġchildren Ġof Ġthe Ġpatriarch Ġwas Ġfresh Ġand Ġsens ual Ġ- Ġthanks Ġto ĠJennifer ĠEh le . &lt;/s&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt; &lt;pad&gt;</td>\n",
       "      <td>negative</td>\n",
       "      <td>positive</td>\n",
       "      <td>0.9127477407455444</td>\n",
       "      <td>2.3215689659118652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;s&gt; ĠThis Ġmovie Ġis Ġhorrible - Ġin Ġa Ġ' so Ġbad Ġit 's Ġgood ' Ġkind Ġof Ġway .&lt; br Ġ/ &gt;&lt; br Ġ/&gt; The Ġstoryline Ġis Ġre h ashed Ġfrom Ġso Ġmany Ġother Ġfilms Ġof Ġthis Ġkind , Ġthat ĠI 'm Ġnot Ġgoing Ġto Ġeven Ġbother Ġdescribing Ġit . ĠIt 's Ġa Ġsword / s or cery Ġpicture , Ġhas Ġa Ġkid Ġhoping Ġto Ġrealize Ġhow Ġimportant Ġhe Ġis Ġin Ġthis Ġworld , Ġhas Ġa Ġ\" nom adic \" Ġadventurer , Ġan Ġevil Ġaide / s orce rer , Ġa Ġprincess , Ġa Ġhairy Ġcreature .... you Ġget Ġthe Ġpoint .&lt; br Ġ/ &gt;&lt; br Ġ/&gt; The Ġfirst Ġtime ĠI Ġcaught Ġthis Ġmovie Ġwas Ġduring Ġa Ġvery Ġharsh Ġwinter . ĠI Ġdon 't Ġknow Ġwhy ĠI Ġdecided Ġto Ġcontinue Ġwatching Ġit Ġfor Ġan Ġextra Ġfive Ġminutes Ġbefore Ġturning Ġthe Ġchannel , Ġbut Ġwhen ĠI Ġcaught Ġsite Ġof ĠGulf ax</td>\n",
       "      <td>positive</td>\n",
       "      <td>negative</td>\n",
       "      <td>0.9033368229866028</td>\n",
       "      <td>2.2247798442840576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interp.plot_top_losses(3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}