solutions/win2022/A3/pytorch_autograd_and_nn.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"DDJwQPZcupab"},"source":["# EECS 498-007/598-005 Assignment 3-3: Pytorch Autograd and NN (from fall2020)\n","\n","Before we start, please put your name and UMID in following format\n","\n",": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"2KMxqLt1h2kx"},"source":["**Your Answer:**   \n","Linxiao WU, #88068092"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"aQW_w1Wzw72f","tags":["pdf-title"]},"source":["# torch.autograd and torch.nn\n","\n","So far, we used PyTorch to accelarate computation using GPU.\n","PyTorch also provides several useful packages, which help to design deep neural networks efficiently.\n","\n","The `torch.autograd` package provides classes and functions implementing automatic differentiation of arbitrary scalar valued functions.\n","\n","This notebook assumes that you are using **PyTorch version 1.3**.\n","In some of the previous versions (e.g. before 0.4), Tensors had to be wrapped in `torch.autograd.Variable` objects to enable autograd;\n","however this class has now been deprecated and merged with `torch.Tensor`.\n","In addition 1.0 also separates a Tensor's datatype from its device, and uses numpy-style factories for constructing Tensors rather than directly invoking Tensor constructors.\n","Now, to obtain gradients for a tensor via autograd from arbitrary scalar valued functions, you can simply set `requires_grad=True`.\n","\n","The `torch.nn` package defines a set of Modules, which you can think of as a neural network layer that has produces output from input and may have some trainable weights.\n","\n","You can also find the detailed [API doc](http://pytorch.org/docs/stable/index.html) here.\n","If you have other questions that are not addressed by the API docs, the [PyTorch forum](https://discuss.pytorch.org/) is a much better place to ask than StackOverflow.\n"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"ED0jpoGyIL_B","tags":["pdf-ignore"]},"source":["# Table of Contents\n","\n","This assignment has 5 parts. You will learn PyTorch on **three different levels of abstraction**, which will help you understand it better.\n","\n","1. Part I, Preparation: As we always do, we will use CIFAR-10 dataset.\n","2. Part II, Barebones PyTorch: **Abstraction level 1**, we will work directly with the lowest-level PyTorch Tensors with autograd.\n","3. Part III, PyTorch Module API: **Abstraction level 2**, we will use `nn.Module` to define an arbitrary neural network architecture. \n","4. Part IV, PyTorch Sequential API: **Abstraction level 3**, we will use `nn.Sequential` to define a fully-connected and convolutional network very conveniently. \n","5. Part V, Residual Network: please implement your own ResNet to get a high accuracy on CIFAR-10.\n","\n","Here is a table of comparison:\n","\n","| API             | Flexibility | Convenience |\n","|-----------------|-------------|-------------|\n","| Barebone        | High        | Low         |\n","| `nn.Module`     | High        | Medium      |\n","| `nn.Sequential` | Low         | High        |"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"7LCmGZ_3IL_V"},"source":["# Part I. Preparation"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"ubB_0e-UAOVK"},"source":["## Install starter code\n","We will continue using the utility functions that we've used for previous assignments: [`coutils` package](https://github.com/deepvision-class/starter-code). Run this cell to download and install it.\n"]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":403},"colab_type":"code","executionInfo":{"elapsed":5992,"status":"ok","timestamp":1573619035590,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"ASkY27ZtA7Is","outputId":"a84301a1-f030-4d74-c1d4-658137d9f191"},"outputs":[{"name":"stdout","output_type":"stream","text":["Collecting git+https://github.com/deepvision-class/starter-code\n","  Cloning https://github.com/deepvision-class/starter-code to /tmp/pip-req-build-rpzxxhn7\n","  Running command git clone -q https://github.com/deepvision-class/starter-code /tmp/pip-req-build-rpzxxhn7\n","Requirement already satisfied: pydrive in /usr/local/lib/python3.6/dist-packages (from Colab-Utils==0.1.dev0) (1.3.1)\n","Requirement already satisfied: PyYAML>=3.0 in /usr/local/lib/python3.6/dist-packages (from pydrive->Colab-Utils==0.1.dev0) (3.13)\n","Requirement already satisfied: google-api-python-client>=1.2 in /usr/local/lib/python3.6/dist-packages (from pydrive->Colab-Utils==0.1.dev0) (1.7.11)\n","Requirement already satisfied: oauth2client>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pydrive->Colab-Utils==0.1.dev0) (4.1.3)\n","Requirement already satisfied: six<2dev,>=1.6.1 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (1.12.0)\n","Requirement already satisfied: httplib2<1dev,>=0.9.2 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (0.11.3)\n","Requirement already satisfied: google-auth>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (1.4.2)\n","Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (3.0.0)\n","Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.6/dist-packages (from google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (0.0.3)\n","Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->pydrive->Colab-Utils==0.1.dev0) (0.2.7)\n","Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->pydrive->Colab-Utils==0.1.dev0) (0.4.7)\n","Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from oauth2client>=4.0.0->pydrive->Colab-Utils==0.1.dev0) (4.0)\n","Requirement already satisfied: cachetools>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth>=1.4.1->google-api-python-client>=1.2->pydrive->Colab-Utils==0.1.dev0) (3.1.1)\n","Building wheels for collected packages: Colab-Utils\n","  Building wheel for Colab-Utils (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for Colab-Utils: filename=Colab_Utils-0.1.dev0-cp36-none-any.whl size=10323 sha256=8352dc453bca8404269d41720725a355434930637072c7a77f04f01a7bd8cdb1\n","  Stored in directory: /tmp/pip-ephem-wheel-cache-ufff2xrt/wheels/63/d1/27/a208931527abb98d326d00209f46c80c9d745851d6a1defd10\n","Successfully built Colab-Utils\n","Installing collected packages: Colab-Utils\n","Successfully installed Colab-Utils-0.1.dev0\n"]}],"source":["!pip install git+https://github.com/deepvision-class/starter-code"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"MzqbYcKdz6ew"},"source":["## Setup code\n","Run some setup code for this notebook."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"Q8o3FxatIL_X","tags":["pdf-ignore"]},"outputs":[],"source":["import coutils\n","from coutils import fix_random_seed\n","\n","from collections import OrderedDict\n","import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","import torch.optim as optim\n","from torch.utils.data import DataLoader\n","from torch.utils.data import sampler\n","\n","import torchvision.datasets as dset\n","import torchvision.transforms as T\n","\n","# for plotting\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n","plt.rcParams['image.interpolation'] = 'nearest'\n","plt.rcParams['image.cmap'] = 'gray'"]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":101},"colab_type":"code","executionInfo":{"elapsed":14539,"status":"ok","timestamp":1573619044146,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"-XB6NUX0IL_f","outputId":"00c40cd6-cf28-4457-c376-d855687dec85","tags":["pdf-ignore"]},"outputs":[{"name":"stderr","output_type":"stream","text":["  0%|          | 0/170498071 [00:00<?, ?it/s]"]},{"name":"stdout","output_type":"stream","text":["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./datasets/cifar-10-python.tar.gz\n"]},{"name":"stderr","output_type":"stream","text":["170500096it [00:01, 85288327.36it/s]                               \n"]},{"name":"stdout","output_type":"stream","text":["Extracting ./datasets/cifar-10-python.tar.gz to ./datasets\n","Files already downloaded and verified\n","Files already downloaded and verified\n"]}],"source":["NUM_TRAIN = 49000\n","\n","# The torchvision.transforms package provides tools for preprocessing data\n","# and for performing data augmentation; here we set up a transform to\n","# preprocess the data by subtracting the mean RGB value and dividing by the\n","# standard deviation of each RGB value; we've hardcoded the mean and std.\n","transform = T.Compose([\n","                T.ToTensor(),\n","                T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))\n","            ])\n","\n","# We set up a Dataset object for each split (train / val / test); Datasets load\n","# training examples one at a time, so we wrap each Dataset in a DataLoader which\n","# iterates through the Dataset and forms minibatches. We divide the CIFAR-10\n","# training set into train and val sets by passing a Sampler object to the\n","# DataLoader telling how it should sample from the underlying Dataset.\n","cifar10_train = dset.CIFAR10('./datasets', train=True, download=True,\n","                             transform=transform)\n","loader_train = DataLoader(cifar10_train, batch_size=64, \n","                          sampler=sampler.SubsetRandomSampler(range(NUM_TRAIN)))\n","\n","cifar10_val = dset.CIFAR10('./datasets', train=True, download=True,\n","                           transform=transform)\n","loader_val = DataLoader(cifar10_val, batch_size=64, \n","                        sampler=sampler.SubsetRandomSampler(range(NUM_TRAIN, 50000)))\n","\n","cifar10_test = dset.CIFAR10('./datasets', train=False, download=True, \n","                            transform=transform)\n","loader_test = DataLoader(cifar10_test, batch_size=64)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"FiookQItIL_p","tags":["pdf-ignore"]},"source":["Note that if CUDA is not enabled, `torch.cuda.is_available()` will return False and this notebook will fallback to CPU mode.\n","\n","The global variables `dtype` and `device` will control the data types throughout this assignment.\n","\n","We will be using `torch.float = torch.float32` for data and `torch.long = torch.int64` for labels.\n","\n","Please refer to https://pytorch.org/docs/stable/tensor_attributes.html#torch-dtype for more details about data types."]},{"cell_type":"code","execution_count":4,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":34},"colab_type":"code","executionInfo":{"elapsed":14602,"status":"ok","timestamp":1573619044214,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"blz1sXlkIL_q","outputId":"a5de8fd5-2f67-440b-c40c-e7399d2527ea","tags":["pdf-ignore-input"]},"outputs":[{"name":"stdout","output_type":"stream","text":["using device: cuda:0\n"]}],"source":["dtype = torch.float\n","ltype = torch.long\n","\n","if torch.cuda.is_available():\n","  device = torch.device('cuda:0')\n","else:\n","  device = torch.device('cpu')\n","\n","# Constant to control how frequently we print train loss\n","print_every = 100\n","\n","print('using device:', device)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"arcbqxO4AmAx"},"source":["Before we start, we define the flatten function for convenience."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"xAhc4L8dzwbM"},"outputs":[],"source":["def flatten(x, start_dim=1, end_dim=-1):\n","  return x.flatten(start_dim=start_dim, end_dim=end_dim)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"mazLauPlIL_5"},"source":["# Part II. Barebones PyTorch\n","\n","PyTorch ships with high-level APIs to help us define model architectures conveniently, which we will cover in Part II of this tutorial. In this section, we will start with the barebone PyTorch elements to understand the autograd engine better. After this exercise, you will come to appreciate the high-level model API more.\n","\n","We will start with a simple fully-connected ReLU network with two hidden layers and no biases for CIFAR classification. \n","This implementation computes the forward pass using operations on PyTorch Tensors, and uses PyTorch autograd to compute gradients. It is important that you understand every line, because you will write a harder version after the example.\n","\n","When we create a PyTorch Tensor with `requires_grad=True`, then operations involving that Tensor will not just compute values; they will also build up a computational graph in the background, allowing us to easily backpropagate through the graph to compute gradients of some Tensors with respect to a downstream loss. Concretely, if `x` is a Tensor with `x.requires_grad == True` then after backpropagation `x.grad` will be another Tensor holding the gradient of `x` with respect to the scalar loss at the end."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"3rPnppVkIMAB","tags":["pdf-ignore"]},"source":["### Barebones PyTorch: Two-Layer Network\n","\n","Here we define a function `two_layer_fc` which performs the forward pass of a two-layer fully-connected ReLU network on a batch of image data. After defining the forward pass we check that it doesn't crash and that it produces outputs of the right shape by running zeros through the network.\n","\n","You don't have to write any code here, but it's important that you read and understand the implementation."]},{"cell_type":"code","execution_count":6,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":34},"colab_type":"code","executionInfo":{"elapsed":14658,"status":"ok","timestamp":1573619044276,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"D6PqRQwlIMAC","outputId":"f23dc566-ba1a-4c23-dee4-06e2f457ddcb","tags":["pdf-ignore-input"]},"outputs":[{"name":"stdout","output_type":"stream","text":["Output size: [64, 10]\n"]}],"source":["def two_layer_fc(x, params):\n","  \"\"\"\n","  A fully-connected neural networks; the architecture is:\n","  NN is fully connected -> ReLU -> fully connected layer.\n","  Note that this function only defines the forward pass; \n","  PyTorch will take care of the backward pass for us.\n","  \n","  The input to the network will be a minibatch of data, of shape\n","  (N, d1, ..., dM) where d1 * ... * dM = D. The hidden layer will have H units,\n","  and the output layer will produce scores for C classes.\n","  \n","  Inputs:\n","  - x: A PyTorch Tensor of shape (N, d1, ..., dM) giving a minibatch of\n","    input data.\n","  - params: A list [w1, w2] of PyTorch Tensors giving weights for the network;\n","    w1 has shape (H, D) and w2 has shape (C, H).\n","  \n","  Returns:\n","  - scores: A PyTorch Tensor of shape (N, C) giving classification scores for\n","    the input data x.\n","  \"\"\"\n","  # first we flatten the image\n","  x = flatten(x)  # shape: [batch_size, C x H x W]\n","  \n","  w1, b1, w2, b2 = params\n","  \n","  # Forward pass: compute predicted y using operations on Tensors. Since w1 and\n","  # w2 have requires_grad=True, operations involving these Tensors will cause\n","  # PyTorch to build a computational graph, allowing automatic computation of\n","  # gradients. Since we are no longer implementing the backward pass by hand we\n","  # don't need to keep references to intermediate values.\n","  # Note that F.linear(x, w, b) is equivalent to x.mm(w.t()) + b\n","  # For ReLU, you can also use `.clamp(min=0)`, equivalent to `F.relu()`\n","  x = F.relu(F.linear(x, w1, b1))\n","  x = F.linear(x, w2, b2)\n","  return x\n","    \n","\n","def two_layer_fc_test():\n","  hidden_layer_size = 42\n","  x = torch.zeros((64, 3, 16, 16), dtype=dtype)  # minibatch size 64, feature dimension 3*16*16\n","  w1 = torch.zeros((hidden_layer_size, 3*16*16), dtype=dtype)\n","  b1 = torch.zeros((hidden_layer_size,), dtype=dtype)\n","  w2 = torch.zeros((10, hidden_layer_size), dtype=dtype)\n","  b2 = torch.zeros((10,), dtype=dtype)\n","  scores = two_layer_fc(x, [w1, b1, w2, b2])\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","\n","two_layer_fc_test()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"muCDvYEKIMAJ"},"source":["### Barebones PyTorch: Three-Layer ConvNet\n","\n","Here you will complete the implementation of the function `three_layer_convnet`, which will perform the forward pass of a three-layer convolutional network. Like above, we can immediately test our implementation by passing zeros through the network. The network should have the following architecture:\n","\n","1. A convolutional layer (with bias) with `channel_1` filters, each with shape `KW1 x KH1`, and zero-padding of two\n","2. ReLU nonlinearity\n","3. A convolutional layer (with bias) with `channel_2` filters, each with shape `KW2 x KH2`, and zero-padding of one\n","4. ReLU nonlinearity\n","5. Fully-connected layer with bias, producing scores for C classes.\n","\n","Note that we have **no softmax activation** here after our fully-connected layer: this is because PyTorch's cross entropy loss performs a softmax activation for you, and by bundling that step in makes computation more efficient.\n","\n","**HINT**: For convolutions: https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.conv2d; pay attention to the shapes of convolutional filters!"]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"FHWVtcaEIMAM"},"outputs":[],"source":["def three_layer_convnet(x, params):\n","  \"\"\"\n","  Performs the forward pass of a three-layer convolutional network with the\n","  architecture defined above.\n","\n","  Inputs:\n","  - x: A PyTorch Tensor of shape (N, C, H, W) giving a minibatch of images\n","  - params: A list of PyTorch Tensors giving the weights and biases for the\n","    network; should contain the following:\n","    - conv_w1: PyTorch Tensor of shape (channel_1, C, KH1, KW1) giving weights\n","      for the first convolutional layer\n","    - conv_b1: PyTorch Tensor of shape (channel_1,) giving biases for the first\n","      convolutional layer\n","    - conv_w2: PyTorch Tensor of shape (channel_2, channel_1, KH2, KW2) giving\n","      weights for the second convolutional layer\n","    - conv_b2: PyTorch Tensor of shape (channel_2,) giving biases for the second\n","      convolutional layer\n","    - fc_w: PyTorch Tensor giving weights for the fully-connected layer. Can you\n","      figure out what the shape should be?\n","    - fc_b: PyTorch Tensor giving biases for the fully-connected layer. Can you\n","      figure out what the shape should be?\n","  \n","  Returns:\n","  - scores: PyTorch Tensor of shape (N, C) giving classification scores for x\n","  \"\"\"\n","  conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b = params\n","  scores = None\n","  ##############################################################################\n","  # TODO: Implement the forward pass for the three-layer ConvNet.              #\n","  # Hint: F.linear, F.conv2d, F.relu, flatten                                  #\n","  ##############################################################################\n","  # Replace \"pass\" statement with your code\n","  h1 = F.relu(F.conv2d(x, conv_w1, bias=conv_b1, stride=1, padding=2))\n","  h2 = F.relu(F.conv2d(h1, conv_w2, bias=conv_b2, stride=1, padding=1))\n","  h2 = flatten(h2)\n","  scores = F.linear(h2, fc_w, fc_b)\n","  ################################################################################\n","  #                                 END OF YOUR CODE                             #\n","  ################################################################################\n","  return scores"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"Z7Cg9qvTIMAR"},"source":["After defining the forward pass of the ConvNet above, run the following cell to test your implementation.\n","\n","When you run this function, scores should have shape (64, 10)."]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":34},"colab_type":"code","executionInfo":{"elapsed":14863,"status":"ok","timestamp":1573619044488,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"1kEMMi4QIMAa","outputId":"e1372028-91b5-4a99-e421-3ab6b525b7c0","tags":["pdf-ignore-input"]},"outputs":[{"name":"stdout","output_type":"stream","text":["Output size: [64, 10]\n"]}],"source":["def three_layer_convnet_test():\n","  x = torch.zeros((64, 3, 32, 32), dtype=dtype)  # minibatch size 64, image size [3, 32, 32]\n","\n","  conv_w1 = torch.zeros((6, 3, 5, 5), dtype=dtype)  # [out_channel, in_channel, kernel_H, kernel_W]\n","  conv_b1 = torch.zeros((6,))  # out_channel\n","  conv_w2 = torch.zeros((9, 6, 3, 3), dtype=dtype)  # [out_channel, in_channel, kernel_H, kernel_W]\n","  conv_b2 = torch.zeros((9,))  # out_channel\n","\n","  # you must calculate the shape of the tensor after two conv layers, before the fully-connected layer\n","  fc_w = torch.zeros((10, 9 * 32 * 32))\n","  fc_b = torch.zeros(10)\n","\n","  scores = three_layer_convnet(x, [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b])\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","three_layer_convnet_test()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"WBStmY2yIMAh"},"source":["### Barebones PyTorch: Kaiming Initialization\n","In this part, we are going to use [Kaiming initialization](https://arxiv.org/abs/1502.01852), which you already implemented in Assignment 3.\n","\n","Fortunately, PyTorch already provides a function, so we will use this:\n","[torch.nn.init.kaiming_normal_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_)\n","\n","By default, `gain = 2`, because this function assumes that ReLU activation follows.\n","This is true in the linear and convolutional layers in the models you are going to implement, except for the last fully-connected layer:\n","in principle, we should give `gain = 1` because ReLU is not applied there.\n","However, as stated in the [original paper](https://arxiv.org/abs/1502.01852), since the factor 1/2 does not matter if it just exists on one layer, we are going to keep using `gain = 2` for simplicity.\n","\n","For more details on initialization methods provided by PyTorch, see https://pytorch.org/docs/stable/nn.init.html."]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":118},"colab_type":"code","executionInfo":{"elapsed":19976,"status":"ok","timestamp":1573619049605,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"_rf9JRh5IMAj","outputId":"66e0fe0a-71e4-4b4c-bf59-dc587c4244e8","tags":["pdf-ignore-input"]},"outputs":[{"name":"stdout","output_type":"stream","text":["tensor([[-0.5848, -0.2690, -1.6721,  0.0918, -0.0764],\n","        [-0.3667, -0.3939, -0.2077, -0.6796, -0.2297],\n","        [-1.0569,  1.4328,  0.1971, -0.1165,  0.8137]], device='cuda:0')\n","tensor([[0., 0., 0., 0., 0.],\n","        [0., 0., 0., 0., 0.],\n","        [0., 0., 0., 0., 0.]], device='cuda:0')\n"]}],"source":["fix_random_seed(0)\n","\n","# create a weight of shape [3 x 5]\n","print(nn.init.kaiming_normal_(torch.empty(3,5, dtype=dtype, device=device)))\n","print(nn.init.zeros_(torch.empty(3,5, dtype=dtype, device=device)))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"akQqCObPIMAo"},"source":["### Barebones PyTorch: Check Accuracy\n","When training the model we will use the following function to check the accuracy of our model on the training or validation sets.\n","\n","When checking accuracy we don't need to compute any gradients; as a result we don't need PyTorch to build a computational graph for us when we compute scores. To prevent a graph from being built we scope our computation under a `torch.no_grad()` context manager."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"l4xAUWASIMAq","tags":["pdf-ignore-input"]},"outputs":[],"source":["def check_accuracy_part2(loader, model_fn, params):\n","  \"\"\"\n","  Check the accuracy of a classification model.\n","  \n","  Inputs:\n","  - loader: A DataLoader for the data split we want to check\n","  - model_fn: A function that performs the forward pass of the model,\n","    with the signature scores = model_fn(x, params)\n","  - params: List of PyTorch Tensors giving parameters of the model\n","  \n","  Returns: Nothing, but prints the accuracy of the model\n","  \"\"\"\n","  split = 'val' if loader.dataset.train else 'test'\n","  print('Checking accuracy on the %s set' % split)\n","  num_correct, num_samples = 0, 0\n","  with torch.no_grad():\n","    for x, y in loader:\n","      x = x.to(device=device, dtype=dtype)  # move to device, e.g. GPU\n","      y = y.to(device=device, dtype=ltype)\n","      scores = model_fn(x, params)\n","      _, preds = scores.max(1)\n","      num_correct += (preds == y).sum()\n","      num_samples += preds.size(0)\n","    acc = float(num_correct) / num_samples\n","    print('Got %d / %d correct (%.2f%%)' % (num_correct, num_samples, 100 * acc))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"8LvPbM4WIMAv"},"source":["### BareBones PyTorch: Training Loop\n","We can now set up a basic training loop to train our network. We will train the model using stochastic gradient descent without momentum. We will use `torch.nn.functional.cross_entropy` to compute the loss; you can [read about it here](https://pytorch.org/docs/stable/nn.html#crossentropyloss).\n","\n","The training loop takes as input the neural network function, a list of initialized parameters (`[w1, w2]` in our example), and learning rate."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"WYGBD0YZIMAx","tags":["pdf-ignore-input"]},"outputs":[],"source":["def train_part2(model_fn, params, learning_rate):\n","  \"\"\"\n","  Train a model on CIFAR-10.\n","  \n","  Inputs:\n","  - model_fn: A Python function that performs the forward pass of the model.\n","    It should have the signature scores = model_fn(x, params) where x is a\n","    PyTorch Tensor of image data, params is a list of PyTorch Tensors giving\n","    model weights, and scores is a PyTorch Tensor of shape (N, C) giving\n","    scores for the elements in x.\n","  - params: List of PyTorch Tensors giving weights for the model\n","  - learning_rate: Python scalar giving the learning rate to use for SGD\n","  \n","  Returns: Nothing\n","  \"\"\"\n","  for t, (x, y) in enumerate(loader_train):\n","    # Move the data to the proper device (GPU or CPU)\n","    x = x.to(device=device, dtype=dtype)\n","    y = y.to(device=device, dtype=ltype)\n","\n","    # Forward pass: compute scores and loss\n","    scores = model_fn(x, params)\n","    loss = F.cross_entropy(scores, y)\n","\n","    # Backward pass: PyTorch figures out which Tensors in the computational\n","    # graph has requires_grad=True and uses backpropagation to compute the\n","    # gradient of the loss with respect to these Tensors, and stores the\n","    # gradients in the .grad attribute of each Tensor.\n","    loss.backward()\n","\n","    # Update parameters. We don't want to backpropagate through the\n","    # parameter updates, so we scope the updates under a torch.no_grad()\n","    # context manager to prevent a computational graph from being built.\n","    with torch.no_grad():\n","      for w in params:\n","        if w.requires_grad:\n","          w -= learning_rate * w.grad\n","\n","          # Manually zero the gradients after running the backward pass\n","          w.grad.zero_()\n","\n","    if t % print_every == 0 or t == len(loader_train)-1:\n","      print('Iteration %d, loss = %.4f' % (t, loss.item()))\n","      check_accuracy_part2(loader_val, model_fn, params)\n","      print()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"gkTNXK9cIMA6"},"source":["### BareBones PyTorch: Train a Two-Layer Network\n","Now we are ready to run the training loop. We need to explicitly allocate tensors for the fully connected weights, `w1` and `w2`. \n","\n","Each minibatch of CIFAR has 64 examples, so the tensor shape is `[64, 3, 32, 32]`. \n","\n","After flattening, `x` shape should be `[64, 3 * 32 * 32]`. This will be the size of the second dimension of `w1`. \n","The first dimension of `w1` is the hidden layer size, which will also be the second dimension of `w2`. \n","\n","Finally, the output of the network is a 10-dimensional vector that represents the probability distribution over 10 classes. \n","\n","You don't need to tune any hyperparameters but you should see accuracies above 40% after training for one epoch."]},{"cell_type":"code","execution_count":12,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":622},"colab_type":"code","executionInfo":{"elapsed":34799,"status":"ok","timestamp":1573619064437,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"OSBSy0JTIMA8","outputId":"7c4d44bb-f091-4e38-fee4-71d3a5177a38"},"outputs":[{"name":"stdout","output_type":"stream","text":["Iteration 0, loss = 3.5316\n","Checking accuracy on the val set\n","Got 119 / 1000 correct (11.90%)\n","\n","Iteration 100, loss = 2.9482\n","Checking accuracy on the val set\n","Got 359 / 1000 correct (35.90%)\n","\n","Iteration 200, loss = 2.1805\n","Checking accuracy on the val set\n","Got 396 / 1000 correct (39.60%)\n","\n","Iteration 300, loss = 1.8965\n","Checking accuracy on the val set\n","Got 408 / 1000 correct (40.80%)\n","\n","Iteration 400, loss = 1.5593\n","Checking accuracy on the val set\n","Got 419 / 1000 correct (41.90%)\n","\n","Iteration 500, loss = 2.1956\n","Checking accuracy on the val set\n","Got 422 / 1000 correct (42.20%)\n","\n","Iteration 600, loss = 1.8713\n","Checking accuracy on the val set\n","Got 415 / 1000 correct (41.50%)\n","\n","Iteration 700, loss = 1.8003\n","Checking accuracy on the val set\n","Got 435 / 1000 correct (43.50%)\n","\n","Iteration 765, loss = 1.5759\n","Checking accuracy on the val set\n","Got 401 / 1000 correct (40.10%)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","\n","w1 = nn.init.kaiming_normal_(torch.empty(hidden_layer_size, C*H*W, dtype=dtype, device=device))\n","w1.requires_grad = True\n","b1 = nn.init.zeros_(torch.empty(hidden_layer_size, dtype=dtype, device=device))\n","b1.requires_grad = True\n","w2 = nn.init.kaiming_normal_(torch.empty(num_classes, hidden_layer_size, dtype=dtype, device=device))\n","w2.requires_grad = True\n","b2 = nn.init.zeros_(torch.empty(num_classes, dtype=dtype, device=device))\n","b2.requires_grad = True\n","\n","train_part2(two_layer_fc, [w1, b1, w2, b2], learning_rate)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"7K_n_BZPIMBB"},"source":["### BareBones PyTorch: Training a ConvNet\n","\n","In the below you should use the functions defined above to train a three-layer convolutional network on CIFAR. The network should have the following architecture:\n","\n","1. Convolutional layer (with bias) with 32 5x5 filters, with zero-padding of 2\n","2. ReLU\n","3. Convolutional layer (with bias) with 16 3x3 filters, with zero-padding of 1\n","4. ReLU\n","5. Fully-connected layer (with bias) to compute scores for 10 classes\n","\n","You should initialize your weight matrices using the `random_weight` function defined above, and you should initialize your bias vectors using the `zero_weight` function above.\n","\n","You don't need to tune any hyperparameters, but if everything works correctly you should achieve an accuracy above 45% after one epoch."]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":622},"colab_type":"code","executionInfo":{"elapsed":50616,"status":"ok","timestamp":1573619080258,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"wBRWytEzIMBC","outputId":"c70bf337-9aac-405f-d289-54686ac4922c"},"outputs":[{"name":"stdout","output_type":"stream","text":["Iteration 0, loss = 2.4780\n","Checking accuracy on the val set\n","Got 111 / 1000 correct (11.10%)\n","\n","Iteration 100, loss = 1.8853\n","Checking accuracy on the val set\n","Got 347 / 1000 correct (34.70%)\n","\n","Iteration 200, loss = 1.9925\n","Checking accuracy on the val set\n","Got 398 / 1000 correct (39.80%)\n","\n","Iteration 300, loss = 1.5757\n","Checking accuracy on the val set\n","Got 416 / 1000 correct (41.60%)\n","\n","Iteration 400, loss = 1.5241\n","Checking accuracy on the val set\n","Got 446 / 1000 correct (44.60%)\n","\n","Iteration 500, loss = 1.7093\n","Checking accuracy on the val set\n","Got 453 / 1000 correct (45.30%)\n","\n","Iteration 600, loss = 1.5381\n","Checking accuracy on the val set\n","Got 481 / 1000 correct (48.10%)\n","\n","Iteration 700, loss = 1.7429\n","Checking accuracy on the val set\n","Got 488 / 1000 correct (48.80%)\n","\n","Iteration 765, loss = 1.3104\n","Checking accuracy on the val set\n","Got 473 / 1000 correct (47.30%)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","channel_1 = 32\n","channel_2 = 16\n","kernel_size_1 = 5\n","kernel_size_2 = 3\n","\n","learning_rate = 3e-3\n","\n","conv_w1 = None\n","conv_b1 = None\n","conv_w2 = None\n","conv_b2 = None\n","fc_w = None\n","fc_b = None\n","\n","################################################################################\n","# TODO: Define and initialize the parameters of a three-layer ConvNet          #\n","#       using nn.init.kaiming_normal_.                                         #\n","################################################################################\n","# Replace \"pass\" statement with your code\n","conv_w1 = nn.init.kaiming_normal_(torch.empty((channel_1, C, kernel_size_1, kernel_size_1), dtype=dtype, device=device))\n","conv_w1.requires_grad = True\n","conv_b1 = nn.init.zeros_(torch.empty(channel_1, dtype=dtype, device=device))\n","conv_b1.requires_grad = True\n","conv_w2 = nn.init.kaiming_normal_(torch.empty((channel_2, channel_1, kernel_size_2, kernel_size_2), dtype=dtype, device=device))\n","conv_w2.requires_grad = True\n","conv_b2 = nn.init.zeros_(torch.empty(channel_2, dtype=dtype, device=device))\n","conv_b2.requires_grad = True\n","fc_w = nn.init.kaiming_normal_(torch.empty(num_classes, channel_2*H*W, dtype=dtype, device=device))\n","fc_w.requires_grad = True\n","fc_b = nn.init.zeros_(torch.empty(num_classes, dtype=dtype, device=device))\n","fc_b.requires_grad = True\n","\n","#train_part2(three_layer_convnet, [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b], learning_rate) \n","################################################################################\n","#                                 END OF YOUR CODE                             #\n","################################################################################\n","\n","params = [conv_w1, conv_b1, conv_w2, conv_b2, fc_w, fc_b]\n","train_part2(three_layer_convnet, params, learning_rate)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"YXGdsYInIMBH"},"source":["# Part III. PyTorch Module API\n","\n","Barebone PyTorch requires that we track all the parameter tensors by hand. This is fine for small networks with a few tensors, but it would be extremely inconvenient and error-prone to track tens or hundreds of tensors in larger networks.\n","\n","PyTorch provides the `nn.Module` API for you to define arbitrary network architectures, while tracking every learnable parameters for you. In Part II, we implemented SGD ourselves. PyTorch also provides the `torch.optim` package that implements all the common optimizers, such as RMSProp, Adagrad, and Adam. It even supports approximate second-order methods like L-BFGS! You can refer to the [doc](http://pytorch.org/docs/master/optim.html) for the exact specifications of each optimizer.\n","\n","To use the Module API, follow the steps below:\n","\n","1. Subclass `nn.Module`. Give your network class an intuitive name like `TwoLayerFC`. \n","\n","2. In the constructor `__init__()`, define all the layers you need as class attributes. Layer objects like `nn.Linear` and `nn.Conv2d` are themselves `nn.Module` subclasses and contain learnable parameters, so that you don't have to instantiate the raw tensors yourself. `nn.Module` will track these internal parameters for you. Refer to the [doc](http://pytorch.org/docs/master/nn.html) to learn more about the dozens of builtin layers. **Warning**: don't forget to call the `super().__init__()` first!\n","\n","3. In the `forward()` method, define the *connectivity* of your network. You should use the attributes defined in `__init__` as function calls that take tensor as input and output the \"transformed\" tensor. Do *not* create any new layers with learnable parameters in `forward()`! All of them must be declared upfront in `__init__`. \n","\n","After you define your Module subclass, you can instantiate it as an object and call it just like the NN forward function in part II.\n","\n","### Module API: Two-Layer Network\n","Here is a concrete example of a 2-layer fully connected network.\n","We use `nn.init.kaiming_normal_` to initialize weights using Kaiming initialization, and `nn.init.kaiming_zeros_` to initialize biases."]},{"cell_type":"code","execution_count":14,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":118},"colab_type":"code","executionInfo":{"elapsed":50613,"status":"ok","timestamp":1573619080259,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"2Ue0_Cf1IMBJ","outputId":"cb634d57-96ff-4373-db55-6c982ef40b0d"},"outputs":[{"name":"stdout","output_type":"stream","text":["Architecture:\n","TwoLayerFC(\n","  (fc1): Linear(in_features=768, out_features=42, bias=True)\n","  (fc2): Linear(in_features=42, out_features=10, bias=True)\n",")\n","Output size: [64, 10]\n"]}],"source":["class TwoLayerFC(nn.Module):\n","  def __init__(self, input_size, hidden_size, num_classes):\n","    super().__init__()\n","    # assign layer objects to class attributes\n","    self.fc1 = nn.Linear(input_size, hidden_size)\n","    self.fc2 = nn.Linear(hidden_size, num_classes)\n","    # nn.init package contains convenient initialization methods\n","    # https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_ \n","    nn.init.kaiming_normal_(self.fc1.weight)\n","    nn.init.kaiming_normal_(self.fc2.weight)\n","    nn.init.zeros_(self.fc1.bias)\n","    nn.init.zeros_(self.fc2.bias)\n","  \n","  def forward(self, x):\n","    # forward always defines connectivity\n","    x = flatten(x)\n","    scores = self.fc2(F.relu(self.fc1(x)))\n","    return scores\n","\n","def test_TwoLayerFC():\n","  input_size = 3*16*16\n","  x = torch.zeros((64, input_size), dtype=dtype)  # minibatch size 64, feature dimension 3*16*16\n","  model = TwoLayerFC(input_size, 42, 10)\n","  scores = model(x)\n","  print('Architecture:')\n","  print(model) # printing `nn.Module` shows the architecture of the module.\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","test_TwoLayerFC()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"CruYc2HJIMBP"},"source":["### Module API: Three-Layer ConvNet\n","It's your turn to implement a 3-layer ConvNet followed by a fully connected layer. The network architecture should be the same as in Part II:\n","\n","1. Convolutional layer with `channel_1` 5x5 filters with zero-padding of 2\n","2. ReLU\n","3. Convolutional layer with `channel_2` 3x3 filters with zero-padding of 1\n","4. ReLU\n","5. Fully-connected layer to `num_classes` classes\n","\n","We assume that the size of the input of this network is `H = W = 32`, and there is no pooing; this information is required when computing the number of input channels in the last fully-connected layer.\n","You should initialize the weight matrices of the model using Kaiming normal initialization, and zero out the bias vectors.\n","\n","**HINT**: [torch.nn.Conv2d](https://pytorch.org/docs/stable/nn.html#conv2d), [torch.nn.init.kaiming_normal_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_), [torch.nn.init.zeros_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.zeros_)\n","\n","After you implement the three-layer ConvNet, the `test_ThreeLayerConvNet` function will run your implementation; it should print `(64, 10)` for the shape of the output scores."]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":118},"colab_type":"code","executionInfo":{"elapsed":50739,"status":"ok","timestamp":1573619080389,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"R58EqBTYIMBU","outputId":"03da4be8-f51c-4da6-a34c-ba9e264d0e5f"},"outputs":[{"name":"stdout","output_type":"stream","text":["ThreeLayerConvNet(\n","  (conv1): Conv2d(3, 12, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n","  (conv2): Conv2d(12, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","  (fc): Linear(in_features=8192, out_features=10, bias=True)\n",")\n","Output size: [64, 10]\n"]}],"source":["class ThreeLayerConvNet(nn.Module):\n","  def __init__(self, in_channel, channel_1, channel_2, num_classes):\n","    super().__init__()\n","    ########################################################################\n","    # TODO: Set up the layers you need for a three-layer ConvNet with the  #\n","    # architecture defined above.                                          #\n","    ########################################################################\n","    # Replace \"pass\" statement with your code\n","    self.conv1 = nn.Conv2d(in_channel, channel_1, 5, padding=2)\n","    self.conv2 = nn.Conv2d(channel_1, channel_2, 3, padding=1)\n","    self.fc = nn.Linear(channel_2*H*W ,num_classes)\n","    nn.init.kaiming_normal_(self.conv1.weight)\n","    nn.init.kaiming_normal_(self.conv2.weight)\n","    nn.init.kaiming_normal_(self.fc.weight)\n","    nn.init.zeros_(self.conv1.bias)\n","    nn.init.zeros_(self.conv2.bias)\n","    nn.init.zeros_(self.fc.bias)\n","    ########################################################################\n","    #                           END OF YOUR CODE                           #     \n","    ########################################################################\n","\n","  def forward(self, x):\n","    scores = None\n","    ########################################################################\n","    # TODO: Implement the forward function for a 3-layer ConvNet. you      #\n","    # should use the layers you defined in __init__ and specify the        #\n","    # connectivity of those layers in forward()                            #\n","    ########################################################################\n","    # Replace \"pass\" statement with your code\n","    h1 = F.relu(self.conv1(x))\n","    h2 = F.relu(self.conv2(h1))\n","    h2 = flatten(h2)\n","    scores = self.fc(h2)\n","    ########################################################################\n","    #                            END OF YOUR CODE                          #\n","    ########################################################################\n","    return scores\n","\n","\n","def test_ThreeLayerConvNet():\n","  x = torch.zeros((64, 3, 32, 32), dtype=dtype)  # minibatch size 64, image size [3, 32, 32]\n","  model = ThreeLayerConvNet(in_channel=3, channel_1=12, channel_2=8, num_classes=10)\n","  scores = model(x)\n","  print(model) # printing `nn.Module` shows the architecture of the module.\n","  print('Output size:', list(scores.size()))  # you should see [64, 10]\n","test_ThreeLayerConvNet()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"3efFpwV_IMBZ"},"source":["### Module API: Check Accuracy\n","Given the validation or test set, we can check the classification accuracy of a neural network. \n","\n","This version is slightly different from the one in part II. You don't manually pass in the parameters anymore."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"LpgKJLVbIMBb"},"outputs":[],"source":["def check_accuracy_part34(loader, model):\n","  if loader.dataset.train:\n","    print('Checking accuracy on validation set')\n","  else:\n","    print('Checking accuracy on test set')   \n","  num_correct = 0\n","  num_samples = 0\n","  model.eval()  # set model to evaluation mode\n","  with torch.no_grad():\n","    for x, y in loader:\n","      x = x.to(device=device, dtype=dtype)  # move to device, e.g. GPU\n","      y = y.to(device=device, dtype=ltype)\n","      scores = model(x)\n","      _, preds = scores.max(1)\n","      num_correct += (preds == y).sum()\n","      num_samples += preds.size(0)\n","    acc = float(num_correct) / num_samples\n","    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))\n","  return acc"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"-tmOMi8SIMBj"},"source":["### Module API: Training Loop\n","We also use a slightly different training loop. Rather than updating the values of the weights ourselves, we use an Optimizer object from the `torch.optim` package, which abstract the notion of an optimization algorithm and provides implementations of most of the algorithms commonly used to optimize neural networks."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"HLJjvtu1IMBm"},"outputs":[],"source":["def adjust_learning_rate(optimizer, lrd, epoch, schedule):\n","  \"\"\"\n","  Multiply lrd to the learning rate if epoch is in schedule\n","  \n","  Inputs:\n","  - optimizer: An Optimizer object we will use to train the model\n","  - lrd: learning rate decay; a factor multiplied at scheduled epochs\n","  - epochs: the current epoch number\n","  - schedule: the list of epochs that requires learning rate update\n","  \n","  Returns: Nothing, but learning rate might be updated\n","  \"\"\"\n","  if epoch in schedule:\n","    for param_group in optimizer.param_groups:\n","      print('lr decay from {} to {}'.format(param_group['lr'], param_group['lr'] * lrd))\n","      param_group['lr'] *= lrd\n","\n","def train_part345(model, optimizer, epochs=1, learning_rate_decay=.1, schedule=[], verbose=True):\n","  \"\"\"\n","  Train a model on CIFAR-10 using the PyTorch Module API.\n","  \n","  Inputs:\n","  - model: A PyTorch Module giving the model to train.\n","  - optimizer: An Optimizer object we will use to train the model\n","  - epochs: (Optional) A Python integer giving the number of epochs to train for\n","  \n","  Returns: Nothing, but prints model accuracies during training.\n","  \"\"\"\n","  model = model.to(device=device)  # move the model parameters to CPU/GPU\n","  num_iters = epochs * len(loader_train)\n","  if verbose:\n","    num_prints = num_iters // print_every + 1\n","  else:\n","    num_prints = epochs\n","  acc_history = torch.zeros(num_prints, dtype=torch.float)\n","  iter_history = torch.zeros(num_prints, dtype=torch.long)\n","  for e in range(epochs):\n","    \n","    adjust_learning_rate(optimizer, learning_rate_decay, e, schedule)\n","    \n","    for t, (x, y) in enumerate(loader_train):\n","      model.train()  # put model to training mode\n","      x = x.to(device=device, dtype=dtype)  # move to device, e.g. GPU\n","      y = y.to(device=device, dtype=ltype)\n","\n","      scores = model(x)\n","      loss = F.cross_entropy(scores, y)\n","\n","      # Zero out all of the gradients for the variables which the optimizer\n","      # will update.\n","      optimizer.zero_grad()\n","\n","      # This is the backwards pass: compute the gradient of the loss with\n","      # respect to each  parameter of the model.\n","      loss.backward()\n","\n","      # Actually update the parameters of the model using the gradients\n","      # computed by the backwards pass.\n","      optimizer.step()\n","\n","      tt = t + e * len(loader_train)\n","\n","      if verbose and (tt % print_every == 0 or (e == epochs-1 and t == len(loader_train)-1)):\n","        print('Epoch %d, Iteration %d, loss = %.4f' % (e, tt, loss.item()))\n","        acc = check_accuracy_part34(loader_val, model)\n","        acc_history[tt // print_every] = acc\n","        iter_history[tt // print_every] = tt\n","        print()\n","      elif not verbose and (t == len(loader_train)-1):\n","        print('Epoch %d, Iteration %d, loss = %.4f' % (e, tt, loss.item()))\n","        acc = check_accuracy_part34(loader_val, model)\n","        acc_history[e] = acc\n","        iter_history[e] = tt\n","        print()\n","  return acc_history, iter_history"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"uFQs2WWKIMBu"},"source":["### Module API: Train a Two-Layer Network\n","Now we are ready to run the training loop. In contrast to part II, we don't explicitly allocate parameter tensors anymore.\n","\n","Simply pass the input size, hidden layer size, and number of classes (i.e. output size) to the constructor of `TwoLayerFC`. \n","\n","You also need to define an optimizer that tracks all the learnable parameters inside `TwoLayerFC`.\n","\n","You don't need to tune any hyperparameters, but you should see model accuracies above 40% after training for one epoch."]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":622},"colab_type":"code","executionInfo":{"elapsed":65511,"status":"ok","timestamp":1573619095169,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"v4Od-a6_IMBv","outputId":"2c43447c-821e-4813-8d7d-2d72262f8ccb"},"outputs":[{"name":"stdout","output_type":"stream","text":["Epoch 0, Iteration 0, loss = 3.3988\n","Checking accuracy on validation set\n","Got 139 / 1000 correct (13.90)\n","\n","Epoch 0, Iteration 100, loss = 2.9729\n","Checking accuracy on validation set\n","Got 328 / 1000 correct (32.80)\n","\n","Epoch 0, Iteration 200, loss = 2.1087\n","Checking accuracy on validation set\n","Got 335 / 1000 correct (33.50)\n","\n","Epoch 0, Iteration 300, loss = 2.1697\n","Checking accuracy on validation set\n","Got 432 / 1000 correct (43.20)\n","\n","Epoch 0, Iteration 400, loss = 1.9787\n","Checking accuracy on validation set\n","Got 434 / 1000 correct (43.40)\n","\n","Epoch 0, Iteration 500, loss = 1.7882\n","Checking accuracy on validation set\n","Got 445 / 1000 correct (44.50)\n","\n","Epoch 0, Iteration 600, loss = 2.1104\n","Checking accuracy on validation set\n","Got 471 / 1000 correct (47.10)\n","\n","Epoch 0, Iteration 700, loss = 1.6796\n","Checking accuracy on validation set\n","Got 472 / 1000 correct (47.20)\n","\n","Epoch 0, Iteration 765, loss = 1.6816\n","Checking accuracy on validation set\n","Got 409 / 1000 correct (40.90)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","weight_decay = 1e-4\n","\n","model = TwoLayerFC(C*H*W, hidden_layer_size, num_classes)\n","\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate,\n","                      weight_decay=weight_decay)\n","\n","_ = train_part345(model, optimizer)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"J_35SZYHIMB4"},"source":["### Module API: Train a Three-Layer ConvNet\n","You should now use the Module API to train a three-layer ConvNet on CIFAR. This should look very similar to training the two-layer network! You don't need to tune any hyperparameters, but you should achieve above above 45% after training for one epoch.\n","\n","You should train the model using stochastic gradient descent without momentum, with L2 weight decay of 1e-4."]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":622},"colab_type":"code","executionInfo":{"elapsed":81543,"status":"ok","timestamp":1573619111205,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"-bIRiwOJIMB6","outputId":"2de30a35-0d91-472e-b0fa-502e939a33b2"},"outputs":[{"name":"stdout","output_type":"stream","text":["Epoch 0, Iteration 0, loss = 3.5309\n","Checking accuracy on validation set\n","Got 120 / 1000 correct (12.00)\n","\n","Epoch 0, Iteration 100, loss = 1.9049\n","Checking accuracy on validation set\n","Got 353 / 1000 correct (35.30)\n","\n","Epoch 0, Iteration 200, loss = 1.7808\n","Checking accuracy on validation set\n","Got 411 / 1000 correct (41.10)\n","\n","Epoch 0, Iteration 300, loss = 1.7485\n","Checking accuracy on validation set\n","Got 461 / 1000 correct (46.10)\n","\n","Epoch 0, Iteration 400, loss = 1.3964\n","Checking accuracy on validation set\n","Got 457 / 1000 correct (45.70)\n","\n","Epoch 0, Iteration 500, loss = 1.6610\n","Checking accuracy on validation set\n","Got 481 / 1000 correct (48.10)\n","\n","Epoch 0, Iteration 600, loss = 1.3303\n","Checking accuracy on validation set\n","Got 488 / 1000 correct (48.80)\n","\n","Epoch 0, Iteration 700, loss = 1.6019\n","Checking accuracy on validation set\n","Got 505 / 1000 correct (50.50)\n","\n","Epoch 0, Iteration 765, loss = 1.5548\n","Checking accuracy on validation set\n","Got 499 / 1000 correct (49.90)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C = 3\n","num_classes = 10\n","\n","channel_1 = 32\n","channel_2 = 16\n","\n","learning_rate = 3e-3\n","weight_decay = 1e-4\n","\n","model = None\n","optimizer = None\n","################################################################################\n","# TODO: Instantiate your ThreeLayerConvNet model and a corresponding optimizer.#\n","################################################################################\n","# Replace \"pass\" statement with your code\n","model = ThreeLayerConvNet(C, channel_1, channel_2, num_classes)\n","\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate,\n","                      weight_decay=weight_decay)\n","################################################################################\n","#                                 END OF YOUR CODE                             \n","################################################################################\n","\n","_ = train_part345(model, optimizer)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"qeTdi4-xIMB_"},"source":["# Part IV. PyTorch Sequential API\n","\n","Part III introduced the PyTorch Module API, which allows you to define arbitrary learnable layers and their connectivity. \n","\n","For simple models like a stack of feed forward layers, you still need to go through 3 steps: subclass `nn.Module`, assign layers to class attributes in `__init__`, and call each layer one by one in `forward()`. Is there a more convenient way? \n","\n","Fortunately, PyTorch provides a container Module called `nn.Sequential`, which merges the above steps into one. It is not as flexible as `nn.Module`, because you cannot specify more complex topology than a feed-forward stack, but it's good enough for many use cases.\n","\n","Before we start, We need to wrap `flatten` function in a module in order to stack it in `nn.Sequential`.\n","As of 1.3.0, PyTorch supports `nn.Flatten`, so this is not required in the latest version.\n","However, let's use the following `Flatten` class for backward compatibility for now."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"eoBUeHDDeM1j"},"outputs":[],"source":["class Flatten(nn.Module):\n","  def forward(self, x):\n","    return flatten(x)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"ZZq-3hjxeYr4"},"source":["### Sequential API: Two-Layer Network\n","Let's see how to rewrite our two-layer fully connected network example with `nn.Sequential`, and train it using the training loop defined above.\n","Here, let's skip weight initialization for simplicity;\n","with a more advanced optimizer than the naive SGD, the default initialization provided in `torch.nn` is good enough for shallow networks.\n","\n","Again, you don't need to tune any hyperparameters here, but you should achieve above 40% accuracy after one epoch of training."]},{"cell_type":"code","execution_count":21,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":739},"colab_type":"code","executionInfo":{"elapsed":96472,"status":"ok","timestamp":1573619126140,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"9smkhciWIMCC","outputId":"885d8ea1-95e3-4444-9d08-287664edd727"},"outputs":[{"name":"stdout","output_type":"stream","text":["Architecture:\n","Sequential(\n","  (flatten): Flatten()\n","  (fc1): Linear(in_features=3072, out_features=4000, bias=True)\n","  (relu1): ReLU()\n","  (fc2): Linear(in_features=4000, out_features=10, bias=True)\n",")\n","Epoch 0, Iteration 0, loss = 2.3772\n","Checking accuracy on validation set\n","Got 137 / 1000 correct (13.70)\n","\n","Epoch 0, Iteration 100, loss = 1.7475\n","Checking accuracy on validation set\n","Got 394 / 1000 correct (39.40)\n","\n","Epoch 0, Iteration 200, loss = 1.7003\n","Checking accuracy on validation set\n","Got 408 / 1000 correct (40.80)\n","\n","Epoch 0, Iteration 300, loss = 1.6132\n","Checking accuracy on validation set\n","Got 442 / 1000 correct (44.20)\n","\n","Epoch 0, Iteration 400, loss = 1.6345\n","Checking accuracy on validation set\n","Got 456 / 1000 correct (45.60)\n","\n","Epoch 0, Iteration 500, loss = 1.5469\n","Checking accuracy on validation set\n","Got 464 / 1000 correct (46.40)\n","\n","Epoch 0, Iteration 600, loss = 1.4817\n","Checking accuracy on validation set\n","Got 470 / 1000 correct (47.00)\n","\n","Epoch 0, Iteration 700, loss = 1.6586\n","Checking accuracy on validation set\n","Got 451 / 1000 correct (45.10)\n","\n","Epoch 0, Iteration 765, loss = 1.2944\n","Checking accuracy on validation set\n","Got 460 / 1000 correct (46.00)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","hidden_layer_size = 4000\n","learning_rate = 1e-2\n","momentum = 0.5\n","\n","# To give a specific name to each module, use OrderedDict.\n","model = nn.Sequential(OrderedDict([\n","  ('flatten', Flatten()),\n","  ('fc1', nn.Linear(C*H*W, hidden_layer_size)),\n","  ('relu1', nn.ReLU()),\n","  ('fc2', nn.Linear(hidden_layer_size, num_classes)),\n","]))\n","\n","print('Architecture:')\n","print(model) # printing `nn.Module` shows the architecture of the module.\n","\n","# you can use Nesterov momentum in optim.SGD\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate, \n","                      weight_decay=weight_decay,\n","                      momentum=momentum, nesterov=True)\n","\n","_ = train_part345(model, optimizer)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"5YMYfFUGIMCH"},"source":["### Sequential API: Three-Layer ConvNet\n","Here you should use `nn.Sequential` to define and train a three-layer ConvNet with the same architecture we used in Part III:\n","\n","1. Convolutional layer (with bias) with 32 5x5 filters, with zero-padding of 2\n","2. ReLU\n","3. Convolutional layer (with bias) with 16 3x3 filters, with zero-padding of 1\n","4. ReLU\n","5. Fully-connected layer (with bias) to compute scores for 10 classes\n","\n","You don't have to re-initialize your weight matrices and bias vectors.\n","\n","You should optimize your model using stochastic gradient descent with Nesterov momentum 0.5, with L2 weight decay of 1e-4..\n","\n","Again, you don't need to tune any hyperparameters but you should see accuracy above 50% after one epoch of training."]},{"cell_type":"code","execution_count":22,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":773},"colab_type":"code","executionInfo":{"elapsed":112964,"status":"ok","timestamp":1573619142636,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"Q2To2-mtIMCJ","outputId":"00c4be3e-eb1e-49a9-b414-b934d09e68ae"},"outputs":[{"name":"stdout","output_type":"stream","text":["Architecture:\n","Sequential(\n","  (conv1): Conv2d(3, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n","  (relu1): ReLU()\n","  (conv2): Conv2d(32, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","  (relu2): ReLU()\n","  (flatten): Flatten()\n","  (fc): Linear(in_features=16384, out_features=10, bias=True)\n",")\n","Epoch 0, Iteration 0, loss = 2.2960\n","Checking accuracy on validation set\n","Got 136 / 1000 correct (13.60)\n","\n","Epoch 0, Iteration 100, loss = 1.6671\n","Checking accuracy on validation set\n","Got 382 / 1000 correct (38.20)\n","\n","Epoch 0, Iteration 200, loss = 1.4600\n","Checking accuracy on validation set\n","Got 480 / 1000 correct (48.00)\n","\n","Epoch 0, Iteration 300, loss = 1.7123\n","Checking accuracy on validation set\n","Got 481 / 1000 correct (48.10)\n","\n","Epoch 0, Iteration 400, loss = 1.6090\n","Checking accuracy on validation set\n","Got 490 / 1000 correct (49.00)\n","\n","Epoch 0, Iteration 500, loss = 1.4304\n","Checking accuracy on validation set\n","Got 499 / 1000 correct (49.90)\n","\n","Epoch 0, Iteration 600, loss = 1.5033\n","Checking accuracy on validation set\n","Got 544 / 1000 correct (54.40)\n","\n","Epoch 0, Iteration 700, loss = 1.3414\n","Checking accuracy on validation set\n","Got 542 / 1000 correct (54.20)\n","\n","Epoch 0, Iteration 765, loss = 1.3172\n","Checking accuracy on validation set\n","Got 538 / 1000 correct (53.80)\n","\n"]}],"source":["fix_random_seed(0)\n","\n","C, H, W = 3, 32, 32\n","num_classes = 10\n","\n","channel_1 = 32\n","channel_2 = 16\n","kernel_size_1 = 5\n","pad_size_1 = 2\n","kernel_size_2 = 3\n","pad_size_2 = 1\n","\n","learning_rate = 1e-2\n","momentum = 0.5\n","\n","model = None\n","optimizer = None\n","################################################################################\n","# TODO: Rewrite the 3-layer ConvNet with bias from Part III with the           #\n","# Sequential API.                                                              #\n","################################################################################\n","# Replace \"pass\" statement with your code\n","model = nn.Sequential(OrderedDict([\n","  ('conv1', nn.Conv2d(C, channel_1, kernel_size_1, padding=pad_size_1)),\n","  ('relu1', nn.ReLU()),\n","  ('conv2', nn.Conv2d(channel_1, channel_2, kernel_size_2, padding=pad_size_2)),\n","  ('relu2', nn.ReLU()),\n","  ('flatten', Flatten()),\n","  ('fc', nn.Linear(channel_2*H*W, num_classes)),\n","]))\n","optimizer = optim.SGD(model.parameters(), lr=learning_rate, \n","                      weight_decay=weight_decay,\n","                      momentum=momentum, nesterov=True)\n","################################################################################\n","#                                 END OF YOUR CODE                             #\n","################################################################################\n","\n","print('Architecture:')\n","print(model) # printing `nn.Module` shows the architecture of the module.\n","\n","_ = train_part345(model, optimizer)"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"CM6FuhGPIMCO"},"source":["# Part V. ResNet for CIFAR-10\n","\n","In this section, you are going to implement [ResNet](https://arxiv.org/abs/1512.03385), one of the state-of-the-art CNN architecture.\n","Specifically, you are going to implement a variation of ResNet called [PreResNet](https://arxiv.org/abs/1603.05027), which locates activation before each convolutional layer (so called pre-activation).\n","You are going to first implement a plain building block, residual block, and then bottleneck block for really deep networks.\n","Finally, you will implement your own ResNet using those blocks.\n","\n","Throughout this part, we will follow the PyTorch default weight initialization for conciseness."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"f8pgYlKN9NLH"},"source":["## Plain block\n","\n","First, let's implement a plain block without residual connection.\n","PreResNet has a different order of layers from the previously implemented ones;\n","BatchNorm and ReLU precedes Conv.\n","The name of PreResNet comes form this pre-activation architecture.\n","Here, for downsampling, we don't introduce MaxPool layers explicitly, but use stride 2 in the first Conv layer in the block.\n","\n","Concretely, a plain block accepts a feature map of shape $C_{in} \\times H_{in} \\times W_{out}$ and produces a feature map of shape $C_{out} \\times H_{out} \\times W_{out}$. If the block performs downsampling, then $W_{out}=W_{in}/2$ and $H_{out}=H_{in}/2$; otherwise $H_{out}=H_{in}$ and $W_{out}=W_{in}$. The plain block consists of the following six layers in order:\n","\n","1. Spatial Batch normalization\n","2. ReLU\n","3. Convolutional layer with `Cout` 3x3 filters, zero-padding of 1, and stride 2 if downsampling; otherwise stride 1\n","4. Spatial Batch normalization\n","5. ReLU\n","6. Convolutional layer with `Cout` 3x3 filters, with zero-padding of 1\n"]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"-c4QBBj5-A3R"},"outputs":[],"source":["class PlainBlock(nn.Module):\n","  def __init__(self, Cin, Cout, downsample=False):\n","    super().__init__()\n","\n","    self.net = None\n","    ############################################################################\n","    # TODO: Implement plain block.                                             #\n","    # Hint: Wrap your layers by nn.Sequential() to output a single module.     #\n","    #       You don't have use OrderedDict.                                    #\n","    # Inputs:                                                                  #\n","    # - Cin: number of input channels                                          #\n","    # - Cout: number of output channels                                        #\n","    # - downsample: add downsampling (a conv with stride=2) if True            #\n","    # Store the result in self.net.                                            #\n","    ############################################################################\n","    # Replace \"pass\" statement with your code\n","    stride = 2 if downsample else 1\n","    self.net = nn.Sequential(\n","      nn.BatchNorm2d(Cin),\n","      nn.ReLU(),\n","      nn.Conv2d(Cin, Cout, 3, stride=stride, padding=1),\n","      nn.BatchNorm2d(Cout),\n","      nn.ReLU(),\n","      nn.Conv2d(Cout, Cout, 3, padding=1)  \n","    )\n","    # optimizer = optim.SGD(model.parameters(), lr=learning_rate, \n","                          # weight_decay=weight_decay,\n","                          # momentum=momentum, nesterov=True)\n","    ############################################################################\n","    #                                 END OF YOUR CODE                         #\n","    ############################################################################\n","\n","  def forward(self, x):\n","    return self.net(x)"]},{"cell_type":"code","execution_count":24,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":50},"colab_type":"code","executionInfo":{"elapsed":113013,"status":"ok","timestamp":1573619142691,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"SofEF-vyAekS","outputId":"4d87876a-4ea2-44b2-f5b2-18cd94df73a8"},"outputs":[{"name":"stdout","output_type":"stream","text":["The output of PlainBlock without downsampling has a *correct* dimension!\n","The output of PlainBlock with downsampling has a *correct* dimension!\n"]}],"source":["data = torch.zeros(2, 3, 5, 6)\n","model = PlainBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of PlainBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of PlainBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","model = PlainBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of PlainBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of PlainBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"ASV5NdNADo3F"},"source":["## Residual block\n","\n","Next, let's implement a residual block.\n","A residual block adds a residual connection to a plain block. Let $\\mathcal{F}$ be a plain block; then the residual version  $\\mathcal{R}$ of the plain block $\\mathcal{F}$ computes:\n","\n","$\\mathcal{R}(x) = \\mathcal{F}(x) + x$\n","\n","However, this implementation will only work if the output from the plain block $\\mathcal{F}(x)$ has the same shape as the input $x$. Based on the plain block that we implemented above, there are two cases where the output of the plain block can have a different shape than the input:\n","\n","1. The number of output channels $C_{out}$ is different from the number of input channels $C_{in}$\n","2. The plain block $\\mathcal{F}$ performs spatial downsampling\n","\n","To deal with these cases, we need generalize our definition of the residual block and add a *shortcut connection* $\\mathcal{G}$:\n","\n","$\\mathcal{R}(x) = \\mathcal{F}(x) + \\mathcal{G}(x)$\n","\n","There are three cases for the shortcut connection $\\mathcal{G}$:\n","\n","1. If $C_{in}=C_{out}$ and $\\mathcal{F}$ does not perform downsampling, then $\\mathcal{F}(x)$ will have the same shape as $x$, so $\\mathcal{G}$ is the identity function: $\\mathcal{G}(x) = x$\n","2. If $C_{in} \\neq C_{out}$ and $\\mathcal{F}$ does not downsample, then $\\mathcal{G}$ is a 1x1 convolution with $C_out$ filters and stride 1.\n","3. If $\\mathcal{F}$ downsamples, then $\\mathcal{G}$ is a 1x1 convolution with $C_{out}$ filters and stride 2.\n","\n","In the code below, implement a residual block using the plain block we just defined:"]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"jzqJCUx6Do3I"},"outputs":[],"source":["class ResidualBlock(nn.Module):\n","  def __init__(self, Cin, Cout, downsample=False):\n","    super().__init__()\n","\n","    self.block = None # F\n","    self.shortcut = None # G\n","    ############################################################################\n","    # TODO: Implement residual block using plain block. Hint: nn.Identity()    #\n","    # Inputs:                                                                  #\n","    # - Cin: number of input channels                                          #\n","    # - Cout: number of output channels                                        #\n","    # - downsample: add downsampling (a conv with stride=2) if True            #\n","    # Store the main block in self.block and the shortcut in self.shortcut.    #\n","    ############################################################################\n","    # Replace \"pass\" statement with your code\n","\n","    if not downsample:\n","      if Cin == Cout:\n","        self.shortcut = nn.Sequential(\n","            nn.Identity()\n","        )\n","      else:\n","        self.shortcut = nn.Sequential(\n","            nn.Conv2d(Cin, Cout, 1)\n","        )\n","    else:\n","      self.shortcut = nn.Sequential(\n","            nn.Conv2d(Cin, Cout, 1, stride=2)\n","        )\n","    self.block = PlainBlock(Cin, Cout, downsample=downsample)\n","    ############################################################################\n","    #                                 END OF YOUR CODE                         #\n","    ############################################################################\n","  \n","  def forward(self, x):\n","    return self.block(x) + self.shortcut(x)"]},{"cell_type":"code","execution_count":26,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":50},"colab_type":"code","executionInfo":{"elapsed":113124,"status":"ok","timestamp":1573619142808,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"TMJ3-eI3Do3M","outputId":"4a33a626-142f-423c-a69e-cc3669069d5a"},"outputs":[{"name":"stdout","output_type":"stream","text":["The output of ResidualBlock without downsampling has a *correct* dimension!\n","The output of ResidualBlock with downsampling has a *correct* dimension!\n"]}],"source":["data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResidualBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of ResidualBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"wrEzQeTBvNso"},"source":["## Residual stage\n","\n","So far, you implemented micro layers, which consists of several convolutional laters.\n","To efficiently build a deep neural network, we define a macro layer by repeating the micro layers.\n","\n","For your convenience, we provide the implementation below."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"NhgWE6qTvVW6"},"outputs":[],"source":["class ResNetStage(nn.Module):\n","  def __init__(self, Cin, Cout, num_blocks, downsample=True,\n","               block=ResidualBlock):\n","    super().__init__()\n","    blocks = [block(Cin, Cout, downsample)]\n","    for _ in range(num_blocks - 1):\n","      blocks.append(block(Cout, Cout))\n","    self.net = nn.Sequential(*blocks)\n","  \n","  def forward(self, x):\n","    return self.net(x)"]},{"cell_type":"code","execution_count":28,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"colab_type":"code","executionInfo":{"elapsed":113121,"status":"ok","timestamp":1573619142811,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"I21i5J3AnbhM","outputId":"0420b090-5d87-4d44-f518-215c74dbf27a"},"outputs":[{"name":"stdout","output_type":"stream","text":["Plain block stage:\n","ResNetStage(\n","  (net): Sequential(\n","    (0): PlainBlock(\n","      (net): Sequential(\n","        (0): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (1): ReLU()\n","        (2): Conv2d(3, 4, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n","        (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (4): ReLU()\n","        (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","      )\n","    )\n","    (1): PlainBlock(\n","      (net): Sequential(\n","        (0): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (1): ReLU()\n","        (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","        (4): ReLU()\n","        (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","      )\n","    )\n","  )\n",")\n","Residual block stage:\n","ResNetStage(\n","  (net): Sequential(\n","    (0): ResidualBlock(\n","      (shortcut): Sequential(\n","        (0): Conv2d(3, 4, kernel_size=(1, 1), stride=(2, 2))\n","      )\n","      (block): PlainBlock(\n","        (net): Sequential(\n","          (0): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (1): ReLU()\n","          (2): Conv2d(3, 4, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n","          (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (4): ReLU()\n","          (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        )\n","      )\n","    )\n","    (1): ResidualBlock(\n","      (shortcut): Sequential(\n","        (0): Identity()\n","      )\n","      (block): PlainBlock(\n","        (net): Sequential(\n","          (0): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (1): ReLU()\n","          (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","          (3): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","          (4): ReLU()\n","          (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        )\n","      )\n","    )\n","  )\n",")\n"]}],"source":["print('Plain block stage:')\n","print(ResNetStage(3, 4, 2, block=PlainBlock))\n","print('Residual block stage:')\n","print(ResNetStage(3, 4, 2, block=ResidualBlock))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"7l3-PNR9mlIb"},"source":["## Residual stem\n","\n","A \"stem\" layer is required at the beginning of the network, which increases the number of channels while keeping the other dimensions.\n","\n","For your convenience, we provide the implementation below."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"eUvbubymmlIc"},"outputs":[],"source":["class ResNetStem(nn.Module):\n","  def __init__(self, Cin=3, Cout=8):\n","    super().__init__()\n","    layers = [\n","        nn.Conv2d(Cin, Cout, kernel_size=3, padding=1, stride=1),\n","        nn.ReLU(),\n","    ]\n","    self.net = nn.Sequential(*layers)\n","    \n","  def forward(self, x):\n","    return self.net(x)"]},{"cell_type":"code","execution_count":30,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":34},"colab_type":"code","executionInfo":{"elapsed":113117,"status":"ok","timestamp":1573619142813,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"UGzh0oVxm2Aw","outputId":"3badc15e-166a-45c6-f3a3-7ee44ec64176"},"outputs":[{"name":"stdout","output_type":"stream","text":["The output of ResidualBlock without downsampling has a *correct* dimension!\n"]}],"source":["data = torch.zeros(2, 3, 5, 6)\n","model = ResNetStem(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResidualBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"QGzn4Gp_sYBp"},"source":["## ResNet class\n","\n","Now, it is time to design the ResNet class using the blocks you implemented above!\n","\n","For general applicability, the class will get a dictionary of the architecture specification as an input, and parse it to build a CNN.\n","\n","Here we provide a couple of examples of specification;\n","`networks` is a collection of pre-defined network specifications, where each can be called by `get_resnet(key)`, where key is the name of the network, e.g., `get_resnet('resnet32')` will return a ResNet with 32 layers.\n","\n","Each specification consists of multiple tuples which correspond to a macro block (`ResNetStage`), and the values in each tuple implies `(num_in_channels, num_out_channels, num_blocks, do_downsample)`.\n","\n","To avoid dependency on the size of the input, ResNet has an average pooling at the end of the convolutional part, such that the size of the input tensor to the linear layer is always `(batch_size, stage_args[-1][1])`.\n","You may want to add an average pooling layer (`nn.AvgPool2d`), but it requires to know the size of the input.\n","Can you relax this requirement?\n","\n","**Hint**: You can perform average pooling in `forward`."]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"1iOOBoSgs-0X"},"outputs":[],"source":["# example of specifications\n","networks = {\n","  'plain32': {\n","    'block': PlainBlock,\n","    'stage_args': [\n","      (8, 8, 5, False),\n","      (8, 16, 5, True),\n","      (16, 32, 5, True),\n","    ]\n","  },\n","  'resnet32': {\n","    'block': ResidualBlock,\n","    'stage_args': [\n","      (8, 8, 5, False),\n","      (8, 16, 5, True),\n","      (16, 32, 5, True),\n","    ]\n","  },\n","}"]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"nVSeEzB7scmW"},"outputs":[],"source":["class ResNet(nn.Module):\n","  def __init__(self, stage_args, Cin=3, block=ResidualBlock, num_classes=10):\n","    super().__init__()\n","\n","    self.cnn = None\n","    ############################################################################\n","    # TODO: Implement the convolutional part of ResNet using ResNetStem,       #\n","    #       ResNetStage, and wrap the modules by nn.Sequential.                #\n","    # Store the model in self.cnn.                                             #\n","    ############################################################################\n","    # Replace \"pass\" statement with your code\n","    blocks = [ResNetStem(Cin, stage_args[0][0])]\n","    for arg in stage_args:\n","      blocks.append(ResNetStage(*arg, block=block))\n","    self.cnn = nn.Sequential(*blocks)\n","    ############################################################################\n","    #                                 END OF YOUR CODE                         #\n","    ############################################################################\n","    self.fc = nn.Linear(stage_args[-1][1], num_classes)\n","  \n","  def forward(self, x):\n","    scores = None\n","    ############################################################################\n","    # TODO: Implement the forward function of ResNet.                          #\n","    # Store the output in `scores`.                                            #\n","    ############################################################################\n","    # Replace \"pass\" statement with your code\n","    out = self.cnn(x)\n","    # print(out.shape)\n","    pool = nn.AvgPool2d((out.shape[2], out.shape[3]))\n","    out = pool(out)\n","    # print(out.shape)\n","    out = flatten(out)\n","    # print(out.shape)\n","    scores = self.fc(out)\n","    ############################################################################\n","    #                                 END OF YOUR CODE                         #\n","    ############################################################################\n","    return scores\n","\n","def get_resnet(name):\n","  return ResNet(**networks[name])"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"uqhp641H0P9Z"},"source":["## Train your model!\n","\n","Now let's train a few epochs of plain and residual networks with 32 layers on CIFAR.\n","You will see that deep non-residual networks don't converge well.\n","\n","**Caution: This takes a long time!**\n","\n","**Disclaimer: The performance of PreResNet-32 you will see here (~ 80%) would be lower than the best performance this model can achieve, because the convergence requires much more training.**"]},{"cell_type":"code","execution_count":33,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"colab_type":"code","executionInfo":{"elapsed":1199247,"status":"ok","timestamp":1573620228951,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"hRyYVBn60A58","outputId":"a1cd2b7e-1fa4-4491-958b-864f5f13a009"},"outputs":[{"name":"stdout","output_type":"stream","text":["plain32 \n","\n","Epoch 0, Iteration 765, loss = 1.5723\n","Checking accuracy on validation set\n","Got 335 / 1000 correct (33.50)\n","\n","Epoch 1, Iteration 1531, loss = 1.5932\n","Checking accuracy on validation set\n","Got 455 / 1000 correct (45.50)\n","\n","Epoch 2, Iteration 2297, loss = 1.4373\n","Checking accuracy on validation set\n","Got 466 / 1000 correct (46.60)\n","\n","Epoch 3, Iteration 3063, loss = 1.1158\n","Checking accuracy on validation set\n","Got 522 / 1000 correct (52.20)\n","\n","Epoch 4, Iteration 3829, loss = 1.3256\n","Checking accuracy on validation set\n","Got 572 / 1000 correct (57.20)\n","\n","Epoch 5, Iteration 4595, loss = 1.0727\n","Checking accuracy on validation set\n","Got 604 / 1000 correct (60.40)\n","\n","lr decay from 0.01 to 0.001\n","Epoch 6, Iteration 5361, loss = 0.6883\n","Checking accuracy on validation set\n","Got 704 / 1000 correct (70.40)\n","\n","Epoch 7, Iteration 6127, loss = 0.7886\n","Checking accuracy on validation set\n","Got 713 / 1000 correct (71.30)\n","\n","lr decay from 0.001 to 0.0001\n","Epoch 8, Iteration 6893, loss = 0.9123\n","Checking accuracy on validation set\n","Got 714 / 1000 correct (71.40)\n","\n","Epoch 9, Iteration 7659, loss = 0.9030\n","Checking accuracy on validation set\n","Got 712 / 1000 correct (71.20)\n","\n","resnet32 \n","\n","Epoch 0, Iteration 765, loss = 1.0867\n","Checking accuracy on validation set\n","Got 531 / 1000 correct (53.10)\n","\n","Epoch 1, Iteration 1531, loss = 1.2877\n","Checking accuracy on validation set\n","Got 589 / 1000 correct (58.90)\n","\n","Epoch 2, Iteration 2297, loss = 1.0020\n","Checking accuracy on validation set\n","Got 622 / 1000 correct (62.20)\n","\n","Epoch 3, Iteration 3063, loss = 0.5386\n","Checking accuracy on validation set\n","Got 728 / 1000 correct (72.80)\n","\n","Epoch 4, Iteration 3829, loss = 0.6438\n","Checking accuracy on validation set\n","Got 698 / 1000 correct (69.80)\n","\n","Epoch 5, Iteration 4595, loss = 0.6657\n","Checking accuracy on validation set\n","Got 746 / 1000 correct (74.60)\n","\n","lr decay from 0.01 to 0.001\n","Epoch 6, Iteration 5361, loss = 0.5477\n","Checking accuracy on validation set\n","Got 815 / 1000 correct (81.50)\n","\n","Epoch 7, Iteration 6127, loss = 0.4939\n","Checking accuracy on validation set\n","Got 805 / 1000 correct (80.50)\n","\n","lr decay from 0.001 to 0.0001\n","Epoch 8, Iteration 6893, loss = 0.7457\n","Checking accuracy on validation set\n","Got 808 / 1000 correct (80.80)\n","\n","Epoch 9, Iteration 7659, loss = 0.4702\n","Checking accuracy on validation set\n","Got 807 / 1000 correct (80.70)\n","\n"]}],"source":["# def init_module(model):\n","#   for m in model.modules():\n","#     if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):\n","#       nn.init.kaiming_normal_(m.weight.data)\n","#       if m.bias is not None: nn.init.zeros_(m.bias.data)\n","#     elif isinstance(m, nn.BatchNorm2d):\n","#       nn.init.ones_(m.weight.data)\n","#       if m.bias is not None: nn.init.zeros_(m.bias.data)\n","\n","names = ['plain32', 'resnet32']\n","acc_history_dict = {}\n","iter_history_dict = {}\n","for name in names:\n","  fix_random_seed(0)\n","  print(name, '\\n')\n","  model = get_resnet(name)\n","#   init_module(model)\n","  \n","  optimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=.9, weight_decay=1e-4)\n","\n","  acc_history, iter_history = train_part345(model, optimizer, epochs=10, schedule=[6, 8], verbose=False)\n","  acc_history_dict[name] = acc_history\n","  iter_history_dict[name] = iter_history"]},{"cell_type":"code","execution_count":34,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":295},"colab_type":"code","executionInfo":{"elapsed":1199560,"status":"ok","timestamp":1573620229269,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"-u89CIFfzWWR","outputId":"5d395546-0f50-4ce3-d1ac-23e9e83859e7"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjUAAAEWCAYAAACexWadAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hVVdr+8e+TkJCEkkACSO9SRcHQxLGOYkV0ULE3bKOjM46ozM9XHWec0cFXZ3zVUXTsKDJWrNgbKBCkNylSEpASSIAkpJ31+2PvhBASCJCTk3PO/bmuXDm7nJ1nb0hyZ+211zLnHCIiIiLhLibUBYiIiIjUBoUaERERiQgKNSIiIhIRFGpEREQkIijUiIiISERQqBEREZGIoFAjIjVmZp3MzJlZg1DXUtvM7CMzuyLUdYjIwVOoEYkiZvaxmd1fxfpzzOyXSAwrNeWcO90592Ko6xCRg6dQIxJdXgQuNTOrtP4yYKJzriQENdUK8+hnmkgU0w8AkejyDpAK/KpshZk1A84CXvKXzzSzOWa23czWmdl9NT24md1lZivNbIeZLTazcyttv9bMllTYPsBf397M3jKzzWaWbWaP++vvM7NXKrx/j9tfZvaVmT1gZtOAfKCLmV1V4WusMrPrK9VwjpnN9c9vpZmdVuFYYyrsd7V/nG1mNtXMOvrrzcweNbNN/jEWmFnfml4jEQkehRqRKOKcKwAmA5dXWH0BsNQ5N89fzvO3pwBnAjea2cgafomVeIEpGfgz8IqZtQYws/OB+/xjNwVGANlmFgu8D6wBOgFtgUkHcFqXAdcBTfxjbMILaU2Bq4BHK4SnQXjhbax/fscBqysf0MzOAf4EnAe0AL4FXvM3n+q/73D/PC8Asg+gXhEJEoUakejzIjDKzBL85cv9dQA4575yzi1wzgWcc/PxfpkfX5MDO+f+65xb77/3dWA5MMjfPAb4h3NulvOscM6t8be3AcY65/Kcc7ucc98dwPm84Jxb5Jwrcc4VO+c+cM6t9L/G18An7G6ZugZ4zjn3qV9jlnNuaRXHvAH4u3NuiX9L7m/AUX5rTTFegOoJmL/PhgOoV0SCRKFGJMr4gWELMNLMuuKFilfLtpvZYDP70r8VlIv3Cz6tJsc2s8v9Wzs5ZpYD9K3w3vZ4LTmVtQfWHEJ/nnWVajjdzH4ws61+DWfUoIbKOgL/qnAeWwED2jrnvgAeB54ANpnZBDNrepC1i0gtUqgRiU4v4bXQXApMdc5trLDtVWAK0N45lww8hfcLfZ/8VoxngJuBVOdcCrCwwnvXAV2reOs6oEM1T17lAUkVlg+rYh9XoYaGwJvAw0Arv4YPa1BDVTVd75xLqfCR6JybDuCce8w5dzTQG+821NgaHFNEgkyhRiQ6vQT8GriWCreefE2Arc65XX4flItreMxGeAFjM4CZXYXXUlPmWeB2Mzva72zbzQ9CM4ENwINm1sjMEsxsmP+eucBxZtbBzJKBcfupIR5o6NdQYman4/WBKfMf4CozO9nMYsysrZn1rOI4TwHjzKyPfy7Jfp8gzGyg35oVhxe6dgGBGl0hEQkqhRqRKOScWw1MxwsiUypt/i1wv5ntAO7B61hck2MuBv4X+B7YCBwBTKuw/b/AA3gtQTvwnsRq7pwrBc4GugFrgUzgQv89nwKvA/OB2XgdivdVww7gFr/mbXiBbEqF7TPxOw8DucDXeLeaKh/nbeAhYJKZbcdrcTrd39wUr0VqG17H5Gxg/H4vkIgEnTnn9r+XiIiISD2nlhoRERGJCAo1IiIiEhEUakRERCQiKNSIiIhIRAi7GXnT0tJcp06dQl2GiIiI1ILZs2dvcc61qI1jhV2o6dSpExkZGaEuQ0RERGqBma2prWPp9pOIiIhEBIUaERERiQgKNSIiIhIRwq5PTVWKi4vJzMxk165doS4lLCUkJNCuXTvi4uJCXYqIiMhBi4hQk5mZSZMmTejUqRNm+51MWCpwzpGdnU1mZiadO3cOdTkiIiIHLSJuP+3atYvU1FQFmoNgZqSmpqqVSyRczZ8Mj/aF+1K8z/NrNP+oSESKiJYaQIHmEOjaiYSp+ZPhvVuguMBbzl3nLQP0uyB0dYmESES01IiIRKXP798daMoUF8DUP8GmJbBzM5SWhKY2kRCImJaa+uqEE07g4YcfJj09vdp9xowZw2233Ubv3r2r3eepp57iiSeeIDY2lsaNGzNhwgR69+7Np59+yl133UVRURHx8fGMHz+ek046KRinIiL1TW5m1evzNsOTQ/wFg8QUSEqDRmmQlOp9NEqrsK55hddpEJdQZ6cQseZP9kJnbiYkt4OT71HrWR2IylDzzpwsxk9dxvqcAtqkJDJ2eA9G9m8bsnqeffbZ/e5z8cUXc8MNNwAwZcoUbrvtNj7++GPS0tJ47733aNOmDQsXLmT48OFkZWUFu2QRCaXCnTDtn9Vvb9QCTv8H5GdD3hbI3+J/zoatq2DdTO+1K636/XGNoFHqnkEnqXmlIFQhHDVsCrVxGztSgkAk3RYMs3+TqAs178zJYtxbCygo9r6Zs3IKGPfWAoBDCjarV6/mtNNO4+ijj+bHH3+kT58+vPTSS3vsc+ONNzJr1iwKCgoYNWoUf/7zn4E9W3MaN27Mrbfeyvvvv09iYiLvvvsurVq1omnTpuXHycvLK+8H079///L1ffr0oaCggMLCQho2bHjQ5yIi9VSgFOa8Al8+ADs3QrtB8Mt8KKnQ0T8uEYb/Dfqet59jBWBXDuRvrRB6yj5XWLdzI2xc7C2XVPNAQUxc1a1ASamVwlHq7oAUE7vnMeo6CARKoaQQSougtNj/XFjhdRGUFFXaXvmjuIpjFMHsF6q+Lfj+72Ht92AxYLH+5xgvEJa/ru7DvGu2r+17rTvEr7HqK5j2mHdd6uLfpBYENdSY2WnAv4BY4Fnn3IOVtncAXgRS/H3ucs59eChf88/vLWLx+u3Vbp+zNoei0sAe6wqKS7njjfm8NnNtle/p3aYp957dZ79fe9myZfznP/9h2LBhXH311Tz55JN7bH/ggQdo3rw5paWlnHzyycyfP59+/frtsU9eXh5DhgzhgQce4I477uCZZ57h7rvvBuCJJ57gkUceoaioiC+++GKvr//mm28yYMAABRqRSLTyC5h6N2xaBO0Hw+hXoV36wf8lHRPj33ZqDnSrWQ1FebvDT/7WSkGowrr1cyAvGwpzqzmQQWKzPYPQqi+rDgIf3A7ZK/YMDiWVwkdp8T4CSTXbXaDq0g5FTBzExkNxXtXbi/JgyXteoHIBcM7/XM0HrvZrPFTFBd7/t2gLNWYWCzwBnAJkArPMbIpzbnGF3e4GJjvn/m1mvYEPgU7BqgnYK9Dsb/2BaN++PcOGDQPg0ksv5bHHHttj++TJk5kwYQIlJSVs2LCBxYsX7xVq4uPjOeusswA4+uij+fTTT8u33XTTTdx00028+uqr/PWvf+XFF18s37Zo0SLuvPNOPvnkk0M+DxGpRzYthU/uhhWfQkpHOP9F6H3O7ts9/S6ou18w8Y28j2Yda7Z/SREUVA4/2ZVui2V7oaWomiBQmAtfP7Q7MMT6nxs03P06Ng5iG/rrE7zbYdVtr/L9FT/2t93fp0Gl48XG7/43ebSv16pRWXJ7+MPCml9v5yoEn9Jqwk81wShQuv999tpe6Wu8OIIqg1V1fbnqgWC21AwCVjjnVgGY2STgHKBiqHFA2X2VZGD9oX7R/bWoDHvwC7JyCvZa3zYlkdevH3pIX7vyo9EVl3/++WcefvhhZs2aRbNmzbjyyiurHBsmLi6u/H2xsbGUlOz95MLo0aO58cYby5czMzM599xzeemll+jateshnYOI1BM7N8NXf4PZL0J8Yzj1rzDoOu+XabhoEA9NDvM+9qfaINAObl3gtSyFi5Pv2fNWGni3BU++58COY+YHpRhC0lskuV31/yb1VDD/l7QFKl6NTH9dRfcBl5pZJl4rze+qOpCZXWdmGWaWsXnz5kMqauzwHiTG7XkvNzEulrHDexzScQHWrl3L999/D8Crr77KscceW75t+/btNGrUiOTkZDZu3MhHH310QMdevnx5+esPPviA7t27A5CTk8OZZ57Jgw8+WN5KJCJhrHgXfPsIPNYffnwJBo6BW+bAMb8Lr0BzoE6+x/vFX1FcIpx8b3gFGvBazs5+zGuZwbzPZz9Wb2/ZVKvaf5MDDGd1KNQdhS8CXnDO/a+ZDQVeNrO+zu15s9M5NwGYAJCenn5INxnLOgMH4+mnHj168MQTT3D11VfTu3dvbrzxRt577z0AjjzySPr370/Pnj33uE1VU48//jifffYZcXFxNGvWrPzW0+OPP86KFSu4//77uf/++wH45JNPaNmy5SGfj4jUIedg4Zvw2X3eX8c9zoBT7oe07qGurG6U/cIPoydt9qkubwsGSxj+m5hzwemI5IeU+5xzw/3lcQDOub9X2GcRcJpzbp2/vAoY4pzbVN1x09PTXUZGxh7rlixZQq9evWr/JA7A6tWrOeuss1i48ADul9Yj9eEaikSttT94A+ZlzYbD+sHwB6DzcaGuSqROmNls51z1g7kdgGC21MwCuptZZyALGA1cXGmftcDJwAtm1gtIAA7t/pKISLjYusprmVn8LjRpDSP/Df1Gh9/tFpF6ImihxjlXYmY3A1PxHtd+zjm3yMzuBzKcc1OAPwLPmNkf8DoNX+mC1XQUZJ06dQrbVhoRqWMF2+Cbh2HG097TMyf8CY652XuySEQOWlD71PhjznxYad09FV4vBtS7VUSiQ2kxZDwHX/0dCnKg/yVw4t3QtHWoKxOJCKHuKCwiEvmcg2Ufwqf3eGOydD7e6zdz2BGhrkwkoijUiIgE0/q53uB5q7+FtMPh4snQ/dTamStJRPagUCMiEgzb18Pnf4F5r3lTEZzxMBx9pdeHRkSCQl3sw1BOTs4e80qtWbOGAQMGcNRRR9GnTx+eeuopAPLz8znzzDPp2bMnffr04a677gpVySLRo3AnfPEAPDYAFr4Bw271Bs8bdK0CjUiQRWeomT/ZG5L7vhTv8/zJtXp45xyBQBAmS/NVDjWtW7fm+++/Z+7cucyYMYMHH3yQ9eu9GSduv/12li5dypw5c5g2bdoBj2QsIjUUKPVGAP6/AfDNP6DnGXBzBpzyZ0hIDnV1IlEh+kJN2fT2uesAt3sq9UMMNqtXr6ZHjx5cfvnl9O3bl5dffpmhQ4cyYMAAzj//fHbu3AnAXXfdRe/evenXrx+33347AFdeeSW33HILxxxzDF26dOGNN94oP+748eMZOHAg/fr149577y0/xsqVKznqqKMYO3Ys8fHx5TNzFxYWlgeqpKQkTjzxRMCbKHPAgAFkZtbfichEwtbKL+Hp42DK77xJJ6/5DEY9V/OJH0WkVkRen5qP7oJfFlS/PXOWNwV9RcUF8O7N3sRxVTnsCDj9wf1+6eXLl/Piiy/SrVs3zjvvPD777DMaNWrEQw89xCOPPMJNN93E22+/zdKlSzEzcnJyyt+7YcMGvvvuO5YuXcqIESMYNWoUn3zyCcuXL2fmzJk45xgxYgTffPMNDz74IAsXLmTu3Lnl71+3bh1nnnkmK1asYPz48bRp02aP2nJycnjvvfe49dZb93seIlJDm5bCp/8Dyz/xwsyo56HPueoELBIikRdq9qdyoNnf+gPQsWNHhgwZwvvvv8/ixYvL53cqKipi6NChJCcnk5CQwDXXXMNZZ53FWWedVf7ekSNHEhMTQ+/evdm4cSPgzeH0ySef0L9/fwB27tzJ8uXL6dChw15fu3379syfP5/169czcuRIRo0aRatWrQAoKSnhoosu4pZbbqFLly6HfJ4iUW/nZm+smdkveDNon/IXGHx9ZE84KRIGIi/U7K9Fpdrp7dvDVR8c0pdu1MgbDdQ5xymnnMJrr7221z4zZ87k888/54033uDxxx/niy++ACi/fVT2/rLP48aN4/rrr9/jGKtXr662hjZt2tC3b1++/fZbRo0aBcB1111H9+7d+f3vf39I5ycS9Yp3wYx/e7NoF+XBwGvg+LugUWqoKxMRorFPTR1MpT5kyBCmTZvGihUrAMjLy+Onn35i586d5ObmcsYZZ/Doo48yb968fR5n+PDhPPfcc+X9cbKysti0aRNNmjRhx44d5ftlZmZSUFAAwLZt2/juu+/o0aMHAHfffTe5ubn885//rLXzE4k6zsGCN+Dxgd5cTR2HwW9/gDPGK9CI1COR11KzP3UwlXqLFi144YUXuOiiiygs9G5r/fWvf6VJkyacc8457Nq1C+ccjzzyyD6Pc+qpp7JkyRKGDh0KQOPGjXnllVfo2rUrw4YNo2/fvpx++umceuqp/PGPf8TMcM5x++23c8QRR5CZmckDDzxAz549GTBgAAA333wzY8aMqbVzFYl4a2f4M2hneP3rzpkCXY4PdVUiUgULt/kj09PTXUZGxh7rlixZQq9evUJUUWTQNRSpZOvP/gza70Djw7w/fo4cDTGxoa5MJKKY2WznXHptHCv6WmpERPalIAe+9WfQjmkAJ4yDY36nGbRFwoBCjYgI+DNoP+/PoL0NjroETtIM2iLhJGJCjXMO09gQByXcbkGK1Ir5k3f3rUtKBYuFvI3Q+Tg49QFo3S/UFYrIAYqIUJOQkEB2djapqakKNgfIOUd2djYJCQmhLkWk7pSNLF7sPTVI/hbAYOjNcOpfNXieSJiKiFDTrl07MjMz2bx5c6hLCUsJCQm0a9cu1GWI1I3iXfDRnbsDTTkHi9+F4Q+EpCwROXQREWri4uLo3LlzqMsQkfrslwXehJPzJ8OunKr3ydXcaCLhLCJCjYhIlXbleoPmzXkZ1s+B2HjoNQJ+/hryqmjZTVaLpUg4U6gRkcjiHKz9Hn58GRa9DSUF0LIPnP4POOJ8SGq+d58aqPWRxUWk7inUiEhk2LkJ5r3m3WLKXgHxTbzB8gZcBm0G7Nn5tw5GFheRuqdQIxJsFR8d1i/P2hUohRWfw48vwk8fQ6AE2g+BY2+DPiP3PWBevwv07yASYRRqRIKp8m2O3HXeMugX6qHYthrmTIS5E2F7FiSlwZAbof/l0OLwUFcnIiGiUCMSTJ/fv/ejw8UF3nqFmgNTUghL3/duL636CjDo9ms47e9w+OnQID7UFYpIiCnUiARTdY8I566Dt2+Ejsd4H827aMC36mxc7D+KPcmbviC5A5zwJzjqYkhpH+rqRKQeUagRCZaiPGjQEEp27b2tQSIsnwrzXvWWm7TeHXA6HgstekR3yCncAQvf8sJMVgbExEGvs2DA5dD5BIiJCXWFIlIPKdSIBMOOX+DVC71AExMHgeLd2+IS4ezHvMeLt/wEq7+DNdNhzTRY+Ka3T1IqdBgKnY71gk6rvhATG5pzqSvOQeYsr9PvwrehOA9a9IThf4N+o6FRaqgrFJF6TqFGpLZtXAQTL/BulVz0OhRur/7ppxY9vI+B13i/1Lf97Aec6V7YWfq+t1/DZOgwxAs4nY6F1kdCbFzozrE25W2BeZO8AfI2L4W4RtD3PBhwBbRLj+4Wqyjzzpwsxk9dxvqcAtqkJDJ2eA9G9m8b6rIOSiSdSzixcJuhOT093WVkZIS6DJGqrfgMJl8JDRvDxa974eNQ5GbubsVZM91r2QGIS4L2g6DjMO+j7dEQF0aTkgYCsOpL7/bS0g+8lqx2A73bS33OhYZNQl2h1LF35mQx7q0FFBSXlq9LjIvl7+cdEXZhINLOJdjhzMxmO+fSa+VYCjUitSTjefjgj9CyF1w8GZKD8MNr56bdLTlrpnmtQjhv+P+26dBpmNea026QF6zqm5x13mPYc17xOksnNvcGyOt/GbTqHerqpA7tKi4lJ7+YnIIituUVc9PE2WzNL95rv8YNG3DJ4A4EnCPgIOAczv9cGvDWOeeq3L57ufK+u7eXvzdQ6dj72b7XvoHdx960YxeBKn61xsYYnVKTaBATQ4NYo0GMERtjNIiJ8T6Xr4vxPu+xj9EgNmaP5bL99nqfvy42xoircOw93lfV1y8/lrf85dJNPPzJMgpLAuXnEIxwplCjUCP1SSAAn98H0/4F3U6B85+vu5aG/K2wbsbufjkb5oErhZgG0Pqo3ber2g+GxJS6qamykiJY9qF3e2nF54CDLid6rTI9z/Q6U0vYKikNkFtQzLb8YnL9gLItv8hfV8S2/GJy8ovIyS/e43XFVoz9SYiLIcaMGDPM8F/jLxuxMexj+57bYmOsfLtV2G+PfSscr+LXiamw3Sptr/j69Yx11Z7Lmf1aU1IaoDTgKAl4Yauk1PtcHAjssVziLxeXLztKA4Hd7ws4SkoDVQaoYGqbksi0u06qtePVZqhRnxqRQ1FcAG9dB0umQPo13vxCsXX4bZXUHHqc7n2A99TQuhl+n5xp8MO/YfpjgMFhfb0nq8qesmqUFtzaNi/zbi/NmwT5W6BpWzj+DjjqEmjWMbhfO4rU1u0B5xw7CkvI8UNJToEXQLbllb321/vBZJu/vGNXSbXHjI0xUhLjSEmKIyUpnrYpCfRp05SUxDiaNYonJSmOZknxpCTG8fvX57JpR+Fex6jtX6B14bsVW8jKKdhrfduURJ64eECtf71AwGstKg8+pRUCkr+8d0CqFKz8gFRx+ZbX5lT59dZXcW71RVB/+prZacC/gFjgWefcg5W2Pwqc6C8mAS2dcyH6c1LkAO3cDK+NhqzZcOoDMPSm0HdqbdjEG5Cu26+95eICyMzw++RMg9kvwIx/e9ta9PQDjn/LqmmbQ//6RXneJJI/vuSFq5gGXuAacAV0PSnyn+CqY5X7bmTlFDDurQUUlwY4tnsa2/K82ztVBZLdrSf++oJiSvfxJ3+ThAZeAPEDSqe0RjRLiic5MY5mSV5I8V7He/s1iqNxfANiYmr2PfGnM3pV2Q9l7PAeh3aRQmDs8B51ei4xMUYMRlwtf3s99NHSKsNZm5TE2v1CtShot5/MLBb4CTgFyARmARc55xZXs//vgP7Ouav3dVzdfpJ6YfMymDjKCza/eQZ6nR3qimqmpAjWz9nd8XjtD1C0w9vWrLMXcMr65aR03DukVTWP1RHnQ9aP/qPYb3nHS+3u3V46cjQ0bln35xklhj34RZW/dPYlIS7GDyfxfotJ3O7XSRVaT/zw0iwpjuTEOBrEBn9soEh6YigSzqWuOjyHRZ8aMxsK3OecG+4vjwNwzv29mv2nA/c65z7d13EVaiTkVn0Nr1/m9QW5eJL35FG4Ki2BjQu8W1VrpsPa6d6j6ODdLiprxek4DDbM3XMeK/DG4GnUEnZkeQMK9j3P6/TbYUjoW62iQOe7PqC6n+B/O/cIL5AkVWg9SYojobb/nJeIFm5PPwXz9lNboGJvqUxgcFU7mllHoDPwRRDrETl0cyZ6v9hTu3lPOIV735DYBtCmv/dxzM1ep+fNS3Y/XbXqK1gw2dvXYsAF9nx/oBjyN8FZj0Lf30BCcp2fQrTK3llIXGwMRaWBvba1TUnk4sEdQlCVRJqR/duGVQtTfekoPBp4wzlXZXd4M7sOuA6gQwd9o0oIOAdfPgDfjIcuJ8D5L4buaaJgiomBVn28j0HXeuedvRLWfAfv3Vr1e0pLIH2fd42lli1an8t1L82mNBAgLtYoLt3dXhOu/VBEakMwb5JmARVnm2vnr6vKaOC16g7knJvgnEt3zqW3aNGiFksUqYHiXfDmGC/Q9L8MLnkjMgNNVcwgrRscfSUkVzN5ZHK7Oi0p2r0/fz2/+fd0As7x9k3DGD/qSNqmJGJ4LTThOMCbSG0JZkvNLKC7mXXGCzOjgYsr72RmPYFmwPdBrEXk4ORlw+uXwNrv4eR74dg/RG9fkZPv2btPTVyit16CLhBw/O+ny3jiy5Uc3bEZ/750AC2bJNCvXYpCjIgvaKHGOVdiZjcDU/Ee6X7OObfIzO4HMpxzU/xdRwOTXLiNAiiRL3ul94RTbhaMet7rBBvNyuarqm4eKwma7buK+cOkuXy+dBMXDWrPfSP60LCBOvyKVKYRhUWqsmY6TLrY6xw7+jXoUGUfd5GgW7V5J2NeymBtdj73jujDpYM7YNHaWigRKVyefhIJT/Mnw7s3eeO0XDIZmncJdUUSpb5ctolbXptDXGwMr4wZzJAuqaEuSaReU6gRKeOc1xn4ywe86QQufNmbhkCkjjnnePqbVTz08VJ6HdaUCZcfTbtmSaEuS6TeU6gRAW+k3fduhXmvQr/RMOL/oEF8qKuSKFRQVMqdb85nyrz1nNmvNeNH9SMpXj+qRWpC3ykiBdu8EYJXfwsnjIPj74zeJ5wkpLJyCrj+5QwWrd/O2OE9+O0JXdV/RuQAKNRIdNv6M0w8H7athnMnwJEXhroiiVIzf97KbyfOprA4wH+uSOeknq1CXZJI2FGokei1bia8dhEESuDyd72JHEVCYOKMNdz77iI6NE9iwnXpdGvZONQliYQlhRqJTovehreuh6Zt4JL/Qlr3UFckUaioJMCf31vExBlrOaFHC/41uj/JiXGhLkskbCnUSHRxDqb9Ez67D9oP9sagaaTHZKXubdlZyG9f+ZGZq7dy4wlduf3UHsTGqP+MyKFQqJHoUVoMH9wGP74Efc6Dkf+GuIRQVyVRaGFWLte9lMHW/CL+NfoozjlK0xyI1AaFGokOu3Jh8hWw6kv41R/hxLu9GalF6ti7c7O48835NE+K540bjqFv2+RQlyQSMRRqJPLlrIWJF0D2chjxOAy4LNQVSRQqDTjGT13GU1+vZFCn5jx56QDSGjcMdVkiEUWhRiJb1mx4dTSUFMKlb0KXE0JdkUSh3IJibp00h6+WbeaSwR249+w+xDdQS6FIbVOokci15H14cww0bgFXvActe4a6IolCKzbt5LqXMli7NZ8Hzu3LJYM7hrokkYilUCORxzn44UmY+v+g7QC4aBI0bhnqqiQKfb5kI7+fNJf4BjG8eu0QBnXWXGIiwaRQI5GltAQ+vhNmPQu9zvZGCY7XRIBSt5xzPPnVSh7+ZBl92jTl6cvSaZuSGOqyRCKeQo1EjsId8MbVsPwTOOZ38Ov79YST1Ln8ohLueGM+78/fwIgj2/DQb/qRGB8b6rJEomQ3n9MAACAASURBVIJCjUSG3Cx49ULYtBjOfAQGXhPqiiQKZW7L57qXZrPkl+2MO70n1x3XRRNSitQhhRoJfxvmeYGmcCdcPBm6/zrUFUkU+mFVNr+d+CPFpQGeu3IgJ/ZQPy6RuqZQI+Htp6nw36sgMQWu/hgO6xvqiiTKOOd45Yc1/Pm9xXRMTeKZy9Pp0kITUoqEgkKNhK+Zz8BHd0Crvl4LTdPWoa5IokxRSYB7pyzktZnrOKlnS/45+iiaJmhCSpFQUaiR8BMohU/u9h7bPvx0+M2z0FB/GUvd2ryjkBtfmU3Gmm3cdGJXbjtFE1KKhJpCjYSXojx481pY9gEMvgGG/w1i9GSJ1K35mTlc//JscvKLefzi/pzVr02oSxIRFGoknOz4xesQ/Mt8OO0hGHJDqCuSKPT2nEzuenMBaY0b8saNQ+nTRhNSitQXCjUSHjYu8ialLNgKo1+FHqeHuiKJMqUBx0MfL2XCN6sY3Lk5T14ygFRNSClSryjUSP00fzJ8fj/kZkKjNNi1HRKbwVUfQZujQl2dRJnc/GJ+N2kO3/y0mcuHduR/zupNXKwGdhSpbxRqpP6ZPxneuwWKC7zlvM2AwbF/UKCROrd84w6ufSmDrJwCHjzvCEYP6hDqkkSkGjX6U8PM3jKzM81Mf5pIcBXuhI/H7Q405Rx8/3hISpLo9enijZz75HR2Fpby2rVDFGhE6rmattQ8CVwFPGZm/wWed84tC15ZEjVKS2D9HFj1Jaz8EjJnQqCk6n1zM+u2Nolazjke/2IF//vpT/Rrl8zTlx1N62RNSClS39Uo1DjnPgM+M7Nk4CL/9TrgGeAV51xxEGuUSOIcZK/0Qsyqr+Dnb6EwFzBo3Q+G3gRzX/VvOVWS3K6uq5UolFdYwtg35vHhgl84t39b/n7eESTEadgAkXBQ4z41ZpYKXApcBswBJgLHAlcAJwSjOIkQeVu8ALPqS1j1NeSu89Ynd4A+50CXE6Hz8dAo1Vvfqu+efWoA4hLh5HvqvHSJLuu25nPtSxn8tHEH/++MXoz5VWdNSCkSRmoUaszsbaAH8DJwtnNug7/pdTPLCFZxEqaKC2DN9N1B5pcF3vqEZOh8HBz7ey/INO8CVf3C6HeB97ns6afkdl6gKVsvEgTTV2zhpld/pDTgeP6qQRx/eItQlyQiB8icc/vfyexE59yXdVDPfqWnp7uMDOWoeiVQ6g2It/JLL8SsnQGlhRATB+0HQ9cToMtJ3pNLGv1X6ol35mQxfuoy1ucU0DSxAdsLSujasjHPXp5Op7RGoS5PJGqY2WznXHptHKumt596m9kc51yOX0Az4CLn3JO1UYSEoW2r/RDzFfz8NRRs89a37AODroUuJ0DHYyBevxyk/nlnThbj3lpAQXEpALkFJcQYjDm2kwKNSBiraai51jn3RNmCc26bmV2L91RUtczsNOBfQCzwrHPuwSr2uQC4D3DAPOfcxTWsSepS/lZY/e3u1phtq731TVp7k0p29fvFNGkV0jJFauLBj5aUB5oyAQf/98VKRg/qGKKqRORQ1TTUxJqZOf9elZnFAvH7eoO/zxPAKUAmMMvMpjjnFlfYpzswDhjmB6WWB3MSEgQlhbBuxu7WmPVzAAfxjaHTr2DIb73WmLTDq+4XI1KPbM0r4vuV2UxfuYXpK7P5ZXthlfutz6k8PpKIhJOahpqP8ToFP+0vX++v25dBwArn3CoAM5sEnAMsrrDPtcATzrltAM65TTUtXGpZIACbFnkBZuWXXkffkgKwWGg3EI6/02uNaXs0xMaFulqRfdpZWMLMn7OZviKbaSuzWbJhOwCNGzZgcOfmbM0rJLdg7/GQ2qRoLBqRcFbTUHMnXpC50V/+FHh2P+9pC6yrsJwJDK60z+EAZjYN7xbVfc65vcKSmV0HXAfQoYNG9Kw1uVm7x4tZ9dXusWHSDocBl3stMZ2OhYSmoatRpAZ2FZfy49ptfL8ym2krtjAvM5fSgCO+QQzpHZtx+6mHc0y3NPq1TaZBbMxefWoAEuNiGTu8RwjPQkQOVU0H3wsA//Y/avvrd8cb56Yd8I2ZHVHWIbnC158ATADv6adariGyVJwIsvKj0Lu2e/1iylpjspd76xu19B6x7nKC95HcNiSli9RUSWmABVm5TPdvKWWs3kZhSYDYGKNfu2RuOL4Lw7qmMaBjsyoHzhvZ3/s/Xvb0U5uURMYO71G+XkTCU03HqekO/B3oDSSUrXfOddnH27KA9hWW2/nrKsoEZvgjEv9sZj/hhZxZNalLKqk8EWTuOnj3Zlj8rtcKk5kBrhQaJEKnYXD0lV6IadVH/WKkXnPOsWzjDqav8ELMjFVb2VHo3T7qeVgTLhnckWHdUhnUuTlNEmp2e3Rk/7YKMSIRpqa3n54H7gUeBU7Emwdqf5NbzgK6m1lnvDAzGqj8ZNM7eNMuPG9maXi3o1bVsCap7PP7954IsrQQlr7v9YUpG/Su/SBo0DA0NYrUgHOOtVvzme7fTvp+ZTbZeUUAdEpN4qwj2zCsWypDuqSS1lj/l0XEU9NQk+ic+9x/AmoNcJ+ZzQaqHbfeOVdiZjcDU/H6yzznnFtkZvcDGc65Kf62U81sMVAKjHXOZR/SGUWrnZt3Tz+wF4Nrv6jTckQO1Mbtu8r7xExfmU2W/yRSq6YNOe7wFhzTNZVjuqXRVp15RaQaNQ01hWYWAyz3g0oW0Hh/b3LOfQh8WGndPRVeO+A2/0MORs5amP5/8OPL1e+jiSClHsrJL+KHVVvLH7NesWknAMmJcQztksoNx3dhaNc0urZopPmXRKRGahpqbgWSgFuAv+DdgroiWEVJDWxaAt/9Exb81+sP0280tDgcvvq7JoKUeim/qIRZq7cx3W+JWbg+F+cgKT6WgZ2ac0F6O47pmkbv1k2JiVGIEZEDt99Q4w+id6Fz7nZgJ15/GgmVdbPgu0dg2YcQlwSDr4ehN+1ujWnSWhNBSr1QVBJg7rqc8j4xc9Zto7jUERdr9O/QjN+ffDjHdEvlyHYpxDfYXxc9EZH922+occ6VmtmxdVGMVMM5WPk5fPsorPkOElLg+Lu8QJPUfM99+12gECNBU3ESyMqPQZcGHIvXb2eafztp1s9bKSguJcagb9tkrjm2C8d0TWVgp+YkxmtiUxGpfTW9/TTHzKYA/wXyylY6594KSlXiCZR6j2N/96g3C3aTNjD8bzDgCmi43y5NIrWq8oB1WTkF3PnmfL5Zvpm8whJ+WLWV3IJiALq3bMyFA9sztGsqQzqnkpykUahFJPhqGmoSgGzgpArrHKBQEwwlhTDvNZj2L9i6ClK7wYjHvRYYPYotITJ+6rK9JoEsLAnw1o9ZtGuWyGl9DuOYbqkM7ZpKyyYJ1RxFRCR4ajqisPrR1IXCHZDxPHz/BOz8BVofBRe8BD3Pghg110vorNi0s/wR68oM+O7Ok6rcJiJSl2o6ovDzeC0ze3DOXV3rFUWjvC0w4ymYOQF25ULn4+Dcp7zRfvUoq4RIUUmAqYt+YeKMNfywamu1+2kSSBGpL2p6++n9Cq8TgHOB9bVfTpTJWeePMfOSNyN2z7Pg2Nug3dGhrkyi2NrsfF6btZbJs9aRnVdE++aJ3HlaTxonxPK3D5ZqEkgRqbdqevvpzYrLZvYa8F1QKooGm5bCNH+MGYB+F8KwW6GFfjlIaJSUBvh86SYmzljLt8s3E2PGyT1bcsmQjvyqW1r5uDFNGsZpEkgRqbdq2lJTWXegZW0WEhUyM+DbR2DZB94YMwOvhWNu1oi/EjIbcguYNHMdr89axy/bd3FY0wRuPbk7Fw5sT+vkvW8raRJIEanPatqnZgd79qn5BbgzKBVFGudg5RfeY9mrv/XHmLkTBl0PjVJDXZ1EoUDA8c3yzUycsZbPl2zEAccf3oK/jOzLiT1a0CBWA+GJSHiq6e2nJsEuJOIESmHJFC/MbJjnjfR76gNw9BXQUJdT6t7mHYX8d/Y6Xpu5lnVbC0hrHM8Nx3flokEdaN88KdTliYgcspq21JwLfOGcy/WXU4ATnHPvBLO4sFRSCPMm+WPMrITmXWHE/3n9ZjTGjNQx5xw/rNrKxBlrmLroF4pLHUO7pHLnaT05tfdhmp5ARCJKTfvU3Ouce7tswTmXY2b3Ago1ZQp3wOwXvDFmdmyA1kfC+S9Cr7M1xozUuZz8It6YncmrM9eyanMeyYlxXD60ExcN6kC3lhqNWkQiU01DTVV/zh1sJ+PIkpddYYyZHOj0Kxj5JHQ5UWPMSJ1yzvHj2hwmzljDB/M3UFgS4OiOzXjkgm6ccURrEuIUrkUkstU0mGSY2SPAE/7yTcDs4JQUJnLWwfePw+wXK4wx8wdolx7qyiTK7NhVzDtz1zPxhzUs/WUHjRs24IL09lw8uAO9WjcNdXkiInWmpqHmd8D/AK/jPQX1KV6wiT6bl8F3/4QFk73lIy7wxphp2TO0dUnUWZiVy8QZa3l3bhb5RaX0adOUv593BCOObEOjhmpIFZHoU9Onn/KAu4JcS/2WORu+ewSWvg8NEmHgGBh6E6R0CHVlEkUKikp5b/56Js5Yy7x1OSTExTDiyDZcMrgj/dolY7rlKSJRrKZPP30KnO+cy/GXmwGTnHPDg1lcyDkHq770Bsxb/S0kJMNxd8Dg66FRWqirkyiyfOMOJs5Yy5s/ZrJjVwndWzbmzyP6MLJ/W5IT40JdnohIvVDTNuq0skAD4JzbZmaRM6Lw/Mnw+f2Qm+mN7nvS3dAgwR9jZi40PgxO+QukX6UxZqTOFJaU8vHCX5j4w1pmrt5KfGwMZxxxGJcM6Uh6x2ZqlRERqaSmoSZgZh2cc2sBzKwTVczaHZbmT4b3boHiAm85dx28fQPgoHkXOPsxOHK0xpiROrN6Sx6vzVzLf2dnsjWviE6pSfzpjJ6MOro9zRvFh7o8EZF6q6ah5v8B35nZ14ABvwKuC1pVdenz+3cHmnIOklLh5gyNMSN1org0wOdLNvoTSm4hNsY4tXcrLhnckWO6ppZPKCkiItWraUfhj80sHS/IzMEbdK9yEghPuZlVr8/fqkAjQZeVU8DrM9cyadY6Nu0opE1yAn885XAuGNieVk0TQl2eiEhYqWlH4THArUA7YC4wBPgeOCl4pdWR5HbeLaeq1ovUgnfmZDF+6jLW5xTQJiWRP55yOM0axTNxxhq+WLoJB5zUoyWXDOnA8Ye3JFatMiIiB6Wmt59uBQYCPzjnTjSznsDfgldWHTr5nj371ADEJXrrRQ7RO3OyGPfWAgqKSwGvZeaP/52HA1o0achNJ3bjwoHtaddME0qKiByqmoaaXc65XWaGmTV0zi01sx5Brayu9LvA+1zx6aeT79m9XuQQjJ+6rDzQlHFA86Q4pt91EnGxmlBSRKS21DTUZPozc78DfGpm24A1wSurjvW7QCFGalUg4Pj6p81k5VTd9WxbfrECjYhILatpR+Fz/Zf3mdmXQDLwcdCqEglThSWlvDtnPc98u4rlm3YSa1BaxeAHbVIS6744EZEId8ATxDjnvg5GISLhLDe/mFdmrOGF6avZvKOQXq2b8s8Lj6I0EODudxbtcQsqMS6WscMj4+6tiEh9olnvRA7Buq35PDftZ16ftY78olKOO7wFj17QhWHdUstH/I2Nidnj6aexw3swsn/bEFcuIhJ5FGpEDsKCzFwmfLuKDxdswIARR7Xh2l91oVfrpnvtO7J/W4UYEZE6oFAjUkNlnX8nfLOK71dl06RhA8Yc25krh3WidbL6yIiIhFpQQ42ZnQb8C4gFnnXOPVhp+5XAeCDLX/W4c+7ZYNYkcqAKS0p5d+56nvnG6/zbOjmB/3dGL0YPak+TBM2QLSJSXwQt1JhZLPAEcAqQCcwysynOucWVdn3dOXdzsOoQOVi5+cVMnLmGF6atZpPf+ffRC4/krH5t9Di2iEg9FMyWmkHACufcKgAzmwScA1QONSL1Sua2fJ77bjWTZq0lv6iUX3VP45ELjtqj86+IiNQ/wQw1bYGKkyplAoOr2O83ZnYc8BPwB+fcXhMxmdl1+LOCd+jQIQilisDCrFye/qZC598j23DtcVV3/hURkfon1B2F3wNec84Vmtn1wItUMUmmc24CMAEgPT29iqHMRA6Oc46vftrMhK/V+VdEJNwFM9RkAe0rLLdjd4dgAJxz2RUWnwX+EcR6RMqVdf599ttV/LRxd+ffCwe1p6k6/4qIhKVghppZQHcz64wXZkYDF1fcwcxaO+c2+IsjgCVBrEdEnX9FRCJY0EKNc67EzG4GpuI90v2cc26Rmd0PZDjnpgC3mNkIoATYClwZrHokulXV+fd/LziSY7ulqfOviEiEMOfCq4tKenq6y8jICHUZEiaq6vw75ldd6N1GnX9FROoDM5vtnEuvjWOFuqOwSK2r3Pm3ccMGXHNsZ65S518RkYimUCMRo3Ln38OaJvCnM3oyelAHdf4VEYkCCjUS9nILipk4Y3fn356HNeHRC4/kzCPaEN9AnX9FRKKFQo2ErbLOv6/PWkueOv+KiEQ9hRoJOwuzcpnwzSo+UOdfERGpQKFG6qV35mQxfuoy1ucU0CYlkdtPPZyURvE8880qpq/c3fn3ymM60SZFnX9FREShRuqhd+ZkMe6tBRQUlwKQlVPAbZPn4UCdf0VEpFoKNRIypQHHtvwituwsZMuOIrLzCtm8o5B/fra8PNCUcUCzpDi+ueNEdf4VEZEqKdRIrSouDZC90w8qOwvZUvZ6R6XlnUVszSskcABjP+bkFyvQiIhItRRqIkzlvihjh/dgZP+2h3TMXcWluwNJeTipGFB2v87JL67yGIlxsaQ1iSe1UUPaNUuif4cU0ho3rPART1qThqQ1asgZj31DVs6uvY6hvjMiIrIvCjURpKq+KOPeWgCwR7BxzpFXVLpHQNm8s4jssoCyY8+wsrOwpMqv16RhAy+INI6ne8vGDO2SSmrj+PKg0qLJ7teNGtb8v9rY4T33OA/wQtHY4T0O5rKIiEiUUKiJIOOnLturL0pBcSl/ensB789fv0dw2VUcqPIYzZLiSPVbTvq2TfbDid+S4geUsuCSEBcblPMoC2C13eIkIiKRTaEmgqzPKahyfX5RKZnbCmjRpCFd0hqR1jjeDy67w0qLJg1p3iieuNj60WdlZP+2CjEiInJAFGoiSJOEBmzftfetorYpiXz8++NCUJGIiEjdqR9/lsshcc7xyKc/sX1XCbGVpgdQXxQREYkWCjVhzjnHQx8v47HPl3NBejvGj+pH25REDK+F5u/nHaHbOCIiEhV0+ymMOef4y/tLeG7az1wyuAN/OacvMTHGeUe3C3VpIiIidU6hJkwFAo57pyzi5R/WcOUxnbj37N6amVpERKKaQk0YCgQcf3p7AZNmreP647pw1+k9FWhERCTqKdSEmdKA44435vPmj5n87qRu3HbK4Qo0IiIiKNSElZLSALdNnseUeeu57ZTDueXk7qEuSUREpN5QqAkTRSUBbp00h48W/sKdp/XkxhO6hrokERGRekWhJgwUlpRy08Q5fLZkI3ef2Ysxv+oS6pJERETqHYWaem5XcSk3vDKbr5Zt5i/n9OGyoZ1CXZKIiEi9pFBTjxUUlXLtSxlMW7mFB887gtGDOoS6JBERkXpLoaaeyiss4eoXZjFr9VYeHnUkv9GAeiIiIvukUFMPbd9VzFXPz2LuuhwevfAozjlK0xyIiIjsj0JNPZObX8zlz89kUVYuj1/Un9OPaB3qkkRERMKCQk09si2viEv/M4PlG3fy70uP5pTerUJdkoiISNhQqKkntuws5NJnZ7BqSx5PX340J/ZoGeqSREREwopCTT2wafsuLn52Bpnb8nnuioEc2z0t1CWJiIiEHYWaENuQW8DFz8xg4/ZdvHDVIIZ0SQ11SSIiImEpJpgHN7PTzGyZma0ws7v2sd9vzMyZWXow66lvMrflc+HTP7B5RyEvXa1AIyIiciiCFmrMLBZ4Ajgd6A1cZGa9q9ivCXArMCNYtdRHa7O9QJOTX8QrYwaT3ql5qEsSEREJa8FsqRkErHDOrXLOFQGTgHOq2O8vwEPAriDWUq+s2ryTC57+nryiEl69dghHtU8JdUkiIiJhL5ihpi2wrsJypr+unJkNANo75z7Y14HM7DozyzCzjM2bN9d+pXVo+cYdXDjhB4pLA7x27RD6tk0OdUkiIiIRIah9avbFzGKAR4A/7m9f59wE51y6cy69RYsWwS8uSJb+sp3RE34AYNJ1Q+jVummIKxIREYkcwQw1WUD7Csvt/HVlmgB9ga/MbDUwBJgSqZ2FF2blctGEH4iLjeH164bQvVWTUJckIiISUYIZamYB3c2ss5nFA6OBKWUbnXO5zrk051wn51wn4AdghHMuI4g1hcTcdTlc/MwPJMU34PXrh9ClReNQlyQiIhJxghZqnHMlwM3AVGAJMNk5t8jM7jezEcH6uvXN7DVbufTZGSQnxfH69UPomNoo1CWJiIhEpKAOvuec+xD4sNK6e6rZ94Rg1hIKP6zK5uoXZtGqaQKvXjuY1smJoS5JREQkYmlE4SCZtmIL17w4i3bNknh1zGBaNk0IdUkiIiIRTaEmCL5atonrX55N57RGvDJmMGmNG4a6JBERkYinUFPLPlu8kd9O/JHurRrzyjWDadYoPtQliYiIRIWQjVMTiT5asIEbXplNr9ZNeHXMEAUaERGROqSWmlry7twsbps8j6Pap/D8VQNpmhAX6pJERESiikJNLXhzdiZj35hHeqfmPHflQBo31GUVERGpa/rte4hen7WWu95awDFdU3nm8nSS4nVJRUREQkG/gQ/By9+v5n/eXcTxh7fg6cuOJiEuNtQliYiIRC2FmoP0n+9+5i/vL+bXvVryxCUDaNhAgUZERCSUFGoOwlNfr+TBj5Zyet/D+Nfo/sQ30ENkIiIioaZQc4Ae+3w5j3z6E2cf2YZHLziSBrEKNCIiIvWBQk0NOed45NOf+L8vVnDegLaMH3UksTEW6rJERETEp1BTA845HvxoKU9/s4rRA9vzt3OPIEaBRkREpF5RqNkP5xz3v7+Y56et5rIhHfnziD4KNCIiIvWQQs0+BAKOe6Ys5JUf1nL1sM78z1m9MFOgERERqY8UaqpRGnD86a0FvJ6xjhuO78qdp/VQoBEREanHFGqqUFIa4I435vPWnCxuObk7f/h1dwUaERGRek6hppLi0gB/eH0u78/fwB9POZzfndw91CWJiIhIDSjUAO/MyWL81GWszymgYVwMu4oDjDu9J9cf3zXUpYmIiEgNRX2oeWdOFuPeWkBBcSkAu4oDxMUarZomhLgyERERORBRPxzu+KnLygNNmeJSx/ipy0JUkYiIiByMqA8163MKDmi9iIiI1E9RH2rapCQe0HoRERGpn6I+1Iwd3oPEuNg91iXGxTJ2eI8QVSQiIiIHI+o7Co/s3xag/OmnNimJjB3eo3y9iIiIhIeoDzXgBRuFGBERkfAW9befREREJDIo1IiIiEhEUKgRERGRiKBQIyIiIhFBoUZEREQigjnnQl3DATGzzcCaUNdRhTRgS6iLqCd0LTy6Dh5dB4+uw266Fh5dB08P51yT2jhQ2D3S7ZxrEeoaqmJmGc659FDXUR/oWnh0HTy6Dh5dh910LTy6Dh4zy6itY+n2k4iIiEQEhRoRERGJCAo1tWdCqAuoR3QtPLoOHl0Hj67DbroWHl0HT61dh7DrKCwiIiJSFbXUiIiISERQqBEREZGIoFCzD2b2nJltMrOFFdY1N7NPzWy5/7mZv97M7DEzW2Fm881sQIX3XOHvv9zMrgjFuRwKM2tvZl+a2WIzW2Rmt/rro+pamFmCmc00s3n+dfizv76zmc3wz/d1M4v31zf0l1f42ztVONY4f/0yMxsemjM6NGYWa2ZzzOx9fzlar8NqM1tgZnPLHk2Ntu8NADNLMbM3zGypmS0xs6HRdh3MrIf//6DsY7uZ/T7argOAmf3B/zm50Mxe839+Bv9nhHNOH9V8AMcBA4CFFdb9A7jLf30X8JD/+gzgI8CAIcAMf31zYJX/uZn/ulmoz+0Ar0NrYID/ugnwE9A72q6Ffz6N/ddxwAz//CYDo/31TwE3+q9/Czzlvx4NvO6/7g3MAxoCnYGVQGyoz+8grsdtwKvA+/5ytF6H1UBapXVR9b3hn8OLwBj/dTyQEo3XocL1iAV+ATpG23UA2gI/A4n+8mTgyrr4GRHyk6/vH0An9gw1y4DW/uvWwDL/9dPARZX3Ay4Cnq6wfo/9wvEDeBc4JZqvBZAE/AgMxhsRtIG/figw1X89FRjqv27g72fAOGBchWOV7xcuH0A74HPgJOB9/7yi7jr4da9m71ATVd8bQDLeLzGL5utQ6dxPBaZF43XACzXr8EJZA/9nxPC6+Bmh208HrpVzboP/+heglf+67B+xTKa/rrr1YclvFuyP10oRddfCv+UyF9gEfIr3l0OOc67E36XiOZWfr789F0glAq4D8E/gDiDgL6cSndcBwAGfmNlsM7vOXxdt3xudgc3A8/4tyWfNrBHRdx0qGg285r+OquvgnMsCHgbWAhvwvudnUwc/IxRqDoHzomPUPBNvZo2BN4HfO+e2V9wWLdfCOVfqnDsKr6ViENAzxCXVOTM7C9jknJsd6lrqiWOdcwOA04GbzOy4ihuj5HujAd6t+n875/oDeXi3WcpFyXUAwO8rMgL4b+Vt0XAd/D5D5+CF3TZAI+C0uvjaCjUHbqOZtQbwP2/y12cB7Svs185fV936sGJmcXiBZqJz7i1/dVReCwDnXA7wJV4TaoqZlc2jVvGcys/X354MZBP+12EYMMLMVgOT8G5B/Yvouw5A+V+lOOc2AW/jhd1o+97IBDKdczP85TfwQk60XYcypwM/Ouc2+svRdh1+DfzsnNvsnCsG3sL7uRH0nxEKNQduClDWE/0KvP4lZesv93uzHTBIkwAABAJJREFUDwFy/ebGqcCpZtbMT6+n+uvChpkZ8B9giXPukQqboupamFkLM0vxXyfi9StaghduRvm7Vb4OZddnFPCF/1faFGC03+O/M9AdmFk3Z3HonHPjnHPtnHOd8JrYv3DOXUKUXQcAM2tkZk3KXuP9n15IlH1vOOd+AdaZWQ9/1cnAYqLsOlRwEbtvPUH0XYe1wBAzS/J/f5T9fwj+z4hQdyiqzx94/yk3AMV4f4lcg3ef73P4/+3dT6gWVRjH8e8PDG+kCFF7EwyhSIuMJIu7iKBwUW6EWuWmggqMCEkQ2l2wTYs2QRCEtCmyFmHRohKjMux61VK60CaiCArJpAh7Wsy5OVxuSvePXub9fmB43/l3ZubADA/nnJmHb4EPgWvbtgFephtjcRy4vVfOTmC6TY9e6euaRz1spWsunQIm2/TAqNUFcAvwVauHE8Detnxdu9Gm6ZqbV7blY21+uq1f1ytrT6uf08D9V/raFlAn41x4+2nk6qFd87E2nQT2tOUjdW+0898EfNnujwN0b+2MYj1cQ9fKsKa3bBTr4QXgVHtWvk73BtOSPyNMkyBJkgbB7idJkjQIBjWSJGkQDGokSdIgGNRIkqRBMKiRJEmDYFAjad6SfNp+1yZ5eJHLfn6uY0nSf/GVbkkLlmQceLaqtv2PfVbUhTwwc60/W1WrFuP8JI0GW2okzVuSs+3vBHB3kskku1riz31JjiSZSvJY2348yaEk79J9YZQkB1oyyJMzCSGTTABXt/L294/Vvr66L8mJJMeT7OiV/VGSN5OcSrK/fc2UJBNJvm7n8uLlrCNJl8+KS28iSZe0m15LTQtOzlTV5iQrgcNJPmjb3gbcXFXftfmdVfVLSz1xJMlbVbU7yZPVJQ+dbTvd12s3Ate1fT5p624FbgJ+AA4DdyX5BngI2FBVNZPqQtLw2FIjaSncR5fTZhL4nO4z8evbui96AQ3A00mOAZ/RJa9bz8VtBd6oLmP6T8DHwOZe2d9X1d906TzWAmeAP4BXk2wHzi346iQtSwY1kpZCgKeqalObbqiqmZaa3//dqBuLcy+wpao20uXWGlvAcf/s/T8PzIzbuYMuc/Q24OACype0jBnUSFoMvwGre/PvA08kuQogyY0ti/Vsa4Bfq+pckg3Anb11f83sP8shYEcbt3M9cA8XydybZBVdcsH3gF103VaSBsgxNZIWwxRwvnUjvQa8RNf1c7QN1v0ZeHCO/Q4Cj7dxL6fpuqBmvAJMJTlaVY/0lr8NbKHLjF3Ac1X1YwuK5rIaeCfJGF0L0jPzu0RJy52vdEuSpEGw+0mSJA2CQY0kSRoEgxpJkjQIBjWSJGkQDGokSdIgGNRIkqRBMKiRJEmD8A/cYC/VQrPu5gAAAABJRU5ErkJggg==","text/plain":["<Figure size 648x288 with 1 Axes>"]},"metadata":{"tags":[]},"output_type":"display_data"}],"source":["plt.title('Val accuracies')\n","for name in names:\n","  plt.plot(iter_history_dict[name], acc_history_dict[name], '-o')\n","plt.legend(names, loc='upper left')\n","plt.xlabel('iterations')\n","plt.ylabel('accuracy')\n","plt.gcf().set_size_inches(9, 4)\n","plt.show()"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"7Z31mvHGQT3y"},"source":["## Residual bottleneck block\n","\n","A bottleneck block is often useful for better efficiency, especially when importing a model to mobile devices.\n","The residual bottleneck block is similar to the standard residual block, but the plain block part has a different architecture:\n","it consists of 3 convolutional layers, and the first two convolutional layers have a smaller number of channels.\n","\n","Here is the specification of the bottleneck block:\n","\n","1. Spatial Batch normalization\n","2. ReLU\n","3. Convolutional layer with `Cout // 4` 1x1 filters, stride 2 if downsampling; otherwise stride 1\n","4. Spatial Batch normalization\n","5. ReLU\n","6. Convolutional layer with `Cout // 4` 3x3 filters, with zero-padding of 1\n","7. Spatial Batch normalization\n","8. ReLU\n","9. Convolutional layer with `Cout` 1x1 filters\n","\n","Don't forget to add the residual connection!"]},{"cell_type":"code","execution_count":0,"metadata":{"colab":{},"colab_type":"code","id":"7pUtZoVsQT31"},"outputs":[],"source":["class ResidualBottleneckBlock(nn.Module):\n","  def __init__(self, Cin, Cout, downsample=False):\n","    super().__init__()\n","\n","    self.block = None\n","    self.shortcut = None\n","    ############################################################################\n","    # TODO: Implement residual bottleneck block.                               #\n","    # Inputs:                                                                  #\n","    # - Cin: number of input channels                                          #\n","    # - Cout: number of output channels                                        #\n","    # - downsample: add downsampling (a conv with stride=2) if True            #\n","    # Store the main block in self.block and the shortcut in self.shortcut.    #\n","    ############################################################################\n","    # Replace \"pass\" statement with your code\n","    stride = 2 if downsample else 1\n","    self.block = nn.Sequential(\n","      nn.BatchNorm2d(Cin),\n","      nn.ReLU(),\n","      nn.Conv2d(Cin, Cout//4, 1, stride=stride),\n","      nn.BatchNorm2d(Cout//4),\n","      nn.ReLU(),\n","      nn.Conv2d(Cout//4, Cout//4, 3, padding=1),\n","      nn.BatchNorm2d(Cout//4),\n","      nn.ReLU(),\n","      nn.Conv2d(Cout//4, Cout, 1) \n","    )\n","    if not downsample:\n","      if Cin == Cout:\n","        self.shortcut = nn.Sequential(\n","            nn.Identity()\n","        )\n","      else:\n","        self.shortcut = nn.Sequential(\n","            nn.Conv2d(Cin, Cout, 1)\n","        )\n","    else:\n","      self.shortcut = nn.Sequential(\n","            nn.Conv2d(Cin, Cout, 1, stride=2)\n","        )\n","    ############################################################################\n","    #                                 END OF YOUR CODE                         #\n","    ############################################################################\n","\n","  def forward(self, x):\n","    return self.block(x) + self.shortcut(x)"]},{"cell_type":"code","execution_count":36,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":50},"colab_type":"code","executionInfo":{"elapsed":1199804,"status":"ok","timestamp":1573620229519,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"vqETnXH5QT37","outputId":"d7def613-7d61-4068-9fbe-a3c5472b67f8"},"outputs":[{"name":"stdout","output_type":"stream","text":["The output of ResidualBlock without downsampling has a *correct* dimension!\n","The output of ResidualBlock with downsampling has a *correct* dimension!\n"]}],"source":["data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBottleneckBlock(3, 10)\n","if list(model(data).shape) == [2, 10, 5, 6]:\n","  print('The output of ResidualBlock without downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock without downsampling has an *incorrect* dimension! expected:', [2, 10, 5, 6], 'got:', list(model(data).shape))\n","\n","data = torch.zeros(2, 3, 5, 6)\n","model = ResidualBottleneckBlock(3, 10, downsample=True)\n","if list(model(data).shape) == [2, 10, 3, 3]:\n","  print('The output of ResidualBlock with downsampling has a *correct* dimension!')\n","else:\n","  print('The output of ResidualBlock with downsampling has an *incorrect* dimension! expected:', [2, 10, 3, 3], 'got:', list(model(data).shape))"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"QnRc5hiHudMP"},"source":["By running the following script, you can check the architecture of ResNet-47 with bottlenecks.\n","\n","Caution: it is long!"]},{"cell_type":"code","execution_count":37,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"colab_type":"code","executionInfo":{"elapsed":1199801,"status":"ok","timestamp":1573620229520,"user":{"displayName":"Linxiao Wu","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mBy2vZ3If8UYdlNoQ8z5dkghCJlzx0iHREISKl0=s64","userId":"09330174145459862569"},"user_tz":300},"id":"md6xmG-Aucrx","outputId":"440b228f-ee02-4bb5-9b5e-864279de9c13"},"outputs":[{"name":"stdout","output_type":"stream","text":["ResNet(\n","  (cnn): Sequential(\n","    (0): ResNetStem(\n","      (net): Sequential(\n","        (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","        (1): ReLU()\n","      )\n","    )\n","    (1): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 8, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(8, 32, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","      )\n","    )\n","    (2): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(32, 16, kernel_size=(1, 1), stride=(2, 2))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Conv2d(32, 64, kernel_size=(1, 1), stride=(2, 2))\n","          )\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","      )\n","    )\n","    (3): ResNetStage(\n","      (net): Sequential(\n","        (0): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(64, 32, kernel_size=(1, 1), stride=(2, 2))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2))\n","          )\n","        )\n","        (1): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (2): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (3): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","        (4): ResidualBottleneckBlock(\n","          (block): Sequential(\n","            (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (1): ReLU()\n","            (2): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))\n","            (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (4): ReLU()\n","            (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n","            (6): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n","            (7): ReLU()\n","            (8): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))\n","          )\n","          (shortcut): Sequential(\n","            (0): Identity()\n","          )\n","        )\n","      )\n","    )\n","  )\n","  (fc): Linear(in_features=128, out_features=10, bias=True)\n",")\n"]}],"source":["# example of specification\n","networks.update({\n","  'resnet47': {\n","    'block': ResidualBottleneckBlock,\n","    'stage_args': [\n","      (32, 32, 5, False),\n","      (32, 64, 5, True),\n","      (64, 128, 5, True),\n","    ],\n","  },\n","})\n","\n","print(get_resnet('resnet47'))"]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"pytorch_autograd_and_nn.ipynb","provenance":[{"file_id":"1mGz6ZqYX6FDsVZYqEB36M-NRv7fl_ds4","timestamp":1573425030670}]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.1"},"toc":{"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"toc_cell":false,"toc_position":{},"toc_section_display":"block","toc_window_display":false},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"nbformat":4,"nbformat_minor":0}