1.2 Densely Connected Neural Networks - MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Using Densely Connected Neural Networks for MNIST (aka \"Hello world of DL\") dataset\n",
    "This demo shows how to use Keras (with TensorFlow in the backend) to train and test a neural network on MNIST handwritten digit recognition data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import RMSprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and process data\n",
    "\n",
    "We first fetch the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset, which is a commonly used dataset for handwritten digit recognition. Keras provides a handy function for loading this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 3s 0us/step\n"
     ]
    }
   ],
   "source": [
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "num_classes = 10  # number of classes (digits) to predict\n",
    "\n",
    "x_train = x_train.reshape(60000, 28 * 28)   # we preprocess data and reshape into the input the neural network expects\n",
    "x_train = x_train.astype('float32')         #transform the type\n",
    "x_train /= 255                              #values were in the [0..255] interval, while neural network expects values to be between 0 and 1\n",
    "\n",
    "x_test = x_test.reshape(10000, 28 * 28)\n",
    "x_test = x_test.astype('float32')\n",
    "x_test /= 255\n",
    "\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "#our labels \"0\", \"1\"..\"9\" should be encoded categorically for both train and test labels\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)  \n",
    "y_test = keras.utils.to_categorical(y_test, num_classes) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a fully connected NN model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, define the model structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(512, activation='relu', input_shape=(28*28,)))  # Layer 1: takes input shape 28x28, uses ReLU activation function, and produces and output space with 512 units \n",
    "model.add(Dense(10, activation='softmax'))                      # Layer 2: 10-way softmax layer, which returns an array of 10 probability scores for a specific class (0..9) \n",
    "\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Layer (type)                 Output Shape              Param # \n",
    "# dense_4 (Dense)              (None, 512)               401920    # (28 * 28 * 1[grayscale])[input values] * 512 [neurons in this layer] + 512 [bias values]\n",
    "# dense_5 (Dense)              (None, 10)                5130      # 512 [input values] * 10 [neurons in this layer] + 10 [bias values]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can fit the model. This should take about 10-15 seconds per epoch on a commodity GPU, or about 2-3 minutes for 10 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 24s 397us/step - loss: 0.2539 - acc: 0.9261 - val_loss: 0.1332 - val_acc: 0.9608\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.1025 - acc: 0.9696 - val_loss: 0.0950 - val_acc: 0.9711\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.0676 - acc: 0.9796 - val_loss: 0.0759 - val_acc: 0.9771\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 2s 28us/step - loss: 0.0495 - acc: 0.9851 - val_loss: 0.0759 - val_acc: 0.9769\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 2s 27us/step - loss: 0.0370 - acc: 0.9889 - val_loss: 0.0702 - val_acc: 0.9795\n"
     ]
    }
   ],
   "source": [
    "#to get neural network ready for training we need to compile it and specify a few metrics as below\n",
    "model.compile(loss='categorical_crossentropy',    # loss: how network measures its performance on the training data (and steers itself into right direction)\n",
    "              optimizer=RMSprop(),                # optimiser: the mechanism how the network will update itself on the data it sees and the loss it regenerates\n",
    "              metrics=['accuracy'])               # metrics: what we care about during training and testing of the model (accuracy: how many images were correctly classified)\n",
    "\n",
    "#now we ready to train the network, and we use fit method in Keras to do it\n",
    "history = model.fit(x_train, y_train,\n",
    "                    batch_size=128,               # the size of the mini-batch used by the network to train\n",
    "                    epochs=5,                     # the number of iterations over the training dataset\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the model\n",
    "\n",
    "We can get test accuracy above `97%` after 5 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.0702366737995\n",
      "Test accuracy: 0.9795\n"
     ]
    }
   ],
   "source": [
    "# now we use our test dataste (that has not been seen by the model before) and score it and evaluate the results\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can we visualise the scoring process?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}