rossman_kaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Kaggle Rossman Cometition\n",
    "\n",
    "This was a kaggle competition to forecast sales at a pharmacy chain/dept store in Europe. It was run back in 2015.\n",
    "\n",
    "The aim of this lab is\n",
    "\n",
    "(a) to see how to \"grid-search\" when we think the data is too large to use cross-validation. This is in opposition to the other way we usually do grid search using pipelines. But we still want to use sklearn/dask pipelines as much as possible so that ALL transformations can be used on validation and test sets\n",
    "(b) to understand some aspects of feature engineering that come in with continuous and categorical variables, and to see some of the new features in sklearn 0.20\n",
    "(c) to capture results from validation\n",
    "(d) to investigate the use of categorical \"embeddings\" to improve performance of a multi-layer percepton\n",
    "(e) if time permits to use dask to do some of this stuff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy.stats\n",
    "import scipy.special\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.mlab as mlab\n",
    "from matplotlib import cm\n",
    "import pandas as pd\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Path('./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We engage in some cleaning. A lot of cleaning of this dataset has already been done for us. Some features have been created. In particular we moved from dates to week-of-year, day-of week, etc. For example the 49th and 50th weeks of the year may have higher sales!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(data/\"train_clean.csv\").drop(['index', 'PromoInterval'], axis=1)\n",
    "test_df = pd.read_csv(data/\"test_clean.csv\").drop(['index', 'PromoInterval'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['Events'] = train_df['Events'].fillna('None')\n",
    "test_df['Events'] = test_df['Events'].fillna('None')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in log-transforming the dependent variable because it is long-tailed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_resp = np.log(train_df['Sales'].copy())\n",
    "train_df = train_df.drop('Sales', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets get some idea about our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Store</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>Date</th>\n",
       "      <th>Customers</th>\n",
       "      <th>Open</th>\n",
       "      <th>Promo</th>\n",
       "      <th>StateHoliday</th>\n",
       "      <th>SchoolHoliday</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>...</th>\n",
       "      <th>AfterStateHoliday</th>\n",
       "      <th>BeforeStateHoliday</th>\n",
       "      <th>AfterPromo</th>\n",
       "      <th>BeforePromo</th>\n",
       "      <th>SchoolHoliday_bw</th>\n",
       "      <th>StateHoliday_bw</th>\n",
       "      <th>Promo_bw</th>\n",
       "      <th>SchoolHoliday_fw</th>\n",
       "      <th>StateHoliday_fw</th>\n",
       "      <th>Promo_fw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>555</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>625</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>821</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>1498</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>2015-07-31</td>\n",
       "      <td>559</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>2015</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 90 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Store  DayOfWeek        Date  Customers  Open  Promo  StateHoliday  \\\n",
       "0      1          5  2015-07-31        555     1      1         False   \n",
       "1      2          5  2015-07-31        625     1      1         False   \n",
       "2      3          5  2015-07-31        821     1      1         False   \n",
       "3      4          5  2015-07-31       1498     1      1         False   \n",
       "4      5          5  2015-07-31        559     1      1         False   \n",
       "\n",
       "   SchoolHoliday  Year  Month    ...     AfterStateHoliday  \\\n",
       "0              1  2015      7    ...                    57   \n",
       "1              1  2015      7    ...                    67   \n",
       "2              1  2015      7    ...                    57   \n",
       "3              1  2015      7    ...                    67   \n",
       "4              1  2015      7    ...                    57   \n",
       "\n",
       "   BeforeStateHoliday  AfterPromo  BeforePromo  SchoolHoliday_bw  \\\n",
       "0                   0           0            0               5.0   \n",
       "1                   0           0            0               5.0   \n",
       "2                   0           0            0               5.0   \n",
       "3                   0           0            0               5.0   \n",
       "4                   0           0            0               5.0   \n",
       "\n",
       "   StateHoliday_bw  Promo_bw  SchoolHoliday_fw  StateHoliday_fw  Promo_fw  \n",
       "0              0.0       5.0               7.0              0.0       5.0  \n",
       "1              0.0       5.0               1.0              0.0       1.0  \n",
       "2              0.0       5.0               5.0              0.0       5.0  \n",
       "3              0.0       5.0               1.0              0.0       1.0  \n",
       "4              0.0       5.0               1.0              0.0       1.0  \n",
       "\n",
       "[5 rows x 90 columns]"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((844338, 90), (41088, 90))"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape, test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         2015-07-31\n",
       "1         2015-07-31\n",
       "2         2015-07-31\n",
       "3         2015-07-31\n",
       "4         2015-07-31\n",
       "5         2015-07-31\n",
       "6         2015-07-31\n",
       "7         2015-07-31\n",
       "8         2015-07-31\n",
       "9         2015-07-31\n",
       "10        2015-07-31\n",
       "11        2015-07-31\n",
       "12        2015-07-31\n",
       "13        2015-07-31\n",
       "14        2015-07-31\n",
       "15        2015-07-31\n",
       "16        2015-07-31\n",
       "17        2015-07-31\n",
       "18        2015-07-31\n",
       "19        2015-07-31\n",
       "20        2015-07-31\n",
       "21        2015-07-31\n",
       "22        2015-07-31\n",
       "23        2015-07-31\n",
       "24        2015-07-31\n",
       "25        2015-07-31\n",
       "26        2015-07-31\n",
       "27        2015-07-31\n",
       "28        2015-07-31\n",
       "29        2015-07-31\n",
       "             ...    \n",
       "844308    2013-01-02\n",
       "844309    2013-01-02\n",
       "844310    2013-01-02\n",
       "844311    2013-01-02\n",
       "844312    2013-01-02\n",
       "844313    2013-01-02\n",
       "844314    2013-01-02\n",
       "844315    2013-01-02\n",
       "844316    2013-01-02\n",
       "844317    2013-01-02\n",
       "844318    2013-01-02\n",
       "844319    2013-01-02\n",
       "844320    2013-01-02\n",
       "844321    2013-01-01\n",
       "844322    2013-01-01\n",
       "844323    2013-01-01\n",
       "844324    2013-01-01\n",
       "844325    2013-01-01\n",
       "844326    2013-01-01\n",
       "844327    2013-01-01\n",
       "844328    2013-01-01\n",
       "844329    2013-01-01\n",
       "844330    2013-01-01\n",
       "844331    2013-01-01\n",
       "844332    2013-01-01\n",
       "844333    2013-01-01\n",
       "844334    2013-01-01\n",
       "844335    2013-01-01\n",
       "844336    2013-01-01\n",
       "844337    2013-01-01\n",
       "Name: Date, Length: 844338, dtype: object"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.Date # latest date first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Store', 'DayOfWeek', 'Date', 'Customers', 'Open', 'Promo',\n",
       "       'StateHoliday', 'SchoolHoliday', 'Year', 'Month', 'Week', 'Day',\n",
       "       'Dayofweek', 'Dayofyear', 'Is_month_end', 'Is_month_start',\n",
       "       'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start',\n",
       "       'Elapsed', 'StoreType', 'Assortment', 'CompetitionDistance',\n",
       "       'CompetitionOpenSinceMonth', 'CompetitionOpenSinceYear', 'Promo2',\n",
       "       'Promo2SinceWeek', 'Promo2SinceYear', 'State', 'file', 'week', 'trend',\n",
       "       'file_DE', 'week_DE', 'trend_DE', 'Date_DE', 'State_DE', 'Month_DE',\n",
       "       'Day_DE', 'Dayofweek_DE', 'Dayofyear_DE', 'Is_month_end_DE',\n",
       "       'Is_month_start_DE', 'Is_quarter_end_DE', 'Is_quarter_start_DE',\n",
       "       'Is_year_end_DE', 'Is_year_start_DE', 'Elapsed_DE', 'Max_TemperatureC',\n",
       "       'Mean_TemperatureC', 'Min_TemperatureC', 'Dew_PointC', 'MeanDew_PointC',\n",
       "       'Min_DewpointC', 'Max_Humidity', 'Mean_Humidity', 'Min_Humidity',\n",
       "       'Max_Sea_Level_PressurehPa', 'Mean_Sea_Level_PressurehPa',\n",
       "       'Min_Sea_Level_PressurehPa', 'Max_VisibilityKm', 'Mean_VisibilityKm',\n",
       "       'Min_VisibilitykM', 'Max_Wind_SpeedKm_h', 'Mean_Wind_SpeedKm_h',\n",
       "       'Max_Gust_SpeedKm_h', 'Precipitationmm', 'CloudCover', 'Events',\n",
       "       'WindDirDegrees', 'StateName', 'CompetitionOpenSince',\n",
       "       'CompetitionDaysOpen', 'CompetitionMonthsOpen', 'Promo2Since',\n",
       "       'Promo2Days', 'Promo2Weeks', 'AfterSchoolHoliday',\n",
       "       'BeforeSchoolHoliday', 'AfterStateHoliday', 'BeforeStateHoliday',\n",
       "       'AfterPromo', 'BeforePromo', 'SchoolHoliday_bw', 'StateHoliday_bw',\n",
       "       'Promo_bw', 'SchoolHoliday_fw', 'StateHoliday_fw', 'Promo_fw'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of variables and cardinality\n",
    "\n",
    "We make a note of which variables are categorical and which are not. This is a choice. If cardinality is not too high, binning or categorizing can be beneficial. Often this will be true for integer valued variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 351,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',\n",
    "    'Promo2Weeks', 'StoreType', 'Assortment', 'CompetitionOpenSinceYear', 'Promo2SinceYear',\n",
    "    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',\n",
    "    'SchoolHoliday_fw', 'SchoolHoliday_bw', 'Promo', 'SchoolHoliday']\n",
    "\n",
    "cont_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
    "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
    "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
    "   'AfterStateHoliday', 'BeforeStateHoliday']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We look for missing data and store the column names where this happend in the continuous data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 354,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompetitionDistance 2075\n",
      "CloudCover 64187\n"
     ]
    }
   ],
   "source": [
    "nacols=[]\n",
    "for v in cont_vars:\n",
    "    if np.sum(trdf[v].isnull()) > 0:\n",
    "        nacols.append(v)\n",
    "        print(v, np.sum(trdf[v].isnull()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And look at some cardinalities: since we have none below 10, we dont engage in binning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 355,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompetitionDistance 655\n",
      "Max_TemperatureC 49\n",
      "Mean_TemperatureC 44\n",
      "Min_TemperatureC 40\n",
      "Max_Humidity 50\n",
      "Mean_Humidity 70\n",
      "Min_Humidity 93\n",
      "Max_Wind_SpeedKm_h 42\n",
      "Mean_Wind_SpeedKm_h 27\n",
      "CloudCover 10\n",
      "trend 67\n",
      "trend_DE 36\n",
      "AfterStateHoliday 136\n",
      "BeforeStateHoliday 147\n"
     ]
    }
   ],
   "source": [
    "for k in cont_vars:\n",
    "    print(k, trdf[k].unique().shape[0])\n",
    "    if trdf[k].unique().shape[0] < 10:\n",
    "        print(trdf[k].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do a similar looksie on the categorical variables. Some of these have many levels. Is there really that much information in 1115 store labels. Can we get some compression to increase our signal-to-noise?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 399,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Store 1115\n",
      "DayOfWeek 7\n",
      "[5 4 3 2 1 7 6]\n",
      "Year 3\n",
      "[2015 2014 2013]\n",
      "Month 12\n",
      "[ 6  5  4  3  2  1 12 11 10  9  8  7]\n",
      "Day 31\n",
      "[19 18 17 16 15 14 13 12 11 10  9  8  7  6  5  4  3  2  1 31 30 29 28 27\n",
      " 26 25 24 23 22 21 20]\n",
      "StateHoliday 2\n",
      "[False  True]\n",
      "CompetitionMonthsOpen 25\n",
      "[ 0 24  2 18  8 15 16  6 23 14 21 10 12  1 22 11  3  9 13 19  7 17  5 20\n",
      "  4]\n",
      "Promo2Weeks 26\n",
      "[25  0 19 11  2  7 18 10  1  6 17  9  5 16  8  4 15 24  3 14 23 13 22 12\n",
      " 21 20]\n",
      "StoreType 4\n",
      "['d' 'c' 'a' 'b']\n",
      "Assortment 3\n",
      "['c' 'a' 'b']\n",
      "CompetitionOpenSinceYear 23\n",
      "[1900 2008 2007 2006 2009 2015 2013 2014 2000 2011 2010 2005 1999 2003\n",
      " 2012 2004 2002 1961 1995 2001 1990 1994 1998]\n",
      "Promo2SinceYear 8\n",
      "[2012 1900 2010 2011 2009 2014 2015 2013]\n",
      "State 12\n",
      "['HE' 'TH' 'NW' 'BE' 'SN' 'SH' 'HB,NI' 'BY' 'BW' 'RP' 'ST' 'HH']\n",
      "Week 52\n",
      "Events 22\n",
      "['Rain' 'None' 'Fog' 'Fog-Rain' 'Rain-Thunderstorm'\n",
      " 'Fog-Rain-Thunderstorm' 'Thunderstorm' 'Rain-Hail' 'Fog-Thunderstorm'\n",
      " 'Rain-Hail-Thunderstorm' 'Rain-Snow' 'Fog-Rain-Hail-Thunderstorm' 'Snow'\n",
      " 'Rain-Snow-Hail' 'Rain-Snow-Hail-Thunderstorm' 'Rain-Snow-Thunderstorm'\n",
      " 'Fog-Rain-Snow' 'Fog-Snow' 'Snow-Hail' 'Fog-Rain-Snow-Hail'\n",
      " 'Fog-Rain-Hail' 'Fog-Snow-Hail']\n",
      "Promo_fw 6\n",
      "[1. 2. 3. 4. 5. 0.]\n",
      "Promo_bw 6\n",
      "[5. 4. 3. 2. 1. 0.]\n",
      "StateHoliday_fw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "StateHoliday_bw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "SchoolHoliday_fw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "SchoolHoliday_bw 8\n",
      "[0. 1. 2. 3. 4. 5. 6. 7.]\n",
      "Promo 2\n",
      "[1 0]\n",
      "SchoolHoliday 2\n",
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "for k in cat_vars:\n",
    "    print(k, trdf[k].unique().shape[0])\n",
    "    if trdf[k].unique().shape[0] < 50:\n",
    "        print(trdf[k].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a validation set\n",
    "\n",
    "The construction of a validation or \"development\" set is not always a `test_train_split` deal. Here we create a validation set of \"latest\" data, cireesponding oin date and size to what we have in the test set. Hopefully this will make sure we have similar distributions of features and outcomes on both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41395"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cut = train_df['Date'][(train_df['Date'] == train_df['Date'][len(test_df)])].index.max()\n",
    "cut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_idx = range(cut)\n",
    "train_idx = list(np.setdiff1d(range(train_df.shape[0]), valid_idx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "trdf = train_df.iloc[train_idx]\n",
    "vadf = train_df.iloc[valid_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943, 90), (41395, 90))"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trdf.shape, vadf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation Pipelines\n",
    "\n",
    "Ok, now we'll use the new `ColumnTransformer`, with imputation, missing-data indicators, the new `OrdinalEncoder`, and the usual Standard Scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 356,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer,MissingIndicator\n",
    "from sklearn.pipeline import make_pipeline, make_union, Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 373,
   "metadata": {},
   "outputs": [],
   "source": [
    "impu = SimpleImputer(strategy=\"median\") # create a median imputer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the missing indicator separately as it creates a new column. It is possible to do this in the pipeline flow in `sklearn` using a union, but subsequent scaling wants to scale this indicator fince the categorical list does not include the new columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {},
   "outputs": [],
   "source": [
    "mi = MissingIndicator() # create, fit, and transform a missingness indicator\n",
    "mi.fit(trdf[nacols])\n",
    "Xtrmi = mi.transform(trdf[nacols])\n",
    "Xvami = mi.transform(vadf[nacols])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False,  True])"
      ]
     },
     "execution_count": 372,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrmi[4460,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, OrdinalEncoder\n",
    "ss = StandardScaler()\n",
    "oe = OrdinalEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {},
   "outputs": [],
   "source": [
    "trdf_cat = trdf[cat_vars]\n",
    "trdf_cont = trdf[cont_vars]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct two pipelines, one for categoricals and one for continuous variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont_pipe = Pipeline([(\"imp\",impu), (\"scale\", ss)])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_pipe = Pipeline([(\"categorify\", oe)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And combine them here in a transformer list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformers = [('cat', cat_pipe, cat_vars),\n",
    "                    ('cont', cont_pipe, cont_vars)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use a `ColumnTransformer` to combine these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 379,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "ct = ColumnTransformer(transformers=transformers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ColumnTransformer(n_jobs=None, remainder='drop', sparse_threshold=0.3,\n",
       "         transformer_weights=None,\n",
       "         transformers=[('cat', Pipeline(memory=None,\n",
       "     steps=[('categorify', OrdinalEncoder(categories='auto', dtype=<class 'numpy.float64'>))]), ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen', 'Promo2Weeks', 'StoreType', 'Assortment', 'CompetitionOpenSinceYear', 'P...ean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE', 'AfterStateHoliday', 'BeforeStateHoliday'])])"
      ]
     },
     "execution_count": 380,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.fit(trdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtr = ct.transform(trdf)\n",
    "Xval = ct.transform(vadf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943, 37), (802943, 2))"
      ]
     },
     "execution_count": 384,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtr.shape, Xtrmi.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We concatenate the old indicators back in. The transformer lists all the categoricals first, since thats the first item in `transformers`, so we pre-pend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(802943, 39)"
      ]
     },
     "execution_count": 400,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = np.concatenate([Xtrmi, Xtr], axis=1)\n",
    "Xtrain.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41395, 39)"
      ]
     },
     "execution_count": 401,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xvalid = np.concatenate([Xvami, Xval], axis=1)\n",
    "Xvalid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn-pipelines lose our nice pandas names. so we bring them back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([(0, ('CompetitionDistance_missing', 'cont')),\n",
       "  (1, ('CloudCover_missing', 'cont')),\n",
       "  (2, ('Store', 'cat')),\n",
       "  (3, ('DayOfWeek', 'cat')),\n",
       "  (4, ('Year', 'cat')),\n",
       "  (5, ('Month', 'cat')),\n",
       "  (6, ('Day', 'cat')),\n",
       "  (7, ('StateHoliday', 'cat')),\n",
       "  (8, ('CompetitionMonthsOpen', 'cat')),\n",
       "  (9, ('Promo2Weeks', 'cat')),\n",
       "  (10, ('StoreType', 'cat')),\n",
       "  (11, ('Assortment', 'cat')),\n",
       "  (12, ('CompetitionOpenSinceYear', 'cat')),\n",
       "  (13, ('Promo2SinceYear', 'cat')),\n",
       "  (14, ('State', 'cat')),\n",
       "  (15, ('Week', 'cat')),\n",
       "  (16, ('Events', 'cat')),\n",
       "  (17, ('Promo_fw', 'cat')),\n",
       "  (18, ('Promo_bw', 'cat')),\n",
       "  (19, ('StateHoliday_fw', 'cat')),\n",
       "  (20, ('StateHoliday_bw', 'cat')),\n",
       "  (21, ('SchoolHoliday_fw', 'cat')),\n",
       "  (22, ('SchoolHoliday_bw', 'cat')),\n",
       "  (23, ('Promo', 'cat')),\n",
       "  (24, ('SchoolHoliday', 'cat')),\n",
       "  (25, ('CompetitionDistance', 'cont')),\n",
       "  (26, ('Max_TemperatureC', 'cont')),\n",
       "  (27, ('Mean_TemperatureC', 'cont')),\n",
       "  (28, ('Min_TemperatureC', 'cont')),\n",
       "  (29, ('Max_Humidity', 'cont')),\n",
       "  (30, ('Mean_Humidity', 'cont')),\n",
       "  (31, ('Min_Humidity', 'cont')),\n",
       "  (32, ('Max_Wind_SpeedKm_h', 'cont')),\n",
       "  (33, ('Mean_Wind_SpeedKm_h', 'cont')),\n",
       "  (34, ('CloudCover', 'cont')),\n",
       "  (35, ('trend', 'cont')),\n",
       "  (36, ('trend_DE', 'cont')),\n",
       "  (37, ('AfterStateHoliday', 'cont')),\n",
       "  (38, ('BeforeStateHoliday', 'cont'))],\n",
       " 39)"
      ]
     },
     "execution_count": 436,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = trdf.columns\n",
    "actcols = []\n",
    "actcolcount = 0\n",
    "nacols_cat = []\n",
    "for k in nacols:\n",
    "    actcols.append((k+'_missing', 'cont'))\n",
    "    nacols_cat.append(k+'_missing')\n",
    "    actcolcount+=1\n",
    "for k in cat_vars+cont_vars:\n",
    "    if k in cat_vars:\n",
    "        actcols.append((k, \"cat\"))\n",
    "        actcolcount+=1\n",
    "    if k in cont_vars:\n",
    "        actcols.append((k, \"cont\"))\n",
    "        actcolcount+=1\n",
    "        \n",
    "list(enumerate(actcols)), actcolcount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time to learn\n",
    "\n",
    "We first plit the y (the log of the y, really)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 390,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((802943,), (41395,))"
      ]
     },
     "execution_count": 390,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ytrain = train_resp[train_idx]\n",
    "yvalid = train_resp[list(valid_idx)]\n",
    "ytrain.shape, yvalid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and import what we need to for Gradient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Peter Prettenhofer, who wrote sklearns GBRT implementation writes in his pydata14 talk (worth watching!)\n",
    "\n",
    ">Hyperparameter tuning I usually follow this recipe to tune the hyperparameters:\n",
    "\n",
    "> \n",
    "- Pick n_estimators as large as (computationally) possible (e.g. 3000)\n",
    "- Tune max_depth, learning_rate, min_samples_leaf, and max_features via grid search\n",
    "- A lower learning_rate requires a higher number of n_estimators. Thus increase n_estimators even more and tune learning_rate again holding the other parameters fixed\n",
    "\n",
    ">This last point is a tradeof between number of iterations or runtime against accuracy. And keep in mind that it might lead to overfitting.\n",
    "\n",
    "Let me add however, that poor learners do rather well. So you might want to not cross-validate max_depth. And min_samples_per_leaf is not independent either, so if you do use cross-val, you might just use one of those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `ParameterGrid` here to construct the entire grid for us! We put the output in a list of dictionaries and then save it in a dataframe. We might want to persist such dataframes to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {'learning_rate': [0.1, 0.01],\n",
    "              'max_depth': [1,2, 3],\n",
    "              'max_features': [0.2, 0.6]\n",
    "              }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ParameterGrid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'learning_rate': 0.1, 'max_depth': 1, 'max_features': 0.2}\n",
      "MSE 0.11952736672561466 0.12841955343258007\n",
      "{'learning_rate': 0.1, 'max_depth': 1, 'max_features': 0.6}\n",
      "MSE 0.11949081835876278 0.12800020775942245\n",
      "{'learning_rate': 0.1, 'max_depth': 2, 'max_features': 0.2}\n",
      "MSE 0.11005260932751748 0.11449586709006311\n",
      "{'learning_rate': 0.1, 'max_depth': 2, 'max_features': 0.6}\n",
      "MSE 0.10579365894338559 0.11001812163105405\n",
      "{'learning_rate': 0.1, 'max_depth': 3, 'max_features': 0.2}\n",
      "MSE 0.09878988052062682 0.10056851339771275\n",
      "{'learning_rate': 0.1, 'max_depth': 3, 'max_features': 0.6}\n",
      "MSE 0.08781969095003936 0.08837837714343463\n",
      "{'learning_rate': 0.01, 'max_depth': 1, 'max_features': 0.2}\n",
      "MSE 0.14663437176401278 0.1567381426802156\n",
      "{'learning_rate': 0.01, 'max_depth': 1, 'max_features': 0.6}\n",
      "MSE 0.13861640989094157 0.1504924776928524\n",
      "{'learning_rate': 0.01, 'max_depth': 2, 'max_features': 0.2}\n",
      "MSE 0.13690674397698613 0.14609089983139759\n",
      "{'learning_rate': 0.01, 'max_depth': 2, 'max_features': 0.6}\n",
      "MSE 0.13050218541652203 0.14060863140763535\n",
      "{'learning_rate': 0.01, 'max_depth': 3, 'max_features': 0.2}\n",
      "MSE 0.12831190858899424 0.13707657023074862\n",
      "{'learning_rate': 0.01, 'max_depth': 3, 'max_features': 0.6}\n",
      "MSE 0.12309304948348057 0.13243375115071154\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'learning_rate': 0.1,\n",
       "  'max_depth': 1,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.11952736672561466,\n",
       "  'msetr': 0.12841955343258007,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.1,\n",
       "  'max_depth': 1,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.11949081835876278,\n",
       "  'msetr': 0.12800020775942245,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.1,\n",
       "  'max_depth': 2,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.11005260932751748,\n",
       "  'msetr': 0.11449586709006311,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.1,\n",
       "  'max_depth': 2,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.10579365894338559,\n",
       "  'msetr': 0.11001812163105405,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.1,\n",
       "  'max_depth': 3,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.09878988052062682,\n",
       "  'msetr': 0.10056851339771275,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.1,\n",
       "  'max_depth': 3,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.08781969095003936,\n",
       "  'msetr': 0.08837837714343463,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 1,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.14663437176401278,\n",
       "  'msetr': 0.1567381426802156,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 1,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.13861640989094157,\n",
       "  'msetr': 0.1504924776928524,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 2,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.13690674397698613,\n",
       "  'msetr': 0.14609089983139759,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 2,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.13050218541652203,\n",
       "  'msetr': 0.14060863140763535,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 3,\n",
       "  'max_features': 0.2,\n",
       "  'mse': 0.12831190858899424,\n",
       "  'msetr': 0.13707657023074862,\n",
       "  'n_estimators': 200},\n",
       " {'learning_rate': 0.01,\n",
       "  'max_depth': 3,\n",
       "  'max_features': 0.6,\n",
       "  'mse': 0.12309304948348057,\n",
       "  'msetr': 0.13243375115071154,\n",
       "  'n_estimators': 200}]"
      ]
     },
     "execution_count": 434,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds=[]\n",
    "for p in ParameterGrid(param_grid):\n",
    "    print(p)\n",
    "    gb = GradientBoostingRegressor(n_estimators=200)\n",
    "    gb.set_params(**p)\n",
    "    gb.fit(Xtrain, ytrain)\n",
    "    ypred = gb.predict(Xvalid)\n",
    "    ypredtrain = gb.predict(Xtrain)\n",
    "    d = p.copy()\n",
    "    d['n_estimators']=200\n",
    "    d['mse'] = mean_squared_error(ypred, yvalid)\n",
    "    d['msetr'] = mean_squared_error(ypredtrain, ytrain)\n",
    "    print(\"MSE\", d['mse'], d['msetr'])\n",
    "    ds.append(d)\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 435,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>max_features</th>\n",
       "      <th>mse</th>\n",
       "      <th>msetr</th>\n",
       "      <th>n_estimators</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.10</td>\n",
       "      <td>3</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.087820</td>\n",
       "      <td>0.088378</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.10</td>\n",
       "      <td>3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.098790</td>\n",
       "      <td>0.100569</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.10</td>\n",
       "      <td>2</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.105794</td>\n",
       "      <td>0.110018</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.10</td>\n",
       "      <td>2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.110053</td>\n",
       "      <td>0.114496</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.10</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.119491</td>\n",
       "      <td>0.128000</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.10</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.119527</td>\n",
       "      <td>0.128420</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.01</td>\n",
       "      <td>3</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.123093</td>\n",
       "      <td>0.132434</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.01</td>\n",
       "      <td>3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.128312</td>\n",
       "      <td>0.137077</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.01</td>\n",
       "      <td>2</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.130502</td>\n",
       "      <td>0.140609</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.01</td>\n",
       "      <td>2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.136907</td>\n",
       "      <td>0.146091</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.01</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.138616</td>\n",
       "      <td>0.150492</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.01</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.146634</td>\n",
       "      <td>0.156738</td>\n",
       "      <td>200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    learning_rate  max_depth  max_features       mse     msetr  n_estimators\n",
       "5            0.10          3           0.6  0.087820  0.088378           200\n",
       "4            0.10          3           0.2  0.098790  0.100569           200\n",
       "3            0.10          2           0.6  0.105794  0.110018           200\n",
       "2            0.10          2           0.2  0.110053  0.114496           200\n",
       "1            0.10          1           0.6  0.119491  0.128000           200\n",
       "0            0.10          1           0.2  0.119527  0.128420           200\n",
       "11           0.01          3           0.6  0.123093  0.132434           200\n",
       "10           0.01          3           0.2  0.128312  0.137077           200\n",
       "9            0.01          2           0.6  0.130502  0.140609           200\n",
       "8            0.01          2           0.2  0.136907  0.146091           200\n",
       "7            0.01          1           0.6  0.138616  0.150492           200\n",
       "6            0.01          1           0.2  0.146634  0.156738           200"
      ]
     },
     "execution_count": 435,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dsdf = pd.DataFrame.from_records(ds)\n",
    "dsdf.sort_values('mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Multi-Layer Perceptron Model\n",
    "\n",
    "This is based on the 3rd prize winning entry, whose authors wrote a [paper](https://arxiv.org/pdf/1604.06737.pdf) afterwords.\n",
    "\n",
    "What we are first going to do is to reduce the cardinality dimensionality of our categorocals by using **embeddings**. This is a technique used often in recommender systems(via matrix factorization), but also in NLP models such as `word2vec`. The idea is to map down to a smaller latent space.\n",
    "\n",
    "Here we divide the carnality by 2 and add 1 to get the embedding (this is a heuristic). If the cardinality is high, we clamp the size of the latent space down at 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 404,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Assortment': (3, 2),\n",
       " 'CloudCover_missing': (2, 2),\n",
       " 'CompetitionDistance_missing': (2, 2),\n",
       " 'CompetitionMonthsOpen': (25, 13),\n",
       " 'CompetitionOpenSinceYear': (23, 12),\n",
       " 'Day': (31, 16),\n",
       " 'DayOfWeek': (7, 4),\n",
       " 'Events': (22, 12),\n",
       " 'Month': (12, 7),\n",
       " 'Promo': (2, 2),\n",
       " 'Promo2SinceYear': (8, 5),\n",
       " 'Promo2Weeks': (26, 14),\n",
       " 'Promo_bw': (6, 4),\n",
       " 'Promo_fw': (6, 4),\n",
       " 'SchoolHoliday': (2, 2),\n",
       " 'SchoolHoliday_bw': (8, 5),\n",
       " 'SchoolHoliday_fw': (8, 5),\n",
       " 'State': (12, 7),\n",
       " 'StateHoliday': (2, 2),\n",
       " 'StateHoliday_bw': (8, 5),\n",
       " 'StateHoliday_fw': (8, 5),\n",
       " 'Store': (1115, 50),\n",
       " 'StoreType': (4, 3),\n",
       " 'Week': (52, 27),\n",
       " 'Year': (3, 2)}"
      ]
     },
     "execution_count": 404,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cards={}\n",
    "for k in nacols_cat:\n",
    "    cards[k] = (2,2)\n",
    "for k in cat_vars :\n",
    "    embed_sz_base = trdf[k].unique().size//2 + 1\n",
    "    embed_sz = (embed_sz_base <=50)*embed_sz_base + 50*((embed_sz_base > 50))\n",
    "    cards[k] = (trdf[k].unique().size, embed_sz)\n",
    "cards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to be careful (very book-keepy) in constructing a model in Keras. We use the Keras Functional API as opposed to the Sequential API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.models import Model as KerasModel\n",
    "from keras.layers import Input, Dense, Activation, Reshape\n",
    "from keras.layers import Concatenate\n",
    "from keras.layers.embeddings import Embedding\n",
    "\n",
    "def build_keras_model():\n",
    "    input_cat = []\n",
    "    output_embeddings = []\n",
    "    for k in nacols_cat+cat_vars:\n",
    "        print('{}_embedding'.format(k))\n",
    "        input_1d = Input(shape=(1,))\n",
    "        output_1d = Embedding(cards[k][0], cards[k][1], name='{}_embedding'.format(k))(input_1d)\n",
    "        output = Reshape(target_shape=(cards[k][1],))(output_1d)\n",
    "        input_cat.append(input_1d)\n",
    "        output_embeddings.append(output)\n",
    "\n",
    "    main_input = Input(shape=(len(cont_vars),), name='main_input')\n",
    "    output_model = Concatenate()([main_input, *output_embeddings])\n",
    "    output_model = Dense(1000, kernel_initializer=\"uniform\")(output_model)\n",
    "    output_model = Activation('relu')(output_model)\n",
    "    output_model = Dense(500, kernel_initializer=\"uniform\")(output_model)\n",
    "    output_model = Activation('relu')(output_model)\n",
    "    output_model = Dense(1)(output_model)\n",
    "    #output_model = Activation('sigmoid')(output_model)\n",
    "    #kmodel = KerasModel(inputs=input_model, outputs=output_model)\n",
    "    kmodel = KerasModel(\n",
    "        inputs=[*input_cat, main_input], \n",
    "        outputs=output_model#[main_output, output_embeddings]\n",
    ")\n",
    "    kmodel.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    return kmodel\n",
    "\n",
    "def fitmodel(kmodel, Xtr, ytr, Xval, yval, epochs, bs):\n",
    "    h = kmodel.fit(Xtr, ytr, validation_data=(Xval, yval),\n",
    "                       epochs=epochs, batch_size=bs)\n",
    "    return h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data input needs to match our construction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_cat_trains=[]\n",
    "list_cat_valids=[]\n",
    "catlen=len(nacols_cat+cat_vars)\n",
    "for i in range(catlen):\n",
    "    list_cat_trains.append(Xtrain[:,i])\n",
    "    list_cat_valids.append(Xvalid[:,i])\n",
    "cont_train=Xtrain[:,catlen:]\n",
    "cont_valid=Xvalid[:,catlen:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 428,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(802943, 14)"
      ]
     },
     "execution_count": 428,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cont_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we run (only a little bit for now!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CompetitionDistance_missing_embedding\n",
      "CloudCover_missing_embedding\n",
      "Store_embedding\n",
      "DayOfWeek_embedding\n",
      "Year_embedding\n",
      "Month_embedding\n",
      "Day_embedding\n",
      "StateHoliday_embedding\n",
      "CompetitionMonthsOpen_embedding\n",
      "Promo2Weeks_embedding\n",
      "StoreType_embedding\n",
      "Assortment_embedding\n",
      "CompetitionOpenSinceYear_embedding\n",
      "Promo2SinceYear_embedding\n",
      "State_embedding\n",
      "Week_embedding\n",
      "Events_embedding\n",
      "Promo_fw_embedding\n",
      "Promo_bw_embedding\n",
      "StateHoliday_fw_embedding\n",
      "StateHoliday_bw_embedding\n",
      "SchoolHoliday_fw_embedding\n",
      "SchoolHoliday_bw_embedding\n",
      "Promo_embedding\n",
      "SchoolHoliday_embedding\n",
      "Train on 802943 samples, validate on 41395 samples\n",
      "Epoch 1/2\n",
      "802943/802943 [==============================] - 216s 269us/step - loss: 0.2259 - val_loss: 0.0265\n",
      "Epoch 2/2\n",
      "802943/802943 [==============================] - 209s 260us/step - loss: 0.0193 - val_loss: 0.0236\n"
     ]
    }
   ],
   "source": [
    "emodel = build_keras_model()\n",
    "history = fitmodel(emodel, [*list_cat_trains, cont_train], ytrain, [*list_cat_valids, cont_valid], yvalid, 2, 256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homework\n",
    "\n",
    "Lets do the GBM on dask. And later, at your leisure, you'll want to `ParameterGrid` Keras."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}