vol_to_non.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This notebook answers the following question(s):  \n",
    "\n",
    "- What is the probability of volunteer making a donation later without volunteering ?\n",
    "- What is the correlation between donating and volunteering ?\n",
    "\n",
    "\n",
    "by Fred Etter - December, 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "from sklearn import linear_model\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import preprocessing\n",
    "import sklearn\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from datetime import datetime\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the data\n",
    "df = pd.read_csv('file1_12_3.csv', low_memory=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year</th>\n",
       "      <th>PledgeTotal</th>\n",
       "      <th>VolType</th>\n",
       "      <th>VolunteerActivityCnt</th>\n",
       "      <th>BirthYear</th>\n",
       "      <th>Gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>874ddbce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>52.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1977.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>9a4ddbce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>80.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1968.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5e4edbce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>10000.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1958.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>c150dbce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>120.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0b53dbce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>500.16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1947.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              ContactId  Year  PledgeTotal VolType  \\\n",
       "0  874ddbce-11cd-e111-941f-00259073dc22  2007        52.00     NaN   \n",
       "1  9a4ddbce-11cd-e111-941f-00259073dc22  2007        80.00     NaN   \n",
       "2  5e4edbce-11cd-e111-941f-00259073dc22  2007     10000.00     NaN   \n",
       "3  c150dbce-11cd-e111-941f-00259073dc22  2007       120.00     NaN   \n",
       "4  0b53dbce-11cd-e111-941f-00259073dc22  2007       500.16     NaN   \n",
       "\n",
       "   VolunteerActivityCnt  BirthYear  Gender  \n",
       "0                   NaN     1977.0  female  \n",
       "1                   NaN     1968.0  female  \n",
       "2                   NaN     1958.0    male  \n",
       "3                   NaN        NaN    male  \n",
       "4                   NaN     1947.0  female  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the first 5 lines of the data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(385722, 7)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the number of rows and columns of the original data\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop Council, Committe, Board members\n",
    "df = df.drop(df[df.VolType == 'Council, Committe or Board'].index).fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year</th>\n",
       "      <th>PledgeTotal</th>\n",
       "      <th>VolType</th>\n",
       "      <th>VolunteerActivityCnt</th>\n",
       "      <th>BirthYear</th>\n",
       "      <th>Gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>135262</td>\n",
       "      <td>98b73991-702e-4027-99ea-31a9713c7d5a</td>\n",
       "      <td>2012</td>\n",
       "      <td>52.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>322854</td>\n",
       "      <td>86320cce-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2013</td>\n",
       "      <td>1091.31</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>344935</td>\n",
       "      <td>bc3cfab3-ecaf-e511-a59d-26d4160798d6</td>\n",
       "      <td>2015</td>\n",
       "      <td>130.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1987.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>103699</td>\n",
       "      <td>e469b7d0-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2008</td>\n",
       "      <td>240.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1959.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>369499</td>\n",
       "      <td>7f0653fb-4789-e911-812e-0050569e2a4f</td>\n",
       "      <td>2018</td>\n",
       "      <td>25.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>373211</td>\n",
       "      <td>5657e8cf-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2019</td>\n",
       "      <td>858.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>249385</td>\n",
       "      <td>3c3915d2-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2014</td>\n",
       "      <td>550.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>277375</td>\n",
       "      <td>cbab9504-ac89-e611-a7f7-26d4160798d6</td>\n",
       "      <td>2018</td>\n",
       "      <td>24.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>57447</td>\n",
       "      <td>03c555d0-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2014</td>\n",
       "      <td>50.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>192763</td>\n",
       "      <td>dbcc55d0-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2007</td>\n",
       "      <td>130.00</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   ContactId  Year  PledgeTotal VolType  \\\n",
       "135262  98b73991-702e-4027-99ea-31a9713c7d5a  2012        52.00       0   \n",
       "322854  86320cce-11cd-e111-941f-00259073dc22  2013      1091.31       0   \n",
       "344935  bc3cfab3-ecaf-e511-a59d-26d4160798d6  2015       130.00       0   \n",
       "103699  e469b7d0-11cd-e111-941f-00259073dc22  2008       240.00       0   \n",
       "369499  7f0653fb-4789-e911-812e-0050569e2a4f  2018        25.00       0   \n",
       "373211  5657e8cf-11cd-e111-941f-00259073dc22  2019       858.00       0   \n",
       "249385  3c3915d2-11cd-e111-941f-00259073dc22  2014       550.00       0   \n",
       "277375  cbab9504-ac89-e611-a7f7-26d4160798d6  2018        24.00       0   \n",
       "57447   03c555d0-11cd-e111-941f-00259073dc22  2014        50.00       0   \n",
       "192763  dbcc55d0-11cd-e111-941f-00259073dc22  2007       130.00       0   \n",
       "\n",
       "        VolunteerActivityCnt  BirthYear  Gender  \n",
       "135262                   0.0        0.0  female  \n",
       "322854                   0.0        0.0    male  \n",
       "344935                   0.0     1987.0  female  \n",
       "103699                   0.0     1959.0  female  \n",
       "369499                   0.0        0.0  female  \n",
       "373211                   0.0        0.0    male  \n",
       "249385                   0.0        0.0    male  \n",
       "277375                   0.0        0.0  female  \n",
       "57447                    0.0        0.0  female  \n",
       "192763                   0.0        0.0    male  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_non_only = df.drop(df[df.VolType == 'Standard'].index).fillna(0)\n",
    "df_std_only = df.drop(df[df.VolType == 0].index).fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a series that captures the earliest year that a non-volunteer donated\n",
    "a = df_std_only.groupby('ContactId')['Year'].transform('min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new column and populate it:\n",
    "df_std_only['min_year'] = a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year</th>\n",
       "      <th>PledgeTotal</th>\n",
       "      <th>VolType</th>\n",
       "      <th>VolunteerActivityCnt</th>\n",
       "      <th>BirthYear</th>\n",
       "      <th>Gender</th>\n",
       "      <th>min_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>383750</td>\n",
       "      <td>3dd755d0-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2013</td>\n",
       "      <td>1352.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1984.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>383325</td>\n",
       "      <td>a1fd81ac-fc25-4ea7-b7c2-ca3d8f39f486</td>\n",
       "      <td>2017</td>\n",
       "      <td>347.88</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>376435</td>\n",
       "      <td>4bfa078e-ab77-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>381533</td>\n",
       "      <td>d7d255d0-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2013</td>\n",
       "      <td>504.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>384702</td>\n",
       "      <td>bb23a584-3475-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>376000</td>\n",
       "      <td>8f55b1d1-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2011</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>384522</td>\n",
       "      <td>fdbc3c6c-4456-e511-97d1-26d4160798d6</td>\n",
       "      <td>2016</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>380193</td>\n",
       "      <td>1e5f22ab-149e-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1969.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>375652</td>\n",
       "      <td>e34cb1d1-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2011</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>383295</td>\n",
       "      <td>e1fdf2d4-62f7-439e-a9c8-d49a22a64d89</td>\n",
       "      <td>2017</td>\n",
       "      <td>130.00</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   ContactId  Year  PledgeTotal   VolType  \\\n",
       "383750  3dd755d0-11cd-e111-941f-00259073dc22  2013      1352.00  Standard   \n",
       "383325  a1fd81ac-fc25-4ea7-b7c2-ca3d8f39f486  2017       347.88  Standard   \n",
       "376435  4bfa078e-ab77-e211-a0e0-4040184c1c1a  2012         0.00  Standard   \n",
       "381533  d7d255d0-11cd-e111-941f-00259073dc22  2013       504.00  Standard   \n",
       "384702  bb23a584-3475-e211-a0e0-4040184c1c1a  2013         0.00  Standard   \n",
       "376000  8f55b1d1-11cd-e111-941f-00259073dc22  2011         0.00  Standard   \n",
       "384522  fdbc3c6c-4456-e511-97d1-26d4160798d6  2016         0.00  Standard   \n",
       "380193  1e5f22ab-149e-e211-a0e0-4040184c1c1a  2012         0.00  Standard   \n",
       "375652  e34cb1d1-11cd-e111-941f-00259073dc22  2011         0.00  Standard   \n",
       "383295  e1fdf2d4-62f7-439e-a9c8-d49a22a64d89  2017       130.00  Standard   \n",
       "\n",
       "        VolunteerActivityCnt  BirthYear  Gender  min_year  \n",
       "383750                   1.0     1984.0  female      2010  \n",
       "383325                   1.0        0.0  female      2017  \n",
       "376435                   1.0        0.0  female      2012  \n",
       "381533                   1.0        0.0    male      2013  \n",
       "384702                   2.0        0.0    male      2012  \n",
       "376000                   1.0        0.0  female      2011  \n",
       "384522                   2.0        0.0  female      2015  \n",
       "380193                   1.0     1969.0  female      2012  \n",
       "375652                   1.0        0.0  female      2011  \n",
       "383295                   1.0        0.0  female      2017  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display a random sample of 10 rows of the dataframe\n",
    "df_std_only.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop rows where Year != min_year\n",
    "df_std_only = df_std_only.drop(df_std_only[df_std_only.Year != df_std_only.min_year].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year</th>\n",
       "      <th>PledgeTotal</th>\n",
       "      <th>VolType</th>\n",
       "      <th>VolunteerActivityCnt</th>\n",
       "      <th>BirthYear</th>\n",
       "      <th>Gender</th>\n",
       "      <th>min_year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>376750</td>\n",
       "      <td>593f15d2-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2011</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>383599</td>\n",
       "      <td>25ef9166-3132-4cc5-ac2d-6f3f3ceeb89f</td>\n",
       "      <td>2014</td>\n",
       "      <td>130.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>382553</td>\n",
       "      <td>1b6cf804-3a75-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>381485</td>\n",
       "      <td>20b962d1-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2013</td>\n",
       "      <td>50.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>382548</td>\n",
       "      <td>905ffb29-8ea3-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>378736</td>\n",
       "      <td>9adfc3a4-7224-e711-98e2-005056975e92</td>\n",
       "      <td>2017</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>379800</td>\n",
       "      <td>23334c7c-a231-e711-b10c-005056975e92</td>\n",
       "      <td>2017</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>male</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>376342</td>\n",
       "      <td>245ce8cf-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>377983</td>\n",
       "      <td>c345e8cf-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2016</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1967.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>376102</td>\n",
       "      <td>4827ede2-76f0-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2011</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   ContactId  Year  PledgeTotal   VolType  \\\n",
       "376750  593f15d2-11cd-e111-941f-00259073dc22  2011          0.0  Standard   \n",
       "383599  25ef9166-3132-4cc5-ac2d-6f3f3ceeb89f  2014        130.0  Standard   \n",
       "382553  1b6cf804-3a75-e211-a0e0-4040184c1c1a  2012          0.0  Standard   \n",
       "381485  20b962d1-11cd-e111-941f-00259073dc22  2013         50.0  Standard   \n",
       "382548  905ffb29-8ea3-e211-a0e0-4040184c1c1a  2012          0.0  Standard   \n",
       "378736  9adfc3a4-7224-e711-98e2-005056975e92  2017          0.0  Standard   \n",
       "379800  23334c7c-a231-e711-b10c-005056975e92  2017          0.0  Standard   \n",
       "376342  245ce8cf-11cd-e111-941f-00259073dc22  2012          0.0  Standard   \n",
       "377983  c345e8cf-11cd-e111-941f-00259073dc22  2016          0.0  Standard   \n",
       "376102  4827ede2-76f0-e211-a0e0-4040184c1c1a  2011          0.0  Standard   \n",
       "\n",
       "        VolunteerActivityCnt  BirthYear  Gender  min_year  \n",
       "376750                   1.0        0.0    male      2011  \n",
       "383599                   1.0        0.0    male      2014  \n",
       "382553                   1.0        0.0    male      2012  \n",
       "381485                   1.0        0.0    male      2013  \n",
       "382548                   1.0        0.0  female      2012  \n",
       "378736                   1.0        0.0  female      2017  \n",
       "379800                   1.0        0.0    male      2017  \n",
       "376342                   1.0        0.0  female      2012  \n",
       "377983                   1.0     1967.0  female      2016  \n",
       "376102                   1.0        0.0  female      2011  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_std_only.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6147, 8)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_std_only.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_std_only = df_std_only.drop_duplicates('ContactId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6147, 8)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_std_only.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(df[df.VolType == 'Standard'].index).fillna(0)\n",
    "df = df.drop_duplicates('ContactId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(115554, 7)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new dataframe\n",
    "new_df = pd.merge(df_std_only, df, how='left', left_on=['ContactId'], right_on = ['ContactId'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6147, 14)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year_x</th>\n",
       "      <th>PledgeTotal_x</th>\n",
       "      <th>VolType_x</th>\n",
       "      <th>VolunteerActivityCnt_x</th>\n",
       "      <th>BirthYear_x</th>\n",
       "      <th>Gender_x</th>\n",
       "      <th>min_year</th>\n",
       "      <th>Year_y</th>\n",
       "      <th>PledgeTotal_y</th>\n",
       "      <th>VolType_y</th>\n",
       "      <th>VolunteerActivityCnt_y</th>\n",
       "      <th>BirthYear_y</th>\n",
       "      <th>Gender_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3481</td>\n",
       "      <td>a41f8a92-e3e3-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4164</td>\n",
       "      <td>bc50e8cf-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2012</td>\n",
       "      <td>120.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1950.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1950.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3848</td>\n",
       "      <td>f09b654f-7e96-e411-b065-26d4160798d6</td>\n",
       "      <td>2018</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1966.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2018</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>450.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1966.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2323</td>\n",
       "      <td>d2e261ab-e86b-e311-a454-4040184c1c1a</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2013</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2157</td>\n",
       "      <td>a548e8cf-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2013</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1958.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2013</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1958.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 ContactId  Year_x  PledgeTotal_x VolType_x  \\\n",
       "3481  a41f8a92-e3e3-e211-a0e0-4040184c1c1a    2013            0.0  Standard   \n",
       "4164  bc50e8cf-11cd-e111-941f-00259073dc22    2012          120.0  Standard   \n",
       "3848  f09b654f-7e96-e411-b065-26d4160798d6    2018            0.0  Standard   \n",
       "2323  d2e261ab-e86b-e311-a454-4040184c1c1a    2013            0.0  Standard   \n",
       "2157  a548e8cf-11cd-e111-941f-00259073dc22    2013            0.0  Standard   \n",
       "\n",
       "      VolunteerActivityCnt_x  BirthYear_x Gender_x  min_year  Year_y  \\\n",
       "3481                     1.0          0.0   female      2013     NaN   \n",
       "4164                     1.0       1950.0   female      2012  2007.0   \n",
       "3848                     1.0       1966.0   female      2018  2014.0   \n",
       "2323                     1.0          0.0   female      2013     NaN   \n",
       "2157                     1.0       1958.0   female      2013  2007.0   \n",
       "\n",
       "      PledgeTotal_y  VolType_y  VolunteerActivityCnt_y  BirthYear_y Gender_y  \n",
       "3481            NaN        NaN                     NaN          NaN      NaN  \n",
       "4164           50.0        0.0                     0.0       1950.0   female  \n",
       "3848          450.0        0.0                     0.0       1966.0   female  \n",
       "2323            NaN        NaN                     NaN          NaN      NaN  \n",
       "2157          100.0        0.0                     0.0       1958.0   female  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df_final = new_df[new_df.Year_x <= new_df.Year_y]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(501, 14)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df_final.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ContactId</th>\n",
       "      <th>Year_x</th>\n",
       "      <th>PledgeTotal_x</th>\n",
       "      <th>VolType_x</th>\n",
       "      <th>VolunteerActivityCnt_x</th>\n",
       "      <th>BirthYear_x</th>\n",
       "      <th>Gender_x</th>\n",
       "      <th>min_year</th>\n",
       "      <th>Year_y</th>\n",
       "      <th>PledgeTotal_y</th>\n",
       "      <th>VolType_y</th>\n",
       "      <th>VolunteerActivityCnt_y</th>\n",
       "      <th>BirthYear_y</th>\n",
       "      <th>Gender_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>4652</td>\n",
       "      <td>e1a59537-3ec3-e811-8101-0050569e2a4f</td>\n",
       "      <td>2018</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2018</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>600.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1843</td>\n",
       "      <td>6053b1d1-11cd-e111-941f-00259073dc22</td>\n",
       "      <td>2010</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1981.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2010</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>208.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1981.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5847</td>\n",
       "      <td>34ca3f5e-d3dc-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2013</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2013</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4406</td>\n",
       "      <td>b826039d-86a3-e211-a0e0-4040184c1c1a</td>\n",
       "      <td>2012</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2012</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>1040.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1074</td>\n",
       "      <td>25712864-6f87-e311-a454-4040184c1c1a</td>\n",
       "      <td>2014</td>\n",
       "      <td>1020.0</td>\n",
       "      <td>Standard</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1987.0</td>\n",
       "      <td>female</td>\n",
       "      <td>2014</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1987.0</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 ContactId  Year_x  PledgeTotal_x VolType_x  \\\n",
       "4652  e1a59537-3ec3-e811-8101-0050569e2a4f    2018            0.0  Standard   \n",
       "1843  6053b1d1-11cd-e111-941f-00259073dc22    2010            0.0  Standard   \n",
       "5847  34ca3f5e-d3dc-e211-a0e0-4040184c1c1a    2013         1000.0  Standard   \n",
       "4406  b826039d-86a3-e211-a0e0-4040184c1c1a    2012            0.0  Standard   \n",
       "1074  25712864-6f87-e311-a454-4040184c1c1a    2014         1020.0  Standard   \n",
       "\n",
       "      VolunteerActivityCnt_x  BirthYear_x Gender_x  min_year  Year_y  \\\n",
       "4652                     1.0          0.0        0      2018  2019.0   \n",
       "1843                     1.0       1981.0   female      2010  2013.0   \n",
       "5847                     2.0          0.0   female      2013  2014.0   \n",
       "4406                     1.0          0.0   female      2012  2017.0   \n",
       "1074                     1.0       1987.0   female      2014  2015.0   \n",
       "\n",
       "      PledgeTotal_y  VolType_y  VolunteerActivityCnt_y  BirthYear_y Gender_y  \n",
       "4652          600.0        0.0                     0.0          0.0        0  \n",
       "1843          208.0        0.0                     0.0       1981.0   female  \n",
       "5847         1000.0        0.0                     0.0          0.0   female  \n",
       "4406         1040.0        0.0                     0.0          0.0   female  \n",
       "1074         1000.0        0.0                     0.0       1987.0   female  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df_final.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08150317227916057"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df_final.shape[0]/new_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion:\n",
    "\n",
    "So, the probability of a volunteer making a donation in a later year without volunteering is about **8.2%**.\n",
    "\n",
    "One caveat: The supporter may make a donation later in 2019 or after 2019.\n",
    "\n",
    "In a previous notebook, it was established that a non-volunteer is about **2.2%** likely to volunteer later.  What do these 2 facts say about the correlation between volunteering and non-volunteering (but making a donation) ?  \n",
    "\n",
    "In short, not much.  \n",
    "\n",
    "It is fair to say that a volunteer is much more likely to make a donation than someone in the general population because presumably, the percentage of donators in the general population is much less than 8.2%.  \n",
    "\n",
    "It is also fair to say that a non-volunteer who makes a donation is more likely to volunteer than someone in the general population because 2.2% would be higher than the volunteer rate of the total population.  \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}