predicting the yearly number of doctor visits with linear regression.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Predicting the yearly number of doctor visits using linear regression"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n#importing data on which we will train the model\ndt=pd.read_csv('../input/learning-lab-mini-challenge-doctors-visits/train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Describe the data"},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"dt.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataframe contains 7 columns:  <br>\n**target variable:** 'visits' numerical dtype: integer <br>\n**numerical features:** age , weight, height dtype: float64 <br>\n**categorical features:** gender, ethnicity dtype: object i.e string/text <br>\n**id** column is not informative but is required in submission <br>\n**number of observations:** 1000 entries <br>\n**memory size:** 54.8+ KB"},{"metadata":{},"cell_type":"markdown","source":"### **Explore more data types**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dt['visits'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Target variable: visits**\n<br> In this sampling, people visit the doctor at least 3 times and at most 30 times a year. <br>the average number of doctor visits is 11."},{"metadata":{"trusted":true},"cell_type":"code","source":"dt['visits'].head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\nsns.set(color_codes=True)\nplt.hist(dt['visits'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(dt['visits'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sample distribution of the feature visits is right-skewed. There are no outliers."},{"metadata":{},"cell_type":"markdown","source":"**Check for null and missing values**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt=dt.dropna() #if you wirte just dt.dropna(), the resulting dataframe will not be memorized and the dt will contain the missing values until you assign it to itself","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"dt.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"62 rows were deleted because they contained missing values. 6,2% of the dataset was eliminated."},{"metadata":{},"cell_type":"markdown","source":"**Randomly mix the data**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.utils import shuffle\nimport random\nrandom.seed(40)\ndt=shuffle(dt,random_state=np.random)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check if data was really randomized"},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encoding categorical features and target numerical feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"#dt_enc is the dataframe that will contain the data including the encoded features\ndt_enc=pd.get_dummies(dt, columns=['gender', 'ethnicity'], dummy_na=False,drop_first=True)\ndt_enc.head(10)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data should be scaled before modeling**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nscaler=StandardScaler()\nscaler.fit(dt_enc)\nscaled = scaler.transform(dt_enc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt_enc.plot(x='age', y='visits', style='o')  \nplt.title('Age vs visits')  \nplt.xlabel('age')  \nplt.ylabel('visits')  \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.plot(x='gender', y='visits', style='o')  \nplt.title('gender vs visits')  \nplt.xlabel('gender')  \nplt.ylabel('visits')  \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.plot(x='ethnicity', y='visits', style='o')  \nplt.title('ethnicity vs visits')  \nplt.xlabel('ethnicity')  \nplt.ylabel('visits')  \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt_enc.corr()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of visits in this sample has the highest positive correlation with the variable age. Visits number is moderately (and positively) related to age"},{"metadata":{},"cell_type":"markdown","source":"### Model the data with simple regression of age as dependent regressor of visits "},{"metadata":{"trusted":true},"cell_type":"code","source":"X = dt_enc[['age', 'ethnicity_group D']]\nY = dt_enc['visits']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, Y_train, Y_test=train_test_split(X,Y, test_size=0.3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import linear_model\nregr = linear_model.LinearRegression()\nregr.fit(X_train, Y_train)\npred = regr.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"regr.intercept_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"regr.coef_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The impact of age on the yearly number of visits is 0.137 <br>\nThe impact of being of ethnicity group D or not on the yearly number of visits is 0.744"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\nimport numpy as np\nprint(np.sqrt(metrics.mean_squared_error(Y_test,pred)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Apply a different linear regression model "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"features","execution_count":48,"outputs":[{"output_type":"execute_result","execution_count":48,"data":{"text/plain":"Index(['id', 'age', 'weight', 'height', 'visits', 'gender_male',\n       'ethnicity_group B', 'ethnicity_group C', 'ethnicity_group D',\n       'ethnicity_group E'],\n      dtype='object')"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt_enc = dt_enc.rename(columns = {'ethnicity_group B': 'ethnicB', 'ethnicity_group C': 'ethnicC','ethnicity_group D': 'ethnicD', 'ethnicity_group E': 'ethnicE'}, errors = 'raise')","execution_count":60,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"import seaborn as sns\nfeatures=dt_enc.columns\nfeatures.tolist()\nsns.pairplot(dt_enc, x_vars=dt_enc[['ethnicity_group B', 'ethnicity_group C','ethnicity_group D']], y_vars=dt_enc['visits'], height=7, aspect=0.7)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import statsmodels.formula.api as smf\nmodel2 = smf.ols(formula='visits ~ age + weight + gender_male', data=dt_enc).fit()\n","execution_count":51,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.params","execution_count":52,"outputs":[{"output_type":"execute_result","execution_count":52,"data":{"text/plain":"Intercept      4.537979\nage            0.141099\nweight         0.003195\ngender_male   -0.066068\ndtype: float64"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"dt_enc.head(4)","execution_count":61,"outputs":[{"output_type":"execute_result","execution_count":61,"data":{"text/plain":"      id   age  weight  height  visits  gender_male  ethnicB  ethnicC  \\\n691  692  32.0   244.0    62.0       7            0        1        0   \n106  107  18.0   223.0    63.0      20            0        1        0   \n50    51  80.0   264.0    62.0      14            1        1        0   \n850  851  37.0   108.0    68.0      18            0        0        0   \n\n     ethnicD  ethnicE  \n691        0        0  \n106        0        0  \n50         0        0  \n850        0        1  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>age</th>\n      <th>weight</th>\n      <th>height</th>\n      <th>visits</th>\n      <th>gender_male</th>\n      <th>ethnicB</th>\n      <th>ethnicC</th>\n      <th>ethnicD</th>\n      <th>ethnicE</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>691</th>\n      <td>692</td>\n      <td>32.0</td>\n      <td>244.0</td>\n      <td>62.0</td>\n      <td>7</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>106</th>\n      <td>107</td>\n      <td>18.0</td>\n      <td>223.0</td>\n      <td>63.0</td>\n      <td>20</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>50</th>\n      <td>51</td>\n      <td>80.0</td>\n      <td>264.0</td>\n      <td>62.0</td>\n      <td>14</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>850</th>\n      <td>851</td>\n      <td>37.0</td>\n      <td>108.0</td>\n      <td>68.0</td>\n      <td>18</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2 = smf.ols(formula='visits ~ age + weight + ethnicD + ethnicE', data=dt_enc).fit()","execution_count":62,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.params","execution_count":63,"outputs":[{"output_type":"execute_result","execution_count":63,"data":{"text/plain":"Intercept    4.474973\nage          0.139602\nweight       0.002419\nethnicD      0.817810\nethnicE      0.355255\ndtype: float64"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.rsquared","execution_count":64,"outputs":[{"output_type":"execute_result","execution_count":64,"data":{"text/plain":"0.2849309704963229"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.summary()","execution_count":65,"outputs":[{"output_type":"execute_result","execution_count":65,"data":{"text/plain":"<class 'statsmodels.iolib.summary.Summary'>\n\"\"\"\n                            OLS Regression Results                            \n==============================================================================\nDep. Variable:                 visits   R-squared:                       0.285\nModel:                            OLS   Adj. R-squared:                  0.282\nMethod:                 Least Squares   F-statistic:                     92.94\nDate:                Thu, 28 May 2020   Prob (F-statistic):           1.51e-66\nTime:                        00:03:16   Log-Likelihood:                -2455.8\nNo. Observations:                 938   AIC:                             4922.\nDf Residuals:                     933   BIC:                             4946.\nDf Model:                           4                                         \nCovariance Type:            nonrobust                                         \n==============================================================================\n                 coef    std err          t      P>|t|      [0.025      0.975]\n------------------------------------------------------------------------------\nIntercept      4.4750      0.570      7.854      0.000       3.357       5.593\nage            0.1396      0.007     18.779      0.000       0.125       0.154\nweight         0.0024      0.002      1.144      0.253      -0.002       0.007\nethnicD        0.8178      0.258      3.175      0.002       0.312       1.323\nethnicE        0.3553      0.334      1.062      0.288      -0.301       1.012\n==============================================================================\nOmnibus:                      208.832   Durbin-Watson:                   1.997\nProb(Omnibus):                  0.000   Jarque-Bera (JB):              374.927\nSkew:                           1.362   Prob(JB):                     3.85e-82\nKurtosis:                       4.475   Cond. No.                     1.13e+03\n==============================================================================\n\nWarnings:\n[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n[2] The condition number is large, 1.13e+03. This might indicate that there are\nstrong multicollinearity or other numerical problems.\n\"\"\"","text/html":"<table class=\"simpletable\">\n<caption>OLS Regression Results</caption>\n<tr>\n  <th>Dep. Variable:</th>         <td>visits</td>      <th>  R-squared:         </th> <td>   0.285</td>\n</tr>\n<tr>\n  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.282</td>\n</tr>\n<tr>\n  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   92.94</td>\n</tr>\n<tr>\n  <th>Date:</th>             <td>Thu, 28 May 2020</td> <th>  Prob (F-statistic):</th> <td>1.51e-66</td>\n</tr>\n<tr>\n  <th>Time:</th>                 <td>00:03:16</td>     <th>  Log-Likelihood:    </th> <td> -2455.8</td>\n</tr>\n<tr>\n  <th>No. Observations:</th>      <td>   938</td>      <th>  AIC:               </th> <td>   4922.</td>\n</tr>\n<tr>\n  <th>Df Residuals:</th>          <td>   933</td>      <th>  BIC:               </th> <td>   4946.</td>\n</tr>\n<tr>\n  <th>Df Model:</th>              <td>     4</td>      <th>                     </th>     <td> </td>   \n</tr>\n<tr>\n  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n</tr>\n</table>\n<table class=\"simpletable\">\n<tr>\n      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n</tr>\n<tr>\n  <th>Intercept</th> <td>    4.4750</td> <td>    0.570</td> <td>    7.854</td> <td> 0.000</td> <td>    3.357</td> <td>    5.593</td>\n</tr>\n<tr>\n  <th>age</th>       <td>    0.1396</td> <td>    0.007</td> <td>   18.779</td> <td> 0.000</td> <td>    0.125</td> <td>    0.154</td>\n</tr>\n<tr>\n  <th>weight</th>    <td>    0.0024</td> <td>    0.002</td> <td>    1.144</td> <td> 0.253</td> <td>   -0.002</td> <td>    0.007</td>\n</tr>\n<tr>\n  <th>ethnicD</th>   <td>    0.8178</td> <td>    0.258</td> <td>    3.175</td> <td> 0.002</td> <td>    0.312</td> <td>    1.323</td>\n</tr>\n<tr>\n  <th>ethnicE</th>   <td>    0.3553</td> <td>    0.334</td> <td>    1.062</td> <td> 0.288</td> <td>   -0.301</td> <td>    1.012</td>\n</tr>\n</table>\n<table class=\"simpletable\">\n<tr>\n  <th>Omnibus:</th>       <td>208.832</td> <th>  Durbin-Watson:     </th> <td>   1.997</td>\n</tr>\n<tr>\n  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td> 374.927</td>\n</tr>\n<tr>\n  <th>Skew:</th>          <td> 1.362</td>  <th>  Prob(JB):          </th> <td>3.85e-82</td>\n</tr>\n<tr>\n  <th>Kurtosis:</th>      <td> 4.475</td>  <th>  Cond. No.          </th> <td>1.13e+03</td>\n</tr>\n</table><br/><br/>Warnings:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The condition number is large, 1.13e+03. This might indicate that there are<br/>strong multicollinearity or other numerical problems."},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"model3 = smf.ols(formula='visits ~ age', data=dt_enc).fit()","execution_count":69,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model3.rsquared","execution_count":70,"outputs":[{"output_type":"execute_result","execution_count":70,"data":{"text/plain":"0.27546663255740467"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"The linear regression models do not fit this dataset because the r squared remains low (around .28) when we use the significantly related features (features that have small p-values : age & ethnicD)\n<br>\nInsight: encode the 'visits' variable (which is numeric) to class intervals ( results in three classes/ binary variables ), then try classification models on the data\nTry imputing the missing values and reapply regression and classification to see if the 6% deleted observations impact the quality of predictions"},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}