higher_order_functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Higher Order Functions\n",
    "\n",
    "In this assignment, I practiced using Python's higher order functions to quickly process two data sets. The goal was to use map, filter, and reduce (without the assistance of global variables) to perform various calculations on the data that could be applied at scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "from functools import reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Student Performance\n",
    "\n",
    "In this task, I used HOFs and NYC's graduation outcomes data set to find the correlation between the percentage of students who dropped out and the percentage of students who graduated with advanced regents for schools in NYC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Bronx', '2001', 0.08713874094123811, 0.21286999039552956),\n",
       " ('Bronx', '2002', 0.08244680851063829, 0.1778590425531915),\n",
       " ('Bronx', '2003', 0.09206279342723005, 0.1813380281690141),\n",
       " ('Bronx', '2004', 0.09711779448621553, 0.16033138401559455),\n",
       " ('Bronx', '2005', 0.10174629324546952, 0.1414827018121911),\n",
       " ('Bronx', '2006', 0.10000641889723345, 0.1541819115475961),\n",
       " ('Brooklyn', '2001', 0.14172636641450828, 0.1776965081909724),\n",
       " ('Brooklyn', '2002', 0.1376874279123414, 0.16190888119953864),\n",
       " ('Brooklyn', '2003', 0.15182338051935876, 0.14990156557607576),\n",
       " ('Brooklyn', '2004', 0.1673600858945108, 0.13300228157294322),\n",
       " ('Brooklyn', '2005', 0.16201692714164168, 0.11544489722806861),\n",
       " ('Brooklyn', '2006', 0.1676060783694819, 0.12314560129864274),\n",
       " ('Manhattan', '2001', 0.14609313338595106, 0.1548539857932123),\n",
       " ('Manhattan', '2002', 0.13904776052885687, 0.1294659436975414),\n",
       " ('Manhattan', '2003', 0.18207363642913754, 0.1245766986094099),\n",
       " ('Manhattan', '2004', 0.18582666754809282, 0.12176902227804588),\n",
       " ('Manhattan', '2005', 0.1687180458246544, 0.10080161585558291),\n",
       " ('Manhattan', '2006', 0.16940789473684212, 0.10258284600389864),\n",
       " ('Queens', '2001', 0.15836811474927986, 0.15848568573276114),\n",
       " ('Queens', '2002', 0.15534990691052458, 0.15419997809659403),\n",
       " ('Queens', '2003', 0.18436057561770297, 0.14759706760792832),\n",
       " ('Queens', '2004', 0.19246995994659546, 0.13377837116154873),\n",
       " ('Queens', '2005', 0.1854338578237917, 0.12480139408538773),\n",
       " ('Queens', '2006', 0.18595970958175684, 0.11534921771142244),\n",
       " ('Staten Island', '2001', 0.2262396694214876, 0.10769628099173553),\n",
       " ('Staten Island', '2002', 0.20827285921625543, 0.10304789550072568),\n",
       " ('Staten Island', '2003', 0.20934091986723566, 0.08866761498340446),\n",
       " ('Staten Island', '2004', 0.248430709802028, 0.09198454852728151),\n",
       " ('Staten Island', '2005', 0.2374439461883408, 0.08116591928251121),\n",
       " ('Staten Island', '2006', 0.25896154681729305, 0.08994134260265045)]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the total for each borough\n",
    "def filterer(x):\n",
    "    if x['Demographic'] == 'Borough Total':\n",
    "        return x\n",
    "\n",
    "# Get the borough, cohort year, and fraction of students with advanced regents and fraction who dropped out\n",
    "def mapper(x):\n",
    "    return x['Borough'], x['Cohort'],\\\n",
    "    float(x['Advanced Regents'])/float(x['Total Cohort']), float(x['Dropped Out'])/float(x['Total Cohort'])\n",
    "\n",
    "# Apply both functions using map and filter\n",
    "with open('data/nyc_grads.csv', 'r') as fi:\n",
    "    reader = csv.DictReader(fi)\n",
    "    output1 = map(mapper, filter(filterer, reader))\n",
    "        \n",
    "output1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Overall Rates\n",
    "\n",
    "Using the output above, I then found the average rates for each borough over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Bronx', 0.0934198082513375, 0.17134384308218617),\n",
       " ('Brooklyn', 0.15470337770864048, 0.14351662251104022),\n",
       " ('Manhattan', 0.16519452307558916, 0.1223416853729485),\n",
       " ('Queens', 0.1769903541049419, 0.13903528573260707),\n",
       " ('Staten Island', 0.23144827521877342, 0.09375060031471814)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dict with each borough as a key and a dict of the total advanced regents rates, dropout rates, and row count\n",
    "# as the value\n",
    "def reducer(x, y):\n",
    "    value = x.get(y[0], {'count':0,'ara':0,'ard':0})\n",
    "    value['count'] += 1\n",
    "    value['ara'] += y[2]\n",
    "    value['ard'] += y[3]\n",
    "    x[y[0]] = value    \n",
    "    return x\n",
    "\n",
    "# Return only each borough, the average advanced regents rate, and average dropout rate\n",
    "def mapper2(x):\n",
    "    return x[0], x[1]['ara']/x[1]['count'], x[1]['ard']/x[1]['count']\n",
    "\n",
    "output2 = sorted(map(mapper2, reduce(reducer, output1, {}).items()))\n",
    "output2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Sales Data\n",
    "\n",
    "Here, I used map and reduce to read a csv with sales data and output the number of unique customers who bought each product and the total reveue brought in per product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('P02291', 16, 1181.97),\n",
       " ('P19498', 17, 989.99),\n",
       " ('P32565', 17, 1006.09),\n",
       " ('P33162', 18, 1210.92),\n",
       " ('P39328', 17, 1129.01),\n",
       " ('P58225', 17, 1349.82),\n",
       " ('P61235', 18, 959.02),\n",
       " ('P76615', 18, 1087.96),\n",
       " ('P82222', 17, 950.05),\n",
       " ('P92449', 14, 966.17)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Store all the product ids and add every unique customer ID and ever revenue value.\n",
    "def sale_reduce(out_dict, in_data):\n",
    "    value = out_dict.get(in_data['Product ID'], {'customers':set(),'revenue':0})\n",
    "    value['customers'].add(in_data['Customer ID'])\n",
    "    value['revenue'] += float(in_data['Item Cost'])\n",
    "    out_dict[in_data['Product ID']] = value \n",
    "    return out_dict\n",
    "\n",
    "# Return each product ID and get the number of unique customers plus the total revenue\n",
    "def sale_mapper(in_dict):\n",
    "    return in_dict[0], len(in_dict[1]['customers']), round(in_dict[1]['revenue'], 2)\n",
    "\n",
    "# Apply both functions to the data\n",
    "with open('data/sale.csv', 'r') as fi:\n",
    "    reader = csv.DictReader(fi)\n",
    "    output3 = sorted(map(sale_mapper, reduce(sale_reduce, reader, {}).items()))\n",
    "\n",
    "output3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myEnv",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}