lectures/notebooks/Lecture_10_Matplotlib_Data_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Fundamentals of Information Systems\n",
    "\n",
    "## Python Programming (for Data Science)\n",
    "\n",
    "### Master's Degree in Data Science\n",
    "\n",
    "#### Gabriele Tolomei\n",
    "<a href=\"mailto:gtolomei@math.unipd.it\">gtolomei@math.unipd.it</a><br/>\n",
    "University of Padua, Italy<br/>\n",
    "2018/2019<br/>\n",
    "November, 26 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 10: Data Visualization with <code>matplotlib</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Motivation\n",
    "\n",
    "-  Making informative visualizations (sometimes called **plots**) is one of the most important tasks in data analysis. \n",
    "\n",
    "-  This may be a part of the \"exploratory process\"; for example, helping identify outliers, needed data transformations, or coming up with ideas for models. \n",
    "\n",
    "-  Python has many add-on libraries for making static or dynamic visualizations, but we will be focusing on [<code>**matplotlib**</code>](http://matplotlib.sourceforge.net) and libraries that build on top of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is <code>matplotlib</code>\n",
    "\n",
    "-  <code>**matplotlib**</code> is a desktop plotting package designed for creating (mostly two-dimensional) plots. \n",
    "\n",
    "-  The project was started by John Hunter in 2002 to enable a MATLAB-like plotting interface in Python. \n",
    "\n",
    "-  The matplotlib and IPython communities have collaborated to simplify interactive plotting from the IPython shell, as well as Jupyter notebook. \n",
    "\n",
    "-  It supports various GUI backends on all operating systems and additionally can export graphics to all of the common vector and raster graphics formats: **PDF**, **SVG**, **JPG**, **PNG**, **BMP**, **GIF**, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A <code>matplotlib</code> API Primer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Again, matplotlib is included in Anaconda and the only thing we need\n",
    "to do for using it is just to import it!\n",
    "Actually, we need to import a \"sub-module\" of the whole matplotlib library, \n",
    "i.e., matplotlib.pyplot (https://matplotlib.org/api/pyplot_summary.html)\n",
    "Let's also import both numpy and pandas, as those might be needed as well.\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "# Adding the following line, allows Jupyter Notebook to visualize plots\n",
    "# produced by matplotlib directly below the code cell which generated those.\n",
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's try to make our first matplotlib plot\n",
    "# When 'plot' is given with just a single input argument, \n",
    "# this is considered as y values, \n",
    "# whilst x values are implicitly taken from the array [0..len(y)-1]\n",
    "# '_' is a convention which is used in CPython interpreter to hold the result \n",
    "# of the last executed expression/statement in an interactive interpreter session.\n",
    "_ = plt.plot(np.arange(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Figures and Subplots\n",
    "\n",
    "-  Plots in <code>**matplotlib**</code> are encapsulated within a <code>**Figure**</code> object. \n",
    "\n",
    "-  A new <code>**Figure**</code> object can be created with <code>**plt.figure**</code>.\n",
    "\n",
    "-  In IPython, an empty plot window will appear, but in Jupyter Notebook nothing will be shown until we use a few more commands. \n",
    "\n",
    "-  <code>**plt.figure**</code> has a number of options, notably <code>**figsize**</code> will guarantee the figure has a certain size and aspect ratio if saved to disk.\n",
    "\n",
    "-  We can't make a plot with a blank figure. We have to create one or more subplots using <code>**add_subplot**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a Figure object\n",
    "fig = plt.figure()\n",
    "# Add a 2x2 subplot and selecting the 1st one (numbering starts from 1)\n",
    "ax1 = fig.add_subplot(2, 2, 1)\n",
    "# Add a 2x2 subplot and selecting the 2nd one\n",
    "ax2 = fig.add_subplot(2, 2, 2)\n",
    "# Add a 2x2 subplot and selecting the 3rd one\n",
    "ax3 = fig.add_subplot(2, 2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If we issue a plt.plot command, matplotlib will draw on the last figure and subplot used\n",
    "(creating one if necessary), thus hiding the figure and subplot creation\n",
    "\"\"\"\n",
    "# Create a Figure object\n",
    "fig = plt.figure()\n",
    "# Add a 2x2 subplot and selecting the 1st one (numbering starts from 1)\n",
    "ax1 = fig.add_subplot(2, 2, 1)\n",
    "# Add a 2x2 subplot and selecting the 2nd one\n",
    "ax2 = fig.add_subplot(2, 2, 2)\n",
    "# Add a 2x2 subplot and selecting the 3rd one\n",
    "ax3 = fig.add_subplot(2, 2, 3)\n",
    "_ = plt.plot(np.arange(42))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "The objects returned by fig.add_subplot above are AxesSubplot objects, \n",
    "We can directly plot on the other empty subplots by calling each one's instance methods.\n",
    "\"\"\"\n",
    "# Set the seed for random experiments\n",
    "np.random.seed(42)\n",
    "# Create a Figure object\n",
    "fig = plt.figure()\n",
    "# Add a 2x2 subplot and selecting the 1st one (numbering starts from 1)\n",
    "ax1 = fig.add_subplot(2, 2, 1)\n",
    "# Add a 2x2 subplot and selecting the 2nd one\n",
    "ax2 = fig.add_subplot(2, 2, 2)\n",
    "# Add a 2x2 subplot and selecting the 3rd one\n",
    "ax3 = fig.add_subplot(2, 2, 3)\n",
    "# This will plot on the latest AxesSubplot (i.e., ax3)\n",
    "_ = plt.plot(np.arange(42))\n",
    "# Call the instance method on each AxesSubplot\n",
    "# The 'k--' is a style option instructing matplotlib to plot a black dashed line.\n",
    "_ = ax1.plot(np.random.randn(50).cumsum(), 'k--')\n",
    "_ = ax2.scatter(np.arange(30), np.arange(30) + 3 * np.random.randn(30))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Since creating a figure with a grid of subplots is such a common task, \n",
    "there is a convenience method 'plt.subplots' that creates a new figure \n",
    "and returns a numpy array containing the created subplot objects.\n",
    "\"\"\"\n",
    "# Create a figure which is made of 6 subplots (2x3)\n",
    "fig, axes = plt.subplots(2, 3)\n",
    "axes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "This is very useful as the axes array can be easily indexed like a two-dimensional array; \n",
    "for example, axes[0, 1].\n",
    "We can also indicate that subplots should have the same X or Y axis \n",
    "using 'sharex' and 'sharey', respectively. This is especially useful \n",
    "when comparing data on the same scale.\n",
    "Otherwise, matplotlib auto-scales plot limits independently.\n",
    "\"\"\"\n",
    "# Set the seed for random experiments\n",
    "np.random.seed(42)\n",
    "# Create a figure whic is made of 6 subplots (2x3)\n",
    "fig, axes = plt.subplots(2, 3)\n",
    "# Plot on the bottom-right corner subplot \n",
    "_ = axes[1, 2].hist(np.random.randn(100), bins=20, color='k', alpha=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>pyplot.subplots</code> Arguments\n",
    "\n",
    "<center>![](./img/matplotlib_subplots.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Colors, Markers, and Line Styles\n",
    "\n",
    "-  The main <code>**plot**</code> function accepts arrays of $x$ and $y$ coordinates, and optionally a string abbreviation indicating color and line style. \n",
    "\n",
    "-  For example, to plot $x$ versus $y$ with green dashes, you would execute:\n",
    "\n",
    "```python\n",
    "ax.plot(x, y, 'g--')\n",
    "```\n",
    "Or, equivalently:\n",
    "```python\n",
    "ax.plot(x, y, linestyle='--', color='g')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Line plots can additionally have markers to highlight the actual data points. \n",
    "Since matplotlib creates a continuous line plot, interpolating between points, \n",
    "it can occasionally be unclear where the points lie. \n",
    "The marker can be part of the style string, which must have color \n",
    "followed by marker type and line style.\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "_ = plt.plot(np.random.randn(30).cumsum(), 'ko--')\n",
    "# Alternatively, we can write the above explicitly as follows\n",
    "#plt.plot(np.random.randn(30).cumsum(), color='k', linestyle='dashed', marker='o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "If we want to change the default interpolation using line plots,\n",
    "we can specify the argument 'drawstyle'\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "\n",
    "data = np.random.randn(30).cumsum()\n",
    "# First plot using the default interpolation\n",
    "_ = plt.plot(data, 'r--', label='Default')\n",
    "# Second plot (on the same Figure object) using steps\n",
    "_ = plt.plot(data, 'b-', drawstyle='steps-post', label='steps-post')\n",
    "# Insert the legend located in the best possible position\n",
    "_ = plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ticks, Labels, and Legends\n",
    "\n",
    "-  2 possible ways of decorating plots: using the procedural  <code>**matplotlib.pyplot**</code> interface or object-oriented **matplotlib API**.\n",
    "\n",
    "-  The pyplot interface exposes methods like <code>**xlim**</code> (range), <code>**xticks**</code> (tick locations), and <code>**xticklabels**</code> (tick labels).\n",
    "\n",
    "-  They can be used in two ways:\n",
    "    -  with no arguments returns the current parameter value. For example, <code>**plt.xlim()**</code> returns the current $x$-axis plotting range;\n",
    "    -  with parameters sets the parameter value. So, <code>**plt.xlim([0, 10])**</code>, sets the $x$-axis range to 0 to 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ticks, Labels, and Legends\n",
    "\n",
    "-  All such methods act on the active or most recently-created <code>**AxesSubplot**</code>. \n",
    "\n",
    "-  Each of them corresponds to two methods on the subplot object itself.\n",
    "\n",
    "-  In the case of <code>**xlim**</code> these are <code>**ax.get_xlim**</code> and <code>**ax.set_xlim**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's clarify the above concepts with an example.\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "# Create a Figure object\n",
    "fig = plt.figure()\n",
    "# Create a 1x1 subplot and select this\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "# Draw some random plot\n",
    "_ = ax.plot(np.random.randn(1000).cumsum())\n",
    "# To change the x-axis ticks, we use 'set_xticks' and 'set_xticklabels'. \n",
    "# The former instructs matplotlib where to place the ticks along the data range; \n",
    "# by default these locations will also be the labels. \n",
    "ticks = ax.set_xticks([0, 250, 500, 750, 1000])\n",
    "# But we can set any other values as the labels using set_xticklabels\n",
    "labels = ax.set_xticklabels(['ONE', 'TWO', 'THREE', 'FOUR', 'FIVE'],\n",
    "                            rotation=30, fontsize='small')\n",
    "# We can also set a title to the plot\n",
    "_ = ax.set_title('My first matplotlib plot')\n",
    "# And x-axis (y-axis) labels\n",
    "_ = ax.set_xlabel('Stages'); _ = ax.set_ylabel('Value')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Suppose we have multiple plots on the same Figure (or AxesSubplot).\n",
    "Legends may help us distinguish between those plots.\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "fig = plt.figure() \n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "_ = ax.plot(np.random.randn(1000).cumsum(), 'r', label='A')\n",
    "_ = ax.plot(np.random.randn(1000).cumsum(), 'g--', label='B')\n",
    "_ = ax.plot(np.random.randn(1000).cumsum(), 'b.', label='C')\n",
    "_ = ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##  Saving Plots to File\n",
    "\n",
    "-  The active figure can be saved to file using <code>**plt.savefig**</code>. \n",
    "\n",
    "-  This method is equivalent to the figure object's <code>**savefig**</code> instance method. \n",
    "\n",
    "-  For example, to save an **SVG** version of a figure, we can do as follows:\n",
    "```python\n",
    "plt.savefig('path/to/plotfile.svg')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Saving Plots to File\n",
    "\n",
    "-  The file type is inferred from the file extension.\n",
    "\n",
    "-  Other important options might be: \n",
    "    -  <code>**dpi**</code>, which controls the dots-per-inch resolution;\n",
    "    -  <code>**bbox_inches**</code>, which can trim the whitespace around the actual figure. \n",
    "    \n",
    "-  To get the same plot as a **PNG** above with minimal whitespace around the plot and at 400 DPI:\n",
    "```\n",
    "plt.savefig('path/to/plotfile.png', dpi=400, bbox_inches='tight')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>savefig<code> Arguments\n",
    "\n",
    "<center>![](./img/matplotlib_savefig.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>matplotlib</code> Configuration\n",
    "\n",
    "-  <code>**matplotlib**</code> comes configured with **color schemes** and defaults options. \n",
    "\n",
    "-  Nearly all of the default behavior can be customized via an extensive set of global parameters governing figure size, subplot spacing, colors, font sizes, grid styles, and so on. \n",
    "\n",
    "-  <code>**matplotlib**</code> configuration system can be accessed either programmatically from Python using the <code>**rc**</code> method, or using an external configuration file (e.g., <code>**~/.matplotlibrc**</code>)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Plotting with <code>pandas</code> and <code>seaborn</code>\n",
    "\n",
    "-  <code>**matplotlib**</code> can be a fairly low level tool, which allows us to assemble a plot from its base components: the data, the type of plot (*line*, *bar*, *box*, *scatter*, *contour*, etc.), legend, title, tick labels, and other annotations. \n",
    " \n",
    "-  In <code>**pandas**</code> we may have multiple columns of data, along with row and column labels.\n",
    "\n",
    "-  <code>**pandas**</code> has built-in methods which simplify creating visualizations from <code>**DataFrame**</code> and <code>**Series**</code> objects. \n",
    "\n",
    "-  On top of that, there is another useful statistical graphics library, called [<code>**seaborn**</code>](https://seaborn.pydata.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Line Plots\n",
    "\n",
    "-  <code>**Series**</code> and <code>**DataFrame**</code> each have a <code>**plot**</code> attribute for making some basic plot types. \n",
    "\n",
    "-  By default, <code>**plot()**</code> makes **line plots**.\n",
    "\n",
    "-  Most of <code>**pandas**</code>'s plotting methods accept an optional <code>**ax**</code> parameter, which can be a matplotlib subplot object. This gives you more flexible placement of subplots in a grid layout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Default line plots with pandas Series\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "# Create a Series\n",
    "s = pd.Series(np.random.randn(10).cumsum(), index=np.arange(0, 100, 10))\n",
    "_ = s.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Line Plots with <code>Series</code>\n",
    "\n",
    "-  The <code>**Series**</code> object's index is passed to <code>**matplotlib**</code> for plotting on the $x$-axis (this can be disabled by passing <code>**use_index=False**</code>). \n",
    "\n",
    "-  The $x$-axis ticks and limits can be adjusted using the <code>**xticks**</code> and <code>**xlim**</code> options, and $y$-axis respectively using <code>**yticks**</code> and <code>**ylim**</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Default line plots with pandas DataFrame\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "# Create a DataFrame\n",
    "df = pd.DataFrame(np.random.randn(10, 4).cumsum(0),\n",
    "                  columns=['A', 'B', 'C', 'D'],\n",
    "                  index=np.arange(0, 100, 10))\n",
    "_ = df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Line Plots with <code>DataFrame</code>\n",
    "\n",
    "-  <code>**DataFrame**</code>'s <code>**plot**</code> method plots each of its columns as a different line on the same subplot, creating a legend automatically.\n",
    "\n",
    "-  The <code>**plot**</code> attribute contains a \"family\" of methods for different plot types. \n",
    "\n",
    "-  For example, <code>**DataFrame.plot()**</code> is equivalent to <code>**DataFrame.plot.line()**</code>. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>pandas.plot</code> Arguments (1 of 2)\n",
    "\n",
    "<center>![](./img/pandas_plot_1.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>pandas.plot</code> Arguments (2 of 2)\n",
    "\n",
    "<center>![](./img/pandas_plot_2.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <code>pandas.plot</code> Arguments (<code>DataFrame</code>-specific)\n",
    "\n",
    "<center>![](./img/pandas_plot_df.png)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bar Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "The plot.bar() and plot.barh() make vertical and horizontal bar plots, respectively. \n",
    "In this case, the Series or DataFrame index will be used as the x (bar) or y (barh) tick.\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "# Create a figure and subplot associated with it\n",
    "fig, axes = plt.subplots(2, 1)\n",
    "# Create a random series of values indexed by letter (i.e., chars)\n",
    "data = pd.Series(np.random.rand(16), index=list('abcdefghijklmnop'))\n",
    "# Specify the subplot where to place this bar plot\n",
    "_ = data.plot.bar(ax=axes[0], color='r', alpha=0.7)\n",
    "# Specify the subplot where to place this horizontal bar plot\n",
    "_ = data.plot.barh(ax=axes[1], color='c', alpha=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "With a DataFrame, bar plots a group of bars for each row.\n",
    "Fixed a row, each group of bars is made of the values contained in that row, side by side\n",
    "(i.e., each bar group contains as many values as the number of columns).\n",
    "\"\"\"\n",
    "np.random.seed(42)\n",
    "\n",
    "df = pd.DataFrame(np.random.rand(6, 4),\n",
    "                  index=['one', 'two', 'three', 'four', 'five', 'six'],\n",
    "                  columns=pd.Index(['A', 'B', 'C', 'D'], name='Genus'))\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "_ = df.plot.bar()\n",
    "# Note that the name \"Genus\" on the DataFrame's columns is used to title the legend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Stacked bar plots are created from a DataFrame by passing stacked=True,\n",
    "resulting in the value in each row being stacked together.\n",
    "\"\"\"\n",
    "_ = df.plot.bar(stacked=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Stacked bar plots are created from a DataFrame by passing stacked=True,\n",
    "resulting in the value in each row being stacked together.\n",
    "\"\"\"\n",
    "# The same as above, yet horizontal\n",
    "_ = df.plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Histograms and Density Plots\n",
    "\n",
    "-  A **histogram** is a kind of bar plot that gives a discretized display of value frequency. \n",
    "\n",
    "-  The data points are split into discrete, evenly spaced **bins**, and the number of data points in each bin is plotted. \n",
    "\n",
    "-  It can be obtained by calling the <code>**plot.hist**</code> method on a <code>**Series**</code> object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Histograms and Density Plots\n",
    "\n",
    "-  A related plot type is a **density plot**, which is formed by computing an **_estimate_** of a **continuous probability distribution** that might have generated the observed data. \n",
    "\n",
    "-  A usual procedure is to approximate this distribution as a **mixture of \"kernels\"**, that is, simpler distributions like the Normal distribution. \n",
    "\n",
    "-  Thus, density plots are also known as **KDE** (kernel density estimate) plots. \n",
    "\n",
    "-  Using <code>**plot.kde**</code> makes a density plot using the conventional mixture-of-normals estimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's go back to our running example DataFrame of users.\n",
    "\"\"\"\n",
    "url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/u.user'\n",
    "# The first line of the file represents the header, and each field\n",
    "# is separated by a pipe\n",
    "\"\"\"\n",
    "We specify the url where the data is located, the character used to separate fields ('|')\n",
    "and the name of the column to use as row label (otherwise, RangeInteger will be used)\n",
    "\"\"\"\n",
    "users = pd.read_csv(url, sep='|', index_col='user_id')\n",
    "print(users.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 1. For each user, extract an average salary value drawn from a Normal distribution.\n",
    "# In other words, we need to drawn a sample from a Normal distribution, \n",
    "# whose size is equal to that of the users in our dataset.\n",
    "# To do so, we need to specify the two parameters of the distribution: \n",
    "# mean (mu) and standard deviation (sigma)\n",
    "np.random.seed(42)\n",
    "mu = 35000\n",
    "sigma = 10000\n",
    "normal_salaries = np.random.normal(loc=mu, scale=sigma, size=users.shape[0])\n",
    "# Let's also round salaries to the 2nd decimal digit\n",
    "normal_salaries = np.round(normal_salaries, decimals=2)\n",
    "# Alternatively:\n",
    "# np.round(normal_salaries, decimals=2, out=normal_salaries)\n",
    "print(normal_salaries[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# 2. Let's add an extra 'salary' column to our 'users' DataFrame\n",
    "# using the just created numpy array\n",
    "users['salary'] = normal_salaries\n",
    "print(users.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's first plot the histogram corresponding to the distribution\n",
    "of 'age' values from our users DataFrame\n",
    "\"\"\"\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10,12))\n",
    "# Using pandas' instance object method 'plot.hist()'\n",
    "_ = users['age'].plot.hist(ax=axes[0,0], color='r')\n",
    "_ = axes[0,0].set_xlabel('Age'); _ = axes[0,0].set_ylabel('Frequency')\n",
    "# Achieving a similar behavior using value_counts() and bar plot\n",
    "_ = users['age'].value_counts(bins=6).sort_index().plot.bar(ax=axes[0,1], color='r', rot=45)\n",
    "_ = axes[0,1].set_xlabel('Age Group'); _ = axes[0,1].set_ylabel('Frequency')\n",
    "# Using matplotlib API\n",
    "_ = axes[1,0].hist(users['age'], color='b')\n",
    "_ = axes[1,0].set_xlabel('Age'); _ = axes[1,0].set_ylabel('Frequency')\n",
    "\n",
    "_ = axes[1,1].bar(users['age'].value_counts().sort_index().index, \n",
    "                  users['age'].value_counts().sort_index().values, color='b')\n",
    "_ = axes[1,1].set_xlabel('Age'); _ = axes[1,1].set_ylabel('Frequency')\n",
    "plt.subplots_adjust(wspace=.3, hspace=.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# NOTE: If we call .hist() on value_counts() we obtain the histogram of the histogram!\n",
    "# In other words, we obtain the frequency distribution of frequency values\n",
    "fig, axes = plt.subplots(2, 1, figsize=(6,8))\n",
    "_ = users['age'].value_counts().sort_index().plot.hist(ax=axes[0], color='g')\n",
    "_ = axes[1].hist(users['age'].value_counts().sort_index(), color='g')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(4,4))\n",
    "# Plot on the same AxesSubplot both histogram and density estimate\n",
    "_ = users.salary.plot.hist(color='k', alpha=.2)\n",
    "_ = users.salary.plot.density(ax=ax, color='r', secondary_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Seaborn makes histograms and density plots even easier through its 'distplot' method, \n",
    "which can plot both a histogram and a continuous density estimate simultaneously.\n",
    "\"\"\"\n",
    "comp1 = np.random.normal(0, 1, size=200)  # Normal(0, 1)\n",
    "comp2 = np.random.normal(10, 2, size=200)  # Normal(10, 4)\n",
    "# Create a Pandas Series from the two normally-distributed samples above\n",
    "values = pd.Series(np.concatenate([comp1, comp2]))\n",
    "_ = sns.distplot(values, bins=100, color='m')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualize Pairwise Relationships\n",
    "\n",
    "-  Point plots (also sometimes called **scatter plots**) can be a useful way of examining the relationship between two one-dimensional series. \n",
    "\n",
    "- It may be useful to look at _all_ the scatter plots among a group of variables; this is known as a **pairs plot** or **scatter plot matrix**. \n",
    "\n",
    "-  <code>**seaborn**</code> has a convenient <code>**pairplot**</code> function; it supports placing histograms or density estimates of each variable along the diagonal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Let's load the iris dataset which comes along with seaborn and\n",
    "use it to demonstrate how 'pairplot' works.\n",
    "\"\"\"\n",
    "# Loading iris dataset\n",
    "iris = sns.load_dataset(\"iris\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(iris, hue=\"species\", markers=[\"o\", \"s\", \"D\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(iris, kind=\"reg\",\n",
    "                 diag_kind='hist',\n",
    "                 diag_kws={'color': 'c', 'alpha': 0.4}, \n",
    "                 plot_kws={'color': 'm'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "_ = sns.pairplot(iris, diag_kind='kde', \n",
    "                 diag_kws={'shade': True, 'color': 'r'}, \n",
    "                 plot_kws={'alpha': 0.4, 'color': 'r'})"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}