notebooks/02.06-Boolean-Arrays-and-Masks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparisons, Masks, and Boolean Logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chapter covers the use of Boolean masks to examine and manipulate values within NumPy arrays.\n",
    "Masking comes up when you want to extract, modify, count, or otherwise manipulate values in an array based on some criterion: for example, you might wish to count all values greater than a certain value, or remove all outliers that are above some threshold.\n",
    "In NumPy, Boolean masking is often the most efficient way to accomplish these types of tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Counting Rainy Days\n",
    "\n",
    "Imagine you have a series of data that represents the amount of precipitation each day for a year in a given city.\n",
    "For example, here we'll load the daily rainfall statistics for the city of Seattle in 2015, using Pandas (see [Part 3](03.00-Introduction-to-Pandas.ipynb)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "365"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from vega_datasets import data\n",
    "\n",
    "# Use DataFrame operations to extract rainfall as a NumPy array\n",
    "rainfall_mm = np.array(\n",
    "    data.seattle_weather().set_index('date')['precipitation']['2015'])\n",
    "len(rainfall_mm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The array contains 365 values, giving daily rainfall in millimeters from January 1 to December 31, 2015.\n",
    "\n",
    "As a first quick visualization, let's look at the histogram of rainy days in the following figure, which was generated using Matplotlib (we will explore this tool more fully in [Part 4](04.00-Introduction-To-Matplotlib.ipynb)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('seaborn-whitegrid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD0CAYAAABtjRZ7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAARgUlEQVR4nO3dfWiVBf/H8c+1s0rbA2NI0LDZTKOpWMxh/5w0KJ2/7syKjakxwZmp2bEVyeacU9k0xfIfnyq5gx9amNMI4VdYCiLLRw6pOI9FaJZuBLbE7bR5dOf6/XHTuVP3oNvZw/fy/fqrXdfZte+X5O3V6TrMcV3XFQDApIT+HgAA0H1EHAAMI+IAYBgRBwDDiDgAGEbEAcCwxL78YcFgsC9/HAB4xrhx49o93qcRlzoe5E6EQiFlZ2fHcZqBgb3s8epuXt1Lsr1bZzfAvJ0CAIYRcQAwjIgDgGFEHAAMI+IAYBgRBwDDiDgAGEbEAcCwPv+wT0/8z/+ek3Su3XO/rPlX3w4DAAMAd+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADCs00cMr1+/rvLycl26dEmRSEQLFizQww8/rHnz5unRRx+VJM2YMUMvvPCCNm7cqAMHDigxMVHl5eUaO3ZsX8wPAPe0TiO+Z88epaWlad26dbpy5YpefvllLVy4ULNnz1ZxcXHsdXV1dTp27JhqamrU0NCgQCCg3bt39/rwAHCv6zTiU6ZMUV5eniTJdV35fD6dPn1a58+f1/79+zVs2DCVl5crGAzK7/fLcRxlZGSora1NjY2NSk9P75MlAOBe1WnEk5KSJEnNzc1atGiRSkpKFIlEVFBQoDFjxmjLli3atGmTUlJSlJaWdtP3NTU1tRvxUCgU3w16+bp9obW11fT8HfHqXpJ3d/PqXpJ3d+vyY/cNDQ1auHChZs6cqalTp+rq1atKTU2VJE2aNElVVVV67rnnFA6HY98TDoeVkpLS7vV69jvu2v/Ifc+v278s/+6/znh1L8m7u3l1L8n2bt3+HZuXL19WcXGxFi9erPz8fEnSnDlzdOrUKUnS4cOHNXr0aOXk5Ki2tlbRaFT19fWKRqO8lQIAfaDTO/GPPvpIV69e1ebNm7V582ZJUllZmVavXq377rtPQ4YMUVVVlZKTk5Wbm6vCwkJFo1FVVlb2yfAAcK/rNOIVFRWqqKi47fiOHTtuOxYIBBQIBOI3GQCgS3zYBwAMI+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADCPiAGAYEQcAw4g4ABhGxAHAMCIOAIYRcQAwjIgDgGFEHAAMI+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADAssbOT169fV3l5uS5duqRIJKIFCxZoxIgRKisrk+M4GjlypJYvX66EhARt3LhRBw4cUGJiosrLyzV27Ni+2gEA7lmdRnzPnj1KS0vTunXrdOXKFb388st64oknVFJSoqefflqVlZXav3+/MjIydOzYMdXU1KihoUGBQEC7d+/uqx0A4J7VacSnTJmivLw8SZLruvL5fKqrq9P48eMlSRMmTND333+vrKws+f1+OY6jjIwMtbW1qbGxUenp6b2/AQDcwzqNeFJSkiSpublZixYtUklJidauXSvHcWLnm5qa1NzcrLS0tJu+r6mpqd2Ih0KhOI7f+9ftC62trabn74hX95K8u5tX95K8u1unEZekhoYGLVy4UDNnztTUqVO1bt262LlwOKzU1FQlJycrHA7fdDwlJaXd62VnZ/dg3HMdnunZdftXKBQyPX9HvLqX5N3dvLqXZHu3YDDY4blOn065fPmyiouLtXjxYuXn50uSRo0apaNHj0qSDh48qNzcXOXk5Ki2tlbRaFT19fWKRqO8lQIAfaDTO/GPPvpIV69e1ebNm7V582ZJ0tKlS1VdXa3169dr+PDhysvLk8/nU25urgoLCxWNRlVZWdknwwPAva7TiFdUVKiiouK249u3b7/tWCAQUCAQiN9kAIAu8WEfADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADCPiAGAYEQcAw4g4ABhGxAHAMCIOAIYRcQAwjIgDgGFEHAAMI+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCG3VHET548qaKiIknSmTNn9Mwzz6ioqEhFRUX6+uuvJUkbN25Ufn6+pk+frlOnTvXexACAmMSuXrB161bt2bNHgwcPliTV1dVp9uzZKi4ujr2mrq5Ox44dU01NjRoaGhQIBLR79+7emxoAIOkO7sQzMzO1YcOG2NenT5/WgQMH9Nprr6m8vFzNzc0KBoPy+/1yHEcZGRlqa2tTY2Njrw4OALiDO/G8vDxdvHgx9vXYsWNVUFCgMWPGaMuWLdq0aZNSUlKUlpYWe01SUpKampqUnp5+2/VCoVB8Ju+j6/aF1tZW0/N3xKt7Sd7dzat7Sd7drcuI32rSpElKTU2N/XNVVZWee+45hcPh2GvC4bBSUlLa/f7s7OxujipJ5zo807Pr9q9QKGR6/o54dS/Ju7t5dS/J9m7BYLDDc3f9dMqcOXNi/+Py8OHDGj16tHJyclRbW6toNKr6+npFo9F278IBAPF113fiK1asUFVVle677z4NGTJEVVVVSk5OVm5urgoLCxWNRlVZWdkbswIAbnFHER86dKh27twpSRo9erR27Nhx22sCgYACgUB8pwMAdIoP+wCAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADCPiAGAYEQcAw4g4ABhGxAHAMCIOAIYRcQAwjIgDgGFEHAAMI+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCG3VHET548qaKiIknShQsXNGPGDM2cOVPLly9XNBqVJG3cuFH5+fmaPn26Tp061XsTAwBiuoz41q1bVVFRoWvXrkmS3n//fZWUlOjzzz+X67rav3+/6urqdOzYMdXU1Gj9+vVauXJlrw8OALiDiGdmZmrDhg2xr+vq6jR+/HhJ0oQJE3To0CEFg0H5/X45jqOMjAy1tbWpsbGx96YGAEiSErt6QV5eni5evBj72nVdOY4jSUpKSlJTU5Oam5uVlpYWe83fx9PT02+7XigUisPYt+ut6/aF1tZW0/N3xKt7Sd7dzat7Sd7drcuI3yoh4b837+FwWKmpqUpOTlY4HL7peEpKSrvfn52d3Y0x/3auwzM9u27/CoVCpufviFf3kry7m1f3kmzvFgwGOzx310+njBo1SkePHpUkHTx4ULm5ucrJyVFtba2i0ajq6+sVjUbbvQsHAMTXXd+Jl5aWatmyZVq/fr2GDx+uvLw8+Xw+5ebmqrCwUNFoVJWVlb0xKwDgFncU8aFDh2rnzp2SpKysLG3fvv221wQCAQUCgfhOBwDoFB/2AQDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADCPiAGAYEQcAw4g4ABhGxAHAMCIOAIYRcQAwjIgDgGFEHAAMI+IAYBgRBwDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgWGJ3v/GVV15RcnKyJGno0KEqLCzUqlWr5PP55Pf79dZbb8VtSABA+7oV8WvXrsl1XW3bti12bNq0adqwYYMeeeQRvfHGGzpz5oxGjRoVt0EBALfr1tspZ8+eVUtLi4qLizVr1iwdP35ckUhEmZmZchxHfr9fhw4divesAIBbdOtOfNCgQZozZ44KCgr0yy+/aO7cuUpNTY2dT0pK0m+//dbu94ZCoe5N2oXeum5faG1tNT1/R7y6l+Td3by6l+Td3boV8aysLA0bNkyO4ygrK0spKSm6cuVK7Hw4HL4p6v+UnZ3drUH/41yHZ3p23f4VCoVMz98Rr+4leXc3r+4l2d4tGAx2eK5bb6fs2rVLa9askST9/vvvamlp0YMPPqhff/1VruuqtrZWubm53ZsWAHDHunUnnp+fryVLlmjGjBlyHEerV69WQkKC3nvvPbW1tcnv9+vJJ5+M96wAgFt0K+L333+/Pvzww9uO79y5s8cDAQDuHB/2AQDDiDgAGEbEAcAwIg4AhhFxADCMiAOAYUQcAAwj4gBgGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADCPiAGAYEQcAw7r1m30GokfL/q/T87+s+VcfTQIAfYc7cQAwjIgDgGFEHAAMI+IAYBgRBwDDPPN0Slc6e3qFJ1cAWMWdOAAYRsQBwLB75u2UnuCDRAAGKiKuriMNAAMVb6cAgGFEHAAM4+2UOIjP2zHnbjvCe+0AuhLXiEejUa1YsUI//vij7r//flVXV2vYsGHx/BEAgH+Ia8T37dunSCSiL774QidOnNCaNWu0ZcuWeP4I/EN/PTUzkJ/WGagf6hqoc8G+uEY8GAzqmWeekSQ99dRTOn36dDwvf8/pzadm+vOJnJ787J4Eryd/+bT/vf99C4wQo784ruu68brY0qVLNXnyZE2cOFGS9Oyzz2rfvn1KTPzP3xXBYDBePwoA7injxo1r93hc78STk5MVDodjX0ej0VjAOxsCANA9cX3EMCcnRwcPHpQknThxQo8//ng8Lw8AuEVc3075++mUn376Sa7ravXq1XrsscfidXkAwC3iGvHe4NXHFk+ePKkPPvhA27Zt04ULF1RWVibHcTRy5EgtX75cCQm2Pod1/fp1lZeX69KlS4pEIlqwYIFGjBhhfi9JamtrU0VFhc6fPy/HcbRy5Uo98MADnthNkv744w+9+uqr+vTTT5WYmOiZvV555RUlJydLkoYOHarCwkKtWrVKPp9Pfr9fb731Vj9PGCfuALd37163tLTUdV3X/eGHH9z58+f380Q998knn7gvvviiW1BQ4Lqu686bN889cuSI67quu2zZMvfbb7/tz/G6ZdeuXW51dbXruq77559/uhMnTvTEXq7rut99951bVlbmuq7rHjlyxJ0/f75ndotEIu6bb77pTp482f355589s1dra6s7bdq0m4699NJL7oULF9xoNOq+/vrrbl1dXf8MF2cD/q9YLz62mJmZqQ0bNsS+rqur0/jx4yVJEyZM0KFDh/prtG6bMmWK3n77bUmS67ry+Xye2EuSnn/+eVVVVUmS6uvrlZqa6pnd1q5dq+nTp+uhhx6S5I0/i5J09uxZtbS0qLi4WLNmzdLx48cViUSUmZkpx3Hk9/vN7narAR/x5ubm2H8SSZLP59ONGzf6caKey8vLu+mpHdd15TiOJCkpKUlNTU39NVq3JSUlKTk5Wc3NzVq0aJFKSko8sdffEhMTVVpaqqqqKk2dOtUTu3355ZdKT0+P3SRJ3vizKEmDBg3SnDlz9O9//1srV67UkiVLNHjw4Nh5y7vdasBHvKvHFr3gn+85hsNhpaam9uM03dfQ0KBZs2Zp2rRpmjp1qmf2+tvatWu1d+9eLVu2TNeuXYsdt7rb7t27dejQIRUVFSkUCqm0tFSNjY2x81b3kqSsrCy99NJLchxHWVlZSklJ0ZUrV2LnLe92qwEf8XvhscVRo0bp6NGjkqSDBw8qNze3nye6e5cvX1ZxcbEWL16s/Px8Sd7YS5K++uorffzxx5KkwYMHy3EcjRkzxvxun332mbZv365t27YpOztba9eu1YQJE8zvJUm7du3SmjVrJEm///67Wlpa9OCDD+rXX3+V67qqra01u9utzDyd4rXHFi9evKh3331XO3fu1Pnz57Vs2TJdv35dw4cPV3V1tXw+X3+PeFeqq6v1zTffaPjw4bFjS5cuVXV1tem9JOmvv/7SkiVLdPnyZd24cUNz587VY489Zv7f2T8VFRVpxYoVSkhI8MRekUhES5YsUX19vRzH0XvvvaeEhAStXr1abW1t8vv9euedd/p7zLgY8BEHAHRswL+dAgDoGBEHAMOIOAAYRsQBwDAiDgCGEXEAMIyIA4BhRBwADPt/qJAX01HyS6UAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(rainfall_mm, 40);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This histogram gives us a general idea of what the data looks like: despite the city's rainy reputation, the vast majority of days in Seattle saw near zero measured rainfall in 2015.\n",
    "But this doesn't do a good job of conveying some information we'd like to see: for example, how many rainy days were there in the year? What was the average precipitation on those rainy days? How many days were there with more than 10 mm of rainfall?\n",
    "\n",
    "One approach to this would be to answer these questions by hand: we could loop through the data, incrementing a counter each time we see values in some desired range.\n",
    "But for reasons discussed throughout this chapter, such an approach is very inefficient from the standpoint of both time writing code and time computing the result.\n",
    "We saw in [Computation on NumPy Arrays: Universal Functions](02.03-Computation-on-arrays-ufuncs.ipynb) that NumPy's ufuncs can be used in place of loops to do fast element-wise arithmetic operations on arrays; in the same way, we can use other ufuncs to do element-wise *comparisons* over arrays, and we can then manipulate the results to answer the questions we have.\n",
    "We'll leave the data aside for now, and discuss some general tools in NumPy to use *masking* to quickly answer these types of questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison Operators as Ufuncs\n",
    "\n",
    "[Computation on NumPy Arrays: Universal Functions](02.03-Computation-on-arrays-ufuncs.ipynb) introduced ufuncs, and focused in particular on arithmetic operators. We saw that using `+`, `-`, `*`, `/`, and other operators on arrays leads to element-wise operations.\n",
    "NumPy also implements comparison operators such as `<` (less than) and `>` (greater than) as element-wise ufuncs.\n",
    "The result of these comparison operators is always an array with a Boolean data type.\n",
    "All six of the standard comparison operations are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3, 4, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True, False, False, False])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x < 3  # less than"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False,  True,  True])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x > 3  # greater than"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True, False, False])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x <= 3  # less than or equal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True,  True,  True])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x >= 3  # greater than or equal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True, False,  True,  True])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x != 3  # not equal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True, False, False])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x == 3  # equal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to do an element-wise comparison of two arrays, and to include compound expressions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False,  True, False, False, False])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(2 * x) == (x ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the case of arithmetic operators, the comparison operators are implemented as ufuncs in NumPy; for example, when you write `x < 3`, internally NumPy uses `np.less(x, 3)`.\n",
    "    A summary of the comparison operators and their equivalent ufuncs is shown here:\n",
    "\n",
    "| Operator    | Equivalent ufunc  | Operator   | Equivalent ufunc |\n",
    "|-------------|-------------------|------------|------------------|\n",
    "|`==`         |`np.equal`         |`!=`        |`np.not_equal`    |\n",
    "|`<`          |`np.less`          |`<=`        |`np.less_equal`   |\n",
    "|`>`          |`np.greater`       |`>=`        |`np.greater_equal`|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as in the case of arithmetic ufuncs, these will work on arrays of any size and shape.\n",
    "Here is a two-dimensional example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9, 4, 0, 3],\n",
       "       [8, 6, 3, 1],\n",
       "       [3, 7, 4, 0]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rng = np.random.default_rng(seed=1701)\n",
    "x = rng.integers(10, size=(3, 4))\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False,  True,  True,  True],\n",
       "       [False, False,  True,  True],\n",
       "       [ True, False,  True,  True]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x < 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In each case, the result is a Boolean array, and NumPy provides a number of straightforward patterns for working with these Boolean results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Boolean Arrays\n",
    "\n",
    "Given a Boolean array, there are a host of useful operations you can do.\n",
    "We'll work with `x`, the two-dimensional array we created earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9 4 0 3]\n",
      " [8 6 3 1]\n",
      " [3 7 4 0]]\n"
     ]
    }
   ],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counting Entries\n",
    "\n",
    "To count the number of `True` entries in a Boolean array, `np.count_nonzero` is useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many values less than 6?\n",
    "np.count_nonzero(x < 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are eight array entries that are less than 6.\n",
    "Another way to get at this information is to use `np.sum`; in this case, `False` is interpreted as `0`, and `True` is interpreted as `1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(x < 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The benefit of `np.sum` is that, like with other NumPy aggregation functions, this summation can be done along rows or columns as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 2, 3])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many values less than 6 in each row?\n",
    "np.sum(x < 6, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This counts the number of values less than 6 in each row of the matrix.\n",
    "\n",
    "If we're interested in quickly checking whether any or all the values are `True`, we can use (you guessed it) `np.any` or `np.all`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are there any values greater than 8?\n",
    "np.any(x > 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are there any values less than zero?\n",
    "np.any(x < 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are all values less than 10?\n",
    "np.all(x < 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are all values equal to 6?\n",
    "np.all(x == 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.all` and `np.any` can be used along particular axes as well. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False,  True])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# are all values in each row less than 8?\n",
    "np.all(x < 8, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here all the elements in the third row are less than 8, while this is not the case for others.\n",
    "\n",
    "Finally, a quick warning: as mentioned in [Aggregations: Min, Max, and Everything In Between](02.04-Computation-on-arrays-aggregates.ipynb), Python has built-in `sum`, `any`, and `all` functions. These have a different syntax than the NumPy versions, and in particular will fail or produce unintended results when used on multidimensional arrays. Be sure that you are using `np.sum`, `np.any`, and `np.all` for these examples!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean Operators\n",
    "\n",
    "We've already seen how we might count, say, all days with less than 20 mm of rain, or all days with more than 10 mm of rain.\n",
    "But what if we want to know how many days there were with more than 10 mm and less than 20 mm of rain? We can accomplish this with Python's *bitwise logic operators*, `&`, `|`, `^`, and `~`.\n",
    "Like with the standard arithmetic operators, NumPy overloads these as ufuncs that work element-wise on (usually Boolean) arrays.\n",
    "\n",
    "For example, we can address this sort of compound question as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum((rainfall_mm > 10) & (rainfall_mm < 20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells us that there were 16 days with rainfall of between 10 and 20 millimeters.\n",
    "\n",
    "The parentheses here are important. Because of operator precedence rules, with the parentheses removed this expression would be evaluated as follows, which results in an error:\n",
    "\n",
    "``` python\n",
    "rainfall_mm > (10 & rainfall_mm) < 20\n",
    "```\n",
    "\n",
    "Let's demonstrate a more complicated expression. Using De Morgan's laws, we can compute the same result in a different manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(~( (rainfall_mm <= 10) | (rainfall_mm >= 20) ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combining comparison operators and Boolean operators on arrays can lead to a wide range of efficient logical operations.\n",
    "\n",
    "The following table summarizes the bitwise Boolean operators and their equivalent ufuncs:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Operator    | Equivalent ufunc  | Operator    | Equivalent ufunc  |\n",
    "|-------------|-------------------|-------------|-------------------|\n",
    "|`&`          |`np.bitwise_and`   |&#124;       |`np.bitwise_or`    |\n",
    "|`^`          |`np.bitwise_xor`   |`~`          |`np.bitwise_not`   |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using these tools, we can start to answer many of the questions we might have about our weather data.\n",
    "Here are some examples of results we can compute when combining masking with aggregations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number days without rain:   221\n",
      "Number days with rain:      144\n",
      "Days with more than 10 mm:  34\n",
      "Rainy days with < 5 mm:     83\n"
     ]
    }
   ],
   "source": [
    "print(\"Number days without rain:  \", np.sum(rainfall_mm == 0))\n",
    "print(\"Number days with rain:     \", np.sum(rainfall_mm != 0))\n",
    "print(\"Days with more than 10 mm: \", np.sum(rainfall_mm > 10))\n",
    "print(\"Rainy days with < 5 mm:    \", np.sum((rainfall_mm > 0) &\n",
    "                                            (rainfall_mm < 5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean Arrays as Masks\n",
    "\n",
    "In the preceding section we looked at aggregates computed directly on Boolean arrays.\n",
    "A more powerful pattern is to use Boolean arrays as masks, to select particular subsets of the data themselves. Let's return to our `x` array from before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9, 4, 0, 3],\n",
       "       [8, 6, 3, 1],\n",
       "       [3, 7, 4, 0]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want an array of all values in the array that are less than, say, 5. We can obtain a Boolean array for this condition easily, as we've already seen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False,  True,  True,  True],\n",
       "       [False, False,  True,  True],\n",
       "       [ True, False,  True,  True]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x < 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to *select* these values from the array, we can simply index on this Boolean array; this is known as a *masking* operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 3, 3, 1, 3, 4, 0])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[x < 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is returned is a one-dimensional array filled with all the values that meet this condition; in other words, all the values in positions at which the mask array is `True`.\n",
    "\n",
    "We are then free to operate on these values as we wish.\n",
    "For example, we can compute some relevant statistics on our Seattle rain data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Median precip on rainy days in 2015 (mm):    3.8\n",
      "Median precip on summer days in 2015 (mm):   0.0\n",
      "Maximum precip on summer days in 2015 (mm):  32.5\n",
      "Median precip on non-summer rainy days (mm): 4.1\n"
     ]
    }
   ],
   "source": [
    "# construct a mask of all rainy days\n",
    "rainy = (rainfall_mm > 0)\n",
    "\n",
    "# construct a mask of all summer days (June 21st is the 172nd day)\n",
    "days = np.arange(365)\n",
    "summer = (days > 172) & (days < 262)\n",
    "\n",
    "print(\"Median precip on rainy days in 2015 (mm):   \",\n",
    "      np.median(rainfall_mm[rainy]))\n",
    "print(\"Median precip on summer days in 2015 (mm):  \",\n",
    "      np.median(rainfall_mm[summer]))\n",
    "print(\"Maximum precip on summer days in 2015 (mm): \",\n",
    "      np.max(rainfall_mm[summer]))\n",
    "print(\"Median precip on non-summer rainy days (mm):\",\n",
    "      np.median(rainfall_mm[rainy & ~summer]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By combining Boolean operations, masking operations, and aggregates, we can very quickly answer these sorts of questions about our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the Keywords and/or Versus the Operators &/|\n",
    "\n",
    "One common point of confusion is the difference between the keywords `and` and `or` on the one hand, and the operators `&` and `|` on the other.\n",
    "When would you use one versus the other?\n",
    "\n",
    "The difference is this: `and` and `or` operate on the object as a whole, while `&` and `|` operate on the elements within the object.\n",
    "\n",
    "When you use `and` or `or`, it is equivalent to asking Python to treat the object as a single Boolean entity.\n",
    "In Python, all nonzero integers will evaluate as `True`. Thus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, False)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bool(42), bool(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bool(42 and 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bool(42 or 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you use `&` and `|` on integers, the expression operates on the bitwise representation of the element, applying the *and* or the *or* to the individual bits making up the number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b101010'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b111011'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(59)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b101010'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(42 & 59)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0b111011'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bin(42 | 59)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the corresponding bits of the binary representation are compared in order to yield the result.\n",
    "\n",
    "When you have an array of Boolean values in NumPy, this can be thought of as a string of bits where `1 = True` and `0 = False`, and `&` and `|` will operate similarly to in the preceding examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True, False,  True,  True])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([1, 0, 1, 0, 1, 0], dtype=bool)\n",
    "B = np.array([1, 1, 1, 0, 1, 1], dtype=bool)\n",
    "A | B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if you use `or` on these arrays it will try to evaluate the truth or falsehood of the entire array object, which is not a well-defined value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/xc/sptt9bk14s34rgxt7453p03r0000gp/T/ipykernel_93010/3447948156.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mA\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mB\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"
     ]
    }
   ],
   "source": [
    "A or B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, when evaluating a Boolean expression on a given array, you should use `|` or `&` rather than `or` or `and`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False,  True,  True,  True, False,\n",
       "       False])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.arange(10)\n",
    "(x > 4) & (x < 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying to evaluate the truth or falsehood of the entire array will give the same `ValueError` we saw previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/xc/sptt9bk14s34rgxt7453p03r0000gp/T/ipykernel_93010/2869511139.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m8\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()"
     ]
    }
   ],
   "source": [
    "(x > 4) and (x < 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, remember this: `and` and `or` perform a single Boolean evaluation on an entire object, while `&` and `|` perform multiple Boolean evaluations on the content (the individual bits or bytes) of an object.\n",
    "For Boolean NumPy arrays, the latter is nearly always the desired operation."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3.9.6 64-bit ('3.9.6')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "513788764cd0ec0f97313d5418a13e1ea666d16d72f976a8acadce25a5af2ffc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}