GrowthViz-adults.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GrowthViz - Adults\n",
    "\n",
    "## Background - growthcleanr\n",
    "\n",
    "[growthcleanr](https://github.com/carriedaymont/growthcleanr) is an automated method for identifying implausible values in anthropometric data from EHRs. It is available as open source software. It was originally designed with a focus on pediatric growth data, as stated in [Automated identification of implausible values in growth data from pediatric electronic health records](https://academic.oup.com/jamia/article/24/6/1080/3767271):\n",
    "\n",
    "> In pediatrics, evaluation of growth is fundamental, and many pediatric research studies include some aspect of growth as an outcome or other variable. The clinical growth measurements obtained in day-to-day care are susceptible to error beyond the imprecision inherent in any anthropometric measurement. Some errors result from minor problems with measurement technique. While these errors can be important in certain analyses, they are often small and generally impossible to detect after measurements are recorded. Larger measurement technique errors can result in values that are biologically implausible and can cause problems for many analyses. \n",
    "\n",
    "The `growthcleanr` package has been expanded to identify implausible height and weight observations among adult subjects, from 18 up to age 65. This notebook supports review of `growthcleanr` results when working with data from **adult** subjects. If you are primarily working with pediatric subjects, please use the `GrowthViz-pediatrics.ipynb` notebook instead.\n",
    "\n",
    "## GrowthViz Purpose\n",
    "\n",
    "The objective of this tool is to allow users to conduct post-processing and data visualization of growthcleanr output. GrowthViz is to be used **after** a data set has been run through growthcleanr.\n",
    "\n",
    "GrowthViz is a [Juypter Notebook](https://jupyter.org/). It provides an environment that includes graphical user interfaces as well as interactive software development to explore data. To achieve this, GrowthViz references different software languages and packages:\n",
    " - [Python programming language](https://www.python.org/) is used to import, transform, visualize and analyze the output of growthcleanr. Some of the code for the tool is directly included in this notebook. Other functions have been placed in external files to minimize the amount of code that users see in order to let them focus on the actual data.\n",
    " - Data analysis is performed using [NumPy](https://numpy.org/) and [Pandas](https://pandas.pydata.org/). The output of growthcleanr will be loaded into a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html). GrowthViz provides functions for transforming DataFrames to support calculation of some values, such as BMI, as well as supporting visualizations. It is expected that users will create views into or copies of the DataFrames built initially by this tool. Adding columns to the DataFrames created by this tool is unlikely to cause problems. Removing columns is likely to break some of the tool's functionality.\n",
    " - Visualization in the tool is provided by [Matplotlib](https://matplotlib.org/) and [Seaborn](http://seaborn.pydata.org/). Users may generate their own charts with these utilities.\n",
    "\n",
    "---\n",
    "# Setting Up the Environment\n",
    "\n",
    "Jupyter Notebooks have documentation cells, such as this one, and code cells like the one below. The notebook server runs the code and provides results (if applicable) back in the notebook. The following code cell loads the libraries necessary for the tool to work. If you would like to incorporate other Python libraries to assist in data exploration, they can be added here. Removing libraries from this cell will very likely break the tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import FileLink, FileLinks\n",
    "import qgrid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next two code cells tell the notebook server to automatically reload the externally defined Python functions created to assist in data analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code cell instructs the notebook to automatically display plots inline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code cell tells the notebook to output plots for high DPI displays, such as 4K monitors, many smartphones or a retina display on Apple hardware. This cell does not need to be run and can be safely removed. If removed, charts will look more \"blocky\" or \"pixelated\" on high DPI displays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Data\n",
    "\n",
    "The following cell imports functions created for the tool to assist in data analysis. Some of the functions generate charts used in this tool. The chart code may be modified to change the appearance of plots without too much risk of breaking things. Other functions transform DataFrames and changing those will very likely cause things to break. If you are unable to tell the difference in the functions by looking at the code, it is probably best to leave them unmodified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from growthviz import charts\n",
    "from growthviz import check_data\n",
    "from growthviz import compare\n",
    "from growthviz import processdata\n",
    "from growthviz import sumstats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell reads in a data set that has been run through the [growthcleanr](https://github.com/carriedaymont/growthcleanr) algorithm. Details of the algorithm can be found in [Automated identification of implausible values in growth data from pediatric electronic health records](https://academic.oup.com/jamia/article/24/6/1080/3767271)\n",
    "\n",
    "<a id=\"input_structure\"></a>\n",
    "Data is expected to be in a CSV file with the following structure:\n",
    "\n",
    "| Column | Description |\n",
    "|--------|-------------|\n",
    "| id | A unique identifier for an observation |\n",
    "| subjid | An identifier for the individual from which the observation was obtained |\n",
    "| sex | A numeric value of 0 (male) or 1 (female). Note that this differs from the CDC Growth Chart Data Tables. |\n",
    "| agedays | Number representing the age of the patient in days when the observation was obtained |\n",
    "| param | The type of observation along with units. Expected values are *HEIGHTCM* and *WEIGHTKG* |\n",
    "| measurement | A decimal number that represents the observation value. |\n",
    "| clean_res | The categorization of the observation by growthcleanr. |\n",
    "\n",
    "This information will be loaded into a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) called `cleaned_obs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "readonly": false
   },
   "outputs": [],
   "source": [
    "cleaned_obs = pd.read_csv(\"growthviz-data/sample-adults-data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell shows what the first five rows look like in the CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cleaned_obs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next cell runs through a series of data checks on the original data file, such as making sure all values of `sex` are either 0 or 1, or no age values are negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "warnings = check_data.check_patient_data(\"growthviz-data/sample-adults-data.csv\")\n",
    "if warnings:\n",
    "    for warning in warnings:\n",
    "        print(warning)\n",
    "else:\n",
    "    print(\"Data looks good!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, the `processdata.setup_individual_obs_df` function performs transformations on the `cleaned_obs` DataFrame. This will create an `age` column, which is a decimal column that represents the patient's age in years at the time of the observation. It changes the `clean_value` column into a [pandas categorical type](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html). It also create an `include` column which contains a boolean value indicating whether growthcleanr states to include (true) or exclude (false) the observation. The resulting DataFrame is assigned to `obs_full`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "obs_full = processdata.setup_individual_obs_df(cleaned_obs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs_full.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, the `charts.make_age_charts` function visually displays the range of ages in the dataset, with those to be excluded identified by the red bars with the **x** pattern, and those that are outside the optimal range of the notebook identified by the orange bars with the **/** pattern. As noted above, if the population in the dataset is primarily pediatrics, you will want to switch to the pediatrics notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "charts.make_age_charts(obs_full, 'adults')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chart above shows adult age ranges supported by the rest of this notebook, any observations that fall outside of those ranges.\n",
    "\n",
    "Now, we will filter the age ranges to match the supported adult ranges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs = processdata.keep_age_range(obs_full, 'adults')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, `charts.weight_distr` creates two visualizations. The first shows a distribution of all of the included weights in the dataset. The second shows weights above a certain threshold to see whether there are spikes at a certain *Included* weights that might indicate that a commonly used scale maxes out at a certain value. This chart is restricted to values of 135kg or higher (rounded to the nearest KG) to make patterns in higher weights easier to identify. This potential issue is important to keep in mind when conducting an analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "charts.weight_distr(obs, 'all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "charts.weight_distr(obs, 'high')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell loads in the [CDC Anthropometric Reference Data for Adults](https://www.cdc.gov/nchs/data/series/sr_03/sr03-046-508.pdf). Rows, which represent decades (e.g., 20-29), are expanded so that there is one record per year. Standard deviation is calculated from the count of examined persons and the standard error. `Sex` is then transformed so that the values align with the values used in growthcleanr, 0 (male) or 1 (female). Finally, percentiles are smoothed across decade changes (e.g., any change happens gradually from 29 to 31). This data is used to plot percentile bands in visualizations in the tool. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# adult percentiles\n",
    "percentiles = pd.read_csv(\"growthviz-data/ext/vdsmeasures.csv\", encoding ='latin1')\n",
    "percentiles_clean = processdata.setup_percentiles_adults(percentiles)\n",
    "\n",
    "# save out smoothed percentiles\n",
    "percentiles_clean.drop(columns={'sd'}).to_csv('smoothed_percentiles.csv')\n",
    "\n",
    "# separate percentiles into different data files\n",
    "bmi_percentiles = percentiles_clean[percentiles_clean['param'] == 'BMI']\n",
    "wt_percentiles = percentiles_clean[percentiles_clean['param'] == 'WEIGHTKG']\n",
    "ht_percentiles = percentiles_clean[percentiles_clean['param'] == 'HEIGHTCM']\n",
    "\n",
    "percentiles_clean.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this cell, the percentiles data are reshaped to provide mean and standard deviation values for each parameter that will later be used for z-score calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "percentiles_wide = sumstats.setup_percentile_zscore_adults(percentiles_clean)\n",
    "percentiles_wide.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In an earlier cell, the tool creates the `obs` DataFrame. In that structure there is one measurement, either height or weight, per row. In this cell, the `processdata.setup_merged_df` function will create a DataFrame where a height observation and weight observation for the same `subjid` on the same `agedays` are combined into a single row. Several new columns are added to the resulting DataFrame:\n",
    "\n",
    "| Column | Description |\n",
    "|--------|-------------|\n",
    "| weight | The `measurement` value for observations that had the `param` of *WEIGHTKG* |\n",
    "| height | The `measurement` value for observations that had the `param` of *HEIGHTCM* |\n",
    "| bmi | The BMI calculated from `weight` and `height` |\n",
    "| weight_cat | The value previously stored in `clean_value` for the weight observation |\n",
    "| include_weight | The value previously stored in `include` for the weight observation |\n",
    "| height_cat | The value previously stored in `clean_value` for the height observation |\n",
    "| include_height | The value previously stored in `include` for the height observation |\n",
    "| include_both | Logical AND of `include_weight` and `include_height` |\n",
    "\n",
    "The result is stored in `merged_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_df = processdata.setup_merged_df(obs)\n",
    "merged_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, `processdata.setup_bmi_adults` calculates BMI for each weight and height pairing to be used in later individual trajectory visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "obs_wbmi = processdata.setup_bmi_adults(merged_df, obs)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exclusion Information\n",
    "\n",
    "The following shows the counts of the values for inclusion/exclusion along with the percentages of total. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processdata.exclusion_information(obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Individuals\n",
    "\n",
    "This next cell creates an interactive tool that can be used to explore patients. The `sumstats.add_mzscored_to_merged_df` function will add modified Z Scores for height, weight and BMI to `merged_df`. The tool uses [Qgrid](https://github.com/quantopian/qgrid) to create the interactive table. Clicking on a row will create a plot for the individual below the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "mdf = sumstats.add_mzscored_to_merged_df_adults(merged_df, percentiles_wide) \n",
    "mdf['wtz'] = (mdf['weight'] - mdf['Mean_weight'])/mdf['sd_weight']\n",
    "mdf['htz'] = (mdf['height'] - mdf['Mean_height'])/mdf['sd_height']\n",
    "mdf['bmiz'] = (mdf['bmi'] - mdf['Mean_bmi'])/mdf['sd_bmi']\n",
    "mdf.head()\n",
    "\n",
    "col_opt = {\n",
    "    'width': 20,\n",
    "}\n",
    "col_def = {\n",
    "    'subjid': { 'width': 80 },\n",
    "    'sex': { 'width': 30 },\n",
    "    'age': { 'width': 30 },\n",
    "    'height': { 'width': 50 },\n",
    "    'height_cat': { 'width': 80 },\n",
    "    'htz': { 'width': 50 },\n",
    "    'weight': { 'width': 50 },\n",
    "    'weight_cat': { 'width': 80 },\n",
    "    'wtz': { 'width': 50 },\n",
    "    'bmi': { 'width': 40 },\n",
    "    'bmiz': { 'width': 30 },\n",
    "}\n",
    "g = qgrid.show_grid(charts.top_ten(mdf, 'weight'), precision=3, column_options=col_opt, column_definitions=col_def)\n",
    "ind_out = widgets.Output()\n",
    "def handle_selection_change(_event, _widget):\n",
    "    df = g.get_changed_df()\n",
    "    if len(_event['old']) > 0:\n",
    "        lastSubj = df.iloc[[_event['old'][0]]]['subjid'].iloc[0]\n",
    "    else:\n",
    "        lastSubj = -1\n",
    "    currSubj = df.iloc[[_event['new'][0]]]['subjid'].iloc[0]\n",
    "    sdf = g.get_selected_df()\n",
    "    if sdf.shape[0] >= 1:\n",
    "        subjid = sdf.subjid.iloc[0]\n",
    "        with ind_out:\n",
    "            if currSubj != lastSubj:\n",
    "                ind_out.clear_output()\n",
    "                charts.overlap_view_adults(obs, subjid, 'WEIGHTKG', True, True, wt_percentiles, bmi_percentiles, ht_percentiles)\n",
    "                display(plt.show())        \n",
    "g.on('selection_changed', handle_selection_change)    \n",
    "widgets.VBox([g, ind_out])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be useful to copy values from the `subjid` column in the results above for use in visualizations in the rest of the tool.\n",
    "\n",
    "---\n",
    "\n",
    "# Visualizing an Individual Trajectory\n",
    "\n",
    "The cell below creates a plot for an individual. It shows either the weight trajectory or height tragectory depending on the `param` value that is selected. The black bands in the diagram represent the 5th and 95th percentile values for age and sex for the given measurement type. For BMI tables, the bands used can be found in the [CDC Definitions of Obesity](https://www.cdc.gov/obesity/adult/defining.html). \n",
    "\n",
    "In this chart, the blue line represents all measurements for an individual. Any values marked for exclusion are represented with a red x. The yellow dashed line represents the trajectory with exclusions removed. Any carried forward values are represented by a blue triangle, unless `include_carry_forward` is set to False, when they will also be represented as a red x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ids = obs['subjid'].unique()\n",
    "val = 2868 if 2868 in all_ids else np.random.choice(all_ids, size=1, replace=False)\n",
    "interactive(charts.overlap_view_adults_show, obs_df=fixed(obs_wbmi), \n",
    "            subjid=widgets.Dropdown(options=all_ids, value=val, description='Subject ID:', disabled=False), \n",
    "            param=['HEIGHTCM', 'WEIGHTKG', 'BMI'], \n",
    "            include_carry_forward=widgets.Checkbox(value=True,description='Include Carry Forward',disabled=False,indent=False), \n",
    "            include_percentiles=widgets.Checkbox(value=True,description='Include Measurement Percentile Bands',disabled=False,indent=False),\n",
    "            wt_df=fixed(wt_percentiles), bmi_df=fixed(bmi_percentiles), ht_df=fixed(ht_percentiles))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(subjid=widgets.Dropdown(options=all_ids, value=val, description='Subject ID:', disabled=False))\n",
    "def all_charts(subjid=val):\n",
    "    charts.overlap_view_adults(obs_df=obs_wbmi, subjid=subjid, param='WEIGHTKG', include_carry_forward=True, \n",
    "            include_percentiles=True, wt_df=wt_percentiles, bmi_df=bmi_percentiles, ht_df=ht_percentiles)\n",
    "\n",
    "    charts.overlap_view_adults(obs_df=obs_wbmi, subjid=subjid, param='BMI', include_carry_forward=True, \n",
    "            include_percentiles=True, wt_df=wt_percentiles, bmi_df=bmi_percentiles, ht_df=ht_percentiles)\n",
    "\n",
    "    charts.overlap_view_adults(obs_df=obs_wbmi, subjid=subjid, param='HEIGHTCM', include_carry_forward=True, \n",
    "            include_percentiles=True, wt_df=wt_percentiles, bmi_df=bmi_percentiles, ht_df=ht_percentiles)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing Multiple Trajectories at Once\n",
    "\n",
    "Next, the tool creates a series that contains the unique set of `subjid`s that have more than one record per category (as determined by `charts.mult_obs`) and stores that in `uniq_ids`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs_wbmi_mult = charts.mult_obs(obs_wbmi)\n",
    "uniq_ids = obs_wbmi_mult['subjid'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the series of unique subjids, the following cell randomly selects 25 individuals and assigns them to `sample`. A random seed, which specifies the start point when a computer generates a random number sequence, is then set so that the random sample is reproducible. The random seed can be changed to change the sample generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "sample = np.random.choice(uniq_ids, size=25, replace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sample` can be passed into the `charts.five_by_five_view` function which will create a [small multiple](https://en.wikipedia.org/wiki/Small_multiple) plot for each of the individuals. Exclusions, including carry forwards, will be represented by a red x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "charts.five_by_five_view(obs_wbmi, sample, 'HEIGHTCM', wt_percentiles, ht_percentiles, bmi_percentiles, 'dotted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Different Sample\n",
    "\n",
    "This tool can be used to create samples that are tailored to specific interests. Views can easily be created on existing DataFrames and be used to generate different samples. Functionality available is described in the [Pandas DataFrame documentation](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html).\n",
    "\n",
    "The cell below selects all observations with a weight exclusion of \"Exclude-Moderate-EWMA\". It then sorts by weight in descending order. The code then takes the top 50 values and selects 25 random, unique `subjids` from that set. Finally it plots the results. If there are fewer than 25 examples, but at least one, each example is shown. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_weight_moderate_ewma_ids = merged_df[merged_df.weight_cat == 'Exclude-Adult-EWMA-Moderate'].sort_values('weight', ascending=False).head(50)['subjid'].unique()\n",
    "if len(top_weight_moderate_ewma_ids) >= 25:\n",
    "    ewma_sample = np.random.choice(top_weight_moderate_ewma_ids, size=25, replace=False)\n",
    "    charts.five_by_five_view(obs, ewma_sample, 'WEIGHTKG', wt_percentiles, ht_percentiles, bmi_percentiles, 'dotted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Top/Bottom 25 for a Given Category\n",
    "\n",
    "The following cell uses the same function as above to create a 5 x 5 set of small multiple charts, but selects the top/bottom 25 individuals by growthcleanr category. The results can be sorted by maximum parameter, minimum parameter, starting age, or size of age range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def edge25(obs, category, group, sort_order, param):\n",
    "    filtered_by_cat = obs[(obs.clean_cat == category) & (obs.param == param)]\n",
    "    # get list of relevant IDs\n",
    "    filtered_sum = filtered_by_cat.groupby('subjid', as_index=False).agg(max_measure=('measurement', 'max'), \n",
    "                                                                         min_measure=('measurement', 'min'), \n",
    "                                                                         start_age=('ageyears', 'min'), \n",
    "                                                                         axis_range=('range', 'mean'))\n",
    "    if group == 'largest':\n",
    "        filtered_sum = filtered_sum.nlargest(25, 'max_measure')\n",
    "    else:\n",
    "        filtered_sum = filtered_sum.nsmallest(25, 'min_measure')\n",
    "    filtered_sum.sort_values(by=[sort_order, 'subjid'], inplace=True)\n",
    "    fig = charts.five_by_five_view(obs, filtered_sum.subjid.values, param, wt_percentiles, ht_percentiles, \n",
    "                                   bmi_percentiles, 'dotted')\n",
    "    plt.show()\n",
    "    \n",
    "interact(edge25, obs=fixed(obs_wbmi_mult), category=obs.clean_cat.unique(), \n",
    "         group=['largest', 'smallest'], sort_order=['max_measure', 'min_measure', 'start_age', 'axis_range'], \n",
    "         param=['WEIGHTKG', 'HEIGHTCM', 'BMI']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing Changes in Trajectory\n",
    "\n",
    "The `charts.param_with_percentiles` function displays a chart showing BMI, height, or weight for an individual over time. Black bands representing the 5th and 95th percentiles for age and sex are shown with the individual's BMI, height, or weight shown in blue. The plot on the left represents all values. The plot on the right is only included values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_ids = obs_wbmi['subjid'].unique()\n",
    "val = 2431 if 2431 in all_ids else np.random.choice(all_ids, size=1, replace=False)\n",
    "interact(charts.param_with_percentiles, merged_df=fixed(obs_wbmi),\n",
    "         subjid=widgets.Dropdown(options=all_ids, value=val,\n",
    "                                 description='Subject ID:', disabled=False), \n",
    "         param=['BMI', 'WEIGHTKG', 'HEIGHTCM'], wt_df=fixed(wt_percentiles), \n",
    "         ht_df=fixed(ht_percentiles), bmi_df=fixed(bmi_percentiles));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary Statistics\n",
    "\n",
    "The next cell generates summary statistics for the data set. The `sumstats.bmi_stats` function uses the `merged_df` DataFrame to compute values on the \"clean\" data, which means only included values as well as the \"raw\" values, which is all values for individuals.\n",
    "\n",
    "| Column Name | Definition |\n",
    "|--------|-------------|\n",
    "| min_clean | Minimum Clean BMI Value |\n",
    "| mean_clean | Average Clean BMI Value |\n",
    "| max_clean | Maximum Clean BMI Value |\n",
    "| sd_clean | Standard Deviation of Clean BMI Values |\n",
    "| count_clean | Count of Patient Records with Clean BMI Values |\n",
    "| min_raw | Minimum Raw BMI Value |\n",
    "| mean_raw | Average Raw BMI Value |\n",
    "| max_raw | Maximum Raw BMI Value |\n",
    "| sd_raw | Standard Deviation of Raw BMI Values |\n",
    "| count_raw | Count of All Patient Records |\n",
    "| count_diff | Count of Patient Records with Excluded BMI Values |\n",
    "\n",
    "The buttons can be used to add or remove columns from the table (all columns are shown by default - clicking a button will remove that column from the table).\n",
    "The checkbox includes \"missing\" values (note: this will impact the raw columns as missing values may cause BMI values of infinity since they divide by 0 when missing). Missing values are not included by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "min_toggle = widgets.ToggleButton(value=True, description='Minimum BMI', \n",
    "                                  tooltip=\"Click to hide or show the minimum BMI columns\")\n",
    "mean_toggle = widgets.ToggleButton(value=True, description='Mean BMI', \n",
    "                                   tooltip=\"Click to hide or show the mean BMI columns\")\n",
    "max_toggle = widgets.ToggleButton(value=True, description='Maximum BMI', \n",
    "                                  tooltip=\"Click to hide or show the maximum BMI columns\")\n",
    "std_toggle = widgets.ToggleButton(value=True, description='BMI Standard Deviation',\n",
    "                                  tooltip=\"Click to hide or show the BMI standard deviation columns\")\n",
    "count_toggle = widgets.ToggleButton(value=True, description='Count',\n",
    "                                    tooltip=\"Click to hide or show the count columns\")\n",
    "diff_toggle = widgets.ToggleButton(value=True, description='Count Difference',\n",
    "                                   tooltip=\"Click to hide or show the count difference column\")\n",
    "age_range = widgets.IntRangeSlider(value=[20, 65], min=20, max=65, step=1, description='Age:', continuous_update=False,\n",
    "                                   orientation='horizontal', readout=True, readout_format='d')\n",
    "include_missing_values = widgets.Checkbox(value=False,description='Include Missing / Zero Heights and Weights',disabled=False,indent=False)\n",
    "hbox = widgets.HBox([min_toggle, mean_toggle, max_toggle, std_toggle, count_toggle, diff_toggle])\n",
    "ui = widgets.VBox([age_range, hbox, include_missing_values])\n",
    "sum_out = widgets.Output()\n",
    "widgets.interactive_output(sumstats.bmi_stats, {'merged_df': fixed(merged_df), 'include_min': min_toggle, \n",
    "        'include_mean': mean_toggle, 'include_max': max_toggle, 'include_std': std_toggle, \n",
    "        'include_mean_diff': diff_toggle, 'include_count': count_toggle,\n",
    "        'out': fixed(sum_out), 'age_range': age_range, 'include_missing': include_missing_values})\n",
    "display(ui, sum_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exporting Data\n",
    "\n",
    "The following code allows you to export a DataFrame as a CSV file. When the cell below is run, the drop down will contain all DataFrames stored in variables in this notebook. Select the desired DataFrame and click Generate CSV. This will create the CSV file and provide a link to download it. DataFrames created in this notebook include:\n",
    "\n",
    "| DataFrame | Description |\n",
    "|--------|-------------|\n",
    "| cleaned_obs | Original growthcleanr output read in from csv file |\n",
    "| obs_full | Slightly modified version of `cleaned_obs` in format needed for use in notebook |\n",
    "| obs | Patient observations within age range allowed for this notebook (18-80) |\n",
    "| percentiles | Original CDC percentiles data |\n",
    "| percentiles_clean | Expanded percentiles data with one row per year and standard deviation |\n",
    "| bmi_percentiles | BMI portion of cleaned percentiles data for use in charts |\n",
    "| wt_percentiles | Weight portion of cleaned percentiles data for use in charts |\n",
    "| ht_percentiles | Height portion of cleaned percentiles data for use in charts |\n",
    "| percentiles_wide | Percentiles data reshaped wide with one row per age and sex |\n",
    "| merged_df | Data by subject and age that contains height, weight, and BMI on one row |\n",
    "| obs_wbmi | Version of `obs` that has BMI values appended at end |\n",
    "| mdf | Version of `merged_df` with added z-scores |\n",
    "| obs_wbmi_mult | Version of `obs_wbmi` that only includes subjects with more than one recorded visit |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_selector = widgets.Dropdown(options=processdata.data_frame_names(locals()), description='Data Frames')\n",
    "generate_button = widgets.Button(description='Generate CSV')\n",
    "ui = widgets.VBox([df_selector, generate_button])\n",
    "csv_out = widgets.Output()\n",
    "\n",
    "l = locals()\n",
    "def on_button_clicked(b):\n",
    "    processdata.export_to_csv(l, df_selector, csv_out)\n",
    "\n",
    "generate_button.on_click(on_button_clicked)\n",
    "    \n",
    "display(ui, csv_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Post Processing Data\n",
    "\n",
    "This tool provides functions that allow the post processing of data. `processdata.clean_swapped_values` will look in a DataFrame for rows where the `height_cat` and `weight_cat` are both flagged for exclusions with \"`Exclude-Adult-Swapped-Measurements`\". It will then swap the `height` and `weight` values for those rows, and recalculate BMI. It will also create two new columns: `postprocess_height_cat` and `postprocess_weight_cat`. The values for these columns is copied from the original categories except in the case where swaps are fixed when it is set to \"`Include-Fixed-Swap`\".\n",
    "\n",
    "The cell below copies the merged DataFrame and then cleans the swapped values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned = merged_df.copy()\n",
    "cleaned = processdata.clean_swapped_values(cleaned)\n",
    "cleaned[cleaned.weight_cat == 'Exclude-Adult-Swapped-Measurements'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing New Visualizations\n",
    "\n",
    "Users may take advantage of the predefined `sumstats.bmi_stats`, `charts.bmi_with_percentiles`, `charts.five_by_five_view`, `charts.overlap_view_adults` and `charts.top_ten` functions. For more information on these functions, execute the function name ending with a \"?\", which will bring up the inline help window. For example, `charts.five_by_five_view`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sumstats.bmi_stats?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, users can take advantage of all of the plotting capabilities of [Matplotlib](https://matplotlib.org/stable/tutorials/index.html) and [Seaborn](https://seaborn.pydata.org/tutorial.html)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}