rapids-plotly-paperspace-tutorial-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAPIDS + Plotly Dash on Paperspace ML-Driven Web Apps - Tutorial 2\n",
    "## Intro\n",
    "In this tutorial, We will build upon whwat we learned in Tutorial 1 by adding a button in your Plot.ly Dash app that that kicks off a series of RAPIDS cuML algorithms (PCA, Kmeans, UMAP) on a 1.5 GB dataset.  This demonstrates capability's speed at scale.\n",
    "\n",
    "\n",
    "## Install Extra Libraries\n",
    "The RAPIDSAI container is our starting point.  However, to build this dashboard, we need some extra libraries from plot.ly and jupyter.  You'll only need to install this once per container.  Leave uncommented if this is your first run.  Otherwise comment the `!pip install` lines to save time running. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "# !pip install jupyter-dash tables plotly\n",
    "# !pip install dash dash-bootstrap-components dash-html-components matplotlib plotly\n",
    "\n",
    "## To download data file\n",
    "# !wget https://rapidsai-data.s3.us-east-2.amazonaws.com/community-examples/plotly-webapp-demo/scrna_data.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "import cudf\n",
    "import cuml\n",
    "import cupy as cp\n",
    "import numpy as np\n",
    "\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "import dash\n",
    "import dash_bootstrap_components as dbc\n",
    "import dash_core_components as dcc\n",
    "import dash_html_components as html\n",
    "from dash.dependencies import Input, Output, State\n",
    "\n",
    "from flask import request\n",
    "\n",
    "from jupyter_dash import JupyterDash\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', 'Expected ')\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "gradient": {}
   },
   "source": [
    "In the previous tutorial we learned:\n",
    " - Plotly plots creation in Python\n",
    " - Create Dash components\n",
    " - Handle UI events\n",
    "\n",
    "In this tutorial the focus will be RAPIDS.\n",
    "\n",
    "# Import the Data\n",
    "We load Single Cell RNA data container cell wise(rows) genes(columns) counts. We will be using RAPIDS to infer information from the gene count matrix. Once the data is loaded, we arbitrarily select the first three genes as MARKER genes. We will use these MARKER genes data for comparision against processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.75 s, sys: 1.38 s, total: 6.13 s\n",
      "Wall time: 6.13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df = cudf.read_hdf('scrna_data.h5')\n",
    "df_markers = df.iloc[:, [0, 1, 2]]\n",
    "\n",
    "MARKERS = df.columns[0:3].tolist()\n",
    "MARKERS = [x + '_val' for x in MARKERS]\n",
    "\n",
    "df_markers.columns = MARKERS\n",
    "additional_cols = [x for x in MARKERS]\n",
    "additional_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "gradient": {}
   },
   "source": [
    "# Process\n",
    "Now, we will use [cuML](https://docs.rapids.ai/api/cuml/stable/api.html) in RAPIDS to process the gene count matrix. First RAPIDS is used to perform PCA for dimensionality reduction and the reduced matrix is used to identify clusters using KMeans and learn the manifolds using UMAP. We also store the original index of the dataframe for correlation later to manage events in the dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.13 s, sys: 452 ms, total: 4.58 s\n",
      "Wall time: 4.57 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(100000, 57)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "n_pca = 50\n",
    "n_clusters = 30\n",
    "\n",
    "def cluster(df, n_pca_comp=50, n_clusters=30, additional_cols=None):\n",
    "\n",
    "    if 'orig_index' in df.columns:\n",
    "        print('Deleting orig_index')\n",
    "        idx = df['orig_index']\n",
    "        df.drop(['orig_index'], inplace=True, axis=1)\n",
    "    else:\n",
    "        idx = df.index\n",
    "    \n",
    "    prop_series, df = remove_non_feature_cols(df, additional_cols)\n",
    "    \n",
    "    pca = cuml.PCA(n_components=n_pca_comp)\n",
    "    pca.fit(df)\n",
    "    embedding = pca.transform(df)\n",
    "\n",
    "    kmeans_cuml = cuml.KMeans(n_clusters=n_clusters)\n",
    "    kmeans_cuml.fit(embedding)\n",
    "    kmeans_labels = kmeans_cuml.predict(embedding)\n",
    "\n",
    "    umap = cuml.manifold.UMAP()\n",
    "    Xt =  umap.fit_transform(embedding)\n",
    "\n",
    "    embedding['labels'] = kmeans_labels\n",
    "    embedding['x'] = Xt[0]\n",
    "    embedding['y'] = Xt[1]\n",
    "    embedding.index = idx\n",
    "    embedding['orig_index'] = idx\n",
    "    \n",
    "    for col in prop_series.keys():\n",
    "        embedding[col] = prop_series[col]\n",
    "    return embedding\n",
    "\n",
    "\n",
    "def remove_non_feature_cols(df, additional_cols):\n",
    "    prop_series = {}\n",
    "    for col_name in additional_cols:\n",
    "        if col_name in df.columns:\n",
    "            prop_series[col_name] = df[col_name]\n",
    "            df = df.drop([col_name], axis=1)\n",
    "\n",
    "    for col_name in ['labels', 'x', 'y']:\n",
    "        if col_name in df.columns:\n",
    "            df.drop([col_name], inplace=True, axis=1)\n",
    "            \n",
    "    return prop_series, df\n",
    "\n",
    "\n",
    "orig_df = cluster(df, n_pca_comp=n_pca, n_clusters=n_clusters, additional_cols=additional_cols)\n",
    "\n",
    "# Add marker genes back\n",
    "orig_df = cudf.concat([orig_df, df_markers], axis=1)\n",
    "orig_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>labels</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>orig_index</th>\n",
       "      <th>Xkr4_val</th>\n",
       "      <th>Gm1992_val</th>\n",
       "      <th>Sox17_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.868425</td>\n",
       "      <td>0.336145</td>\n",
       "      <td>1.421227</td>\n",
       "      <td>-2.847102</td>\n",
       "      <td>-0.633875</td>\n",
       "      <td>3.891086</td>\n",
       "      <td>3.511035</td>\n",
       "      <td>-3.105289</td>\n",
       "      <td>-3.548421</td>\n",
       "      <td>0.086103</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845912</td>\n",
       "      <td>1.195730</td>\n",
       "      <td>-1.716885</td>\n",
       "      <td>23</td>\n",
       "      <td>-0.592015</td>\n",
       "      <td>-3.349333</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.139578</td>\n",
       "      <td>-0.005757</td>\n",
       "      <td>0.049112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.660085</td>\n",
       "      <td>-0.411820</td>\n",
       "      <td>2.132230</td>\n",
       "      <td>-4.078625</td>\n",
       "      <td>-1.024646</td>\n",
       "      <td>3.650779</td>\n",
       "      <td>3.149760</td>\n",
       "      <td>-5.575364</td>\n",
       "      <td>0.577942</td>\n",
       "      <td>-0.278277</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.046324</td>\n",
       "      <td>-0.700149</td>\n",
       "      <td>-0.310466</td>\n",
       "      <td>9</td>\n",
       "      <td>-2.226587</td>\n",
       "      <td>-2.708250</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.130967</td>\n",
       "      <td>0.001012</td>\n",
       "      <td>0.045747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.293064</td>\n",
       "      <td>3.452299</td>\n",
       "      <td>-5.101409</td>\n",
       "      <td>-3.459800</td>\n",
       "      <td>0.248678</td>\n",
       "      <td>0.371149</td>\n",
       "      <td>1.881885</td>\n",
       "      <td>-2.333594</td>\n",
       "      <td>0.430030</td>\n",
       "      <td>2.331941</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.615356</td>\n",
       "      <td>-2.687400</td>\n",
       "      <td>-1.101635</td>\n",
       "      <td>12</td>\n",
       "      <td>2.128305</td>\n",
       "      <td>-5.388611</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.133182</td>\n",
       "      <td>0.000788</td>\n",
       "      <td>0.018643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-4.962597</td>\n",
       "      <td>-1.094800</td>\n",
       "      <td>-1.184917</td>\n",
       "      <td>12.477499</td>\n",
       "      <td>0.852307</td>\n",
       "      <td>-2.062974</td>\n",
       "      <td>-0.705089</td>\n",
       "      <td>-7.453671</td>\n",
       "      <td>-0.326200</td>\n",
       "      <td>-1.523405</td>\n",
       "      <td>...</td>\n",
       "      <td>2.053468</td>\n",
       "      <td>-0.682838</td>\n",
       "      <td>1.046343</td>\n",
       "      <td>1</td>\n",
       "      <td>9.932784</td>\n",
       "      <td>1.685898</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.155867</td>\n",
       "      <td>-0.009243</td>\n",
       "      <td>-0.116312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-9.520083</td>\n",
       "      <td>0.326495</td>\n",
       "      <td>2.546679</td>\n",
       "      <td>-5.504425</td>\n",
       "      <td>-0.621280</td>\n",
       "      <td>0.145227</td>\n",
       "      <td>-0.697787</td>\n",
       "      <td>-4.823132</td>\n",
       "      <td>3.346006</td>\n",
       "      <td>0.645233</td>\n",
       "      <td>...</td>\n",
       "      <td>0.951622</td>\n",
       "      <td>-0.019756</td>\n",
       "      <td>-0.619416</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.899145</td>\n",
       "      <td>-0.292542</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.159265</td>\n",
       "      <td>-0.004439</td>\n",
       "      <td>-0.252790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99995</th>\n",
       "      <td>-9.652540</td>\n",
       "      <td>1.606910</td>\n",
       "      <td>1.734879</td>\n",
       "      <td>-6.240180</td>\n",
       "      <td>-0.202734</td>\n",
       "      <td>0.062667</td>\n",
       "      <td>-0.580834</td>\n",
       "      <td>-2.403271</td>\n",
       "      <td>-1.482028</td>\n",
       "      <td>-0.425373</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.917124</td>\n",
       "      <td>0.642069</td>\n",
       "      <td>-1.700373</td>\n",
       "      <td>3</td>\n",
       "      <td>-3.825832</td>\n",
       "      <td>3.444439</td>\n",
       "      <td>99995</td>\n",
       "      <td>-0.163394</td>\n",
       "      <td>-0.006726</td>\n",
       "      <td>-0.268847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99996</th>\n",
       "      <td>30.457289</td>\n",
       "      <td>-78.483887</td>\n",
       "      <td>8.533363</td>\n",
       "      <td>-2.817974</td>\n",
       "      <td>12.796712</td>\n",
       "      <td>-6.034056</td>\n",
       "      <td>5.341434</td>\n",
       "      <td>-1.726742</td>\n",
       "      <td>0.550985</td>\n",
       "      <td>1.117306</td>\n",
       "      <td>...</td>\n",
       "      <td>6.830244</td>\n",
       "      <td>-2.166545</td>\n",
       "      <td>0.219191</td>\n",
       "      <td>11</td>\n",
       "      <td>6.222085</td>\n",
       "      <td>9.552871</td>\n",
       "      <td>99996</td>\n",
       "      <td>-0.215264</td>\n",
       "      <td>-0.012333</td>\n",
       "      <td>4.465612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99997</th>\n",
       "      <td>-4.483392</td>\n",
       "      <td>-1.274549</td>\n",
       "      <td>-0.444018</td>\n",
       "      <td>-2.021665</td>\n",
       "      <td>0.546830</td>\n",
       "      <td>-0.759780</td>\n",
       "      <td>-1.122950</td>\n",
       "      <td>3.151691</td>\n",
       "      <td>3.470231</td>\n",
       "      <td>0.876152</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.367942</td>\n",
       "      <td>-0.497674</td>\n",
       "      <td>0.254078</td>\n",
       "      <td>29</td>\n",
       "      <td>-3.237031</td>\n",
       "      <td>1.565403</td>\n",
       "      <td>99997</td>\n",
       "      <td>-0.143622</td>\n",
       "      <td>-0.005804</td>\n",
       "      <td>-0.007047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99998</th>\n",
       "      <td>-5.490015</td>\n",
       "      <td>-0.840302</td>\n",
       "      <td>-2.735395</td>\n",
       "      <td>12.840818</td>\n",
       "      <td>0.953613</td>\n",
       "      <td>-3.606477</td>\n",
       "      <td>-1.729116</td>\n",
       "      <td>-4.340821</td>\n",
       "      <td>0.075027</td>\n",
       "      <td>1.233104</td>\n",
       "      <td>...</td>\n",
       "      <td>0.866716</td>\n",
       "      <td>0.753365</td>\n",
       "      <td>-0.080494</td>\n",
       "      <td>1</td>\n",
       "      <td>10.964851</td>\n",
       "      <td>-0.196928</td>\n",
       "      <td>99998</td>\n",
       "      <td>-0.151278</td>\n",
       "      <td>-0.005351</td>\n",
       "      <td>-0.123351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99999</th>\n",
       "      <td>-7.254598</td>\n",
       "      <td>0.646109</td>\n",
       "      <td>-0.179532</td>\n",
       "      <td>0.263951</td>\n",
       "      <td>-0.581387</td>\n",
       "      <td>5.371624</td>\n",
       "      <td>4.401132</td>\n",
       "      <td>-4.866256</td>\n",
       "      <td>-8.212704</td>\n",
       "      <td>1.468548</td>\n",
       "      <td>...</td>\n",
       "      <td>0.289684</td>\n",
       "      <td>1.470201</td>\n",
       "      <td>-1.899853</td>\n",
       "      <td>23</td>\n",
       "      <td>4.258865</td>\n",
       "      <td>-9.562527</td>\n",
       "      <td>99999</td>\n",
       "      <td>-0.166677</td>\n",
       "      <td>-0.009215</td>\n",
       "      <td>-0.269248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100000 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               0          1         2          3          4         5  \\\n",
       "0       1.868425   0.336145  1.421227  -2.847102  -0.633875  3.891086   \n",
       "1       0.660085  -0.411820  2.132230  -4.078625  -1.024646  3.650779   \n",
       "2      13.293064   3.452299 -5.101409  -3.459800   0.248678  0.371149   \n",
       "3      -4.962597  -1.094800 -1.184917  12.477499   0.852307 -2.062974   \n",
       "4      -9.520083   0.326495  2.546679  -5.504425  -0.621280  0.145227   \n",
       "...          ...        ...       ...        ...        ...       ...   \n",
       "99995  -9.652540   1.606910  1.734879  -6.240180  -0.202734  0.062667   \n",
       "99996  30.457289 -78.483887  8.533363  -2.817974  12.796712 -6.034056   \n",
       "99997  -4.483392  -1.274549 -0.444018  -2.021665   0.546830 -0.759780   \n",
       "99998  -5.490015  -0.840302 -2.735395  12.840818   0.953613 -3.606477   \n",
       "99999  -7.254598   0.646109 -0.179532   0.263951  -0.581387  5.371624   \n",
       "\n",
       "              6         7         8         9  ...        47        48  \\\n",
       "0      3.511035 -3.105289 -3.548421  0.086103  ...  0.845912  1.195730   \n",
       "1      3.149760 -5.575364  0.577942 -0.278277  ... -0.046324 -0.700149   \n",
       "2      1.881885 -2.333594  0.430030  2.331941  ... -0.615356 -2.687400   \n",
       "3     -0.705089 -7.453671 -0.326200 -1.523405  ...  2.053468 -0.682838   \n",
       "4     -0.697787 -4.823132  3.346006  0.645233  ...  0.951622 -0.019756   \n",
       "...         ...       ...       ...       ...  ...       ...       ...   \n",
       "99995 -0.580834 -2.403271 -1.482028 -0.425373  ... -0.917124  0.642069   \n",
       "99996  5.341434 -1.726742  0.550985  1.117306  ...  6.830244 -2.166545   \n",
       "99997 -1.122950  3.151691  3.470231  0.876152  ... -0.367942 -0.497674   \n",
       "99998 -1.729116 -4.340821  0.075027  1.233104  ...  0.866716  0.753365   \n",
       "99999  4.401132 -4.866256 -8.212704  1.468548  ...  0.289684  1.470201   \n",
       "\n",
       "             49  labels          x         y  orig_index  Xkr4_val  \\\n",
       "0     -1.716885      23  -0.592015 -3.349333           0 -0.139578   \n",
       "1     -0.310466       9  -2.226587 -2.708250           1 -0.130967   \n",
       "2     -1.101635      12   2.128305 -5.388611           2 -0.133182   \n",
       "3      1.046343       1   9.932784  1.685898           3 -0.155867   \n",
       "4     -0.619416       3  -0.899145 -0.292542           4 -0.159265   \n",
       "...         ...     ...        ...       ...         ...       ...   \n",
       "99995 -1.700373       3  -3.825832  3.444439       99995 -0.163394   \n",
       "99996  0.219191      11   6.222085  9.552871       99996 -0.215264   \n",
       "99997  0.254078      29  -3.237031  1.565403       99997 -0.143622   \n",
       "99998 -0.080494       1  10.964851 -0.196928       99998 -0.151278   \n",
       "99999 -1.899853      23   4.258865 -9.562527       99999 -0.166677   \n",
       "\n",
       "       Gm1992_val  Sox17_val  \n",
       "0       -0.005757   0.049112  \n",
       "1        0.001012   0.045747  \n",
       "2        0.000788   0.018643  \n",
       "3       -0.009243  -0.116312  \n",
       "4       -0.004439  -0.252790  \n",
       "...           ...        ...  \n",
       "99995   -0.006726  -0.268847  \n",
       "99996   -0.012333   4.465612  \n",
       "99997   -0.005804  -0.007047  \n",
       "99998   -0.005351  -0.123351  \n",
       "99999   -0.009215  -0.269248  \n",
       "\n",
       "[100000 rows x 57 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orig_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "# Utility function\n",
    "def generate_colors(num_colors):\n",
    "    \"\"\"\n",
    "    Generates evenly disributed colors\n",
    "    \"\"\"\n",
    "    a = ((np.random.random(size=num_colors) * 255))\n",
    "    b = ((np.random.random(size=num_colors) * 255))\n",
    "    return [\"#%02x%02x%02x\" % (int(r), int(g), 125) for r, g in zip(a, b)]\n",
    "\n",
    "colors = generate_colors(24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def generate_main_graph(df):\n",
    "    fig = go.Figure(layout = {'colorscale' : {}})\n",
    "    label_count = len(df['labels'].unique())\n",
    "    \n",
    "    for i in range(0, label_count):\n",
    "        si = str(i)\n",
    "        gdf = df.query('labels == ' + si)\n",
    "        fig.add_trace(\n",
    "            go.Scattergl({\n",
    "            'x': gdf['x'].to_array(),\n",
    "            'y': gdf['y'].to_array(),\n",
    "            'text': gdf['labels'].to_array(),\n",
    "            'customdata': gdf['orig_index'].to_array(),\n",
    "            'name': 'Cluster ' + si,\n",
    "            'mode': 'markers',\n",
    "            'marker': {'size': 2, 'color': colors[i % len(colors)]}\n",
    "        }))\n",
    "\n",
    "    fig.update_layout(\n",
    "            showlegend=True, clickmode='event', title='UMAP', dragmode='select',\n",
    "            annotations=[\n",
    "                dict(x=0.5, y=-0.07, showarrow=False, text='UMAP_1', xref=\"paper\", yref=\"paper\"),\n",
    "                dict(x=-0.05, y=0.5, showarrow=False, text=\"UMAP_2\", xref=\"paper\", yref=\"paper\", textangle=-90, )])\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def update_scatter_plots_by_points(df, selected_point_indexes):\n",
    "    localdf = df[df['orig_index'].isin(selected_point_indexes)]\n",
    "    \n",
    "    fig_marker1 = graph_scatter(localdf, MARKERS[0])\n",
    "    fig_marker2 = graph_scatter(localdf, MARKERS[1])\n",
    "    fig_marker3 = graph_scatter(localdf, MARKERS[2])\n",
    "\n",
    "    return fig_marker1, fig_marker2, fig_marker3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def graph_scatter(df, label):\n",
    "    \n",
    "    scatter_fig = go.Figure(layout = {'colorscale' : {}})\n",
    "    df[label] = df[label].round(3)\n",
    "    gdf = df.query(label + ' != 0').groupby(['labels',label], as_index=False).agg({'x':'count'})\n",
    "    \n",
    "    xmax = gdf['x'].max()\n",
    "    xmin = gdf['x'].min()\n",
    "    gdf = df.groupby(['labels', label], as_index=False).agg({'x':'count'})\n",
    "    \n",
    "    for i in range(0, len(gdf['labels'].unique())):\n",
    "        sdf = gdf.query('labels == ' + str(i))\n",
    "        _t = sdf['x'].astype('int')\n",
    "        _ace = (sdf[label] * 10).astype('int')\n",
    "        _z = cp.zeros(21)\n",
    "        cp.put(_z, _ace, _t)\n",
    "        _z[_z == 0] = None\n",
    "        scatter_fig.add_trace(\n",
    "                go.Scattergl({\n",
    "                'x': sdf[label].to_array(),\n",
    "                'y': sdf['labels'].to_array(),\n",
    "                'text': cp.asnumpy(_z),\n",
    "                'mode': 'markers',\n",
    "                'marker':{\n",
    "                    'size':4,\n",
    "                    'color':cp.asnumpy(_z),\n",
    "                    'colorscale':'Viridis',\n",
    "                    'cmin': xmin,\n",
    "                    'cmax': xmax,\n",
    "                    'showscale':True\n",
    "                }\n",
    "            }))\n",
    "\n",
    "    scatter_fig.update_layout(showlegend=False, clickmode='select', title=label, dragmode='select',\n",
    "        annotations=[\n",
    "            dict(x=0.5,   y=-0.15, showarrow=False, xref=\"paper\", yref=\"paper\", text=label),\n",
    "            dict(x=-0.15, y=0.5,   showarrow=False, xref=\"paper\", yref=\"paper\", text=\"Clusters\", textangle=-90)\n",
    "        ],)\n",
    "    return scatter_fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>labels</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>orig_index</th>\n",
       "      <th>Xkr4_val</th>\n",
       "      <th>Gm1992_val</th>\n",
       "      <th>Sox17_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.868425</td>\n",
       "      <td>0.336145</td>\n",
       "      <td>1.421227</td>\n",
       "      <td>-2.847102</td>\n",
       "      <td>-0.633875</td>\n",
       "      <td>3.891086</td>\n",
       "      <td>3.511035</td>\n",
       "      <td>-3.105289</td>\n",
       "      <td>-3.548421</td>\n",
       "      <td>0.086103</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845912</td>\n",
       "      <td>1.195730</td>\n",
       "      <td>-1.716885</td>\n",
       "      <td>23</td>\n",
       "      <td>-0.592015</td>\n",
       "      <td>-3.349333</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.139578</td>\n",
       "      <td>-0.005757</td>\n",
       "      <td>0.049112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.660085</td>\n",
       "      <td>-0.411820</td>\n",
       "      <td>2.132230</td>\n",
       "      <td>-4.078625</td>\n",
       "      <td>-1.024646</td>\n",
       "      <td>3.650779</td>\n",
       "      <td>3.149760</td>\n",
       "      <td>-5.575364</td>\n",
       "      <td>0.577942</td>\n",
       "      <td>-0.278277</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.046324</td>\n",
       "      <td>-0.700149</td>\n",
       "      <td>-0.310466</td>\n",
       "      <td>9</td>\n",
       "      <td>-2.226587</td>\n",
       "      <td>-2.708250</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.130967</td>\n",
       "      <td>0.001012</td>\n",
       "      <td>0.045747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.293064</td>\n",
       "      <td>3.452299</td>\n",
       "      <td>-5.101409</td>\n",
       "      <td>-3.459800</td>\n",
       "      <td>0.248678</td>\n",
       "      <td>0.371149</td>\n",
       "      <td>1.881885</td>\n",
       "      <td>-2.333594</td>\n",
       "      <td>0.430030</td>\n",
       "      <td>2.331941</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.615356</td>\n",
       "      <td>-2.687400</td>\n",
       "      <td>-1.101635</td>\n",
       "      <td>12</td>\n",
       "      <td>2.128305</td>\n",
       "      <td>-5.388611</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.133182</td>\n",
       "      <td>0.000788</td>\n",
       "      <td>0.018643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-4.962597</td>\n",
       "      <td>-1.094800</td>\n",
       "      <td>-1.184917</td>\n",
       "      <td>12.477499</td>\n",
       "      <td>0.852307</td>\n",
       "      <td>-2.062974</td>\n",
       "      <td>-0.705089</td>\n",
       "      <td>-7.453671</td>\n",
       "      <td>-0.326200</td>\n",
       "      <td>-1.523405</td>\n",
       "      <td>...</td>\n",
       "      <td>2.053468</td>\n",
       "      <td>-0.682838</td>\n",
       "      <td>1.046343</td>\n",
       "      <td>1</td>\n",
       "      <td>9.932784</td>\n",
       "      <td>1.685898</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.155867</td>\n",
       "      <td>-0.009243</td>\n",
       "      <td>-0.116312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-9.520083</td>\n",
       "      <td>0.326495</td>\n",
       "      <td>2.546679</td>\n",
       "      <td>-5.504425</td>\n",
       "      <td>-0.621280</td>\n",
       "      <td>0.145227</td>\n",
       "      <td>-0.697787</td>\n",
       "      <td>-4.823132</td>\n",
       "      <td>3.346006</td>\n",
       "      <td>0.645233</td>\n",
       "      <td>...</td>\n",
       "      <td>0.951622</td>\n",
       "      <td>-0.019756</td>\n",
       "      <td>-0.619416</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.899145</td>\n",
       "      <td>-0.292542</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.159265</td>\n",
       "      <td>-0.004439</td>\n",
       "      <td>-0.252790</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0         1         2          3         4         5         6  \\\n",
       "0   1.868425  0.336145  1.421227  -2.847102 -0.633875  3.891086  3.511035   \n",
       "1   0.660085 -0.411820  2.132230  -4.078625 -1.024646  3.650779  3.149760   \n",
       "2  13.293064  3.452299 -5.101409  -3.459800  0.248678  0.371149  1.881885   \n",
       "3  -4.962597 -1.094800 -1.184917  12.477499  0.852307 -2.062974 -0.705089   \n",
       "4  -9.520083  0.326495  2.546679  -5.504425 -0.621280  0.145227 -0.697787   \n",
       "\n",
       "          7         8         9  ...        47        48        49  labels  \\\n",
       "0 -3.105289 -3.548421  0.086103  ...  0.845912  1.195730 -1.716885      23   \n",
       "1 -5.575364  0.577942 -0.278277  ... -0.046324 -0.700149 -0.310466       9   \n",
       "2 -2.333594  0.430030  2.331941  ... -0.615356 -2.687400 -1.101635      12   \n",
       "3 -7.453671 -0.326200 -1.523405  ...  2.053468 -0.682838  1.046343       1   \n",
       "4 -4.823132  3.346006  0.645233  ...  0.951622 -0.019756 -0.619416       3   \n",
       "\n",
       "          x         y  orig_index  Xkr4_val  Gm1992_val  Sox17_val  \n",
       "0 -0.592015 -3.349333           0 -0.139578   -0.005757   0.049112  \n",
       "1 -2.226587 -2.708250           1 -0.130967    0.001012   0.045747  \n",
       "2  2.128305 -5.388611           2 -0.133182    0.000788   0.018643  \n",
       "3  9.932784  1.685898           3 -0.155867   -0.009243  -0.116312  \n",
       "4 -0.899145 -0.292542           4 -0.159265   -0.004439  -0.252790  \n",
       "\n",
       "[5 rows x 57 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tdf = orig_df.copy()\n",
    "tdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding the RAPIDS ML function to the button\n",
    "Now we will add in the callbacks necessary to run a process similar to the `cluster` function that we did above, but in the app.  In the last callback for the new button, `Replot on Selected Points`,  the `recluster` function will run and reform the data so that it can be properly displayed by Plot.ly Dash.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b style='color: red'>Please click on <a href='/proxy/8080/' target='_blank'>here</a> to open the dash</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dash app running on http://127.0.0.1:8080/proxy/8080/\n"
     ]
    }
   ],
   "source": [
    "# Please change the proxy_port if the port is already in use.\n",
    "proxy_port = 8080\n",
    "\n",
    "app = JupyterDash(__name__, \n",
    "                  external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css', dbc.themes.BOOTSTRAP], \n",
    "                  requests_pathname_prefix='/proxy/' + str(proxy_port) + '/')\n",
    "fig = generate_main_graph(tdf)\n",
    "fig_ace = graph_scatter(tdf, MARKERS[0])\n",
    "fig_tmp = graph_scatter(tdf, MARKERS[1])\n",
    "fig_epc = graph_scatter(tdf, MARKERS[2])\n",
    "\n",
    "app.layout = html.Div(id='site_body', children=[\n",
    "    html.Div(className='row', children=[\n",
    "        html.Div([\n",
    "            dcc.Graph(id='basic-interactions', figure=fig),\n",
    "        ], className='nine columns', style={'verticalAlign': 'text-top',}),\n",
    "\n",
    "        html.Div([\n",
    "            html.Div(className='row', children=[\n",
    "                dbc.Button(\"Directions\", id=\"bt_open_directions\"),\n",
    "                dbc.Modal([\n",
    "                        dbc.ModalHeader(\"Directions\"),\n",
    "                        dbc.ModalBody(\n",
    "                            dcc.Markdown(\"\"\"\n",
    "                                The main scatterplot shows the UMAP visualization of single cells. \n",
    "                                \n",
    "                                ### Re-running Plotting and Visualization\n",
    "                                #### Reclustering by Selecting Points: \n",
    "                                1. Use the **Box Select** or **Lasso Select** tool to select your points of interest. A number of points will populate the inthe Selected Points field.\n",
    "                                2. Click **Replot on Points**.\n",
    "                                \n",
    "                                You can also manually add either Cluster number or selected points by typing it in. To do so, enter one or more numbers, e.g. “1” or “1,2,3” in the text box above the button for the operation you want to perform.\n",
    "                                \n",
    "                                ### Using the Toolbar\n",
    "                                Hover the mouse over the top right corner of the screen to see a toolbar. Hover over each tool to see its name. The tool options from left to right are:\n",
    "                                - **Camera:** download a snapshot of the current view as .png\n",
    "                                - **Zoom:** Click and drag to select a region of the plot to zoom into\n",
    "                                - **Pan:** Click and drag to shift the current view to a different region of the plot\n",
    "                                - **Box Select/Lasso Select:** both these tools can be used to select a region on the plot. The selected points are exported under ‘selection data’. See below to export the selected points to a dataframe.\n",
    "                                - **Zoom In/Zoom Out:** Zoom in and out centered on the current view.\n",
    "                            \"\"\"),\n",
    "                        ),\n",
    "                        dbc.ModalFooter(\n",
    "                            dbc.Button(\"Close\", id=\"bt_close_directions\", className=\"ml-auto\")\n",
    "                        ),\n",
    "                    ], id=\"md_directions\"),\n",
    "            ]),\n",
    "                        \n",
    "            html.Div(className='row', children=[\n",
    "                dcc.Markdown(\"\"\"\n",
    "                    **Selection Data**\n",
    "\n",
    "                    Choose the lasso or rectangle tool in the graph's menu\n",
    "                    bar and then select points in the graph.\n",
    "                \"\"\"),\n",
    "            ], style={'marginTop': 18,}),\n",
    "            html.Div(className='row', children=[\n",
    "                html.Div(dcc.Input(id='point_index', type='text')),\n",
    "            ]),\n",
    "            \n",
    "            html.Div(className='row', children=[html.Div(id='point_index_cnt'),]),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dbc.Button('Replot on Selected Points', id='recluster_by_point', n_clicks=0),\n",
    "            ], style={'marginTop': 6,}),\n",
    "\n",
    "            html.Div(className='row', children=[dbc.Button(\"Exit\", id=\"bt_close\"),], style={'marginTop': 6,}),\n",
    "\n",
    "        ], className='three columns', style={'marginTop': 90, 'verticalAlign': 'text-top',}),\n",
    "    ]),\n",
    "\n",
    "    html.Div(className='row', children=[\n",
    "        html.Div([dcc.Graph(id='ace-interactions',figure=fig_ace)], className='four columns'),\n",
    "        html.Div([dcc.Graph(id='tmp-interactions',figure=fig_tmp)], className='four columns'),\n",
    "        html.Div([dcc.Graph(id='epc-interactions',figure=fig_epc)], className='four columns')\n",
    "    ]),\n",
    "])\n",
    "\n",
    "\n",
    "@app.callback(Output(\"site_body\", \"children\"), Input('bt_close', 'n_clicks'))\n",
    "def close_app(export_clicks):\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "\n",
    "    func = request.environ.get('werkzeug.server.shutdown')\n",
    "    if func is None:\n",
    "        raise RuntimeError('Not running with the Werkzeug Server')\n",
    "    func()\n",
    "    return 'Closed'\n",
    "\n",
    "\n",
    "@app.callback(\n",
    "    Output(\"md_directions\", \"is_open\"),\n",
    "    [Input(\"bt_open_directions\", \"n_clicks\"), Input(\"bt_close_directions\", \"n_clicks\")],\n",
    "    [State(\"md_directions\", \"is_open\")])\n",
    "def toggle_directions_dialog(n1, n2, is_open):\n",
    "    if n1 or n2:\n",
    "        return not is_open\n",
    "    return is_open\n",
    "\n",
    "\n",
    "@app.callback(\n",
    "    [Output('point_index_cnt', 'children'),\n",
    "     Output('point_index', 'value')],\n",
    "    [Input('basic-interactions', 'clickData'),\n",
    "     Input('basic-interactions', 'selectedData')],\n",
    "    [State('point_index', 'value')])\n",
    "def display_selected_data(clicked_cluster, selected_point_index, point_index):\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "\n",
    "    comp_id, event_type = dash.callback_context.triggered[0]['prop_id'].split('.')\n",
    "    print('Event %s from %s' % (event_type, comp_id))\n",
    "\n",
    "    point_cnt_str = ''\n",
    "    point_indexes = ''\n",
    "\n",
    "    if comp_id == 'basic-interactions' and event_type == 'selectedData':\n",
    "        # Event - On selection on the main scatterplot\n",
    "        print('Event - On selecting points on the main scatterplot')\n",
    "        if not selected_point_index:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "\n",
    "        selected_point_indexes = []\n",
    "        for point in selected_point_index['points']:\n",
    "            selected_point_indexes.append(point['customdata'])\n",
    "\n",
    "        if len(selected_point_indexes) <= 1:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "\n",
    "        point_cnt_str = str(len(selected_point_indexes)) + ' points selected'\n",
    "        point_indexes = ', '.join(map(str, selected_point_indexes))\n",
    "    else:\n",
    "        print('Unhandled event')\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "        \n",
    "    return point_cnt_str, point_indexes\n",
    "\n",
    "\n",
    "@app.callback([Output('basic-interactions', 'figure'),\n",
    "               Output('ace-interactions', 'figure'),\n",
    "               Output('tmp-interactions', 'figure'),\n",
    "               Output('epc-interactions', 'figure')],\n",
    "              [Input('recluster_by_point', 'n_clicks')],\n",
    "              [State('point_index', 'value')])\n",
    "def recluster(recluster_by_point, point_index):\n",
    "    global tdf, fig_ace, fig_tmp, fig_epc\n",
    "\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "    comp_id, event_type = dash.callback_context.triggered[0]['prop_id'].split('.')\n",
    "    print('Event %s from %s' % (event_type, comp_id))\n",
    "\n",
    "    if comp_id == 'recluster_by_point' and event_type == 'n_clicks':\n",
    "        if not point_index:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "        # Event - On click 'recluster' button\n",
    "        selected_point_indexes = list(map(int, point_index.split(\",\")))\n",
    "\n",
    "        fig_ace, fig_tmp, fig_epc = update_scatter_plots_by_points(tdf, selected_point_indexes)\n",
    "    else:\n",
    "        print('Unhandled event')\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "    return fig, fig_ace, fig_tmp, fig_epc\n",
    "\n",
    "js = \"<b style='color: red'>Please click on <a href='/proxy/\" + str(proxy_port) + \"/' target='_blank'>here</a> to open the dash</b>\"\n",
    "display(HTML(js))\n",
    "\n",
    "srv = app.run_server(debug=True, use_reloader=False, port=proxy_port)\n",
    "srv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}