examples/dataframe_batch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataFrame Batch Training\n",
    "\n",
    "This notebook explores the new batch training feature in Gretel Synthetics. This interface will create N synthetic training configurations, where N is a specific number of batches of column names. We break down the source DataFrame into smaller DataFrames that have the same number of rows, but only a subset of total columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you are using Colab, you may wish to mount your Google Drive, once that is done, you can create a symlinked\n",
    "# directory that you can use to store the checkpoint directories in.\n",
    "#\n",
    "# For this example we are using some Google data that can be learned and trained relatively quickly\n",
    "# \n",
    "# NOTE: Gretel Synthetic paths must NOT contain whitespaces, which is why we have to symlink to a more local directory\n",
    "# in /content. Unfortunately, Google Drive mounts contain whitespaces either in the \"My drive\" or \"Shared drives\" portion\n",
    "# of the path\n",
    "#\n",
    "# !ln -s \"/content/drive/Shared drives[My Drive]/YOUR_TARGET_DIRECTORY\" checkpoints\n",
    "#\n",
    "# !pip install -U gretel-synthetics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from gretel_synthetics.batch import DataFrameBatch\n",
    "\n",
    "source_df = pd.read_csv(\"https://gretel-public-website.s3-us-west-2.amazonaws.com/tests/synthetics/data/USAdultIncome14K.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14000, 15)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we create a dict with our config params, these are identical to when creating a LocalConfig object\n",
    "#\n",
    "# NOTE: We do not specify a ``input_data_path`` as this is automatically created for each batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "checkpoint_dir = str(Path.cwd() / \"test-model-2\")\n",
    "\n",
    "config_template = {\n",
    "    \"epochs\": 10,\n",
    "    \"max_line_len\": 2048,\n",
    "    \"vocab_size\": 200000,\n",
    "    \"field_delimiter\": \",\",\n",
    "    \"overwrite\": True,\n",
    "    \"checkpoint_dir\": checkpoint_dir\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 12:58:50,096 : MainThread : INFO : Creating directory structure for batch jobs...\n"
     ]
    }
   ],
   "source": [
    "# Create our batch handler. During construction, checkpoint directories are automatically created\n",
    "# based on the configured batch size\n",
    "batcher = DataFrameBatch(df=source_df, config=config_template, batch_size=5)\n",
    "\n",
    "# Optionally, you can also provide your own batches, which can be a list of lists of strings:\n",
    "#\n",
    "# my_batches = [[\"col1\", \"col2\"], [\"col3\", \"col4\", \"col5\"]]\n",
    "# batcher = DataFrameBatch(df=source_df, batch_headers=my_batches, config=config_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 12:58:51,411 : MainThread : INFO : Generating training DF and CSV for batch 0\n",
      "2021-02-04 12:58:51,454 : MainThread : INFO : Generating training DF and CSV for batch 1\n",
      "2021-02-04 12:58:51,500 : MainThread : INFO : Generating training DF and CSV for batch 2\n"
     ]
    }
   ],
   "source": [
    "# Next we generate our actual training DataFrames and Training text files\n",
    "#\n",
    "# Each batch directory will now have it's own \"train.csv\" file\n",
    "# Each Batch object now has a ``training_df`` associated with it\n",
    "batcher.create_training_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 12:58:53,708 : MainThread : INFO : Loading training data from /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_0/train.csv\n",
      "2021-02-04 12:58:53,739 : MainThread : INFO : Training SentencePiece tokenizer\n",
      "2021-02-04 12:58:54,116 : MainThread : INFO : Loading tokenizer from: m.model\n",
      "2021-02-04 12:58:54,127 : MainThread : INFO : Tokenizer model vocabulary size: 5536 tokens\n",
      "2021-02-04 12:58:54,128 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'30<d>?<d>157289<d>11th<d>7<n>'\n",
      " ---- sample tokens mapped to pieces ---- > \n",
      "▁, 3, 0, <d>, ?, <d>, 1, 5, 72, 89, <d>, 1, 1, th, <d>, 7, <n>\n",
      "\n",
      "2021-02-04 12:58:54,128 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'30<d>?<d>157289<d>11th<d>7<n>'\n",
      " ---- sample tokens mapped to int ---- > \n",
      "25, 47, 4, 55, 4, 1140, 13, 4, 50, 41, 4, 34, 3\n",
      "\n",
      "2021-02-04 12:58:54,131 : MainThread : WARNING : ***** GPU not found, CPU will be used instead! *****\n",
      "2021-02-04 12:58:54,132 : MainThread : INFO : Tokenizing training data\n",
      "100%|██████████| 14000/14000 [00:00<00:00, 46378.37it/s]\n",
      "2021-02-04 12:58:54,436 : MainThread : INFO : Creating and shuffling tensorflow dataset\n",
      "2021-02-04 12:58:55,224 : MainThread : INFO : Initializing synthetic model\n",
      "2021-02-04 12:58:55,649 : MainThread : INFO : Using keras.optimizers.RMSprop optimizer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (64, None, 256)           1417216   \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (64, None, 5536)          1422752   \n",
      "=================================================================\n",
      "Total params: 3,890,592\n",
      "Trainable params: 3,890,592\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/10\n",
      "48/48 [==============================] - 41s 799ms/step - loss: 5.2312 - accuracy: 0.1426\n",
      "Epoch 2/10\n",
      "48/48 [==============================] - 41s 854ms/step - loss: 3.8833 - accuracy: 0.1941\n",
      "Epoch 3/10\n",
      "48/48 [==============================] - 45s 917ms/step - loss: 1.6319 - accuracy: 0.6281\n",
      "Epoch 4/10\n",
      "48/48 [==============================] - 46s 936ms/step - loss: 1.0442 - accuracy: 0.7637\n",
      "Epoch 5/10\n",
      "48/48 [==============================] - 45s 915ms/step - loss: 0.9645 - accuracy: 0.7735\n",
      "Epoch 6/10\n",
      "48/48 [==============================] - 42s 868ms/step - loss: 0.9236 - accuracy: 0.7817\n",
      "Epoch 7/10\n",
      "48/48 [==============================] - 41s 853ms/step - loss: 0.8996 - accuracy: 0.7849\n",
      "Epoch 8/10\n",
      "48/48 [==============================] - 42s 869ms/step - loss: 0.8804 - accuracy: 0.7877\n",
      "Epoch 9/10\n",
      "48/48 [==============================] - 43s 890ms/step - loss: 0.8650 - accuracy: 0.7909\n",
      "Epoch 10/10\n",
      "48/48 [==============================] - 41s 851ms/step - loss: 0.8490 - accuracy: 0.7930\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 13:06:03,825 : MainThread : INFO : Saving model history to model_history.csv\n",
      "2021-02-04 13:06:03,828 : MainThread : INFO : Saving model to /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_0/synthetic\n",
      "2021-02-04 13:06:03,833 : MainThread : INFO : Loading training data from /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_1/train.csv\n",
      "2021-02-04 13:06:03,864 : MainThread : INFO : Training SentencePiece tokenizer\n",
      "2021-02-04 13:06:04,262 : MainThread : INFO : Loading tokenizer from: m.model\n",
      "2021-02-04 13:06:04,265 : MainThread : INFO : Tokenizer model vocabulary size: 118 tokens\n",
      "2021-02-04 13:06:04,266 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'Never-married<d>?<d>Unmarried<d>White<d>Male<n>'\n",
      " ---- sample tokens mapped to pieces ---- > \n",
      "▁, N, e, ve, r, -, ma, r, ried, <d>, ?, <d>, U, n, m, arried, <d>, W, hi, t, e, <d>, M, ale, <n>\n",
      "\n",
      "2021-02-04 13:06:04,267 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'Never-married<d>?<d>Unmarried<d>White<d>Male<n>'\n",
      " ---- sample tokens mapped to int ---- > \n",
      "7, 14, 5, 54, 13, 6, 12, 33, 4, 80, 4, 69, 29, 12, 33, 4, 9, 8, 10, 4, 11, 17, 3\n",
      "\n",
      "2021-02-04 13:06:04,270 : MainThread : WARNING : ***** GPU not found, CPU will be used instead! *****\n",
      "2021-02-04 13:06:04,271 : MainThread : INFO : Tokenizing training data\n",
      "100%|██████████| 14000/14000 [00:00<00:00, 37728.11it/s]\n",
      "2021-02-04 13:06:04,644 : MainThread : INFO : Creating and shuffling tensorflow dataset\n",
      "2021-02-04 13:06:05,734 : MainThread : INFO : Initializing synthetic model\n",
      "2021-02-04 13:06:06,134 : MainThread : INFO : Using keras.optimizers.RMSprop optimizer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (64, None, 256)           30208     \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_4 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (64, None, 118)           30326     \n",
      "=================================================================\n",
      "Total params: 1,111,158\n",
      "Trainable params: 1,111,158\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/10\n",
      "69/69 [==============================] - 38s 504ms/step - loss: 3.8326 - accuracy: 0.1691\n",
      "Epoch 2/10\n",
      "69/69 [==============================] - 36s 517ms/step - loss: 0.5031 - accuracy: 0.8578\n",
      "Epoch 3/10\n",
      "69/69 [==============================] - 35s 505ms/step - loss: 0.2446 - accuracy: 0.9152\n",
      "Epoch 4/10\n",
      "69/69 [==============================] - 35s 504ms/step - loss: 0.2116 - accuracy: 0.9226\n",
      "Epoch 5/10\n",
      "69/69 [==============================] - 36s 508ms/step - loss: 0.2009 - accuracy: 0.9250\n",
      "Epoch 6/10\n",
      "69/69 [==============================] - 36s 513ms/step - loss: 0.1953 - accuracy: 0.9264\n",
      "Epoch 7/10\n",
      "69/69 [==============================] - 37s 529ms/step - loss: 0.1926 - accuracy: 0.9269\n",
      "Epoch 8/10\n",
      "69/69 [==============================] - 36s 511ms/step - loss: 0.1890 - accuracy: 0.9275\n",
      "Epoch 9/10\n",
      "69/69 [==============================] - 33s 470ms/step - loss: 0.1899 - accuracy: 0.9278\n",
      "Epoch 10/10\n",
      "69/69 [==============================] - 32s 462ms/step - loss: 0.1864 - accuracy: 0.9289\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 13:12:01,853 : MainThread : INFO : Saving model history to model_history.csv\n",
      "2021-02-04 13:12:01,855 : MainThread : INFO : Saving model to /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_1/synthetic\n",
      "2021-02-04 13:12:01,857 : MainThread : INFO : Loading training data from /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_2/train.csv\n",
      "2021-02-04 13:12:01,883 : MainThread : INFO : Training SentencePiece tokenizer\n",
      "2021-02-04 13:12:02,124 : MainThread : INFO : Loading tokenizer from: m.model\n",
      "2021-02-04 13:12:02,127 : MainThread : INFO : Tokenizer model vocabulary size: 265 tokens\n",
      "2021-02-04 13:12:02,128 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'0<d>0<d>40<d>United-States<d><=50K<n>'\n",
      " ---- sample tokens mapped to pieces ---- > \n",
      "▁, 0, <d>, 0, <d>, 4, 0, <, d, >, U, ni, te, d, -, S, t, ate, s, <, d, >, <, =, 50, K, <n>\n",
      "\n",
      "2021-02-04 13:12:02,129 : MainThread : INFO : Mapping first line of training data\n",
      "\n",
      "'0<d>0<d>40<d>United-States<d><=50K<n>'\n",
      " ---- sample tokens mapped to int ---- > \n",
      "9, 5, 4, 5, 4, 19, 4, 14, 15, 6, 13, 10, 12, 18, 6, 11, 4, 16, 17, 7, 8, 3\n",
      "\n",
      "2021-02-04 13:12:02,131 : MainThread : WARNING : ***** GPU not found, CPU will be used instead! *****\n",
      "2021-02-04 13:12:02,132 : MainThread : INFO : Tokenizing training data\n",
      "100%|██████████| 14000/14000 [00:00<00:00, 57709.19it/s]\n",
      "2021-02-04 13:12:02,377 : MainThread : INFO : Creating and shuffling tensorflow dataset\n",
      "2021-02-04 13:12:03,032 : MainThread : INFO : Initializing synthetic model\n",
      "2021-02-04 13:12:03,411 : MainThread : INFO : Using keras.optimizers.RMSprop optimizer\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (64, None, 256)           67840     \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_6 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_7 (LSTM)                (64, None, 256)           525312    \n",
      "_________________________________________________________________\n",
      "dropout_11 (Dropout)         (64, None, 256)           0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (64, None, 265)           68105     \n",
      "=================================================================\n",
      "Total params: 1,186,569\n",
      "Trainable params: 1,186,569\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/10\n",
      "46/46 [==============================] - 22s 419ms/step - loss: 3.8962 - accuracy: 0.1165\n",
      "Epoch 2/10\n",
      "46/46 [==============================] - 20s 433ms/step - loss: 2.3947 - accuracy: 0.3537\n",
      "Epoch 3/10\n",
      "46/46 [==============================] - 21s 457ms/step - loss: 0.3964 - accuracy: 0.9121\n",
      "Epoch 4/10\n",
      "46/46 [==============================] - 22s 465ms/step - loss: 0.2767 - accuracy: 0.9385\n",
      "Epoch 5/10\n",
      "46/46 [==============================] - 23s 479ms/step - loss: 0.2525 - accuracy: 0.9443\n",
      "Epoch 6/10\n",
      "46/46 [==============================] - 22s 460ms/step - loss: 0.2412 - accuracy: 0.9460\n",
      "Epoch 7/10\n",
      "46/46 [==============================] - 20s 418ms/step - loss: 0.2391 - accuracy: 0.9460\n",
      "Epoch 8/10\n",
      "46/46 [==============================] - 21s 444ms/step - loss: 0.2365 - accuracy: 0.9466\n",
      "Epoch 9/10\n",
      "46/46 [==============================] - 26s 551ms/step - loss: 0.2339 - accuracy: 0.9469\n",
      "Epoch 10/10\n",
      "46/46 [==============================] - 22s 470ms/step - loss: 0.2344 - accuracy: 0.9469\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-02-04 13:15:40,605 : MainThread : INFO : Saving model history to model_history.csv\n",
      "2021-02-04 13:15:40,607 : MainThread : INFO : Saving model to /Users/jtm/gretel/gretel-synthetics/examples/test-model-2/batch_2/synthetic\n"
     ]
    }
   ],
   "source": [
    "# Now we can trigger each batch to train\n",
    "batcher.train_all_batches()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Next, we can trigger all batched models to create output. This loops over each model and will attempt to generate\n",
    "# ``gen_lines`` valid lines for each model. This method returns a dictionary of bools that is indexed by batch number\n",
    "# and tells us if, for each batch, we were able to generate the requested number of valid lines\n",
    "status = batcher.generate_all_batch_lines(num_lines=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batcher.batches[2].gen_data_stream.getvalue()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can grab a DataFrame for each batch index\n",
    "batcher.batch_to_df(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally, we can re-assemble all synthetic batches into our new synthetic DF\n",
    "batcher.batches_to_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read only mode\n",
    "\n",
    "If you've already created a model(s) and simply want to load that data to generate more lines, you can use the read-only mode for the batch interface. No input DataFrame is required and it will automatically try and load model information from a primary checkpoint directory.\n",
    "\n",
    "Additionally, you can also control the number of lines you wish to generate with the ``num_lines`` parameter for generation. This option exists for write mode as well and overrides the number of lines specified in the synthetic config that was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_batch = DataFrameBatch(mode=\"read\", checkpoint_dir=checkpoint_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_batch.generate_all_batch_lines(num_lines=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_batch.batches_to_df()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}