text_demo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Formatting refined audio\n",
    "\n",
    "In this tutorial, you will learn how to apply `VocalForge.text` pipelines on audio files either manually processed or through through `VocalForge.audio`.\n",
    "\n",
    "Each step is a step toward converting plain audio file(s) into a standardized format that is perfect for TTS training, speaker identification, or content Analysis (and more). These can be used independently of eachother, so if theres a step that isn't useful to you, feel free to simply skip it. This demo is just showing all capabilities of VocalForge. The final output of this dataset follows the same format to [LJSpeech](https://keithito.com/LJ-Speech-Dataset/) but the class can be pretty easily modified to export another format.\n",
    "\n",
    "The pipelines are as follows:\n",
    "\n",
    "- `Transcribe` Using OpenAI's Whisper, it goes through each file and, well, transcribes them into a text document. Pretty self explicatory. \n",
    "\n",
    "- `NormalizeText` Taking the raw text from Whisper, NormalizeText will output three text files, along with a copied version of the input audio file. The three text files split the text into utterances, such as sentences or exclamations. Each file will vary in terms of text normalization in order to expand compatibility with any range of preprocessors.\n",
    "\n",
    "- `Segment` generates timestamps (and confidence levels) for each utterence generated from `NormalizeText` that matches to segments of the audio file.\n",
    "\n",
    "- `SplitAudio` is also very straightforward. It splits the audio file based on timestamps, assuming the segment reaches a certain confidence level (alongwith offsets and padding, but we will get to that)\n",
    "\n",
    "- `Export` combines the text generated from `NormalizeText` and the audio files from `SplitAudio` in a nice little folder, alongwith a metadata file.\n",
    "\n",
    "More pipelines are coming soon™\n",
    "\n",
    "NOTE: It is highly reccomended to run this on a conda enviroment if running locally by running the command\n",
    "`conda create -n VocalForge python=3.8 pytorch=1.11.0 torchvision=0.12.0 torchaudio=0.11.0 cudatoolkit=11.3.1 -c pytorch`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Started"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's get to creating our work directory and installing `VocalForge`'s text features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/rio/Desktop/VocalForgeDev/VocalForge\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "root_path = Path.cwd()\n",
    "print(root_path)\n",
    "\n",
    "work_text_path = root_path / 'work' / 'text'\n",
    "\n",
    "work_text_path.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this might take a while\n",
    "%pip install .[text]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[NeMo W 2023-08-26 19:55:18 optimizers:54] Apex was not found. Using the lamb or fused_adam optimizer will error out.\n",
      "[NeMo W 2023-08-26 19:55:18 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.\n"
     ]
    }
   ],
   "source": [
    "from VocalForge.text.text_utils import create_core_folders\n",
    "\n",
    "root_path = Path.cwd()\n",
    "work_text_path = root_path / 'work' / 'text'\n",
    "\n",
    "folder_names = ['Input_Audio', 'Transcription', 'Normalized', 'Segments', 'Sliced_Audio', 'Dataset']\n",
    "\n",
    "create_core_folders(folder_names, workdir=str(work_text_path))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import cuda\n",
    "cuda.empty_cache()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to put your audio in the 'Input_Audio' folder at this point!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transcription"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now all the sitting around is done, we can finally get to...pushing a button and then waiting a little while longer! Welcome to AI. There are a few things we can do however before pressing the play button to make our process a little bit better."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's talk models. Whisper can use models of varying size. Personally, I'd go with the largest model you can get your grubby hands on, which depends on the VRAM your GPU has. `large` requires ~10gb, while `medium` and `small` require 5GB and 2GB respectively.  \n",
    "\n",
    "The other thing to take note of is the prompt. This allows us to guide the diction Whisper will include in the transcription. For instance, if the speaker(s) stutter or use filler words frequently, and our goal is to have as accurate of a transcription as possible, we can include an example of that (as seen below). This also works on slang that isn't in the standard lexicon. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VocalForge.text import Transcribe\n",
    "\n",
    "transcriber = Transcribe(\n",
    "    input_dir=str(work_text_path / 'Input_Audio'),\n",
    "    output_dir=str(work_text_path / 'Transcription'),\n",
    "    model='small',\n",
    "    prompt=\"uh, um, I...I think that what, what you're saying is terrible, mhm. Ya'know?\",\n",
    "    do_write=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcriber.run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so this one is pretty straightforward 🙌 Still some things to know:\n",
    "\n",
    "`max_length` and `min_length` are pretty self explanatory. The only thing to note is that `min_length` is the minimum length of the utterance after normalization. This is to prevent the utterance from being too short, which can cause issues with the model.\n",
    "\n",
    "`lang` can be changed to any language supported by en, es, ru, cn, and probably more. This will change the normalization process to be more accurate to the language and its phonetics.\n",
    "\n",
    "`model` can be changed pending a model that is an upgrade from the current one or to better fit your language. The list can be found [here](https://catalog.ngc.nvidia.com/models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VocalForge.text import NormalizeText\n",
    "\n",
    "normalizer = NormalizeText(\n",
    "    input_dir=str(work_text_path / 'Transcription'),\n",
    "    output_dir=str(work_text_path / 'Normalized'),\n",
    "    audio_dir=str(work_text_path / 'Input_Audio'),\n",
    "    max_length=25,\n",
    "    min_length=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Splitting text in /home/rio/Desktop/VocalForgeDev/VocalForge/work/text/Transcription/DATA3.txt into sentences.\n",
      "Using NeMo normalization tool...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:01<00:00, 59.68it/s]\n",
      " 27%|██▋       | 27/100 [00:02<00:10,  6.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING! Your input is too long and could take a long time to normalize.Use split_text_into_sentences() to make the input shorter and then call normalize_list().\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:06<00:00, 16.66it/s]\n",
      "100%|██████████| 21/21 [00:00<00:00, 30.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/rio/Desktop/VocalForgeDev/VocalForge/work/text/Normalized/DATA3/DATA3\n",
      "/home/rio/Desktop/VocalForgeDev/VocalForge/work/text/Normalized/DATA3/DATA3_with_punct_normalized\n",
      "/home/rio/Desktop/VocalForgeDev/VocalForge/work/text/Normalized/DATA3/DATA3_with_punct\n"
     ]
    }
   ],
   "source": [
    "normalizer.run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Segmentation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to align the text and audio! Using another NVIDIA ASR model, we can generate timestamps for each utterance. There's not much to say here, but the `window_size` may be changed based on the length of each audio file, as too small a window size can hinder the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VocalForge.text import Segment\n",
    "\n",
    "segmenter = Segment(\n",
    "    input_dir=str(work_text_path / 'Normalized'),\n",
    "    output_dir=str(work_text_path / 'Segments'),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/rio/Desktop/VocalForgeDev/VocalForge/work/text/Normalized/DATA3/DATA3.wav\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd569d2e1a7f441a8d73ba390c9e833a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Transcribing:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 221/221 [00:00<00:00, 4838.45it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average segmentation loss: -0.8618766769169445\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "segmenter.run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have these segments, things won't be perfect. Theres some things we can do to make it better, such as padding and offsets. I made a little function to help with this.\n",
    "\n",
    "`threshold` is the confidence level that the segment must reach in order to be included in the final dataset. The closer to 0, the more confident the model is of the segment timing to be included. Feel free to change this around to your liking (it won't affect the dataset, just the previews)\n",
    "\n",
    "This will iterate through each file, prompting you to input an offset and padding value. Type in the values and press enter. The function will then display an audio clip with the specified offset and padding. Try to find the best values for each file. Once you have, type \"stop\" and it will move on to the next file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.io.wavfile as wav\n",
    "import IPython.display as ipd\n",
    "threshold = 0.75\n",
    "\n",
    "def test_offset_padding(alignment_file: str):\n",
    "    offset = 0\n",
    "    padding = 0\n",
    "    alignment_path = Path(alignment_file)\n",
    "    if not alignment_path.exists():\n",
    "        raise ValueError(f\"{alignment_file} not found\")\n",
    "\n",
    "    print(f\"Reading {alignment_file}\")\n",
    "    with alignment_path.open(\"r\") as f:\n",
    "        sample = [next(f) for _ in range(10)]\n",
    "        offset = input('Input offset (type \"stop\" to end): ')\n",
    "        padding = input(\"Input padding: \")\n",
    "        padding = float(padding)\n",
    "\n",
    "        while True:\n",
    "            print(f\"offset: {offset}, padding: {padding}\")\n",
    "            for line in sample:\n",
    "                line = line.split(\"|\")\n",
    "                if len(line) == 1:\n",
    "                    audio_file = line[0].strip()\n",
    "                    continue\n",
    "                text = line[2]\n",
    "                line = line[0].split()\n",
    "                sampling_rate, signal = wav.read(audio_file)\n",
    "                if float(line[2]) < -threshold:\n",
    "                    continue\n",
    "                segment = [\n",
    "                    float(line[0]) + float(offset) + padding,\n",
    "                    float(line[1]) + float(offset) - padding,\n",
    "                ]\n",
    "                if float(line[0]) + float(offset) < 0:\n",
    "                    segment[0] = 0\n",
    "                st, end = segment\n",
    "                audio = signal[round(st * sampling_rate) : round(end * sampling_rate)]\n",
    "                print(f\"loss: {line[2]}\")\n",
    "                print(f\"sample text: {text}\")\n",
    "                ipd.display(ipd.Audio(audio, rate=sampling_rate))\n",
    "            previous_offset = offset\n",
    "            offset = input('Input offset (type \"stop\" to end): ')\n",
    "            if offset == \"stop\":\n",
    "                offset = previous_offset\n",
    "                print(\n",
    "                    f\"final values for {alignment_file}: offset: {offset}, padding: {padding}\"\n",
    "                )\n",
    "                break\n",
    "            padding = float(input(\"Input padding: \"))\n",
    "        return offset, padding\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'test_offset_padding' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[3], line 7\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[39mfor\u001b[39;00m file \u001b[39min\u001b[39;00m get_files(segment_dir, ext\u001b[39m=\u001b[39m\u001b[39m'\u001b[39m\u001b[39m.txt\u001b[39m\u001b[39m'\u001b[39m):\n\u001b[1;32m      6\u001b[0m     file_dir \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mjoin(segment_dir, file)\n\u001b[0;32m----> 7\u001b[0m     offset, padding \u001b[39m=\u001b[39m test_offset_padding(alignment_file\u001b[39m=\u001b[39mfile_dir)\n\u001b[1;32m      8\u001b[0m     offsets\u001b[39m.\u001b[39mappend(offset)\n\u001b[1;32m      9\u001b[0m     paddings\u001b[39m.\u001b[39mappend(padding)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'test_offset_padding' is not defined"
     ]
    }
   ],
   "source": [
    "from VocalForge.text import get_files\n",
    "\n",
    "offsets = []\n",
    "paddings = []\n",
    "work_text_path = Path.cwd() / 'work' / 'text'\n",
    "segment_dir = work_text_path / 'Segments'\n",
    "\n",
    "for file in get_files(str(segment_dir), ext='.txt'):\n",
    "    offset, padding = test_offset_padding(alignment_file=str(segment_dir / file))\n",
    "    offsets.append(offset)\n",
    "    paddings.append(padding)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting Audio"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now move onto cutting the audio files. Passing along the padding and offset values, alongwith the confidence threshold and max length each split audio file (in s), we can now cut the audio files into segments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VocalForge.text.split_audio import SplitAudio\n",
    "\n",
    "threshold = 0.6\n",
    "max_duration = 20\n",
    "\n",
    "split = SplitAudio(\n",
    "    input_dir=str(segment_dir),\n",
    "    output_dir=str(work_text_path / 'Sliced_Audio'),\n",
    "    threshold=threshold,\n",
    "    offsets=offsets,\n",
    "    paddings=paddings,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original duration  : 31min\n",
      "High score segments: 12min (39%)\n",
      "Low score segments : 9min (30%)\n"
     ]
    }
   ],
   "source": [
    "split.run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Creation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost done! This step essentially combines the audio we just split, alongwith with the text normalization we did earlier. The product of this will be a folder containing the audio files and a metadata file that contains the normalized text and the name of the corresponding audio file.\n",
    "\n",
    "Note that currently it will only export as a LJSpeech format, but it can be easily modified to export in any format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VocalForge.text.create_dataset import GenerateDataset\n",
    "\n",
    "work_text_path = Path.cwd() / 'work' / 'text'\n",
    "segment_dir = work_text_path / 'Segments'\n",
    "threshold = 0.6\n",
    "\n",
    "dataset = GenerateDataset(\n",
    "    segment_dir=str(segment_dir),\n",
    "    sliced_aud_dir=str(work_text_path / 'Sliced_Audio'),\n",
    "    output_dir=str(work_text_path / 'Dataset'),\n",
    "    threshold=threshold,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.run()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, every now and again there might be an extra audio file or metadata entry that will throw any preprocesser into a fit. But don't fret! This function will help you find and remove them. This can also sync mannual changes in either removing metadata entries or audio files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['DATA3_0105.wav', 'DATA3_0088.wav', 'DATA3_0036.wav']\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "bad_files = []\n",
    "work_text_path = Path.cwd() / 'work' / 'text'\n",
    "dataset_dir = work_text_path / 'Dataset'\n",
    "\n",
    "# Iterating through .wav files, checking their size\n",
    "for file_path in (dataset_dir / 'wavs').iterdir():\n",
    "    if file_path.stat().st_size < 50000:\n",
    "        bad_files.append(file_path.name)\n",
    "\n",
    "print(bad_files)\n",
    "\n",
    "# Reading and cleaning dataset\n",
    "df = pd.read_csv(dataset_dir / 'metadata.csv', sep='|', on_bad_lines='skip')\n",
    "for index, row in df.iterrows():\n",
    "    if row[0] + '.wav' in bad_files:\n",
    "        df.drop(index, inplace=True)\n",
    "        (dataset_dir / 'wavs' / (row[0] + '.wav')).unlink()\n",
    "\n",
    "df.to_csv(dataset_dir / 'metadata.csv', sep='|', index=False)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And done! Get curatin'!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}