ESPnet2/Course/CMU_SpeechRecognition_Fall2022/new_task_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNbxe3gyfbPi"
      },
      "source": [
        "# CMU 11751/18781 Fall 2022: ESPnet Tutorial2 (New task)\n",
        "\n",
        "[ESPnet](https://github.com/espnet/espnet) is a widely-used end-to-end speech processing toolkit. It has supported various speech processing tasks. ESPnet uses PyTorch as a main deep learning engine, and also follows Kaldi style data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments.\n",
        "\n",
        "Main references:\n",
        "- [ESPnet repository](https://github.com/espnet/espnet)\n",
        "- [ESPnet documentation](https://espnet.github.io/espnet/)\n",
        "- [ESPnet tutorial in Speech Recognition and Understanding (Fall 2021)](https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_tutorial_2021_CMU_11751_18781.ipynb)\n",
        "- [Recitation in Multilingual NLP (Spring 2022)](https://colab.research.google.com/drive/1tY6PxF_M5Nx5n488x0DrpujJOyqW-ATi?usp=sharing)\n",
        "- [ESPnet tutorial1 in Speech Recognition and Understanding (Fall 2022)](https://colab.research.google.com/drive/1d3yMY3xv_hpr2mznskZpF2Cj-bEooDv-?usp=sharing)\n",
        "\n",
        "\n",
        "Author: Jiatong Shi (jiatongs@andrew.cmu.edu)\n",
        "\n",
        "We would like to thank You (Neil) Zhang for kindly helping the hands-on tutorial and sharing his knowledge on the task. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pwxgDdlMUE60"
      },
      "source": [
        "## ❗Important Notes❗\n",
        "- We are using Colab to show the demo. However, Colab has some constraints on the total GPU runtime. If you use too much GPU time, you may not be able to use GPU for some time.\n",
        "- There are multiple in-class checkpoints ✅ throughout this tutorial. There will also be some after-class excersices 📗 after the tutorial. **Your participation points are based on these tasks.** Please try your best to follow all the steps! If you encounter issues, please notify the TAs as soon as possible so that we can make an adjustment for you.\n",
        "- Please submit PDF files of your completed notebooks to Gradescope. You can print the notebook using `File -> Print` in the menu bar.\n",
        "- This tutorial covers some advanced usage of ESPnet, which is the extension of the first tutorial."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGg1N9jufpf2"
      },
      "source": [
        "## Useful links\n",
        "\n",
        "- Installation https://espnet.github.io/espnet/installation.html\n",
        "- Usage https://espnet.github.io/espnet/espnet2_tutorial.html\n",
        "- Reference of task class in ESPnet https://espnet.github.io/espnet/espnet2_task.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wl9JFMNJ5iYu"
      },
      "source": [
        "## Objectives\n",
        "After this tutorial, you are expected to know:\n",
        "- How to add new task in ESPnet2\n",
        "- How to add new models in ESPnet2\n",
        "- How to create a new recipe (and template) of a new task from scratch\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff1e4zYNaBcO"
      },
      "source": [
        "## Function to print date and time\n",
        "\n",
        "We first define a function to print the current date and time, which will be used in multiple places below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U674ZP6DaA3M"
      },
      "outputs": [],
      "source": [
        "def print_date_and_time():\n",
        "  from datetime import datetime\n",
        "  import pytz\n",
        "\n",
        "  now = datetime.now(pytz.timezone(\"America/New_York\"))\n",
        "  print(\"=\" * 60)\n",
        "  print(f' Current date and time: {now.strftime(\"%m/%d/%Y %H:%M:%S\")}')\n",
        "  print(\"=\" * 60)\n",
        "\n",
        "# example output\n",
        "print_date_and_time()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0LaeC3RzECk"
      },
      "source": [
        "# Install ESPnet (Almost same procedure as your first tutorial)\n",
        "\n",
        "## Download ESPnet\n",
        "\n",
        "We use `git clone` to download the source code of ESPnet and then go to a specific commit.\n",
        "\n",
        "**Important:** In other versions of ESPnet, you may encounter errors related to imcompatible package versions (`numba`). Please use the same commit to avoid such issues.\n",
        "\n",
        "Note that we are using another branch `espnet_tutorial_asvspoof` instead of \"master\". You can also use your own fork to proceed the following sections if you want to use Github to save your code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "St7lke2P0GUP"
      },
      "outputs": [],
      "source": [
        "# It takes a few seconds\n",
        "!git clone --depth 5 -b 2022fall_new_task_tutorial https://github.com/espnet/espnet\n",
        "\n",
        "# We use a specific commit just for reproducibility.\n",
        "%cd /content/espnet\n",
        "!git checkout 9cff98a78ceaa4d85843be0a50b369ec826b27f6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lZGnBSEaz1Zt"
      },
      "source": [
        "## Setup Python environment based on anaconda + Install ESPnet\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4F80yqAIz86B"
      },
      "outputs": [],
      "source": [
        "# It takes 30 seconds\n",
        "%cd /content/espnet/tools\n",
        "!./setup_anaconda.sh anaconda espnet 3.9\n",
        "\n",
        "# It may take 12 minutes\n",
        "%cd /content/espnet/tools\n",
        "!make TH_VERSION=1.12.1 CUDA_VERSION=11.6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41alrKGO4d3v"
      },
      "source": [
        "# What we provide you and what you need to proceed\n",
        "\n",
        "We have provide you most of the files needed for ASVSpoof recipe. So you do not need to add any additional files. However, noted that some of the files are not complete and need your completion to proceed. For a quick overview of the whole layout of the new task, please refer to https://github.com/espnet/espnet/compare/master...2022fall_new_task_tutorial \n",
        "\n",
        "As elaborated in the warming-up, we have shown that there are two core components for a new task in ESPnet: a **task library** and correponding **recipe setups**. For the following of the section, we will briefly show the overall layout of adding the ASVSpoof task in ESPnet. The listed files are almost the minimum requirements to add a new task in ESPnet."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6uBU3Mahsaj"
      },
      "source": [
        "**Task library for ASVSpoof**\n",
        "\n",
        "Followings are a list of files adding to ESPnet for ASVSpoof (files in \"\" are ones that need modifications)\n",
        "```\n",
        "- espnet2\n",
        "  - bin\n",
        "    - asvspoof_train.py # Major entry point for asvspoof\n",
        "    - \"asvspoof_inference.py\" (Checkpoint 4) # Inference scripts for asvspoof\n",
        "  - asvspoof\n",
        "    - decoder\n",
        "      - __init__.py\n",
        "      - abs_decoder.py # abstract class for decoder in ASVSpoof\n",
        "      - \"linear_decoder.py\" (Checkpoint 3) # simple linear decoder for ASVSpoof\n",
        "    - loss\n",
        "      - __init__.py\n",
        "      - abs_loss.py # abstract class for loss in ASVSpoof\n",
        "      - binary_loss.py # naive binary class loss for ASVSpoof\n",
        "      - am_softmax.py\n",
        "      - \"oc_softmax.py\" (Bouns)\n",
        "    - __init__.py\n",
        "    - \"espnet_model.py\" (Bouns)\n",
        "  - tasks\n",
        "    - \"asvspoof.py\" (Checkpoint 2)\n",
        "```\n",
        "\n",
        "To help you understand more, we would recommend you to check the layout of other tasks (e.g., ASR, TTS, ST, etc.) to understand how the codebase is functioning."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Recipe for ASVSpoof**\n",
        "\n",
        "Followings are a list of files adding to ESPnet for ASVSpoof (files in boldface are ones that need modifications)\n",
        "```\n",
        "- egs2\n",
        "  - TEMPLATE\n",
        "    - asvspoof1\n",
        "      - \"asvspoof.sh\" (Checkpoint 1)\n",
        "      - others\n",
        "  - espnet_tutorial\n",
        "    - asvspoof11\n",
        "      - conf\n",
        "      - \"asvspoof.sh” (Checkpoint 1)\n",
        "      - local\n",
        "        - \"data_prep.py\" (Bouns)\n",
        "        - \"data.sh\" (Bouns)\n",
        "      - \"run.sh\" (Checkpoint 5)\n",
        "      - scripts\n",
        "      - pyscripts\n",
        "      - utils\n",
        "      - steps\n",
        "      - path.sh\n",
        "      - db.sh\n",
        "      - cmd.sh\n",
        "```\n",
        "\n",
        "Noted that because of the [symlink](https://www.freecodecamp.org/news/symlink-tutorial-in-linux-how-to-create-and-remove-a-symbolic-link/), the `asvspoof.sh` is essentially the same for checkpoint 1."
      ],
      "metadata": {
        "id": "lZ12-sIq9YXJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ASVSpoof data preparation\n",
        "\n",
        "As discussed in the warm-up session, ASVSpoof aims to conduct a binary classfication. As the task layout is a bit different from the ASR task we touched on the first tutorial, so we need to use a different format to formulate the data. For here, to keep the simplicity, we stil use the exact same file as the first tutorial:\n",
        "```\n",
        "wav.scp text utt2spk spk2utt\n",
        "```\n",
        "But on the other hand, we change the format of text into\n",
        "```\n",
        "utt_id1 0\n",
        "utt_id2 1\n",
        "utt_id3 0\n",
        "```\n",
        "where 0 represents real speech and 1 stands for fake speech."
      ],
      "metadata": {
        "id": "35armcrL_9p7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Download dataset\n",
        "\n",
        "We first download the data from google drive. Noted that the data is a subset of the ASVSpoof2019 Challenge."
      ],
      "metadata": {
        "id": "PbW_n91zBZOm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# a few seconds\n",
        "%cd /content/espnet/egs2/espnet_tutorial/asvspoof1/\n",
        "!gdown 1HRdjjmGXBTXOqOq9iijuXPCA4y_46OzP\n",
        "!unzip espnet_tutorial_asvspoof.zip"
      ],
      "metadata": {
        "id": "u1CB5FBCBYjp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prepare data (Stage1 & Stage2)\n",
        "\n",
        "This time, we make the task template to be as simple as possible. The data preparation will be only two stages, including basic data preparation and wave format. "
      ],
      "metadata": {
        "id": "08P8jvHUBq4W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# It may take around 6 minutes\n",
        "!./asvspoof.sh --stage 1 --stop_stage 2 --train_set train --valid_set dev --test_sets \"eval\""
      ],
      "metadata": {
        "id": "MRomfZb2Ix93"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ASVSpoof collect stats (✅ Checkpint 1 (1 point))\n",
        "Similar to the previous tutorial, we collect the statisitcs for the data. \n",
        "\n",
        "In the process, the data will be passed into a iterable loader. However, remember that the `text` file is no longer the format as the ASR recipe. Therefore, we will need to use another data loader to load the corresponding information.\n",
        "\n",
        "Fortunately, we have a wide range of data loaders for choices, which is listing in [here](https://github.com/espnet/espnet/blob/a7bd6522b32ec6472c13f6a2289dcdff4a846c12/espnet2/train/dataset.py#L130-L249). Please choose the correct file format and replace the `[REPLACE_ME]` token in `asvspoof.sh`\n",
        "\n",
        "After the replacement, you should be able to run the following blocks"
      ],
      "metadata": {
        "id": "bGUbnUhOKRdD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# It takes less than 2 minutes\n",
        "!./asvspoof.sh --stage 3 --stop_stage 3 --train_set train --valid_set dev --test_sets \"dev eval\" --asvspoof_config conf/checkpoint1_dummy.yaml\n",
        "\n",
        "# NOTE: Checkpoint 1\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "ztg17tGRM1RF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ASVSpoof Model\n",
        "\n",
        "In this section, we will define the ASVSpoof model and use the model to conduct the training of ASVSpoof task. For easier understanding, we first use an `encoder` to convert speech features into hidden representations and then use a `decoder` to conduct the classification.\n",
        "\n",
        "### Encoder  (✅ Checkpint 2 (1 point))\n",
        "\n",
        "First, we are going to focus on the encoder part. There has been a long history over the discussion of the speech encoder in our community. Given the sequential perspective, people firstly investigated recurrent neural networks. More recently, we are focusing on [conformer block](https://arxiv.org/abs/2010.13956), which is an extension to the transformer block. In the previous settings, we used a transformer block to collect stats. However, we would want to switch to conformer.\n",
        "\n",
        "Code-reusibility is one of the major benefits of using ESPnet as a toolkit for speech tasks. As ESPnet already support conformer block in ASR, it is easy to import into this new task.\n",
        "\n",
        "In ESPnet, adding modules that we already have can be as simple as two-line codes. Please add lines into `/content/espnet/espnet2/tasks/asvspoof.py`. We have marked `TODO` in the scripts for your convenience.\n",
        "\n"
      ],
      "metadata": {
        "id": "sZv2mf-b-Nor"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# It takes less than 2 minutes\n",
        "!./asvspoof.sh --stage 3 --stop_stage 3 --train_set train --valid_set dev --test_sets \"dev eval\" --asvspoof_config conf/checkpoint2.yaml\n",
        "\n",
        "# NOTE: Checkpoint 2\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "k1-DLj5Snec5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Decoder (✅ Checkpint 3 (1 point))\n",
        "\n",
        "In this stage, we will finally start the training. As the previous tutorial, we can use the Tensorboard to monitor the process."
      ],
      "metadata": {
        "id": "dYQAg_uoQQyi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the TensorBoard notebook extension\n",
        "%reload_ext tensorboard\n",
        "\n",
        "# Launch tensorboard before training\n",
        "%tensorboard --logdir /content/espnet/egs2/espnet_tutorial/asvspoof1/exp"
      ],
      "metadata": {
        "id": "KYWPpq7PSRTM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After we finished the encoder, we also need to create a decoder to conduct the prediciton. As the encoder will generate hidden representations, we want to have a simple decoder to conduct mean-pooling to all the hidden representation at the time-axis. There should be another linear layer to conclude the models into binary classification. Please fill the missing part in `/conent/espnet/espnet2/asvspoof/decoder/linear_decoder.py` to finally start the training. For people who are not familiar with Pytorch, please refer the related resources for details.\n",
        "\n",
        "Related resources that could be helpful for this checkpoint:\n",
        "- https://pytorch.org/docs/stable/generated/torch.mean.html\n",
        "- https://pytorch.org/docs/stable/generated/torch.nn.Linear.html\n",
        "- https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html"
      ],
      "metadata": {
        "id": "4TJgO_4YSXHf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi\n",
        "\n",
        "# Training takes around 2 minutes\n",
        "!./asvspoof.sh --stage 4 --stop_stage 4 --train_set train --valid_set dev --test_sets \"dev eval\" --asvspoof_config conf/checkpoint2.yaml --inference_config conf/decode_asvspoof.yaml\n",
        "\n",
        "# NOTE: Checkpoint 3\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "R8_CkObvRnfU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model Inference \n",
        "### (✅ Checkpint 4 (1 point))\n",
        "As the training is finished, we expect to conduct ASVSpoof on the test set. To approach that, we first have to finish the inference codebase. For our task specifically, we need the log-probability of the prediction to compute [equal error rate (EER)](https://www.igi-global.com/dictionary/equal-error-rate-eer/35389). Therefore the output should be a float number for each utterance.\n",
        "\n",
        "Please fill the missing parts with `TODO`s in `/content/espnet/espnet2/bin/asvspoof_inference.py`"
      ],
      "metadata": {
        "id": "vLOM4e1mSjSR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./asvspoof.sh --stage 5 --stop_stage 5 --train_set train --valid_set dev --test_sets \"eval\" --asvspoof_config conf/checkpoint2.yaml --inference_nj 1 --gpu_inference true\n",
        "\n",
        "# NOTE: Checkpoint 4\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "aMJCYgcAUJx_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Scoring\n",
        "### (✅ Checkpint 5 (1 point))\n",
        "\n",
        "We have prepred the scoring script for you. We can get the EER by the following code-block"
      ],
      "metadata": {
        "id": "w2wcW4MtUjAX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./asvspoof.sh --stage 6 --stop_stage 6 --train_set train --valid_set dev --test_sets \"eval\" --asvspoof_config conf/checkpoint2.yaml \n",
        "!chmod +x scripts/utils/show_asvspoof_result.sh\n",
        "# NOTE: Checkpoint 5\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "6KVy2H4nVeBn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 📗 Exercise 1 (1 point bonus)\n",
        "\n",
        "In the data you just downloaded, we have some extra data for training (`/content/espnet/egs2/espnet_tutorial/asvspoof1/espnet_asvspoof_tutorial/extend_train`). Please try to combine them with the training set and then conduct experiments the augmented set. You are also encouraged to change the model configuration. **If you achieve a better equal error rate (EER) than the previous experiments, you can get a bonus point.**"
      ],
      "metadata": {
        "id": "kSgGR0VJV15c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO\n",
        "\n",
        "# NOTE: Exercise 1\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "rKiu7VZwXm2g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 📗 Exercise 2 (1 point bonus)\n",
        "\n",
        "One main issue of speech anti-spoofing research is the generalization to unseen attacks, i.e., synthesis methods not seen in training the anti-spoofing models. In fact, the test set in our scenario is exact in the same case. Recently, there is a [one-class learning method](https://arxiv.org/pdf/2010.13995.pdf) that compacts the natural speech representations and separate them from the fake speech with a certain margin in the embedding space.\n",
        "\n",
        "We have implemented the AM softmax method located in `/content/espnet/espnet2/asvspoof/loss/am_softmax_loss.py` and also prepared the template `/content/espnet/espnet2/asvspoof/loss/oc_softmax_loss.py` for your implementation. You can follow the TODOs to implement the methods (note that the inference/train_config should change accordingly).\n",
        "\n",
        "**If you successfully implement the OC-softmax and get similar/better EER, you can get a bouns point**\n",
        "\n"
      ],
      "metadata": {
        "id": "YcQmPp_nX7v1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO\n",
        "\n",
        "# NOTE: Exercise 2\n",
        "print_date_and_time()"
      ],
      "metadata": {
        "id": "kk69G9DUq62p"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}