codes/MMSelfSup_tutorials/【3】自监督预训练模型的评估：“分类” 下游任务 之 线性评估.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/mmselfsup_logo.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自监督预训练模型的评估：“分类” 下游任务 之 线性评估\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/open-mmlab/OpenMMLabCourse/blob/main/codes/MMSelfSup_tutorials/【3】自监督预训练模型的评估：“分类”%20下游任务%20之%20线性评估.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MMSelfSup Repo**：[https://github.com/open-mmlab/mmselfsup](https://github.com/open-mmlab/mmselfsup)\n",
    "\n",
    "**MMSelfSup 官方文档链接**：[https://mmselfsup.readthedocs.io/en/latest](https://mmselfsup.readthedocs.io/en/latest)\n",
    "\n",
    "**作者**：OpenMMLab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 任务介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在使用 “分类” 下游任务对自监督预训练模型进行评估的方法中，MMSelfSup 目前实现了以下几种方法：(详细请参考官方[文档](https://mmselfsup.readthedocs.io/zh_CN/latest/tutorials/6_benchmarks.html#id2))\n",
    "+ VOC SVM / Low-shot SVM\n",
    "+ **线性评估**\n",
    "+ ImageNet 半监督分类\n",
    "+ ImageNet 最邻近分类\n",
    "\n",
    "\n",
    "**注意：下游任务的 backbone 必须和自监督预训练模型的 backbone 保持一致，才能进行自监督预训练权值的迁移！**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性评估是评估自监督预训练模型最通用的基准之一。它的 **`基本流程`** 如下：\n",
    "1. 在自监督预训练模型的 backbone 后面接上一层新的线性分类层\n",
    "2. 固定自监督预训练模型的 backbone 权值，使用下游任务（分类）的数据集对新的线性分类层进行训练，以此验证自监督预训练模型提取图片特征的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 环境配置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 查看 Python、PyTorch 和 Torchvision 的版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check nvcc version\n",
    "!nvcc -V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check GCC version\n",
    "!gcc --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check PyTorch installation\n",
    "import torch, torchvision\n",
    "print(torch.__version__)\n",
    "print(torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 安装 MMSelfSup 的依赖库：MMCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install openmim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mim install mmcv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3  安装 MMSelfSup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/open-mmlab/mmselfsup.git\n",
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install MMSelfSup from source\n",
    "!pip install -e . "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 检查安装是否正确"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mmselfsup\n",
    "print(mmselfsup.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 准备数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目前，线性评估所支持的数据集包括 **ImageNet**、**Places205** 和 **iNaturalist18**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.0 数据集介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本教程将在 `Tiny ImageNet` 数据集上对自监督预训练好的模型进行线性评估。\n",
    "\n",
    "Tiny ImageNet 数据集是 ImageNet 的一个子集。\n",
    "\n",
    "该数据集包含 200 个类别，每个类别有 500 张训练图片、50 张验证图片和 50 张测试图片，共 120,000 张图像。每张图片均为 64×64 彩色图片。\n",
    "\n",
    "数据集官方下载地址：http://cs231n.stanford.edu/tiny-imagenet-200.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 下载数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 GNU [Wget](https://www.gnu.org/software/wget/) 工具从斯坦福官方网站下载：http://cs231n.stanford.edu/tiny-imagenet-200.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "%cd data\n",
    "!wget http://cs231n.stanford.edu/tiny-imagenet-200.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 解压数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip -q tiny-imagenet-200.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf tiny-imagenet-200.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 查看数据集目录"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check data directory\n",
    "!apt-get install tree\n",
    "!tree -d /content/mmselfsup/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 准备标注文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了减少大家重写 `加载数据集` 代码的负担，我们整理好了标注文件，复制到数据集根目录 `mmselfsup/data/tiny-imagenet-200` 下即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/wangruohui/OpenMMLabCourse/main/codes/MMSelfSup_tutorials/anno_files/train.txt -P tiny-imagenet-200\n",
    "!wget https://raw.githubusercontent.com/wangruohui/OpenMMLabCourse/main/codes/MMSelfSup_tutorials/anno_files/val.txt -P tiny-imagenet-200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 准备自监督预训练模型的 backbone 权值文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 针对自监督预训练过程中保存的 checkpoint 文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用第一个教程 `模型自监督预训练 之 SimCLR` 中训练保存下来的 `epoch_1.pth` 文件进行演示，该文件可以从 [这里](https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth) 下载，存放在文件夹 `mmselfsup/work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200` 里。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir -p work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200\n",
    "!wget https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth  -P  work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以使用命令 `tools/model_converters/extract_backbone_weights.py` 来提取自监督预训练模型的 backbone 权值，代码如下：\n",
    "\n",
    "```python\n",
    "python tools/model_converters/extract_backbone_weights.py {CHECKPOINT} {MODEL_FILE}\n",
    "```\n",
    "\n",
    "参数:\n",
    "- CHECKPOINT：自监督预训练过程中保存下来（名为 `epoch_*.pth`）的模型文件路径\n",
    "- MODEL_FILE：输出 backbone 权重文件的保存路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ./tools/model_converters/extract_backbone_weights.py \\\n",
    "work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200/epoch_1.pth \\\n",
    "work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200/backbone.pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 针对 MMSelfSup 模型库里的模型文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：MMSelfSup 的 [模型库](https://github.com/open-mmlab/mmselfsup/blob/master/docs/en/model_zoo.md) 中的模型文件都已经提取过 backbone 权值，不需要再次提取！**我们直接使用即可。\n",
    "\n",
    "在模型库中找到在 SimCLR 的预训练模型文件 `simclr_resnet50_8xb32-coslr-200e_in1k`，下载放在 `checkpoints` 文件夹里 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir checkpoints\n",
    "!wget https://download.openmmlab.com/mmselfsup/simclr/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth -P checkpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 自监督预训练模型的线性评估（Linear Evaluation）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 写自监督预训练模型的线性评估配置文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 新建一个名为 `resnet50_linear-1xb512-coslr-5e_tinyin200.py` 的配置文件。（配置文件命名要求 & 含义可参考[这里](https://mmsegmentation.readthedocs.io/zh_CN/latest/tutorials/config.html#id3))\n",
    "\n",
    "\n",
    "\n",
    "2. `resnet50_linear-1xb512-coslr-5e_tinyin200.py` 训练配置文件的内容：\n",
    "    1. 继承 [resnet50_linear-8xb512-coslr-90e_in1k.py](https://github.com/open-mmlab/mmselfsup/blob/master/configs/benchmarks/classification/imagenet/resnet50_linear-8xb512-coslr-90e_in1k.py) 配置文件\n",
    "    2. 根据需求修改参数 samples_per_gpu（单个 GPU 的 Batch size）和 workers_per_gpu （单个 GPU 分配的数据加载线程数）\n",
    "    3. 修改数据集路径和数据标注文件路径\n",
    "    4. 根据 batch size 调整学习率（调整原则请参考：[这里](https://mmselfsup.readthedocs.io/zh_CN/latest/get_started.html#id2)）\n",
    "    5. 修改训练的总轮数 epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile /content/mmselfsup/configs/benchmarks/classification/imagenet/resnet50_linear-1xb512-coslr-5e_tinyin200.py\n",
    "_base_ = 'resnet50_linear-8xb512-coslr-90e_in1k.py'\n",
    "\n",
    "data = dict(\n",
    "    samples_per_gpu=512,  \n",
    "    workers_per_gpu=2,\n",
    "    train=dict(\n",
    "        data_source=dict(\n",
    "            data_prefix='data/tiny-imagenet-200/train',\n",
    "            ann_file='data/tiny-imagenet-200/train.txt')),\n",
    "    val=dict(\n",
    "        data_source=dict(\n",
    "            data_prefix='data/tiny-imagenet-200/val',\n",
    "            ann_file='data/tiny-imagenet-200/val.txt'))\n",
    "    )\n",
    "\n",
    "# optimizer\n",
    "optimizer = dict(\n",
    "    lr=1.6 * ((512 * 1) / (512 * 8)),\n",
    ")\n",
    "\n",
    "# runtime settings\n",
    "runner = dict(type='EpochBasedRunner', max_epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 开始线性评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在运行线性评估时，我们使用 [dist_train_linear.sh](https://github.com/open-mmlab/mmselfsup/blob/master/tools/benchmarks/classification/dist_train_linear.sh) 脚本来启动训练。\n",
    "\n",
    "```shell\n",
    "GPUS=${GPUS} bash tools/benchmarks/classification/dist_train_linear.sh ${CONFIG} ${PRETRAIN}\n",
    "```\n",
    "\n",
    "参数：\n",
    "- GPUS：默认使用的 GPU 数量是 8，如果需要调整，需要加上该环境变量。\n",
    "- CONFIG：线性评估所使用的配置文件，位于 `configs/benchmarks/classification/` 对应数据集目录下\n",
    "- PRETRAIN：自监督预训练模型的 backbone 权重文件所在的路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!GPUS=1 bash tools/benchmarks/classification/dist_train_linear.sh \\\n",
    "configs/benchmarks/classification/imagenet/resnet50_linear-1xb512-coslr-5e_tinyin200.py \\\n",
    "checkpoints/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}