modules/transfer_mmar.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BazmryhqKyZk"
   },
   "source": [
    "# Transfer learning from Clara Train models\n",
    " \n",
    "In this tutorial, we demonstrate transfer learning from a pretrained model in [Clara Train's Medical Model Archive format](https://docs.nvidia.com/clara/clara-train-sdk/pt/mmar.html).\n",
    "\n",
    "The `monai.apps.load_from_mmar` command implements the following: \n",
    "- fetch a remote model if it is not cached (from Nvidia GPU cloud https://ngc.nvidia.com/)\n",
    "- cache a local model file copy\n",
    "- instantiate a network model with pre-trained weights.\n",
    "\n",
    "The APIs enable direct access to the MMAR models in a pythonic way.\n",
    "\n",
    "\n",
    "In this demo, we show that a PyTorch native pipeline could be configured to use the pretrained models,\n",
    "and compare the following three cases:\n",
    "1. regular training with a randomly initialised model (best mean Dice: 0.84)\n",
    "<p><img src=\"../figures/mmar_1.png\" width=\"50%\" alt='mmar_1'></p>\n",
    "2. loading from a pretrained model, without any model updating (best mean Dice: 0.86)\n",
    "<p><img src=\"../figures/mmar_2.png\" width=\"50%\" alt='mmar_2'></p>\n",
    "3. loading from a pretrained model, with finetuning (best mean Dice: 0.92)\n",
    "<p><img src=\"../figures/mmar_0.png\" width=\"50%\" alt='mmar_0'></p>\n",
    "\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Project-MONAI/tutorials/blob/main/modules/transfer_mmar.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H3dniV1cKyZp"
   },
   "source": [
    "## Setup environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2709,
     "status": "ok",
     "timestamp": 1625519079692,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "fUOXcmazKyZq",
    "outputId": "1ecb32b7-afe7-406e-cd97-eb7be42ec437"
   },
   "outputs": [],
   "source": [
    "!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel, lmdb, tqdm]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "NZdJy4-FKyZr",
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kGQY14M9KyZr"
   },
   "source": [
    "## Setup imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1222,
     "status": "ok",
     "timestamp": 1625519080911,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "E1VgZdItKyZr",
    "outputId": "19f244ef-b133-4de6-d9a0-bd6c27c03c90",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Copyright 2020 - 2021 MONAI Consortium\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "import os\n",
    "import tempfile\n",
    "import glob\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import numpy as np\n",
    "from monai.apps import download_and_extract\n",
    "from monai.networks.nets import UNet\n",
    "from monai.networks.layers import Norm\n",
    "from monai.losses import DiceFocalLoss\n",
    "from monai.metrics import DiceMetric\n",
    "from monai.inferers import sliding_window_inference\n",
    "from monai.data import (\n",
    "    LMDBDataset,\n",
    "    DataLoader,\n",
    "    decollate_batch,\n",
    ")\n",
    "from monai.apps import load_from_mmar\n",
    "from monai.transforms import (\n",
    "    AsDiscrete,\n",
    "    EnsureChannelFirstd,\n",
    "    Compose,\n",
    "    LoadImaged,\n",
    "    ScaleIntensityRanged,\n",
    "    Spacingd,\n",
    "    Orientationd,\n",
    "    CropForegroundd,\n",
    "    RandCropByPosNegLabeld,\n",
    "    RandAffined,\n",
    "    RandRotated,\n",
    "    EnsureType,\n",
    "    EnsureTyped,\n",
    ")\n",
    "from monai.utils import set_determinism\n",
    "from monai.apps.mmars import RemoteMMARKeys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the flag, it will change the initial model object.\n",
    "PRETRAINED = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gybHjcBLKyZs"
   },
   "source": [
    "## Setup data directory\n",
    "\n",
    "You can specify a directory with the `MONAI_DATA_DIRECTORY` environment variable.  \n",
    "This allows you to save results and reuse downloads.  \n",
    "If not specified a temporary directory will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1625519080912,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "IxiXEVmEKyZs",
    "outputId": "b22f9026-7df7-471d-99ff-f13b7523a32f",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./mmar_cache\n"
     ]
    }
   ],
   "source": [
    "directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n",
    "root_dir = tempfile.mkdtemp() if directory is None else directory\n",
    "print(root_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-bFK3xLcKyZt"
   },
   "source": [
    "## Download dataset\n",
    "\n",
    "Downloads and extracts the dataset.\n",
    "The dataset comes from http://medicaldecathlon.com/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3026,
     "status": "ok",
     "timestamp": 1625519083935,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "musXbLIAKyZt",
    "outputId": "dabb2d74-285f-42f1-abc5-7b5b622ac8c1",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Verified 'Task09_Spleen.tar', md5: 410d4a301da4e5b2f6f86ec3ddba524e.\n",
      "File exists: ./mmar_cache/Task09_Spleen.tar, skipped downloading.\n",
      "Non-empty folder exists in ./mmar_cache/Task09_Spleen, skipped extracting.\n"
     ]
    }
   ],
   "source": [
    "resource = \"https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar\"\n",
    "md5 = \"410d4a301da4e5b2f6f86ec3ddba524e\"\n",
    "\n",
    "compressed_file = os.path.join(root_dir, \"Task09_Spleen.tar\")\n",
    "download_and_extract(resource, compressed_file, root_dir, md5)\n",
    "data_dir = os.path.join(root_dir, \"Task09_Spleen\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ypEvK0aAKyZu"
   },
   "source": [
    "## Set MSD Spleen dataset path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "fQeUN3eyKyZu",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_images = sorted(\n",
    "    glob.glob(os.path.join(data_dir, \"imagesTr\", \"*.nii.gz\")))\n",
    "train_labels = sorted(\n",
    "    glob.glob(os.path.join(data_dir, \"labelsTr\", \"*.nii.gz\")))\n",
    "data_dicts = [\n",
    "    {\"image\": image_name, \"label\": label_name}\n",
    "    for image_name, label_name in zip(train_images, train_labels)\n",
    "]\n",
    "train_files, val_files = data_dicts[:-9], data_dicts[-9:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UkQNk9yxKyZv"
   },
   "source": [
    "# Setup transforms for training and validation\n",
    "Relatively heavy data augmentations are used here to simulate a difficult training/validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "U1-_K-26KyZv",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_transforms = Compose(\n",
    "    [\n",
    "        LoadImaged(keys=[\"image\", \"label\"]),\n",
    "        EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n",
    "        Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n",
    "        Spacingd(keys=[\"image\", \"label\"], pixdim=(\n",
    "            1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n",
    "        ScaleIntensityRanged(\n",
    "            keys=[\"image\"], a_min=-57, a_max=164,\n",
    "            b_min=0.0, b_max=1.0, clip=True,\n",
    "        ),\n",
    "        CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n",
    "        RandCropByPosNegLabeld(\n",
    "            keys=[\"image\", \"label\"],\n",
    "            label_key=\"label\",\n",
    "            spatial_size=(96, 96, 96),\n",
    "            pos=1,\n",
    "            neg=1,\n",
    "            num_samples=4,\n",
    "            image_key=\"image\",\n",
    "            image_threshold=0,\n",
    "        ),\n",
    "        RandAffined(\n",
    "            keys=['image', 'label'],\n",
    "            mode=('bilinear', 'nearest'),\n",
    "            prob=0.5,\n",
    "            spatial_size=(96, 96, 96),\n",
    "            rotate_range=(np.pi/18, np.pi/18, np.pi/5),\n",
    "            scale_range=(0.05, 0.05, 0.05)\n",
    "        ),\n",
    "        EnsureTyped(keys=[\"image\", \"label\"]),\n",
    "    ]\n",
    ")\n",
    "val_transforms = Compose(\n",
    "    [\n",
    "        LoadImaged(keys=[\"image\", \"label\"]),\n",
    "        EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n",
    "        Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n",
    "        Spacingd(keys=[\"image\", \"label\"], pixdim=(\n",
    "            1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n",
    "        ScaleIntensityRanged(\n",
    "            keys=[\"image\"], a_min=-57, a_max=164,\n",
    "            b_min=0.0, b_max=1.0, clip=True,\n",
    "        ),\n",
    "        RandRotated(\n",
    "            keys=['image', 'label'],\n",
    "            mode=('bilinear', 'nearest'),\n",
    "            range_x=np.pi/18,\n",
    "            range_y=np.pi/18,\n",
    "            range_z=np.pi/5,\n",
    "            prob=1.0,\n",
    "            padding_mode=('reflection', 'reflection'),\n",
    "        ),\n",
    "        CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n",
    "        EnsureTyped(keys=[\"image\", \"label\"]),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XMudxL8c6oG9"
   },
   "source": [
    "# Define LMDBDataset and DataLoader for training and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "8cgC2sZ6KyZw"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 32/32 [00:00<00:00, 46138.79it/s]\n",
      "100%|██████████| 9/9 [00:00<00:00, 29959.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accessing lmdb file: /workspace/mmar_cache/monai_cache.lmdb.\n",
      "Accessing lmdb file: /workspace/mmar_cache/monai_cache.lmdb.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_ds = LMDBDataset(data=train_files, transform=train_transforms, cache_dir=root_dir)\n",
    "# initialize cache and print meta information\n",
    "print(train_ds.info())\n",
    "\n",
    "# use batch_size=2 to load images and use RandCropByPosNegLabeld\n",
    "# to generate 2 x 4 images for network training\n",
    "train_loader = DataLoader(train_ds, batch_size=2, shuffle=True, num_workers=2)\n",
    "\n",
    "# the validation data loader will be created on the fly to ensure \n",
    "# a deterministic validation set for demo purpose.\n",
    "val_ds = LMDBDataset(data=val_files, transform=val_transforms, cache_dir=root_dir)\n",
    "# initialize cache and print meta information\n",
    "print(val_ds.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'map_addr': 0, 'map_size': 1099511627776, 'last_pgno': 885751, 'last_txnid': 41, 'max_readers': 126, 'num_readers': 0, 'size': 32, 'filename': '/workspace/mmar_cache/monai_cache.lmdb'}\n"
     ]
    }
   ],
   "source": [
    "print(train_ds.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_IbQ5UcDeSRm"
   },
   "source": [
    "# Download the pretrained model from Nvidia Clara Train\n",
    "\n",
    "By setting the global variable `PRETRAINED` to `True`, the following sections\n",
    "will use a pretrained model from the `clara_pt_spleen_ct_segmentation` MMAR.\n",
    "\n",
    "\n",
    "For more details, please see https://docs.monai.io/en/latest/apps.html#monai.apps.load_from_mmar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "9Sf7H2GWmSeS"
   },
   "outputs": [],
   "source": [
    "mmar = {\n",
    "    RemoteMMARKeys.ID: \"clara_pt_spleen_ct_segmentation_1\",\n",
    "    RemoteMMARKeys.NAME: \"clara_pt_spleen_ct_segmentation\",\n",
    "    RemoteMMARKeys.FILE_TYPE: \"zip\",\n",
    "    RemoteMMARKeys.HASH_TYPE: \"md5\",\n",
    "    RemoteMMARKeys.HASH_VAL: None,\n",
    "    RemoteMMARKeys.MODEL_FILE: os.path.join(\"models\", \"model.pt\"),\n",
    "    RemoteMMARKeys.CONFIG_FILE: os.path.join(\"config\", \"config_train.json\"),\n",
    "    RemoteMMARKeys.VERSION: 1,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3622,
     "status": "ok",
     "timestamp": 1625519087552,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "_CuUelsjOaeP",
    "outputId": "502b1e14-d5ae-41b6-f405-1d80a1f409d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using a pretrained model.\n",
      "Expected md5 is None, skip md5 check for file ./mmar_cache/clara_pt_spleen_ct_segmentation_1.zip.\n",
      "File exists: ./mmar_cache/clara_pt_spleen_ct_segmentation_1.zip, skipped downloading.\n",
      "Non-empty folder exists in ./mmar_cache/clara_pt_spleen_ct_segmentation_1, skipped extracting.\n",
      "\n",
      "*** \"clara_pt_spleen_ct_segmentation_1\" available at ./mmar_cache/clara_pt_spleen_ct_segmentation_1.\n",
      "*** Model: <class 'monai.networks.nets.unet.UNet'>\n",
      "*** Model params: {'dimensions': 3, 'in_channels': 1, 'out_channels': 2, 'channels': [16, 32, 64, 128, 256], 'strides': [2, 2, 2, 2], 'num_res_units': 2, 'norm': 'batch'}\n",
      "\n",
      "---\n",
      "For more information, please visit https://ngc.nvidia.com/catalog/models/nvidia:med:clara_pt_spleen_ct_segmentation\n",
      "\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "if PRETRAINED:\n",
    "    print(\"using a pretrained model.\")\n",
    "    unet_model = load_from_mmar(\n",
    "        item=mmar[RemoteMMARKeys.NAME],\n",
    "        mmar_dir=root_dir,\n",
    "        map_location=device,\n",
    "        pretrained=True)\n",
    "    model = unet_model\n",
    "else:\n",
    "    print(\"using a randomly init. model.\")\n",
    "    model = UNet(\n",
    "        spatial_dims=3,\n",
    "        in_channels=1,\n",
    "        out_channels=2,\n",
    "        channels=(16, 32, 64, 128, 256),\n",
    "        strides=(2, 2, 2, 2),\n",
    "        num_res_units=2,\n",
    "        norm=Norm.BATCH,\n",
    "    )\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RdzdCygTPdc9"
   },
   "source": [
    "### Alternative transfer learning setup\n",
    "Using the MONAI modules, it is also possible to set\n",
    "the learning rates for different layers for transfer learning.\n",
    "\n",
    "```python\n",
    "from monai.networks.utils import copy_model_state\n",
    "from monai.optimizers import generate_param_groups\n",
    "\n",
    "# copy all the pretrained weights except for variables whose name matches \"model.0.conv.unit0\"\n",
    "pretrained_dict, updated_keys, unchanged_keys = copy_model_state(\n",
    "    model, unet_model, exclude_vars=\"model.0.conv.unit0\")\n",
    "print(\"num. var. using the pretrained\", len(updated_keys), \", random init\", len(unchanged_keys), \"variables.\")\n",
    "model.load_state_dict(pretrained_dict)\n",
    "\n",
    "print([x[0] for x in model.named_parameters()])\n",
    "print(unchanged_keys)\n",
    "\n",
    "# stop gradients for the pretrained weights\n",
    "for x in model.named_parameters():\n",
    "    if x[0] in updated_keys:\n",
    "        x[1].requires_grad = False\n",
    "params = generate_param_groups(\n",
    "    network=model,\n",
    "    layer_matches=[lambda x: x[0] in updated_keys],\n",
    "    match_types=[\"filter\"],\n",
    "    lr_values=[1e-4],\n",
    "    include_others=False\n",
    ")\n",
    "optimizer = torch.optim.Adam(params, 1e-5)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GcfAgnCbPLvN"
   },
   "source": [
    "## Create an optimizer and a loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "x6W5lOcMJbo8"
   },
   "outputs": [],
   "source": [
    "loss_function = DiceFocalLoss(to_onehot_y=True, softmax=True)\n",
    "optimizer = torch.optim.Adam(model.parameters(), 5e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6FQ5DHsxKyZw"
   },
   "source": [
    "## Model training\n",
    "\n",
    "Execute a typical PyTorch training that run epoch loop and step loop, and do validation after other epoch.  \n",
    "Will save the model weights to file if got best validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gmrPkMd_KyZw",
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_epochs = 50\n",
    "val_interval = 2\n",
    "best_metric = -1\n",
    "best_metric_epoch = -1\n",
    "epoch_loss_values = []\n",
    "metric_values = []\n",
    "post_pred = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=2)])\n",
    "post_label = Compose([EnsureType(), AsDiscrete(to_onehot=2)])\n",
    "dice_metric = DiceMetric(include_background=False, reduction=\"mean\", get_not_nans=False)\n",
    "\n",
    "for epoch in range(max_epochs):\n",
    "    print(\"-\" * 10)\n",
    "    print(f\"epoch {epoch + 1}/{max_epochs}\")\n",
    "    model.train()\n",
    "    epoch_loss = 0\n",
    "    step = 0\n",
    "    set_determinism(seed=42)\n",
    "    for batch_data in train_loader:\n",
    "        step += 1\n",
    "        inputs, labels = (\n",
    "            batch_data[\"image\"].to(device),\n",
    "            batch_data[\"label\"].to(device),\n",
    "        )\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(inputs)\n",
    "        loss = loss_function(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        epoch_loss += loss.item()\n",
    "        print(\n",
    "            f\"{step}/{len(train_ds) // train_loader.batch_size}, \"\n",
    "            f\"train_loss: {loss.item():.4f}\")\n",
    "    epoch_loss /= step\n",
    "    epoch_loss_values.append(epoch_loss)\n",
    "    print(f\"epoch {epoch + 1} average loss: {epoch_loss:.4f}\")\n",
    "\n",
    "    if (epoch + 1) % val_interval == 0:\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            set_determinism(seed=42)\n",
    "            for val_data in DataLoader(val_ds, batch_size=1, num_workers=2):\n",
    "                val_inputs, val_labels = (\n",
    "                    val_data[\"image\"].to(device),\n",
    "                    val_data[\"label\"].to(device),\n",
    "                )\n",
    "                roi_size = (160, 160, 160)\n",
    "                sw_batch_size = 4\n",
    "                val_outputs = sliding_window_inference(\n",
    "                    val_inputs, roi_size, sw_batch_size, model, overlap=0.5)\n",
    "                val_outputs = [post_pred(i) for i in decollate_batch(val_outputs)]\n",
    "                val_labels = [post_label(i) for i in decollate_batch(val_labels)]\n",
    "                dice_metric(y_pred=val_outputs, y=val_labels)\n",
    "            metric = dice_metric.aggregate().item()\n",
    "            dice_metric.reset()\n",
    "            metric_values.append(metric)\n",
    "            if metric > best_metric:\n",
    "                best_metric = metric\n",
    "                best_metric_epoch = epoch + 1\n",
    "                torch.save(model.state_dict(), os.path.join(\n",
    "                    root_dir, \"best_metric_model.pth\"))\n",
    "                print(\"saved new best metric model\")\n",
    "            print(\n",
    "                f\"current epoch: {epoch + 1} current mean dice: {metric:.4f}\"\n",
    "                f\"\\nbest mean dice: {best_metric:.4f} \"\n",
    "                f\"at epoch: {best_metric_epoch}\"\n",
    "            )\n",
    "print(\n",
    "    f\"train completed, best_metric: {best_metric:.4f} \"\n",
    "    f\"at epoch: {best_metric_epoch}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b-P6bmD9KyZx"
   },
   "source": [
    "## Plot the loss and metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 232
    },
    "executionInfo": {
     "elapsed": 16,
     "status": "error",
     "timestamp": 1625521679645,
     "user": {
      "displayName": "Wenqi Li GB",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ghje4if0W0HBe6qgJYvWDgJIf2jatF9cjTAGMR0=s64",
      "userId": "00176608654401740282"
     },
     "user_tz": -60
    },
    "id": "u7ge6508KyZx",
    "outputId": "2ba5aa42-fec4-4177-c93a-705d51fdd04e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAGDCAYAAAAh/naNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyCElEQVR4nO3deZhcd33n+/e3qnrT2rIlG2vxgi1iDCE2CAMxJA5LYhjGZm5Y7AQCE8CTTExgyJ3EzuQSQu7MXJJ5SJgbJ+AQLpAAxiEhoxDnGsIaSAAJMIttbAsvaMHWYm0tqdf6zh/ndKvctKSW1FvVeb8e1VNnqzrfU1361ad+dZbITCRJkqSqqc13AZIkSdJ8MAhLkiSpkgzCkiRJqiSDsCRJkirJICxJkqRKMghLkiSpkgzCmlURkRFx0XzXIUk6eZ3YhkfEXRFx5XzXoYXBIFwhEfFQRByJiIGW25/Md12TRcTrysb3VfNdy+mKiPPLbWnMdy2Sqici/v+IeMcU06+JiEdOp22KiM+X7dtPTJr+iXL6laf63KdYz3h7O/759mhEfDIiXtS6XGY+JTM/P5e1aeEyCFfPv83MJS23G+a7oCm8FngM+KXZeHJDqaQK+SDw6oiISdNfA3w4M0dP8/nvo6WtjogzgecAu07zeU9Hf2YuAX4C+DTwiYh43TzWowXMICxgohf2yxHxJxGxPyK+FxEvaJm/OiI2RsRjEbElIt7YMq8eEb8dEd+PiIMR8fWIWNfy9C+MiPsjYl9E3DxFg9xax3nATwPXAz8XEU8op/9ZRPyPScv+r4h4a0t9fxMRuyLiwYj49Zbl3h4RH4+Iv4qIA8DrIuLyiPjXsqYfltvd3fKYn42Ie8vX4k8j4gsR8YaW+b8cEfdExN6IuKOs+2Rf8+O9ppdHxOaIOFD2aryrnN5bbseesvZNEXH2ya5bUmX8HXAm8LzxCRGxAngp8KETtYXT8GHgVRFRL8evAz4BDLesrxYRN5afEXsi4raIOKNl/l+XvdP7I+KLEfGUlnkfKD83/qH8fPlqRFw4ncIy85HMfDfwduCdEVErn/OhiHhhOXzMz6+IuDgiPl220fdGxCtP4nVRmzAIq9WzgO8DK4HfBf62pbG6FdgGrAZeDvy3iHh+Oe+tFI3fS4BlwC8Dh1ue96XAM4GnAa8Efu44NfwSsDkz/wa4B/jFcvpHKRrbgImG/GeBW8vG7e+BbwFrgBcAb4mI1vVcA3wc6KdouMeA/1Ru63PKx/zH8rlXlsveRPEBci/wk+NPFBHXAL8N/B/AKuCfy/pO1vFe03cD787MZcCFwG3l9NcCy4F1ZW2/Ahw5hXVLqoDMPELRfrT+wvZK4HuZ+S2O0xZO0w7gbor2mHI9H5q0zJuAl1F0cqwG9gI3t8z/R2A9cBbwDYo2utW1wO8BK4AtwH89ifoA/rZ87h+bYt6Un18RsZiiN/kj5WOvBf40Ii45yXVroctMbxW5AQ8BA8C+ltsby3mvo2jQomX5r1H8fLaOorFc2jLvvwMfKIfvBa45xjoTeG7L+G3Ajcep8X7gLeXwTcC3yuEAfgD8VDn+RuCz5fCzgB9Mep6bgP+vHH478MUTvDZvAT5RDv8S8K8t8wLYCryhHP9H4PUt82sUwf+8KZ73/PI1aEyafqLX9IsUDf/KSY/7ZeBfgKfN9/vJmzdv7XEDnlu2973l+JeB/3SMZSfawnI8gYuOsezngTcAr6boDLgYuK+ctw24shy+B3hBy+POAUYmt4vlvP5yncvL8Q8A72uZ/xKKED9VPcdqb3vL6VeU4w8BLyyHp/z8Al4F/POkae8Ffne+/57eZvZmj3D1vCwz+1tuf94yb3uW/9tLD1N8e18NPJaZByfNW1MOr6PoST6WR1qGDwNLplooIq4ALqDoKYXim/iPR8SlZV23UnxzB/gFjvYanAesLn/a2xcR+yh6bFt3Gdg6aV1PiuIgikfK3SX+G0WPCOX2Tixfrntby8PPA97dsq7HKMLyGqbvRK/p64EnAd8rd394aTn9L4E7KHrCd0TEH0RE10msV1LFZOaXgN3Ay8rdCi6naF9P1BZO198CzwduoGijJjuPYj/d8TbzHoqOgLPLXRP+n3LXhAMUIZVJNUzrM+Q4xtvVx6aYd6zPr/OAZ036XPlF4AknuW4tcAZhtVozvutB6VyKXuIdwBkRsXTSvO3l8FaKn+9P12spAuWdEfEI8NWW6VD0OLy83B/3WcDftKz/wUkBf2lmvqTluVsDPsCfAd8D1mex+8Fvl+sG+CGwdnzB8jVZ2/LYrcB/mLS+vsz8l5PY1uO+ppl5f2ZeR/GT3DuBj0fE4swcyczfy8xLKHbXeCmzdFChpI7yIYq24tXAHZn5aDn9eG3htGTmYYpfyn6VqYPwVuDFk9rM3szcTtGpcQ3wQordvs4vH3NSNZzAvwN2UvT+TlXbVJ9fW4EvTKp5SWb+6gzWpQXAIKxWZwG/HhFdEfEK4MnA7Zm5leLn+P9eHqz1NIoey78qH/c+4PcjYn0UnhbFkcPTFhG9FPutXQ9c2nJ7E/ALEdHIzG9S9Gq8j6Ih31c+/GvAwYj4rYjoK3sYnhoRzzzOKpcCB4CBiLiYogEf9w8UPdEvi+IME7/G43sB3gPcNH5AR0QsL1+v4+kpX7veclu3c5zXNCJeHRGrMrNJ8ZMmQDMifiYifrw8MOUAxc+LzROsW5I+RBE230hxJolxx2sLT8ZvAz+dmQ9NMe89wH8tOzGIiFXlsRbj6x8C9gCLKHqkZ0REnB0RN1Ac83JT2Z5OdqzPr08CT4qI15SfiV0R8cyIePJM1aeFwSBcPX8fjz+P8Cda5n2V4oCF3RQHI7w8M/eU866j+Ka+g+KI4N/NzH8q572LYt/fT1E0qH8B9J1kXS+jOOjrQ1kc6ftIZj4CvB9oAFeVy32EojH/yPgDM3OMomf0UuBBjobl5cdZ3/9J0RNxEPhz4GMtz7cbeAXwBxSN8yXAZorGmsz8BEUv7a3lT3nfBV58gu0bKLdv/PZ8jv+aXgXcFREDFAfOXZvFQS9PoDiQ7wDFz4tfYOoeGEmaUAbUfwEWAxtbZh2zLTzJ599R7oIxlXeX6/xURBwEvkLxqx4UAf1his6Bu8t5p2tfRBwCvkOxT/ErMvP9x1h2ys+vcre1n6U4SG4Hxe4Z7wR6ZqA+LSDx+F1CVVVRnGPxDZn53PmuZaEpz0qxDfjFzPzcfNcjSZJmhj3C0hQi4ucioj8ieji6z9xM9FRIkqQFYlpBOCKuKk8mvSUibpxi/h9FxJ3l7b7y6EqpnT2H4kji3cC/pTjbhufrVVuIiPdHxM6I+O4x5kdE/M+yTf92RDx9rmuUpIXghLtGlAfl3Ae8iOLn4U3AdZl59zGWfxNwWWb+8gzXKkmahoj4KYr90j+UmU+dYv5LKA5EfQnFvprvzsxnTV5OkjrddHqELwe2ZOYDmTlMcS7Xa46z/HWc2lW2JEkzIDO/yNTnTB13DUVIzsz8CtAfEefMTXWStHBMJwiv4fEXI9jGMS4cUJ4a5QLgs6dfmiRplky7XZekTtaY4ee7Fvh4eTqrHxER11OcJ5bFixc/4+KLL57h1UvS7Pv617++OzNXzXcds802W1KnOFa7PZ0gvJ3iEoTj1nL0imKTXUtx8YEpZeYtwC0AGzZsyM2bN09j9ZK0sETEw/Ndw2maVrtumy2pUxyr3Z7OrhGbgPURcUFEdFOE3Y2TFyqvSLMC+NfTKVSSNOs2Ar9Unj3i2cD+zPzhfBclSXPthD3CmTlaXqLwDqAOvD8z74qIdwCbM3M8FF8L3JpeoUOS5lVEfBS4ElgZEdsoLjHbBZCZ7wFupzhjxBbgMPDv56dSSZpf09pHODNvp2g4W6e9bdL422euLEnSqcrM604wPznObmySVBVeWU6SJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJTXmuwBJUrVlJvfvHOCrDz7G/sPDDI82GRprMjTSZHisyfBoy22sydDoGMOjTQCW9DRY0tvFkp4GS3sbxXhPgyW9DZaW98W8Lpb1NljW10VvV33Wt6nZTJqZjGXSbNIynDSzGM+EpLwfn1Y+Fo4uMzg6xqGhUQaGxjg8NMrA0CiHhkY5NFxMbx0+MjLGWDMnnq91uLgVr/fj6moW42PN4tZsJqPj9Tdbbpktf7OWv9/j/phHBxv1oK+rTm9Xnb7uOn1dxa23u05fV60Y7y7mj/9Nxre59XUan95af29XnZVLejhzcTdnLunhzCXdrFxc3C/qrhMRx/zbHBwc4Yf7B4vbviPl8JGJaV31Gpeu6+fp5/Zz2bkreOLKxdRqx34+tTeDsCRpzj16YJAvb9nNl+7fzZe27GbnwaHHze9p1Ohu1Ir7+vhwne5yene9+EFzz6FhHt5zmINDowwMFkHwRLobNZb3FcF4eV8Xy/q6yvHifnlfFz1dNQ4PjwfPMQ4PPz54Hh4fHh7l8NAYw2NNMpkIlHOlXgsWd9dZ0tNgcU+Dvu469VpQi6AWEBHUatBVq1GLKMaDifm1COq1llvLeG3SeL0WPC4ORuvg0ZHxDDo61uTIyBhHhpsMjoyVw2McODLCzgNjHB4upg0OjzE4OkYQxKTamDQeUSwzODzGwaHRKV+T3q4aZy7uYeWSIiQv622w59Awj5RBd2DS4yJg5ZIeVi/v5cJVizk8PMYnv72Dj37tBwAs7+sqg/EKLju3n59Y18/yvq6Z+PNNODA4wo59R9ix7wjb9w2yY98RHtk/SDNz4u/SqBd/g0atVt5Hy33x/6J/URf95Xt4+aIu+hd109/XdcIvB1VmEJYkzbqBoVG+9uAe/vn+3Xx5y27ue3QAgDMWd3PFRSt57kVn8pMXruTsZb101eOUP7RHx5ocGhrjwOAIA2Xv6cDgKAcGRzg4OMr+IyMcGBzhwJGRYvjIKHsGhnlw96FyfITWHBsBi7sbLCrD5qKeOou6G6xa2sN5Zy4q5vXU6a7XqNWKwFaPKIdjIpTWa+OBrgx1ZagcD3a1KMZbw1+U6+/tqrO4p8Hi7uJ+SU9Rz+KeBj2NWmUDzuDIGI8dGmbPwDC7Dw2xZ2CYPQND7Dk0zO6BYnznwUHue3SEMxd388RVi7niopWcs7yXc/r7OGd5L09Y1svZy3rpbjx+T9FmM3lg9wDfeHgf39y6l288vI8//sx9ZBZ/k4tWLZkIxUt6GsXfkWLeeKAfHx//xhABR4bH2F4G3uJWhN7Job5Ri4n/C6PNx/fMj07cNxlrJiNjJ/7i1agF/YuOftHrX9TNWUt7eMLy3uJ1WF6+Hst7WVpuz3RkJoeGx9gzMMTultd/sPxlYrSZjI41W2r+0fGxsfFtap5wW8eayc8+5Qn8yk9fOK36psMgLEmaFd/dvp/P3LOTL2/ZzTd+sJfRZtLTqHH5BWfw809fy3PXr+TJT1g2oz87N+o1li+qsXzRqfXYZSYDQ6MMjjRZ3FP8lF/VoLnQ9XbVWd3fx+r+vhl/7lotuOispVx01lJe+cx1QLFLxbe37ecbD+/lm1v38em7H+W2zdtO6flXLOpidX8f5565iOdceCar+3sntmVNfx8rl/RQP4n/F81mMjg6xv4jI+w7XNz2Hxlh/5HhieF9R0bYXw4/emCQ72zfz+6Bocft5gKwuLteBuS+iaC8pKfBY4eHH/dlY89A8YVjqNxN6URqweN6s4se7hr1WjF9vMd7/FeI8flHe75r9HYFPY2ZPbzNICxJmhUf/urD3LppK09dvZw3/tQTed5FK3n6eSvmZB/dUxURLO3tYmnvfFeihWZpbxdXXLSSKy5aCRRfmrbvO8LQaLMMk+P7ffMj+3+PDxfhvZdF3TMbv2q1YFF3g0XdDc5ZPv0vBsOjTXYeHJzYbeSR/YPs2H9kYvxL9+9m58FBmgnd9RpnLukubot7uOisJVPup33Gkm4WddVp1B+/G8dC3c/aICxJmhW//oL1/ObPXcyKxd3zXYo04yKCtSsWzXcZp6W7UWPtikXH3Y7xfb2XnMQuE+3EICxJmhUn0zMlaWFq1GssrXfu2XY7d8skSZKk4zAIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSppWEI6IqyLi3ojYEhE3HmOZV0bE3RFxV0R8ZGbLlCRJkmZW40QLREQduBl4EbAN2BQRGzPz7pZl1gM3AVdk5t6IOGu2CpYkSZJmwnR6hC8HtmTmA5k5DNwKXDNpmTcCN2fmXoDM3DmzZUqSJEkzazpBeA2wtWV8Wzmt1ZOAJ0XElyPiKxFx1VRPFBHXR8TmiNi8a9euU6tYkiRJmgEzdbBcA1gPXAlcB/x5RPRPXigzb8nMDZm5YdWqVTO0akmSJOnkTScIbwfWtYyvLae12gZszMyRzHwQuI8iGEuSJEkL0nSC8CZgfURcEBHdwLXAxknL/B1FbzARsZJiV4kHZq5MSZIkaWadMAhn5ihwA3AHcA9wW2beFRHviIiry8XuAPZExN3A54D/nJl7ZqtoSZIk6XSd8PRpAJl5O3D7pGlvaxlO4K3lTZIkSVrwvLKcJEmSKskgLEkd5kRXA42IcyPicxHxzYj4dkS8ZD7qlKT5ZhCWpA7ScjXQFwOXANdFxCWTFvsdiuM9LqM4APpP57ZKSVoYDMKS1FmmczXQBJaVw8uBHXNYnyQtGNM6WE6S1DamuhrosyYt83bgUxHxJmAx8MK5KU2SFhZ7hCWpeq4DPpCZa4GXAH8ZET/yeRAR10fE5ojYvGvXrjkvUpJmm0FYkjrLdK4G+nrgNoDM/FegF1g5+Yky85bM3JCZG1atWjVL5UrS/DEIS1Jnmc7VQH8AvAAgIp5MEYTt8pVUOQZhSeog07wa6G8Ab4yIbwEfBV5XXhhJkirFg+UkqcNM42qgdwNXzHVdkrTQ2CMsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaqkaQXhiLgqIu6NiC0RceMU818XEbsi4s7y9oaZL1WSJEmaOY0TLRARdeBm4EXANmBTRGzMzLsnLfqxzLxhFmqUJEmSZtx0eoQvB7Zk5gOZOQzcClwzu2VJkiRJs2s6QXgNsLVlfFs5bbKfj4hvR8THI2LdVE8UEddHxOaI2Lxr165TKFeSJEmaGTN1sNzfA+dn5tOATwMfnGqhzLwlMzdk5oZVq1bN0KolSZKkkzedILwdaO3hXVtOm5CZezJzqBx9H/CMmSlPkiRJmh3TCcKbgPURcUFEdAPXAhtbF4iIc1pGrwbumbkSJUmSpJl3wrNGZOZoRNwA3AHUgfdn5l0R8Q5gc2ZuBH49Iq4GRoHHgNfNYs2SJEnSaTthEAbIzNuB2ydNe1vL8E3ATTNbmiRJkjR7vLKcJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEmSpEoyCEuSJKmSDMKSJEmqJIOwJEmSKskgLEkdJiKuioh7I2JLRNx4jGVeGRF3R8RdEfGRua5RkhaCxnwXIEmaORFRB24GXgRsAzZFxMbMvLtlmfXATcAVmbk3Is6an2olaX7ZIyxJneVyYEtmPpCZw8CtwDWTlnkjcHNm7gXIzJ1zXKMkLQgGYUnqLGuArS3j28pprZ4EPCkivhwRX4mIq6Z6ooi4PiI2R8TmXbt2zVK5kjR/DMKSVD0NYD1wJXAd8OcR0T95ocy8JTM3ZOaGVatWzW2FkjQHDMKS1Fm2A+taxteW01ptAzZm5khmPgjcRxGMJalSDMKS1Fk2Aesj4oKI6AauBTZOWubvKHqDiYiVFLtKPDCHNUrSgmAQlqQOkpmjwA3AHcA9wG2ZeVdEvCMiri4XuwPYExF3A58D/nNm7pmfiiVp/nj6NEnqMJl5O3D7pGlvaxlO4K3lTZIqyx5hSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJIkSZVkEJYkSVIlTSsIR8RVEXFvRGyJiBuPs9zPR0RGxIaZK1GSJEmaeScMwhFRB24GXgxcAlwXEZdMsdxS4M3AV2e6SEmSJGmmTadH+HJgS2Y+kJnDwK3ANVMs9/vAO4HBGaxPkiRJmhXTCcJrgK0t49vKaRMi4unAusz8hxmsTZIkSZo1p32wXETUgHcBvzGNZa+PiM0RsXnXrl2nu2pJkiTplE0nCG8H1rWMry2njVsKPBX4fEQ8BDwb2DjVAXOZeUtmbsjMDatWrTr1qiVJkqTTNJ0gvAlYHxEXREQ3cC2wcXxmZu7PzJWZeX5mng98Bbg6MzfPSsWSJEnSDDhhEM7MUeAG4A7gHuC2zLwrIt4REVfPdoGSJEnSbGhMZ6HMvB24fdK0tx1j2StPvyxJkiRpdnllOUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUmSJFWSQViSJEmVZBCWJElSJRmEJUmSVEkGYUnqMBFxVUTcGxFbIuLG4yz38xGREbFhLuuTpIXCICxJHSQi6sDNwIuBS4DrIuKSKZZbCrwZ+OrcVihJC4dBWJI6y+XAlsx8IDOHgVuBa6ZY7veBdwKDc1mcJC0kBmFJ6ixrgK0t49vKaRMi4unAusz8h+M9UURcHxGbI2Lzrl27Zr5SSZpnBmFJqpCIqAHvAn7jRMtm5i2ZuSEzN6xatWr2i5OkOWYQlqTOsh1Y1zK+tpw2binwVODzEfEQ8GxgowfMSaoig7AkdZZNwPqIuCAiuoFrgY3jMzNzf2auzMzzM/N84CvA1Zm5eX7KlaT5YxCWpA6SmaPADcAdwD3AbZl5V0S8IyKunt/qJGlhacx3AZKkmZWZtwO3T5r2tmMse+Vc1CRJC5E9wpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaokg7AkSZIqySAsSZKkSjIIS5IkqZIMwpIkSaqkaQXhiLgqIu6NiC0RceMU838lIr4TEXdGxJci4pKZL1WSJEmaOScMwhFRB24GXgxcAlw3RdD9SGb+eGZeCvwB8K6ZLnRcZs7WU0uSJKlCptMjfDmwJTMfyMxh4FbgmtYFMvNAy+hiYFbS6jU3f5nf/Pi3Z+OpJUmSVDGNaSyzBtjaMr4NeNbkhSLi14C3At3A86d6ooi4Hrge4Nxzzz3ZWumqBVv3Hj7px0mSJEmTzdjBcpl5c2ZeCPwW8DvHWOaWzNyQmRtWrVp10utYs6KPHfsGT7NSSZIkaXpBeDuwrmV8bTntWG4FXnYaNR3T6v4+frj/CM2m+wlLkiTp9EwnCG8C1kfEBRHRDVwLbGxdICLWt4z+G+D+mSvxqNX9fYyMJbsGhmbj6SVJklQhJ9xHODNHI+IG4A6gDrw/M++KiHcAmzNzI3BDRLwQGAH2Aq+djWLX9vcBsG3vEc5e1jsbq5AkSVJFTOdgOTLzduD2SdPe1jL85hmua0qryyC8Y98RnnHeirlYpSRJkjpUW11ZbnV/0Qu8fd+Rea5EkiRJ7a6tgvDS3i6W9TbYYRCWJEnSaWqrIAywZsUitu81CEuSJOn0tF8Q7u911whJkiSdtjYMwn3uGiFJkqTT1nZBeHV/HwcGRzk4ODLfpUiSJKmNtV0QXrNi/BRqXmpZkiRJp67tgvD4uYS37zs8z5VIkiSpnbVdEF47EYTtEZYkSdKpa7sgvHJJD1318BRqkiRJOi1tF4RrteCc5Z45QpIkSaen7YIwFKdQ81zCkiRJOh1tGYRXey5hSZIknaa2DMJrVvTx6IFBRsaa812KJEmS2lR7BuH+XpoJj+z3zBGSJEk6NW0ahBcBuHuEJEmSTllbBuHV/b0AHjAnSZKkU9amQXj8MssGYUmSJJ2atgzCvV11Vi7ptkdYkiRJp6wtgzCMn0vYg+UkSZJ0ato2CK/u72P73sPzXYYkSZLaVFsH4R37BsnM+S5FkiRJbahtg/Ca/j6OjIyx9/DIfJciSZKkNtS2QdgzR0iSJOl0tG0QXruiCMKeOUKSJEmnom2D8HiP8Pa9BmFJkiSdvLYNwisWddHXVXfXCEmSJJ2Stg3CEcHq/l53jZAkSdIpadsgDLBmxSJ7hCVJknRK2jsI2yMsSZKkU9TmQbiP3QPDDI6MzXcpkiRJajNtHYQ9l7AkSZJOVVsH4TUTQXhwniuRJElSu2nrIDxxLuF9h+e5EkmSJLWbtg7CT1jeSy1guz3CkjQhIq6KiHsjYktE3DjF/LdGxN0R8e2I+ExEnDcfdUrSfGvrINxVr3H2sl73EZakUkTUgZuBFwOXANdFxCWTFvsmsCEznwZ8HPiDua1SkhaGtg7CUOwe4WWWJWnC5cCWzHwgM4eBW4FrWhfIzM9l5vg+ZV8B1s5xjZK0ILR9EF7T38eO/QZhSSqtAba2jG8rpx3L64F/nNWKJGmBavsgvLq/jx/uG6TZzPkuRZLaSkS8GtgA/OEx5l8fEZsjYvOuXbvmtjhJmgNtH4TXrOhjeKzJ7oGh+S5FkhaC7cC6lvG15bTHiYgXAv8FuDozp2xAM/OWzNyQmRtWrVo1K8VK0nxq/yDc3wvANg+YkySATcD6iLggIrqBa4GNrQtExGXAeylC8M55qFGSFoS2D8Lrz1oKwD/ft3ueK5Gk+ZeZo8ANwB3APcBtmXlXRLwjIq4uF/tDYAnw1xFxZ0RsPMbTSVJHa8x3Aadr3RmLeOGTz+ID//Igb/ypC1jU3fabJEmnJTNvB26fNO1tLcMvnPOiJGkBavseYYBf+ekL2Xt4hNs2bT3xwpIkSRIdEoQ3nH8Gzzx/BX/+zw8yMtac73IkSZLUBqYVhNvhcp2/euWFbN93hL//1o65XrUkSZLa0AmDcLtcrvNnfuwsfuzspbznC9/3nMKSJEk6oen0CLfF5Tojgl+58onc9+gAn7vXswFJkiTp+KYThGfscp2zfZWilz5tNWv6+/izz39/xp9bkiRJnWVGD5Y70eU6Z/sqRV31Gm983gVsfngvmx56bMafX5IkSZ1jOkF4xi7XORde9cxzOWNxN++xV1iSJEnHMZ0g3FaX6+zrrvO6nzyfz3xvJ9975MB8liJJkqQF7IRBuB0v1/lLzzmPRd11/vRz3yfTM0hIkiTpR03resTtdrnO/kXdvOY55/HeLzwAwP/9757Kst6uea5KkiRJC8m0gnA7+s2fu5gl3Q3++DP3882te3n3tZfx9HNXzHdZkiRJWiA64hLLU6nXgje9YD23/Ydn02zCK97zr9z8uS2MebENSZIk0cFBeNwzzjuD29/8PF781Cfwh3fcy6vf91Ue3nNovsuSJEnSPOv4IAywvK+L//e6y/iDn38ad27dx5X/4/O85i++yu3f+SHDo835Lk+SJEnzoGP3EZ4sInjlM9fxvCet5GObtnLbpq38xw9/gzMXd/PyZ6zlVc9cxxNXLZnvMiVJkjRHKhOEx52zvI+3vPBJvOn56/ni/bu49Ws/4H1fepD3fvEBnnzOMl50ydm86Mln89Q1y4iI+S5XkiRJs6RyQXhcvRb8zI+dxc/82FnsPDjIxjt38Km7HuVPPns///Mz93PO8l5e+OSzef7FZ/GM81d4+jVJkqQOU9kg3Oqspb284XlP5A3PeyJ7Bob47Pd28um7H+XjX9/GX37lYWoBFz9hGZdfcAaXX3AGzzz/DFYt7ZnvsiVJknQaDMKTnLmkh1dsWMcrNqxjcGSMrz+8l689+BibHnqMj23aygf+5SEA1vT38eRzlvGU1cu4ZPUyLjlnGWtX9AEwONLk0PAoh4ZGOTQ0Rk9XjXPPWERXvRLHJkqSJLUFg/Bx9HbVueKilVxx0UoARsaafHf7fjY/tJfv7tjPXTsO8NnvPcr4qYl7GjVGxppMdariRi04f+ViLly1mIvOWsKFq5bw5HOWcdFZSwzIkiRJ88AgfBK66jUuO3cFl7Vcoe7I8Bj3PnqQu3cc4KE9h+hp1FjU3WBxT53F5f3A0Bjf3zXA93cOcP/OAf7pnp0TF/bobtS4+AlLecrq5Tx1zTKedPZSFnc36Omq0dtVp6dR3Pd11anXPHhPkiRpphiET1Nfd51L1/Vz6br+aT9meLTJw3sOcfcPD3DXjgN8d/t+/uHbO/jo135wzMfUAs5Y3MPKJd2sXFLcn7mkh6W9jTJwHw3fi7rr9HTV6e2q0dM4et/TVaO7XtxqU4TqsWYyPNqcOLfy0t7GlMtJkiR1AoPwPOhu1Fh/9lLWn72Uay5dA0Bmsm3vEbbsGmBoZIzBkSZDo2MMjTYZHBnj4OAouweG2D0wzO6BIR7+wSH2DAxzeHjslGqo14Lueo1GPRhrJkOjzR+5/HSjFpyxuAjc4wF8aW+D3q46vY1aGbaLoN2oBfVajXoNahE0asVzj/dojy/X26iTMBG4h8fGJta9qLteBPruRhHwexruNiJJkmaNQXiBiAjWnbGIdWcsOqnHjTWTw8PFQXmtB+gNjbaE6ZEmg6NjZfBsMjKaDI+NMTJW9AB31YPuRo3uep3uRo2eRo1mJo8dGmbPwDB7DhUB/MHdhxgYGmWwDOpzobtRo6/cNWRRdxGo+7rrNGpBLYJ6LYhgYvho8B7vDa/TXQ9qtaAe5X05HFG87rWA4Ojw+LL12tFbJgyVr+FQ+ToOjzYn1ttVL74IdNWDrnqNxT0NlvYUYX5JTxHs67XiS0czk9Fm0mwmY5mMjiUjY8WXgZGxnPhCsrinPvGFYEn5BaFWC7J8/FizvB9LBkfHJv4uR0aK4aA4+HPVkh6W9TVO6rzYmUkzi/viNX78Y8eayaHhUQYGi/fcwaFRhkaaLO/r4ozF3fQv6qK3qz6D7wRJkmaeQbjN1WvB0t4uls7xeY4zi17k8ZA9HuzGA1oR6h4fxAdHit7tWhRhsbtR3sqe6SPDYwwMjTIwVISrgcFRDg0Xoe7w8ChHRpocGR7jyMgoo2PJaLPJ8Bg08+i6h0abE18CBkeKdQ+Pdc5ltGvBlAdjnkhXPThzcQ8rl3ZTjyhenzI8j/8dx0P6WCY5xTrqtfJLQ8S0Lk2+uLvOisXdLO3tonv8y1ajRle9uI0H+vEvAqPl+rvL90ZPo/gy092oUa8FzUwo/pGZJFCPoFF++eiql79M1IOg/KID5X2U74+jX2aGRpsMj47xy1dcwLOeeObJv6iSpLZnENYpiYiJXR6Ws/AvNjLe+zoR9ppFj2cRrI72fjaTifkTt0wC6OmqPy6kdddrJEz05h4N582il3Sw6J0fGBrh4OAoY818XC/zeM90o/wi0Kgd3aUkk4me1ole18ERmlkE0vHAN75LyvhuJ0WPeTHcTNhzaIhdB4/uUrNnYIiEoz3n5b7jPY0a9VqNWowH3uIWcfSLRjNhLIvXr7fx+N7qJb0Neuo1DgyO8NihER47NMRjh0bYe3iYg4OjjJQ96EMjTQ4Ojk70phc96cVr0NtVo1aG7IODo+wpv9QMjzUZG0tiohe/+AUAmHjdR8aaE2F6tJkTf9dsCc6Neo2eeq3c3uJA1O5GjUPDo/P3xpQkzSuDsCqhVgtqBLPxa3295i4AkiS1I49EkiRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlTStIBwRV0XEvRGxJSJunGL+T0XENyJiNCJePvNlSpKmaxptdk9EfKyc/9WIOH8eypSkeXfCIBwRdeBm4MXAJcB1EXHJpMV+ALwO+MhMFyhJmr5pttmvB/Zm5kXAHwHvnNsqJWlhmE6P8OXAlsx8IDOHgVuBa1oXyMyHMvPbQHMWapQkTd8J2+xy/IPl8MeBF0REzGGNkrQgTCcIrwG2toxvK6dJkhae6bTZE8tk5iiwHzhzTqqTpAWkMZcri4jrgevL0YGIuPc4i68Eds9+VfOq07ex07cPOn8bO3374NS28bzZKGShOU6b7fuic7idnaMK2winvp1TttvTCcLbgXUt42vLaSctM28BbpnOshGxOTM3nMp62kWnb2Onbx90/jZ2+vZBR27jdNrs8WW2RUQDWA7smfxEx2qzO/A1+xFV2EZwOztJFbYRZn47p7NrxCZgfURcEBHdwLXAxpkqQJI0o6bTZm8EXlsOvxz4bGbmHNYoSQvCCYNwuf/YDcAdwD3AbZl5V0S8IyKuBoiIZ0bENuAVwHsj4q7ZLFqSNLXptNnAXwBnRsQW4K3Aj5xiTZKqYFr7CGfm7cDtk6a9rWV4E8XPbzNpWrtQtLlO38ZO3z7o/G3s9O2DDtzGabTZgxQdF6eq416zKVRhG8Ht7CRV2EaY4e0Mfw2TJElSFXmJZUmSJFXSggzCJ7o8aDuKiPdHxM6I+G7LtDMi4tMRcX95v2I+azwdEbEuIj4XEXdHxF0R8eZyekdsY0T0RsTXIuJb5fb9Xjn9gvIStVvKS9Z2z3etpyMi6hHxzYj4ZDneadv3UER8JyLujIjN5bSOeI/OhU5sm6Hz22fo/DZ6XFXaauj89hrmps1ecEE4pnd50Hb0AeCqSdNuBD6TmeuBz9DeB6yMAr+RmZcAzwZ+rfy7dco2DgHPz8yfAC4FroqIZ1NcmvaPykvV7qW4dG07ezPFAVbjOm37AH4mMy9tOf1Op7xHZ1UHt83Q+e0zdH4bPa4qbTVUo72GWW6zF1wQZnqXB207mflF4LFJk1svc/pB4GVzWdNMyswfZuY3yuGDFP8519Ah25iFgXK0q7wl8HyKS9RCG28fQESsBf4N8L5yPOig7TuOjniPzoGObJuh89tn6Pw2elwV2mqodHsNM/yeXYhBuEqXdD47M39YDj8CnD2fxcyUiDgfuAz4Kh20jeXPUHcCO4FPA98H9pWnq4L2f6/+MfCbQLMcP5PO2j4oPhA/FRFfj+KqadBB79FZVqW2GTr4fdGpbfS4CrTVUI32GuagzZ7TSyzr2DIzI6LtT+EREUuAvwHekpkHii+phXbfxswcAy6NiH7gE8DF81vRzImIlwI7M/PrEXHlPJczm56bmdsj4izg0xHxvdaZ7f4e1ezopPdFJ7fR4zq5rYZKtdcwB232QuwRnrFLOreBRyPiHIDyfuc813NaIqKLooH9cGb+bTm5o7YRIDP3AZ8DngP0R3GJWmjv9+oVwNUR8RDFT97PB95N52wfAJm5vbzfSfEBeTkd+B6dJVVqm6ED3xdVaaPHdWhbDRVpr2Fu2uyFGISrdEnn1sucvhb4X/NYy2kp90/6C+CezHxXy6yO2MaIWFX2LhARfcCLKPax+xzFJWqhjbcvM2/KzLWZeT7F/7nPZuYv0iHbBxARiyNi6fgw8LPAd+mQ9+gcqFLbDB32vuj0Nnpcp7fVUI32Guawzc7MBXcDXgLcR7Ffz3+Z73pmaJs+CvwQGKHYd+f1FPv0fAa4H/gn4Iz5rvM0tu+5FPvyfBu4s7y9pFO2EXga8M1y+74LvK2c/kTga8AW4K+BnvmudQa29Urgk522feW2fKu83TXetnTKe3SOXsOOa5vL7ero9rncxo5uo1u2szJtdbldHdlet2zPrLfZXllOkiRJlbQQd42QJEmSZp1BWJIkSZVkEJYkSVIlGYQlSZJUSQZhSZIkVZJBWJUVEVdGxCfnuw5J0onZZms2GIQlSZJUSQZhLXgR8eqI+FpE3BkR742IekQMRMQfRcRdEfGZiFhVLntpRHwlIr4dEZ+IiBXl9Isi4p8i4lsR8Y2IuLB8+iUR8fGI+F5EfLi8+pIk6RTZZqudGIS1oEXEk4FXAVdk5qXAGPCLwGJgc2Y+BfgC8LvlQz4E/FZmPg34Tsv0DwM3Z+ZPAD9JcRUpgMuAtwCXUFzF5opZ3iRJ6li22Wo3jfkuQDqBFwDPADaVX/z7gJ1AE/hYucxfAX8bEcuB/sz8Qjn9g8Bfl9cqX5OZnwDIzEGA8vm+lpnbyvE7gfOBL836VklSZ7LNVlsxCGuhC+CDmXnT4yZG/F+TljvVa4UPtQyP4f8JSTodttlqK+4aoYXuM8DLI+IsgIg4IyLOo3jvvrxc5heAL2XmfmBvRDyvnP4a4AuZeRDYFhEvK5+jJyIWzeVGSFJF2GarrfhNSgtaZt4dEb8DfCoiasAI8GvAIeDyct5Oin3SAF4LvKdsNB8A/n05/TXAeyPiHeVzvGION0OSKsE2W+0mMk/11wlp/kTEQGYume86JEknZputhcpdIyRJklRJ9ghLkiSpkuwRliRJUiUZhCVJklRJBmFJkiRVkkFYkiRJlWQQliRJUiUZhCVJklRJ/xtPbjbGjFGkiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(\"train\", (12, 6))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.title(\"Epoch Average Loss\")\n",
    "x = [i + 1 for i in range(len(epoch_loss_values))]\n",
    "y = epoch_loss_values\n",
    "plt.xlabel(\"epoch\")\n",
    "plt.ylim([0.1, 0.7])\n",
    "plt.plot(x, y)\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.title(\"Val Mean Dice\")\n",
    "x = [val_interval * (i + 1) for i in range(len(metric_values))]\n",
    "y = metric_values\n",
    "plt.xlabel(\"epoch\")\n",
    "plt.ylim([0, 1.0])\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zIO0xaNDKyZ0"
   },
   "source": [
    "## Cleanup data directory\n",
    "\n",
    "Remove directory if a temporary was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "qJpZYdCgKyZ0"
   },
   "outputs": [],
   "source": [
    "# if directory is None:\n",
    "#     shutil.rmtree(root_dir)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "transfer_mmar.ipynb",
   "provenance": [
    {
     "file_id": "https://github.com/Project-MONAI/tutorials/blob/main/2d_classification/mednist_tutorial.ipynb",
     "timestamp": 1607545340801
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}