demo/denas/computer_vision/DENAS_ViT_DEMO.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42cbc49e",
   "metadata": {},
   "source": [
    "[![open in colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/68a5a83f25022d0f1ba6610b18f620d0cdde9d68/demo/denas/computer_vision/DENAS_ViT_DEMO.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6ce66fb",
   "metadata": {},
   "source": [
    "# AIOK DE-NAS for ViT Demo\n",
    "\n",
    "DE-NAS is a multi-model, hardware-aware, train-free NAS to construct compact model architectures for target platform directly. DE-NAS includes CNN-based search space for CV domain and Transformer-based search space for CV/NLP/ASR domains, and leverages hardware-aware train-free scoring method to evaluate the performance of the candidate architecture without training.\n",
    "\n",
    "This demo mainly introduces CV integration with DE-NAS to search lighter, faster, higher performance transformer-based model in a training-free way."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c8671985",
   "metadata": {},
   "source": [
    "# Content\n",
    "* [Overview](#overview)\n",
    "    * [DE-NAS on CV(ViT) Domain](#de-nas-on-cvvit-domain)\n",
    "    * [Performance](#performance)\n",
    "* [Getting Started](#getting-started)\n",
    "    * [1. Environment Setup](#1-environment-setup)\n",
    "    * [2. Workflow Prepare](#2-workflow-prepare)\n",
    "    * [3. Data Prepare](#3-data-prepare)\n",
    "    * [4. Launch Search](#4-launch-search)\n",
    "    * [5. Launch Training with Best Searched Model Structure](#5-launch-training-with-best-searched-model-structure)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "480c73af",
   "metadata": {},
   "source": [
    "# Overview\n",
    "## DE-NAS on CV(ViT) Domain\n",
    "For VIT models, the basic structure is the transformer based model which generated from the unified transformer based search space, the Search Space for VIT domain as following:\n",
    "\n",
    "``` yaml\n",
    "SUPERNET:\n",
    "  MLP_RATIO: 4.0 # Linear layer ratio\n",
    "  NUM_HEADS: 10 # num of attention heads\n",
    "  EMBED_DIM: 640 # Q,K,V embedding dimision\n",
    "  DEPTH: 16 # number of transformer layers\n",
    "SEARCH_SPACE:\n",
    "  MLP_RATIO:[3.0,3.5,4.0]\n",
    "  NUM_HEADS:[3,4,5,6,7,8,9,10]\n",
    "  DEPTH:[12,13,14,15,16]\n",
    "  EMBED_DIM:[192,216,240,320,384,448,528,576,624]\n",
    "```\n",
    "\n",
    "## Performance\n",
    "<center>\n",
    "<img src=\"./img/denas_vit_stock.png\" width=\"500\"/><figure>DE-NAS ViT performance over stock model</figure>\n",
    "</center>\n",
    "\n",
    "<center>\n",
    "<img src=\"./img/denas_vit_autoformer.png\" width=\"500\"/><figure>DE-NAS ViT performance over Autoformer</figure>\n",
    "</center>\n",
    "\n",
    "* Testing methodology\n",
    "  * Dataset: CIFAR10, Metric: Top-1 accuracy  0.9482\n",
    "  * Baseline: model AutoFormer\n",
    "  * Search best model structure with highest DE-score\n",
    "  * DE-NAS model is highest DE-Score model, total searched candidates is 1000\n",
    "  * Training epoch: 200\n",
    "* DENAS ViT delivered 35.63x search and 4.44x training speedup over SOTA NAS (AutoFormer) with 5% accuracy loss (0.90 vs. 0.9482)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2c248fc3",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "## 1. Environment Setup\n",
    "\n",
    "### (Option 1) Use Pip install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d7172b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install e2eAIOK-denas --pre"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "def012f3",
   "metadata": {},
   "source": [
    "### (Option 2) Use Docker\n",
    "\n",
    "Step1. prepare code\n",
    "``` bash\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "git submodule update --init –recursive\n",
    "python3 scripts/start_e2eaiok_docker.py -b pytorch112 -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "```\n",
    "\n",
    "Step2. build docker image\n",
    "```bash\n",
    "python3 scripts/start_e2eaiok_docker.py -b pytorch112 -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "```\n",
    "\n",
    "Step3. run docker and start conda env\n",
    "``` bash\n",
    "sshpass -p docker ssh ${host0} -p 12347\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8724642e",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "### Configuration\n",
    "\n",
    "* Create config file for VIT model search\n",
    "\n",
    "``` yaml\n",
    "model_type: transformer\n",
    "search_engine: EvolutionarySearchEngine # Options: random search, Evolution algorithm, SigOpt search\n",
    "batch_size: 64\n",
    "random_max_epochs: 1000\n",
    "max_epochs: 10\n",
    "select_num: 50\n",
    "population_num: 50\n",
    "m_prob: 0.2\n",
    "s_prob: 0.4\n",
    "crossover_num: 25\n",
    "mutation_num: 25\n",
    "max_param_limits: 100\n",
    "min_param_limits: 1\n",
    "supernet_cfg: /home/vmagent/app/e2eaiok/conf/denas/cv/supernet_vit/supernet_large.conf\n",
    "img_size: 224\n",
    "patch_size: 16 # patch number for input image\n",
    "drop_rate: 0.0 # dropout ratio\n",
    "drop_path_rate: 0.1\n",
    "max_relative_position: 14 #max distance in relative position embedding\n",
    "gp: True\n",
    "relative_position: True\n",
    "change_qkv: True\n",
    "abs_pos: True\n",
    "seed: 0\n",
    "expressivity_weight: 0 # weight for train free score of expressivity \n",
    "complexity_weight: 0 # weight for train free score of complexity\n",
    "diversity_weight: 1 # weight for train free score of diversity score\n",
    "saliency_weight: 1 # weight for train free score of salience score\n",
    "latency_weight: 10000 # weight for latency setup according to different platforms\n",
    "```\n",
    "\n",
    "* Create config file for VIT model train\n",
    "\n",
    "``` yaml\n",
    "domain: vit\n",
    "train_epochs: 1\n",
    "eval_epochs: 1\n",
    "input_size: 32\n",
    "best_model_structure: /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/best_vit_model_structure.txt\n",
    "num_classes: 10\n",
    "dist_backend: gloo\n",
    "train_batch_size: 128\n",
    "eval_batch_size: 128\n",
    "data_path: ~/data/pytorch_cifar10\n",
    "data_set: CIFAR10\n",
    "output_dir: ./\n",
    "num_workers: 10\n",
    "pin_mem: True\n",
    "eval_metric: \"accuracy\"\n",
    "learning_rate: 0.001\n",
    "momentum: 0.9\n",
    "weight_decay: 0.01\n",
    "optimizer: \"SGD\"\n",
    "criterion: \"CrossEntropyLoss\"\n",
    "lr_scheduler: \"CosineAnnealingLR\"\n",
    "print_freq: 10\n",
    "mode: \"train\"\n",
    "gp: True\n",
    "change_qkv: True \n",
    "relative_position: True # whether to use relative position embedding\n",
    "drop_path: 0.1 #Drop path rate\n",
    "max_relative_position: 14 #max distance in relative position embedding\n",
    "no_abs_pos: False\n",
    "patch_size: 16 # patch size for input image\n",
    "drop: 0.0\n",
    "metric_threshold: 94 # early stop target accuracy\n",
    "SUPERNET:\n",
    "  MLP_RATIO: 4.0\n",
    "  NUM_HEADS: 10\n",
    "  EMBED_DIM: 640\n",
    "  DEPTH: 16\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1a8eec30",
   "metadata": {},
   "source": [
    "## 3. Data Prepare\n",
    "\n",
    "### CIFAR(10/100) Dataset Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88843f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "\n",
    "data_folder = \"~/data/pytorch_cifar\"\n",
    "is_train = True\n",
    "transform = None\n",
    "\n",
    "# Download Cifar10 Dataset\n",
    "dataset = datasets.CIFAR10(data_folder, train=is_train, transform=transform, download=True)\n",
    "\n",
    "# Download Cifar100 Dataset\n",
    "dataset = datasets.CIFAR100(data_folder, train=is_train, transform=transform, download=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "01b0f3e6",
   "metadata": {},
   "source": [
    "## 4. Launch Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37ecaa5a",
   "metadata": {},
   "source": [
    "### Launch Search for ViT\n",
    "The input is the configuration for VIT domain, edit the configuration file of `e2eaiok_denas_vit.conf` for vit model search, and then run below commond line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "94740c7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "paths: /home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/asr/utils, /home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/asr\n",
      "['/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/opt/intel/oneapi/advisor/2022.1.0/pythonapi', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python39.zip', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/lib-dynload', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/site-packages', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/site-packages/warprnnt_pytorch-0.1-py3.9-linux-x86_64.egg', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/site-packages/e2eAIOK-0.2.4-py3.9.egg', '', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/asr']\n",
      "12/02/2022 01:58:42 - INFO - DENAS -   epoch = 0\n",
      "/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/module/cv/multihead_super.py:51: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "  distance_mat_v = (range_vec_k[None, :] // int(length_q ** 0.5 )  - range_vec_q[:, None] // int(length_q ** 0.5 ))\n",
      "12/02/2022 01:58:50 - INFO - DENAS -   random 1/50 structure (15, 3.5, 3.0, 3.5, 4.0, 3.5, 3.5, 3.5, 3.5, 3.5, 4.0, 3.0, 4.0, 3.0, 3.5, 3.0, 10, 3, 7, 10, 5, 7, 9, 10, 7, 4, 5, 3, 9, 3, 10, 448) nas_score 374.1955871582031 params 33.208394\n",
      "12/02/2022 01:58:57 - INFO - DENAS -   random 2/50 structure (15, 4.0, 3.0, 3.5, 3.5, 3.5, 4.0, 4.0, 3.0, 4.0, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 3, 3, 9, 10, 6, 9, 10, 10, 9, 9, 3, 7, 6, 10, 10, 448) nas_score 237.9915008544922 params 35.390666\n",
      "12/02/2022 01:59:03 - INFO - DENAS -   random 3/50 structure (13, 4.0, 3.5, 4.0, 3.0, 3.0, 4.0, 3.0, 3.0, 3.0, 4.0, 3.5, 3.0, 3.0, 5, 7, 6, 3, 5, 7, 5, 9, 3, 7, 5, 10, 7, 320) nas_score 285.9417724609375 params 15.950218\n",
      "12/02/2022 01:59:12 - INFO - DENAS -   random 4/50 structure (16, 4.0, 4.0, 3.5, 3.5, 3.0, 4.0, 3.5, 3.5, 4.0, 3.0, 3.5, 3.0, 3.0, 3.0, 3.0, 4.0, 9, 5, 6, 3, 3, 9, 10, 4, 4, 3, 10, 3, 9, 10, 7, 9, 528) nas_score 352.50457763671875 params 45.742258\n",
      "12/02/2022 01:59:20 - INFO - DENAS -   random 5/50 structure (16, 3.5, 4.0, 3.0, 3.0, 4.0, 4.0, 3.5, 4.0, 3.5, 3.5, 3.5, 4.0, 4.0, 4.0, 3.5, 3.0, 5, 7, 3, 6, 7, 9, 5, 10, 4, 4, 3, 9, 5, 3, 9, 4, 448) nas_score 216.00521850585938 params 34.595978\n",
      "12/02/2022 01:59:25 - INFO - DENAS -   random 6/50 structure (13, 3.5, 3.5, 3.0, 3.0, 3.0, 4.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3, 3, 3, 9, 4, 7, 10, 7, 3, 6, 3, 5, 10, 216) nas_score 203.23052978515625 params 8.681074\n",
      "12/02/2022 01:59:31 - INFO - DENAS -   random 7/50 structure (12, 4.0, 3.0, 3.0, 3.0, 4.0, 3.0, 3.5, 3.0, 4.0, 3.0, 4.0, 3.0, 7, 6, 7, 3, 10, 5, 3, 4, 3, 9, 5, 5, 528) nas_score 423.9903259277344 params 32.319106\n",
      "12/02/2022 01:59:38 - INFO - DENAS -   random 8/50 structure (13, 3.0, 4.0, 3.5, 3.0, 4.0, 3.0, 4.0, 3.5, 3.0, 3.5, 3.5, 4.0, 3.5, 10, 7, 4, 9, 9, 4, 10, 3, 10, 9, 4, 10, 4, 448) nas_score 371.549560546875 params 29.541674\n",
      "12/02/2022 01:59:45 - INFO - DENAS -   random 9/50 structure (16, 3.5, 3.0, 4.0, 3.5, 4.0, 3.0, 3.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.5, 3.5, 10, 9, 5, 4, 7, 9, 3, 6, 9, 3, 5, 9, 3, 7, 5, 4, 320) nas_score 394.6971435546875 params 20.308266\n",
      "12/02/2022 01:59:53 - INFO - DENAS -   random 10/50 structure (15, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 4.0, 4.0, 3.0, 4.0, 3.5, 3.5, 4.0, 3.5, 4.0, 3, 3, 7, 4, 9, 5, 4, 4, 4, 9, 6, 6, 9, 9, 7, 528) nas_score 234.67391967773438 params 43.980058\n",
      "12/02/2022 02:00:00 - INFO - DENAS -   random 11/50 structure (12, 3.5, 4.0, 4.0, 3.5, 4.0, 4.0, 3.0, 3.0, 3.5, 3.0, 3.5, 4.0, 4, 6, 7, 6, 7, 7, 10, 3, 3, 6, 5, 5, 576) nas_score 348.6418762207031 params 39.442762\n",
      "12/02/2022 02:00:05 - INFO - DENAS -   random 12/50 structure (12, 3.5, 3.0, 4.0, 4.0, 3.0, 4.0, 3.0, 3.0, 3.5, 4.0, 3.5, 3.5, 3, 4, 3, 9, 10, 3, 10, 9, 7, 7, 3, 4, 216) nas_score 201.36280822753906 params 8.242498\n",
      "12/02/2022 02:00:13 - INFO - DENAS -   random 13/50 structure (16, 4.0, 3.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 3.0, 3.5, 4, 9, 7, 10, 9, 9, 5, 3, 10, 4, 5, 10, 3, 3, 3, 4, 384) nas_score 215.16859436035156 params 26.28193\n",
      "12/02/2022 02:00:20 - INFO - DENAS -   random 14/50 structure (16, 3.5, 3.5, 4.0, 3.0, 4.0, 4.0, 4.0, 4.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.5, 3.5, 4.0, 4, 4, 6, 7, 5, 3, 4, 4, 5, 5, 5, 10, 5, 9, 3, 5, 192) nas_score 271.71490478515625 params 8.909386\n",
      "12/02/2022 02:00:26 - INFO - DENAS -   random 15/50 structure (12, 3.5, 3.0, 3.0, 4.0, 3.5, 3.5, 3.5, 4.0, 3.0, 3.5, 3.0, 3.5, 9, 4, 3, 5, 4, 10, 7, 9, 3, 5, 6, 10, 448) nas_score 485.7431335449219 params 25.655114\n",
      "12/02/2022 02:00:32 - INFO - DENAS -   random 16/50 structure (14, 3.5, 3.0, 3.0, 4.0, 3.5, 3.5, 3.5, 3.5, 3.0, 3.5, 3.0, 4.0, 3.5, 3.5, 3, 5, 5, 9, 9, 4, 4, 9, 7, 6, 10, 9, 3, 3, 216) nas_score 187.4281768798828 params 9.59845\n",
      "12/02/2022 02:00:39 - INFO - DENAS -   random 17/50 structure (13, 4.0, 3.0, 3.0, 3.5, 3.0, 3.0, 3.5, 4.0, 4.0, 3.5, 3.5, 3.5, 4.0, 9, 9, 4, 6, 3, 5, 4, 5, 7, 10, 4, 6, 4, 448) nas_score 479.0843811035156 params 27.588714\n",
      "12/02/2022 02:00:44 - INFO - DENAS -   random 18/50 structure (12, 3.0, 4.0, 3.0, 4.0, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 3.5, 3.0, 9, 3, 10, 4, 7, 4, 3, 4, 6, 6, 5, 7, 240) nas_score 341.97784423828125 params 9.268426\n",
      "12/02/2022 02:00:51 - INFO - DENAS -   random 19/50 structure (12, 4.0, 3.5, 3.0, 4.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.5, 4.0, 3.5, 10, 6, 6, 9, 9, 10, 4, 7, 7, 4, 3, 5, 448) nas_score 313.03704833984375 params 27.63601\n",
      "12/02/2022 02:00:57 - INFO - DENAS -   random 20/50 structure (14, 3.0, 4.0, 3.0, 3.5, 4.0, 3.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.5, 3.0, 3.0, 7, 10, 3, 3, 4, 4, 3, 5, 3, 10, 10, 6, 5, 10, 240) nas_score 264.6897888183594 params 11.134666\n",
      "12/02/2022 02:01:04 - INFO - DENAS -   random 21/50 structure (13, 4.0, 3.5, 3.5, 4.0, 3.5, 4.0, 4.0, 3.0, 4.0, 3.0, 4.0, 4.0, 4.0, 3, 9, 6, 10, 4, 5, 7, 7, 6, 10, 6, 10, 10, 528) nas_score 266.75323486328125 params 40.314082\n",
      "12/02/2022 02:01:13 - INFO - DENAS -   random 22/50 structure (16, 4.0, 3.0, 3.0, 4.0, 3.0, 4.0, 3.0, 3.5, 4.0, 4.0, 3.5, 3.5, 3.0, 3.5, 3.5, 3.5, 10, 6, 6, 10, 6, 3, 4, 9, 4, 4, 5, 9, 10, 5, 3, 3, 576) nas_score 390.81036376953125 params 52.255114\n",
      "12/02/2022 02:01:20 - INFO - DENAS -   random 23/50 structure (15, 3.0, 3.5, 4.0, 4.0, 3.0, 4.0, 4.0, 3.0, 3.5, 3.5, 3.0, 4.0, 3.5, 3.5, 3.5, 3, 3, 6, 10, 4, 3, 3, 7, 7, 4, 9, 5, 3, 9, 7, 448) nas_score 310.7445373535156 params 31.42753\n",
      "12/02/2022 02:01:26 - INFO - DENAS -   random 24/50 structure (13, 3.5, 3.5, 3.0, 3.0, 4.0, 4.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.0, 4.0, 9, 7, 5, 10, 10, 4, 6, 10, 5, 9, 10, 9, 10, 216) nas_score 335.2095642089844 params 10.49473\n",
      "12/02/2022 02:01:33 - INFO - DENAS -   random 25/50 structure (16, 3.0, 3.0, 3.5, 3.5, 3.5, 3.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.0, 4.0, 9, 7, 7, 4, 4, 3, 10, 10, 9, 10, 7, 9, 7, 6, 7, 5, 216) nas_score 236.29281616210938 params 11.68447\n",
      "12/02/2022 02:01:39 - INFO - DENAS -   random 26/50 structure (13, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 3.0, 3.0, 3.5, 3.0, 4.0, 3.0, 3.5, 6, 9, 6, 3, 6, 10, 5, 9, 5, 3, 5, 3, 3, 448) nas_score 527.647216796875 params 26.440362\n",
      "12/02/2022 02:01:44 - INFO - DENAS -   random 27/50 structure (12, 3.5, 3.5, 3.0, 3.0, 3.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 3.0, 9, 9, 9, 3, 9, 3, 5, 5, 9, 3, 7, 4, 240) nas_score 360.819091796875 params 9.41125\n",
      "12/02/2022 02:01:50 - INFO - DENAS -   random 28/50 structure (13, 3.5, 3.0, 3.5, 3.5, 4.0, 3.5, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 3.5, 5, 7, 9, 5, 4, 7, 3, 3, 7, 7, 5, 4, 6, 240) nas_score 286.1578369140625 params 9.870394\n",
      "12/02/2022 02:01:57 - INFO - DENAS -   random 29/50 structure (13, 3.0, 3.5, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 3.0, 3.0, 4.0, 3, 6, 3, 9, 3, 4, 6, 5, 7, 6, 9, 4, 7, 528) nas_score 219.6857452392578 params 34.96009\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12/02/2022 02:02:02 - INFO - DENAS -   random 30/50 structure (14, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.0, 3.5, 3.5, 3.0, 3.5, 9, 5, 6, 4, 5, 5, 6, 3, 4, 10, 3, 5, 3, 7, 240) nas_score 260.6387023925781 params 10.64161\n",
      "12/02/2022 02:02:09 - INFO - DENAS -   random 31/50 structure (15, 3.5, 3.0, 3.5, 3.5, 3.0, 3.0, 3.5, 3.5, 4.0, 3.0, 3.5, 3.5, 4.0, 4.0, 3.0, 6, 10, 4, 5, 3, 10, 9, 6, 7, 6, 3, 4, 5, 3, 10, 384) nas_score 289.3059997558594 params 24.696586\n",
      "12/02/2022 02:02:16 - INFO - DENAS -   random 32/50 structure (16, 4.0, 3.0, 3.5, 4.0, 3.5, 3.0, 4.0, 3.5, 3.0, 4.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.5, 9, 3, 3, 5, 6, 7, 7, 9, 6, 6, 3, 9, 5, 5, 5, 10, 320) nas_score 244.2968292236328 params 20.000586\n",
      "12/02/2022 02:02:26 - INFO - DENAS -   random 33/50 structure (16, 3.0, 3.0, 3.0, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 4, 7, 7, 9, 4, 7, 3, 6, 9, 7, 6, 9, 10, 10, 5, 5, 576) nas_score 235.0663604736328 params 52.550986\n",
      "12/02/2022 02:02:31 - INFO - DENAS -   random 34/50 structure (14, 4.0, 4.0, 3.0, 3.0, 3.0, 4.0, 3.0, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 4.0, 5, 5, 9, 6, 9, 6, 4, 6, 6, 9, 7, 5, 6, 9, 192) nas_score 282.96173095703125 params 8.510314\n",
      "12/02/2022 02:02:38 - INFO - DENAS -   random 35/50 structure (15, 3.5, 3.0, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.5, 3.5, 3.5, 3.0, 4.0, 3.0, 4.0, 10, 3, 9, 9, 9, 6, 3, 5, 3, 3, 7, 3, 5, 10, 9, 384) nas_score 430.8814392089844 params 24.992074\n",
      "12/02/2022 02:02:44 - INFO - DENAS -   random 36/50 structure (13, 3.0, 4.0, 3.5, 3.0, 3.0, 3.5, 3.5, 4.0, 3.5, 3.5, 3.0, 3.5, 3.5, 9, 9, 6, 4, 6, 9, 4, 7, 5, 4, 4, 4, 6, 448) nas_score 324.3466491699219 params 27.301738\n",
      "12/02/2022 02:02:52 - INFO - DENAS -   random 37/50 structure (15, 3.5, 3.5, 3.5, 4.0, 3.0, 3.0, 3.5, 3.0, 4.0, 4.0, 3.0, 3.5, 3.0, 3.0, 4.0, 3, 10, 4, 5, 3, 9, 9, 9, 4, 4, 7, 5, 7, 3, 9, 624) nas_score 279.7934265136719 params 55.474306\n",
      "12/02/2022 02:02:58 - INFO - DENAS -   random 38/50 structure (14, 3.5, 3.5, 3.5, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.5, 4.0, 3.5, 6, 5, 9, 6, 4, 5, 5, 3, 3, 3, 4, 5, 3, 5, 192) nas_score 256.7184143066406 params 7.44913\n",
      "12/02/2022 02:03:04 - INFO - DENAS -   random 39/50 structure (13, 3.0, 3.5, 4.0, 3.0, 4.0, 3.5, 4.0, 3.0, 3.5, 3.0, 3.5, 4.0, 3.0, 7, 3, 10, 5, 5, 7, 6, 5, 5, 3, 7, 3, 4, 448) nas_score 405.6549072265625 params 26.698506\n",
      "12/02/2022 02:03:09 - INFO - DENAS -   random 40/50 structure (12, 3.0, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 4.0, 4.0, 3.5, 3.5, 4.0, 10, 6, 5, 6, 5, 10, 9, 4, 3, 5, 7, 3, 192) nas_score 241.88507080078125 params 7.074634\n",
      "12/02/2022 02:03:15 - INFO - DENAS -   random 41/50 structure (15, 3.5, 4.0, 3.5, 3.0, 3.5, 3.5, 3.5, 3.5, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.5, 4, 6, 7, 3, 10, 6, 7, 10, 10, 6, 3, 4, 4, 6, 4, 240) nas_score 276.23577880859375 params 11.63293\n",
      "12/02/2022 02:03:23 - INFO - DENAS -   random 42/50 structure (14, 3.5, 3.5, 3.5, 3.0, 4.0, 3.5, 4.0, 4.0, 3.0, 4.0, 3.5, 3.5, 4.0, 4.0, 6, 7, 9, 5, 5, 4, 3, 3, 7, 10, 9, 3, 4, 9, 528) nas_score 268.3005065917969 params 40.50193\n",
      "12/02/2022 02:03:29 - INFO - DENAS -   random 43/50 structure (13, 3.0, 3.5, 4.0, 4.0, 3.0, 4.0, 4.0, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 5, 3, 9, 7, 5, 4, 9, 9, 9, 7, 9, 5, 4, 320) nas_score 210.8346710205078 params 16.95569\n",
      "12/02/2022 02:03:34 - INFO - DENAS -   random 44/50 structure (12, 4.0, 3.5, 4.0, 3.5, 3.0, 3.5, 4.0, 3.5, 3.5, 4.0, 3.0, 3.0, 10, 3, 7, 7, 5, 5, 7, 9, 3, 10, 4, 6, 240) nas_score 291.62109375 params 9.934642\n",
      "12/02/2022 02:03:41 - INFO - DENAS -   random 45/50 structure (13, 4.0, 3.0, 3.0, 3.0, 3.5, 4.0, 4.0, 3.0, 3.0, 4.0, 3.0, 3.0, 3.5, 5, 7, 7, 4, 9, 6, 3, 7, 10, 3, 7, 7, 5, 576) nas_score 359.5926818847656 params 41.742154\n",
      "12/02/2022 02:03:48 - INFO - DENAS -   random 46/50 structure (15, 3.5, 3.0, 3.0, 4.0, 4.0, 3.0, 4.0, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 3.0, 3.5, 4, 7, 6, 9, 4, 10, 5, 10, 5, 6, 3, 4, 3, 5, 9, 320) nas_score 256.27984619140625 params 18.82129\n",
      "12/02/2022 02:03:53 - INFO - DENAS -   random 47/50 structure (14, 3.5, 4.0, 4.0, 4.0, 4.0, 3.0, 3.5, 3.5, 4.0, 3.0, 3.5, 3.0, 3.5, 4.0, 7, 3, 9, 9, 4, 5, 3, 6, 9, 3, 3, 4, 3, 3, 320) nas_score 355.25054931640625 params 16.636202\n",
      "12/02/2022 02:04:02 - INFO - DENAS -   random 48/50 structure (14, 3.0, 3.0, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 3.5, 3.0, 3.5, 3.5, 3.0, 4.0, 5, 9, 10, 9, 6, 6, 3, 6, 5, 7, 7, 10, 9, 9, 576) nas_score 288.3834533691406 params 46.846282\n",
      "12/02/2022 02:04:07 - INFO - DENAS -   random 49/50 structure (12, 4.0, 3.0, 3.5, 3.5, 3.0, 3.0, 4.0, 3.0, 4.0, 4.0, 4.0, 4.0, 9, 4, 3, 5, 4, 10, 10, 10, 4, 4, 3, 10, 240) nas_score 343.5627746582031 params 9.992362\n",
      "12/02/2022 02:04:15 - INFO - DENAS -   random 50/50 structure (16, 3.0, 4.0, 3.0, 3.5, 4.0, 3.0, 4.0, 4.0, 3.5, 4.0, 3.0, 4.0, 4.0, 3.0, 3.5, 3.5, 10, 9, 9, 6, 3, 9, 3, 6, 3, 5, 5, 9, 4, 6, 4, 7, 448) nas_score 295.0278625488281 params 34.768522\n",
      "12/02/2022 02:04:15 - INFO - DENAS -   random_num = 50\n",
      "12/02/2022 02:04:21 - INFO - DENAS -   mutation 1/25 structure (14, 3.0, 4.0, 3.0, 3.5, 4.0, 3.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.5, 3.0, 3.0, 7, 10, 3, 3, 4, 4, 3, 5, 4, 10, 10, 6, 9, 10, 240) nas_score 233.85121154785156 params 11.442826\n",
      "12/02/2022 02:04:28 - INFO - DENAS -   mutation 2/25 structure (15, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.0, 3.0, 4.0, 3.5, 3.5, 4.0, 3.5, 4.0, 3, 3, 7, 4, 9, 5, 4, 3, 4, 9, 6, 6, 9, 9, 6, 528) nas_score 252.20050048828125 params 43.151242\n",
      "12/02/2022 02:04:33 - INFO - DENAS -   mutation 3/25 structure (12, 3.5, 3.0, 3.0, 4.0, 3.0, 4.0, 3.0, 3.0, 3.5, 4.0, 3.5, 3.5, 6, 4, 3, 9, 10, 3, 4, 3, 7, 7, 3, 4, 216) nas_score 475.2862548828125 params 7.649578\n",
      "12/02/2022 02:04:40 - INFO - DENAS -   mutation 4/25 structure (14, 3.0, 3.0, 3.0, 4.0, 3.5, 3.5, 3.5, 3.5, 3.0, 3.5, 3.0, 4.0, 3.5, 3.5, 3, 5, 5, 9, 3, 4, 4, 4, 7, 6, 10, 9, 3, 3, 528) nas_score 268.6385498046875 params 37.330354\n",
      "12/02/2022 02:04:47 - INFO - DENAS -   mutation 5/25 structure (15, 3.5, 3.0, 3.5, 4.0, 3.5, 3.5, 3.0, 3.5, 3.5, 4.0, 3.0, 4.0, 3.0, 3.5, 3.0, 10, 3, 7, 10, 5, 7, 5, 10, 7, 4, 5, 3, 9, 3, 6, 448) nas_score 439.08984375 params 32.088426\n",
      "12/02/2022 02:04:54 - INFO - DENAS -   mutation 6/25 structure (13, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.0, 4.0, 9, 9, 6, 6, 3, 5, 4, 5, 7, 10, 3, 4, 7, 528) nas_score 458.34429931640625 params 37.16749\n",
      "12/02/2022 02:05:01 - INFO - DENAS -   mutation 7/25 structure (14, 3.0, 3.0, 3.0, 3.5, 3.0, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 4, 6, 3, 5, 6, 7, 3, 6, 9, 7, 6, 9, 10, 10, 576) nas_score 378.5887451171875 params 44.705674\n",
      "12/02/2022 02:05:07 - INFO - DENAS -   mutation 8/25 structure (16, 3.0, 4.0, 3.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 3.5, 3.0, 3.0, 3.0, 3.5, 3.5, 4, 3, 10, 4, 7, 6, 3, 4, 7, 6, 7, 6, 3, 4, 10, 4, 192) nas_score 186.947021484375 params 8.626282\n",
      "12/02/2022 02:05:13 - INFO - DENAS -   mutation 9/25 structure (14, 3.5, 3.0, 3.0, 4.0, 3.5, 3.5, 3.5, 3.5, 3.0, 3.5, 3.0, 4.0, 4.0, 3.5, 3, 5, 5, 9, 9, 4, 4, 9, 7, 6, 10, 9, 3, 7, 216) nas_score 228.6171875 params 9.867166\n",
      "12/02/2022 02:05:20 - INFO - DENAS -   mutation 10/25 structure (16, 4.0, 3.5, 3.5, 3.5, 3.0, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 3.5, 3.5, 3.5, 4.0, 3.0, 9, 5, 7, 5, 4, 7, 10, 10, 4, 4, 3, 7, 7, 7, 3, 9, 240) nas_score 218.8852081298828 params 13.243522\n",
      "12/02/2022 02:05:27 - INFO - DENAS -   mutation 11/25 structure (12, 4.0, 3.0, 3.0, 3.0, 4.0, 3.0, 3.5, 3.0, 4.0, 3.0, 4.0, 3.5, 7, 6, 9, 3, 10, 5, 3, 4, 3, 9, 5, 5, 576) nas_score 382.042724609375 params 38.114506\n",
      "12/02/2022 02:05:36 - INFO - DENAS -   mutation 12/25 structure (15, 4.0, 3.5, 3.5, 4.0, 3.5, 4.0, 4.0, 3.0, 4.0, 3.0, 4.0, 4.0, 3.0, 3.5, 3.5, 3, 9, 6, 10, 4, 5, 7, 7, 7, 10, 10, 10, 10, 9, 4, 528) nas_score 181.0482940673828 params 46.120834\n",
      "12/02/2022 02:05:42 - INFO - DENAS -   mutation 13/25 structure (13, 4.0, 3.5, 3.0, 3.0, 4.0, 4.0, 3.0, 4.0, 3.0, 4.0, 3.5, 3.0, 3.0, 5, 7, 6, 3, 5, 3, 5, 9, 3, 7, 5, 10, 7, 216) nas_score 260.8351745605469 params 8.698522\n",
      "12/02/2022 02:05:48 - INFO - DENAS -   mutation 14/25 structure (12, 4.0, 3.5, 3.0, 4.0, 3.5, 4.0, 4.0, 3.5, 4.0, 3.5, 4.0, 3.0, 10, 6, 6, 9, 9, 10, 4, 7, 7, 4, 3, 5, 448) nas_score 389.149169921875 params 27.435082\n",
      "12/02/2022 02:05:54 - INFO - DENAS -   mutation 15/25 structure (13, 3.0, 4.0, 3.5, 3.0, 3.5, 3.5, 3.5, 4.0, 3.5, 3.5, 3.0, 3.5, 3.5, 9, 9, 6, 6, 6, 9, 4, 7, 5, 4, 3, 4, 6, 240) nas_score 250.47857666015625 params 10.355626\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12/02/2022 02:06:00 - INFO - DENAS -   mutation 16/25 structure (13, 3.5, 4.0, 3.0, 3.0, 4.0, 4.0, 3.5, 3.0, 3.5, 3.5, 3.5, 4.0, 4.0, 5, 7, 3, 4, 7, 9, 5, 10, 4, 4, 3, 9, 5, 448) nas_score 369.3968505859375 params 27.87569\n",
      "12/02/2022 02:06:06 - INFO - DENAS -   mutation 17/25 structure (12, 3.0, 3.5, 4.0, 3.0, 4.0, 3.5, 4.0, 3.0, 3.5, 3.0, 3.0, 4.0, 7, 3, 10, 7, 5, 7, 6, 5, 5, 3, 7, 3, 448) nas_score 416.1216735839844 params 25.051882\n",
      "12/02/2022 02:06:12 - INFO - DENAS -   mutation 18/25 structure (12, 4.0, 3.5, 4.0, 3.0, 3.0, 4.0, 3.5, 3.0, 3.0, 4.0, 3.5, 3.0, 5, 5, 6, 3, 3, 3, 5, 9, 6, 7, 6, 6, 448) nas_score 309.3833923339844 params 24.592362\n",
      "12/02/2022 02:06:17 - INFO - DENAS -   mutation 19/25 structure (12, 3.5, 3.0, 4.0, 4.0, 3.0, 4.0, 3.0, 3.0, 3.5, 4.0, 3.5, 3.5, 3, 4, 3, 9, 3, 4, 10, 9, 7, 7, 3, 4, 216) nas_score 297.0036926269531 params 7.90957\n",
      "12/02/2022 02:06:25 - INFO - DENAS -   mutation 20/25 structure (16, 3.0, 3.0, 3.0, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 4, 7, 7, 4, 6, 7, 3, 6, 9, 9, 9, 9, 10, 10, 7, 5, 576) nas_score 269.2431335449219 params 53.141578\n",
      "12/02/2022 02:06:33 - INFO - DENAS -   mutation 21/25 structure (16, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 3.0, 3.5, 3.5, 3.0, 4.0, 3.0, 3.5, 3.5, 3.5, 3.0, 4, 9, 6, 3, 6, 10, 5, 9, 5, 7, 5, 3, 3, 9, 6, 10, 448) nas_score 313.2721252441406 params 33.792714\n",
      "12/02/2022 02:06:39 - INFO - DENAS -   mutation 22/25 structure (14, 3.0, 4.0, 3.0, 4.0, 3.5, 3.5, 3.0, 3.0, 3.5, 3.0, 3.5, 3.0, 3.5, 4.0, 3, 3, 10, 4, 7, 4, 3, 4, 6, 6, 5, 7, 7, 10, 240) nas_score 223.1712646484375 params 10.714978\n",
      "12/02/2022 02:06:45 - INFO - DENAS -   mutation 23/25 structure (14, 4.0, 3.5, 4.0, 3.5, 3.0, 3.5, 4.0, 3.5, 3.5, 4.0, 3.0, 3.5, 4.0, 3.0, 10, 3, 7, 7, 5, 6, 7, 4, 3, 10, 4, 3, 4, 6, 240) nas_score 278.6750793457031 params 11.003578\n",
      "12/02/2022 02:06:54 - INFO - DENAS -   mutation 24/25 structure (16, 4.0, 4.0, 3.5, 4.0, 3.5, 3.5, 4.0, 4.0, 3.0, 4.0, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 3, 3, 7, 4, 9, 5, 4, 10, 4, 9, 6, 6, 9, 9, 7, 5, 624) nas_score 218.985595703125 params 62.770474\n",
      "12/02/2022 02:07:00 - INFO - DENAS -   mutation 25/25 structure (13, 4.0, 4.0, 4.0, 3.5, 4.0, 3.5, 4.0, 3.5, 3.5, 4.0, 3.0, 3.0, 3.0, 10, 3, 7, 7, 5, 6, 7, 9, 4, 10, 4, 6, 10, 216) nas_score 360.4787292480469 params 9.606922\n",
      "12/02/2022 02:07:00 - INFO - DENAS -   mutation_num = 25\n",
      "12/02/2022 02:07:06 - INFO - DENAS -   crossover 1/25 structure (13, 3.0, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 4.0, 3.5, 3.5, 3.0, 3.0, 4.0, 3, 9, 3, 4, 6, 4, 4, 7, 5, 6, 4, 4, 7, 448) nas_score 238.2905731201172 params 26.038058\n",
      "12/02/2022 02:07:12 - INFO - DENAS -   crossover 2/25 structure (13, 3.5, 3.5, 4.0, 3.0, 3.0, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 3.0, 5, 7, 6, 5, 5, 7, 5, 9, 7, 7, 5, 10, 6, 320) nas_score 417.0663146972656 params 16.258218\n",
      "12/02/2022 02:07:17 - INFO - DENAS -   crossover 3/25 structure (14, 3.5, 3.5, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.5, 4.0, 4.0, 5, 5, 9, 6, 9, 6, 5, 3, 6, 9, 7, 5, 3, 5, 192) nas_score 218.207275390625 params 8.287978\n",
      "12/02/2022 02:07:24 - INFO - DENAS -   crossover 4/25 structure (14, 4.0, 4.0, 3.0, 4.0, 3.0, 3.5, 3.0, 3.5, 3.0, 3.5, 3.0, 4.0, 4.0, 4.0, 3, 5, 9, 6, 9, 6, 4, 9, 7, 9, 7, 9, 6, 3, 216) nas_score 166.65028381347656 params 10.07167\n",
      "12/02/2022 02:07:31 - INFO - DENAS -   crossover 5/25 structure (13, 4.0, 4.0, 3.5, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 10, 9, 4, 9, 4, 4, 10, 7, 6, 9, 4, 10, 4, 448) nas_score 486.32501220703125 params 29.799818\n",
      "12/02/2022 02:07:38 - INFO - DENAS -   crossover 6/25 structure (14, 3.5, 3.5, 3.5, 3.0, 4.0, 3.5, 4.0, 4.0, 4.0, 3.0, 3.5, 3.5, 4.0, 3.5, 6, 7, 9, 5, 5, 4, 6, 3, 4, 10, 3, 3, 4, 9, 528) nas_score 378.204345703125 params 39.410722\n",
      "12/02/2022 02:07:44 - INFO - DENAS -   crossover 7/25 structure (13, 3.0, 3.5, 4.0, 3.0, 4.0, 3.5, 4.0, 3.0, 3.5, 3.0, 3.0, 4.0, 3.0, 5, 3, 10, 5, 5, 7, 6, 5, 9, 7, 9, 3, 4, 448) nas_score 283.6556091308594 params 27.416618\n",
      "12/02/2022 02:07:51 - INFO - DENAS -   crossover 8/25 structure (13, 3.5, 3.5, 3.0, 3.0, 4.0, 3.5, 3.5, 4.0, 3.5, 3.5, 3.5, 3.0, 4.0, 7, 3, 10, 10, 5, 4, 6, 5, 5, 3, 7, 3, 10, 448) nas_score 251.9680938720703 params 27.818474\n",
      "12/02/2022 02:07:58 - INFO - DENAS -   crossover 9/25 structure (16, 4.0, 3.0, 3.5, 3.5, 3.5, 3.0, 4.0, 3.5, 4.0, 3.0, 3.0, 3.0, 4.0, 3.5, 3.5, 4.0, 9, 3, 3, 9, 4, 7, 7, 6, 6, 7, 3, 9, 10, 5, 5, 10, 320) nas_score 333.23223876953125 params 20.411146\n",
      "12/02/2022 02:08:04 - INFO - DENAS -   crossover 10/25 structure (14, 3.5, 4.0, 3.0, 3.5, 4.0, 3.0, 3.5, 3.5, 4.0, 3.0, 4.0, 3.0, 3.5, 3.0, 7, 10, 9, 3, 4, 4, 3, 5, 9, 3, 10, 4, 3, 3, 320) nas_score 351.113037109375 params 16.718634\n",
      "12/02/2022 02:08:09 - INFO - DENAS -   crossover 11/25 structure (13, 3.5, 3.5, 4.0, 4.0, 3.0, 4.0, 3.0, 4.0, 4.0, 4.0, 3.0, 3.5, 4.0, 5, 3, 3, 7, 5, 7, 9, 7, 3, 7, 9, 5, 4, 216) nas_score 287.04559326171875 params 8.876854\n",
      "12/02/2022 02:08:17 - INFO - DENAS -   crossover 12/25 structure (15, 3.5, 3.0, 3.5, 4.0, 3.5, 3.0, 3.5, 3.5, 4.0, 4.0, 3.0, 4.0, 3.0, 3.0, 4.0, 3, 10, 7, 10, 5, 7, 9, 10, 7, 4, 5, 3, 9, 3, 10, 448) nas_score 229.3458251953125 params 33.409322\n",
      "12/02/2022 02:08:24 - INFO - DENAS -   crossover 13/25 structure (15, 3.5, 4.0, 3.5, 3.0, 4.0, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 3.0, 4.0, 3.0, 3.5, 4, 6, 7, 3, 4, 10, 7, 10, 5, 6, 3, 4, 4, 5, 4, 320) nas_score 239.87864685058594 params 18.266954\n",
      "12/02/2022 02:08:32 - INFO - DENAS -   crossover 14/25 structure (16, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 3.5, 3.0, 4.0, 3.0, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 4, 7, 7, 4, 4, 7, 10, 10, 9, 10, 6, 9, 7, 10, 5, 5, 216) nas_score 157.80230712890625 params 11.777998\n",
      "12/02/2022 02:08:39 - INFO - DENAS -   crossover 15/25 structure (16, 3.5, 3.0, 4.0, 3.5, 4.0, 3.0, 3.0, 3.5, 3.5, 3.5, 4.0, 4.0, 3.5, 4.0, 3.5, 3.5, 10, 9, 7, 10, 7, 9, 5, 3, 9, 3, 5, 9, 3, 7, 3, 4, 384) nas_score 302.71661376953125 params 27.51265\n",
      "12/02/2022 02:08:46 - INFO - DENAS -   crossover 16/25 structure (13, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 3.5, 6, 9, 6, 9, 6, 4, 5, 3, 5, 3, 4, 3, 3, 448) nas_score 349.4648742675781 params 26.238986\n",
      "12/02/2022 02:08:55 - INFO - DENAS -   crossover 17/25 structure (16, 4.0, 3.0, 3.0, 3.5, 4.0, 4.0, 3.0, 3.5, 4.0, 3.5, 3.5, 4.0, 3.0, 4.0, 3.5, 3.5, 10, 9, 5, 4, 7, 3, 4, 9, 9, 3, 5, 9, 3, 7, 3, 3, 576) nas_score 382.349609375 params 52.32865\n",
      "12/02/2022 02:09:01 - INFO - DENAS -   crossover 18/25 structure (14, 3.5, 4.0, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 3.0, 3.5, 3.5, 3.5, 4.0, 5, 9, 9, 9, 6, 5, 3, 6, 9, 3, 7, 10, 9, 9, 320) nas_score 305.6822509765625 params 18.730218\n",
      "12/02/2022 02:09:08 - INFO - DENAS -   crossover 19/25 structure (13, 3.0, 3.5, 4.0, 3.5, 4.0, 3.5, 4.0, 4.0, 3.5, 3.0, 3.5, 3.5, 3.0, 7, 3, 10, 5, 5, 7, 6, 5, 5, 10, 7, 3, 4, 448) nas_score 415.7505798339844 params 27.904522\n",
      "12/02/2022 02:09:15 - INFO - DENAS -   crossover 20/25 structure (13, 3.5, 3.0, 3.5, 3.0, 3.0, 3.5, 4.0, 3.0, 3.5, 4.0, 3.0, 3.0, 3.5, 6, 9, 6, 3, 9, 10, 5, 9, 10, 3, 5, 7, 3, 448) nas_score 285.16448974609375 params 27.818922\n",
      "12/02/2022 02:09:21 - INFO - DENAS -   crossover 21/25 structure (13, 4.0, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 4.0, 4.0, 3.0, 3.5, 3.5, 4.0, 9, 9, 4, 6, 3, 5, 4, 5, 5, 10, 4, 3, 4, 448) nas_score 458.79364013671875 params 27.41617\n",
      "12/02/2022 02:09:28 - INFO - DENAS -   crossover 22/25 structure (15, 3.5, 3.0, 3.5, 4.0, 3.5, 3.0, 3.5, 3.5, 4.0, 3.0, 3.0, 4.0, 3.0, 4.0, 3.0, 6, 10, 4, 10, 5, 7, 9, 6, 7, 4, 5, 3, 5, 3, 10, 384) nas_score 280.4824523925781 params 24.992074\n",
      "12/02/2022 02:09:36 - INFO - DENAS -   crossover 23/25 structure (13, 4.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 3.5, 3.0, 3.0, 4.0, 5, 7, 7, 6, 9, 5, 3, 5, 10, 3, 4, 7, 4, 576) nas_score 317.3668212890625 params 42.000106\n",
      "12/02/2022 02:09:43 - INFO - DENAS -   crossover 24/25 structure (13, 3.0, 3.5, 4.0, 4.0, 3.0, 3.5, 4.0, 4.0, 3.5, 3.0, 3.0, 3.5, 3.0, 7, 3, 10, 7, 5, 4, 6, 9, 5, 7, 9, 5, 4, 448) nas_score 453.1164245605469 params 27.962186\n",
      "12/02/2022 02:09:50 - INFO - DENAS -   crossover 25/25 structure (13, 3.0, 3.5, 4.0, 3.0, 3.0, 4.0, 4.0, 3.5, 3.5, 4.0, 3.0, 3.0, 4.0, 5, 3, 3, 7, 5, 4, 6, 9, 7, 6, 9, 4, 7, 528) nas_score 263.1604309082031 params 36.203314\n",
      "12/02/2022 02:09:50 - INFO - DENAS -   crossover_num = 25\n",
      "DE-NAS search best structure took 668.1732305698097 sec\n",
      "12/02/2022 02:09:50 - INFO - DENAS -   best structure (13, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 3.0, 3.0, 3.5, 3.0, 4.0, 3.0, 3.5, 6, 9, 6, 3, 6, 10, 5, 9, 5, 3, 5, 3, 3, 448) nas_score 527.647216796875 params 26.440362\n",
      "DE-NAS completed, best structure is (13, 3.5, 4.0, 3.5, 3.0, 3.0, 3.5, 3.0, 3.0, 3.5, 3.0, 4.0, 3.0, 3.5, 6, 9, 6, 3, 6, 10, 5, 9, 5, 3, 5, 3, 3, 448)\n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.DeNas.search.utils import parse_config\n",
    "from e2eAIOK.DeNas.search.SearchEngineFactory import SearchEngineFactory\n",
    "from e2eAIOK.DeNas.cv.supernet_transformer import Vision_TransformerSuper\n",
    "\n",
    "\n",
    "params = parse_config('/home/vmagent/app/e2eaiok/conf/denas/cv/e2eaiok_denas_vit.conf')\n",
    "\n",
    "# construct supernet and search space\n",
    "super_net = Vision_TransformerSuper(img_size=params.img_size,\n",
    "                                    patch_size=params.patch_size,\n",
    "                                    embed_dim=params.SUPERNET.EMBED_DIM, depth=params.SUPERNET.DEPTH,\n",
    "                                    num_heads=params.SUPERNET.NUM_HEADS,mlp_ratio=params.SUPERNET.MLP_RATIO,\n",
    "                                    qkv_bias=True, drop_rate=params.drop_rate,\n",
    "                                    drop_path_rate=params.drop_path_rate,\n",
    "                                    gp=params.gp,\n",
    "                                    num_classes=params.num_classes,\n",
    "                                    max_relative_position=params.max_relative_position,\n",
    "                                    relative_position=params.relative_position,\n",
    "                                    change_qkv=params.change_qkv, abs_pos=params.abs_pos)\n",
    "search_space = {'num_heads': params.SEARCH_SPACE.NUM_HEADS, 'mlp_ratio': params.SEARCH_SPACE.MLP_RATIO, 'embed_dim': params.SEARCH_SPACE.EMBED_DIM , 'depth': params.SEARCH_SPACE.DEPTH}\n",
    "\n",
    "# create DE-NAS searcher and trigger the search process\n",
    "searcher = SearchEngineFactory.create_search_engine(params = params, super_net = super_net, search_space = search_space)\n",
    "\n",
    "# trigger the search process\n",
    "searcher.search()\n",
    "best_structure = searcher.get_best_structures()\n",
    "print(f\"DE-NAS completed, best structure is {best_structure}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e465cdeb",
   "metadata": {},
   "source": [
    "## 5. Launch Training with Best Searched Model Structure"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ad47332c",
   "metadata": {},
   "source": [
    "### Train the best searched ViT model\n",
    "The input is the configuration for VIT domain best model train, edit the configuration file of `e2eaiok_denas_train_vit.conf` for VIT best model train.\n",
    "\n",
    "Noted: Below performance result is using sample dataset and small iterations to demonstrate its function. The actual performance result please refers to the [performance section](#performance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fa21d00f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/runpy.py:127: RuntimeWarning: 'intel_extension_for_pytorch.cpu.launch' found in sys.modules after import of package 'intel_extension_for_pytorch.cpu', but prior to execution of 'intel_extension_for_pytorch.cpu.launch'; this may result in unpredictable behaviour\n",
      "  warn(RuntimeWarning(msg))\n",
      "2022-12-02 02:33:35,564 - __main__ - INFO - MASTER_ADDR=127.0.0.1\n",
      "2022-12-02 02:33:35,564 - __main__ - INFO - MASTER_PORT=29500\n",
      "2022-12-02 02:33:35,565 - __main__ - INFO - I_MPI_PIN_DOMAIN=[0xfffff0,0xfffff0000000,]\n",
      "2022-12-02 02:33:35,565 - __main__ - WARNING - Both TCMalloc and JeMalloc are not found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /root/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance\n",
      "2022-12-02 02:33:35,565 - __main__ - INFO - OMP_NUM_THREADS=20\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - Using Intel OpenMP\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - KMP_BLOCKTIME=1\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - LD_PRELOAD=/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/libiomp5.so\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - CCL_WORKER_COUNT=4\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - CCL_WORKER_AFFINITY=0,1,2,3,24,25,26,27\n",
      "2022-12-02 02:33:35,566 - __main__ - INFO - ['mpiexec.hydra', '-l', '-np', '2', '-ppn', '2', '-genv', 'I_MPI_PIN_DOMAIN=[0xfffff0,0xfffff0000000,]', '-genv', 'OMP_NUM_THREADS=20', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/bin/python', '-u', '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/train.py', '--domain', 'vit', '--conf', '/home/vmagent/app/e2eaiok/e2eAIOK/conf/denas/cv/e2eaiok_denas_train_vit.conf']\n",
      "[1] world_size:2,rank:1\n",
      "[1] extend_distributed backend:gloo\n",
      "[0] world_size:2,rank:0\n",
      "[0] extend_distributed backend:gloo\n",
      "[0] 12/02/2022 02:33:37 - INFO - torch.distributed.distributed_c10d -   Added key: store_based_barrier_key:1 to store for rank: 0\n",
      "[1] 12/02/2022 02:33:37 - INFO - torch.distributed.distributed_c10d -   Added key: store_based_barrier_key:1 to store for rank: 1\n",
      "[0] 12/02/2022 02:33:37 - INFO - torch.distributed.distributed_c10d -   Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.\n",
      "[1] 12/02/2022 02:33:37 - INFO - torch.distributed.distributed_c10d -   Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.\n",
      "[0] Running on 2 ranks using gloo backend\n",
      "[0] 12/02/2022 02:33:37 - INFO - Trainer -   building model\n",
      "[1] 12/02/2022 02:33:37 - INFO - Trainer -   building model\n",
      "[1] model parameters size: 26354346\n",
      "[1] 12/02/2022 02:33:38 - INFO - Trainer -   model created: Vision_TransformerSuper(\n",
      "[1]   (patch_embed_super): PatchembedSuper(\n",
      "[1]     (proj): Conv2d(3, 640, kernel_size=(16, 16), stride=(16, 16))\n",
      "[1]   )\n",
      "[1]   (blocks): ModuleList(\n",
      "[1]     (0): TransformerEncoderLayer(\n",
      "[1]       (drop_path): Identity()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (1): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (2): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (3): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (4): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (5): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (6): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (7): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (8): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (9): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (10): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (11): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (12): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (13): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (14): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (15): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]   )\n",
      "[1]   (norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]   (head): LinearSuper(in_features=640, out_features=10, bias=True)\n",
      "[1] )\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] model parameters size: 26354346\r\n",
      "[0] 12/02/2022 02:33:39 - INFO - Trainer -   model created: Vision_TransformerSuper(\r\n",
      "[0]   (patch_embed_super): PatchembedSuper(\r\n",
      "[0]     (proj): Conv2d(3, 640, kernel_size=(16, 16), stride=(16, 16))\r\n",
      "[0]   )\r\n",
      "[0]   (blocks): ModuleList(\r\n",
      "[0]     (0): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): Identity()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (1): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (2): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (3): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (4): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (5): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (6): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (7): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (8): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (9): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (10): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (11): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (12): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (13): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (14): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (15): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]   )\r\n",
      "[0]   (norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]   (head): LinearSuper(in_features=640, out_features=10, bias=True)\r\n",
      "[0] )\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] Files already downloaded and verified\n",
      "[0] Files already downloaded and verified\n",
      "[1] Files already downloaded and verified\n",
      "[0] Files already downloaded and verified\n",
      "[1] /opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 10 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "[1]   warnings.warn(_create_warning_msg(\n",
      "[1] model:Vision_TransformerSuper(\n",
      "[1]   (patch_embed_super): PatchembedSuper(\n",
      "[1]     (proj): Conv2d(3, 640, kernel_size=(16, 16), stride=(16, 16))\n",
      "[1]   )\n",
      "[1]   (blocks): ModuleList(\n",
      "[1]     (0): TransformerEncoderLayer(\n",
      "[1]       (drop_path): Identity()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (1): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (2): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (3): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (4): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (5): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (6): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (7): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (8): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (9): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (10): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (11): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (12): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (13): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (14): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]     (15): TransformerEncoderLayer(\n",
      "[1]       (drop_path): DropPath()\n",
      "[1]       (attn): AttentionSuper(\n",
      "[1]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\n",
      "[1]         (rel_pos_embed_k): RelativePosition2D_super()\n",
      "[1]         (rel_pos_embed_v): RelativePosition2D_super()\n",
      "[1]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\n",
      "[1]         (attn_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]         (proj_drop): Dropout(p=0.0, inplace=False)\n",
      "[1]       )\n",
      "[1]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\n",
      "[1]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\n",
      "[1]     )\n",
      "[1]   )\n",
      "[1]   (norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\n",
      "[1]   (head): LinearSuper(in_features=640, out_features=10, bias=True)\n",
      "[1] )\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] /opt/intel/oneapi/intelpython/latest/envs/pytorch-1.10.0/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 10 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\r\n",
      "[0]   warnings.warn(_create_warning_msg(\r\n",
      "[0] model:Vision_TransformerSuper(\r\n",
      "[0]   (patch_embed_super): PatchembedSuper(\r\n",
      "[0]     (proj): Conv2d(3, 640, kernel_size=(16, 16), stride=(16, 16))\r\n",
      "[0]   )\r\n",
      "[0]   (blocks): ModuleList(\r\n",
      "[0]     (0): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): Identity()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (1): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (2): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (3): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (4): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (5): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (6): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (7): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (8): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (9): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (10): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (11): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (12): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (13): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (14): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]     (15): TransformerEncoderLayer(\r\n",
      "[0]       (drop_path): DropPath()\r\n",
      "[0]       (attn): AttentionSuper(\r\n",
      "[0]         (qkv): qkv_super(in_features=640, out_features=1920, bias=True)\r\n",
      "[0]         (rel_pos_embed_k): RelativePosition2D_super()\r\n",
      "[0]         (rel_pos_embed_v): RelativePosition2D_super()\r\n",
      "[0]         (proj): LinearSuper(in_features=640, out_features=640, bias=True)\r\n",
      "[0]         (attn_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]         (proj_drop): Dropout(p=0.0, inplace=False)\r\n",
      "[0]       )\r\n",
      "[0]       (attn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (ffn_layer_norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]       (fc1): LinearSuper(in_features=640, out_features=2560, bias=True)\r\n",
      "[0]       (fc2): LinearSuper(in_features=2560, out_features=640, bias=True)\r\n",
      "[0]     )\r\n",
      "[0]   )\r\n",
      "[0]   (norm): LayerNormSuper((640,), eps=1e-05, elementwise_affine=True)\r\n",
      "[0]   (head): LinearSuper(in_features=640, out_features=10, bias=True)\r\n",
      "[0] )\r\n",
      "[0] 12/02/2022 02:33:40 - INFO - Trainer -   Trainer config: {'domain': 'vit', 'train_epochs': 1, 'eval_epochs': 1, 'input_size': 32, 'best_model_structure': '/home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/best_vit_model_structure.txt', 'num_classes': 10, 'dist_backend': 'gloo', 'train_batch_size': 128, 'eval_batch_size': 128, 'data_path': '~/data/pytorch_cifar10', 'data_set': 'CIFAR10', 'output_dir': './', 'num_workers': 10, 'pin_mem': True, 'eval_metric': 'accuracy', 'learning_rate': 0.001, 'momentum': 0.9, 'weight_decay': 0.01, 'optimizer': 'SGD', 'criterion': 'CrossEntropyLoss', 'lr_scheduler': 'CosineAnnealingLR', 'print_freq': 10, 'mode': 'train', 'gp': True, 'change_qkv': True, 'relative_position': True, 'drop_path': 0.1, 'max_relative_position': 14, 'no_abs_pos': False, 'patch_size': 16, 'drop': 0.0, 'metric_threshold': 94, 'SUPERNET': {'MLP_RATIO': 4.0, 'NUM_HEADS': 10, 'EMBED_DIM': 640, 'DEPTH': 16}}\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] /home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/module/cv/multihead_super.py:51: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[1]   distance_mat_v = (range_vec_k[None, :] // int(length_q ** 0.5 )  - range_vec_q[:, None] // int(length_q ** 0.5 ))\n",
      "[0] /home/vmagent/app/e2eaiok/e2eAIOK/e2eAIOK/DeNas/module/cv/multihead_super.py:51: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').\n",
      "[0]   distance_mat_v = (range_vec_k[None, :] // int(length_q ** 0.5 )  - range_vec_q[:, None] // int(length_q ** 0.5 ))\n",
      "[1] Epoch: [1]  [  0/196]  eta: 0:22:23  lr: 0.001000  loss: 2.3453 (2.3453)  time: 6.8559  data: 5.2393\n",
      "[0] Epoch: [1]  [  0/196]  eta: 0:22:25  lr: 0.001000  loss: 2.3404 (2.3404)  time: 6.8649  data: 5.5334\n",
      "[1] Epoch: [1]  [ 10/196]  eta: 0:03:40  lr: 0.001000  loss: 2.2744 (2.2759)  time: 1.1846  data: 0.4948\n",
      "[0] Epoch: [1]  [ 10/196]  eta: 0:03:40  lr: 0.001000  loss: 2.2699 (2.2903)  time: 1.1846  data: 0.5214\n",
      "[1] Epoch: [1]  [ 20/196]  eta: 0:02:40  lr: 0.001000  loss: 2.2375 (2.2317)  time: 0.6148  data: 0.0201\n",
      "[0] Epoch: [1]  [ 20/196]  eta: 0:02:40  lr: 0.001000  loss: 2.2439 (2.2509)  time: 0.6144  data: 0.0201\n",
      "[1] Epoch: [1]  [ 30/196]  eta: 0:02:16  lr: 0.001000  loss: 2.1719 (2.2012)  time: 0.6193  data: 0.0195\n",
      "[0] Epoch: [1]  [ 30/196]  eta: 0:02:16  lr: 0.001000  loss: 2.1828 (2.2145)  time: 0.6193  data: 0.0195\n",
      "[0] Epoch: [1]  [ 40/196]  eta: 0:02:00  lr: 0.001000  loss: 2.1048 (2.1862)  time: 0.6225  data: 0.0198\n",
      "[1] Epoch: [1]  [ 40/196]  eta: 0:02:00  lr: 0.001000  loss: 2.1013 (2.1716)  time: 0.6225  data: 0.0196\n",
      "[0] Epoch: [1]  [ 50/196]  eta: 0:01:47  lr: 0.001000  loss: 2.0699 (2.1583)  time: 0.6146  data: 0.0206\n",
      "[1] Epoch: [1]  [ 50/196]  eta: 0:01:47  lr: 0.001000  loss: 2.0606 (2.1510)  time: 0.6148  data: 0.0205\n",
      "[0] Epoch: [1]  [ 60/196]  eta: 0:01:37  lr: 0.001000  loss: 2.0355 (2.1403)  time: 0.6101  data: 0.0208\n",
      "[1] Epoch: [1]  [ 60/196]  eta: 0:01:37  lr: 0.001000  loss: 2.0682 (2.1372)  time: 0.6101  data: 0.0205\n",
      "[1] Epoch: [1]  [ 70/196]  eta: 0:01:28  lr: 0.001000  loss: 2.0569 (2.1236)  time: 0.6105  data: 0.0205\n",
      "[0] Epoch: [1]  [ 70/196]  eta: 0:01:28  lr: 0.001000  loss: 2.0355 (2.1270)  time: 0.6107  data: 0.0208\n",
      "[1] Epoch: [1]  [ 80/196]  eta: 0:01:20  lr: 0.001000  loss: 2.0321 (2.1118)  time: 0.6104  data: 0.0205\n",
      "[0] Epoch: [1]  [ 80/196]  eta: 0:01:20  lr: 0.001000  loss: 2.0246 (2.1113)  time: 0.6105  data: 0.0207\n",
      "[0] Epoch: [1]  [ 90/196]  eta: 0:01:12  lr: 0.001000  loss: 2.0281 (2.1009)  time: 0.6119  data: 0.0206\n",
      "[1] Epoch: [1]  [ 90/196]  eta: 0:01:12  lr: 0.001000  loss: 2.0311 (2.1013)  time: 0.6121  data: 0.0205\n",
      "[0] Epoch: [1]  [100/196]  eta: 0:01:04  lr: 0.001000  loss: 1.9885 (2.0883)  time: 0.6158  data: 0.0203\n",
      "[1] Epoch: [1]  [100/196]  eta: 0:01:04  lr: 0.001000  loss: 2.0190 (2.0952)  time: 0.6159  data: 0.0205\n",
      "[0] Epoch: [1]  [110/196]  eta: 0:00:57  lr: 0.001000  loss: 1.9779 (2.0807)  time: 0.6131  data: 0.0205\n",
      "[1] Epoch: [1]  [110/196]  eta: 0:00:57  lr: 0.001000  loss: 2.0033 (2.0861)  time: 0.6131  data: 0.0204\n",
      "[1] Epoch: [1]  [120/196]  eta: 0:00:50  lr: 0.001000  loss: 1.9823 (2.0782)  time: 0.6077  data: 0.0206\n",
      "[0] Epoch: [1]  [120/196]  eta: 0:00:50  lr: 0.001000  loss: 1.9698 (2.0719)  time: 0.6078  data: 0.0208\n",
      "[0] Epoch: [1]  [130/196]  eta: 0:00:43  lr: 0.001000  loss: 1.9880 (2.0671)  time: 0.6065  data: 0.0209\n",
      "[1] Epoch: [1]  [130/196]  eta: 0:00:43  lr: 0.001000  loss: 1.9940 (2.0716)  time: 0.6064  data: 0.0206\n",
      "[0] Epoch: [1]  [140/196]  eta: 0:00:36  lr: 0.001000  loss: 1.9928 (2.0616)  time: 0.6074  data: 0.0207\n",
      "[1] Epoch: [1]  [140/196]  eta: 0:00:36  lr: 0.001000  loss: 1.9940 (2.0647)  time: 0.6076  data: 0.0204\n",
      "[1] Epoch: [1]  [150/196]  eta: 0:00:30  lr: 0.001000  loss: 1.9840 (2.0581)  time: 0.6088  data: 0.0203\n",
      "[0] Epoch: [1]  [150/196]  eta: 0:00:30  lr: 0.001000  loss: 1.9690 (2.0557)  time: 0.6088  data: 0.0205\n",
      "[1] Epoch: [1]  [160/196]  eta: 0:00:23  lr: 0.001000  loss: 1.9483 (2.0502)  time: 0.6095  data: 0.0204\n",
      "[0] Epoch: [1]  [160/196]  eta: 0:00:23  lr: 0.001000  loss: 1.9392 (2.0492)  time: 0.6097  data: 0.0207\n",
      "[1] Epoch: [1]  [170/196]  eta: 0:00:16  lr: 0.001000  loss: 1.9038 (2.0421)  time: 0.6101  data: 0.0208\n",
      "[0] Epoch: [1]  [170/196]  eta: 0:00:16  lr: 0.001000  loss: 1.9411 (2.0448)  time: 0.6102  data: 0.0204\n",
      "[1] Epoch: [1]  [180/196]  eta: 0:00:10  lr: 0.001000  loss: 1.9270 (2.0374)  time: 0.5998  data: 0.0154\n",
      "[0] Epoch: [1]  [180/196]  eta: 0:00:10  lr: 0.001000  loss: 1.9541 (2.0395)  time: 0.5998  data: 0.0151\n",
      "[0] Epoch: [1]  [190/196]  eta: 0:00:03  lr: 0.001000  loss: 1.9553 (2.0360)  time: 0.5795  data: 0.0057\n",
      "[1] Epoch: [1]  [190/196]  eta: 0:00:03  lr: 0.001000  loss: 1.9462 (2.0340)  time: 0.5797  data: 0.0055\n",
      "[0] Epoch: [1]  [195/196]  eta: 0:00:00  lr: 0.001000  loss: 1.9601 (2.0330)  time: 0.5679  data: 0.0012\n",
      "[1] Epoch: [1]  [195/196]  eta: 0:00:00  lr: 0.001000  loss: 1.9282 (2.0337)  time: 0.5682  data: 0.0012\n",
      "[0] Epoch: [1] Total time: 0:02:05 (0.6397 s / it)\n",
      "[1] Epoch: [1] Total time: 0:02:05 (0.6397 s / it)\n",
      "[1] Test:[ 0/40]eta: 0:04:33loss: 1.9061 (1.9061)acc1: 30.4688 (30.4688)time: 6.8387data: 6.3137\n",
      "[0] Test:[ 0/40]eta: 0:04:35loss: 1.8868 (1.8868)acc1: 39.0625 (39.0625)time: 6.8877data: 6.3939\n",
      "[1] Test:[10/40]eta: 0:00:21loss: 1.8835 (1.8958)acc1: 32.0312 (31.4631)time: 0.7118data: 0.5911\n",
      "[0] Test:[10/40]eta: 0:00:21loss: 1.9169 (1.9207)acc1: 31.2500 (31.2500)time: 0.7185data: 0.5984\n",
      "[1] Test:[20/40]eta: 0:00:08loss: 1.8858 (1.9093)acc1: 31.2500 (30.4315)time: 0.0868data: 0.0172\n",
      "[0] Test:[20/40]eta: 0:00:08loss: 1.9235 (1.9227)acc1: 29.6875 (30.4315)time: 0.0894data: 0.0172\n",
      "[1] Test:[30/40]eta: 0:00:02loss: 1.9383 (1.9217)acc1: 28.9062 (30.2923)time: 0.0666data: 0.0083\n",
      "[0] Test:[30/40]eta: 0:00:02loss: 1.9221 (1.9187)acc1: 28.1250 (30.1411)time: 0.0689data: 0.0083\n",
      "[1] Test:[39/40]eta: 0:00:00loss: 1.9176 (1.9159)acc1: 30.4688 (30.3906)time: 0.0576data: 0.0010\n",
      "[1] Test: Total time: 0:00:09 (0.2445 s / it)\n",
      "[0] Test:[39/40]eta: 0:00:00loss: 1.9178 (1.9141)acc1: 27.3438 (30.0000)time: 0.0596data: 0.0010\n",
      "[0] Test: Total time: 0:00:09 (0.2481 s / it)\n",
      "[0] * Acc@1 30.195 loss 1.915\n",
      "[1] * Acc@1 30.195 loss 1.915\n",
      "[0] 12/02/2022 02:35:56 - INFO - Trainer -   Evaluate time:10.769713401794434\n",
      "[0] 12/02/2022 02:35:56 - INFO - Trainer -   Epoch 1 training time:136.14972710609436\n",
      "[0] 12/02/2022 02:35:56 - INFO - Trainer -   Total time:136.1497721672058\n",
      "[0] 12/02/2022 02:35:56 - INFO - Trainer -   Trainer complete\n"
     ]
    }
   ],
   "source": [
    "import e2eAIOK.common.trainer.utils.utils as utils\n",
    "from e2eAIOK.DeNas.cv.model_builder_denas_cv import ModelBuilderCVDeNas\n",
    "from e2eAIOK.common.trainer.data.cv.data_builder_cifar import DataBuilderCIFAR\n",
    "from e2eAIOK.common.trainer.data.cv.data_builder_imagenet import DataBuilderImageNet\n",
    "from e2eAIOK.DeNas.cv.cv_trainer import CVTrainer\n",
    "from e2eAIOK.DeNas.search.utils import parse_config\n",
    "\n",
    "# parse DE-NAS train configure\n",
    "cfg = parse_config(\"/home/vmagent/app/e2eaiok/conf/denas/cv/e2eaiok_denas_train_vit.conf\")\n",
    "\n",
    "# construct model, dataloader, optimizer, criterion, scheduler and metric\n",
    "model = ModelBuilderCVDeNas(cfg).create_model()\n",
    "train_dataloader, eval_dataloader = (DataBuilderImageNet(cfg) if cfg.data_set == 'ImageNet' else DataBuilderCIFAR(cfg)).get_dataloader()\n",
    "optimizer = utils.create_optimizer(model, cfg)\n",
    "criterion = utils.create_criterion(cfg)\n",
    "scheduler = utils.create_scheduler(optimizer, cfg)\n",
    "metric = utils.create_metric(cfg)\n",
    "\n",
    "# create DE-NAS trainer\n",
    "trainer = CVTrainer(cfg, model, train_dataloader, eval_dataloader, optimizer, criterion, scheduler, metric)\n",
    "\n",
    "# trigger the training process\n",
    "trainer.fit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "denas",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Jun  1 2022, 11:38:51) \n[GCC 7.5.0]"
  },
  "vscode": {
   "interpreter": {
    "hash": "960de6b0d90e8c1b5b9252244488628d965641900559c86d48ac5737e2de9daf"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}