Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Reacher.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Size of each action: 4\n",
      "There are 1 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726671e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=False)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from ddpg_agent import Agent # Agent taken from https://github.com/udacity/deep-reinforcement-learning/blob/master/ddpg-pendulum/ddpg_agent.py\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 1.1099999751895666\n",
      "Episode 1\tAverage Score: 1.11\n",
      "--- 14.468080282211304 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.30999999307096004\n",
      "Episode 2\tAverage Score: 0.71\n",
      "--- 18.227809190750122 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.8499999810010195\n",
      "Episode 3\tAverage Score: 0.76\n",
      "--- 18.22128129005432 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.1099999751895666\n",
      "Episode 4\tAverage Score: 0.84\n",
      "--- 18.343459844589233 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.5099999886006117\n",
      "Episode 5\tAverage Score: 0.78\n",
      "--- 18.394530534744263 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.7499999608844519\n",
      "Episode 6\tAverage Score: 0.94\n",
      "--- 18.441455841064453 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.35999999195337296\n",
      "Episode 7\tAverage Score: 0.86\n",
      "--- 18.310797691345215 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.4999999888241291\n",
      "Episode 8\tAverage Score: 0.81\n",
      "--- 18.53895354270935 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.1299999747425318\n",
      "Episode 9\tAverage Score: 0.85\n",
      "--- 18.07156276702881 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.8299999814480543\n",
      "Episode 10\tAverage Score: 0.85\n",
      "--- 18.10022783279419 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.1399999298155308\n",
      "Episode 11\tAverage Score: 1.05\n",
      "--- 17.425050020217896 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.0899999756366014\n",
      "Episode 12\tAverage Score: 1.06\n",
      "--- 18.487801790237427 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.8699999805539846\n",
      "Episode 13\tAverage Score: 1.04\n",
      "--- 18.641813039779663 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.6799999848008156\n",
      "Episode 14\tAverage Score: 1.02\n",
      "--- 17.447125673294067 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.319999948143959\n",
      "Episode 15\tAverage Score: 1.10\n",
      "--- 18.34191393852234 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.3499999921768904\n",
      "Episode 16\tAverage Score: 1.06\n",
      "--- 18.504340410232544 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.9199999794363976\n",
      "Episode 17\tAverage Score: 1.05\n",
      "--- 18.697978496551514 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.339999947696924\n",
      "Episode 18\tAverage Score: 1.12\n",
      "--- 17.84971594810486 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.9299999568611383\n",
      "Episode 19\tAverage Score: 1.16\n",
      "--- 18.673616647720337 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.389999968931079\n",
      "Episode 20\tAverage Score: 1.17\n",
      "--- 18.54567837715149 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.909999979659915\n",
      "Episode 21\tAverage Score: 1.16\n",
      "--- 18.858104944229126 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.9999999329447746\n",
      "Episode 22\tAverage Score: 1.25\n",
      "--- 18.77624249458313 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.7499999608844519\n",
      "Episode 23\tAverage Score: 1.27\n",
      "--- 18.740246534347534 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.279999904334545\n",
      "Episode 24\tAverage Score: 1.39\n",
      "--- 18.197457551956177 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.299999948590994\n",
      "Episode 25\tAverage Score: 1.43\n",
      "--- 18.488171339035034 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.1899999734014273\n",
      "Episode 26\tAverage Score: 1.42\n",
      "--- 18.50605058670044 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.7999999597668648\n",
      "Episode 27\tAverage Score: 1.43\n",
      "--- 18.830175161361694 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.1199999749660492\n",
      "Episode 28\tAverage Score: 1.42\n",
      "--- 18.850664377212524 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.8899999354034662\n",
      "Episode 29\tAverage Score: 1.47\n",
      "--- 18.759321928024292 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.009999955072999\n",
      "Episode 30\tAverage Score: 1.49\n",
      "--- 17.914933681488037 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.1299999747425318\n",
      "Episode 31\tAverage Score: 1.48\n",
      "--- 18.843690395355225 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.459999945014715\n",
      "Episode 32\tAverage Score: 1.51\n",
      "--- 18.795750617980957 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.759999915957451\n",
      "Episode 33\tAverage Score: 1.58\n",
      "--- 18.726770639419556 seconds ---\n",
      "Total score (averaged over agents) this episode: 0.46999998949468136\n",
      "Episode 34\tAverage Score: 1.55\n",
      "--- 18.995666980743408 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.9799999333918095\n",
      "Episode 35\tAverage Score: 1.59\n",
      "--- 19.037928342819214 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.7799999602138996\n",
      "Episode 36\tAverage Score: 1.59\n",
      "--- 19.041298151016235 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.50999996624887\n",
      "Episode 37\tAverage Score: 1.59\n",
      "--- 18.832498788833618 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.8799999579787254\n",
      "Episode 38\tAverage Score: 1.60\n",
      "--- 18.94263505935669 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.889999957755208\n",
      "Episode 39\tAverage Score: 1.61\n",
      "--- 19.003694534301758 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.5999999418854713\n",
      "Episode 40\tAverage Score: 1.63\n",
      "--- 19.027209043502808 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.0099999774247408\n",
      "Episode 41\tAverage Score: 1.61\n",
      "--- 18.832478284835815 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.879999913275242\n",
      "Episode 42\tAverage Score: 1.67\n",
      "--- 18.8283429145813 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.559999942779541\n",
      "Episode 43\tAverage Score: 1.69\n",
      "--- 18.959089994430542 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.3299999255687\n",
      "Episode 44\tAverage Score: 1.73\n",
      "--- 18.13172674179077 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.0899999756366014\n",
      "Episode 45\tAverage Score: 1.71\n",
      "--- 17.177372932434082 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.4999999217689037\n",
      "Episode 46\tAverage Score: 1.75\n",
      "--- 18.87566041946411 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.629999941214919\n",
      "Episode 47\tAverage Score: 1.77\n",
      "--- 18.93104124069214 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.5499999206513166\n",
      "Episode 48\tAverage Score: 1.81\n",
      "--- 18.97263765335083 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.389999879524112\n",
      "Episode 49\tAverage Score: 1.88\n",
      "--- 18.987049341201782 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.699999939650297\n",
      "Episode 50\tAverage Score: 1.90\n",
      "--- 19.030738592147827 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.7699999380856752\n",
      "Episode 51\tAverage Score: 1.91\n",
      "--- 18.99098300933838 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.4299999456852674\n",
      "Episode 52\tAverage Score: 1.92\n",
      "--- 18.929575204849243 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.799999915063381\n",
      "Episode 53\tAverage Score: 1.96\n",
      "--- 18.268283128738403 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.709999917075038\n",
      "Episode 54\tAverage Score: 1.99\n",
      "--- 19.082947492599487 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.819999936968088\n",
      "Episode 55\tAverage Score: 2.01\n",
      "--- 19.23667287826538 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.469999900087714\n",
      "Episode 56\tAverage Score: 2.05\n",
      "--- 19.153364419937134 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.549999898299575\n",
      "Episode 57\tAverage Score: 2.09\n",
      "--- 17.268362283706665 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.1999999284744263\n",
      "Episode 58\tAverage Score: 2.11\n",
      "--- 18.018298149108887 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.379999902099371\n",
      "Episode 59\tAverage Score: 2.15\n",
      "--- 19.152935028076172 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.759999871253967\n",
      "Episode 60\tAverage Score: 2.21\n",
      "--- 18.998114347457886 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.499999899417162\n",
      "Episode 61\tAverage Score: 2.25\n",
      "--- 19.004183053970337 seconds ---\n",
      "Total score (averaged over agents) this episode: 1.3499999698251486\n",
      "Episode 62\tAverage Score: 2.24\n",
      "--- 19.08118772506714 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 3.659999918192625\n",
      "Episode 63\tAverage Score: 2.26\n",
      "--- 17.81393551826477 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.8599999360740185\n",
      "Episode 64\tAverage Score: 2.27\n",
      "--- 19.188938856124878 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.7399999164044857\n",
      "Episode 65\tAverage Score: 2.29\n",
      "--- 17.98105549812317 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.659999918192625\n",
      "Episode 66\tAverage Score: 2.31\n",
      "--- 17.007182598114014 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.829999892041087\n",
      "Episode 67\tAverage Score: 2.35\n",
      "--- 19.158616304397583 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.919999822974205\n",
      "Episode 68\tAverage Score: 2.43\n",
      "--- 19.11681318283081 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.749999849125743\n",
      "Episode 69\tAverage Score: 2.49\n",
      "--- 19.143723011016846 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.169999906793237\n",
      "Episode 70\tAverage Score: 2.52\n",
      "--- 19.207127571105957 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.569999920204282\n",
      "Episode 71\tAverage Score: 2.53\n",
      "--- 19.223772287368774 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.419999901205301\n",
      "Episode 72\tAverage Score: 2.56\n",
      "--- 19.247843265533447 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.8099999148398638\n",
      "Episode 73\tAverage Score: 2.57\n",
      "--- 19.087318420410156 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.8399999141693115\n",
      "Episode 74\tAverage Score: 2.59\n",
      "--- 19.10648250579834 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.29999990388751\n",
      "Episode 75\tAverage Score: 2.61\n",
      "--- 19.086743354797363 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.829999892041087\n",
      "Episode 76\tAverage Score: 2.64\n",
      "--- 19.21485209465027 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.0499999318271875\n",
      "Episode 77\tAverage Score: 2.65\n",
      "--- 19.301231145858765 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.799999937415123\n",
      "Episode 78\tAverage Score: 2.65\n",
      "--- 19.365373611450195 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.069999886676669\n",
      "Episode 79\tAverage Score: 2.68\n",
      "--- 19.325016975402832 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.7399999164044857\n",
      "Episode 80\tAverage Score: 2.69\n",
      "--- 19.05771541595459 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.129999885335565\n",
      "Episode 81\tAverage Score: 2.73\n",
      "--- 19.321712493896484 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.859999891370535\n",
      "Episode 82\tAverage Score: 2.75\n",
      "--- 19.28875160217285 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.359999880194664\n",
      "Episode 83\tAverage Score: 2.78\n",
      "--- 18.550721168518066 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.179999951273203\n",
      "Episode 84\tAverage Score: 2.78\n",
      "--- 19.36008381843567 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.8799999356269836\n",
      "Episode 85\tAverage Score: 2.78\n",
      "--- 19.20715880393982 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.209999928250909\n",
      "Episode 86\tAverage Score: 2.78\n",
      "--- 18.28900957107544 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.769999893382192\n",
      "Episode 87\tAverage Score: 2.80\n",
      "--- 19.331242084503174 seconds ---\n",
      "Total score (averaged over agents) this episode: 2.9799999333918095\n",
      "Episode 88\tAverage Score: 2.81\n",
      "--- 19.406428337097168 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.479999855160713\n",
      "Episode 89\tAverage Score: 2.85\n",
      "--- 17.944603204727173 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.969999888911843\n",
      "Episode 90\tAverage Score: 2.87\n",
      "--- 18.227560997009277 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.959999889135361\n",
      "Episode 91\tAverage Score: 2.89\n",
      "--- 19.366244554519653 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.329999836161733\n",
      "Episode 92\tAverage Score: 2.94\n",
      "--- 19.172011852264404 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.409999901428819\n",
      "Episode 93\tAverage Score: 2.96\n",
      "--- 19.44970417022705 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.379999857395887\n",
      "Episode 94\tAverage Score: 2.99\n",
      "--- 19.286346435546875 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.8399999141693115\n",
      "Episode 95\tAverage Score: 3.00\n",
      "--- 19.325572967529297 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.899999912828207\n",
      "Episode 96\tAverage Score: 3.01\n",
      "--- 19.39788818359375 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.52999989874661\n",
      "Episode 97\tAverage Score: 3.03\n",
      "--- 19.29539179801941 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.3199999257922173\n",
      "Episode 98\tAverage Score: 3.03\n",
      "--- 19.40443706512451 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.289999881759286\n",
      "Episode 99\tAverage Score: 3.05\n",
      "--- 19.182802200317383 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.8399999141693115\n",
      "Episode 100\tAverage Score: 3.06\n",
      "--- 19.266639471054077 seconds ---\n",
      "Episode 100\tAverage Score: 3.06\n",
      "Total score (averaged over agents) this episode: 3.9099999126046896\n",
      "Episode 101\tAverage Score: 3.09\n",
      "--- 19.515523433685303 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.069999909028411\n",
      "Episode 102\tAverage Score: 3.13\n",
      "--- 19.585858583450317 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.5499999206513166\n",
      "Episode 103\tAverage Score: 3.15\n",
      "--- 19.483012437820435 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.6299999188631773\n",
      "Episode 104\tAverage Score: 3.18\n",
      "--- 18.71720814704895 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.29999990388751\n",
      "Episode 105\tAverage Score: 3.22\n",
      "--- 19.492945432662964 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.449999900534749\n",
      "Episode 106\tAverage Score: 3.24\n",
      "--- 19.499966859817505 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.729999916628003\n",
      "Episode 107\tAverage Score: 3.28\n",
      "--- 19.442830085754395 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.729999894276261\n",
      "Episode 108\tAverage Score: 3.32\n",
      "--- 19.23344326019287 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.599999874830246\n",
      "Episode 109\tAverage Score: 3.37\n",
      "--- 19.73094868659973 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.309999881312251\n",
      "Episode 110\tAverage Score: 3.41\n",
      "--- 19.526914358139038 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.109999841079116\n",
      "Episode 111\tAverage Score: 3.45\n",
      "--- 19.463491201400757 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.319999881088734\n",
      "Episode 112\tAverage Score: 3.49\n",
      "--- 19.62657856941223 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.599999897181988\n",
      "Episode 113\tAverage Score: 3.53\n",
      "--- 18.24008083343506 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.409999856725335\n",
      "Episode 114\tAverage Score: 3.59\n",
      "--- 19.372790098190308 seconds ---\n",
      "Total score (averaged over agents) this episode: 3.469999922439456\n",
      "Episode 115\tAverage Score: 3.60\n",
      "--- 19.586588621139526 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.909999890252948\n",
      "Episode 116\tAverage Score: 3.64\n",
      "--- 19.569617748260498 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.379999902099371\n",
      "Episode 117\tAverage Score: 3.68\n",
      "--- 19.891460418701172 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.529999831691384\n",
      "Episode 118\tAverage Score: 3.73\n",
      "--- 18.276715755462646 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.829999869689345\n",
      "Episode 119\tAverage Score: 3.77\n",
      "--- 17.62792205810547 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.889999778941274\n",
      "Episode 120\tAverage Score: 3.85\n",
      "--- 16.432538986206055 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.539999853819609\n",
      "Episode 121\tAverage Score: 3.91\n",
      "--- 15.277198791503906 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.009999820962548\n",
      "Episode 122\tAverage Score: 3.96\n",
      "--- 19.31202006340027 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.929999822750688\n",
      "Episode 123\tAverage Score: 4.02\n",
      "--- 19.712429761886597 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.7499998938292265\n",
      "Episode 124\tAverage Score: 4.03\n",
      "--- 19.668007373809814 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 7.039999842643738\n",
      "Episode 125\tAverage Score: 4.07\n",
      "--- 19.733855962753296 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.14999988488853\n",
      "Episode 126\tAverage Score: 4.11\n",
      "--- 19.851556062698364 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.679999895393848\n",
      "Episode 127\tAverage Score: 4.14\n",
      "--- 19.749101161956787 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.199999816715717\n",
      "Episode 128\tAverage Score: 4.21\n",
      "--- 19.810270309448242 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.96999984420836\n",
      "Episode 129\tAverage Score: 4.25\n",
      "--- 19.714149951934814 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.319999858736992\n",
      "Episode 130\tAverage Score: 4.30\n",
      "--- 19.67421007156372 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.189999816939235\n",
      "Episode 131\tAverage Score: 4.37\n",
      "--- 19.320748567581177 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.5299997869879\n",
      "Episode 132\tAverage Score: 4.44\n",
      "--- 19.66479229927063 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.779999826103449\n",
      "Episode 133\tAverage Score: 4.48\n",
      "--- 18.292088747024536 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.899999845772982\n",
      "Episode 134\tAverage Score: 4.54\n",
      "--- 16.89153265953064 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.479999899864197\n",
      "Episode 135\tAverage Score: 4.56\n",
      "--- 20.071826934814453 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.589999807998538\n",
      "Episode 136\tAverage Score: 4.63\n",
      "--- 19.934908628463745 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.83999989181757\n",
      "Episode 137\tAverage Score: 4.66\n",
      "--- 19.773096799850464 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.559999831020832\n",
      "Episode 138\tAverage Score: 4.72\n",
      "--- 19.848233699798584 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.149999728426337\n",
      "Episode 139\tAverage Score: 4.82\n",
      "--- 19.88651442527771 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.6399998515844345\n",
      "Episode 140\tAverage Score: 4.86\n",
      "--- 20.043908834457397 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.89999982342124\n",
      "Episode 141\tAverage Score: 4.93\n",
      "--- 18.83262348175049 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.259999815374613\n",
      "Episode 142\tAverage Score: 4.97\n",
      "--- 17.6087167263031 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.789999870583415\n",
      "Episode 143\tAverage Score: 5.00\n",
      "--- 19.418318271636963 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.459999855607748\n",
      "Episode 144\tAverage Score: 5.04\n",
      "--- 20.075003385543823 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.819999802857637\n",
      "Episode 145\tAverage Score: 5.11\n",
      "--- 20.006756067276 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.659999806433916\n",
      "Episode 146\tAverage Score: 5.17\n",
      "--- 20.048791885375977 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.3099997472018\n",
      "Episode 147\tAverage Score: 5.25\n",
      "--- 20.0892391204834 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.569999808445573\n",
      "Episode 148\tAverage Score: 5.30\n",
      "--- 19.999412536621094 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.509999809786677\n",
      "Episode 149\tAverage Score: 5.33\n",
      "--- 20.03203797340393 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.789999848231673\n",
      "Episode 150\tAverage Score: 5.37\n",
      "--- 19.99698758125305 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.629999874159694\n",
      "Episode 151\tAverage Score: 5.40\n",
      "--- 20.031007289886475 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.199999794363976\n",
      "Episode 152\tAverage Score: 5.47\n",
      "--- 18.092224597930908 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.119999796152115\n",
      "Episode 153\tAverage Score: 5.52\n",
      "--- 16.03906512260437 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.979999866336584\n",
      "Episode 154\tAverage Score: 5.55\n",
      "--- 19.898754119873047 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.379999857395887\n",
      "Episode 155\tAverage Score: 5.58\n",
      "--- 18.738500356674194 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.239999860525131\n",
      "Episode 156\tAverage Score: 5.60\n",
      "--- 19.822359085083008 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.529999831691384\n",
      "Episode 157\tAverage Score: 5.63\n",
      "--- 20.07650375366211 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.459999654442072\n",
      "Episode 158\tAverage Score: 5.75\n",
      "--- 20.147128343582153 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.169999839738011\n",
      "Episode 159\tAverage Score: 5.78\n",
      "--- 19.96934151649475 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.0699998419731855\n",
      "Episode 160\tAverage Score: 5.79\n",
      "--- 20.00975489616394 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.449999833479524\n",
      "Episode 161\tAverage Score: 5.82\n",
      "--- 20.160763263702393 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.179999839514494\n",
      "Episode 162\tAverage Score: 5.88\n",
      "--- 17.813501596450806 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.159999705851078\n",
      "Episode 163\tAverage Score: 5.98\n",
      "--- 15.635966062545776 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.239999815821648\n",
      "Episode 164\tAverage Score: 6.03\n",
      "--- 18.46022081375122 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.48999978788197\n",
      "Episode 165\tAverage Score: 6.09\n",
      "--- 20.07595944404602 seconds ---\n",
      "Total score (averaged over agents) this episode: 4.599999897181988\n",
      "Episode 166\tAverage Score: 6.10\n",
      "--- 20.0336012840271 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.679999805986881\n",
      "Episode 167\tAverage Score: 6.14\n",
      "--- 19.778006076812744 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.449999811127782\n",
      "Episode 168\tAverage Score: 6.14\n",
      "--- 19.105822324752808 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.459999833256006\n",
      "Episode 169\tAverage Score: 6.15\n",
      "--- 20.12589430809021 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.069999819621444\n",
      "Episode 170\tAverage Score: 6.19\n",
      "--- 20.148319244384766 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.139999773353338\n",
      "Episode 171\tAverage Score: 6.25\n",
      "--- 20.06866455078125 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.539999853819609\n",
      "Episode 172\tAverage Score: 6.27\n",
      "--- 20.33134365081787 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.559999831020832\n",
      "Episode 173\tAverage Score: 6.31\n",
      "--- 20.030165910720825 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.509999832138419\n",
      "Episode 174\tAverage Score: 6.35\n",
      "--- 20.054584741592407 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.0699998419731855\n",
      "Episode 175\tAverage Score: 6.38\n",
      "--- 20.261282205581665 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.309999814257026\n",
      "Episode 176\tAverage Score: 6.41\n",
      "--- 20.08662724494934 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.429999789223075\n",
      "Episode 177\tAverage Score: 6.47\n",
      "--- 20.17706847190857 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.029999865218997\n",
      "Episode 178\tAverage Score: 6.51\n",
      "--- 20.27971386909485 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.299999814480543\n",
      "Episode 179\tAverage Score: 6.54\n",
      "--- 20.131041765213013 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.969999754801393\n",
      "Episode 180\tAverage Score: 6.61\n",
      "--- 20.047717809677124 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.17999979481101\n",
      "Episode 181\tAverage Score: 6.65\n",
      "--- 20.388076543807983 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.789999848231673\n",
      "Episode 182\tAverage Score: 6.67\n",
      "--- 20.290260553359985 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.459999788552523\n",
      "Episode 183\tAverage Score: 6.71\n",
      "--- 20.232828617095947 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.479999788105488\n",
      "Episode 184\tAverage Score: 6.78\n",
      "--- 17.101780891418457 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.399999812245369\n",
      "Episode 185\tAverage Score: 6.84\n",
      "--- 18.279712915420532 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.239999771118164\n",
      "Episode 186\tAverage Score: 6.91\n",
      "--- 20.376339197158813 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 10.209999771788716\n",
      "Episode 187\tAverage Score: 6.96\n",
      "--- 20.291811227798462 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.44999978877604\n",
      "Episode 188\tAverage Score: 7.03\n",
      "--- 20.263349056243896 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.739999759942293\n",
      "Episode 189\tAverage Score: 7.07\n",
      "--- 20.408308029174805 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.129999795928597\n",
      "Episode 190\tAverage Score: 7.11\n",
      "--- 20.430460214614868 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.609999829903245\n",
      "Episode 191\tAverage Score: 7.14\n",
      "--- 20.345086812973022 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.989999821409583\n",
      "Episode 192\tAverage Score: 7.15\n",
      "--- 20.331172943115234 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.369999745860696\n",
      "Episode 193\tAverage Score: 7.22\n",
      "--- 20.354143619537354 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.159999795258045\n",
      "Episode 194\tAverage Score: 7.24\n",
      "--- 20.370949745178223 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.019999708980322\n",
      "Episode 195\tAverage Score: 7.34\n",
      "--- 20.339751958847046 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.94999977760017\n",
      "Episode 196\tAverage Score: 7.40\n",
      "--- 20.405635595321655 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.4299998339265585\n",
      "Episode 197\tAverage Score: 7.43\n",
      "--- 16.828906059265137 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.279999792575836\n",
      "Episode 198\tAverage Score: 7.48\n",
      "--- 16.22739315032959 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.749999715015292\n",
      "Episode 199\tAverage Score: 7.56\n",
      "--- 20.47688341140747 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.989999709650874\n",
      "Episode 200\tAverage Score: 7.65\n",
      "--- 20.473605394363403 seconds ---\n",
      "Episode 200\tAverage Score: 7.65\n",
      "Total score (averaged over agents) this episode: 13.379999700933695\n",
      "Episode 201\tAverage Score: 7.75\n",
      "--- 20.52319598197937 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.249999770894647\n",
      "Episode 202\tAverage Score: 7.81\n",
      "--- 20.447340488433838 seconds ---\n",
      "Total score (averaged over agents) this episode: 5.959999866783619\n",
      "Episode 203\tAverage Score: 7.83\n",
      "--- 20.58545207977295 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.249999636784196\n",
      "Episode 204\tAverage Score: 7.96\n",
      "--- 20.455551862716675 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.929999800398946\n",
      "Episode 205\tAverage Score: 8.00\n",
      "--- 20.478822708129883 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.67999978363514\n",
      "Episode 206\tAverage Score: 8.06\n",
      "--- 20.54849910736084 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.979999799281359\n",
      "Episode 207\tAverage Score: 8.11\n",
      "--- 20.479853630065918 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.819999825209379\n",
      "Episode 208\tAverage Score: 8.14\n",
      "--- 20.59453296661377 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.779999803751707\n",
      "Episode 209\tAverage Score: 8.17\n",
      "--- 20.565120458602905 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.709999760612845\n",
      "Episode 210\tAverage Score: 8.23\n",
      "--- 20.503438711166382 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.37999976798892\n",
      "Episode 211\tAverage Score: 8.26\n",
      "--- 20.584089040756226 seconds ---\n",
      "Total score (averaged over agents) this episode: 6.589999852702022\n",
      "Episode 212\tAverage Score: 8.27\n",
      "--- 20.768860578536987 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.519999653100967\n",
      "Episode 213\tAverage Score: 8.38\n",
      "--- 20.542384386062622 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.95999975502491\n",
      "Episode 214\tAverage Score: 8.43\n",
      "--- 17.074920415878296 seconds ---\n",
      "Total score (averaged over agents) this episode: 14.469999676570296\n",
      "Episode 215\tAverage Score: 8.54\n",
      "--- 17.013373136520386 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.58999983035028\n",
      "Episode 216\tAverage Score: 8.56\n",
      "--- 17.977282762527466 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.459999833256006\n",
      "Episode 217\tAverage Score: 8.59\n",
      "--- 15.960340738296509 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.609999785199761\n",
      "Episode 218\tAverage Score: 8.61\n",
      "--- 16.15818738937378 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.139999706298113\n",
      "Episode 219\tAverage Score: 8.69\n",
      "--- 17.518543004989624 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.869999712333083\n",
      "Episode 220\tAverage Score: 8.72\n",
      "--- 20.302499294281006 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.409999744966626\n",
      "Episode 221\tAverage Score: 8.77\n",
      "--- 20.827083349227905 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.119999751448631\n",
      "Episode 222\tAverage Score: 8.80\n",
      "--- 20.765795707702637 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.659999806433916\n",
      "Episode 223\tAverage Score: 8.80\n",
      "--- 20.801762580871582 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.329999724403024\n",
      "Episode 224\tAverage Score: 8.88\n",
      "--- 20.770320415496826 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.759999714791775\n",
      "Episode 225\tAverage Score: 8.94\n",
      "--- 20.819826364517212 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.1899998392909765\n",
      "Episode 226\tAverage Score: 8.96\n",
      "--- 20.47398018836975 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.359999813139439\n",
      "Episode 227\tAverage Score: 8.99\n",
      "--- 20.763503313064575 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.089999752119184\n",
      "Episode 228\tAverage Score: 9.02\n",
      "--- 20.764477252960205 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.1899998392909765\n",
      "Episode 229\tAverage Score: 9.03\n",
      "--- 20.653546810150146 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.3099998366087675\n",
      "Episode 230\tAverage Score: 9.04\n",
      "--- 19.215530395507812 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.799999803304672\n",
      "Episode 231\tAverage Score: 9.04\n",
      "--- 20.24719476699829 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.699999738484621\n",
      "Episode 232\tAverage Score: 9.06\n",
      "--- 20.8572518825531 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.459999721497297\n",
      "Episode 233\tAverage Score: 9.11\n",
      "--- 20.80027675628662 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.40999978967011\n",
      "Episode 234\tAverage Score: 9.13\n",
      "--- 20.883383989334106 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.129999751225114\n",
      "Episode 235\tAverage Score: 9.20\n",
      "--- 19.722732543945312 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.92999973334372\n",
      "Episode 236\tAverage Score: 9.23\n",
      "--- 20.891215562820435 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.939999755471945\n",
      "Episode 237\tAverage Score: 9.30\n",
      "--- 20.99108862876892 seconds ---\n",
      "Total score (averaged over agents) this episode: 8.879999801516533\n",
      "Episode 238\tAverage Score: 9.31\n",
      "--- 20.91264271736145 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.469999698922038\n",
      "Episode 239\tAverage Score: 9.32\n",
      "--- 21.145960092544556 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.169999772682786\n",
      "Episode 240\tAverage Score: 9.36\n",
      "--- 20.86007857322693 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.209999727085233\n",
      "Episode 241\tAverage Score: 9.40\n",
      "--- 21.053279638290405 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.87999964505434\n",
      "Episode 242\tAverage Score: 9.48\n",
      "--- 20.944496393203735 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.80999962426722\n",
      "Episode 243\tAverage Score: 9.59\n",
      "--- 21.043944835662842 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.139999773353338\n",
      "Episode 244\tAverage Score: 9.62\n",
      "--- 19.00027370452881 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.119999662041664\n",
      "Episode 245\tAverage Score: 9.69\n",
      "--- 21.02108383178711 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.289999702945352\n",
      "Episode 246\tAverage Score: 9.73\n",
      "--- 21.01116180419922 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.25999963656068\n",
      "Episode 247\tAverage Score: 9.78\n",
      "--- 21.153605937957764 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 8.569999808445573\n",
      "Episode 248\tAverage Score: 9.78\n",
      "--- 21.13817286491394 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.57999916002154\n",
      "Episode 249\tAverage Score: 10.07\n",
      "--- 20.439104318618774 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.539999719709158\n",
      "Episode 250\tAverage Score: 10.13\n",
      "--- 21.109346389770508 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.74999958090484\n",
      "Episode 251\tAverage Score: 10.26\n",
      "--- 21.06857919692993 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.349999723955989\n",
      "Episode 252\tAverage Score: 10.29\n",
      "--- 20.00658917427063 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.939999643713236\n",
      "Episode 253\tAverage Score: 10.36\n",
      "--- 21.102973461151123 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.179999638348818\n",
      "Episode 254\tAverage Score: 10.46\n",
      "--- 21.069141387939453 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.78999942354858\n",
      "Episode 255\tAverage Score: 10.66\n",
      "--- 21.23817801475525 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.569999719038606\n",
      "Episode 256\tAverage Score: 10.72\n",
      "--- 19.113175868988037 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.89999957755208\n",
      "Episode 257\tAverage Score: 10.83\n",
      "--- 18.476094007492065 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.03999948501587\n",
      "Episode 258\tAverage Score: 10.91\n",
      "--- 19.138022661209106 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.659999582916498\n",
      "Episode 259\tAverage Score: 11.03\n",
      "--- 21.081840991973877 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.36999941058457\n",
      "Episode 260\tAverage Score: 11.22\n",
      "--- 21.20981001853943 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.909999443218112\n",
      "Episode 261\tAverage Score: 11.39\n",
      "--- 19.09208393096924 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.9799998216331005\n",
      "Episode 262\tAverage Score: 11.40\n",
      "--- 21.233104467391968 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.34999947808683\n",
      "Episode 263\tAverage Score: 11.50\n",
      "--- 21.337695360183716 seconds ---\n",
      "Total score (averaged over agents) this episode: 17.629999605938792\n",
      "Episode 264\tAverage Score: 11.60\n",
      "--- 21.269019842147827 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.529999630525708\n",
      "Episode 265\tAverage Score: 11.67\n",
      "--- 20.550985097885132 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.159999549388885\n",
      "Episode 266\tAverage Score: 11.82\n",
      "--- 18.392873525619507 seconds ---\n",
      "Total score (averaged over agents) this episode: 11.61999974027276\n",
      "Episode 267\tAverage Score: 11.85\n",
      "--- 21.39024043083191 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.879999555647373\n",
      "Episode 268\tAverage Score: 11.97\n",
      "--- 20.69042181968689 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.269999658688903\n",
      "Episode 269\tAverage Score: 12.04\n",
      "--- 16.73805570602417 seconds ---\n",
      "Total score (averaged over agents) this episode: 17.529999608173966\n",
      "Episode 270\tAverage Score: 12.14\n",
      "--- 20.038822412490845 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.229999592527747\n",
      "Episode 271\tAverage Score: 12.22\n",
      "--- 21.4896981716156 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.999999463558197\n",
      "Episode 272\tAverage Score: 12.39\n",
      "--- 21.037883281707764 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.119999416172504\n",
      "Episode 273\tAverage Score: 12.58\n",
      "--- 19.250282049179077 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.079999774694443\n",
      "Episode 274\tAverage Score: 12.61\n",
      "--- 21.611103057861328 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.729999559000134\n",
      "Episode 275\tAverage Score: 12.73\n",
      "--- 21.500081062316895 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.269999457523227\n",
      "Episode 276\tAverage Score: 12.89\n",
      "--- 21.660892724990845 seconds ---\n",
      "Total score (averaged over agents) this episode: 14.089999685063958\n",
      "Episode 277\tAverage Score: 12.94\n",
      "--- 21.56301784515381 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.16999957151711\n",
      "Episode 278\tAverage Score: 13.07\n",
      "--- 21.5001323223114 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.899999689310789\n",
      "Episode 279\tAverage Score: 13.13\n",
      "--- 21.693296432495117 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.969999397173524\n",
      "Episode 280\tAverage Score: 13.29\n",
      "--- 21.595614194869995 seconds ---\n",
      "Total score (averaged over agents) this episode: 10.389999767765403\n",
      "Episode 281\tAverage Score: 13.30\n",
      "--- 21.679675579071045 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.15999959409237\n",
      "Episode 282\tAverage Score: 13.41\n",
      "--- 21.551403045654297 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.829999646171927\n",
      "Episode 283\tAverage Score: 13.48\n",
      "--- 18.983242511749268 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.399999499320984\n",
      "Episode 284\tAverage Score: 13.61\n",
      "--- 21.665344715118408 seconds ---\n",
      "Total score (averaged over agents) this episode: 12.729999715462327\n",
      "Episode 285\tAverage Score: 13.65\n",
      "--- 19.74875259399414 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.879999466240406\n",
      "Episode 286\tAverage Score: 13.78\n",
      "--- 20.014562845230103 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.249999346211553\n",
      "Episode 287\tAverage Score: 13.98\n",
      "--- 21.708059310913086 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.679999694228172\n",
      "Episode 288\tAverage Score: 14.02\n",
      "--- 21.694428205490112 seconds ---\n",
      "Total score (averaged over agents) this episode: 9.63999978452921\n",
      "Episode 289\tAverage Score: 14.01\n",
      "--- 20.927899837493896 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.39999970048666\n",
      "Episode 290\tAverage Score: 14.05\n",
      "--- 19.17921757698059 seconds ---\n",
      "Total score (averaged over agents) this episode: 13.139999706298113\n",
      "Episode 291\tAverage Score: 14.10\n",
      "--- 21.76124668121338 seconds ---\n",
      "Total score (averaged over agents) this episode: 7.4299998339265585\n",
      "Episode 292\tAverage Score: 14.10\n",
      "--- 21.876319408416748 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.169999638572335\n",
      "Episode 293\tAverage Score: 14.15\n",
      "--- 21.748192071914673 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.939999375492334\n",
      "Episode 294\tAverage Score: 14.33\n",
      "--- 20.186780214309692 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.03999964147806\n",
      "Episode 295\tAverage Score: 14.36\n",
      "--- 19.603488445281982 seconds ---\n",
      "Total score (averaged over agents) this episode: 14.689999671652913\n",
      "Episode 296\tAverage Score: 14.41\n",
      "--- 21.67781925201416 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.099999506026506\n",
      "Episode 297\tAverage Score: 14.56\n",
      "--- 21.636799335479736 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.63999953866005\n",
      "Episode 298\tAverage Score: 14.67\n",
      "--- 21.96920657157898 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.949999643489718\n",
      "Episode 299\tAverage Score: 14.70\n",
      "--- 21.838940143585205 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.739999536424875\n",
      "Episode 300\tAverage Score: 14.78\n",
      "--- 21.939162731170654 seconds ---\n",
      "Episode 300\tAverage Score: 14.78\n",
      "Total score (averaged over agents) this episode: 19.62999956123531\n",
      "Episode 301\tAverage Score: 14.84\n",
      "--- 22.008304595947266 seconds ---\n",
      "Total score (averaged over agents) this episode: 15.83999964594841\n",
      "Episode 302\tAverage Score: 14.90\n",
      "--- 22.090864181518555 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.84999962337315\n",
      "Episode 303\tAverage Score: 15.01\n",
      "--- 22.083512783050537 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.659999381750822\n",
      "Episode 304\tAverage Score: 15.12\n",
      "--- 22.036430597305298 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.339999567717314\n",
      "Episode 305\tAverage Score: 15.23\n",
      "--- 22.081328868865967 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.55999954044819\n",
      "Episode 306\tAverage Score: 15.34\n",
      "--- 22.02152705192566 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.04999948479235\n",
      "Episode 307\tAverage Score: 15.48\n",
      "--- 22.0263614654541 seconds ---\n",
      "Total score (averaged over agents) this episode: 32.549999272450805\n",
      "Episode 308\tAverage Score: 15.72\n",
      "--- 22.06818914413452 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 26.67999940365553\n",
      "Episode 309\tAverage Score: 15.90\n",
      "--- 21.99590516090393 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.98999955318868\n",
      "Episode 310\tAverage Score: 16.00\n",
      "--- 22.160805702209473 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.209999369457364\n",
      "Episode 311\tAverage Score: 16.17\n",
      "--- 22.178287744522095 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.529999451711774\n",
      "Episode 312\tAverage Score: 16.35\n",
      "--- 18.858344316482544 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.01999957486987\n",
      "Episode 313\tAverage Score: 16.39\n",
      "--- 20.88957667350769 seconds ---\n",
      "Total score (averaged over agents) this episode: 18.729999581351876\n",
      "Episode 314\tAverage Score: 16.47\n",
      "--- 22.162920713424683 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.139999505132437\n",
      "Episode 315\tAverage Score: 16.54\n",
      "--- 21.295772552490234 seconds ---\n",
      "Total score (averaged over agents) this episode: 21.129999527707696\n",
      "Episode 316\tAverage Score: 16.68\n",
      "--- 21.668349742889404 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.519999563694\n",
      "Episode 317\tAverage Score: 16.80\n",
      "--- 21.24219584465027 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.369999477639794\n",
      "Episode 318\tAverage Score: 16.94\n",
      "--- 17.512089252471924 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.019999507814646\n",
      "Episode 319\tAverage Score: 17.03\n",
      "--- 19.69947361946106 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.16999943740666\n",
      "Episode 320\tAverage Score: 17.15\n",
      "--- 17.77419924736023 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.879999332129955\n",
      "Episode 321\tAverage Score: 17.33\n",
      "--- 20.20187473297119 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.289999501779675\n",
      "Episode 322\tAverage Score: 17.45\n",
      "--- 22.321819305419922 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.509999496862292\n",
      "Episode 323\tAverage Score: 17.58\n",
      "--- 22.329005241394043 seconds ---\n",
      "Total score (averaged over agents) this episode: 17.369999611750245\n",
      "Episode 324\tAverage Score: 17.63\n",
      "--- 22.315459489822388 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.68999949283898\n",
      "Episode 325\tAverage Score: 17.73\n",
      "--- 22.29913067817688 seconds ---\n",
      "Total score (averaged over agents) this episode: 31.399999298155308\n",
      "Episode 326\tAverage Score: 17.98\n",
      "--- 22.359636306762695 seconds ---\n",
      "Total score (averaged over agents) this episode: 14.82999966852367\n",
      "Episode 327\tAverage Score: 18.04\n",
      "--- 22.357582569122314 seconds ---\n",
      "Total score (averaged over agents) this episode: 21.039999529719353\n",
      "Episode 328\tAverage Score: 18.14\n",
      "--- 22.46626329421997 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.529999451711774\n",
      "Episode 329\tAverage Score: 18.31\n",
      "--- 22.726075649261475 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.449999565258622\n",
      "Episode 330\tAverage Score: 18.43\n",
      "--- 22.450828552246094 seconds ---\n",
      "Total score (averaged over agents) this episode: 14.129999684169888\n",
      "Episode 331\tAverage Score: 18.49\n",
      "--- 17.581072092056274 seconds ---\n",
      "Total score (averaged over agents) this episode: 32.439999274909496\n",
      "Episode 332\tAverage Score: 18.70\n",
      "--- 19.382945775985718 seconds ---\n",
      "Total score (averaged over agents) this episode: 21.709999514743686\n",
      "Episode 333\tAverage Score: 18.79\n",
      "--- 22.53237748146057 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.01999932900071\n",
      "Episode 334\tAverage Score: 18.99\n",
      "--- 21.18466329574585 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.29999954625964\n",
      "Episode 335\tAverage Score: 19.09\n",
      "--- 22.28905987739563 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.969999419525266\n",
      "Episode 336\tAverage Score: 19.23\n",
      "--- 22.726637840270996 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.909999353811145\n",
      "Episode 337\tAverage Score: 19.41\n",
      "--- 22.706538438796997 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.37999938800931\n",
      "Episode 338\tAverage Score: 19.59\n",
      "--- 22.67214012145996 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.909999443218112\n",
      "Episode 339\tAverage Score: 19.70\n",
      "--- 22.62744927406311 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.739999424666166\n",
      "Episode 340\tAverage Score: 19.86\n",
      "--- 22.69064712524414 seconds ---\n",
      "Total score (averaged over agents) this episode: 17.749999603256583\n",
      "Episode 341\tAverage Score: 19.92\n",
      "--- 22.65397334098816 seconds ---\n",
      "Total score (averaged over agents) this episode: 24.759999446570873\n",
      "Episode 342\tAverage Score: 20.00\n",
      "--- 22.66414523124695 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.77999922260642\n",
      "Episode 343\tAverage Score: 20.18\n",
      "--- 22.775907039642334 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.739999402314425\n",
      "Episode 344\tAverage Score: 20.35\n",
      "--- 22.71750020980835 seconds ---\n",
      "Total score (averaged over agents) this episode: 19.36999956704676\n",
      "Episode 345\tAverage Score: 20.39\n",
      "--- 22.6868634223938 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.3099994789809\n",
      "Episode 346\tAverage Score: 20.49\n",
      "--- 22.713110208511353 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.149999482557178\n",
      "Episode 347\tAverage Score: 20.56\n",
      "--- 22.99161434173584 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.16999932564795\n",
      "Episode 348\tAverage Score: 20.78\n",
      "--- 22.903417825698853 seconds ---\n",
      "Total score (averaged over agents) this episode: 21.739999514073133\n",
      "Episode 349\tAverage Score: 20.62\n",
      "--- 22.8080894947052 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.12999932654202\n",
      "Episode 350\tAverage Score: 20.80\n",
      "--- 22.721760749816895 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.379999365657568\n",
      "Episode 351\tAverage Score: 20.89\n",
      "--- 22.913438320159912 seconds ---\n",
      "Total score (averaged over agents) this episode: 33.67999924719334\n",
      "Episode 352\tAverage Score: 21.11\n",
      "--- 23.01722264289856 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.319999635219574\n",
      "Episode 353\tAverage Score: 21.11\n",
      "--- 22.809110403060913 seconds ---\n",
      "Total score (averaged over agents) this episode: 16.4199996329844\n",
      "Episode 354\tAverage Score: 21.11\n",
      "--- 22.85406184196472 seconds ---\n",
      "Total score (averaged over agents) this episode: 35.0799992159009\n",
      "Episode 355\tAverage Score: 21.20\n",
      "--- 22.843838930130005 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.559999406337738\n",
      "Episode 356\tAverage Score: 21.34\n",
      "--- 23.10499906539917 seconds ---\n",
      "Total score (averaged over agents) this episode: 33.019999261945486\n",
      "Episode 357\tAverage Score: 21.49\n",
      "--- 22.594006538391113 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.079999439418316\n",
      "Episode 358\tAverage Score: 21.51\n",
      "--- 23.083561182022095 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.38999934308231\n",
      "Episode 359\tAverage Score: 21.61\n",
      "--- 23.004646062850952 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.839999333024025\n",
      "Episode 360\tAverage Score: 21.65\n",
      "--- 22.003156185150146 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.209999347105622\n",
      "Episode 361\tAverage Score: 21.69\n",
      "--- 23.203140020370483 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.06999941729009\n",
      "Episode 362\tAverage Score: 21.87\n",
      "--- 23.115750551223755 seconds ---\n",
      "Total score (averaged over agents) this episode: 32.45999927446246\n",
      "Episode 363\tAverage Score: 21.96\n",
      "--- 23.13686990737915 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.839999310672283\n",
      "Episode 364\tAverage Score: 22.09\n",
      "--- 23.100118398666382 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.14999923668802\n",
      "Episode 365\tAverage Score: 22.27\n",
      "--- 23.004777193069458 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.489999318495393\n",
      "Episode 366\tAverage Score: 22.37\n",
      "--- 23.22490167617798 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.59999942779541\n",
      "Episode 367\tAverage Score: 22.51\n",
      "--- 23.237839937210083 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.06999939493835\n",
      "Episode 368\tAverage Score: 22.59\n",
      "--- 19.77165198326111 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.10999923758209\n",
      "Episode 369\tAverage Score: 22.77\n",
      "--- 18.24742364883423 seconds ---\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 27.279999390244484\n",
      "Episode 370\tAverage Score: 22.87\n",
      "--- 19.31864881515503 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.11999923735857\n",
      "Episode 371\tAverage Score: 23.03\n",
      "--- 23.26488423347473 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.069999327883124\n",
      "Episode 372\tAverage Score: 23.09\n",
      "--- 23.342623472213745 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.81999946758151\n",
      "Episode 373\tAverage Score: 23.07\n",
      "--- 23.354874849319458 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.25999912247062\n",
      "Episode 374\tAverage Score: 23.36\n",
      "--- 23.558867931365967 seconds ---\n",
      "Total score (averaged over agents) this episode: 38.56999913789332\n",
      "Episode 375\tAverage Score: 23.55\n",
      "--- 23.400435209274292 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.719999313354492\n",
      "Episode 376\tAverage Score: 23.61\n",
      "--- 22.35279130935669 seconds ---\n",
      "Total score (averaged over agents) this episode: 20.44999954290688\n",
      "Episode 377\tAverage Score: 23.68\n",
      "--- 23.45312809944153 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.269999323412776\n",
      "Episode 378\tAverage Score: 23.79\n",
      "--- 23.403786182403564 seconds ---\n",
      "Total score (averaged over agents) this episode: 35.96999919600785\n",
      "Episode 379\tAverage Score: 24.01\n",
      "--- 23.44035840034485 seconds ---\n",
      "Total score (averaged over agents) this episode: 31.83999928832054\n",
      "Episode 380\tAverage Score: 24.06\n",
      "--- 23.39005947113037 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.41999916359782\n",
      "Episode 381\tAverage Score: 24.33\n",
      "--- 23.400357961654663 seconds ---\n",
      "Total score (averaged over agents) this episode: 23.689999470487237\n",
      "Episode 382\tAverage Score: 24.38\n",
      "--- 23.499940872192383 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.529999339953065\n",
      "Episode 383\tAverage Score: 24.52\n",
      "--- 23.453511953353882 seconds ---\n",
      "Total score (averaged over agents) this episode: 35.339999210089445\n",
      "Episode 384\tAverage Score: 24.65\n",
      "--- 23.532572269439697 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.99999921768904\n",
      "Episode 385\tAverage Score: 24.87\n",
      "--- 23.758793354034424 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.249999368563294\n",
      "Episode 386\tAverage Score: 24.92\n",
      "--- 23.340779304504395 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.699999403208494\n",
      "Episode 387\tAverage Score: 24.89\n",
      "--- 22.990949630737305 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.21999939158559\n",
      "Episode 388\tAverage Score: 25.03\n",
      "--- 23.416661500930786 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.019999396055937\n",
      "Episode 389\tAverage Score: 25.20\n",
      "--- 23.518293857574463 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.83999937772751\n",
      "Episode 390\tAverage Score: 25.34\n",
      "--- 23.5359365940094 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.88999937660992\n",
      "Episode 391\tAverage Score: 25.49\n",
      "--- 23.73458504676819 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.129999371245503\n",
      "Episode 392\tAverage Score: 25.70\n",
      "--- 23.610947847366333 seconds ---\n",
      "Total score (averaged over agents) this episode: 35.89999919757247\n",
      "Episode 393\tAverage Score: 25.90\n",
      "--- 23.660623788833618 seconds ---\n",
      "Total score (averaged over agents) this episode: 36.82999917678535\n",
      "Episode 394\tAverage Score: 25.98\n",
      "--- 23.643476486206055 seconds ---\n",
      "Total score (averaged over agents) this episode: 26.06999941729009\n",
      "Episode 395\tAverage Score: 26.08\n",
      "--- 23.690812826156616 seconds ---\n",
      "Total score (averaged over agents) this episode: 22.41999949887395\n",
      "Episode 396\tAverage Score: 26.16\n",
      "--- 23.720998764038086 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.419999342411757\n",
      "Episode 397\tAverage Score: 26.24\n",
      "--- 23.54376721382141 seconds ---\n",
      "Total score (averaged over agents) this episode: 27.21999939158559\n",
      "Episode 398\tAverage Score: 26.30\n",
      "--- 23.67394709587097 seconds ---\n",
      "Total score (averaged over agents) this episode: 34.409999230876565\n",
      "Episode 399\tAverage Score: 26.49\n",
      "--- 23.761392831802368 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.70999915711582\n",
      "Episode 400\tAverage Score: 26.66\n",
      "--- 23.780913829803467 seconds ---\n",
      "Episode 400\tAverage Score: 26.66\n",
      "Total score (averaged over agents) this episode: 34.059999238699675\n",
      "Episode 401\tAverage Score: 26.80\n",
      "--- 23.66612195968628 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.279999166727066\n",
      "Episode 402\tAverage Score: 27.01\n",
      "--- 23.82118320465088 seconds ---\n",
      "Total score (averaged over agents) this episode: 33.9299992416054\n",
      "Episode 403\tAverage Score: 27.18\n",
      "--- 23.763355016708374 seconds ---\n",
      "Total score (averaged over agents) this episode: 32.469999274238944\n",
      "Episode 404\tAverage Score: 27.23\n",
      "--- 23.363755702972412 seconds ---\n",
      "Total score (averaged over agents) this episode: 25.67999942600727\n",
      "Episode 405\tAverage Score: 27.30\n",
      "--- 23.8284330368042 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.07999935001135\n",
      "Episode 406\tAverage Score: 27.38\n",
      "--- 22.61115288734436 seconds ---\n",
      "Total score (averaged over agents) this episode: 36.95999917387962\n",
      "Episode 407\tAverage Score: 27.52\n",
      "--- 22.02437162399292 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.409999320283532\n",
      "Episode 408\tAverage Score: 27.50\n",
      "--- 23.93305516242981 seconds ---\n",
      "Total score (averaged over agents) this episode: 38.03999914973974\n",
      "Episode 409\tAverage Score: 27.61\n",
      "--- 23.913720846176147 seconds ---\n",
      "Total score (averaged over agents) this episode: 30.05999932810664\n",
      "Episode 410\tAverage Score: 27.71\n",
      "--- 23.89115858078003 seconds ---\n",
      "Total score (averaged over agents) this episode: 38.55999913811684\n",
      "Episode 411\tAverage Score: 27.82\n",
      "--- 23.95799422264099 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.2699991222471\n",
      "Episode 412\tAverage Score: 27.96\n",
      "--- 22.157433032989502 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.59999915957451\n",
      "Episode 413\tAverage Score: 28.15\n",
      "--- 23.892780780792236 seconds ---\n",
      "Total score (averaged over agents) this episode: 29.889999331906438\n",
      "Episode 414\tAverage Score: 28.26\n",
      "--- 21.330154180526733 seconds ---\n",
      "Total score (averaged over agents) this episode: 28.099999371916056\n",
      "Episode 415\tAverage Score: 28.32\n",
      "--- 22.176992893218994 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.00999912805855\n",
      "Episode 416\tAverage Score: 28.50\n",
      "--- 22.92396593093872 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.41999911889434\n",
      "Episode 417\tAverage Score: 28.70\n",
      "--- 23.471389770507812 seconds ---\n",
      "Total score (averaged over agents) this episode: 37.68999915756285\n",
      "Episode 418\tAverage Score: 28.84\n",
      "--- 24.03108501434326 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.34999912045896\n",
      "Episode 419\tAverage Score: 29.02\n",
      "--- 22.901410818099976 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.52999911643565\n",
      "Episode 420\tAverage Score: 29.16\n",
      "--- 24.002310514450073 seconds ---\n",
      "Total score (averaged over agents) this episode: 31.90999928675592\n",
      "Episode 421\tAverage Score: 29.18\n",
      "--- 22.86046528816223 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.439999118447304\n",
      "Episode 422\tAverage Score: 29.35\n",
      "--- 23.898436784744263 seconds ---\n",
      "Total score (averaged over agents) this episode: 36.64999918080866\n",
      "Episode 423\tAverage Score: 29.49\n",
      "--- 20.725200653076172 seconds ---\n",
      "Total score (averaged over agents) this episode: 38.669999135658145\n",
      "Episode 424\tAverage Score: 29.71\n",
      "--- 23.94243860244751 seconds ---\n",
      "Total score (averaged over agents) this episode: 38.5799991376698\n",
      "Episode 425\tAverage Score: 29.86\n",
      "--- 23.690083503723145 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.37999911978841\n",
      "Episode 426\tAverage Score: 29.94\n",
      "--- 20.74186372756958 seconds ---\n",
      "Total score (averaged over agents) this episode: 39.65999911352992\n",
      "Episode 427\tAverage Score: 30.19\n",
      "--- 23.58912467956543 seconds ---\n",
      "SOLVED!!  Episode 427\tAverage Score: 30.19\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2deZhcZZX/v+fequp9SXe6kyYLnY0kLCHBQIjse8BRURkRl0GHEZmRQX+iAqIO7uCMREUGjaAoIiKiwBAWIQkQJCwJhJB9IyF7d6fT+1Lb+/vj3vfWe997b1V1p6t6O5/nydNVd337Jjnvud9z3nNICAGGYRhm9GAM9gAYhmGY/MKGn2EYZpTBhp9hGGaUwYafYRhmlMGGn2EYZpQRGuwBZMPYsWNFfX39YA+DYRhmWLFmzZomIUSNvn1YGP76+nqsXr16sIfBMAwzrCCi3X7bWephGIYZZbDhZxiGGWWw4WcYhhll5NzwE5FJRG8R0ZP29ylE9BoRbSeih4kokusxMAzDMCny4fF/CcAm5fsdABYLIaYDOALgmjyMgWEYhrHJqeEnookAPgDgXvs7ATgfwF/sQ34H4PJcjoFhGIZxk2uP/6cAvg4gaX+vBtAihIjb3/cCmJDjMTAMwzAKOTP8RPRPABqEEGv6ef61RLSaiFY3NjYO8OgYhmGGNj2xBG57YgP2Huka8Gvn0uM/A8CHiGgXgD/Bknh+BqCSiOTCsYkA9vmdLIRYIoSYL4SYX1PjWXjGMAwz7EkmBTbub8Pmg234xJJVeGlrI/Y0W4b+D6/uxv2v7MK+I90Dft+crdwVQtwC4BYAIKJzAXxVCPEpInoEwBWwJoOrATyeqzEwDMNkwxNv78c5x9Wgoiic83s9/c4BHDe+DNNqSvHdJzfi/ld2Ofte3fk6AODdH12Ge1e+izOmV2PB1OoBH8Ng5PHfBOArRLQdluZ/3yCMgWEYBgDQ2N6LGx56C5/77es5v1dPLIF/f/BNfPyXqwDAZfT1MR1s68FFs8flZBx5qdUjhHgBwAv2550ATsvHfRmGYTKRtNvPvvleS87vtelAGwDgcGcUbT2x4OMOtgMAZtWV52QcvHKXYZhRTTKPfcfX72sFAEyoLEJPLBF43K1/ewcAMGt8WU7GwYafYZhRTVKx+43tvTm914b9lsdfEDKQSAZPOHvtgG5lcW4KG7DhZxhmVJNUDHBHbzzNkUdPe491/cOd0bSGHwAeuW5hzsbBhp9hmFGNKvUkksk0Rx4da/e0OBNLa3cMPbH095p/7JicjWVYNGJhGIbJFarjHUvkRu8/2NqDy+/+h2tbOlkpZBCsCje5gT1+hmFGNarkkkl+ycT6fa342fPbPNvbfTJ4GjvSGH4zd0YfYMPPMEOaP6/eg/qbl+JIZ3SwhzJiEYrUE0uk5Je9R7oQT/RN+vnQL17G4ue3ZjWBSI9/7qRKXHrieNe+sJFb08yGn2GGMH941WqZ+l7zwNdrYSwSiuGP2wa7ob0HZ96xArc/vblP15L2/verduFwgEdfFDYBAN1RS++/7pyp+Nkn5rmOYY+fYRjkL9N89KHGc+O2xt/Ubr1hvby9qV/X/M7/bcQNf3rL+a7+/RWGLbMbjVs3Ng0DIcNt6EMme/wMwzA5I+ny+C1jLKWaTJ53Q3tPoAx3qM3y+B98bTcuXvySs73Q9vh7HMMPGJrhDxvs8TMMw+QMl+G3PX45AZgZMmtO+8EynHnHct998sxb/7betV0a/l575a7po+ezx88wo5jc+n0M4E7nlBq/9PjNNJ63XPjVGfUvvRA0ZxSELLPbGw+eXD59+uT0gz5KOI+fYYYBIo/1ZEYbbo/fMsZyAgilya450NbTr/s5Hr9t+PVbfPMDs3HNmVP6de1sYY+fYZhRjVqyIdYHj39nYwcAoK9yvAzu9satNwV9cikMmzldvAWw4WcYZpSjSj2JZBK//ce7+NS9rwFIH9zd0WAZ/vrqEt/9FCDUpTT+VHBXJZzjVE6ADT/DDG1y7PkxbqknlhD4zv9tdL6n8/gb7AVYY0r6VkGzMKRJPdrfcTp5aaDIZbP1QiJ6nYjeJqINRPQde/v9RPQuEa21/8zN1RgYZqTACn/uSKYp2ZAuqyehyULZkknqyfXiLSC3wd1eAOcLITqIKAzgZSJ62t73NSHEX3J4b4ZhmKxwZfVoJRpMg3DbExuwcFo1LjnBXVZBFnTLNAEQAWpsPlNwd1h7/MKiw/4atv+w48Iw/YAFn+zYdqgdb+xq7tM5utSjEjIJ97+yC194YI3rzQDwLvbq1jpqSY9e/7tz0jkdjV9ftTvMNX4iMoloLYAGAM8JIV6zd/2AiNYR0WIiKsjlGBhmJMAeU3ZctPgl/LPdyDxbEsIt9ah2WF1ctXavuyevTPmUE0dX1N3EpcvO79czdMKmAdMgRerR9w9zwy+ESAgh5gKYCOA0IjoRwC0AZgE4FUAVgJv8ziWia4loNRGtbmxszOUwGWbIwp5+igOt3Tjrx8uxZ4AL1rmqcyaTLg9ctcGtXe7SynrOf7e2kEt+1/8OTZMQMmhkBndVhBAtAFYAWCSEOGDLQL0AfgvgtIBzlggh5gsh5tfU1ORjmAzDDGF2H+7CnuZu7GzqHNDrJrQibaohVpWfqKb/y/IOUgLq0gx/VywBIYQnMStkuA3/iJJ6iKiGiCrtz0UALgKwmYjq7G0E4HIA64OvwjAMYyG19L7WyM9EUivLrBp+vzo+6rHqT13jTySFZ7IALI8+ZBpKrR6vFJRrcpnVUwfgd0Rkwppg/iyEeJKIlhNRDaw3oLUArsvhGBhmRMAVG1IGtr/tEVdsacDf3tyHn181D6t3NeO95i5s2N+G2XXlqXsk3FJPUJMWazzu4K4ss6zSHU3YC7lS1wkZhLBJ6OwN8PhzXJkTyKHhF0KsAzDPZ/v5ubonw4w0eP1WiqTjYffP4//cb98AACy+ci6uUALAU8amVt7Gk25pRjXmuuGXE5B8K9DfCABb/tH+Di2N33AmMn2tQD40fi7SxjDMsECvnKnyzt5WFEXMrK7To0kynb2pbBxd4++JpYx9XE/n1IK7MXtC+st1C3GwrQfX//EttHTFPMHdkEHuAPIgaPxs+BmGGRYkbMPqJ/V88BcvZ30d3fCrBj2uZfWox3qlHndwV3r8BSET48sLAViNWrzBXcOVsunV+NnwMwwDgDP5U9k3Rxvc1YOwbjnHncffq+1TSTVtcQedQyahokga/l5PsbaQSa5GK3r3rRGTzskwzNHBwd2Uth/LojaOvphKRff41cybRDKpST3pPH7re9KReqyfYZNQW26tS21s7/V4/Kadzinx9tsdxumcDMMMHGz3lZo4WXj8x3/7Wby41X/hZ3dUz8dXdHxN41ffDvQ3DSfmIDSP3zBQGDZRVhhCQ1uPr8avGnd9AVc+0jnZ8DPMMIA9fiWPP8tqmKt2HPbdrks98nKRkJVpo16/b1KPu0F7bVmBJfX4ZO2oco6u8acrBT1QsOFnmCGMNAHJIW75W7tj+OKDb+JIZ/SornOwtSewymXiKPL4tzd0OJ91wy8pMA3Ek0lE46n96aQe+T2pZfVIj722rNDW+N2ETHIHd3WPnzV+hmGAoe/xb9zfhqXvHMA7+1r7fY1DbT04/UfL8D9/3+K7X0oqiSzz+FXH+cI7X3Q+6zV1JJGQgVhCuCaW3jTpnAmP1CP79Fo3ri6NWBOhj8YvvXoin+Aua/wMwwBDv9l6yhvvf8bN4Q7rbWHF5oYM93A/i6BnE7T4TQ/uSiIhA4mkcP0OauBXX5krJwIhLK8/5mT1GM71oomkJ0Bjrdy1jvFr9MKGn2EYAEM/uCu93v6WUwBShjpojktp6m4D7FcPB0j1vNUnhiCpJxIyEI0nA2MI+n3VCSIhUrEBKeOEDQOxhPd6lsZvHeOn57PUwzCjHBkYHOoaf2pxVf89fsfwB0xzQaUR/GrkqNfTYwaBUo9ppE0D1e+rXjeRFK6sHgAIhwjxhPBMGKaSx68a/h9fMQeTqoo80k8uYMPPMMOAIW73HaN4NIZfpjUGevwBWT1Bhl+iv4Wk8/hlaeXrzpmG+upiZ1/YJM+bhXpdSyJye/yhQI+ffD3+j8+fhJVfz08pMzb8DDMMGPoef98N/9PvHED9zUvRbGcCZcpgCirLHCj12BOJvj+dxi8nhbqKQowpiTj7isKmT1lmXeqxyj3I+4ZNq+a+/usY5O/x5xM2/AwzDBjaZj/lhUf7oPHf+/K7AICdjVaqZUrq8ceZXLL0+KVJ1SeK9FKPtS8SMlyB15KCkGdSSyjlHRJ2NpC6CjdsGq51ABIhgLD0+Aep/CobfoYZBgyXrJ6+1NGRnn1qgVOWUk9APr2OvGxfpB45KYRNw6W1F0VMz3ViySQKQlZF0ISwsnrUVbehNCtwzTTB3XzAhp9hhgFD3O73S+qRjrvu9AZNcskAjd/PqwZSWT36mAIXcIVSwd1IyHCtAyiJhDxB2nhCoCBsmVAruCtcC7PCaYw6Sz0Mw2Rk+Bj+PgzU/qVSQV07Lz7gcL00giRTVo/U+K87Zxqm1ZS46u+rREJGqnyDmVpkFTLIXtyVuo+w0zcLQorhTyZdXn44FGxe5QSh1+nJF7nsuVtIRK8T0dtEtIGIvmNvn0JErxHRdiJ6mIgima7FMKOV4VKywdH4M2TYqCQcw299l0Y3OLgrG59owd2Aeya0iWLOxAoURUx0BBh+VaYJm4ZjlMOmlXevZ/EASEk9dlaP6uUHtVAUEE7KZz4Wa/mRS4+/F8D5QoiTAcwFsIiITgdwB4DFQojpAI4AuCaHY2CYEcHQNvv9y+OX9tvQ1ioEzXGpevzWAQ+s2oVvPbYetz+z2ff4uDamkEEoCJno6PWXetTCaW7D7/X45URX6JJ6NI9f+Xz1wmMxzi7VDKQM/ogL7goLWRkpbP8RAM4H8Bd7++8AXJ6rMTDMSGGoB3eDcux1Nu5vQ0uXlb6Z1KQe+X3vkW7M/ObTnnNTHr913Lce34AHXt2Nt95rCRzTkc6oEwMIhwxETMNX6jEIUGOxanvESMjy+FWJSU4CESn1CIFYUrg8eNXwz5lYiWvOnAIAmDim2HkbyMdiLT9y2oGLiEwAawBMB3A3gB0AWoQQ8snvBTAh4NxrAVwLAJMnT87lMBlmyDPE7b4jfWSSei77+UpMqynBshvPdX4nuVJX/R39ArbxPgaQ9zR3Yd73nsMlJ4wDYKVrFoQNdAVIPaZWKlna5IhpIGy6PX5d6knaHr9abkGdBEIm4fNnTcXlcyegtrzQOW+wyGlwVwiREELMBTARwGkAZvXh3CVCiPlCiPk1NTU5GyPDDGVI07+HKtlk9ci3lh2NndZ3zeBnimMElWwI4pn1BwEAz244BMDywCOm4avxJ4Xm8ZuUknpCXsMv9X4p9cTtrB63x+/2/okItXYv3qqSMACgvSeW1e8y0OQlq0cI0QJgBYCFACqJSL5pTASwLx9jYJjhTFD9moGmob2nX+dl443rMpAezM00uUmDH1SvX5fL9cNCJqEgbDqLtHRUjd+glNQTNg2ETPI0ZQdSHv8197+BZZsbAjV+PW1Trgpu7R5hhp+Iaoio0v5cBOAiAJtgTQBX2IddDeDxXI2BYUYK+fD4X9jSgNN+sAzLNx/q87l6Bo0funwj8/IzZfPo94gF1OPPVNUyYnv8QXEINbUyZLizesKmgZgyfvl7ynTO/a099hjc13DGpmXvVBVbhr8nln0wfCDJpcdfB2AFEa0D8AaA54QQTwK4CcBXiGg7gGoA9+VwDAwzIshHcPfN3UcAAG/v6XszlVTJhmBD1utpeShcP/XfcX9LN57fmJqE9IYnOn6LodRNYVvjD0KVaUyDnMBrJGQgbJKrVEQ86Tb8ftdQjX1Im5TUOkCDQc6Cu0KIdQDm+WzfCUvvZxgmA5ShjMFAoteT7wvZpHPqk4Lj6Sf9pZ6rfv0qdh/uwvYfXIqQ4qkH3SNkEqApJ1UlBWjq6HX2R9KUUXB5/CbBdIK7hJBhaE3Z3VKP8zspQ1OlnuKI+7jqQTb8vHKXYYYB+dD4pZSSrsZMECmjnEbq0WQN+TvpE4DkveYuAHCqd8r9QRp/2GfcMogKpLJ6glAXXBlEXqkn4ePxa9friqUCx6r3Xxh2G/7KYjb8DMNkIB8evzRsQStO05HMIrjr1fjtnwHB3WLbWDa0Wx57prUCfsMeoxjYsGmgIJ3Hr626NQzV8JN7AVfCX+rpUhaHpfP4I2nKOeQDNvwMMwzIR3BXZqr0x/BLY3ykK4r6m5di6boDnmP0HP9MGn9xgaVEN9pSTaaUUb9nVKVIKiF7BW4Q6u9tGuSsqo2EDKd/blILMOtSj5ox5Db8OV0y1WfY8DPMMCAfwV1pvE3FYPXEErjzua2BzUsk0ijvaLBy9O9avs1zTG/cukaqNo+ex+8+XnrJTe1uw+8n9Vx20njfrCA1iBo2jbQLp0xDD+5anyOmgTHFEQgBtHTHEI0n8dBr74EIOFbp0gUAnUrrRnUiKYp471tXUYgFU6oCx5NLhtY0xDCMmwwNyAeShI/U8/tVu/DzZdsQMQnXnz/Dc05DWw/GlhZ45Bc/r1xKPdLAylMSSbfnLymypR6vx+99GBHT8MQIgFTapDwmncdv6lKPUqunpsyqs9PU0Yu7lm/DI2v24lMLJmPm+DLXNboDPP6isNfwr7rlgsCx5Br2+BlmGJCP4G7MR+qReeZ+JRQOtfXgtB8uw+LntzqThjTefsZZSj16b11d8pHIyaTR0fj9q3MClpEVInihlHUMeTR5FV3qUYO70vA3tvdi1Y7DOHFCOb774RM9ZZXVCVDNjhpsTV9naI2GYRhf8qHxS4/aLzvGD2mQl21qcAyenCD8OnFJqUcaZ6FJPfpbTZu9qlXeR9p7vzz+cMhAUghPgLesMCVqmEZ6jV9fwKUWaRtbahn+A6092NHYgTOn11hxgDTxkGyf42AwdEfGMIxDXvL4bYOqOrHZ3lf31v167zpSj1aNM8jjb++x9HJZ1iCdxx8xrSYqpHngqodPRB6Nv7bMWyoZAAwDvh7/Vx95G7GEwCxb4tED4ZXFqfRRNvwMw/SLXDViaeuJ4ct/estVK0Ya1GxvpTZH1zV+P+Ms8/hlmmRK24frp0S2SGyzJ4B0ZSEitsevo+fPqx7/s18+G7/57KnOd2/JhtQ55YXucOhx48pcvwsATBxThKU3nJW6xiA1WckGNvwMMwwYaIf/Ny+/i8fW7sd9K3c629JlzfiRWlUsnJW7Ej/j3JuQGj/s89z3DZrcZAVLp2RDUniynEyDIIQ3+0nX9NXvx1YXu/LrPemcRiq4q75J3P7RkzC7zuvxnzuzBhMqi5zvmWoHDSac1cMww4CBTud0LqcYtFgivQFOh27ofbN6YlLjtwyiPCOl9fvft8P2+NV76MFmg6xx61dI5/EXhk2XHGN40jltjd+0rjGjthRFEROfOC3VH0R/S1AJh4aux8+Gn2GGAUG2ePWuZry4tRE3Xjyzb9ezf6qmyU/qSdcZMNUrQHjeEtKnc6bOs37C9VOlvDDkaP3qhNStlVY2iJw3AhXd49eDu6rhd5dsSL2ZSAP+9/93tmeManBXr3GkTwRDiaE7MoZhHIK84St+uQp3Ld+etvNVIinw6Jq9buMsvIFcp959lh6/er2gWvsqcoxSIsoU3AWA8RWF6I4lEEskXffojnkNvyX1AF+6YAZK7VW/usevTwRhrSKnhEhZuWtPDkTeLB51stBrHPWn2F2+YMPPMMOAINlddoCSFSj9+N0ru3DjI2/j4Tf2pL1HJq096HghsjvHSfWU98lQqwcAxtkdqzp7466Jxs/wq59TjVIMV6aNNPzkePPBzVLUWj1BuD1+93F6htFQgg0/wwwDpMnriSXwiSWrsH6fVTNfrkyVhcz82H240zk3HbFksAH2I6F46tm0Q5R5/NIoZ6rVA8BJo/zp89tc9/BKPe7PTk/csImVXz8Pq795IYCUXl9ZZE0Grh65uuFXavUEob5RhNPk9A812PAzzBDGSZm0jeLGA214dWczbn1sPYDUytSGtuCWie12j1l1MVNK408Zq4Sj8Wdn+dUSCdlkAkmpR67ylWfonr+KXDh1/yu7sK+l29kuJxGJGpg1jFSbxMKQgbLCsHMdyaQqq8aOKsfoq3DNLDx+1fAHlbM+ferg1ONJR86Cu0Q0CcDvAYyD9Xe8RAjxMyK6DcDnATTah35DCPFUrsbBMCMBfXWrNFGy+mQ6j18GR12GX17HT+PP0uWXxtXK48/cQlBKPbEAj9/vElVaw5KIaVXJ1Pvmqr8HUer3K9A0/klVRbj50lm4fO4EAFp9Hk2Td+SgNFp9uuAuALz1rYt8C7QNNrnM6okDuFEI8SYRlQFYQ0TP2fsWCyH+J4f3ZpgRRcobdgdlK2zJIp3hl6UPVMlCJj66s3r6JvU4Hr/IbrJIlXPwL9Xg5/Hrhr/ALo/sl9Xj97kw5NXdrztnmuu733kAPMHdTPiVsx7sFotB5EzqEUIcEEK8aX9uh9VofUKu7scwI5lUzrv1k7Ttje3BUk+HLfVkcspTwdqUAU6n+rg9fu+B+mQgYwz6Aiy9PLPKGK1Tlex45Q3uuj9ffPw4AH3rJqanX6q1etLhFyge6uRlpERUD6v/7mv2puuJaB0R/YaIxgSccy0RrSai1Y2NjX6HMMyIR099TK27ssse2N6zbE/oh18evB8y997Pe/cTO9Tgrt85ejC5S6lVr04UCU3yUTmmstD1Xdba0ds46h7/XZ+chzduvdBn1MHoaffyGWequSMDxEN5pa5OzkdKRKUAHgXwZSFEG4B7AEwDMBfAAQA/8TtPCLFECDFfCDG/pqYm18NkmCGNI4fYBlN6uKkMmeBzZckDnzR+X42/z1IP/CcL3SvvVNoSutYAJAT+vHqP76KvsaUF+K1STyfI4/cWZzOdjKBs8Xj8jsaf3kzK2MBQrs2jk9OVu0QUhmX0HxRC/BUAhBCHlP2/BvBkLsfAMCMBPdNGvgnEEl55Rkd6/EIINHdGUVkU9rw5AKrG38fgrq3xW31pg1Mu1e+qkX/o9few+WA7ptWUeO4hG51LZEMTPbjrlnr6Z4B1+56SetJfT2r7/WlSP1jkbKRk/Yu6D8AmIcSdyvY65bCPAFifqzEwzEjBCYRq27Px+KWB3t/ag1O+9xz+94Xtvnq6TOf062TlR1KZKOJJ4Vklq3vwaltCNSdfBqb95KpIyHBlzkjDr3v8plZuoT+Yhv8CLJn7H4ScmIZTHn8uPf4zAHwGwDtEtNbe9g0AVxHRXFj/hncB+EIOx8AwIwI9uCtF976kYO5p7gIALNvcgAVTqq3LHIXUo5Z2SCQFisKm83YBAFHN8HdHE06qZUyJNMecqp1ewxk2yZUmKVMje9NIPUY/DbCelePk8Wfw+LPJ9x9q5MzwCyFehn9MiHP2GaaPpIK7usafvTwjDbFf2mHQtdK1fFRLNsSTSa/HH3ef2xmNo7wwjNbumMvjl4bfT6EJG9l5/Oqv1N9SCUHpnBmDu/Z+1vgZhhlQUsFd66fU+GWLwyC7r8o2sbg0/IavQfer1ZOucqaa/plIpuoGSaIJdzC3J5ZETVkBWrtjrrLKMafzl9dwGga5DK+cXNLn8XvHmo4/XLMAJQWmx3BXl0YQMQ2nvEMQ8rzh5PEPn5EyzChEGmgZvNUrZ0ov/XBnFL9ftcsT5FUlFcfjN8nRjlwlG3zSKuXE4VexU83jTySTjjcukcZdCIHvPbkRAFBeaBlRNdVTTiBB9lo1yPIeL2x1p3gfTXD3zBljMW/yGM9558+qxcs3n4fq0vTZQU5wdxhp/Gz4GWYIowd1ZQBW2igpmWw60IZvP74B6/e1uc5Xs2yi8ZTUk8rqkfcR0FshAspk4OPyJ11Sj19w19q/s6kT97+yC0BqpbFfwbgge60aVKnxN2orlekoPH6/+8hr1pYVBhydIiX1DB9zylIPwwxhpLmVXrg09AYR6m9e6jle176jcW8QVc9esa6vfE76ePx+K3OVt4B4Qnhq0vj1CEh5/N59QT0F1Px6fXKRqN56fzV+vSxzX89jj59hmAEh1ZbQ+p4pe0cvlqamVPbGvcFd+Um9rir1xNNIPU7z86RAdyzhePMSachV797x+ONej99vMgDgG9wFgB9fMcf5PDB5/P3MBrI9/Wwb2AwF2PAzzBAmVcDM+ikNsZ4qKdEnBtWLlmUOQiZ5YgF+AV31en5Sj9zX0WutDNbr6shJRzXo5UWWyLB6V7Pnen6TAeAOmhZFUp/LClKCxdEEdyX9Nfwyfz+bngRDBTb8DDOEcfL34Q686s3GJR7Dr0wQUgYKGeQq2bD1UDuu+d0bvtdIBgSV1eOkYa/WKlH6efxlttRz94odnusFOcxBHn+JYvhpED3+6tKIfd9+nT4osMbPMEMYXeqRXmWQHq57nTE/w68FIb/+l3VYu6fFc08gZdwTSWB7Qzvqq0tS0oY2yVTZBtA0yErfjCfwyOo9KFckoPEVmYOlOuoCrkKX4U99dvfL7fMtrGv088Q7PjYH84/di/cd61tvckiStcdPREVENDOXg2EYxo1T8l563lLqCZBF9PLI6iIqmfuuZvUkhbfOjyvQa+870NqNC+98yUnLBLxvAbINZLEd5P3Lmr342l/W4efLtjnHnHBMOa6cP8n/lw3A9MnqAVKVOoHgevx9uk8/F2BVFkfw+bOnDukeuzpZGX4i+iCAtQCesb/PJaIncjkwhmG8pRqkYQ+SevQVvKrUI8siq3nxiaTwLM7SSzFY51qTxt83Hkrt094uZNOR4oi7dPKG/VaK6acWTMbxdeWYMa7Ud+xB+BVpA9yTwEAEd4dTVs7Rkq3HfxuA0wC0AIAQYi2AKTkaE8MwEmdRlfVV5vEHGX69MJoqCUmph5DS+IUQnsnCLfXIn9Y2tZCax+N3DL+lIOv290sXzgARoaCPDUuCNP7CsImQQfj3c6cNSB5/fyeM4Ui2fwMxIUSrtm34hLAZZpiiB3elAdaLlEl03V2dCGQQNiGEEiz2lmNQM0JTweSE/TOpHOc+saIoDNOwDDsR0Gq3fJRIo7r+L28AACAASURBVJ2uo1XIIE8DFdUTV3voFoYMbP/hZbhp0awByeMfTR5/tsHdDUT0SQAmEc0AcAOAV3I3LIZhAG9bwr6mc/o1N0lqWTu6xu8n9egdr9SxSMoKQwgZhJBJiJgGWrrchr8wC8NvGORZD6Aa8ohP3R7A23qxP/Q3q2c4kq3H/58ATgDQC+CPAFoBfDlXg2IYxkKVZADFEAdl9aTJ41ePUdcH6GmUqvSTEMH3UycI0yAUhU1ETAMhw0DENFyriIlSWn26+vYGpTfAaolkt+E/+uDucArOHi0ZPX4iMgEsFUKcB+DW3A+JYRhJ0AKuoJz3uK7xB3j86sIsr8bvPhZIST0A8PK2JswYV+p6cygtCIGIEA4ZCBlkefVKOR31mmk9fqK0HrtaviEohXMYtb4dNDIafiFEgoiSRFTho/MzDJNDPAu4MpRsiHry+L3Hx5PCVYlTP8LVD9fnDePT972GiqIwPnrKBGdbWaFlSsImwdRKKetkMvzpPO9wQMrlQGj8o4lsNf4OWJ20ngPQKTcKIW4IOoGIJgH4PYBxsP79LhFC/IyIqgA8DKAeVgeujwshjvRr9AwzwhFaVo8u5ejo0o6fxp9ICqVtojcF1K8ss15Ns7U75pqE5Ipc2SM3nXGPpJkUMtnsoAqYLsOf/hIZKYmkb7U4EsjW8P/V/tMX4gBuFEK8SURlANbYE8dnASwTQtxORDcDuBnATX28NsOMClIav/UzU5G27mgc9//jXXzq9GMRNg1fjT+hyDtCCI9spH5Pp/Grk1BtmVWzPmxaHbPSGf6CcHqPX7JgSpVnf1Bf24HI4weAB645DVNr+rbOYDiSleEXQvyOiCIAjrM3bRFCxDKccwDAAftzOxFtAjABwIcBnGsf9jsAL4ANP8P4ojdiyeTx3/vyu+iKJhBNJPGvZ0zx9fjjSeHk5yd98vjVySWRJqaQ8DX8ZP+xjPvY0gI0dbhr56fz+KUBf/3WC5wSzipBEpLaZ/doDP9ZM2r6fe5wItuVu+cC2AbgbgD/C2ArEZ2d7U2IqB7APACvARhnTwoAcBCWFOR3zrVEtJqIVjc2NvodwjAjHq/H75/NI5ErbH/41GYs+tlK/+CuYuwTyfRZPel6+TYozVDGlVs1eK5430RcemKd4/HX+dTmSbeASxrt2rJC39r7QX1tB6I652gi2/j3TwBcLIQ4RwhxNoBLACzO5kQiKgXwKIAvCyFc7YGE5cb4/ssSQiwRQswXQsyvqRkdszDD6OhN1gPS933Z3tDhSD2q9BJXs3p8PH6/ssx+7DrshPtQW255/NeePQ0fe99ERGwDXeyjl6tjkUZavgVkCswGevwD0Gx9NJGt4Q8LIbbIL0KIrQDSdyAGQERhWEb/QSGEjBEcIqI6e38dgIa+DZlhRg9qMTUgs8evE0skrUVVimVMZtD43cHd4GsfUco3SKlHIo27rKcztabEs8+6l/VTVtrM5K0Hra4diJINo4lsg7urieheAH+wv38KwOp0J5D1N3EfgE1CiDuVXU8AuBrA7fbPx/s0YoYZTWhSTyaNXyeWEAibhksKWbP7iGMcM2X1pOsq1dmbyvSZN9ldklh68MURE5u/t8h1fz+Nv6QghCNdsYz6vGkQPjCnDuO0Xriu4C5b/oxka/j/HcAXYZVqAICVsLT+dJwB4DOw0kDX2tu+Acvg/5mIrgGwG8DH+zRihhlFpKpzulfuZks0nkTYJJeeqq6oPRqpJ5pIYnptKZ7/yjmefVKSKQqHPFq9X8ZPiV3YLZPNJiLc/clTPNtZ4+8b2Rr+EICfSc/dXs1bkO4EIcTLCE6pvSDrETLMKEYa5X/saEJDW0+fPf5oIolIyPT04lWvr16SyFvLJx1BzUvmTR6Dv288hO2NHZ59vobflnr6q8/zAq6+ka3GvwxAkfK9CMDzAz8chmFUpN091NaLD/3iH74rd9PlzMfiSURMCpRQkkl3GeawabibrWfoIxtUV+fTp08GAJw9Y6x3vAFSD9D/cgsD0XpxNJGtx18ohHCmbiFEBxEV52hMDMPYqAUVDmoe/8mTKnH1wmPxzr5W/PYfu3zPjyWSCIeMwGqeSS24GzYo63ROINjwlxWGsem7i1Dos1jLzyMvlYZ/ADx+lnoyk+382klEjrBGRPMBdOdmSAzDSHS7q2ruBSEDHz1lYtq6ONFEEmHTCJQ/3tjVjMNKdk44ZLgyeTLFFNIFUosiZtayS8nRGn7lEbDHn5lsPf4vA3iEiPbb3+sAXJmbITEMI9ENv6rVp3Lfg89/t6kLx1QWoaMn7rt/1+Eu13dd6kmX1QMMXPMS6fEPRKN0tvuZSevxE9GpRDReCPEGgFmwiqvFYPXefTcP42OYUY2nSYrijctKlek83K2H2nHyxIqs5Q+P1JPB4w8K7vaVVB7/0dfSZ48/M5mknl8BkO+BC2GlY94N4AiAJTkcF8Mw8C5rVxdwyUqV6Yx6Iilw0oSKrCWXkGlAjeeqHr/q3cu3jYHqWpWSevz3nzuzxtOZS2WgirSNFjJJPaYQotn+fCWs0sqPAnhUyc1nGCZHeKWe1IaIY/jTG7rZdeVZG+iwSa63DFXvD5nk3L+8KIymjt6BM/yR9Br//Z87Le35HNztG5k8fpOI5ORwAYDlyr5s4wMMw/STdJUzpdSTyZuvLo1krXt7NH71fkoEtaJIpl/2z8ounFqNf37fROd7iaPxcx5/PshkvB8C8CIRNcHK4lkJAEQ0HVbfXYZhcogu9cST7px7ILPOXuRT5TKIsGm47uGSepTKmFJ26W9w96FrTwcAPLJmLwCgNMtaPUG48/j7d43RRFrDL4T4AREtg5XF83eRegc0YDVgZxhmAHlm/UE0d0bxyQXWAihPAbWkaogza/yA5QFnyseXhE1y1fD3ux+QKr42UHr60adzcnC3L2TTc/dVn21bczMchhndXPeHNQDgGH7d53dr/HZWj230qksirpx8lSztvi31pFI/VY9f7X4l3yJqytJWbsmaTMHdTHBwt29wP3qGGcLo2ZQJH6lH2rmxpV4jLA1iXwy/qwNXwt/jl03d66sHZgF/6YBq/AMypBENG36GScODr+3GB+96OS/3aulye+ufWLIKzZoHH/eVemSQF1h328Wu4wtClmeurwcIwsrqSX0P0vgb2noAAMdWp+rsHw3FkaPT+KX0VFEU9u36xbjhzByGScOtf1uft3vtaOx0fX91Z7PnmO0N7c5nR+pRjKXep1Y2Ns+2qGcoy6yeQ7bhrx87MB6/LN3cX5mmvDCMl752HuoqC11vJow//IQYZoiwq6kz4zFCAJecYLWpDmeRxy/72wr/DqceIqbh8vITSeFIJ7FkEqdMrnRl8kyuGhjDLyuMHo0+P7m6OG3dIiYFe/wMkwXJpPDkrO9v6cb48sIB6/jU1hPLeExZYQhVJREAVkE1IJPhtzzpbD3+sEnOoi0hBOJJgYljirD3SDcOd0SxzG66svlgO1btOIziyMCYkGzqDjEDR86mRyL6DRE1ENF6ZdttRLSPiNbafy7L1f2Z4c+yTYdwwrefQVfUv8BYPolpjUz2NHfh/bcvx0+XbRuwe8jOWAYF18gpjoQc+UV63unmHcfjT2P4v/mB2c7nkGk48QA5hAmVViuO1u4YiAhEhNl15fjXM6dk/qWyJNtVyMzAkMv3ovsBLPLZvlgIMdf+81QO788Mc378zBZ0RhN4r7kr88E5Rm9I0tjRCwB4cUvDgN2jJyoNP2GHT+cqACgMG06hNkceSWP568fK4Guw5VflkZBBjtQjK4FOGFPke95AYhiEsEn9bsTC9I2cPWYhxEsAvNEphsmSbHXpfKAbfultZypb3Bekxx9PCly0+CXfY4oiphN8TaVz+hv+j50yET/5+MkA0ks9araOtdjL+ix/54mVuTf8gOX1s8efHwZjfr2eiNbZUtCYoIOI6FoiWk1EqxsbG/M5PobxoEs9sjhZptaEfaEn5t8lS6UobDpSTzjDyt0r3jfRyfJJt3JXDdaaRir1Uy3Ilg8ioeCGMczAkm/Dfw+AaQDmAjgA4CdBBwohlggh5gsh5tfU1ORrfAzji27gCbbH38fm5+mQHn86CsOm85aRqR6/6smnezEJKfqKoXj8+gSTC+67ej5+8JETAViGn+vs5Ie8ZvUIIQ7Jz0T0awBP5vP+DNNfYlrPWulBb2vowFVLXnWKjh0N2Rj+gpDpBH6lwQ4q0qZ68ukWcKkThEHkGHyp8ZsG4ekvneUsshpILpg9zvlsGX62/Pkgr4afiOqEEAfsrx8BkL/VMcywZSC96v4S18agfl+18/CA3EMGd9MRNlOGWTri0lbqMonqyffN47cOVrOHZteVZxzb0WJp/Dm/DYMcGn4iegjAuQDGEtFeAP8F4FwimgsrxWAXgC/k6v7M8EdoksNgEtc8/kQysx7fV7Lx+EOmgVjCSm+V3nE2Uo805nUVhTjQ2hN4nEGp5y7lrYFqtpKJ2rJCVBZH8nKv0U7ODL8Q4iqfzffl6n7MyEX3tgeDmKbxD2RQV5KV4TdUj19W5ww+ViJH+4OPnAgC4XP3v+HsC6uG31ClHtvjN/Nj+O/59ClcbiFP8FNmhjy5MLJ9HkNS9/hzYPizkHqsPHvrs2G4PX7dPJuGN7hbWhDGebNq8cT1ZzgTg6lJPQlH6kl69ueSyuKIU6WTyS38lJkhj25088HG/W2uFcMej38ADf/9/3gX97+yC4BshJI+ECuDu6ZTldPfI1ezcaTUIyeDORMrUVEUxuHOqKvOfiRkIBpP4lcv7sBx48us67DwPuJgw88MeaTHv72hAyUFJuoqcrugSAiBy36+UhtDZo//QGs3isJmn3Xq2/5vIwCgtqwAZYVhTylmFdNH6gnK6vHz+FX5R3r2qrxSaFfz/NHTm32vw4wMWOphhizStEpDd+GdL2Lhj5bn5F6/enEH1u+z2khv2N/m2a97+Hp6JwCcdccKzP3uc1nXvtdp6Y6hrDDli82ZWOE5JmSkqmemgrv+13Pl8cMbqE0mvdsKQ96UzXxp/Ez+YMPPDHlyHdwVQuBHT2/GP9kNV376vFV4TTWI+hj8PH55zEvbmgAArV0xvLO3NetxRONJl+E/c/pYzzEhgzwGO0jqUdM05XD93gLU4G6hT2P2fGn8TP7gv1FmyKPLLANNbzx1/YOtPXh+k7XOMGySkyOvj0GfCIQQTtG0zQesN4Zz/2cFPviLvnXvUhuphHxc+ZBSL18qNPIw3f67V+66K3oCitRjGE4PXSn1uK7DUs+Igw0/M+TJtcffq9TIabdr4k+vLUVPLOl4xXrAVff4E0nhaO0N7b2IxpM40uVfX/+bj72DB1bt8t2nevx+VTfDJuETp04CkGp7GJjHr8o6Ph5/Uin98Oo3LsAbt14Y4PGz4R9pcHCXyQvvNnViXHlBvxp35DqrpyeeSqPsslMqq0si2J5mDF7NXzjXaWjvxbq9Lc6+RFI4xjOZFPjDq+8BAD6zsN4zFj+P/7Pvr8ep9VX44h/fxKn1VTj7uBpceepk57ggJcbPYLvkn2TquAq7EJufxx9mjX/EwYafyTlCCJz3Py9g4dTqPtW0capE5jiPX82fl4uoxpYWuI7Rx6Cv3O3ojTtvBw1tPa5uWrFEEqZhedK7ld4CXdG4ZwKpUCphSo+/IGzgA3PqcMb0i3wzhgLTOX1mBHVTQivvDPgHd1njH3mw4WdyjjRu/a1pk+uSDarHLyeB6lK3gdWzeHSDrRr6xvZedEdTx/fGk46EstHOGCICFv10Jd5r7gIpZRJqylITTkh5SwAQmCYalM7pJxW5A77elbkFPlIPa/wjD57KmZyjBk/7Qywpcmr8/Tz+6hLN48+Q1dPeYy32Ko6YaGjvdZVfUCeNLQctw39MRZHTWUzV6NU3Dbk9U4zDWbmbhX32y+pxpXP6SD2s8Y882PAzOac3ixo06Ugkkr558wNBTyyBl7Y2Od+lxj+2zO1de7J6NOmnrdvy+CdXFaOjN47mzl5nX1SZ+GTAVx4PuCcRP48/06Sn2+W7P3kKLpxd63usn/euSkJ+wV32+EcebPiZnBPk8b+68zDqb17qCoT6EU+KnBn+bz++Houf3+p8D/L4M2X1SI//lGOtpnKPr92vnJsae2evdVx7r38DedXjN7M0/LrG/4E5dbj36lN9j/WVfzLm8bPhH2mw4WdyTpDhf36jlS//yg5/7V+aO8vw913q2dfSHdi0XLLlYLvre7ddn2espvFnyuqRGv+iE8ZjRm2pa/XvE2v3o/7mpWjriaEz6m/wJep9ZVA1XdtEQMnj95Rp8+K7NkD1+EN+WT1sJkYa/DfK5Jze+FFKPf30+M+4fTku+MmLvvv2t3QD8Bq1jl5rrFUlenA3ZXwTSeHJ6pHSTUmBiROOcTct+clz1hvFe4e70Nmb/lkUKV2u5NAyZTVJLz6b5vS+KZ7s8Y862PAzOac3oIm4NFOBZsVZPJV06eRHG+h9dsNBvP/25Vi5rdFj+A939CISMlBW6G4wLo3vloPtmPaNp/DClkYAwOIrTwaQknoKw6bnXIlpEDo0iUdPG1XHIz3+REaPP3vD7O/xpzf8rPGPPHJm+InoN0TUQETrlW1VRPQcEW2zf47J1f2ZoYOUevpqP9QWgFHF44/2MUtIL5r2+rvNAKxibHoBssMdURSFTZQUuA2glHo27Ldq76zefQQGAWPsFEsp9RSFTdfqW5VYIukq9WwtnHIf6zb81s9sg7vZSD1+3rsaI/Dbzx7/yCOXHv/9ABZp224GsEwIMQPAMvs7M8KRUo8ehJT2OMiQS3lFD+7q0lFDe0/aipgykLqjsQM3PPQW/u9tK/BqEiGie/ydvSiOmJ6FTHIsPcrbS8gwHEMtpZ7iSCjQ4++JJdHZm3BSJgtDBkq0xiOqkXU8/j4Gd7M99vrzpmd1TogXcI04cvY3KoR4CUCztvnDAH5nf/4dgMtzdX9m6CClHt1xjCYsA94REPCUXnY8kUQsnjJ+6kSx+3AnTvvBMix5aWfg/Y/Y9e1XbG7AE2/vR0O7lWrZ1hPzeLPS4zcMcuW0y3TOI12pWvmGkZJBpNSTzuPviSXQGY2jtqwQgCWrqNKK2tD8tPoqZ2FWRsOfdq9FgU/Q9quXzMSu2z+Q8VyTSzaMOPI9lY8TQhywPx8EMC7oQCK6lohWE9HqxsbG/IyOycg9L+zA2T9e0adzpNSjSxEdtrHsDEhtVHu/RhPulbCSd5s6AQAvb29CELKxid7gpKmj1xM0bmzvdQKsqlGWWTytWv592DaoUuopjBiBhr87lkBnb9zJ1VcN/3998Hj83/VnWL/LTefh/n89Net0Tkk6x//pL53lxCP6Cmv8I49BK9kghBBEFPgvWgixBMASAJg/f/7gN11lAAB3PLM54zHd0QQiIcMxXNKz111TGeiUE4COI/UkdKnHq/frko0qB0mDr3rrANDYHnUWbEnae+NOiWLVS77/lV348NxjnLcHOT65+KmtOw7TsKSjIMPf3hNHLCFQUyoNv4Ei+62ipCDkdMKaOKYYQPZ5/Nn855haU4qpNaVZHOmFDf/II98e/yEiqgMA+2dDnu/PDBBBxiiZFJj97WfwX084Mf1AqUfKIx0BKY6qx68a/gvvfNEpsyAngQJFlnl87T4s25T6p/Xom3txwU9ewEOv73GlaTZ2uEsrSKTHr2f8fPOx9Z5SyzI43NIdRXHYBBEFavyHO3pd1y8Mm84k41dvxwnu9rOjV3/5+qKZ+O8r5ijjYMM/0si34X8CwNX256sBPJ7n+zMDRFeALn/Y9ohl6WEgWOqRi5mk1LP5YBue23jIKUomjX0i6S3Z0Gjr9HLyUD3+Hz21Gb9Yniqq/NQ7B7Gj0ZKEZtSW4gtnT8WkqiI0tfd6PH4AjjHWDX8iKdCivTXIksWHO6IotytrBnn88tmcOKECxRETX7tkpiP19Pisdcg6uJt2b9/5j3On45/nT0pdvw/BY2Z4kMt0zocArAIwk4j2EtE1AG4HcBERbQNwof2dGYZ0+xhMADjU1gPA7SWmsnrcx3Y4Hr/18ztPbMTnf78aD71hTRpSV48nBKJxt/GTqZ4t3ZYxLbCzcKLxJA619zgBXJ0xxRHcctlsXHpiHZo6en1/D2nAdYljX0s3mruirt9NTg7xpHBKKgd5/E22xz+uvAAbv7sI586sdQy/3ziyrdXDOijTV3Km8QshrgrYdUGu7snkDz+JBLBaFwLu5h0pqSe1bdWOw9h12KpOKT1+aeB2NnZCCBEo9QBwgr2ttvQiV60ebO2BECkjq1Ns5+fXlBagN55EY3svPnbKRNz2oeMx77vPIZ4UjhwU0jz+9p442nvimDimCHuPdHuOqcjk8XdYk5Sawllt38tvEVa21Tkl7Jcz2cL1+Jl+4SeRAMBB2+OX0suKLQ1Oxovqud7457XO5wOtVh6+fDPoiiZcxi7uI/V89jev4/9ddJyTZdNtTy57W7qQjm6t+mY0kUR1aQRlhWGETEI8KVBZbBlwOXl95vRjcfKkSnz1kbcBABMqU4Y/rHj/0vCXBHQZk29D5crE8PmzpyIpgE+dPtlzvNq1i2EGEl6ZwfSLQMNve/yGQVi/rxWf++0b+PXKdwG4q1SOsT3ds2aMRXcsgR89vdl5i+iKxl2ThCX1uA3//tYefO0v67DT1u6lQd/f0uM6rkgrQSBlJbVUgpRbZBZRlb0aV0ot02tLccrkSud4Nd8+7OPxmwbhKxcdh7/+x/td95aTRU1poeveX7pwhiNVqUyqKgIAXHR8YNYzAO/KZIbJBHv8TL/QNek33zuCyqKw4/G3dcc86ZPxpMBPn99qNzJP4ANz6rD443Nx5h3LselAm2P4O3sTrkkiXZG213dZawR77HP32cZVUldRiJ1NnVg4tRqrdh7Gp08/FoDb8BfbWTZyshmjST2mQTimssg5/njF8KslHyqKU9r+DRfM8IxVTjpqzf101FUU4Z3bLkZpAf83ZQYW/hfF9IsHXt2FuspCTLNzwz/6v68AAM4+rgYAkBRWbrvOT5/fBsAyfuWFIURCBk6aUIEDrT1Ou8LuWNxVkTKWFIja302DfIOdctLYp0k9E8YUYWdTJ+rHFrv6/arGtzji9rZl/R0p9ZgGBa6w9fP401FWEHJV4Mx4fECgmGGOBpZ6hjBXLXm1z6tk88WzGw7hgp+86EnrVBc4NWsev2RSVRHae2KOUasqiaC5M+rUwu/sdWv8ajrnwwHN2uUbyL6WlMdvEDCu3JJVdK95jNK/Vm+6UlUis3pSHr/K+IqUVKNm/pSnMfwy5lFTnp233xcmjLHeRj40d8KAX5sZmbDHP4Tpb3PyXOEXZGzujKJYCWYe6YpifHkhDrb1YO8R/0BrUdhETyzpBDmrSi3DH7Nr83RHE67GJ/GEQMzW+Et9MmYiIQMN7T3Y09zlknpKIiHH4KcrhnbuzBrXPo/Hb2fXXHfONOw50qXVzPcGd/2YMrYEWw61O6t2B5LaskJs+f4iz+plhgmCDf8wIBpPIuJTZCsbZODvaBfhCCEw9RtPebbrWv+B1h4sOmE8lr5zABuVLlSuY+wArPT4q0sirlo8ndG4qyhbQ3uv08ykOOz9J3vm9LFYvrkBZ9lvR5GQgWg8iZBJTkqkLucAwCUnjENZYdgzKVQUuT1+qePffOksAO7+u+pzTWf4pf5fW14YeMzR4BccZpgg2PAPAw629mBydXG/zr3xkbfx1zf3ZVWFMR1B7RM7o95A7KzxZXhxayPW72v1PUeWSS63a9HrUktXNIHGjlR2jizEBrhLM1x79lSEDPIUeZs6tgSbD7YjbBppewD86jPzXd//+4o5eGTNXieoq9fql+j5/c98+SzcvWIH5kyoCLyXlMBqswzsDgX+8/zpzloLZmTB74bDgCDJJBv++ua+Pp8jhMCsbz2N+15+19kWlL55zwvbsW6v28CPKYlgem2pp66NTlmBrfErfWYrisLoisZxwE4LPXGCu42hKmecO7MGX180CwVayqb0qsOmkWpLmEXG4z/Pn4Q/f2Gh891ZlZuh9eGs8eW466p5TjaQSmlBCHMnVeLUKVUA4GQVDQduvHgm7rpq3mAPg8kB7PEPA/a2dGc+KAN9kYs6euPoiSXxvSc34pozpwAIrs3z7IZDeHbDIde2McWW4V+7pyXtfWQwtFoxmNWlEexs7HS0+l9cdQruf2UX/mlOHWbXlbvKPshMG+lNf+2SmdjX0o15kyrx0tZGhM1UdaD+rIGSgVu90XpfeOe2iyGEVVbhtg+e0G/JjmEGEv5XOERRF+Uc0BYl9Yegmvd+HOn0eupBHr8fY0rCgZJGpZLrLksbqDn1chLY2diJ4oiJY6uLcduHTsD8+iqUFIRcHr/skiXr8swcV4YffuQkJ1UzZBo4d2YtAGDB1Kqsxy+Rkk4sg8efDiKCYZBVspmNPjNE4H+JQxQ12ClLHvSFB1btwk+f3+p8f31XsysomQ41DXPN7mbsbOzom+Evjnjyzz948jE4vq4c/37ONGebNPzqJCH1/mc3HsT4ikJPUFrV12WHrHo7/jG1pgQAnCyjsGlg4bRqbPvBpThlct/bOzsef8BzO3lisKbPMEMZlnqGKF1KjfqgRiXp+NbjG1zfv/DAGtxw/nR85eKZGc9t7kwVOPvYPasAAH/8twXOthsvOg6XnDgeFy9+yfd8y/C7/2nNqC3FXVfNw7JNKVlI9cwl1bbe39IVQ3mGxUtS6rnlstn48LwJTqMRGZQNOz/759/I6/gVSdvxw8u4KBozbGHDP0TpVDT1jj7INOlYF5Blo9OcQeo5prII9dUlnmPu/Zf5iCWSGF9R6FnMJPPp1TcBvxRENbXyX8+oTztOafgLw6bLo5fB2KPtHKWWXNbh5iTMcIYN/xAgkRRo7oy6ygiohrZ9gAx/tobwSKd3xa06EXVGOpMZwgAAEt5JREFU4756dW15AeZMtIqZ6R6/NPwyn36ctoI1YhqIJpKYZss1v/rM+3DJCePTjlNthq4yq64MlcVhfDWLt5t0nGSnZ07vZ8tChhmqsOEfAvzP37fgnhd24K1vXeSkBKrB2I5+aPx+ZOOldvTG8cTb+z3b1YVa58+q9T1XrYRZrhl+6cnX2gb/i+dNd+2vKongYFsPFk4dize/dZGrRWIQQYuWygvDWPvtizOen4kPnnwMZteVYXpt2VFfi2GGEoMS3CWiXUT0DhGtJaLVgzGGfCKEwGq7iqQfS9cdAADsb02lbUqPv6okkpXU09wZDaxgKZErUdNx1/JteMdHEuq0x/P2ty92moHrqKUM9OBuqRPILcSm7y7Cvyysd+3/wUdORE1ZAWrLC7Iy+kB+5BY2+sxIZDCzes4TQswVQszPfOjw5verduOKX67Cis3+veWlAZO17IGU4R9XXpgxuNvZG8cp33sOP3pqc/qB2HZy6boD+H8Pr/U95FCrf+polz35pKssqXr8Xqkntc/vGhfMHoc3br3QVQWTYZjcwOmceUDWrDkQYFSl47rfZfgtQzuuvCBQ47935U587ZG3ccJ/PQsgc1E3Kdd88Y9v4m9v7fMtbywbmXzvwye4tnfFEgib6XPR03n8XEuGYYYOg2X4BYC/E9EaIrrW7wAiupaIVhPR6sbGxjwPb2CREkw4oPaLNMAHlBW6nXY6Z21ZATp6464FXY+v3Ye9R7rw/aWb8Miavc72E46xyhvsPpyqbaPSrsUKDvv0pd3Z1IEr50/Cpxa4Swvc88KOjKtfCxXjXqJ59ZwFwzBDh8Ey/GcKIU4BcCmALxLR2foBQoglQoj5Qoj5NTU13isMI+RiLN1uJpJWn9kmuwm3+kYgu1cdU1kEIVLSz76WbnzpT2vxx9fe89wnkRTY3tCBc/77BQDAJxdMdtWvb9ckI7niVdLaHUNTRxRTa0qcGjf69dOhnqMuvJpUVYQpY73pnwzDDA6DktUjhNhn/2wgor8BOA2A/2qgEYDsF6tm6jR3RnH+T15AUdh0grf7FY9/X0s3xhSHUVtmFRxbtrkBx9eV47V3LTlHNdonT6xATyyJrmjc5e2fcEw5Fkytdr63dsdQf/NS53tDew8AK2Vx1Y7DeOh1azKRC6Eun3sMHlvrzfCRrLrlfMTiAvtbu9MGr1d+/fzAff1h1vgy7Gzyf6thGCYzeTf8RFQCwBBCtNufLwbw3XyPIxt6Ygnc/Og63HjxTEyq6l9ZZCAl9aiG/5UdTWjpiqEFKfmltTv1ed+RbkwYU+Rkw9zw0FsAUo23VQNfWRxBW08MXdGEq3yyDLaWFYTQ3hv3xBga2qzJY2djB6769avOdln64M6Pz8WZM2rw1UfeBgCPvl9XYXV+mlxdjNOVCUby/ctPxDGVA19//qkbzvK8PTEMkz2D4fGPA/A3WwoIAfijEOKZQRhHRv6xvQmPrd2P1u4Yfvu50/p9HenRt2sev8qEyiKnbyxgefzTa0o9zT1WbrPiHXqd9OKIia5owqXby8VS73znEvz3s5tx94odrnOe3XAQO5s6PQXVJtuTnGGQk49fEDLwxjcvzPI3tshVCWI/GYphmOzJu+EXQuwEcHK+79sfDCO4VkuPbaT19MOmjl5878mN+P7lJzqZLdLId/bG0RNL4O09LWjU9PXptaXYYGf/CCGw70g3zjmuBice465HL1sWqucLAEXhEA53dLm2q2MrLfDWvVmxpRErtjRiQmWRa7ta20YuvBpXXpixdg7DMMMDTudMg+wx69eI46ZH1+H6P77p2b7kpZ14fO1+Ry8H4DQk6eiJ4+E39uDKJa/iruXbXedNGVuCpo5e/PCpTbjgzhfRHUtgQmURqrUerRMqi3CxVspgWk0JSgpMdMcSaOxIvUmovXDnTa7E9Fp36YG5k6zyCvtaunGxLSHpyBRNv9aFDMMMT9jwp0FKNH7ZLJsPtGPvEW+DFOllt/fE8eZ7R/CJJascj7+jN473mlMSzXil/6qUVJa8tBM7Gy39/uzjrGwmWf53y/cX4eWbznN56F+7ZCZuWjQLxRETnb0Jl8evLqg6fWo1nv/KOfjoKRMAAHd87CQ89sUzcLJt/C8+YTwevvZ0PP8Vd4KVbHKuL8hiGGb4wv+b0yDTH/06MB1o7XYaaKuU2dLIXcu3e7z6jt44DrWlAqzVpVZ9GgAoirj/KhZfebLjoT/4+dNxpDPqLIKqVloVfmTeBBSGTRRHQuiOxtGkaPxFEe+8LsstyEDtL66ahyNdUae4ms7cyZX44MnH4CsXHee7n2GY4ceoNfwPrNqFbz2+ARu/e4lLEpEc6Yzirfes1oG6x98VjaOtx6pQuae5y5XxEwto0ydr7qjXarEloNPqq1BS4JZSptekasSUFoSc6paAVe9eIssfF0dMdEYT2Hao3dnnt1r2i+dNQ1VxGB862fL8J1UVp81YKgiZ3HeVYUYYo1bq+eWLOwHAE2SVXPHLV/Dom9aq2J6YZcyXbz6EHz61yamp09QRxVk/XoFXtjc55wW1ODxpQgX2HenG7sNdjnTT0hXF5u8twoOfX+CZfKbVBi94mjU+NSmUOBq8dX5nNOHIN2pte0lByMRnz5jCK2kZZhQzag2/LFQZVPlyR2MqT76l29Lol647iHtX7sRuLZVyze4jAKwia1sPdfheb3x5IY50xdDQ3ov59Vb/1+vPn4HCsImwabiCp1fOn+T7FiKZMa4Mp0yuRCRkOCtk5fllhSE8et1CvHzTeVlXuWQYZnQxaqUewzaYrV2Za91LSaa5sxdJYS2+UpE9cU//0TLf88+cPhZXLZiMh1fvAWAVydx1+wdcx6iG/44r5mQc0yPXvd+V9y/PP25cGUKmEVg6mWEYZvR6/Lbhb+mO4d6VO3HFPa9g6boDqL95KVq63IureuNJdEcTaLYngJXb3Ia/vSfupH4CQHVJBGdMt1ayvn9aNf7wbwswd1IlXvjquQgZhEUnejtLpfPw/TANcun+spYPd4tiGCYTo9bjlzXEWrpi+P7STQCAAruV37MbDiJkkGvh1sG2HqcJ+eaD7a5rHe6M4l2lhMKEMUWYf2wV/rH9sMug148twfYfXuY7nqPNk5etDC+Y7d8di2EYRjJqDb8j9Sj1cWR5hCfXHUChUjwNAJZvbsA+n7x9ANjR0IHP/fYN53tSCCfbhrKMoR6t4b/khPFYduM5mMYeP8MwGRg1Uk8iKVB/81LctWwbgFTFTBm4BeAEZl/b2exqLg4A33tyo6sevZoVs7Op07UwqzuacCYRtY5+OvwycPoCEbHRZxgmK0aN4bdKEAM/X24ZftmURA3ubm/oQGHYQDSRhBBWWeKXvnae6zqV9qKtyT657w9cYxVy64klXa0GsyFdS0OGYZiBZNQYflnr3iCCEMJZlatXyfynOcegwF7Vesb0sZhc7Tbw84+1UjHVxiI//MhJePDfFmCsXVenO5ZwpKQsHX4U2+UVZteVZziSYRjm6BjRhn/xc1vx4bv/gZXbGvGxe1YBsCSaz93/hhO43d7ozrufXJWqLV9pr5Adq5RIOG6cJadMq0kZ/k8umIwzpo9NGf5owpGCsi0hHDIN/PHzC/Dgvy3o8+/JMAzTF0Z0cNc0CG/vacFn7nvd2dYVTeCFLVZNeyI4BdEktWUFOPu4Gry4tdHR6ZfdeC7ae2J47K19uPbsaairKMSpU6rw65Xvus6VC6auPHUSzppRgyvnT8INF87Ierzvnza2X78nwzBMXxjRhn+mUtrAjwf+dQG+v3SjKz1z5vgyTBlbgob2Hpw8ySqtUFEURkVRGNefbxnxzyys90hEgDXRbPruIhSEDBgGZbUQi2EYJt8MitRDRIuIaAsRbSeim3N1n9njg/XydbddjDNnjMXPr5qH82elct9PmlCByuIIbrl0tm+RM0lQ+mVRxOQOUQzDDGnybviJyARwN4BLARwP4CoiOj4X95o4JlW3/lefeR8un3sMAOCsGWOdblLHjSvDbz57qnNcyMzukRSEDJw1Yyx+/S/zB3DEDMMwuWcwpJ7TAGy3WzCCiP4E4MMANg70jQyD8MOPnITJVcU4c8ZYnDezFotOrPNd3fqX6xY6wdlsICI8cA0HYhmGGX4MhuGfAGCP8n0vAI8FJaJrAVwLAJMnT+73zT65IHVuJGT41skB4FTMZBiGGekM2XROIcQSIcR8IcT8mpqawR4OwzDMiGEwDP8+AJOU7xPtbQzDMEweGAzD/waAGUQ0hYgiAD4B4IlBGAfDMMyoJO8avxAiTkTXA3gWgAngN0KIDfkeB8MwzGhlUBZwCSGeAvDUYNybYRhmtDNkg7sMwzBMbmDDzzAMM8pgw88wDDPKoGw7RA0mRNQIYHc/Th0LoCnjUaMXfj7B8LNJDz+fYIbSszlWCOFZCDUsDH9/IaLVQgguphMAP59g+Nmkh59PMMPh2bDUwzAMM8pgw88wDDPKGOmGf8lgD2CIw88nGH426eHnE8yQfzYjWuNnGIZhvIx0j59hGIbRYMPPMAwzyhixhj9ffX2HMkT0GyJqIKL1yrYqInqOiLbZP8fY24mIfm4/r3VEdMrgjTz3ENEkIlpBRBuJaAMRfcnePuqfDxEVEtHrRPS2/Wy+Y2+fQkSv2c/gYbu6LoiowP6+3d5fP5jjzwdEZBLRW0T0pP19WD2bEWn489nXd4hzP4BF2rabASwTQswAsMz+DljPaob951oA9+RpjINFHMCNQojjAZwO4Iv2vxF+PkAvgPOFECcDmAtgERGdDuAOAIuFENMBHAFwjX38NQCO2NsX28eNdL4EYJPyfXg9GyHEiPsDYCGAZ5XvtwC4ZbDHNUjPoh7AeuX7FgB19uc6AFvsz78CcJXfcaPhD4DHAVzEz8fzXIoBvAmrPWoTgJC93fk/BqvE+kL7c8g+jgZ77Dl8JhNhOQXnA3gSAA23ZzMiPX749/WdMEhjGWqME0IcsD8fBDDO/jxqn5n9+j0PwGvg5wPAkTLWAmgA8ByAHQBahBBx+xD193eejb2/FUB1fkecV34K4OsAkvb3agyzZzNSDT+TBcJyQ0Z1Pi8RlQJ4FMCXhRBt6r7R/HyEEAkhxFxY3u1pAGYN8pCGBET0TwAahBBrBnssR8NINfzc1zeYQ0RUBwD2zwZ7+6h7ZkQUhmX0HxRC/NXezM9HQQjRAmAFLPmikohk8yb193eejb2/AsDhPA81X5wB4ENEtAvAn2DJPT/DMHs2I9Xwc1/fYJ4AcLX9+WpY2rbc/i929srpAFoVyWPEQUQE4D4Am4QQdyq7Rv3zIaIaIqq0PxfBin1sgjUBXGEfpj8b+cyuALDcflsacQghbhFCTBRC1MOyK8uFEJ/CcHs2gx1kyGEA5jIAW2Fpk7cO9ngG6Rk8BOAAgBgs3fEaWPriMgDbADwPoMo+lmBlQu0A8A6A+YM9/hw/mzNhyTjrAKy1/1zGz0cAwBwAb9nPZj2Ab9vbpwJ4HcB2AI8AKLC3F9rft9v7pw7275Cn53QugCeH47Phkg0MwzCjjJEq9TAMwzABsOFnGIYZZbDhZxiGGWWw4WcYhhllsOFnGIYZZbDhZ0Y0RJQgorXKn7SVWonoOiL6lwG47y4iGtuP8y4hou/YVUKfPtpxMIwfocyHMMywpltYpQeyQgjxy1wOJgvOgrUY6CwALw/yWJgRCnv8zKjE9sh/TETv2LXnp9vbbyOir9qfb7Dr9a8joj/Z26qI6DF726tENMfeXk1Ef7fr198La8GXvNen7XusJaJf2WXD9fFcaRdFuwFWEbBfA/gcEfGKc2bAYcPPjHSKNKnnSmVfqxDiJAC/gGVsdW4GME8IMQfAdfa27wB4y972DQC/t7f/F4CXhRAnAPgbgMkAQESzAVwJ4Az7zSMB4FP6jYQQD8OqELreHtM79r0/dDS/PMP4wVIPM9JJJ/U8pPxc7LN/HYAHiegxAI/Z284E8DEAEEIstz39cgBnA/iovX0pER2xj78AwPsAvGGVB0IRUoXfdI4DsNP+XCKEaM/i92OYPsOGnxnNiIDPkg/AMugfBHArEZ3Uj3sQgN8JIW5JexDRagBjAYSIaCOAOlv6+U8hxMp+3JdhAmGphxnNXKn8XKXuICIDwCQhxAoAN8Eqp1sKYCVsqYaIzgXQJKw6/i8B+KS9/VIAY+xLLQNwBRHV2vuqiOhYfSBCiPkAlgL4MIAfwyosOJeNPpML2ONnRjpFtucseUYIIVM6xxDROlg9Zq/SzjMB/IGIKmB57T8XQrQQ0W0AfmOf14VUyd3vAHiIiDYAeAXAewAghNhIRN8E8Hd7MokB+CKA3T5jPQVWcPc/ANzps59hBgSuzsmMSuxGGvOFEE2DPRaGyTcs9TAMw4wy2ONnGIYZZbDHzzAMM8pgw88wDDPKYMPPMAwzymDDzzAMM8pgw88wDDPK+P9yHl4uAKiaegAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=1000, print_every=100):\n",
    "    scores_deque = deque(maxlen=print_every)\n",
    "    scores_history = []\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        start_time = time.time()\n",
    "        env_info = env.reset(train_mode=True)[brain_name]      # reset the environment  \n",
    "        agent.reset()\n",
    "        states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        while True:\n",
    "            actions = agent.act(states, add_noise=False)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            scores += env_info.rewards                         # update the score (for each agent)\n",
    "            agent.step(states, actions, rewards, next_states, dones)\n",
    "            states = next_states                               # roll over states to next time step\n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "        scores_deque.append(scores)\n",
    "        scores_history.append(scores)\n",
    "        print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "        print(\"--- %s seconds ---\" % (time.time() - start_time))\n",
    "        torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "        torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "        if i_episode % print_every == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "        if np.mean(scores_deque) > 30.0:\n",
    "            print('\\rSOLVED!!  Episode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            break\n",
    "        \n",
    "    return scores_history\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Evaluate on trained network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=2)\n",
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 28.399999365210533\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = agent.act(states, add_noise=False)\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    agent.step(states, actions, rewards, next_states, dones)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}