{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "qQB3ABjahv-W" }, "source": [ "# DQN for Atari Pacman\n", "\n", "In this Notebook, we will explore the Fitted Q-Iteration algorithm with a replay buffer and a target network, which defines the so-called DQN algorithm:\n", "\n", "1. Take action $a_j$ w.r.t. some policy (e.g. $\\varepsilon$-greedy w.r.t. $Q_\\varphi$). Observe transition $(s_j, a_j, r_j, s'_j)$ and add it in the Buffer $B$; \n", "2. Sample a minibatch of transitions $\\{(s_j, a_j, r_j, s'_j)\\}_j$ from $B$; \n", "3. Compute $y_j = r(s_j,a_j) + \\gamma\\max_{a'_j}Q_{ \\varphi'}(s'_j,a'_j)$ using the **target network** $Q_{ \\varphi'}$;\n", "4. Update parameters of main network:\n", "$\\varphi\\leftarrow \\varphi - \\alpha \\sum_j \\left[\\nabla_\\varphi Q_\\varphi(s_j,a_j)\\right]\\left(Q_\\varphi(s_j,a_j) - y_j\\right)$;\n", "5. Update target network $\\varphi'\\leftarrow \\varphi$.\n", "\n", "We apply the algorithm to teach an RL agent play the Atari video game *Pacman* from images of the screen.

\n", "\n", "Since DQN is quite time consuming, this notebook is to be run on from Google Colab with a GPU runtime type. For the same reasons, your goal is to explore the main part of the algorithm ahead of time. Running the training takes about 5 hours using a GPU runtime to produce a mean reward/score of about 2000 per episode. \n", "\n", "\n", "
\n", "--------------------------\n", "\n", "*Many thanks to Jiahao Yao from UC Berkeley for helping me prepare this notebook!*" ] }, { "cell_type": "markdown", "metadata": { "id": "qy3nVlpn-Q4M" }, "source": [ "## Setup\n", "\n", "You will need to make a copy of this notebook in your Google Drive before you can edit the homework files. You can do so with **File → Save a copy in Drive**." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "tdaE2act-WDX" }, "outputs": [], "source": [ "#@title Jax with TPU \n", "\n", "#@markdown *(uncomment and run this block first)*\n", "\n", "#@markdown **CAVEAT:** this is currently slower than GPU, but this will supposedly change very soon: see this [notebook](https://github.com/google/jax/blob/master/cloud_tpu_colabs/JAX_NeurIPS_2020_demo.ipynb) and this [NeurIPS 2020 video](https://drive.google.com/file/d/1jKxefZT1xJDUxMman6qrQVed7vWI0MIn/view). \n", "\n", "# # get the latest JAX and jaxlib\n", "# !pip install --upgrade -q jax jaxlib\n", "\n", "# # Colab runtime set to TPU accel\n", "# import requests\n", "# import os\n", "# if 'TPU_DRIVER_MODE' not in globals():\n", "# url = 'http://' + os.environ['COLAB_TPU_ADDR'].split(':')[0] + ':8475/requestversion/tpu_driver_nightly'\n", "# resp = requests.post(url)\n", "# TPU_DRIVER_MODE = 1\n", "\n", "# # TPU driver as backend for JAX\n", "# from jax.config import config\n", "# config.FLAGS.jax_xla_backend = \"tpu_driver\"\n", "# config.FLAGS.jax_backend_target = \"grpc://\" + os.environ['COLAB_TPU_ADDR']\n", "# print(config.FLAGS.jax_backend_target)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "mmuROkmK-1FH", "outputId": "7f262a26-60cd-43ad-9ce0-84e5ef3ec7e4" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mounted at /content/gdrive\n" ] } ], "source": [ "#@title mount your Google Drive\n", "#@markdown Your work will be stored in a folder called `DQN_atari` by default to prevent Colab instance timeouts from deleting your edits.\n", "\n", "import os\n", "from google.colab import drive\n", "drive.mount('/content/gdrive')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VsGATE7T_X5B", "outputId": "514c7af7-fa02-4ebc-e072-448cbd41d865" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Gym Env: MsPacman-v0\n" ] } ], "source": [ "#@title Atari Environments \n", "#@markdown We will use the Gym Pacman environment.\n", "\n", "env_name = 'pacman' \n", "gym_name_map = {\n", " 'pacman': 'MsPacman-v0', \n", "}\n", "gym_name = gym_name_map[env_name]\n", "print('Gym Env:', gym_name)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3p_R65Q3_rhW", "outputId": "f9817700-48d5-4294-a23a-c2cc64204a1f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/content\n", "/content/gdrive/My Drive/DQN_atari\n" ] } ], "source": [ "#@title set up mount symlink\n", "%cd /content\n", "DRIVE_PATH = '/content/gdrive/My\\ Drive/DQN_atari'\n", "DRIVE_PYTHON_PATH = DRIVE_PATH.replace('\\\\', '')\n", "if not os.path.exists(DRIVE_PYTHON_PATH):\n", " %mkdir $DRIVE_PATH\n", "\n", "\n", "## the space in `My Drive` causes some issues,\n", "## make a symlink to avoid this\n", "SYM_PATH = '/content/DQN_atari'\n", "if not os.path.exists(SYM_PATH):\n", " !ln -s $DRIVE_PATH $SYM_PATH\n", "\n", "%cd $SYM_PATH" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "bJltyIhoAdQK", "outputId": "8fa2ab94-a394-4995-b132-9a32e632cd55" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33m\r", "0% [Working]\u001b[0m\r", " \r", "Get:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n", "\u001b[33m\r", "0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Connecting to security.u\u001b[0m\u001b[33m\r", "0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Connecting to security.u\u001b[0m\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\r", " \r", "Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease\n", "\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\r", " \r", "Ign:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease\n", "\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\r", " \r", "Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n", "\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\r", " \r", "Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release [697 B]\n", "\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\u001b[33m\r", "0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\u001b[0m\r", " \r", "Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B]\n", "Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release.gpg [836 B]\n", "Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B]\n", "Get:9 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ Packages [40.7 kB]\n", "Hit:10 http://archive.ubuntu.com/ubuntu bionic InRelease\n", "Get:11 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n", "Get:12 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n", "Ign:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages\n", "Get:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [506 kB]\n", "Get:14 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [66.1 kB]\n", "Get:15 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n", "Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n", "Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,372 kB]\n", "Get:18 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,699 kB]\n", "Get:19 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,816 kB]\n", "Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,244 kB]\n", "Get:21 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [237 kB]\n", "Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [15.3 kB]\n", "Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,136 kB]\n", "Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB]\n", "Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [53.8 kB]\n", "Get:26 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [870 kB]\n", "Get:27 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [46.5 kB]\n", "Fetched 11.7 MB in 4s (3,130 kB/s)\n", "Reading package lists... Done\n", "Building dependency tree \n", "Reading state information... Done\n", "59 packages can be upgraded. Run 'apt list --upgradable' to see them.\n", "Collecting opencv-python==3.4.0.12\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/50/f9/5c454f0f52788a913979877e6ed9b2454a9c7676581a0ee3a2d81db784a6/opencv_python-3.4.0.12-cp36-cp36m-manylinux1_x86_64.whl (24.9MB)\n", "\u001b[K |████████████████████████████████| 24.9MB 140kB/s \n", "\u001b[?25hRequirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python==3.4.0.12) (1.19.4)\n", "\u001b[31mERROR: dopamine-rl 1.0.5 has requirement opencv-python>=3.4.1.15, but you'll have opencv-python 3.4.0.12 which is incompatible.\u001b[0m\n", "\u001b[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001b[0m\n", "Installing collected packages: opencv-python\n", " Found existing installation: opencv-python 4.1.2.30\n", " Uninstalling opencv-python-4.1.2.30:\n", " Successfully uninstalled opencv-python-4.1.2.30\n", "Successfully installed opencv-python-3.4.0.12\n" ] } ], "source": [ "#@title apt install requirements\n", "\n", "#@markdown Run each section with Shift+Enter\n", "\n", "#@markdown Double-click on section headers to show code.\n", "\n", "#@markdown If you see some ERRORs, they are caused by dependencies which are preinstalled in Google Colab; you may ignore them. \n", "\n", "!apt update \n", "!apt install -y --no-install-recommends \\\n", " build-essential \\\n", " curl \\\n", " git \\\n", " gnupg2 \\\n", " make \\\n", " cmake \\\n", " ffmpeg \\\n", " swig \\\n", " libz-dev \\\n", " unzip \\\n", " zlib1g-dev \\\n", " libglfw3 \\\n", " libglfw3-dev \\\n", " libxrandr2 \\\n", " libxinerama-dev \\\n", " libxi6 \\\n", " libxcursor-dev \\\n", " libgl1-mesa-dev \\\n", " libgl1-mesa-glx \\\n", " libglew-dev \\\n", " libosmesa6-dev \\\n", " lsb-release \\\n", " ack-grep \\\n", " patchelf \\\n", " wget \\\n", " xpra \\\n", " xserver-xorg-dev \\\n", " xvfb \\\n", " python-opengl \\\n", " ffmpeg > /dev/null 2>&1\n", "\n", "!pip install opencv-python==3.4.0.12" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UBoSm896A5G7", "outputId": "7c85f529-9dac-404a-e452-06aa9b169ae2" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting gym[atari]==0.17.2\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/b3/99/7cc3e510678119cdac91f33fb9235b98448f09a6bdf0cafea2b108d9ce51/gym-0.17.2.tar.gz (1.6MB)\n", "\u001b[K |████████████████████████████████| 1.6MB 6.9MB/s \n", "\u001b[?25hCollecting pyvirtualdisplay==1.3.2\n", " Downloading https://files.pythonhosted.org/packages/d0/8a/643043cc70791367bee2d19eb20e00ed1a246ac48e5dbe57bbbcc8be40a9/PyVirtualDisplay-1.3.2-py2.py3-none-any.whl\n", "Collecting box2d-py\n", "\u001b[?25l Downloading https://files.pythonhosted.org/packages/06/bd/6cdc3fd994b0649dcf5d9bad85bd9e26172308bbe9a421bfc6fdbf5081a6/box2d_py-2.3.8-cp36-cp36m-manylinux1_x86_64.whl (448kB)\n", "\u001b[K |████████████████████████████████| 450kB 17.5MB/s \n", "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (1.4.1)\n", "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (1.19.4)\n", "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (1.5.0)\n", "Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (1.3.0)\n", "Requirement already satisfied: atari_py~=0.2.0 in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (0.2.6)\n", "Requirement already satisfied: Pillow in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (7.0.0)\n", "Requirement already satisfied: opencv-python in /usr/local/lib/python3.6/dist-packages (from gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (3.4.0.12)\n", "Collecting EasyProcess\n", " Downloading https://files.pythonhosted.org/packages/48/3c/75573613641c90c6d094059ac28adb748560d99bd27ee6f80cce398f404e/EasyProcess-0.3-py2.py3-none-any.whl\n", "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (0.16.0)\n", "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from atari_py~=0.2.0->gym[atari]==0.17.2->-r requirements_colab.txt (line 1)) (1.15.0)\n", "Building wheels for collected packages: gym\n", " Building wheel for gym (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for gym: filename=gym-0.17.2-cp36-none-any.whl size=1650891 sha256=af0e5507afaaa84689c21e9e9ea732b5a2b2a6709bad1c0ac7cb363431e975f6\n", " Stored in directory: /root/.cache/pip/wheels/87/e0/91/f56e44e8062f8cd549673da49f59e1d4fe8b17398119b1d221\n", "Successfully built gym\n", "\u001b[31mERROR: dopamine-rl 1.0.5 has requirement opencv-python>=3.4.1.15, but you'll have opencv-python 3.4.0.12 which is incompatible.\u001b[0m\n", "Installing collected packages: gym, EasyProcess, pyvirtualdisplay, box2d-py\n", " Found existing installation: gym 0.17.3\n", " Uninstalling gym-0.17.3:\n", " Successfully uninstalled gym-0.17.3\n", "Successfully installed EasyProcess-0.3 box2d-py-2.3.8 gym-0.17.2 pyvirtualdisplay-1.3.2\n", "/content/gdrive/My Drive/DQN_atari\n" ] } ], "source": [ "#@title download the JAX DQN_pacman codebase \n", "import os\n", "from google_drive_downloader import GoogleDriveDownloader as gdd\n", "\n", "# download the DQN_pacman codebase -- DO NOT MODIFY THIS CELL\n", "gdd.download_file_from_google_drive(file_id='1TXLk-eeKwuaxrhc7gYLhw4VE94Il6zl_', \n", " dest_path='./DQN_pacman.tar.gz', \n", " unzip=True, \n", " )\n", "\n", "# install JAX DQN codebase requirements from requirements_colab.txt \n", "%pip install -r requirements_colab.txt \n", "expt_dir = '/content/DQN_atari/'\n", "video_path = os.path.join(expt_dir, 'video')\n", "os.chdir(expt_dir)\n", "!pwd\n", "\n", "required_files = ['atari_wrappers.py', \n", " 'buffer.py', 'main.py', \n", " 'NN.py', 'env_utils.py', \n", " 'colab_utils.py', \n", " 'video_utils.py',\n", " ]\n", "for f in required_files:\n", " assert os.path.isfile(f)\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "7NXjuL0eCA9M" }, "outputs": [], "source": [ "#@title set up virtual display\n", "\n", "from pyvirtualdisplay import Display\n", "\n", "display = Display(visible=0, size=(1400, 900))\n", "display.start()\n", "\n", "# For later\n", "from colab_utils import (\n", " wrap_env_demo,\n", " show_video_demo, \n", " show_video\n", ")\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 438 }, "id": "nu7FvolKCJdR", "outputId": "57b07c23-ce69-418e-fbc4-675f8c2eeeec" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading video...\n" ] }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "#@title test virtual display\n", "\n", "#@markdown If you see a video, setup is complete!\n", "\n", "import gym\n", "import matplotlib\n", "\n", "env = wrap_env_demo(gym.make(gym_name))\n", "\n", "observation = env.reset()\n", "for i in range(10):\n", " env.render(mode='rgb_array')\n", " obs, rew, term, _ = env.step(env.action_space.sample() ) \n", " if term:\n", " break;\n", " \n", "env.close()\n", "print('Loading video...')\n", "show_video_demo()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "ZlUAzMqMQDvK" }, "outputs": [], "source": [ "#@title imports\n", "\n", "import os.path as osp\n", "import sys, time\n", "from functools import partial\n", "\n", "import gym\n", "from gym import wrappers\n", "\n", "import numpy as np\n", "import random\n", "\n", "from atari_wrappers import *\n", "from buffer import ReplayBuffer\n", "\n", "from NN import Neural_Net\n", "from env_utils import episode_step\n", "from video_utils import learning_logger\n", "\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "wxT2y4BQU1bf" }, "outputs": [], "source": [ "#@title hyperparameters\n", "\n", "#@markdown Set the seed and define the hyperparameters for DQN.\n", "\n", "# seed\n", "seed = 0\n", "np.random.seed(seed)\n", "random.seed(seed)\n", "np.random.RandomState(seed)\n", "\n", "# Q-learning & network\n", "N_iterations = 3000000 # 200 # \n", "\n", "# discount factor\n", "gamma = 0.99\n", "\n", "# Q network update frequency\n", "update_frequency = 4\n", "\n", "# frame history length\n", "agent_history_length = 4\n", "\n", "\n", "use_target = True\n", "# target network update frequency\n", "target_update = 10000 # 100 # \n", "minibatch_size = 32\n", "\n", "# replay buffer parameters\n", "replay_memory_size = 1000000 # 10000 # \n", "\n", "# buffer prefilling steps\n", "replay_start_size = 50000 # 500 # \n", "\n", "# adam parameters\n", "step_size = 1e-4\n", "adam_beta1 = 0.9\n", "adam_beta2 = 0.999\n", "adam_eps = 1e-4\n", "adam_params=dict(N_iterations=N_iterations,\n", " step_size=step_size,\n", " b1=adam_beta1,\n", " b2=adam_beta2,\n", " eps=adam_eps,\n", " )\n", "\n", "# exploration (epsilon-greedy) schedule\n", "eps_schedule_step = [0, 1e6, 2.5e6]\n", "#eps_schedule_val = [1.0, 0.1, 0.01]\n", "eps_schedule_val = [0.2, 0.1, 0.01]\n", "eps_schedule_args = dict(\n", " eps_schedule_step=eps_schedule_step, eps_schedule_val=eps_schedule_val\n", ")\n", "\n", "# video logging: default is to not log video so that logs are small enough\n", "# in units of epidoes\n", "video_log_freq = 1000 #-1" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "83DGs7KwVOqs" }, "outputs": [], "source": [ "#@title load the pacman gym environment\n", "\n", "def get_env(seed):\n", " env = gym.make(\"MsPacman-v0\")\n", " env.seed(seed)\n", " env.action_space.np_random.seed(seed)\n", " expt_dir = \"./\"\n", "\n", " # the video recorder only captures a sampling of episodes\n", " # (those with episodes numbers which are perfect cubes: 1, 8, 27, 64, ... and then every `video_log_freq`-th).\n", " def capped_cubic_video_schedule(episode_id):\n", " if episode_id < video_log_freq:\n", " return int(round(episode_id ** (1.0 / 3))) ** 3 == episode_id\n", " else:\n", " return episode_id % video_log_freq == 0\n", "\n", " env = wrappers.Monitor(\n", " env,\n", " osp.join(expt_dir, \"video\"),\n", " force=True,\n", " video_callable=(capped_cubic_video_schedule if video_log_freq > 0 else False),\n", " )\n", "\n", " # configure environment for DeepMind-style Atari\n", " env = wrap_deepmind(env) \n", " return env\n", "\n", "\n", "##### Create a breakout environment\n", "# fix env seeds\n", "env = get_env(seed)\n", "# reset environment to initial state\n", "frame = env.reset()\n", "\n", "# get the size of the action space\n", "n_actions = env.action_space.n\n", "\n", "# define logger\n", "rl_logger = learning_logger(env, eps_schedule_args)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "hUzgoBmrVh5S" }, "outputs": [], "source": [ "#@title create the data buffer\n", "\n", "frame_shape = (env.observation_space.shape[0], env.observation_space.shape[1])\n", "replay_buffer = ReplayBuffer(replay_memory_size, agent_history_length, lander=False)\n", "# channel last format of the input\n", "input_shape = (1,) + frame_shape + (agent_history_length,)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "idnILGLQVthP", "outputId": "c3577857-ddbd-40d5-dd09-fe10b0662549" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "build the Q learning network.\n", "\n", "\n", "DQN input shape: (1, 84, 84, 4).\n" ] } ], "source": [ "#@build deep Q-network\n", "\n", "print(\"build the Q learning network.\\n\")\n", "##### Create deep neural net\n", "model = Neural_Net(\n", " n_actions,\n", " input_shape,\n", " adam_params,\n", " use_target=use_target,\n", " seed=seed\n", " )" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ollgJjtYWeCB", "outputId": "105ce7e1-8da7-4e43-eb04-2b7a1ea95547" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Start prefilling the buffer.\n", "\n", "\n", "Finished prefilling the buffer.\n", "\n" ] } ], "source": [ "#@title Pre-fill buffer\n", "\n", "print(\"Start prefilling the buffer.\\n\")\n", "\n", "tot_time = time.time()\n", "\n", "##### prefill buffer using the random policy\n", "pre_iteration = 0\n", "while pre_iteration < replay_start_size:\n", " # reset environment\n", " state = env.reset()\n", " is_terminal = False\n", "\n", " while not is_terminal:\n", "\n", " # store state in buffer\n", " buffer_index = replay_buffer.store_frame(state)\n", " last_obs_encode = replay_buffer.encode_recent_observation()\n", " state_enc = np.expand_dims(last_obs_encode, 0)\n", "\n", " # take environment step and overwrite state; reward is not used to prefill buffer\n", " state, reward, is_terminal = episode_step(\n", " pre_iteration,\n", " env,\n", " model,\n", " replay_buffer,\n", " buffer_index,\n", " state_enc,\n", " prefill_buffer=True,\n", " )\n", " pre_iteration += 1\n", "\n", "print(\"\\nFinished prefilling the buffer.\\n\")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IB2c3DfjWnqw", "outputId": "1a900a29-569f-43b5-883b-62203184fc8b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1;30;43mStreaming output truncated to the last 5000 lines.\u001b[0m\n", "running time 0.014043\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2945662.\n", "mean reward (over 100 episodes) 1866.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11298.\n", "exploration eps 0.010000.\n", "running time 0.012596\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2945790.\n", "mean reward (over 100 episodes) 1866.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11298.\n", "exploration eps 0.010000.\n", "running time 0.012825\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2945825.\n", "mean reward (over 100 episodes) 1866.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11298.\n", "exploration eps 0.010000.\n", "running time 0.003934\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2945924.\n", "mean reward (over 100 episodes) 1870.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11299.\n", "exploration eps 0.010000.\n", "running time 0.009833\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946081.\n", "mean reward (over 100 episodes) 1870.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11299.\n", "exploration eps 0.010000.\n", "running time 0.015376\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946105.\n", "mean reward (over 100 episodes) 1870.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11299.\n", "exploration eps 0.010000.\n", "running time 0.002629\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946249.\n", "mean reward (over 100 episodes) 1884.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11300.\n", "exploration eps 0.010000.\n", "running time 0.014086\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946431.\n", "mean reward (over 100 episodes) 1884.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11300.\n", "exploration eps 0.010000.\n", "running time 0.017843\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946463.\n", "mean reward (over 100 episodes) 1884.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11300.\n", "exploration eps 0.010000.\n", "running time 0.003195\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946494.\n", "mean reward (over 100 episodes) 1887.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11301.\n", "exploration eps 0.010000.\n", "running time 0.003254\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946610.\n", "mean reward (over 100 episodes) 1887.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11301.\n", "exploration eps 0.010000.\n", "running time 0.011369\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946770.\n", "mean reward (over 100 episodes) 1887.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11301.\n", "exploration eps 0.010000.\n", "running time 0.016193\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2946908.\n", "mean reward (over 100 episodes) 1882.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11302.\n", "exploration eps 0.010000.\n", "running time 0.013356\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947092.\n", "mean reward (over 100 episodes) 1882.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11302.\n", "exploration eps 0.010000.\n", "running time 0.018324\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947182.\n", "mean reward (over 100 episodes) 1882.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11302.\n", "exploration eps 0.010000.\n", "running time 0.008742\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947218.\n", "mean reward (over 100 episodes) 1897.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11303.\n", "exploration eps 0.010000.\n", "running time 0.003407\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947336.\n", "mean reward (over 100 episodes) 1897.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11303.\n", "exploration eps 0.010000.\n", "running time 0.011798\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947386.\n", "mean reward (over 100 episodes) 1897.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11303.\n", "exploration eps 0.010000.\n", "running time 0.005090\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947417.\n", "mean reward (over 100 episodes) 1888.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11304.\n", "exploration eps 0.010000.\n", "running time 0.003165\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947616.\n", "mean reward (over 100 episodes) 1888.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11304.\n", "exploration eps 0.010000.\n", "running time 0.019578\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947678.\n", "mean reward (over 100 episodes) 1888.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11304.\n", "exploration eps 0.010000.\n", "running time 0.006353\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947717.\n", "mean reward (over 100 episodes) 1889.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11305.\n", "exploration eps 0.010000.\n", "running time 0.003932\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947808.\n", "mean reward (over 100 episodes) 1889.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11305.\n", "exploration eps 0.010000.\n", "running time 0.009233\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947901.\n", "mean reward (over 100 episodes) 1889.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11305.\n", "exploration eps 0.010000.\n", "running time 0.009409\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2947944.\n", "mean reward (over 100 episodes) 1889.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11306.\n", "exploration eps 0.010000.\n", "running time 0.004847\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948047.\n", "mean reward (over 100 episodes) 1889.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11306.\n", "exploration eps 0.010000.\n", "running time 0.010224\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948204.\n", "mean reward (over 100 episodes) 1889.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11306.\n", "exploration eps 0.010000.\n", "running time 0.015326\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948369.\n", "mean reward (over 100 episodes) 1888.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11307.\n", "exploration eps 0.010000.\n", "running time 0.015744\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948571.\n", "mean reward (over 100 episodes) 1888.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11307.\n", "exploration eps 0.010000.\n", "running time 0.019566\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948624.\n", "mean reward (over 100 episodes) 1888.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11307.\n", "exploration eps 0.010000.\n", "running time 0.005308\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948687.\n", "mean reward (over 100 episodes) 1899.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11308.\n", "exploration eps 0.010000.\n", "running time 0.006160\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948912.\n", "mean reward (over 100 episodes) 1899.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11308.\n", "exploration eps 0.010000.\n", "running time 0.021874\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2948961.\n", "mean reward (over 100 episodes) 1899.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11308.\n", "exploration eps 0.010000.\n", "running time 0.004867\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949000.\n", "mean reward (over 100 episodes) 1900.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11309.\n", "exploration eps 0.010000.\n", "running time 0.004250\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949132.\n", "mean reward (over 100 episodes) 1900.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11309.\n", "exploration eps 0.010000.\n", "running time 0.012770\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949183.\n", "mean reward (over 100 episodes) 1900.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11309.\n", "exploration eps 0.010000.\n", "running time 0.004999\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949226.\n", "mean reward (over 100 episodes) 1893.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11310.\n", "exploration eps 0.010000.\n", "running time 0.004164\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949368.\n", "mean reward (over 100 episodes) 1893.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11310.\n", "exploration eps 0.010000.\n", "running time 0.014174\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949424.\n", "mean reward (over 100 episodes) 1893.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11310.\n", "exploration eps 0.010000.\n", "running time 0.005653\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949443.\n", "mean reward (over 100 episodes) 1890.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11311.\n", "exploration eps 0.010000.\n", "running time 0.001861\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949564.\n", "mean reward (over 100 episodes) 1890.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11311.\n", "exploration eps 0.010000.\n", "running time 0.012223\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949612.\n", "mean reward (over 100 episodes) 1890.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11311.\n", "exploration eps 0.010000.\n", "running time 0.004716\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949669.\n", "mean reward (over 100 episodes) 1892.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11312.\n", "exploration eps 0.010000.\n", "running time 0.005739\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949813.\n", "mean reward (over 100 episodes) 1892.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11312.\n", "exploration eps 0.010000.\n", "running time 0.013955\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949851.\n", "mean reward (over 100 episodes) 1892.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11312.\n", "exploration eps 0.010000.\n", "running time 0.003940\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2949891.\n", "mean reward (over 100 episodes) 1896.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11313.\n", "exploration eps 0.010000.\n", "running time 0.003956\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950029.\n", "mean reward (over 100 episodes) 1896.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11313.\n", "exploration eps 0.010000.\n", "running time 0.014048\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950047.\n", "mean reward (over 100 episodes) 1896.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11313.\n", "exploration eps 0.010000.\n", "running time 0.001797\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950170.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11314.\n", "exploration eps 0.010000.\n", "running time 0.011868\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950274.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11314.\n", "exploration eps 0.010000.\n", "running time 0.010367\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950346.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11314.\n", "exploration eps 0.010000.\n", "running time 0.007093\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950455.\n", "mean reward (over 100 episodes) 1895.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11315.\n", "exploration eps 0.010000.\n", "running time 0.010799\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950589.\n", "mean reward (over 100 episodes) 1895.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11315.\n", "exploration eps 0.010000.\n", "running time 0.013372\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950643.\n", "mean reward (over 100 episodes) 1895.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11315.\n", "exploration eps 0.010000.\n", "running time 0.005376\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950688.\n", "mean reward (over 100 episodes) 1888.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11316.\n", "exploration eps 0.010000.\n", "running time 0.004722\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950847.\n", "mean reward (over 100 episodes) 1888.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11316.\n", "exploration eps 0.010000.\n", "running time 0.015437\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950879.\n", "mean reward (over 100 episodes) 1888.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11316.\n", "exploration eps 0.010000.\n", "running time 0.003273\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2950996.\n", "mean reward (over 100 episodes) 1893.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11317.\n", "exploration eps 0.010000.\n", "running time 0.011642\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951153.\n", "mean reward (over 100 episodes) 1893.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11317.\n", "exploration eps 0.010000.\n", "running time 0.015526\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951207.\n", "mean reward (over 100 episodes) 1893.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11317.\n", "exploration eps 0.010000.\n", "running time 0.005417\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951260.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11318.\n", "exploration eps 0.010000.\n", "running time 0.005495\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951419.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11318.\n", "exploration eps 0.010000.\n", "running time 0.015882\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951507.\n", "mean reward (over 100 episodes) 1898.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11318.\n", "exploration eps 0.010000.\n", "running time 0.008561\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951565.\n", "mean reward (over 100 episodes) 1903.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11319.\n", "exploration eps 0.010000.\n", "running time 0.006192\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951678.\n", "mean reward (over 100 episodes) 1903.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11319.\n", "exploration eps 0.010000.\n", "running time 0.011073\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951861.\n", "mean reward (over 100 episodes) 1903.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11319.\n", "exploration eps 0.010000.\n", "running time 0.018422\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2951960.\n", "mean reward (over 100 episodes) 1908.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11320.\n", "exploration eps 0.010000.\n", "running time 0.009767\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952193.\n", "mean reward (over 100 episodes) 1908.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11320.\n", "exploration eps 0.010000.\n", "running time 0.022903\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952311.\n", "mean reward (over 100 episodes) 1908.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11320.\n", "exploration eps 0.010000.\n", "running time 0.011496\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952341.\n", "mean reward (over 100 episodes) 1911.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11321.\n", "exploration eps 0.010000.\n", "running time 0.003257\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952482.\n", "mean reward (over 100 episodes) 1911.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11321.\n", "exploration eps 0.010000.\n", "running time 0.013822\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952510.\n", "mean reward (over 100 episodes) 1911.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11321.\n", "exploration eps 0.010000.\n", "running time 0.002928\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952548.\n", "mean reward (over 100 episodes) 1911.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11322.\n", "exploration eps 0.010000.\n", "running time 0.003787\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952612.\n", "mean reward (over 100 episodes) 1911.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11322.\n", "exploration eps 0.010000.\n", "running time 0.006998\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952714.\n", "mean reward (over 100 episodes) 1911.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11322.\n", "exploration eps 0.010000.\n", "running time 0.010198\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952802.\n", "mean reward (over 100 episodes) 1904.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11323.\n", "exploration eps 0.010000.\n", "running time 0.008943\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952905.\n", "mean reward (over 100 episodes) 1904.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11323.\n", "exploration eps 0.010000.\n", "running time 0.010345\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2952939.\n", "mean reward (over 100 episodes) 1904.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11323.\n", "exploration eps 0.010000.\n", "running time 0.003329\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953038.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11324.\n", "exploration eps 0.010000.\n", "running time 0.010471\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953147.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11324.\n", "exploration eps 0.010000.\n", "running time 0.011176\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953238.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11324.\n", "exploration eps 0.010000.\n", "running time 0.009241\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953281.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11325.\n", "exploration eps 0.010000.\n", "running time 0.004488\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953414.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11325.\n", "exploration eps 0.010000.\n", "running time 0.013200\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953443.\n", "mean reward (over 100 episodes) 1901.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11325.\n", "exploration eps 0.010000.\n", "running time 0.002917\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953532.\n", "mean reward (over 100 episodes) 1895.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11326.\n", "exploration eps 0.010000.\n", "running time 0.009015\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953757.\n", "mean reward (over 100 episodes) 1895.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11326.\n", "exploration eps 0.010000.\n", "running time 0.022033\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953825.\n", "mean reward (over 100 episodes) 1895.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11326.\n", "exploration eps 0.010000.\n", "running time 0.006754\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2953873.\n", "mean reward (over 100 episodes) 1895.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11327.\n", "exploration eps 0.010000.\n", "running time 0.004700\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954004.\n", "mean reward (over 100 episodes) 1895.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11327.\n", "exploration eps 0.010000.\n", "running time 0.013139\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954057.\n", "mean reward (over 100 episodes) 1895.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11327.\n", "exploration eps 0.010000.\n", "running time 0.005369\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954212.\n", "mean reward (over 100 episodes) 1886.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11328.\n", "exploration eps 0.010000.\n", "running time 0.015155\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954306.\n", "mean reward (over 100 episodes) 1886.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11328.\n", "exploration eps 0.010000.\n", "running time 0.009203\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954389.\n", "mean reward (over 100 episodes) 1886.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11328.\n", "exploration eps 0.010000.\n", "running time 0.008216\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954443.\n", "mean reward (over 100 episodes) 1887.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11329.\n", "exploration eps 0.010000.\n", "running time 0.005454\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954570.\n", "mean reward (over 100 episodes) 1887.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11329.\n", "exploration eps 0.010000.\n", "running time 0.012569\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954629.\n", "mean reward (over 100 episodes) 1887.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11329.\n", "exploration eps 0.010000.\n", "running time 0.005914\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954669.\n", "mean reward (over 100 episodes) 1876.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11330.\n", "exploration eps 0.010000.\n", "running time 0.004104\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954797.\n", "mean reward (over 100 episodes) 1876.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11330.\n", "exploration eps 0.010000.\n", "running time 0.012567\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954855.\n", "mean reward (over 100 episodes) 1876.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11330.\n", "exploration eps 0.010000.\n", "running time 0.005579\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2954897.\n", "mean reward (over 100 episodes) 1876.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11331.\n", "exploration eps 0.010000.\n", "running time 0.004418\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955039.\n", "mean reward (over 100 episodes) 1876.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11331.\n", "exploration eps 0.010000.\n", "running time 0.014024\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955100.\n", "mean reward (over 100 episodes) 1876.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11331.\n", "exploration eps 0.010000.\n", "running time 0.006360\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955232.\n", "mean reward (over 100 episodes) 1898.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11332.\n", "exploration eps 0.010000.\n", "running time 0.012908\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955329.\n", "mean reward (over 100 episodes) 1898.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11332.\n", "exploration eps 0.010000.\n", "running time 0.009611\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955422.\n", "mean reward (over 100 episodes) 1898.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11332.\n", "exploration eps 0.010000.\n", "running time 0.008998\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955475.\n", "mean reward (over 100 episodes) 1900.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11333.\n", "exploration eps 0.010000.\n", "running time 0.005330\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955646.\n", "mean reward (over 100 episodes) 1900.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11333.\n", "exploration eps 0.010000.\n", "running time 0.017301\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955834.\n", "mean reward (over 100 episodes) 1900.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11333.\n", "exploration eps 0.010000.\n", "running time 0.018087\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2955972.\n", "mean reward (over 100 episodes) 1924.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11334.\n", "exploration eps 0.010000.\n", "running time 0.013567\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956163.\n", "mean reward (over 100 episodes) 1924.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11334.\n", "exploration eps 0.010000.\n", "running time 0.018738\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956311.\n", "mean reward (over 100 episodes) 1924.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11334.\n", "exploration eps 0.010000.\n", "running time 0.014347\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956389.\n", "mean reward (over 100 episodes) 1937.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11335.\n", "exploration eps 0.010000.\n", "running time 0.007709\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956512.\n", "mean reward (over 100 episodes) 1937.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11335.\n", "exploration eps 0.010000.\n", "running time 0.011974\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956597.\n", "mean reward (over 100 episodes) 1937.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11335.\n", "exploration eps 0.010000.\n", "running time 0.008796\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956617.\n", "mean reward (over 100 episodes) 1933.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11336.\n", "exploration eps 0.010000.\n", "running time 0.002242\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956769.\n", "mean reward (over 100 episodes) 1933.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11336.\n", "exploration eps 0.010000.\n", "running time 0.015446\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956866.\n", "mean reward (over 100 episodes) 1933.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11336.\n", "exploration eps 0.010000.\n", "running time 0.009686\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2956922.\n", "mean reward (over 100 episodes) 1936.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11337.\n", "exploration eps 0.010000.\n", "running time 0.005611\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957096.\n", "mean reward (over 100 episodes) 1936.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11337.\n", "exploration eps 0.010000.\n", "running time 0.016980\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957147.\n", "mean reward (over 100 episodes) 1936.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11337.\n", "exploration eps 0.010000.\n", "running time 0.005179\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957175.\n", "mean reward (over 100 episodes) 1939.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11338.\n", "exploration eps 0.010000.\n", "running time 0.002786\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957305.\n", "mean reward (over 100 episodes) 1939.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11338.\n", "exploration eps 0.010000.\n", "running time 0.013352\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957359.\n", "mean reward (over 100 episodes) 1939.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11338.\n", "exploration eps 0.010000.\n", "running time 0.005530\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957400.\n", "mean reward (over 100 episodes) 1933.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11339.\n", "exploration eps 0.010000.\n", "running time 0.004191\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957548.\n", "mean reward (over 100 episodes) 1933.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11339.\n", "exploration eps 0.010000.\n", "running time 0.014476\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957591.\n", "mean reward (over 100 episodes) 1933.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11339.\n", "exploration eps 0.010000.\n", "running time 0.004129\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957633.\n", "mean reward (over 100 episodes) 1947.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11340.\n", "exploration eps 0.010000.\n", "running time 0.004348\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957743.\n", "mean reward (over 100 episodes) 1947.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11340.\n", "exploration eps 0.010000.\n", "running time 0.011010\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957846.\n", "mean reward (over 100 episodes) 1947.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11340.\n", "exploration eps 0.010000.\n", "running time 0.010108\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2957984.\n", "mean reward (over 100 episodes) 1949.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11341.\n", "exploration eps 0.010000.\n", "running time 0.013483\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958144.\n", "mean reward (over 100 episodes) 1949.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11341.\n", "exploration eps 0.010000.\n", "running time 0.015546\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958163.\n", "mean reward (over 100 episodes) 1949.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11341.\n", "exploration eps 0.010000.\n", "running time 0.002026\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958195.\n", "mean reward (over 100 episodes) 1938.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11342.\n", "exploration eps 0.010000.\n", "running time 0.003241\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958294.\n", "mean reward (over 100 episodes) 1938.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11342.\n", "exploration eps 0.010000.\n", "running time 0.009871\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958386.\n", "mean reward (over 100 episodes) 1938.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11342.\n", "exploration eps 0.010000.\n", "running time 0.009116\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958510.\n", "mean reward (over 100 episodes) 1937.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11343.\n", "exploration eps 0.010000.\n", "running time 0.012474\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958662.\n", "mean reward (over 100 episodes) 1937.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11343.\n", "exploration eps 0.010000.\n", "running time 0.015310\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958695.\n", "mean reward (over 100 episodes) 1937.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11343.\n", "exploration eps 0.010000.\n", "running time 0.003507\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958735.\n", "mean reward (over 100 episodes) 1937.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11344.\n", "exploration eps 0.010000.\n", "running time 0.003988\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958885.\n", "mean reward (over 100 episodes) 1937.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11344.\n", "exploration eps 0.010000.\n", "running time 0.015142\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2958949.\n", "mean reward (over 100 episodes) 1937.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11344.\n", "exploration eps 0.010000.\n", "running time 0.006565\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959010.\n", "mean reward (over 100 episodes) 1921.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11345.\n", "exploration eps 0.010000.\n", "running time 0.006359\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959126.\n", "mean reward (over 100 episodes) 1921.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11345.\n", "exploration eps 0.010000.\n", "running time 0.011457\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959168.\n", "mean reward (over 100 episodes) 1921.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11345.\n", "exploration eps 0.010000.\n", "running time 0.004506\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959268.\n", "mean reward (over 100 episodes) 1920.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11346.\n", "exploration eps 0.010000.\n", "running time 0.009693\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959435.\n", "mean reward (over 100 episodes) 1920.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11346.\n", "exploration eps 0.010000.\n", "running time 0.016064\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959462.\n", "mean reward (over 100 episodes) 1920.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11346.\n", "exploration eps 0.010000.\n", "running time 0.002846\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959557.\n", "mean reward (over 100 episodes) 1922.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11347.\n", "exploration eps 0.010000.\n", "running time 0.009467\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959803.\n", "mean reward (over 100 episodes) 1922.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11347.\n", "exploration eps 0.010000.\n", "running time 0.023837\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959875.\n", "mean reward (over 100 episodes) 1922.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11347.\n", "exploration eps 0.010000.\n", "running time 0.007047\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2959890.\n", "mean reward (over 100 episodes) 1929.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11348.\n", "exploration eps 0.010000.\n", "running time 0.001704\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960076.\n", "mean reward (over 100 episodes) 1929.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11348.\n", "exploration eps 0.010000.\n", "running time 0.019000\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960147.\n", "mean reward (over 100 episodes) 1929.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11348.\n", "exploration eps 0.010000.\n", "running time 0.007222\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960277.\n", "mean reward (over 100 episodes) 1914.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11349.\n", "exploration eps 0.010000.\n", "running time 0.013111\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960471.\n", "mean reward (over 100 episodes) 1914.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11349.\n", "exploration eps 0.010000.\n", "running time 0.018814\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960512.\n", "mean reward (over 100 episodes) 1914.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11349.\n", "exploration eps 0.010000.\n", "running time 0.004265\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960683.\n", "mean reward (over 100 episodes) 1935.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11350.\n", "exploration eps 0.010000.\n", "running time 0.016968\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960850.\n", "mean reward (over 100 episodes) 1935.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11350.\n", "exploration eps 0.010000.\n", "running time 0.016476\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2960883.\n", "mean reward (over 100 episodes) 1935.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11350.\n", "exploration eps 0.010000.\n", "running time 0.003331\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961002.\n", "mean reward (over 100 episodes) 1945.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11351.\n", "exploration eps 0.010000.\n", "running time 0.011920\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961274.\n", "mean reward (over 100 episodes) 1945.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11351.\n", "exploration eps 0.010000.\n", "running time 0.026620\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961337.\n", "mean reward (over 100 episodes) 1945.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11351.\n", "exploration eps 0.010000.\n", "running time 0.006195\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961384.\n", "mean reward (over 100 episodes) 1943.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11352.\n", "exploration eps 0.010000.\n", "running time 0.004702\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961525.\n", "mean reward (over 100 episodes) 1943.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11352.\n", "exploration eps 0.010000.\n", "running time 0.013915\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961572.\n", "mean reward (over 100 episodes) 1943.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11352.\n", "exploration eps 0.010000.\n", "running time 0.004915\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961624.\n", "mean reward (over 100 episodes) 1941.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11353.\n", "exploration eps 0.010000.\n", "running time 0.005506\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961797.\n", "mean reward (over 100 episodes) 1941.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11353.\n", "exploration eps 0.010000.\n", "running time 0.016838\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961872.\n", "mean reward (over 100 episodes) 1941.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11353.\n", "exploration eps 0.010000.\n", "running time 0.007456\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2961936.\n", "mean reward (over 100 episodes) 1942.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11354.\n", "exploration eps 0.010000.\n", "running time 0.006677\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962055.\n", "mean reward (over 100 episodes) 1942.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11354.\n", "exploration eps 0.010000.\n", "running time 0.012923\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962169.\n", "mean reward (over 100 episodes) 1942.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11354.\n", "exploration eps 0.010000.\n", "running time 0.011406\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962213.\n", "mean reward (over 100 episodes) 1952.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11355.\n", "exploration eps 0.010000.\n", "running time 0.004609\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962352.\n", "mean reward (over 100 episodes) 1952.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11355.\n", "exploration eps 0.010000.\n", "running time 0.014325\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962390.\n", "mean reward (over 100 episodes) 1952.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11355.\n", "exploration eps 0.010000.\n", "running time 0.003825\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962496.\n", "mean reward (over 100 episodes) 1953.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11356.\n", "exploration eps 0.010000.\n", "running time 0.010532\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962632.\n", "mean reward (over 100 episodes) 1953.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11356.\n", "exploration eps 0.010000.\n", "running time 0.013820\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962661.\n", "mean reward (over 100 episodes) 1953.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11356.\n", "exploration eps 0.010000.\n", "running time 0.002898\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962764.\n", "mean reward (over 100 episodes) 1953.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11357.\n", "exploration eps 0.010000.\n", "running time 0.010531\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2962919.\n", "mean reward (over 100 episodes) 1953.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11357.\n", "exploration eps 0.010000.\n", "running time 0.015564\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963018.\n", "mean reward (over 100 episodes) 1953.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11357.\n", "exploration eps 0.010000.\n", "running time 0.009960\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963120.\n", "mean reward (over 100 episodes) 1950.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11358.\n", "exploration eps 0.010000.\n", "running time 0.009997\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963243.\n", "mean reward (over 100 episodes) 1950.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11358.\n", "exploration eps 0.010000.\n", "running time 0.012285\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963376.\n", "mean reward (over 100 episodes) 1950.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11358.\n", "exploration eps 0.010000.\n", "running time 0.013377\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963421.\n", "mean reward (over 100 episodes) 1954.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11359.\n", "exploration eps 0.010000.\n", "running time 0.004734\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963532.\n", "mean reward (over 100 episodes) 1954.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11359.\n", "exploration eps 0.010000.\n", "running time 0.011096\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963572.\n", "mean reward (over 100 episodes) 1954.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11359.\n", "exploration eps 0.010000.\n", "running time 0.004130\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963606.\n", "mean reward (over 100 episodes) 1966.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11360.\n", "exploration eps 0.010000.\n", "running time 0.003706\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963802.\n", "mean reward (over 100 episodes) 1966.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11360.\n", "exploration eps 0.010000.\n", "running time 0.019220\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963839.\n", "mean reward (over 100 episodes) 1966.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11360.\n", "exploration eps 0.010000.\n", "running time 0.003617\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2963943.\n", "mean reward (over 100 episodes) 1958.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11361.\n", "exploration eps 0.010000.\n", "running time 0.010016\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964107.\n", "mean reward (over 100 episodes) 1958.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11361.\n", "exploration eps 0.010000.\n", "running time 0.016109\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964157.\n", "mean reward (over 100 episodes) 1958.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11361.\n", "exploration eps 0.010000.\n", "running time 0.004944\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964207.\n", "mean reward (over 100 episodes) 1964.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11362.\n", "exploration eps 0.010000.\n", "running time 0.004964\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964341.\n", "mean reward (over 100 episodes) 1964.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11362.\n", "exploration eps 0.010000.\n", "running time 0.013467\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964389.\n", "mean reward (over 100 episodes) 1964.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11362.\n", "exploration eps 0.010000.\n", "running time 0.004836\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964505.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11363.\n", "exploration eps 0.010000.\n", "running time 0.011444\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964577.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11363.\n", "exploration eps 0.010000.\n", "running time 0.007187\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964718.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11363.\n", "exploration eps 0.010000.\n", "running time 0.013953\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964842.\n", "mean reward (over 100 episodes) 1954.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11364.\n", "exploration eps 0.010000.\n", "running time 0.012526\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2964986.\n", "mean reward (over 100 episodes) 1954.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11364.\n", "exploration eps 0.010000.\n", "running time 0.014196\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965024.\n", "mean reward (over 100 episodes) 1954.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11364.\n", "exploration eps 0.010000.\n", "running time 0.003750\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965047.\n", "mean reward (over 100 episodes) 1952.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11365.\n", "exploration eps 0.010000.\n", "running time 0.002172\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965270.\n", "mean reward (over 100 episodes) 1952.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11365.\n", "exploration eps 0.010000.\n", "running time 0.021676\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965319.\n", "mean reward (over 100 episodes) 1952.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11365.\n", "exploration eps 0.010000.\n", "running time 0.005013\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965382.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11366.\n", "exploration eps 0.010000.\n", "running time 0.006158\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965589.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11366.\n", "exploration eps 0.010000.\n", "running time 0.020687\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965644.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11366.\n", "exploration eps 0.010000.\n", "running time 0.005642\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965699.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11367.\n", "exploration eps 0.010000.\n", "running time 0.005417\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965801.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11367.\n", "exploration eps 0.010000.\n", "running time 0.010403\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965864.\n", "mean reward (over 100 episodes) 1960.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11367.\n", "exploration eps 0.010000.\n", "running time 0.006204\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2965977.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11368.\n", "exploration eps 0.010000.\n", "running time 0.011209\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966129.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11368.\n", "exploration eps 0.010000.\n", "running time 0.014940\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966203.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11368.\n", "exploration eps 0.010000.\n", "running time 0.007170\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966261.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11369.\n", "exploration eps 0.010000.\n", "running time 0.005893\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966411.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11369.\n", "exploration eps 0.010000.\n", "running time 0.014791\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966430.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11369.\n", "exploration eps 0.010000.\n", "running time 0.001953\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966476.\n", "mean reward (over 100 episodes) 1954.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11370.\n", "exploration eps 0.010000.\n", "running time 0.004823\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966590.\n", "mean reward (over 100 episodes) 1954.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11370.\n", "exploration eps 0.010000.\n", "running time 0.011439\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966649.\n", "mean reward (over 100 episodes) 1954.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11370.\n", "exploration eps 0.010000.\n", "running time 0.006426\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966801.\n", "mean reward (over 100 episodes) 1955.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11371.\n", "exploration eps 0.010000.\n", "running time 0.014575\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966905.\n", "mean reward (over 100 episodes) 1955.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11371.\n", "exploration eps 0.010000.\n", "running time 0.010308\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2966939.\n", "mean reward (over 100 episodes) 1955.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11371.\n", "exploration eps 0.010000.\n", "running time 0.003266\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967102.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11372.\n", "exploration eps 0.010000.\n", "running time 0.015774\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967205.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11372.\n", "exploration eps 0.010000.\n", "running time 0.011172\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967303.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11372.\n", "exploration eps 0.010000.\n", "running time 0.009864\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967337.\n", "mean reward (over 100 episodes) 1960.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11373.\n", "exploration eps 0.010000.\n", "running time 0.003396\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967535.\n", "mean reward (over 100 episodes) 1960.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11373.\n", "exploration eps 0.010000.\n", "running time 0.019586\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967592.\n", "mean reward (over 100 episodes) 1960.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11373.\n", "exploration eps 0.010000.\n", "running time 0.005666\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967628.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11374.\n", "exploration eps 0.010000.\n", "running time 0.003627\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967723.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11374.\n", "exploration eps 0.010000.\n", "running time 0.009623\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967808.\n", "mean reward (over 100 episodes) 1955.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11374.\n", "exploration eps 0.010000.\n", "running time 0.008592\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967869.\n", "mean reward (over 100 episodes) 1958.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11375.\n", "exploration eps 0.010000.\n", "running time 0.005984\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2967974.\n", "mean reward (over 100 episodes) 1958.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11375.\n", "exploration eps 0.010000.\n", "running time 0.010027\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968179.\n", "mean reward (over 100 episodes) 1958.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11375.\n", "exploration eps 0.010000.\n", "running time 0.019805\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968244.\n", "mean reward (over 100 episodes) 1973.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11376.\n", "exploration eps 0.010000.\n", "running time 0.006381\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968393.\n", "mean reward (over 100 episodes) 1973.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11376.\n", "exploration eps 0.010000.\n", "running time 0.014576\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968495.\n", "mean reward (over 100 episodes) 1973.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11376.\n", "exploration eps 0.010000.\n", "running time 0.009875\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968569.\n", "mean reward (over 100 episodes) 1967.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11377.\n", "exploration eps 0.010000.\n", "running time 0.007244\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968768.\n", "mean reward (over 100 episodes) 1967.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11377.\n", "exploration eps 0.010000.\n", "running time 0.019504\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968809.\n", "mean reward (over 100 episodes) 1967.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11377.\n", "exploration eps 0.010000.\n", "running time 0.003969\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968862.\n", "mean reward (over 100 episodes) 1973.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11378.\n", "exploration eps 0.010000.\n", "running time 0.005162\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968964.\n", "mean reward (over 100 episodes) 1973.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11378.\n", "exploration eps 0.010000.\n", "running time 0.010378\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968981.\n", "mean reward (over 100 episodes) 1973.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11378.\n", "exploration eps 0.010000.\n", "running time 0.001776\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2968996.\n", "mean reward (over 100 episodes) 1967.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11379.\n", "exploration eps 0.010000.\n", "running time 0.001708\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969155.\n", "mean reward (over 100 episodes) 1967.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11379.\n", "exploration eps 0.010000.\n", "running time 0.015876\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969200.\n", "mean reward (over 100 episodes) 1967.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11379.\n", "exploration eps 0.010000.\n", "running time 0.004695\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969271.\n", "mean reward (over 100 episodes) 1958.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11380.\n", "exploration eps 0.010000.\n", "running time 0.006645\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969483.\n", "mean reward (over 100 episodes) 1958.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11380.\n", "exploration eps 0.010000.\n", "running time 0.020273\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969536.\n", "mean reward (over 100 episodes) 1958.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11380.\n", "exploration eps 0.010000.\n", "running time 0.005381\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969587.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11381.\n", "exploration eps 0.010000.\n", "running time 0.005033\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969791.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11381.\n", "exploration eps 0.010000.\n", "running time 0.020121\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969828.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11381.\n", "exploration eps 0.010000.\n", "running time 0.003663\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969859.\n", "mean reward (over 100 episodes) 1970.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11382.\n", "exploration eps 0.010000.\n", "running time 0.002942\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2969987.\n", "mean reward (over 100 episodes) 1970.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11382.\n", "exploration eps 0.010000.\n", "running time 0.012417\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970023.\n", "mean reward (over 100 episodes) 1970.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11382.\n", "exploration eps 0.010000.\n", "running time 0.003511\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970069.\n", "mean reward (over 100 episodes) 1956.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11383.\n", "exploration eps 0.010000.\n", "running time 0.004883\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970265.\n", "mean reward (over 100 episodes) 1956.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11383.\n", "exploration eps 0.010000.\n", "running time 0.019737\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970297.\n", "mean reward (over 100 episodes) 1956.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11383.\n", "exploration eps 0.010000.\n", "running time 0.003221\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970356.\n", "mean reward (over 100 episodes) 1965.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11384.\n", "exploration eps 0.010000.\n", "running time 0.005940\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970470.\n", "mean reward (over 100 episodes) 1965.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11384.\n", "exploration eps 0.010000.\n", "running time 0.011250\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970577.\n", "mean reward (over 100 episodes) 1965.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11384.\n", "exploration eps 0.010000.\n", "running time 0.010647\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970604.\n", "mean reward (over 100 episodes) 1961.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11385.\n", "exploration eps 0.010000.\n", "running time 0.002641\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970745.\n", "mean reward (over 100 episodes) 1961.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11385.\n", "exploration eps 0.010000.\n", "running time 0.014421\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970830.\n", "mean reward (over 100 episodes) 1961.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11385.\n", "exploration eps 0.010000.\n", "running time 0.008577\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2970876.\n", "mean reward (over 100 episodes) 1949.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11386.\n", "exploration eps 0.010000.\n", "running time 0.004537\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971047.\n", "mean reward (over 100 episodes) 1949.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11386.\n", "exploration eps 0.010000.\n", "running time 0.016823\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971218.\n", "mean reward (over 100 episodes) 1949.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11386.\n", "exploration eps 0.010000.\n", "running time 0.017063\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971397.\n", "mean reward (over 100 episodes) 1950.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11387.\n", "exploration eps 0.010000.\n", "running time 0.018369\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971551.\n", "mean reward (over 100 episodes) 1950.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11387.\n", "exploration eps 0.010000.\n", "running time 0.015247\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971592.\n", "mean reward (over 100 episodes) 1950.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11387.\n", "exploration eps 0.010000.\n", "running time 0.004487\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971706.\n", "mean reward (over 100 episodes) 1950.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11388.\n", "exploration eps 0.010000.\n", "running time 0.011401\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971810.\n", "mean reward (over 100 episodes) 1950.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11388.\n", "exploration eps 0.010000.\n", "running time 0.010651\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971847.\n", "mean reward (over 100 episodes) 1950.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11388.\n", "exploration eps 0.010000.\n", "running time 0.003722\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2971964.\n", "mean reward (over 100 episodes) 1939.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11389.\n", "exploration eps 0.010000.\n", "running time 0.011909\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972066.\n", "mean reward (over 100 episodes) 1939.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11389.\n", "exploration eps 0.010000.\n", "running time 0.010331\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972224.\n", "mean reward (over 100 episodes) 1939.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11389.\n", "exploration eps 0.010000.\n", "running time 0.015535\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972262.\n", "mean reward (over 100 episodes) 1936.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11390.\n", "exploration eps 0.010000.\n", "running time 0.003847\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972418.\n", "mean reward (over 100 episodes) 1936.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11390.\n", "exploration eps 0.010000.\n", "running time 0.015921\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972477.\n", "mean reward (over 100 episodes) 1936.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11390.\n", "exploration eps 0.010000.\n", "running time 0.006010\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972509.\n", "mean reward (over 100 episodes) 1943.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11391.\n", "exploration eps 0.010000.\n", "running time 0.003294\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972679.\n", "mean reward (over 100 episodes) 1943.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11391.\n", "exploration eps 0.010000.\n", "running time 0.016727\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972774.\n", "mean reward (over 100 episodes) 1943.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11391.\n", "exploration eps 0.010000.\n", "running time 0.009614\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2972839.\n", "mean reward (over 100 episodes) 1944.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11392.\n", "exploration eps 0.010000.\n", "running time 0.006350\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973021.\n", "mean reward (over 100 episodes) 1944.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11392.\n", "exploration eps 0.010000.\n", "running time 0.018064\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973106.\n", "mean reward (over 100 episodes) 1944.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11392.\n", "exploration eps 0.010000.\n", "running time 0.008422\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973156.\n", "mean reward (over 100 episodes) 1936.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11393.\n", "exploration eps 0.010000.\n", "running time 0.005357\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973375.\n", "mean reward (over 100 episodes) 1936.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11393.\n", "exploration eps 0.010000.\n", "running time 0.021259\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973423.\n", "mean reward (over 100 episodes) 1936.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11393.\n", "exploration eps 0.010000.\n", "running time 0.004869\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973551.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11394.\n", "exploration eps 0.010000.\n", "running time 0.012548\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973771.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11394.\n", "exploration eps 0.010000.\n", "running time 0.021689\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973809.\n", "mean reward (over 100 episodes) 1957.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11394.\n", "exploration eps 0.010000.\n", "running time 0.003820\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2973895.\n", "mean reward (over 100 episodes) 1966.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11395.\n", "exploration eps 0.010000.\n", "running time 0.008437\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974053.\n", "mean reward (over 100 episodes) 1966.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11395.\n", "exploration eps 0.010000.\n", "running time 0.015551\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974224.\n", "mean reward (over 100 episodes) 1966.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11395.\n", "exploration eps 0.010000.\n", "running time 0.016625\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974273.\n", "mean reward (over 100 episodes) 1943.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11396.\n", "exploration eps 0.010000.\n", "running time 0.005024\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974411.\n", "mean reward (over 100 episodes) 1943.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11396.\n", "exploration eps 0.010000.\n", "running time 0.013413\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974453.\n", "mean reward (over 100 episodes) 1943.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11396.\n", "exploration eps 0.010000.\n", "running time 0.004332\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974495.\n", "mean reward (over 100 episodes) 1941.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11397.\n", "exploration eps 0.010000.\n", "running time 0.004071\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974620.\n", "mean reward (over 100 episodes) 1941.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11397.\n", "exploration eps 0.010000.\n", "running time 0.012289\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974656.\n", "mean reward (over 100 episodes) 1941.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11397.\n", "exploration eps 0.010000.\n", "running time 0.003630\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974738.\n", "mean reward (over 100 episodes) 1934.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11398.\n", "exploration eps 0.010000.\n", "running time 0.007901\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974879.\n", "mean reward (over 100 episodes) 1934.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11398.\n", "exploration eps 0.010000.\n", "running time 0.013913\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2974942.\n", "mean reward (over 100 episodes) 1934.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11398.\n", "exploration eps 0.010000.\n", "running time 0.006206\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975067.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11399.\n", "exploration eps 0.010000.\n", "running time 0.012127\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975248.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11399.\n", "exploration eps 0.010000.\n", "running time 0.017998\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975303.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11399.\n", "exploration eps 0.010000.\n", "running time 0.005686\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975338.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11400.\n", "exploration eps 0.010000.\n", "running time 0.003500\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975526.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11400.\n", "exploration eps 0.010000.\n", "running time 0.018347\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975696.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11400.\n", "exploration eps 0.010000.\n", "running time 0.017078\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975786.\n", "mean reward (over 100 episodes) 1928.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11401.\n", "exploration eps 0.010000.\n", "running time 0.008659\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2975896.\n", "mean reward (over 100 episodes) 1928.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11401.\n", "exploration eps 0.010000.\n", "running time 0.010934\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976022.\n", "mean reward (over 100 episodes) 1928.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11401.\n", "exploration eps 0.010000.\n", "running time 0.012597\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976107.\n", "mean reward (over 100 episodes) 1925.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11402.\n", "exploration eps 0.010000.\n", "running time 0.008307\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976254.\n", "mean reward (over 100 episodes) 1925.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11402.\n", "exploration eps 0.010000.\n", "running time 0.014625\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976348.\n", "mean reward (over 100 episodes) 1925.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11402.\n", "exploration eps 0.010000.\n", "running time 0.009395\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976449.\n", "mean reward (over 100 episodes) 1924.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11403.\n", "exploration eps 0.010000.\n", "running time 0.009982\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976542.\n", "mean reward (over 100 episodes) 1924.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11403.\n", "exploration eps 0.010000.\n", "running time 0.009840\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976635.\n", "mean reward (over 100 episodes) 1924.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11403.\n", "exploration eps 0.010000.\n", "running time 0.009179\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976788.\n", "mean reward (over 100 episodes) 1927.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11404.\n", "exploration eps 0.010000.\n", "running time 0.015323\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2976915.\n", "mean reward (over 100 episodes) 1927.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11404.\n", "exploration eps 0.010000.\n", "running time 0.012481\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977014.\n", "mean reward (over 100 episodes) 1927.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11404.\n", "exploration eps 0.010000.\n", "running time 0.009777\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977120.\n", "mean reward (over 100 episodes) 1932.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11405.\n", "exploration eps 0.010000.\n", "running time 0.010524\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977469.\n", "mean reward (over 100 episodes) 1932.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11405.\n", "exploration eps 0.010000.\n", "running time 0.034221\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977516.\n", "mean reward (over 100 episodes) 1932.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11405.\n", "exploration eps 0.010000.\n", "running time 0.004651\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977718.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11406.\n", "exploration eps 0.010000.\n", "running time 0.020113\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977814.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11406.\n", "exploration eps 0.010000.\n", "running time 0.009911\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977904.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11406.\n", "exploration eps 0.010000.\n", "running time 0.009057\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2977940.\n", "mean reward (over 100 episodes) 1931.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11407.\n", "exploration eps 0.010000.\n", "running time 0.003661\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978039.\n", "mean reward (over 100 episodes) 1931.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11407.\n", "exploration eps 0.010000.\n", "running time 0.009984\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978155.\n", "mean reward (over 100 episodes) 1931.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11407.\n", "exploration eps 0.010000.\n", "running time 0.011391\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978244.\n", "mean reward (over 100 episodes) 1923.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11408.\n", "exploration eps 0.010000.\n", "running time 0.008693\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978384.\n", "mean reward (over 100 episodes) 1923.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11408.\n", "exploration eps 0.010000.\n", "running time 0.013656\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978438.\n", "mean reward (over 100 episodes) 1923.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11408.\n", "exploration eps 0.010000.\n", "running time 0.005409\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978524.\n", "mean reward (over 100 episodes) 1918.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11409.\n", "exploration eps 0.010000.\n", "running time 0.008564\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978726.\n", "mean reward (over 100 episodes) 1918.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11409.\n", "exploration eps 0.010000.\n", "running time 0.019630\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978836.\n", "mean reward (over 100 episodes) 1918.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11409.\n", "exploration eps 0.010000.\n", "running time 0.010820\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978874.\n", "mean reward (over 100 episodes) 1935.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11410.\n", "exploration eps 0.010000.\n", "running time 0.003883\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2978967.\n", "mean reward (over 100 episodes) 1935.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11410.\n", "exploration eps 0.010000.\n", "running time 0.009154\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979044.\n", "mean reward (over 100 episodes) 1935.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11410.\n", "exploration eps 0.010000.\n", "running time 0.007912\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979102.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11411.\n", "exploration eps 0.010000.\n", "running time 0.005967\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979225.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11411.\n", "exploration eps 0.010000.\n", "running time 0.012067\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979280.\n", "mean reward (over 100 episodes) 1935.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11411.\n", "exploration eps 0.010000.\n", "running time 0.005426\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979328.\n", "mean reward (over 100 episodes) 1935.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11412.\n", "exploration eps 0.010000.\n", "running time 0.004968\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979484.\n", "mean reward (over 100 episodes) 1935.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11412.\n", "exploration eps 0.010000.\n", "running time 0.015314\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979546.\n", "mean reward (over 100 episodes) 1935.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11412.\n", "exploration eps 0.010000.\n", "running time 0.006087\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979648.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11413.\n", "exploration eps 0.010000.\n", "running time 0.010263\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979889.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11413.\n", "exploration eps 0.010000.\n", "running time 0.023762\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979975.\n", "mean reward (over 100 episodes) 1935.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11413.\n", "exploration eps 0.010000.\n", "running time 0.008416\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2979995.\n", "mean reward (over 100 episodes) 1937.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11414.\n", "exploration eps 0.010000.\n", "running time 0.001975\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980158.\n", "mean reward (over 100 episodes) 1937.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11414.\n", "exploration eps 0.010000.\n", "running time 0.016223\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980214.\n", "mean reward (over 100 episodes) 1937.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11414.\n", "exploration eps 0.010000.\n", "running time 0.005646\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980274.\n", "mean reward (over 100 episodes) 1937.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11415.\n", "exploration eps 0.010000.\n", "running time 0.006123\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980458.\n", "mean reward (over 100 episodes) 1937.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11415.\n", "exploration eps 0.010000.\n", "running time 0.018228\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980488.\n", "mean reward (over 100 episodes) 1937.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11415.\n", "exploration eps 0.010000.\n", "running time 0.003192\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980550.\n", "mean reward (over 100 episodes) 1945.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11416.\n", "exploration eps 0.010000.\n", "running time 0.005957\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980755.\n", "mean reward (over 100 episodes) 1945.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11416.\n", "exploration eps 0.010000.\n", "running time 0.021020\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980795.\n", "mean reward (over 100 episodes) 1945.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11416.\n", "exploration eps 0.010000.\n", "running time 0.003997\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2980896.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11417.\n", "exploration eps 0.010000.\n", "running time 0.009736\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981085.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11417.\n", "exploration eps 0.010000.\n", "running time 0.018894\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981128.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11417.\n", "exploration eps 0.010000.\n", "running time 0.004323\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981201.\n", "mean reward (over 100 episodes) 1940.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11418.\n", "exploration eps 0.010000.\n", "running time 0.007103\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981371.\n", "mean reward (over 100 episodes) 1940.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11418.\n", "exploration eps 0.010000.\n", "running time 0.016453\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981416.\n", "mean reward (over 100 episodes) 1940.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11418.\n", "exploration eps 0.010000.\n", "running time 0.004558\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981600.\n", "mean reward (over 100 episodes) 1938.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11419.\n", "exploration eps 0.010000.\n", "running time 0.018811\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981832.\n", "mean reward (over 100 episodes) 1938.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11419.\n", "exploration eps 0.010000.\n", "running time 0.023170\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981887.\n", "mean reward (over 100 episodes) 1938.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11419.\n", "exploration eps 0.010000.\n", "running time 0.005610\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2981945.\n", "mean reward (over 100 episodes) 1939.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11420.\n", "exploration eps 0.010000.\n", "running time 0.005980\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982058.\n", "mean reward (over 100 episodes) 1939.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11420.\n", "exploration eps 0.010000.\n", "running time 0.011352\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982159.\n", "mean reward (over 100 episodes) 1939.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11420.\n", "exploration eps 0.010000.\n", "running time 0.010186\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982212.\n", "mean reward (over 100 episodes) 1944.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11421.\n", "exploration eps 0.010000.\n", "running time 0.005446\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982404.\n", "mean reward (over 100 episodes) 1944.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11421.\n", "exploration eps 0.010000.\n", "running time 0.019053\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982440.\n", "mean reward (over 100 episodes) 1944.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11421.\n", "exploration eps 0.010000.\n", "running time 0.003901\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982525.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11422.\n", "exploration eps 0.010000.\n", "running time 0.008324\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982668.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11422.\n", "exploration eps 0.010000.\n", "running time 0.013907\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982694.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11422.\n", "exploration eps 0.010000.\n", "running time 0.002449\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982753.\n", "mean reward (over 100 episodes) 1943.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11423.\n", "exploration eps 0.010000.\n", "running time 0.005835\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2982859.\n", "mean reward (over 100 episodes) 1943.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11423.\n", "exploration eps 0.010000.\n", "running time 0.010314\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983001.\n", "mean reward (over 100 episodes) 1943.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11423.\n", "exploration eps 0.010000.\n", "running time 0.014474\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983062.\n", "mean reward (over 100 episodes) 1942.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11424.\n", "exploration eps 0.010000.\n", "running time 0.006037\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983122.\n", "mean reward (over 100 episodes) 1942.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11424.\n", "exploration eps 0.010000.\n", "running time 0.006357\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983200.\n", "mean reward (over 100 episodes) 1942.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11424.\n", "exploration eps 0.010000.\n", "running time 0.007737\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983230.\n", "mean reward (over 100 episodes) 1934.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11425.\n", "exploration eps 0.010000.\n", "running time 0.003109\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983387.\n", "mean reward (over 100 episodes) 1934.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11425.\n", "exploration eps 0.010000.\n", "running time 0.015511\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983425.\n", "mean reward (over 100 episodes) 1934.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11425.\n", "exploration eps 0.010000.\n", "running time 0.003801\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983490.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11426.\n", "exploration eps 0.010000.\n", "running time 0.006404\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983660.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11426.\n", "exploration eps 0.010000.\n", "running time 0.016820\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983724.\n", "mean reward (over 100 episodes) 1948.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11426.\n", "exploration eps 0.010000.\n", "running time 0.006275\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983762.\n", "mean reward (over 100 episodes) 1958.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11427.\n", "exploration eps 0.010000.\n", "running time 0.003767\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2983983.\n", "mean reward (over 100 episodes) 1958.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11427.\n", "exploration eps 0.010000.\n", "running time 0.021523\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984040.\n", "mean reward (over 100 episodes) 1958.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11427.\n", "exploration eps 0.010000.\n", "running time 0.005662\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984082.\n", "mean reward (over 100 episodes) 1952.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11428.\n", "exploration eps 0.010000.\n", "running time 0.004211\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984190.\n", "mean reward (over 100 episodes) 1952.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11428.\n", "exploration eps 0.010000.\n", "running time 0.010944\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984247.\n", "mean reward (over 100 episodes) 1952.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11428.\n", "exploration eps 0.010000.\n", "running time 0.005617\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984333.\n", "mean reward (over 100 episodes) 1951.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11429.\n", "exploration eps 0.010000.\n", "running time 0.008842\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984502.\n", "mean reward (over 100 episodes) 1951.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11429.\n", "exploration eps 0.010000.\n", "running time 0.016511\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984615.\n", "mean reward (over 100 episodes) 1951.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11429.\n", "exploration eps 0.010000.\n", "running time 0.011170\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984680.\n", "mean reward (over 100 episodes) 1961.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11430.\n", "exploration eps 0.010000.\n", "running time 0.006495\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984812.\n", "mean reward (over 100 episodes) 1961.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11430.\n", "exploration eps 0.010000.\n", "running time 0.013157\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984900.\n", "mean reward (over 100 episodes) 1961.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11430.\n", "exploration eps 0.010000.\n", "running time 0.008410\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2984947.\n", "mean reward (over 100 episodes) 1965.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11431.\n", "exploration eps 0.010000.\n", "running time 0.004609\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985107.\n", "mean reward (over 100 episodes) 1965.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11431.\n", "exploration eps 0.010000.\n", "running time 0.015734\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985131.\n", "mean reward (over 100 episodes) 1965.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11431.\n", "exploration eps 0.010000.\n", "running time 0.002359\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985185.\n", "mean reward (over 100 episodes) 1961.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11432.\n", "exploration eps 0.010000.\n", "running time 0.005507\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985356.\n", "mean reward (over 100 episodes) 1961.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11432.\n", "exploration eps 0.010000.\n", "running time 0.016883\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985417.\n", "mean reward (over 100 episodes) 1961.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11432.\n", "exploration eps 0.010000.\n", "running time 0.006132\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985438.\n", "mean reward (over 100 episodes) 1955.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11433.\n", "exploration eps 0.010000.\n", "running time 0.002093\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985624.\n", "mean reward (over 100 episodes) 1955.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11433.\n", "exploration eps 0.010000.\n", "running time 0.018522\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985661.\n", "mean reward (over 100 episodes) 1955.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11433.\n", "exploration eps 0.010000.\n", "running time 0.003771\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985840.\n", "mean reward (over 100 episodes) 1940.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11434.\n", "exploration eps 0.010000.\n", "running time 0.018350\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985930.\n", "mean reward (over 100 episodes) 1940.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11434.\n", "exploration eps 0.010000.\n", "running time 0.008997\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2985953.\n", "mean reward (over 100 episodes) 1940.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11434.\n", "exploration eps 0.010000.\n", "running time 0.002434\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986048.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11435.\n", "exploration eps 0.010000.\n", "running time 0.009495\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986194.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11435.\n", "exploration eps 0.010000.\n", "running time 0.014443\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986245.\n", "mean reward (over 100 episodes) 1926.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11435.\n", "exploration eps 0.010000.\n", "running time 0.005090\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986377.\n", "mean reward (over 100 episodes) 1929.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11436.\n", "exploration eps 0.010000.\n", "running time 0.013593\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986540.\n", "mean reward (over 100 episodes) 1929.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11436.\n", "exploration eps 0.010000.\n", "running time 0.015882\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986638.\n", "mean reward (over 100 episodes) 1929.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11436.\n", "exploration eps 0.010000.\n", "running time 0.009942\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986790.\n", "mean reward (over 100 episodes) 1937.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11437.\n", "exploration eps 0.010000.\n", "running time 0.015141\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986908.\n", "mean reward (over 100 episodes) 1937.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11437.\n", "exploration eps 0.010000.\n", "running time 0.011790\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2986976.\n", "mean reward (over 100 episodes) 1937.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11437.\n", "exploration eps 0.010000.\n", "running time 0.007068\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987056.\n", "mean reward (over 100 episodes) 1935.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11438.\n", "exploration eps 0.010000.\n", "running time 0.008062\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987220.\n", "mean reward (over 100 episodes) 1935.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11438.\n", "exploration eps 0.010000.\n", "running time 0.016234\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987430.\n", "mean reward (over 100 episodes) 1935.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11438.\n", "exploration eps 0.010000.\n", "running time 0.020385\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987481.\n", "mean reward (over 100 episodes) 1950.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11439.\n", "exploration eps 0.010000.\n", "running time 0.005195\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987578.\n", "mean reward (over 100 episodes) 1950.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11439.\n", "exploration eps 0.010000.\n", "running time 0.009440\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987662.\n", "mean reward (over 100 episodes) 1950.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11439.\n", "exploration eps 0.010000.\n", "running time 0.008446\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987709.\n", "mean reward (over 100 episodes) 1950.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11440.\n", "exploration eps 0.010000.\n", "running time 0.004927\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987866.\n", "mean reward (over 100 episodes) 1950.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11440.\n", "exploration eps 0.010000.\n", "running time 0.015443\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987923.\n", "mean reward (over 100 episodes) 1950.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11440.\n", "exploration eps 0.010000.\n", "running time 0.005793\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2987964.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11441.\n", "exploration eps 0.010000.\n", "running time 0.004187\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988076.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11441.\n", "exploration eps 0.010000.\n", "running time 0.011093\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988173.\n", "mean reward (over 100 episodes) 1949.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11441.\n", "exploration eps 0.010000.\n", "running time 0.009531\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988330.\n", "mean reward (over 100 episodes) 1962.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11442.\n", "exploration eps 0.010000.\n", "running time 0.015798\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988606.\n", "mean reward (over 100 episodes) 1962.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11442.\n", "exploration eps 0.010000.\n", "running time 0.027186\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988728.\n", "mean reward (over 100 episodes) 1962.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11442.\n", "exploration eps 0.010000.\n", "running time 0.011800\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988788.\n", "mean reward (over 100 episodes) 1986.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11443.\n", "exploration eps 0.010000.\n", "running time 0.005745\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988950.\n", "mean reward (over 100 episodes) 1986.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11443.\n", "exploration eps 0.010000.\n", "running time 0.016088\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2988986.\n", "mean reward (over 100 episodes) 1986.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11443.\n", "exploration eps 0.010000.\n", "running time 0.003784\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989006.\n", "mean reward (over 100 episodes) 1987.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11444.\n", "exploration eps 0.010000.\n", "running time 0.002207\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989110.\n", "mean reward (over 100 episodes) 1987.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11444.\n", "exploration eps 0.010000.\n", "running time 0.010545\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989150.\n", "mean reward (over 100 episodes) 1987.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11444.\n", "exploration eps 0.010000.\n", "running time 0.004305\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989219.\n", "mean reward (over 100 episodes) 1992.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11445.\n", "exploration eps 0.010000.\n", "running time 0.007149\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989316.\n", "mean reward (over 100 episodes) 1992.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11445.\n", "exploration eps 0.010000.\n", "running time 0.009772\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989351.\n", "mean reward (over 100 episodes) 1992.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11445.\n", "exploration eps 0.010000.\n", "running time 0.003441\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989506.\n", "mean reward (over 100 episodes) 1998.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11446.\n", "exploration eps 0.010000.\n", "running time 0.015128\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989607.\n", "mean reward (over 100 episodes) 1998.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11446.\n", "exploration eps 0.010000.\n", "running time 0.010167\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989707.\n", "mean reward (over 100 episodes) 1998.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11446.\n", "exploration eps 0.010000.\n", "running time 0.009704\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989751.\n", "mean reward (over 100 episodes) 1988.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11447.\n", "exploration eps 0.010000.\n", "running time 0.004691\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989882.\n", "mean reward (over 100 episodes) 1988.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11447.\n", "exploration eps 0.010000.\n", "running time 0.013013\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989931.\n", "mean reward (over 100 episodes) 1988.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11447.\n", "exploration eps 0.010000.\n", "running time 0.004924\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2989945.\n", "mean reward (over 100 episodes) 1978.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11448.\n", "exploration eps 0.010000.\n", "running time 0.001639\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990065.\n", "mean reward (over 100 episodes) 1978.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11448.\n", "exploration eps 0.010000.\n", "running time 0.011857\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990133.\n", "mean reward (over 100 episodes) 1978.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11448.\n", "exploration eps 0.010000.\n", "running time 0.006743\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990190.\n", "mean reward (over 100 episodes) 1975.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11449.\n", "exploration eps 0.010000.\n", "running time 0.005692\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990348.\n", "mean reward (over 100 episodes) 1975.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11449.\n", "exploration eps 0.010000.\n", "running time 0.015550\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990412.\n", "mean reward (over 100 episodes) 1975.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11449.\n", "exploration eps 0.010000.\n", "running time 0.006327\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990490.\n", "mean reward (over 100 episodes) 1956.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11450.\n", "exploration eps 0.010000.\n", "running time 0.007563\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990602.\n", "mean reward (over 100 episodes) 1956.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11450.\n", "exploration eps 0.010000.\n", "running time 0.010843\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990725.\n", "mean reward (over 100 episodes) 1956.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11450.\n", "exploration eps 0.010000.\n", "running time 0.011921\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990786.\n", "mean reward (over 100 episodes) 1952.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11451.\n", "exploration eps 0.010000.\n", "running time 0.006203\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2990935.\n", "mean reward (over 100 episodes) 1952.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11451.\n", "exploration eps 0.010000.\n", "running time 0.014679\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991032.\n", "mean reward (over 100 episodes) 1952.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11451.\n", "exploration eps 0.010000.\n", "running time 0.009515\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991088.\n", "mean reward (over 100 episodes) 1950.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11452.\n", "exploration eps 0.010000.\n", "running time 0.005638\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991257.\n", "mean reward (over 100 episodes) 1950.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11452.\n", "exploration eps 0.010000.\n", "running time 0.017147\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991349.\n", "mean reward (over 100 episodes) 1950.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11452.\n", "exploration eps 0.010000.\n", "running time 0.009123\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991399.\n", "mean reward (over 100 episodes) 1948.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11453.\n", "exploration eps 0.010000.\n", "running time 0.005064\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991660.\n", "mean reward (over 100 episodes) 1948.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11453.\n", "exploration eps 0.010000.\n", "running time 0.025740\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991714.\n", "mean reward (over 100 episodes) 1948.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11453.\n", "exploration eps 0.010000.\n", "running time 0.005300\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991746.\n", "mean reward (over 100 episodes) 1953.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11454.\n", "exploration eps 0.010000.\n", "running time 0.003304\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991868.\n", "mean reward (over 100 episodes) 1953.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11454.\n", "exploration eps 0.010000.\n", "running time 0.012456\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2991929.\n", "mean reward (over 100 episodes) 1953.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11454.\n", "exploration eps 0.010000.\n", "running time 0.005942\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992014.\n", "mean reward (over 100 episodes) 1942.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11455.\n", "exploration eps 0.010000.\n", "running time 0.008459\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992197.\n", "mean reward (over 100 episodes) 1942.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11455.\n", "exploration eps 0.010000.\n", "running time 0.018227\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992287.\n", "mean reward (over 100 episodes) 1942.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11455.\n", "exploration eps 0.010000.\n", "running time 0.009054\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992381.\n", "mean reward (over 100 episodes) 1944.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11456.\n", "exploration eps 0.010000.\n", "running time 0.009145\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992566.\n", "mean reward (over 100 episodes) 1944.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11456.\n", "exploration eps 0.010000.\n", "running time 0.018094\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992689.\n", "mean reward (over 100 episodes) 1944.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11456.\n", "exploration eps 0.010000.\n", "running time 0.012055\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992756.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11457.\n", "exploration eps 0.010000.\n", "running time 0.006541\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992898.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11457.\n", "exploration eps 0.010000.\n", "running time 0.013911\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992941.\n", "mean reward (over 100 episodes) 1944.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11457.\n", "exploration eps 0.010000.\n", "running time 0.004379\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2992987.\n", "mean reward (over 100 episodes) 1940.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11458.\n", "exploration eps 0.010000.\n", "running time 0.004661\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993229.\n", "mean reward (over 100 episodes) 1940.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11458.\n", "exploration eps 0.010000.\n", "running time 0.023907\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993276.\n", "mean reward (over 100 episodes) 1940.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11458.\n", "exploration eps 0.010000.\n", "running time 0.004790\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993341.\n", "mean reward (over 100 episodes) 1968.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11459.\n", "exploration eps 0.010000.\n", "running time 0.006594\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993510.\n", "mean reward (over 100 episodes) 1968.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11459.\n", "exploration eps 0.010000.\n", "running time 0.016554\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993530.\n", "mean reward (over 100 episodes) 1968.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11459.\n", "exploration eps 0.010000.\n", "running time 0.002221\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993559.\n", "mean reward (over 100 episodes) 1971.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11460.\n", "exploration eps 0.010000.\n", "running time 0.003189\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993661.\n", "mean reward (over 100 episodes) 1971.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11460.\n", "exploration eps 0.010000.\n", "running time 0.010633\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993773.\n", "mean reward (over 100 episodes) 1971.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11460.\n", "exploration eps 0.010000.\n", "running time 0.010751\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993851.\n", "mean reward (over 100 episodes) 1975.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11461.\n", "exploration eps 0.010000.\n", "running time 0.007601\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2993947.\n", "mean reward (over 100 episodes) 1975.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11461.\n", "exploration eps 0.010000.\n", "running time 0.009442\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994097.\n", "mean reward (over 100 episodes) 1975.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11461.\n", "exploration eps 0.010000.\n", "running time 0.014715\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994145.\n", "mean reward (over 100 episodes) 1972.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11462.\n", "exploration eps 0.010000.\n", "running time 0.004620\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994254.\n", "mean reward (over 100 episodes) 1972.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11462.\n", "exploration eps 0.010000.\n", "running time 0.010775\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994345.\n", "mean reward (over 100 episodes) 1972.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11462.\n", "exploration eps 0.010000.\n", "running time 0.009079\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994391.\n", "mean reward (over 100 episodes) 1972.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11463.\n", "exploration eps 0.010000.\n", "running time 0.004682\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994522.\n", "mean reward (over 100 episodes) 1972.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11463.\n", "exploration eps 0.010000.\n", "running time 0.013292\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994653.\n", "mean reward (over 100 episodes) 1972.100000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11463.\n", "exploration eps 0.010000.\n", "running time 0.013103\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994697.\n", "mean reward (over 100 episodes) 1968.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11464.\n", "exploration eps 0.010000.\n", "running time 0.004558\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2994952.\n", "mean reward (over 100 episodes) 1968.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11464.\n", "exploration eps 0.010000.\n", "running time 0.024819\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995000.\n", "mean reward (over 100 episodes) 1968.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11464.\n", "exploration eps 0.010000.\n", "running time 0.004806\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995028.\n", "mean reward (over 100 episodes) 1972.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11465.\n", "exploration eps 0.010000.\n", "running time 0.003009\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995173.\n", "mean reward (over 100 episodes) 1972.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11465.\n", "exploration eps 0.010000.\n", "running time 0.014561\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995220.\n", "mean reward (over 100 episodes) 1972.900000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11465.\n", "exploration eps 0.010000.\n", "running time 0.004811\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995259.\n", "mean reward (over 100 episodes) 1967.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11466.\n", "exploration eps 0.010000.\n", "running time 0.003905\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995424.\n", "mean reward (over 100 episodes) 1967.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11466.\n", "exploration eps 0.010000.\n", "running time 0.016298\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995450.\n", "mean reward (over 100 episodes) 1967.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11466.\n", "exploration eps 0.010000.\n", "running time 0.002669\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995506.\n", "mean reward (over 100 episodes) 1977.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11467.\n", "exploration eps 0.010000.\n", "running time 0.005628\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995711.\n", "mean reward (over 100 episodes) 1977.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11467.\n", "exploration eps 0.010000.\n", "running time 0.020130\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995791.\n", "mean reward (over 100 episodes) 1977.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11467.\n", "exploration eps 0.010000.\n", "running time 0.007965\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2995869.\n", "mean reward (over 100 episodes) 1983.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11468.\n", "exploration eps 0.010000.\n", "running time 0.008225\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996095.\n", "mean reward (over 100 episodes) 1983.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11468.\n", "exploration eps 0.010000.\n", "running time 0.022220\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996114.\n", "mean reward (over 100 episodes) 1983.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11468.\n", "exploration eps 0.010000.\n", "running time 0.002462\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996131.\n", "mean reward (over 100 episodes) 1991.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11469.\n", "exploration eps 0.010000.\n", "running time 0.001918\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996235.\n", "mean reward (over 100 episodes) 1991.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11469.\n", "exploration eps 0.010000.\n", "running time 0.010472\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996357.\n", "mean reward (over 100 episodes) 1991.000000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11469.\n", "exploration eps 0.010000.\n", "running time 0.012005\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996539.\n", "mean reward (over 100 episodes) 1990.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11470.\n", "exploration eps 0.010000.\n", "running time 0.017516\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996662.\n", "mean reward (over 100 episodes) 1990.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11470.\n", "exploration eps 0.010000.\n", "running time 0.012530\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996775.\n", "mean reward (over 100 episodes) 1990.200000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11470.\n", "exploration eps 0.010000.\n", "running time 0.011193\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996831.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11471.\n", "exploration eps 0.010000.\n", "running time 0.005797\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996940.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11471.\n", "exploration eps 0.010000.\n", "running time 0.010746\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2996997.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11471.\n", "exploration eps 0.010000.\n", "running time 0.005886\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997090.\n", "mean reward (over 100 episodes) 1985.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11472.\n", "exploration eps 0.010000.\n", "running time 0.009064\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997196.\n", "mean reward (over 100 episodes) 1985.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11472.\n", "exploration eps 0.010000.\n", "running time 0.010948\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997237.\n", "mean reward (over 100 episodes) 1985.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11472.\n", "exploration eps 0.010000.\n", "running time 0.004008\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997321.\n", "mean reward (over 100 episodes) 1985.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11473.\n", "exploration eps 0.010000.\n", "running time 0.008230\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997477.\n", "mean reward (over 100 episodes) 1985.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11473.\n", "exploration eps 0.010000.\n", "running time 0.015476\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997567.\n", "mean reward (over 100 episodes) 1985.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11473.\n", "exploration eps 0.010000.\n", "running time 0.008904\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997619.\n", "mean reward (over 100 episodes) 1985.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11474.\n", "exploration eps 0.010000.\n", "running time 0.005098\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997784.\n", "mean reward (over 100 episodes) 1985.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11474.\n", "exploration eps 0.010000.\n", "running time 0.016395\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997889.\n", "mean reward (over 100 episodes) 1985.300000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11474.\n", "exploration eps 0.010000.\n", "running time 0.010472\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2997958.\n", "mean reward (over 100 episodes) 1987.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11475.\n", "exploration eps 0.010000.\n", "running time 0.006677\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998076.\n", "mean reward (over 100 episodes) 1987.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11475.\n", "exploration eps 0.010000.\n", "running time 0.011809\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998166.\n", "mean reward (over 100 episodes) 1987.800000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11475.\n", "exploration eps 0.010000.\n", "running time 0.008658\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998202.\n", "mean reward (over 100 episodes) 1969.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11476.\n", "exploration eps 0.010000.\n", "running time 0.003841\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998375.\n", "mean reward (over 100 episodes) 1969.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11476.\n", "exploration eps 0.010000.\n", "running time 0.017036\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998415.\n", "mean reward (over 100 episodes) 1969.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11476.\n", "exploration eps 0.010000.\n", "running time 0.004292\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998473.\n", "mean reward (over 100 episodes) 1971.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11477.\n", "exploration eps 0.010000.\n", "running time 0.005854\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998535.\n", "mean reward (over 100 episodes) 1971.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11477.\n", "exploration eps 0.010000.\n", "running time 0.006190\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998662.\n", "mean reward (over 100 episodes) 1971.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11477.\n", "exploration eps 0.010000.\n", "running time 0.012740\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998694.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11478.\n", "exploration eps 0.010000.\n", "running time 0.003334\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998830.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11478.\n", "exploration eps 0.010000.\n", "running time 0.013979\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998873.\n", "mean reward (over 100 episodes) 1964.600000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11478.\n", "exploration eps 0.010000.\n", "running time 0.004328\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2998914.\n", "mean reward (over 100 episodes) 1968.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11479.\n", "exploration eps 0.010000.\n", "running time 0.004380\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999101.\n", "mean reward (over 100 episodes) 1968.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11479.\n", "exploration eps 0.010000.\n", "running time 0.018460\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999245.\n", "mean reward (over 100 episodes) 1968.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11479.\n", "exploration eps 0.010000.\n", "running time 0.014326\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999287.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11480.\n", "exploration eps 0.010000.\n", "running time 0.004229\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999426.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11480.\n", "exploration eps 0.010000.\n", "running time 0.013808\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999475.\n", "mean reward (over 100 episodes) 1981.700000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11480.\n", "exploration eps 0.010000.\n", "running time 0.004830\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999524.\n", "mean reward (over 100 episodes) 1966.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11481.\n", "exploration eps 0.010000.\n", "running time 0.005045\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999632.\n", "mean reward (over 100 episodes) 1966.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11481.\n", "exploration eps 0.010000.\n", "running time 0.010749\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999720.\n", "mean reward (over 100 episodes) 1966.400000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11481.\n", "exploration eps 0.010000.\n", "running time 0.008808\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999780.\n", "mean reward (over 100 episodes) 1961.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11482.\n", "exploration eps 0.010000.\n", "running time 0.006143\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999923.\n", "mean reward (over 100 episodes) 1961.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11482.\n", "exploration eps 0.010000.\n", "running time 0.014201\n", "------------------------------\n", "\n", "------------------------------\n", "Timestep 2999957.\n", "mean reward (over 100 episodes) 1961.500000.\n", "best mean reward (ever) 2124.900000.\n", "episode # 11482.\n", "exploration eps 0.010000.\n", "running time 0.003571\n", "------------------------------\n", "\n", "\n", "total time: 17709.96412205696\n" ] } ], "source": [ "#@title train DQN\n", "\n", "# reset environment\n", "state = env.reset()\n", "\n", "#####\n", "print(\"Start learning.\\n\")\n", "##### run DQN\n", "for iteration in range(N_iterations):\n", " \n", " # store state in buffer and compute its encoding\n", " buffer_index = replay_buffer.store_frame(state)\n", " last_obs_encode = replay_buffer.encode_recent_observation()\n", " state_enc = np.expand_dims(last_obs_encode, 0)\n", " \n", " # take one episode step \n", " state, reward, is_terminal = episode_step(\n", " iteration,\n", " env,\n", " model,\n", " replay_buffer,\n", " buffer_index,\n", " state_enc,\n", " eps_schedule_args=eps_schedule_args,\n", " )\n", "\n", " # update deep Q-net\n", " if iteration % update_frequency == 0:\n", " model.update_Qnet(replay_buffer, minibatch_size, gamma)\n", "\n", " # update target Q-net\n", " if iteration % target_update == 0:\n", " model.update_Qnet_target()\n", "\n", " if is_terminal:\n", " # print stats\n", " rl_logger.stats(iteration)\n", " \n", " # reset environment\n", " state = env.reset()\n", "\n", "print(\"\\n\\ntotal time: {}\".format(time.time() - tot_time))\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 295 }, "id": "B7o8a0evXbkj", "outputId": "dd331c23-3e7d-404b-e881-c3ee15df03fe" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydZ3hURduA79n0AgmEXgMCoQQSIHRCbwKKKEqRYvlEXxXri4IiKhbsBVEQXgELBhQFlaJ06SVBEKRICyR0CJDeduf7cXY3u9makCVA5r6uvXLOzJw5syGc58xThZQShUKhUCicoSvtBSgUCoXixkcJC4VCoVC4RAkLhUKhULhECQuFQqFQuEQJC4VCoVC4RAkLhUKhULhECQuFQqFQuEQJC0WZRAiRKITIFUJUKtT+lxBCCiHCnVzbTQhhEEKkCyHShBCHhBAPenrNCkVpooSFoixzHBhuOhFCNAcC3bz2tJQyGCgPvAjMFkI0LfklKhQ3BkpYKMoy3wKjLc7HAN+YToQQ/YUQ+427h1NCiP8WnkBqLAEuA02FEAOMu5NUIUSSEOI1y/FCiM5CiC1CiCvG/geM7fOEEF8IIVYYdyybhRDVhBCfCCEuCyEOCiFaWswzQQhx1Li2/UKIwRZ9DwghNgkhPjBee1wIcXsJ/c4UZRQlLBRlmW1AeSFEEyGEFzAM+M6i/yvgUSllOSASWFt4AiGEzvigDgX2AhloAigUGAD8Rwhxl3FsXWAF8BlQGYgGdltMdx8wCagE5ABbgV3G80XARxZjjwKxQAjwOvCdEKK6RX874JDx2veAr4QQoii/HIXCEiUsFGUd0+6iN3AAOGXRl4e2Wygvpbwspdxl0VdDCHEFuAi8CoySUh6SUq6XUu6VUhqklH8DcUBX4zUjgNVSyjgpZZ6U8pKU0lJYLJZSJkgps4HFQLaU8hsppR5YCJh3FlLKH6WUp433WQgcBtpazHVCSjnbeO3XQHWg6jX+rhRlGCUsFGWdb9Ee4g9goYIycg/QHzghhPhTCNHBou+0lDJUSllRShktpVwAIIRoJ4RYJ4S4IIS4CjyG9nYPUBttR+CIcxbHWXbOg00nQojRQojdRnXWFbSdj6Wx/qzpQEqZaTwMRqEoJkpYKMo0UsoTaIbu/sDPhfp2SikHAVWAJcAPbkz5PfArUFtKGQLMBEzqnyTgtmtds1GdNRt4EgiTUoYC+yzuo1CUOEpYKBTwMNBDSplh0eYrhLhfCBEipcwDUgGDG3OVA1KklNlCiLZouxYT84FeQoj7hBDeQogwIUR0MdYbBEjgAoDRbTeyGPMoFG6jhIWizCOlPCqljLfTNQpIFEKkoqmT7ndjuseBKUKINGAyFrsRKeVJtB3M80AKmnE7qhjr3Q98iGYAPwc0BzYXdR6FoigIVfxIoVAoFK5QOwuFQqFQuEQJC4VCoVC4RAkLhUKhULhECQuFQqFQuMS7tBfgKSpVqiTDw8NLexkKhUJx05CQkHBRSlnZXt8tKyzCw8OJj7fnDalQKBQKewghTjjqU2oohUKhULhECQuFQqFQuEQJC4VCoVC45Ja1WdgjLy+P5ORksrOzS3spCoVd/P39qVWrFj4+PqW9FIXCijIlLJKTkylXrhzh4eGoOjCKGw0pJZcuXSI5OZl69eqV9nIUCivKlBoqOzubsLAwJSgUNyRCCMLCwtTOV3FDUqaEBaAEheKGRv19Km5UypQaSqFQKBxiMIA+p7RXYZeL509zefMcGlYKdD3YNwg6P1Pia1DCohRYsmQJgwcP5sCBAzRu3Li0l3PdWL9+PYMGDbLSx3/wwQf06tWrROafN28e8fHxTJ8+3em4bt26cebMGfz9/fH19WX27NlERxenBtG18dprrxEcHMx///vf635vhTXbtm2m8Yp7CRUZrgeXApWwrJnrYvcZXEUJi1uFuLg4OnfuTFxcHK+//vo1z6fX6/Hy8iqBlZUs+fn5eHtb/4nFxsaydOnSUlpRAfPnzycmJoa5c+cyfvx4Vq1a5dH7SSmRUqLTlTnN703BD7/+yke+GSzWd6JbbFcqBPqW9pKseGfFQRJlNX43tCXxnQGlswjTH/Gt9mndurUszP79+23arjdpaWmyRo0a8tChQ7JRo0ZSSilXrFghhwwZYh6zbt06OWDAACmllH/88Yds3769bNmypRwyZIhMS0uTUkpZt25d+cILL8iWLVvKuLg4OWvWLBkTEyNbtGgh7777bpmRkSGllPLIkSOyXbt2MjIyUr788ssyKCjIfJ/33ntPxsTEyObNm8vJkyfbXW9QUJB85plnZNOmTWWPHj3k+fPnzfP27dtXtmrVSnbu3FkeOHBASinlmDFj5KOPPirbtm0rn332Wau5LL+XJcePH5cRERFyxIgRsnHjxvKee+4xr3/16tUyOjpaRkZGygcffFBmZ2dLKaXcsWOH7NChg2zRooVs06aNTE1NlXPnzpWDBw+Wffv2lQ0aNJDjx4+3+526du0qd+7cKaWU8sCBA7JJkyZSSinT09Plgw8+KNu0aSOjo6PlkiVLpJRS9u/fX+7Zs0dKKWV0dLR8/fXXpZRSvvLKK3LWrFkyLS1N9ujRQ7Zs2VJGRkaarzt+/Lhs1KiRHDVqlGzatKlMTEyUb775pmzYsKHs1KmTHDZsmHz//fdt1ncj/J2WNSa/NE7KV8vL6Be/l4fPpZb2cmyo++JS88eTAPHSwTO1zO4sXv/tH/afTi3ROZvWKM+rdzRzOuaXX36hX79+NGrUiLCwMBISEujVqxdjx44lIyODoKAgFi5cyLBhw7h48SJvvvkmq1evJigoiHfffZePPvqIyZMnAxAWFsauXbsAuHTpEo888ggAkyZN4quvvmLcuHE8/fTTPP300wwfPpyZM2ea17Fy5UoOHz7Mjh07kFJy5513smHDBrp06WK13oyMDGJiYvj444+ZMmUKr7/+OtOnT2fs2LHMnDmThg0bsn37dh5//HHWrl0LaC7KW7Zssbvb2bhxo5XK56effsLLy4tDhw7x1Vdf0alTJx566CG++OILnnzySR544AHWrFlDo0aNGD16NDNmzODxxx9n6NChLFy4kDZt2pCamkpAQAAAu3fv5q+//sLPz4+IiAjGjRtH7dq1Hf57/P7779x1110AvPXWW/To0YM5c+Zw5coV2rZtS69evYiNjWXjxo3UrVsXb29vNm/ebP4uM2fOxN/fn8WLF1O+fHkuXrxI+/btufPOOwE4fPgwX3/9Ne3btychIYEFCxawe/du8vPzadWqFa1bt3b696K4PtQvp4dsSCOQ01eyaVClXGkvyYrwsEASL2WW6hrKrLAoLeLi4nj66acBGDZsGHFxcbRu3Zp+/frx22+/MWTIEJYtW8Z7773Hn3/+yf79++nUqRMAubm5dOjQwTzX0KFDzcf79u1j0qRJXLlyhfT0dPr27QvA1q1bWbJkCQAjRoww68dXrlzJypUradmyJQDp6ekcPnzYRljodDrzfUaOHMndd99Neno6W7Zs4d577zWPy8kpMAzee++9DtVi9tRQiYmJ1K5d2/w9R44cybRp0+jduzf16tWjUaNGAIwZM4bPP/+cnj17Ur16ddq0aQNA+fLlzXP17NmTkJAQAJo2bcqJEyfsCov777+f3Nxc0tPT2b17t/l38uuvv/LBBx8Amqv1yZMniY2NZdq0adSrV48BAwawatUqMjMzOX78OBEREeTl5fHSSy+xYcMGdDodp06d4ty5cwDUrVuX9u3bA5pwGTx4MIGBmpHSJFAUpY+/Pp0s6Us+3vh533iqwk4NKpF46WSprqHMCgtXOwBPkJKSwtq1a9m7dy9CCPR6PUII3n//fYYNG8b06dOpWLEiMTExlCtXDiklvXv3Ji4uzu58QUFB5uMHHniAJUuWEBUVxbx581i/fr3TtUgpmThxIo8++miRvoMQAoPBQGhoqPkh62xdRZnX2bm7+Pn5mY+9vLzIz8+3O27+/Pm0bt2a8ePHM27cOH7++WeklPz0009ERERYjc3NzSU+Pp769evTu3dvLl68yOzZs827gvnz53PhwgUSEhLw8fEhPDzcHCtRnN+F4voT5p1Deq62O83K05fyamwxyNJeQRmMsyhNFi1axKhRozhx4gSJiYkkJSVRr149Nm7cSNeuXdm1axezZ89m2LBhALRv357Nmzdz5MgRQFMJ/fvvv3bnTktLo3r16uTl5TF//nxze/v27fnpp58AWLBggbm9b9++zJkzh/T0dABOnTrF+fPnbeY1GAwsWrQIgO+//57OnTtTvnx56tWrx48//ghogmfPnj3X9Ls5efIkW7dutbpPREQEiYmJ5u//7bff0rVrVyIiIjhz5gw7d+40f3dHQsEZQgjeeOMNtm3bxsGDB+nbty+fffYZmuoW/vrrLwB8fX2pXbs2P/74Ix06dCA2NpYPPvjAvAu7evUqVapUwcfHh3Xr1nHihP0sz126dGHJkiVkZWWRlpbGb7/9VuQ1KzxDmHc2qVLb8T0wd2cpr8aWK5m5pb0EJSyuJ3FxcQwePNiq7Z577iEuLg4vLy8GDhzIihUrGDhwIACVK1dm3rx5DB8+nBYtWtChQwcOHjxod+433niDdu3a0alTJyt33E8++YSPPvqIFi1acOTIEbOKpk+fPowYMYIOHTrQvHlzhgwZQlpams28QUFB7Nixg8jISNauXWu2l8yfP5+vvvqKqKgomjVrxi+//OLW78BkszB9TIIoIiKCzz//nCZNmnD58mX+85//4O/vz9y5c7n33ntp3rw5Op2Oxx57DF9fXxYuXMi4ceOIioqid+/exY56DggI4Pnnn+f999/nlVdeIS8vjxYtWtCsWTNeeeUV87jY2FiqVKlCQEAAsbGxJCcnExsbC2gqrfj4eJo3b84333zj0B26VatWDB06lKioKG6//XazGk1RumTn5NAybR1pBJT2UhyyYt9Z87G+lLYZwvQWdasRExMjCxc/OnDgAE2aNCmlFZUOmZmZBAQEIIRgwYIFxMXFuf1gBwgODjbvPjxFYmIiAwcOZN++fR69z81CWfw7LU22JPxFx9+6sVLfmrF5zwOUnnuqHaSU1Ju43Hz++zOxNK5W3skVxUcIkSCljLHXV2ZtFmWFhIQEnnzySaSUhIaGMmfOnNJekuIWZe+ubQSt+i/1Kvg4CBtz8WLq8sXVSf81XBuVqQXi/ayPJaZuBRfzXH8OnrXe8f+dfNVjwsIZSljc4sTGxl6TPcHTuwrQSuCqXcXNz6oVi3guby/pFTsTHOBEpWN0XsjO04OArFx9oSA4J84NLh0fin5thi6bNZerstMQQV0pOZd6Y6X8+GztYavzPL2hVNbhMWEhhKgNfANURRPrs6SUnwohKgILgXAgEbhPSnlZaO4vnwL9gUzgASnlLuNcY4BJxqnflFJ+7al1KxSK4lEx6wR4Q6ujj/DvO3e5HN/xjVWkZGiG2/+NjqFX06qeXqJd4vee4aljWrxShI+Xa3kEXEzPYcJPe/nwvihCAjxbe2T7sRSr84qlFF3uSQN3PvC8lLIp0B54QgjRFJgArJFSNgTWGM8BbgcaGj9jgRkARuHyKtAOaAu8KoS48faKCkUZJ8/47pmLew9Pk6AA+L9v4p2M9CymuIrv/68doYE+nLnq2lliwLSNrD5wjqcX/EXcDs/GP9QN07y0GlUNBmD9oQsevZ8jPCYspJRnTDsDKWUacACoCQwCTDuDrwHTK8gg4Btj1Pk2IFQIUR3oC6ySUqZIKS8Dq4B+nlq3QqEoOvFb1vCI93LOyIoAHL94Yybks0d2nqbWCQv2IyTABx8v11sLk6pq/aELTPx5L1cz8zy2vha1QgGYPVqzO7cOL5135eviOiuECAdaAtuBqlLKM8aus2hqKtAESZLFZcnGNkftCoXiBiFzxasArNVrGQG6f7C+FFdTNLKNQXj+Pjoyc/Vk5xkoqpfoX0mXPbE0ANJz8qkR4o+Pl/a4drS2Fxf9TcepazB4yLXW48JCCBEM/AQ8I6W0SsZkTFxVYt9MCDFWCBEvhIi/cKF0tmqu8PLyIjo6mqioKFq1asWWLVuKNc8nn3xCZqb9XDHdunWjTp06Vn9Ud911F8HBwcW6l7vMmzePypUrEx0dTePGjfn44489ej9HJCYmEhkZWSr3LquEiAy2GZrwcv7Dpb2UIpOdbxIWXvyy+zQAZ1M1VVT4hGWMmbPD5RxvLz/gsfWlZecR7O+Nt3HHc/qKfTXZwvgkTl/NpvErv3tkHR4VFkIIHzRBMV9K+bOx+ZxRvYTxpyls+BRgmcSnlrHNUbsNUspZUsoYKWVM5cqVS+6LlCABAQHs3r2bPXv2MHXqVCZOnFiseZwJC4DQ0FBzwrsrV65w5swZh2NLkqFDh7J79242b97MW2+9RVJSkuuLrpHiRG8rSgaD3sD2ueOJ0h3jorz+7pwlgUkN5eetY3xfLdWL6S0e4M9/rV88M3Js/97y9ZJ8vcEjb/W7Tl4hXy/J02tzf7rG2jvKYJBWL4a5HvKW8piwMHo3fQUckFJ+ZNH1KzDGeDwG+MWifbTQaA9cNaqr/gD6CCEqGA3bfYxtNz2pqalUqFCgf3z//fdp06YNLVq04NVXtW19RkYGAwYMICoqisjISBYuXMi0adM4ffo03bt3p3v37nbnHjZsmDm9x88//8zdd99t1W/vXqDtQFq3bk2zZs2YNWuWuT04OJiXX36ZqKgo2rdvb06U54iwsDAaNGhgFlLfffcdbdu2JTo6mkcffRS9Xs+PP/7Ic889B8Cnn35K/fr1ATh27Jg5qeCUKVNo06YNkZGRjB071vyfolu3bjzzzDPExMTw6aefkpCQQFRUFFFRUXz++ecufvOKkmLL/qO0OzGLi7I8K/TtrPqych3nWLKnSrmc4fmUFudTs8nNt36YZhof/oG+3lQM0jyNnK09007fsYsZNHh5BfVfWk74hGUlFmUtpeRCWg7HLmZQI8QfgAEtqluNqf/ScqugPU/hyTiLTsAoYK8QwpRx7iXgHeAHIcTDwAngPmPfcjS32SNorrMPAkgpU4QQbwCmhC1TpJTWvmTFYcUEOLv3mqexolpzuP0dp0OysrKIjo4mOzubM2fOmNN6O0oZfuHCBWrUqMGyZcsALQ9RSEgIH330EevWraNSpUp279OzZ08eeeQR9Ho9CxYsYNasWbzxxhtO79WlSxfmzJlDxYoVycrKok2bNtxzzz2EhYWRkZFB+/bteeutt3jhhReYPXs2kyZNsntv0HI9ZWdn06JFCw4cOMDChQvZvHkzPj4+PP7448yfP58+ffrw3nvvAVoakLCwME6dOsXGjRvNeZeefPJJc4qRUaNGsXTpUu644w6gIMEfQIsWLZg+fTpdunRh/Pjxbv1zKa6dE6fO0hl4N38Yywztrfr+OX2VmPCKdq8zvc13j6jMOqN3z0+7kvm/2PoeW2tOvp62b69hSOtafHBvlLn94Lk0dAJ8vXVcTNMM17HvrbOK4tYbJF46TQ3kTJCY2HL0IrENr127YdpNgJbLzN9HR83QghiWT1cftneZR/CYsJBSbsJxhExPO+Ml8ISDueYAt0TosUkNBVr68NGjR7Nv3z6HKcNjY2N5/vnnefHFFxk4cKA5H5ErvLy86Ny5MwsWLCArK4vw8HBzn7P05NOmTWPx4sUAJCUlcfjwYcLCwvD19TXnrGrdurXDynILFy5kw4YNHDx4kOnTp+Pv78+aNWtISEgw50LKysqiSpUqVKtWjfT0dNLS0khKSmLEiBFs2LCBjRs3mndC69at47333iMzM5OUlBSaNWtmFham1OlXrlzhypUrZgEzatQoVqxY4d4/iMIlebnZ/LXgTVrX8DM/ME00OaYleUyTtrWhh8zc6jBtxjdbEwHrt/RmNUJKZsFGwicsI7JmeZaOi+WjVf8yzai+WZSQbCUszqdmozMGV6zcb3/H/Mg38cx5QPv7PZniuq5EUkrWtS4f0AQcwJPdGwCakF24M4mX+mvpYL7adKxE7uMOZTeC28UO4HrQoUMHLl68yIULF5ymDN+1axfLly9n0qRJ9OzZ0/ym7Yphw4YxePBgXnvtNat2R/dav349q1evZuvWrQQGBtKtWzdzgj4fHx9z2nBnqb+HDh3K9OnTiY+Pp0+fPtx5551IKRkzZgxTp061Gd+xY0fmzp1LREQEsbGxzJkzh61bt/Lhhx+SnZ3N448/Tnx8PLVr1+a1116zShio0n9fH+KmjGS09yo4BghrzXW0hHTpz1FZo0hzXjKqnFrWqcD245qiIN2OLaAoSCnNf6MnjYWC9p1KxWCQZkFhoueH61nzfDdAM2xXKaeltrdUUV3NKnCHXXtQM60aDJKRX213uZbzaUVPbPnxqn9pUSuEnk0KghNNO7CqRhUUWNtMUrNtf2edG9jXNlwrKutsKXLw4EH0ej1hYWEOU4afPn2awMBARo4cyfjx482V8cqVK2c3S6wlsbGxTJw4keHDh1u1O7rX1atXqVChAoGBgRw8eJBt27YV+7vFxMQwatQoPv30U3r27MmiRYvMKdBTUlLMabwt0323bNmSdevW4efnR0hIiFkwVKpUifT0dHOG2sKEhoYSGhrKpk2bAKxStCuunariMpdlMBNbbIRXL1t96md/R2TOHA7LWmx6sTs7X+7l1pxNqmuV6Ia2qc2rdzQFMHv7FJd6E5dz35dbbdrrv2Srzz96oSAOJE9voFYFbWf0f7H1zO1frD9ic527xuMrmXl8uPKQlcBxxadrDvPw19bBiaadhSlwsH6lIPpFVgMcu9COaFfH7XsWhbK7syglTDYL0P6xv/76a7y8vOjTpw8HDhwwV8ILDg7mu+++48iRI4wfPx6dToePjw8zZswAYOzYsfTr148aNWqwbt06u/cSQpgr41ni6F79+vVj5syZNGnShIiICHOFt+Ly4osv0qpVK1566SXefPNN+vTpg8FgwMfHh88//5y6desSGxtLUlISXbp0wcvLi9q1a5tTfIeGhvLII48QGRlJtWrVnKb0njt3Lg899BBCCPr06XNN61ZYU45MDsuaxO1IYurdLczthR9WNUIC0OkEe17tQ9TrKwGtDkOonfQUJ4xv/v4+OtoY7Rp5+UXz4knNzqPFayv5akyM+W18x/EUwicsszECO+JqZh7bLNJp1KtUsFv98k9bFU9hYeHrpbMrQOZtSQTgzNVsK5WXI37ZbdfB08pTy4RJdZfvwIjev7l7372oqBTlCsUNxo32d/r35GguyhAeynvBygax7dglhs0q2H1a9j08bydrjKqbp3o0YFDLmtxWuSDO57Vf/2HelkT+eb0vp65k0efjDXw+opXbD3nAyg6xdFxnBn62qUjfq129itSvHETcjiTz+vP0Bhq+vIKKQb5W6UgANk/ogZ+3jpg3V2v3vy+Kzg0q0fbtNQ7v0bdZVb4cZTfjtxXhE5aZj3dP7m0WsP+cvsqAaZuYObI1/SKrmcclvjOArFw9TSZbx1R8/0g7Ot5WfDWUSlGuUCgckpGawuGvnyQ8LYFQO0nxmojTrJBtaVvP2rPJUlBsf8naZ8XybXva2iP8EJ/MNosxpjdvP2+dOaahqNlUV/5TUBCoqIICYPvxFC6mW2eY9fHS0bxmCHtPXWVk+zp8t60g75Oft47lewvilaJqh1KlvD/OsPRmcpfky1lmYZFj3G35+9haDP6w+P7ThrdEbzBck6BwhRIWCkUZJvHkScLnNCfa1ND0fpsxS+KT+EHfjZ3HHXusl/O3fpRsPHzR6vxsajYGg0RXyJvK20tnzsVU1GCywnUeikN5o3CceHtBdcO9p64C8N22kwT5ejGuZ0PeWXGQfL3kh/iCINPCWpnWdSuQcMI67celYsSODPxsk3mXtv+0lvTC1yhQW9YJ5Zwx0eEzCzWvyl5NqnJnVNEcDIpDmTNw36pqN8WtwfX++/z+y7cB+FPfgrbZn8NdX9h8xuc/xk6pPUxNBtvdSVes5vH1cv0oeX/lIbvtpmuzcvX8EJ9kztXkaWpVCOCvk9r3eMRBfEdGrt7sCTV/+wlGtK1r7iu8q7D3O9Abri2aetISrc7LMWNixmA/b05fzXYr1qOkKVPCwt/fn0uXLimBobghkVJy6dIl/P2dqzZKkijdUQAeyHuB89jPZmpZr+ERYyrxJ+bvshrj7YawmLFeu1dhYWBSQy3+6xQvLPqbhTvdSxHTpVHRgt6OvHU7ie8MML+1J18uiIUovOOxpL7R6J14KZNDZ41v+t46yvtrv5fG1Qo8u1oXqrS371QqS/6yb7x2RmautUtsoK8XULBjm7L0H3Oft5O1lyRlSg1Vq1YtkpOTuVGTDCoU/v7+1KpV67rdz2B8X5QO3hv1Bmnl/nnsgtHd+orzoLMKgT5ctpO2W0rJ/zZaexn5GD19TLuV01fdC2jb8K/r/8crn+1Cn483sHRcZ7cEmj2e6N6ABTuT6NKwEuMX/Q3AtokF9pel4zqz43gKHRtU4q6WNa2M1aCpi+5q6ThRtr18Ulez8gj0LXg8Fy4MtSgh2XxsEiSepkwJCx8fH+rVq+d6oEJRRgjW5bHPEO6w39KIDHAxPZeEE9a2i9fvbGZz3ZiO4XxiJxXFnM2JfLDyX6u2wvUjvvzzGH5eOp7rE+FwXduPXTIf392qJj/vsv/23qhqObtR5DVDAxwKvNXPdaHXRxvM5wHGh/H4RX+bjd+mHFKg7ao6XkMgnGkd5f29zUF2O46nWAXXmXYxJkyG85qhAbx6h+3v3xOUKTWUQqGwpppPBlloD77qIbbqr2p22v6vUODYmI7hNmOe7tmQtc93ZWCL6hx8o6BW2RtL95uPezWpAoCPzvYxNG2tbUCcJefTCryY7BUeahNeged6N3J4fffGjlVYDaqUszoP8Cl4czcZv53xcGfnL6Qj/7fdSo1nilx/b0hBDMuepKu0NrroWrLosQ5W5493v42QQM+WdTWhhIVCUYapmadF0vdoXIULaTk2/S/+9LdNm6V6yVHuJyEE9SsHM31EK/x97KtJHjfmO3JkLyjs1mqix4frGRf3l/n8y1Gtbcb8+FhHnurZ0O71AIstdiKjO9R1OOUYw/AAACAASURBVK5ikK+VsHCHB+wIT0s2HbnIMgsX3BXG45SMPIKMu5g5m4/bvbZwYkZ7gtZTlCk1lEJxq7D2mzdpd/wLgnyL/19YIikvsjhnqGD2+EnJyLVSsfx7TrNRRNYsz75TqXbnKS6t6jgvDxrz5mqOT+3PukPn6dqoCl46gZSSYxesS7Z6e+nY9GJ3jl7IoKubRu9HutQ3q8mmDHJcKOuL+1vZCLMejas4nbtWhQCzAPpm6wmH485ezaZaiL95F3XobCpfP9SWITOtU5YUFj71KgWRmpXHpYzca06RUhSUsFAobkLSD28iTwdEjyj2HHl6A99uO8mP+q7mttNXsqyEhYkHO9bj+R/3FPtejgzeJupXCjK7h1oStyOJlxbvZVyPBvRoXIXBX1hXltwwXqvnUqtCoDm/kzuYPJjC7HxXSyzTgZuwFyBniRDCLIC8dIJF8QXG6C//PGo+bj91DRMs4jvujalNsxq2BaQGFopqrxkawCXjrqu4RvvioISFQnETEkwWJ2UV/r7t+SK7kJr499RV3thkHfmc4yA/U0S1cjZt9mwcjiic2txUkc7Es70bWamWTLy0WKs589naI3xmx44RYifi3B0ycvROr/fz1pGTb6BWBU1YzBzZise+0+wMqVnuZ8f19daRYxFsOHXFQav+dyzOfb115qy5ljQsZEPZdKQg4NGd+JaSQtksFIqbkGCRRboMYHExfPhN2EsJ7shnP7Kmba2JyQObun2vdvXCrM6fMNorTNwRVcOh/cMZhSPH3cVkILe3mwH4+fGOvHlXpPnh3bBqwQPb8mHtiquZeeTmG0jLtr+ram7xe83Jsy+onRmww4Kd74xKEiUsFIqbjD8+fYy2ukOkEWjXKF2YpJRMwics47SFq+j51Gyr+InPR7QCwGAnYDXYz/4D+fYiZDedek9z87GpdkRJ4CyYzhn/F1sPIWDVs13s9jerEcLI9gWGby+LN/76ld2vo7LAGGDY/LWVdr22+ljET9xWxXZeVwK0gp2Mvp5CCQuF4iaj6iWt+M6s/AF21UOFmfyLljLCpNJJycil7dtrePTbBPOY8gGaQLBXO9reDqSogWDl/X0Y1qY2YO32Wpil4zrTtVFltw3VxcXHS8fxqQOsdgzOsFSjubJzOGKKhduwiQ9XFcScmILwNk/o4XQeS/WfPddmT6GEhUJxE3H58mWidcf4Vd+BBBnBV5vsu1haYqp7YMqDZK8gj+lhaK9GgqnGwxgLF9PMYuQmmni7lnbdsjZDYSJrhvD1Q225kln0BHyexFJYONpp2eOz4S3Nxz/t0gzd3jrhNEVHzdAAwoJ8eaiT/XiNzS8WCJOg6xS9DcrArVDc0Jw6e4aMOXdTP0TgrdPhnanp2PcYbnN7jq6NKrPx8EWqltfUP/YeVN5Gf/3CO4uaoQHmfEevD4rkayeuoK4ICfRh84QeVHJDzx7iRL2ydWIPOkxd6zL4rSSxFBbli2BUt1fitG5YIMueiqXxKwW1KJpWt/aCSnilt8M5LVVv9gzinsJjOwshxBwhxHkhxD6LtoVCiN3GT6IQYrexPVwIkWXRN9PimtZCiL1CiCNCiGniev52FIpSZsGXb9Modz/J5y5ywbsqmeXq8Yu+I4v1nc1jXNWuNr0J/3sunfAJy5hdKDfTzJGtMTnVFE7yl6s32KTjAK1wUHGoGRqAn7frt2HLvE8rno616qseEkD8pF681P/6FYjSWTx2XMVZWBJkZxfi46WzCVR8fdD1SdlxLXhSDTUP6GfZIKUcKqWMllJGAz8BP1t0HzX1SSkfs2ifATwCNDR+rOZUKG5ldLnaTqJn7ge0OfoQ+7vO5Om8J0mhvNnfP/LVP/jTSVK9wkbgwoFi/SKrmQPuHv46ngYWNavz9QZzVlhLxnaxn9K7pBjZXqsjvXRcZyuPn/uN9aUrBfvZuON6EsvdWFFqR/jaUbldtqNiq+aiiNKNgMeEhZRyA2C3Wopxd3AfEOdsDiFEdaC8lHKb1PKKfwPcVdJrVShuVJpW9SNXeqFHexP9bfdpAOIeaW+VDvv5H/bQcspKdiba/pcrXHvCHpa7k3yD5LKxaM/lzDxzvWxLOjf0XEU2gNfvjOSPZ7oQWTPEykbwTC/H+Z48SUmqfjo30Iz3T/UocB8uqqH646FRdtOceJLSMnDHAueklJZpKesJIf4SQvwphDDtO2sCyRZjko1tdhFCjBVCxAsh4lUacsWtQDkvPbkU6Mh/NsZVhAb6WKWwvpiew+XMPO4tlCoC3PPeube1dVr0A2dSzQJkn53kee6okq4FL50we3pZ5maqXIJut0WhJIPf3hqsRXc/1yeCWaNas21iT7u7N2cMblmLvs2qldia3KG0DNzDsd5VnAHqSCkvCSFaA0uEEEVW4kkpZwGzAGJiYlSFI8VNzYnDf9PxwkKuYOt/X6dioMOAtPAJy/h0WDSDorX3KnuRz4UpbLT97e/T5vQclnaMva/1ua7qH9De5GMbViqRMqrFxaTyu1aX3nX/7WZlr+hznR/418J1FxZCCG/gbsC8h5JS5gA5xuMEIcRRoBFwCrB85allbFMobgr2/bWNwMVjCNLlEeiro5yf6aFsfJexCoKzbquboSX3226wNeQG+Hg5jXV4esFuBkXXJPlygQrpgY7hzNuSCMA9rWqZXTnB1p01bkcSdcM0IXW/RXBaOf/rkw67MN8+3K5U7mtCCMGWCT3s5s1yxQ+PdiAlI5d+kTePYLBHaewsegEHpZTmv1QhRGUgRUqpF0LURzNkH5NSpgghUoUQ7YHtwGjgs1JYs0JRLNb9vohxurP8pm9PZqY/Q5vVAoxv5la678JtgvnbT7Jb3sbfle5geN1Q4nYUlBvV6YTL1Nk/7EziBYsU46/d2Yxtxy5x8GwaPZtUsRIW9vTwprxFsR62T9ws1LCTVNAd2hbTc+xGw2PCQggRB3QDKgkhkoFXpZRfAcOwNWx3AaYIIfIAA/CYlNJkqXsczbMqAFhh/CgUpY40GNi99Auah+Q4DLIanT0fBIzPe5Rs/Bg6yP38Ry9v1spzVs3K5fU7I62EBUCGi8C4F+zUojCpcqqU8+PFfo05mWJrvC5MUfXpilsTjwkLKeVwB+0P2Gn7Cc2V1t74eMBxwnmFopT46dVBDPHa4HRMiIB9hnCyKb5h9lxqjt1Yh++3nyz2nLl6A//p5l5g383g1qnwPCqCW6Fwk51b1xG4cjy3VQ3B38eLIV5ajqaRYXF895j9fD4Rr6wg18l/s4vpOaw5cI57W9dm0i/7GNMh3G6+JyG0FBH20nEUhRn3t+I/83fRoX6YwzHl/LxJs3ClvZ75hxQ3LkpYKMosK1/pQXfdbgpq2RQ2Ols/mNsYf56+WJvKNeuzVd+cafmDqVKxKvjYf6Dm4NggujvpClN++4ddJ6/w2dojJF/O4vvtJx1mGi0sKPo3r8byvWet2v4c342u7693eM/bm1d3OP+79zQnyM+b2yOrc5tFYJ6zXE6KsoMSFopSw6A38MOrg+ld8TxhQUYvG8sHtfnZ6OAhLqXdvgtpOQghqWTyXLFzXb5B0lN3gr2yPtGdTXGe1kbmz9cfNT6gBRKQUrDR0JyaEd0Y16MBoz/WVFBTnLylVy3vx7nUgiyrUkqEEMzZdNwqC2ny5Sx7l9vFVFt66t0tWL73LC/2a8y7vx+kU4MwswdTYaJrh7qcd2ibOnbbVYYdBShhoShFEg78yzDv9ZAKVO1r5Qmk/RDY9RyyO854LgTbz51BAnfcVsPBHILfdp/CQG3m5N/Osp5P2F3f+yuX2W2vkJNvVajG2Zt34XoT09ceYVzPhmw/fsnhNaAJFRPLn7LOjZRvrLwWEuBj3iW4sj98cG8Lp/2FeapnQ6atOex6oKLMoISFotQI0ml68fF5Y3n//vdLZM7sPD1PJmjZPAcO6e/wrfjZnfYFgTtUKe/PHdMLypF+v/2k3bfy7Dw9hU0MH676l3E9G/LHP+ec3iPLGAj3Yr/GNC1Ul/kOF7mJomuH4uMl+PGxjoRP0L5ngyru1W0w8VzvRkpYKKxQwkJRaviiRQjnyJKr9nXFohpZTr7BJrunPUyqIXcp7Jm0J/kqBoO0Sdj3wiJb11VXmCKFTxnVUvZ2La5cWZc80anI97XHnAdiaFC5aEJGceuihIWiVFg2dTht83YAkIMPeoMskTQSlmVBM3P1bgmLXL3Bbq6jxtXK2U0xYW+VlzNzCQvW3GNz8vXk6yW/7jlt7m9RK4S/k21zLBUmO8/AzD+PsvGwlttszubjPFSobkNIEeop/PZkZ/addn1fe/RoXNX1IEWZQbk5KEqF7tlrqGy4yPf5PdhuaGy3eltROJeaTfLlTDq+s9bclpnrvM6DifTsfKauOED4hGVsP1ZgS/BzIGjsFQD6eVdBFppB0zfT7NU/zOe+3jpevaMg1Vn0lJVO1/POioPmqnb2chEVpaRp81ohDG9r33CtUBQFJSwU1xeDgYzMTAJFDh/n3cNL+f/HFcqRbzC4vtYBu5Ou0O7tNXR+d51V+9MLdjN81jamrjhg1Z6nt77XmavZfPmnVhBo6Kxt5nZ9Edb01vKCexTejcRP6mWVTtxSVdYmvAL2MKmfBrYosE+YosSLWv9aoSgJlBpKUeJkZqaTfmgDlQK9yMnXKq0ZJKRdOkvY6qfNOVRTCTRfU7icZ1H4MT7JbnvCicsAbD12yVz/GeCnhGSrcf4+1u9MyZczqVUhkINnXGc5rRjkS0qG83rRQb6O/5tZGqEtMWV8bVmnwOX1j2e7sON4inJlVZQKamehKHF+nPkGVX4Zji7uPgJ+HIb3gqH4LhxK2OqnAViqb8e7ecP4RV9giM3Xa8Ii4cRlVu8v8BQ6cj6N+dtPkJSSSfiEZTw+P8HmfpuPXHS5piyLPEqF7RjZedY7iNs/2YjBIN2Klp49OsblGFe2mEdiNZvE0nGdbfos13pb5WClUlKUGmpnoShx/FIOgjcMypli05dOAEelbf0q087inhlbAMzxA70+ss69VDhiGeDemNq8/8chp2u6lJFDLV9tJ2PyJvpPt9uYsf4oadnWto20nHxSswtURQ92Cmfu5kSbOfe93teqipsrnurRgGkWtSVMhXxeHtCUlwc0BSCqdih73Khsp1Bcb9TOQlHilBOZHDbUZI9sYPOxJygAPl79Lx+t+tet+Qurbco7KAJkySerD/Pe71rK7UlL9gJaHWeA4bO32YzPNO5E2oZX5KkeDc3tHW8riNYuLCj0BmmjTrO0SdwZbR0f8ULfCJv7KkGhuFFRwkJR4gSSQ2YRs6z+svu0VRBYvt65cdnSTuCOJ9WihGS+WH+UrFy92R5QKdhxfIfJq6pGqL9V8PjcB9vYjDUJnTy9wSqnElirkQpXo7sWO41Ccb1RaihFkfn1lX4M1G1DJ+w/7Lp7wRZ902u6R4OXVzjVz7d6YxVH3+7P3M3H+WBlwY5kz+Q+bDl6kf/M32X3uiaTfzcfe+tcvys1qlbOqjqcvVrMj3Wtz5vLDnD6im1+p6FtapuPAwsZunedvMwwZYNQ3CQoYaEoMj11u9AJySf5d/NMz0ZWfZ+tO4zeIPlTH3XN94nbYV2v4fU7m/Hqr/+Yz+/6fDN7TxUEnB19uz9eOsHtzauzZ3IfPl9/hFkbjjmcv2uE63rKNUIC8NIJfn2yE946ndkTqXODgupxpmC7Hh/+aXN9F4s4Cf9C0dimGtmOCFIusoobCCUsFEVCb5B4YWBm/h18kj+EZ7oXpLs+cSmDD/9Y73KOLRN60PGdtTZ1E1wxpmO4lbCwFBRg7XUUEujDE90bOBUWgXaC7r5/pB0jZm83n5tKYraoVeDC+uf4blQpV5CS3DJS28SjXerzYKd6lLfYlXh76Xh7cHMqBPrw657TxNiJsfh0WDRPL9gNQLfGVRyuXaG43iibhaJI5OkN+JBPLrYP2vWHLrg1R43QABLfGUDj6p7NOxQS4MPa57s67C8crtCrSVXqVAy0arNXd7luWBABFm/96//bzWbMf/tG2C0aNKJdHW5vXp0ZI1vbTTEyKLomx6f2Z/qIlnx037XvzhSKkkLtLBRFIi8vD38hyZO2fzqWb/2WbJ7Qg5qhAYRPWEbtigUP30+HtWTT4Yt2a0U7oleTKqw+cN7t8fUrBzvsE0IwpHUt7oquSWTN8gT4elm50T7ZvYFb96gbFmjTdi11q4UQVpHbCsWNgNpZKIpETmY6AEEim77NXCeae7l/E2oa387/eb0v657vZu6rERrAfRYGYEcMaV3LrL//aGi0W/d1xf+MwXQf3BtF54aVCA30xc/biwAL1dSzvRs5utyKwhHVznYzCsXNiseEhRBijhDivBBin0Xba0KIU0KI3cZPf4u+iUKII0KIQ0KIvhbt/YxtR4QQEzy1XoV75GVqcQCJspo56toZHSziEoL8vPF28cYdU7cCj3atb9X2wb1R/DOlHwDl/X3sGoYHRRftTdxyXZYE+XmzeUIPDr3Zr0hZcP/bRxMsIQE+TnczCsXNiifVUPOA6cA3hdo/llJ+YNkghGgKDAOaATWA1UII02vd50BvIBnYKYT4VUq5H8V14+ieDdTmPL5eOnzOahlX02UAO46nuLzWnVgCXy8duca4iv1nUvnm4bbmxH4//aejzfgedgy/nw5r6XD+v1/rg49Ou0fU61rGV2fJ+GrasVO44onuDQj282Zwy1pFvlahuBnwmLCQUm4QQoS7OXwQsEBKmQMcF0IcAdoa+45IKY8BCCEWGMcqYeEmKVfTOPthJxr4X7UbI+AKKQ3cll0QVWxyGD0tw0jLyWfV/nP0bmqtFnplYFN+Skhm/5lUt/IrbZ7Qg7eW7WfJ7tNk5uoJ9PXm8Fu3k5Wnt/ImMuHv48Xyp2LpP20jAG8MamYzxhLTHAF48e49zbmSmVfiyfiEEDzQqZ7rgQrFTUppGLifFEKMBuKB56WUl4GagGXOhWRjG0BSofZ2jiYWQowFxgLUqXNrBzsd2x+PWPwodUO80Dl58PlmZtFUd4I/s1rQtb3DX51DLqTl8Pu+M6w1tGTaY3dy94wtZONHstTiByb/so/eTauy6XBBMr+HO9fDx0sw+Zd/7HoEFaZyOT8aVy8Pu0+bs6z6eOmcGonrWBiVP1j5L6M6hLv1feyVP1UoFK653sJiBvAGII0/PwQeKqnJpZSzgFkAMTExt3QuhSW//sxzeUe45NedsJDydsecT8shRZ/LP6m1mZp3P/EDRhT5Pjv/PsPk3Vo09LdHAzgirdUsZ65mAzDyq+1W7aPa1+X2yOrmZHmu+NPodmsq+uMKy11SSZURVSgUjrmuwkJKac49LYSYDSw1np4CLN1iahnbcNJepqmU/i94w8HO0+jUxPZtOV9voO3LK675Pk98X5A2Y5exPgRoAXDO7BFCCLcFBWg1J4qCr0U0dL1KQU5GKhSKkuC6us4KIapbnA4GTJ5SvwLDhBB+Qoh6QENgB7ATaCiEqCeE8EUzgv96Pdd8o2J6TBu87Rtjz6XluDePdH8DtuZgQXzDwrHtAWhS3XpXM7il8xQWjihnzBw7zA1XWhOH3uzHjpd7Fut+CoWiaHjSdTYO2ApECCGShRAPA+8JIfYKIf4GugPPAkgp/wF+QDNc/w48IaXUSynzgSeBP4ADwA/GsWWajMxMxniv4qihutl9dd3B86yzeJhnupFGY+w38dSbuJyklEy7/Y52Dj5egphwLQ3GgTOpVn1nrtom03OHgS2094i7W7nvTeTn7WWVdkOhUHgOT3pDDbfT/JWT8W8Bb9lpXw4st72i7DJ1xmzeBA7IOjw5byd7X+vDg/N2AlC/chBrn+9GTr7zFN9SSlYaK9IdvZBO7Yq2UciOUn9/PDTa6jwzt0AwbTvm2p3WHhP6NaFFrVCHNakVCkXpoiK4b0I6XfkNgGn5dwPQ/LWV5r5jFzIAyMnXW11TOL5s69ECG0HhMqMmPl93xG57TF1tV2EqB2qqDwFwX0zx4gxCAn0Y3raOqi+tUNygKGFxE1JVaIbmIw6qzgF8se6o+bhLo8oYJPxvY0EG1pUWda59vOw/oE1FfR4qFD9gcoc1xVCM/Sbe3FfYhqFQKG4NlLC4yThxcBetdEf4Td8eg5N/vsiaIebjzg201BZvLjtgbsuzqET30y77DmZhxkpyd7eyL5QuGI3o/5wusFsoG4JCcWuihMVNxlffzgNgg6GFwzGHzqbxqbFEaUiAD1m5tvaLpjUKdgDl/OybrryNuit/H51NG9hmVq0Y5Gs3FYdCobj5USnKb1Dy8vI4vnczVf31hPjpwKAHqSdap6mXFus7O7y27ycbzMeLH+/I4C+22IzJyi2waXy54RgT+zexGZNpHGNZO9qyvkNuISP6va1rWdV5UCgUtw5OhYUQopWzfiml/ULHimvmuy+m8ODlaTbtd3vBeRnK6E4NmbP5uMt5qocE8MsTnej2wXqr9vRCrrVrD55jZ+Jlejetym2VggkJ9GHSEi0MJsjXm8kDmzJl6X6aWtgkCifju715dRQKxa2Jq53Fh8af/kAMsAcQQAu03E4dPLe0sk3YxZ3gBSNzJ/Ldo11B5wU6L6atO868fXnsuqMpv+w+xaWMXPM1/ZtXY/nes1bzBPh6EV4piOjaoexOuoKUkn9Op/LJ6sNW4x6apxmpZ6zXdi7dLOpTB/l582CncAxSMqJdQbR44SSBde243yoUilsDpzYLKWV3KWV34AzQSkoZI6VsDbREpd3wKPXFGfKljk2G5vx3RyDUbgs1W/Plv8GkoL3dVylfYEx+9Y6mTutL7E7Sci59t/0kAz/b5PL+hUukCiH4v9j6BPoWvF+Yoq5NVAjydf3FFArFTYm7Bu4IKeVe04mUch9gq+RWlAiXU9OJ1CWy0qBVc1uUkMzRC1qFugwLW0PcI+1oUCWYQ2/248FO9ZgxsrXVPM9ZVHoz1ZZ+Zck+qzHfPtwWe1RxI6/Tf7rdZj6+u5hpPhQKxc2BuwbuvUKI/wHfGc/vB9wvnKwoEl8v/o1ngKOyoPrb+dQcbitUgS000JfVzxWU8Cxc2e2png3Nxy3rhHLSTlqPTrdVsmkDLWMtwPC2jnM1VQ8J4Njb/Vm0K5l7ipCmQ6FQ3Hy4u7N4APgHeNr42Q886KE1lXlCMxMBWKsvqP62+UhBvYiYuo5TYnQ32hoaVytn1W6v5sWXo1qjc1E6tJyd4kNW8+oE98XULlIJUoVCcfPhcmchhPACVhhtFx97fkmKeqE+cA7eGt2b/l8nAjB93RG+3qIdt3IiLOY+2JbcfIPNw3tUh7os/svazNS3WTWXa7F0sVUoFGUXlzsLKaUeMAghQlyNVZQMFTK0tBx1a1nbAdKM7q6zNhyzucYSX2+djbBoVacCbw9ubnf85IFNzcd/v9bHqs+e6kqhUJQ93FVDpaPZLb4SQkwzfTy5sLLMlTwtfiEwOISESb1KbN4BDuIgHupckPupcM3rP/+9UHi4QqEog7hr4P7Z+FFcB3T6bNKlP8FCEBbsfrU5VwT6eZl3HIWFULeIyiRf1mpRPNqlPl8ady+mOhMKhaJs45awkFJ+7emFKAoQ+dnk4EOwg/6NL3Qv1rw+Xjp2TepNsL+3jZpq3oMFLrQv9GvMHVE1yDdIG0O5QqEom7glLIQQDYGpQFO0aG4ApJT1PbSuMs3ZS5cJ1xUEuCW+M4AtRy4y4n/bAewWKnKXkEDn3k2gueBaZq1VKBQKd20Wc4EZQD5aOdRvKIi5UJQwIaSTKoOs2q5FQCgUCsW14q6wCJBSrgGElPKElPI1YIDnllW2qSUukoqtsBjdoS6LH+9YSqtSKBRlGXcN3DlCCB1wWAjxJFpeKEcqdcU14kcuFXxt4xumDIoshdUoFAqF+zuLp4FA4CmgNTASGOPsAiHEHCHEeSHEPou294UQB4UQfwshFgshQo3t4UKILCHEbuNnpsU1rYUQe4UQR4wuu7dkqHBubi4pBzfC0bUEi2z259hPw6FQKBSlgbs7ixQpZTpavIW7aT7mAdPR7BsmVgETpZT5Qoh3gYnAi8a+o1LKaDvzzAAeAbYDy4F+wAo313DTsHjaswxN18xAlQUky8ourlAoFIrrh7s7izlCiKNCiAVCiCeEEPZDgS2QUm4AUgq1rZRSmqrubAOcZp8TQlQHykspt0kpJZrgucvNNd9UtEjVqtvt77uAO3PeYHOt/yvlFSkUCkUB7sZZdBVC+AJtgG7AMiFEsJSy4jXc+yFgocV5PSHEX0AqMElKuRGoCSRbjEk2ttlFCDEWGAtQp04dR8NuSBqJJHYYIrjvFwNwGySmlfaSFAqFwoy7cRadgVjjJxRYCmws7k2FEC+jueHONzadAepIKS8JIVoDS4QQzYo6r5RyFjALICYmxnEloBuMP7ftoKuQJBuU6kmhUNyYuGuzWA8koAXmLZdS5jof7hghxAPAQKCnUbWElDIHyDEeJwghjgKN0LyuLFVVtbgFK/TtOXSUrsDv+jbmtkHRNRxfoFAoFNcZd20WlYApaDW3fxdCrBZCvFHUmwkh+gEvAHdKKTMt2isbU6EjhKgPNASOSSnPAKlCiPZGL6jRwC9Fve+NTueqeQBckQXeyM/2auRouEKhUFx33LVZXBFCHANqo73ddwSc5o0QQsSh2TcqCSGSgVfRvJ/8gFVGD9htUsrHgC7AFCFEHmAAHpNSmozjj6N5VgWgeUHdcp5Q+pwMADIpSBpokDeNFk2hUJQB3LVZHAMOApvQXFkfdKWKklIOt9P8lYOxPwE/OeiLB27paDT/hJmgg9OyILYiLKjkss0qFArFteKuzaKBlNLg0ZWUYZrrErkig0ihPAA1QwPcSvinUCgU1wt3bRYNhBBrTNHYQogWQohJHlxXmUHqtbCTPyyM25sn9Cit5SgUCoVd3BUWs9HsDXkAUsq/gWGeWlRZIicri5CjlAAAFHdJREFUHYAjsgYJk3oVu1aFQqFQeBJ3hUWglHJHobZ8uyMVRWLT7n8ACPXTquKpVOQKheJGxF2bxUUhxG2ABBBCDEELpCvzZGamM/P9CTwVWwNvnbuyt4DQfQcAuBTcoKSXplAoFCWGu8LiCbTI6MZCiFPAceB+j63qJuK9tybyms+3sKF418cAGdKPvl06l+i6FAqFoiRxN87iGNBLCBGEprrKRLNZnPDg2m4KRnqtBqBh9jccnnpnka8Pn7gMgN9rNS7RdSkUCkVJ4lRvIoQoL4SYKISYLoTojSYkxgBHgPuuxwJvVHJyc9k5uQ0NdKdJlpXIwxuEKPoH7ePrVXQVlkKhUFwvXD2hvgUigL1oNSXWAfcCg6WUgzy8thuai5cu0Eb3L1nSl5G5EwEIn7CMPH3xwlEqBPqW5PIUCoWiRHElLOpLKR+QUn4JDAeaAn2llLs9v7QbG4PR5XVy/gMkyurm9reWHXB/DkNBSg+V3EOhUNzIuBIWeaYDKaUeSJZSZnt2STcH+dlavYlM6W/VPm9LIr/tOe3WHMmXs8zHFVTEtkKhuIFxJSyihBCpxk8a0MJ0LIRIvR4LvFHJy9KERQa2OZzGxf1ldX4uNduueuqpBQXjbtHS4gqF4hbBqbCQUnpJKcsbP+WklN4Wx+Wv1yJvRPIyNVmZIQOcjsvO09Pu7TVM/HmvTd/upCsAtKt3LQUHFQqFwvMoF5xisOznb6m9+jEA0gng4c71bMbojfaI9Bxj7qd9Z636j11INx9n5ek9tVSFQqEoEdwNylNYYPjrO8p7ZfJu3jAG9+3F2G6NeGVgU46cT6fXR38C8Omaw5Tz8+at5ZrBOy3HOjtKSkZBhvfneqtCRwqF4sZGCYsiIrOvUo4szslQZujv5NcGVcx9QX5e5uMf45M4c9WxL8DMP4+Zj7tFVHE4TqFQKG4ElBqqCGzZvA7D1Lp089rDYUNNAFrUCjX35+sLHGDtCYpsC3XT6gPnPLhShUKhKFnUzsJNTp/4l2YrR+AlJJ/m381SfXubMZXLOa9ul5KRS41Q5wZxhUKhuBFROws3+XjWbEJEJvsNdfk8fxCHZS2bMf4+XnauLGDbsUs2bf8bHVNia1QoFApP4VFhIYSYI4Q4b6qwZ2yrKIRYJYQ4bPxZwdguhBDThBBHhBB/CyFaWVwzxjj+sBBijCfXbI/t373Gw14rALgv9xVycRxAd1d0DYd9ufm2sRa9mla95vUpFAqFp/H0zmIe0K9Q2wRgjZSyIbDGeA5wO9DQ+BkLzABNuACvAu2AtsCrJgFzvQg/PJcq4jJL9e1Ix7ka6cXbHWePnVAo1sLHSwXiKRSKmwOPCgsp5QYgpVDzIOBr4/HXwF0W7d9IjW1AqBCiOtAXWCWlTJFSXgZWYSuAPIaUkkByWKLvzJN5T6NlidVYMNbWblE9xFqYdI+oTHTtAiO4lJIhM7YAkKdXGaEUCsXNQWkYuKtKKU1V9s4CJj1MTSDJYlyysc1R+3UhX28gkGwy8Lfpa18/zO41R9/uz/K9Z+jeuArBft5sOXKREf/bDsCljFziT1z26JoVCoWipClVA7eUUlKCCVeFEGOFEPFCiPgLFy6UyJw52Zl4CUmWdO7pZImXTnBHVA2C/TRZbClULqXnOrpMoVAoblhKQ1icM6qXMP48b2w/BdS2GFfL2Oao3QYp5SwpZYyUMqZy5colsticTFPCwIKdRZ2KgTzVw/2a2Tpdgeqq7yfFrL+qUCgUpUhpCItf0artYfz5i0X7aKNXVHvgqlFd9QfQRwhRwWjY7mNsuy6cPX0SgEyL7LIbXujOc30iijRPsxq2eRcja5bpXIwKheImwtOus3HAViBCCJEshHgYeAfoLYQ4DPQyngMsB46hlWydDTwOIKVMAd4Adho/U4xt14WlW7Q6T2ky8Jrmub9dXavzfs2qsXRc7DXNqVAoFNcLjxq4pZTDHXT1tDNWAk84mGcOMKcEl+YWWRnpPHhuKgg4JGu7vsAJwf7Wv+p7WtsG9SkUCsWNiorgdkJ8wg6qiCuclRVIktdmA+lTKPjualaeg5EKhUJx46GEhRMyMjTj9gt5Y8k3bsJ2vGSzKXKLwqlAyvmrtFwKheLmQQkLJ5STmrAwuc0+17sRVcrbxlsUh+4qLblCobiJUK+3Tjhy+gKdAAOCDeO7U6vCtWeM1Ql4sFM9fL2VnFYoFDcPSlg4oVm1YEiGKwRTJ+zavKEA9k/pi4+XDh8vJSgUCsXNhRIWzsjTChhF1atWItMF+qpft0KhuDlRr7hOEPlZALwwILqUV6JQKBSlixIWTkhNTwfAy69kjNoKhUJxs6KEhRMOJGlpq4SPKoWqUCjKNkpYOKFL6EUAKpUPLuWVKBQKxf+3d/dBdtX1Hcffn31IQp5DssU8oEEmZAakxJBJsTJWhwEVO0QLSpyOEqTiWLBlOtrB/lGplXbqAyit1UkVG2t9QCqSIhBRqVgrJJECSXiMBJoseVg2Tchms5u9u9/+cX4Xbjf37r0ke/fcu/t5zdzZc8/93Xu+vz2793vP75z7/eXLyWIE3YV0RCHPaGdmE5uTxQimtAzQGeUnODIzm0icLEbQOthPtEzKOwwzs9w5WYxg+sB+CmrPOwwzs9w5WYxAA7204eqwZmZOFiOYX9jJswPz8g7DzCx3ThYjKNCKiLzDMDPLnZPFCNoZ5OkTnCHPzGw8cLKoYGgomKEjnDxrZt6hmJnlzsmigqP9WRHBxSdPzjkSM7P8jXmykLRU0iMlt5ckXSfpBkmdJesvLnnOJyVtl/SUpLePRZx9PQcBOKRpY7E5M7OGNuYTLETEU8AyAEmtQCdwB3AlcHNEfL60vaQzgdXAWcAC4CeSzoiIwXrGOXAkm1J18qz59dyMmVlTyHsY6gLgNxHx/AhtVgHfjYj+iNgBbAdW1juwgd79AEzy9yzMzHJPFquB75Tcv1bSY5JulTQnrVsI7CxpsyutO4akqyVtlrS5q6vrhAIbGMiSRGH6ghN6HTOz8SC3ZCFpEnAJ8P206ivA6WRDVLuBL7za14yItRGxIiJWdHR0nFB8PT2HARhscbkPM7M8jyzeCTwcEXsBImJvRAxGxBDwT7wy1NQJlH7ZYVFaV1cayo4s2to9S56ZWZ7J4v2UDEFJKj2T/B5ga1peD6yWNFnSacASYGO9g9vT1Q3AoULeI3VmZvkb86uhACRNAy4EPlKy+rOSlgEBPFd8LCK2SboNeBwoANfU+0oogO4X9wDQMXNqvTdlZtbwckkWEXEYmDts3QdGaH8jcGO94yq1cNZk2AULFrrch5mZx1gqiP4eAKZMn5VzJGZm+XOyqORoShZTnSzMzJwsKlD/IfqjnZZ2T6tqZuZkUUHLwGEO66S8wzAzawhOFhW0FXo44mRhZgY4WVTUVuilT75s1swMnCwqmnvkOfpbnSzMzMDJoqL+1qmcNNiTdxhmZg3ByaKCSUN97J+6OO8wzMwagpNFBW1DfQy1+QS3mRk4WZQVQ0Ms4EU6PQplZgY4WZT1Quf/ANDdp5wjMTNrDE4WZfQd2AvAc1PPzjkSM7PG4GRRxuBAHwCX/s7pOUdiZtYYnCzKKCaL1vbJOUdiZtYYnCzKGDxaTBaeUtXMDJwsyhoc6AdwxVkzs8TJooxC32EA2idPyzkSM7PG4GRRxmDfIQCmzpyTcyRmZo3ByaKMwSMHAThp6oycIzEzawy5JQtJz0naIukRSZvTupMl3SfpmfRzTlovSbdI2i7pMUnL6xnbjicfBmDqjNn13IyZWdPI+8jibRGxLCJWpPvXAz+NiCXAT9N9gHcCS9LtauAr9Qxq+rTsXMWkKS5RbmYG+SeL4VYB69LyOuDdJeu/GZkHgdmS5tcriJkt/exlbr1e3sys6eSZLAL4saRfS7o6rTslInan5T3AKWl5IbCz5Lm70rr/R9LVkjZL2tzV1XXcgc2Jg/Tg71iYmRXlmSzOj4jlZENM10h6S+mDERFkCaVmEbE2IlZExIqOjo7jDuys3k30DPnb22ZmRbkli4joTD/3AXcAK4G9xeGl9HNfat4JnFry9EVpXV20EASuOGtmVpRLspA0TdKM4jJwEbAVWA9ckZpdAdyZltcDH0xXRZ0HHCwZrhpVhUKBFgW75p1fj5c3M2tKbTlt9xTgDknFGL4dEfdK2gTcJukq4Hngfan93cDFwHagF7iyXoH1HullJvBbc2bVaxNmZk0nl2QREc8C55RZ3w1cUGZ9ANeMQWgcOXSAmUCb60KZmb2s0S6dzV3P/mx062ibv2NhZlbkZDHM7T/fDEDvjNfnHImZWeNwshjm0AvPZAtzTss3EDOzBuJkMcx7Wv8TgN871/Nvm5kVOVmUKhzlNO1m51AHra2teUdjZtYw8rp0tjHFEDcV3kvv7KXclHcsZmYNxMmiVPsUPnPjF/OOwsys4XgYyszMqnKyMDOzqpwszMysKicLMzOrysnCzMyqcrIwM7OqnCzMzKwqJwszM6tK2VQR44+kLrIJlI7HPODFUQwnT+OlL+OlH+C+NKLx0g84sb68LiI6yj0wbpPFiZC0OSJW5B3HaBgvfRkv/QD3pRGNl35A/friYSgzM6vKycLMzKpysihvbd4BjKLx0pfx0g9wXxrReOkH1KkvPmdhZmZV+cjCzMyqcrIwM7OqJnSykPQOSU9J2i7p+jKPT5b0vfT4Q5IWj32U1dXQjzWSuiQ9km5/lEec1Ui6VdI+SVsrPC5Jt6R+PiZp+VjHWKsa+vJWSQdL9slfjnWMtZJ0qqT7JT0uaZukPy3TpuH3TY39aIr9ImmKpI2SHk19+asybUb3/SsiJuQNaAV+A7wemAQ8Cpw5rM0fA19Ny6uB7+Ud93H2Yw3wD3nHWkNf3gIsB7ZWePxi4B5AwHnAQ3nHfAJ9eStwV95x1tiX+cDytDwDeLrM31jD75sa+9EU+yX9nqen5XbgIeC8YW1G9f1rIh9ZrAS2R8SzEXEU+C6walibVcC6tHw7cIEkjWGMtailH00hIh4A9o/QZBXwzcg8CMyWNH9sont1auhL04iI3RHxcFo+BDwBLBzWrOH3TY39aArp99yT7ran2/CrlUb1/WsiJ4uFwM6S+7s49g/n5TYRUQAOAnPHJLra1dIPgEvT8MDtkk4dm9BGXa19bRZvSsMI90g6K+9gapGGMt5I9km2VFPtmxH6AU2yXyS1SnoE2AfcFxEV98lovH9N5GQxkfw7sDgifhu4j1c+bVh+Hiarw3MO8PfAD3OOpypJ04F/A66LiJfyjud4VelH0+yXiBiMiGXAImClpDfUc3sTOVl0AqWfsBeldWXbSGoDZgHdYxJd7ar2IyK6I6I/3f0acO4YxTbaatlnTSEiXioOI0TE3UC7pHk5h1WRpHayN9h/jYgflGnSFPumWj+abb8ARMQB4H7gHcMeGtX3r4mcLDYBSySdJmkS2Qmg9cParAeuSMuXAT+LdLaogVTtx7Cx40vIxmqb0Xrgg+nKm/OAgxGxO++gjoek1xTHjyWtJPtfbLQPIkB2pRPwdeCJiLipQrOG3ze19KNZ9oukDkmz0/JJwIXAk8Oajer7V9vxPrHZRURB0rXABrIrim6NiG2SPg1sjoj1ZH9Y/yJpO9nJytX5RVxejf34E0mXAAWyfqzJLeARSPoO2dUo8yTtAj5FduKOiPgqcDfZVTfbgV7gynwira6GvlwGfFRSATgCrG7ADyJFbwY+AGxJY+QAfwG8Fppq39TSj2bZL/OBdZJayRLabRFxVz3fv1zuw8zMqprIw1BmZlYjJwszM6vKycLMzKpysjAzs6qcLMzMxoFqxSvLtH9fSVHFb1dr72RhE56kuSVVRvdI6kzLPZL+sQ7bWyrpP9I2npC0Nq1fJuni0d6eTRj/zLFfzCtL0hLgk8CbI+Is4Lpqz5mw37MwK4qIbmAZgKQbgJ6I+HwdN3kLcHNE3Jm2eXZavwxYQfadBbNXJSIeGF6GXNLpwJeBDrLvv3w4Ip4EPgx8OSL+Nz13X7XX95GFWQVpboO70vINktZJ+oWk5yX9gaTPStoi6d5URgJJ50r6uaRfS9pQofLqfLJCewBExJb07ftPA5enI47LJU1LQwsbJf23pFVpG2sk3ZmOTp6R9Km0fpqkH6UieFslXV7v35E1vLXAxyLiXODjQPFI+QzgDEm/lPSgpKpHJD6yMKvd6cDbgDOBXwGXRsSfS7oDeJekH5EVn1sVEV3pzfpG4EPDXudm4GeS/gv4MfCNiDigbKKdFRFxLYCkvyEr0fChVNpho6SfpNdYCbyB7NPiprTt1wEvRMS70vNn1esXYY1PWcHE3wW+r1cqk09OP9uAJWRVBhYBD0g6O9WZKsvJwqx290TEgKQtZKVV7k3rtwCLgaVkb+D3pX/OVuCY+kgR8Q1JG8jGl1cBH5F0TpntXQRcIunj6f4UUmkKspLU3QCSfgCcTzZ89QVJf0c2gc8vTrC/1txagAOpMu1wu8gmqBoAdkh6mix5bBrpxcysNv0AETEEDJTUDBoi++AlYFtELEu3syPionIvFBEvRMStEbGKrGZXufLSIjt6Kb7eayOiWARyeJ2eiIinyWbn2wJ8Rg06JaiNjVR+fYek98LLU98WP5T8kOyoAmVVdc8Anh3p9ZwszEbPU0CHpDdBVg5bZSbPUTZnevEcx2vIJqTpBA6RTfdZtAH4WEkV1DeWPHahpJNTxdF3A7+UtADojYhvAZ8jSxw2QaTilb8ClkraJekq4A+BqyQ9CmzjlVk0NwDdkh4nK2/+ieKRaiUehjIbJRFxVNJlwC3pfEEb8EWyf9JSFwFfktSX7n8iIvZIuh+4PlVE/Vvgr9PzH5PUAuwAfj89ZyPZvAyLgG9FxGZJbwc+J2kIGAA+WrfOWsOJiPdXeOiYk9fpqPjP0q0mrjpr1mQkraHkRLjZWPAwlJmZVeUjCzMzq8pHFmZmVpWThZmZVeVkYWZmVTlZmJlZVU4WZmZW1f8B/PrLgiHNagEAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light", "tags": [] }, "output_type": "display_data" } ], "source": [ "#@title plot learning curves\n", "\n", "# plot results\n", "rl_logger.plot(env.spec._env_name)" ] }, { "cell_type": "markdown", "metadata": { "id": "MigCVgr-Nr0k" }, "source": [ "### Questions\n", "\n", "1. Try learning without the target network. What do you observe?\n", "2. Play with the hyperparameters to see if you can substantially improve the performance of the algorithm. \n", "3. Implement Double DQN; Does it perform better?\n", "4. Check if the Q-network makes correct predictions, i.e. if the predicted expected return matches the observed values.\n", "5. Try out Boltzmann exploration, where instead of an $\\varepsilon$-greedy policy, one can use $\\pi_\\beta\\propto \\exp(Q(s,a)/T)$ for some temperature $T$ which follows some decay schedule. " ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "Notebook_9_Deep_Q_learning.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" }, "latex_metadata": { "affiliation": "Faculty of Physics, Sofia University, 5 James Bourchier Blvd., 1164 Sofia, Bulgaria", "author": "Marin Bukov", "title": "Reinforcement Learning Course: WiSe 2020/21" } }, "nbformat": 4, "nbformat_minor": 1 }