{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning in JAX\n",
    "\n",
    "This notebook provides an introduction to basic Deep Supervised Learning using [JAX](https://jax.readthedocs.io/en/latest/). In particular, we will cover the MNIST classification problem -- one of the first big success stories in ML.\n",
    "\n",
    "Our goal is to become familiar with the typical pipelines of Deep Learning in JAX. This includes:\n",
    "\n",
    "* building models: shallow and deep networks with nonlinear activations, fully-connected and convolutional layers, etc;\n",
    "* defining cost functions and figure-of-merit functions which test the model accuracy;\n",
    "* computing derivatives of model parameters;\n",
    "* understanding the optimizer pipeline and how to update the model parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Problems\n",
    "\n",
    "The \"harmonic oscillator\" of Deep Learning is the MNIST problem.\n",
    "\n",
    "### The MNIST Dataset\n",
    "\n",
    "The MNIST classification problem is one of the classical ML problems for learning classification on high-dimensional data with a fairly sizeable number of examples. Yann LeCun and collaborators collected and processed  70 000  handwritten digits (60 000 are used for training and 10 000 for testing) to produce what became known as one of the most widely used datasets in ML: the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). Each handwritten digit comes in a grayscale square image in the shape of a  28×28  pixel grid. Every pixel takes a value in the range  [0, 255], representing  256  nuances of the gray color from black to white. The problem of image classification finds applications in a wide range of fields and is important for numerous industry applications of ML: there have exists a number of much more challenging datasets, such as [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) or [ImageNet](http://www.image-net.org/).\n",
    "\n",
    "\n",
    "### Data Preprocesssing\n",
    "\n",
    "The first two codeblocks below download the MNIST dataset from the web, and preprocess the MNIST data. In particular, the data are separated into a training set, and a test set, and the labels are encoded in one-hot form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### download MNIST data and store it in under directory _DATA\n",
    "\n",
    "import array\n",
    "import gzip\n",
    "import os\n",
    "from os import path\n",
    "import struct\n",
    "import urllib.request\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# path to data directory\n",
    "_DATA = \"/tmp/jax_example_data/\"\n",
    "\n",
    "\n",
    "def _download(url, filename):\n",
    "    \"\"\"Download a url to a file in the JAX data temp directory.\"\"\"\n",
    "    if not path.exists(_DATA):\n",
    "        os.makedirs(_DATA)\n",
    "    out_file = path.join(_DATA, filename)\n",
    "    if not path.isfile(out_file):\n",
    "        urllib.request.urlretrieve(url, out_file)\n",
    "        print(\"downloaded {} to {}\".format(url, _DATA))\n",
    "\n",
    "\n",
    "def _partial_flatten(x):\n",
    "    \"\"\"Flatten all but the first dimension of an ndarray.\"\"\"\n",
    "    return np.reshape(x, (x.shape[0], -1))\n",
    "\n",
    "\n",
    "def _one_hot(x, k, dtype=np.float32):\n",
    "    \"\"\"Create a one-hot encoding of x of size k.\"\"\"\n",
    "    return np.array(x[:, None] == np.arange(k), dtype)\n",
    "\n",
    "\n",
    "def mnist_raw():\n",
    "    \"\"\"Download and parse the raw MNIST dataset.\"\"\"\n",
    "    # CVDF mirror of http://yann.lecun.com/exdb/mnist/\n",
    "    base_url = \"https://storage.googleapis.com/cvdf-datasets/mnist/\"\n",
    "\n",
    "    def parse_labels(filename):\n",
    "        with gzip.open(filename, \"rb\") as fh:\n",
    "            _ = struct.unpack(\">II\", fh.read(8))\n",
    "            return np.array(array.array(\"B\", fh.read()), dtype=np.uint8)\n",
    "\n",
    "    def parse_images(filename):\n",
    "        with gzip.open(filename, \"rb\") as fh:\n",
    "            _, num_data, rows, cols = struct.unpack(\">IIII\", fh.read(16))\n",
    "            return np.array(array.array(\"B\", fh.read()),\n",
    "                          dtype=np.uint8).reshape(num_data, rows, cols)\n",
    "\n",
    "    for filename in [\"train-images-idx3-ubyte.gz\", \"train-labels-idx1-ubyte.gz\",\n",
    "                   \"t10k-images-idx3-ubyte.gz\", \"t10k-labels-idx1-ubyte.gz\"]:\n",
    "        _download(base_url + filename, filename)\n",
    "\n",
    "    train_images = parse_images(path.join(_DATA, \"train-images-idx3-ubyte.gz\"))\n",
    "    train_labels = parse_labels(path.join(_DATA, \"train-labels-idx1-ubyte.gz\"))\n",
    "    test_images = parse_images(path.join(_DATA, \"t10k-images-idx3-ubyte.gz\"))\n",
    "    test_labels = parse_labels(path.join(_DATA, \"t10k-labels-idx1-ubyte.gz\"))\n",
    "\n",
    "    return train_images, train_labels, test_images, test_labels\n",
    "\n",
    "\n",
    "def mnist(permute_train=False):\n",
    "    \"\"\"Download, parse and process MNIST data to unit scale and one-hot labels.\"\"\"\n",
    "    train_images, train_labels, test_images, test_labels = mnist_raw()\n",
    "\n",
    "    train_images = _partial_flatten(train_images) / np.float32(255.)\n",
    "    test_images = _partial_flatten(test_images) / np.float32(255.)\n",
    "    train_labels = _one_hot(train_labels, 10)\n",
    "    test_labels = _one_hot(test_labels, 10)\n",
    "\n",
    "    if permute_train:\n",
    "        perm = np.random.RandomState(0).permutation(train_images.shape[0])\n",
    "        train_images = train_images[perm]\n",
    "        train_labels = train_labels[perm]\n",
    "\n",
    "    return train_images, train_labels, test_images, test_labels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minibatches\n",
    "\n",
    "Now that we've written the functions to download and preprocess the MNIST dataset, let's prepare it for training. Since we'll be using some variant of SGD (e.g., ADAM, etc.), we need to feed data into our machine learning model (e.g., a DNN) in random minibatches. \n",
    "\n",
    "The function `data_stream` creates a python generator which returns one minibatch of the randomized training set at a time, until all training datapoints are exhausted. To do this efficiently, `data_stream()` defines a python generator. Generators are functions containing loops which `yield` a result one at a time until the loop is exhausted, rather than `return` the output. \n",
    "\n",
    "If you're not familiar with generators, explore carefully the code below first. Make sure to explore the effect of the `while`-loop statement which is currently commented out. What is the purpose of having generators? What are generetors good/useful for? Explore the `data_stream()` generator below by printing a few small minibatches. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object my_generator at 0x7fe594e767d0>\n",
      "\n",
      "first loop:\n",
      "\n",
      "0 0\n",
      "1 1\n",
      "2 2\n",
      "3 3\n",
      "\n",
      "second loop:\n",
      "\n",
      "0 4\n",
      "1 5\n",
      "2 6\n",
      "3 7\n",
      "4 8\n",
      "5 9\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-e0008476830e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def my_generator():\n",
    "    #while True:\n",
    "        for j in range(10):\n",
    "            yield j\n",
    "            \n",
    "gen = my_generator()\n",
    "print(gen) # shows a generator object\n",
    "\n",
    "print('\\nfirst loop:\\n')   \n",
    "\n",
    "# call generator\n",
    "for i in range(4):\n",
    "    print(i, next(gen) )\n",
    "    \n",
    "print('\\nsecond loop:\\n')    \n",
    "    \n",
    "for i in range(10):\n",
    "    print(i, next(gen) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "train data: image shape: (60000, 784), label shape: (60000, 10).\n",
      "test data : image shape: (10000, 784), label shape: (10000, 10).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### define minibatches\n",
    "\n",
    "# fix seed\n",
    "seed=0\n",
    "np.random.seed(seed)\n",
    "\n",
    "##### define data variables and the minibatch generator\n",
    "# load MNIST data\n",
    "train_images, train_labels, test_images, test_labels = mnist()\n",
    "\n",
    "print('\\ntrain data: image shape: {}, label shape: {}.'.format(train_images.shape, train_labels.shape ))\n",
    "print('test data : image shape: {}, label shape: {}.\\n'.format(test_images.shape, test_labels.shape ))\n",
    "\n",
    "\n",
    "# size of a single minibatch\n",
    "batch_size=128 \n",
    "# size of the trining set\n",
    "num_train = train_images.shape[0] \n",
    "# define number of complete minibatches (data size need not be muptiple of batch_size)\n",
    "num_complete_batches, leftover = divmod(num_train, batch_size)\n",
    "# total number of minibatches is the smallest integer to fit all minibatches in the dataset\n",
    "num_batches = num_complete_batches + bool(leftover)\n",
    "\n",
    "\n",
    "def data_stream():\n",
    "    \"\"\"\n",
    "    This function defines a generator which produces random batches of data, one at a time.\n",
    "    \n",
    "    \"\"\"\n",
    "    rng = np.random.RandomState(0)\n",
    "    while True:\n",
    "        perm = rng.permutation(num_train) # compute a random permutation\n",
    "        for i in range(num_batches):\n",
    "            batch_idx = perm[i * batch_size:(i + 1) * batch_size]\n",
    "            yield train_images[batch_idx], train_labels[batch_idx]\n",
    "\n",
    "# define the batches generator\n",
    "batches = data_stream()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the Data\n",
    "\n",
    "The codeblock below plots one of the datapoints. Pay attention to the title of the plot where the corresponding label in one-hot form is given. Make sure you familiarize yourself with the data size: shapes, dimensions, etc. We said earlier that each pixel takes on the values $[0,1,\\dots,255]$, but below it seems like they are squeeze in the interval $[0,1]$: Where in the code did we do that squeezing? Why did we do it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAEICAYAAACZA4KlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAARcElEQVR4nO3dfYwc9X3H8fcHQgwCp8XxhRjjcuFJ1KkKRCuHyk1IBKUOUgVulci0si6qVYNK1OZBCpZRsVs3KSl5qKM2UUxxw0MwiSBWrJa0UJOSRsFpjuD4oRcTxzLE4NjnEIzdOI6Nv/1j59rl2J19mNmH8+/zkla3O9/57Xxv5z43uzO7O4oIzOzkd0q/GzCz3nDYzRLhsJslwmE3S4TDbpYIh90sEQ67WSIGOuySdku6psV5Q9JFHS6n5bFZT0ck3dfJssyakXSNpMOSTrT699+KgQ77APu9iFg8cUPSsKRvSPq5pB+0s4IkTZO0VtLLkn4i6cPtNCLpQ9m4g9n9TGtj7NVZvz/P+j+/jbGrJG2VdFzSyjZ7nqq/c0/Wc0T8e0ScBTzX6v23wmEvxzrgaeCNwG3AQ5KGWhy7ErgYOB94N/BRSQtaGSjpd4FlwNXAMHAB8Jctjp0JfBX4C2AGMAp8ucWeAXYCHwX+pY0xE1YyNX/nvqzn0kTEwF6A3cA12fV5wJPAS8Be4O+B19fMG8CfAbuAA8CdwCk19T8GxoCfAf8GnD9p7EXt9pTdvgQ4CkyvmfafwM0t3t/zwLU1t1cBD7Y49gHg4zW3rwZ+0uLYpcC3a26fCRwBLm1zHd0PrGxzzJT7nfuxnif/rRW9TKUt+yvAh4CZwG9RXcl/OmmehUAFeBtwPdWAI+kGYDnw+8AQ1ZW0rt5CJP2hpC1t9PVWYFdEHKqZ9v1sei5JZwPnZvO3NbZm2ZPHniPpje2OjYj/AX7UxrI7MoV/536u51JMmbBHxFMRsSkijkfEbuALwFWTZvtERLwYEc8BfwfcmE2/CfibiBiLiOPAx4HL671ei4gHIuI322jtLODgpGkHgektjp2Yv92x9ZY9cb3VZXfadxFT9Xfu53ouxZQJu6RLJP1ztnPjZaqBnTlpth/XXH+W6n9TqL5OWi3pJUkvAS8CAmaX0Nph4A2Tpr0BOFRn3npjJ+Zvd2y9ZU9cb3XZnfZdxFT9nfu5nksxZcIOfB74AXBxRLyB6tNyTZpnTs31XwNeyK7/GLgpIn615nJGRHy7hL62AxdIqv0vfVk2PVdE/Izq/ofL2h1bs+zJY/dFxE/bHSvpTODCNpbdkSn8O/dzPZejrBf/3bjw6h10/wXcTjXglwI7gG/VzBvARuBsqqH/AbA0qy0EtgFvzW7/CvDeSWM72kGXTdsEfBI4PVvWS8BQi/d3B/BE1velVP8oFrQ4dgHwE2BuNv5x4I4Wxw5RfSr5B1nfnwA2tbFuTsvGPQD8dXb91JP8d+7peq73t1YoT2XdUTcuk8L+zizAh6nuYPurOmGf2Bv/U+BTtX98wGJgK/Ay1S392kljL8qu/xGwvZ0VQPUQ0H9Q3bO7g1fvrX8HcDjn/qYBa7O+9gEfnlQ/DLwjZ/yHs3EvA/8ETKupfR1YnjP2muwxPZL1P1xTWw58PWfsF7PHrfby/pP8d+7peq73t1bkouxOrUWSdgCzgPURMdLvfuzkI+lq4GGq/yCui4hvlHK/DrtZGqbSDjozK8BhN0vE63q5sJkzZ8bw8HAvF2mWlN27d3PgwIHJh6SBgmHP3si/GjgV+MeIuCNv/uHhYUZHR4ss0sxyVCqVhrWOn8ZLOhX4B+A9VI953ihpbqf3Z2bdVeQ1+zxgZ0TsiohfAg9S/fCJmQ2gImGfzavfi76HOu81l7RU0qik0fHx8QKLM7MiioS93k6A1xy0j4g1EVGJiMrQUKuf8zezshUJ+x5e/cGT8/j/D56Y2YApEvbvAhdLeouk1wOLgA3ltGVmZev40FtEHJf0Aapf8XQq1Q+W9PYje2bWskLH2SPiEeCRknoxsy7y22XNEuGwmyXCYTdLhMNulgiH3SwRDrtZIhx2s0Q47GaJcNjNEuGwmyXCYTdLhMNulgiH3SwRDrtZIhx2s0Q47GaJcNjNEuGwmyXCYTdLhMNulgiH3SwRDrtZIhx2s0Q47GaJcNjNEuGwmyXCYTdLhMNulgiH3SwRDrtZIgqdslnSbuAQ8ApwPCIqZTRlZuUrFPbMuyPiQAn3Y2Zd5KfxZokoGvYAHpX0lKSl9WaQtFTSqKTR8fHxgoszs04VDfv8iHgb8B7gFknvnDxDRKyJiEpEVIaGhgouzsw6VSjsEfFC9nM/sB6YV0ZTZla+jsMu6UxJ0yeuA9cC28pqzMzKVWRv/DnAekkT9/NARPxrKV3Zqxw7diy3ftdddzWsLVmyJHfstGnTOuppEKxbty63fttttzWs7dq1q+x2Bl7HYY+IXcBlJfZiZl3kQ29miXDYzRLhsJslwmE3S4TDbpaIMj4IY122bNmy3Prq1asb1mbMmJE7dtGiRR31NAj27NmTW88OC1vGW3azRDjsZolw2M0S4bCbJcJhN0uEw26WCIfdLBE+zj4FjI2N9buFgeTj6O3xlt0sEQ67WSIcdrNEOOxmiXDYzRLhsJslwmE3S4SPs/fA0aNHc+uf/exnc+tPPPFEme0MjEOHDuXW77zzztz6jh07ymznpOctu1kiHHazRDjsZolw2M0S4bCbJcJhN0uEw26WCB9n74GDBw/m1pt9L3wzV111VcPaIH8v/GOPPZZb/9jHPlbo/kdGRgqNP9k03bJLWitpv6RtNdNmSHpM0g+zn2d3t00zK6qVp/FfBBZMmrYM2BgRFwMbs9tmNsCahj0ivgm8OGny9cA92fV7gBvKbcvMytbpDrpzImIvQPbzTY1mlLRU0qik0fHx8Q4XZ2ZFdX1vfESsiYhKRFSGhoa6vTgza6DTsO+TNAsg+7m/vJbMrBs6DfsGYOK4xgjwtXLaMbNuaXqcXdI64F3ATEl7gBXAHcBXJC0BngPe280mp7pVq1bl1ot+//n8+fMLje+X559/Prde9HE577zzCo0/2TQNe0Tc2KB0dcm9mFkX+e2yZolw2M0S4bCbJcJhN0uEw26WCH/EtQc+97nP5dabHWIaHh7OrS9fvrzdlgZCt09FPXfu3K7e/1TjLbtZIhx2s0Q47GaJcNjNEuGwmyXCYTdLhMNulggfZ58Cbr/99tz6GWec0aNO2vf00083rN1///097MS8ZTdLhMNulgiH3SwRDrtZIhx2s0Q47GaJcNjNEuHj7CU4evRobv3EiRO59WanLs47JTPAL37xi4a1008/PXfskSNHcuvHjh3LrT/zzDO59Xnz5jWsFf2q6Dlz5uTWp+pXbHeLt+xmiXDYzRLhsJslwmE3S4TDbpYIh90sEQ67WSJ8nL1FBw4caFi79dZbc8eeckr+/9TFixfn1q+44orc+t69exvWrrzyytyxjz76aG792Wefza03k3csvdlx9gULFuTW169fn1s/7bTTcuupabpll7RW0n5J22qmrZT0vKTN2eW67rZpZkW18jT+i0C9f7GfiYjLs8sj5bZlZmVrGvaI+CbwYg96MbMuKrKD7gOStmRP889uNJOkpZJGJY2Oj48XWJyZFdFp2D8PXAhcDuwFPtVoxohYExGViKgMDQ11uDgzK6qjsEfEvoh4JSJOAHcBjT/aZGYDoaOwS5pVc3MhsK3RvGY2GJoeZ5e0DngXMFPSHmAF8C5JlwMB7AZu6l6Lg2HmzJkNa6tXr84d2+wz5XnH8AEefvjh3HqeLVu25NaLfqa8iBUrVuTWFy1alFv3cfT2NA17RNxYZ/LdXejFzLrIb5c1S4TDbpYIh90sEQ67WSIcdrNEKCJ6trBKpRKjo6M9W95Ucfz48dz6zp07c+sbNmxoWGu2fpsdehsbG8ut33fffbn16dOnN6xt3749d+y5556bW7fXqlQqjI6O1l2p3rKbJcJhN0uEw26WCIfdLBEOu1kiHHazRDjsZonwV0kPgNe9Ln81XHrppYXqRTz55JO59XvvvTe3PjIy0rDm4+i95S27WSIcdrNEOOxmiXDYzRLhsJslwmE3S4TDbpYIH2e3XA899FBuvdnn4WfPnl1mO1aAt+xmiXDYzRLhsJslwmE3S4TDbpYIh90sEQ67WSJaOWXzHOBe4M3ACWBNRKyWNAP4MjBM9bTN74uIn3WvVeuGI0eO5NYff/zxQvfv8wQMjla27MeBj0TErwNXArdImgssAzZGxMXAxuy2mQ2opmGPiL0R8b3s+iFgDJgNXA/ck812D3BDl3o0sxK09Zpd0jBwBfAd4JyI2AvVfwjAm0rvzsxK03LYJZ0FPAx8MCJebmPcUkmjkkbHx8c76dHMStBS2CWdRjXoX4qIr2aT90maldVnAfvrjY2INRFRiYjK0NBQGT2bWQeahl3VjzXdDYxFxKdrShuAia8OHQG+Vn57ZlaWVj7iOh9YDGyVtDmbthy4A/iKpCXAc8B7u9KhddWKFSty61u3bi10/6tWrSo03srTNOwR8S2g0YeWry63HTPrFr+DziwRDrtZIhx2s0Q47GaJcNjNEuGwmyXCXyWduEOHDuXWIyK3vnDhwtz6JZdc0nZP1h3espslwmE3S4TDbpYIh90sEQ67WSIcdrNEOOxmifBx9sQ1O+Vys/q8efPKbMe6yFt2s0Q47GaJcNjNEuGwmyXCYTdLhMNulgiH3SwRDrtZIhx2s0Q47GaJcNjNEuGwmyXCYTdLhMNulgiH3SwRTT/PLmkOcC/wZuAEsCYiVktaCfwJMJ7NujwiHulWo9Ydb3/723PrmzZtyq3ffPPNZbZjXdTKl1ccBz4SEd+TNB14StJjWe0zEfHJ7rVnZmVpGvaI2Avsza4fkjQGzO52Y2ZWrrZes0saBq4AvpNN+oCkLZLWSjq7wZilkkYljY6Pj9ebxcx6oOWwSzoLeBj4YES8DHweuBC4nOqW/1P1xkXEmoioRERlaGioeMdm1pGWwi7pNKpB/1JEfBUgIvZFxCsRcQK4C/A3D5oNsKZhV/XrRe8GxiLi0zXTZ9XMthDYVn57ZlaWVvbGzwcWA1slbc6mLQdulHQ5EMBu4KYu9GddNjIyUqhuU0cre+O/BdT78nAfUzebQvwOOrNEOOxmiXDYzRLhsJslwmE3S4TDbpYIh90sEQ67WSIcdrNEOOxmiXDYzRLhsJslwmE3S4TDbpYIRUTvFiaNA8/WTJoJHOhZA+0Z1N4GtS9wb50qs7fzI6Lu97/1NOyvWbg0GhGVvjWQY1B7G9S+wL11qle9+Wm8WSIcdrNE9Dvsa/q8/DyD2tug9gXurVM96a2vr9nNrHf6vWU3sx5x2M0S0ZewS1ogaYeknZKW9aOHRiTtlrRV0mZJo33uZa2k/ZK21UybIekxST/MftY9x16felsp6fnssdss6bo+9TZH0jckjUnaLunPs+l9fexy+urJ49bz1+ySTgWeAX4H2AN8F7gxIv67p400IGk3UImIvr8BQ9I7gcPAvRHxG9m0vwVejIg7sn+UZ0fErQPS20rgcL9P452drWhW7WnGgRuA99PHxy6nr/fRg8etH1v2ecDOiNgVEb8EHgSu70MfAy8ivgm8OGny9cA92fV7qP6x9FyD3gZCROyNiO9l1w8BE6cZ7+tjl9NXT/Qj7LOBH9fc3sNgne89gEclPSVpab+bqeOciNgL1T8e4E197meypqfx7qVJpxkfmMeuk9OfF9WPsNc7ldQgHf+bHxFvA94D3JI9XbXWtHQa716pc5rxgdDp6c+L6kfY9wBzam6fB7zQhz7qiogXsp/7gfUM3qmo902cQTf7ub/P/fyfQTqNd73TjDMAj10/T3/ej7B/F7hY0lskvR5YBGzoQx+vIenMbMcJks4ErmXwTkW9AZg4teoI8LU+9vIqg3Ia70anGafPj13fT38eET2/ANdR3SP/I+C2fvTQoK8LgO9nl+397g1YR/Vp3TGqz4iWAG8ENgI/zH7OGKDe7gO2AluoBmtWn3r7baovDbcAm7PLdf1+7HL66snj5rfLmiXC76AzS4TDbpYIh90sEQ67WSIcdrNEOOxmiXDYzRLxv29bfwpe9en4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "# static plots\n",
    "%matplotlib inline \n",
    "\n",
    "### show the first data point as an example\n",
    "n=1111 # test data point number\n",
    "\n",
    "plt.imshow(test_images[n].reshape(28,28),cmap='Greys')\n",
    "plt.title('label: {}'.format(test_labels[n]) )\n",
    "\n",
    "plt.show()\n",
    "\n",
    "# print the array which generates the image above and expore it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SoftMax Regression\n",
    "\n",
    "Our goal is to train a ML model that classifies the images in the test set, by only using the data in the training set. We will explore shallow and deep networks, as well as fully-connected and convolutional layers.\n",
    "\n",
    "Instead of learning directly the catecory an imamge falls into, learning is more stable if we model the probability to be in a certain category $i$. To do this, we will use a SoftMax output nonlinearity activaion, which can be thought of as a statistical model that assigns the probability that a given input image corresponds to any of the 10 handwritten digits. This layer is a multi-categorical generalization of the logistic regression and reads as:\n",
    "\n",
    "$$ \n",
    "p\\left(y=i|x,\\{\\boldsymbol{\\theta}_k\\}_{k=0}^9 \\right) = \\frac{\\exp(x^T\\cdot \\boldsymbol{\\theta}_i)}{\\sum_{j=0}^9 \\exp(x^T\\cdot \\boldsymbol{\\theta}_j) }\n",
    "$$\n",
    "\n",
    "Where  $p\\left(y=i|x,\\{\\boldsymbol{\\theta}_k\\}_{k=0}^9 \\right)$  is the probability that input $x$  is the  $i$-th digit,  $i\\in\\{0,1,2,\\dots,9\\}$. The model parameters (sometimes called trainables or learnables) are denoted by $\\boldsymbol{\\theta}$: in the simplest model, there's one $\\theta_j$ per category. One can use this model for prediction by taking that value of $y$ for which the probability is maximized:\n",
    "\n",
    "$$\n",
    "y_\\mathrm{pred} = \\arg\\max_{j} p\\left(y=j|x,\\{\\boldsymbol{\\theta}_k\\}_{k=0}^9 \\right).\n",
    "$$ \n",
    "\n",
    "In practice, it is often more convenient to use the logarithm of the softmax function to learn probabilities. Using a log scale allows the model to more easily capture probabilities which differ by several orders of magnitude. Often, there is a special function for this in ML packages, with improved performance. \n",
    "\n",
    "\n",
    "### Three ML Models\n",
    "\n",
    "Below, you have to use the JAX library to build three different models:\n",
    "\n",
    "A. SoftMax Logistic Regression: no correlations between pixels, no info about image dimensionality\n",
    "\n",
    "B. DNN: a fully-connected (fc) deep neural network: no info about image dimensionality\n",
    "\n",
    "C. CNN: a convolutional (conv) neural network with a fully-connected head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SoftMax Logistic Regression\n",
    "\n",
    "We begin with the SoftMax Logistic Regression. The code structure will later allow us to easily generalize and re-use the functions we write to the DNN and CNN models. We will proceed in the following steps:\n",
    "\n",
    "1. Define the ML model. \n",
    "2. Define the `loss` function, and a function which measures the `accuracy` of the model predictions. \n",
    "3. Define generalized gradient descent optimizer\n",
    "4. Define the training loop and train the model\n",
    "\n",
    "Fill-in the code snippet below. To do this, you will have to explore and read the JAX documentation:\n",
    "* start with [`jax.numpy`](https://jax.readthedocs.io/en/latest/jax.numpy.html) and make yourself familiar with the functions in there. Why is there a need to a JAX version of numpy, i.e. why not use the ordinary numpy library? You may als check out `jax.scipy`.\n",
    "* look up [`jax.random`](https://jax.readthedocs.io/en/latest/jax.random.html): random numbers work a bit differently in JAX, compared to numpy, but this is not hat important for understanding how to do deep learning in JAX.\n",
    "* neural network architectures, like the fc and conv layers we will need, are described in [`jax.experimental.stax`](https://jax.readthedocs.io/en/latest/jax.experimental.stax.html); make sure you understand the difference between required/compusory and optional arguments. The [`jax.nn`](https://jax.readthedocs.io/en/latest/jax.nn.html) package contains the nonlinear activation functions. To apply the activations elementwise, use either the [`elementwise` function](https://jax.readthedocs.io/en/latest/_modules/jax/experimental/stax.html#elementwise), or capital-letter activations instead.   \n",
    "\n",
    "\n",
    "#### Define the ML model\n",
    "\n",
    "We will first construct the SoftMax model. To do so, we think of it as a single layer NN. To construct the layer, use the `init_fun, apply_fun = stax.serial()` function. This function returns two other functions: `init_fun, apply_fun` (you may give them whatever names you want). \n",
    "* `output_shape, params = init_fun(rng, input_shape)` is used to initialize the parameters of the model: it returns the shape of the output at the topmost layer, and a tuple of nested tuples `params` which contains the model parameters. \n",
    "* `predictions = predict(params, data)` makes use of the model as defined by `params` and the `data` to apply the model on data and produce the model output. \n",
    "\n",
    "Use a subset of the training data which contains, say three, data points. We will use it to debug and explore the model we have defined. \n",
    "\n",
    "1.0. build a model consisting of a single `Dense` layer, followed by the `LogSoftMax` activation. \n",
    "\n",
    "1.1. compute the model predictions on that toy subset\n",
    "\n",
    "1.2. check the shape of the output. Does it agree with the `output_shape` tuple returned by `init_fun`?\n",
    "\n",
    "1.3. print out the prediction values themselves: how many values does each of the three toyset datapoints have? Do these values represent a well-defined probability distribution? -- check the conservation of probability. \n",
    "\n",
    "1.4. print the `params` variable and explore it carefully. It defines a so-called JAX tree. Manipulating this ordered list of nested lists can be very annoying if you want to do that from scratch. Instead, explore and use the functions of the [`jax.tree_utils`](https://jax.readthedocs.io/en/latest/jax.tree_util.html) package. Understanding this is especially important if you want to access the parameters of a specific layer in the model. Extract the weights and biases from the single layer model, and check their shape/sizes of the parameters; waht is their data type `type(variable)`; why is this new datatype needed and how can you transfer data from an ordinary numpy datatype to such a datatype back and forth?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "output shape of the model is (-1, 10).\n",
      "\n",
      "actual output shape is: (3, 10)\n",
      "log(softmax) values: [-2.2292223 -2.4820766 -2.7947464 -2.1780825 -1.9830028 -2.2069645\n",
      " -1.9345262 -2.6357243 -2.065387  -3.1278682]\n",
      "conservation of probability [0.9999999 1.0000001 1.0000001]\n"
     ]
    }
   ],
   "source": [
    "import jax.numpy as jnp # jax's numpy version with GPU support\n",
    "from jax import random # used to define a RNG key to control the random input in JAX\n",
    "from jax.experimental import stax # neural network library\n",
    "from jax.experimental.stax import Dense, Relu, LogSoftmax # neural network layers\n",
    "\n",
    "# set key for the RNG (see JAX docs)\n",
    "rng = random.PRNGKey(seed)\n",
    "\n",
    "# cast data into 1D image format suitable for fc layers: the shape should be (N_datapoints, 28*28)\n",
    "train_images = train_images.reshape(-1,28*28) # -1: number of data points, (28*28): (height*width) dimensions of image\n",
    "test_images = test_images.reshape(-1,28*28)\n",
    "\n",
    "# define functions which initialize the parameters and evaluate the model\n",
    "initialize_params, predict = stax.serial(\n",
    "                                            ### SoftMax Regression\n",
    "                                            Dense(10), # 10 output neurons\n",
    "                                            LogSoftmax # NB: computes the log-probability\n",
    "    \n",
    "#                                             ### fully connected DNN\n",
    "#                                             Dense(512), # 512 hidden neurons\n",
    "#                                             Relu,\n",
    "#                                             Dense(256), # 256 hidden neurons\n",
    "#                                             Relu,\n",
    "#                                             Dense(10), # 10 output neurons\n",
    "#                                             LogSoftmax # NB: computes the log-probability\n",
    "                                        )\n",
    "\n",
    "# initialize the model parameters\n",
    "output_shape, inital_params = initialize_params(rng, (-1, 28 * 28)) # fcc layer 28x28 pixes in each image\n",
    "\n",
    "print('\\noutput shape of the model is {}.\\n'.format(output_shape))\n",
    "\n",
    "# check how network works on 3 examples\n",
    "predictions = predict(inital_params, test_images[0:3])\n",
    "\n",
    "# print shape of output\n",
    "print(\"actual output shape is:\", predictions.shape)\n",
    "\n",
    "# check if probability is conserved\n",
    "print('log(softmax) values:', predictions[0])\n",
    "print('conservation of probability', np.sum(jnp.exp(predictions), axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the loss/cost function\n",
    "\n",
    "Next, we define the loss/cost function and the accuracy function to measure the performance of the model. Defining these functions in JAX works the same way as in ordinary python. The only difference is that one has to use `jax.numpy` instead of ordinary `numpy` (for ordinary `numpy` is not optimized for GPUs). \n",
    "\n",
    "2.0. Complete the `loss(params, batch)` function which computes the cross entropy, given the model `params` and the data of a minibatch `batch`. Using functions like `jnp.mean` or `jnp.sum` can make the `loss` agnostic to the size of the `batch`. \n",
    "\n",
    "2.1. Complete the `mean_accuracy(params, batch)` function. It computes the mean number of datapoints which produce a correct preduction.\n",
    "\n",
    "2.2. Test the `loss` and `mean_accuracy` functions on the toy dataset. This helps spotting errors and debugging them.\n",
    "\n",
    "2.3. (optional): use the `tree_flatten` function from the `jax.tree_util` package and add an `L2` regularizer to the `loss` function. To do so, define another function `l2_regularizer(params, lmbda)` which computes the L2 norm of the model parameters and weighs it by the regulazation strength `lmbda`. Is there a JAX function that computes the L2 norm?\n",
    "\n",
    "2.4. explore the [`jax.grad`](https://jax.readthedocs.io/en/latest/jax.html#jax.grad) function. Test it on the loss function by computing the values of the gradient of the model parameters for the points at the toyset. Check the form of the gradient output. Check the shapes/dimensions of the gradients themselves. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "### define loss and accuracy functions\n",
    "\n",
    "from jax import grad\n",
    "from jax.tree_util import tree_flatten # jax params are stored as nested tuples; use this to manipulate tuples\n",
    "\n",
    "\n",
    "def l2_regularizer(params, lmbda):\n",
    "    \"\"\"\n",
    "    Define l2 regularizer: $\\lambda \\ sum_j ||theta_j||^2 $ for every parameter in the model $\\theta_j$\n",
    "    \n",
    "    \"\"\"\n",
    "    return lmbda*jnp.sum(jnp.array([jnp.sum(jnp.abs(theta)**2) for theta in tree_flatten(params)[0] ]))\n",
    "\n",
    "\n",
    "def loss(params, batch):\n",
    "    \"\"\"\n",
    "    Define cost (or lost) function for softmax classification. \n",
    "    \n",
    "    \"\"\"\n",
    "    inputs, targets = batch\n",
    "    preds = predict(params, inputs)\n",
    "    return -jnp.mean(jnp.sum(preds * targets, axis=1)) + l2_regularizer(params, 0.001)\n",
    "\n",
    "\n",
    "def mean_accuracy(params, batch):\n",
    "    \"\"\"\n",
    "    Define accuracy function: the mean number of datapoints which have correct preductions. \n",
    "    This function is not used for training; only to test the performance. \n",
    "    \n",
    "    \"\"\"\n",
    "    inputs, targets = batch\n",
    "    target_class = jnp.argmax(targets, axis=1)\n",
    "    predicted_class = jnp.argmax(predict(params, inputs), axis=1)\n",
    "    return jnp.mean(predicted_class == target_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the optimizer\n",
    "\n",
    "Next, we define the optimizer. Make sure to read the documentation of [`jax.experimental.optimizers`](https://jax.readthedocs.io/en/latest/jax.experimental.optimizers.html). In particular, you may want to check out the source code for \n",
    "[SGD](https://jax.readthedocs.io/en/latest/_modules/jax/experimental/optimizers.html#sgd), \n",
    "[SGD with momentum](https://jax.readthedocs.io/en/latest/_modules/jax/experimental/optimizers.html#momentum), and \n",
    "[ADAM](https://jax.readthedocs.io/en/latest/_modules/jax/experimental/optimizers.html#adam).\n",
    "\n",
    "3.0. define the optimizer hyperparameters (step size/learning rate, etc.)\n",
    "\n",
    "3.1. call the `optimizers.momentum` constructor and obtain the `opt_init, opt_update, get_params` functions. Makes sure you read the documentation to understand what they do, and how they are used.\n",
    "\n",
    "3.2. Complete the `update(i, opt_state, batch)` function. The only nontrivial step is the the computation of the gradient of the loss function, which we explored in part 2.4. above. Which line does the actual update of the model parameters (e.g. the SGD step) take place in?\n",
    "\n",
    "3.3. add a `@jit` decorator (Just-In-Time compiler) to the update function. This will make jax compile the `update()` function to give you speed (even on the CPU!). Explore the documentation for [`jax.jit`](https://jax.readthedocs.io/en/latest/jax.html#jax.jit). One caveat is that any functions (and subroutines) used under the `jit` decorator must be using `jax.numpy` or `jax.scipy`; using ordinary `numpy` _and_ `jit` will throw an error (thy that out by modifying the `loss` function!). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "### define generalized gradient descent optimizer and a function to update model parameters\n",
    "\n",
    "from jax.experimental import optimizers # gradient descent optimizers\n",
    "from jax import jit\n",
    "\n",
    "step_size = 0.001 # step size or learning rate \n",
    "momentum_mass = 0.9 # \"gamma\" parameter in GD+momentum\n",
    "\n",
    "# compute optimizer functions\n",
    "opt_init, opt_update, get_params = optimizers.momentum(step_size, mass=momentum_mass)\n",
    "\n",
    "\n",
    "# define function which updates the parameters using the change computed by the optimizer\n",
    "@jit # Just In Time compilation speeds up the code; requires to use jnp everywhere; remove when debugging\n",
    "def update(i, opt_state, batch):\n",
    "    \"\"\"\n",
    "    i: int,\n",
    "        counter to count how many update steps we have performed\n",
    "    opt_state: object,\n",
    "        the state of the optimizer\n",
    "    batch: np.array\n",
    "        batch containing the data used to update the model\n",
    "        \n",
    "    Returns: \n",
    "    opt_state: object,\n",
    "        the new state of the optimizer\n",
    "        \n",
    "    \"\"\"\n",
    "    # get current parameters of the model\n",
    "    current_params = get_params(opt_state)\n",
    "    # compute gradients\n",
    "    grad_params = grad(loss)(current_params, batch)\n",
    "    # use the optimizer to perform the update using opt_update\n",
    "    return opt_update(i, grad_params, opt_state)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model\n",
    "\n",
    "At last, we have built all ingredients and we can start training our model. We train the model in epochs. In every epoch, we loop over the number of minibatches to exhaust the training set. We update the model parameters for each minibatch (hence the number of epochs is not the same of the number of updates). Therefore, we use a variable `itercount` to count the number of updates; in fact, `itercount` will be a trivial generator similar to the ones discussed above. Once we've done the update, we can read off the model parameters and check the current loss and model accuracy ***on the test set*** for the given epoch. \n",
    "\n",
    "Then we move to the second epoch and repeat the procedure. The model learns if the loss on the test data goes down, and the accuracy on the test data goes up. We can monitor these quantities durin training.\n",
    "\n",
    "4.0. define placeholders for the `train_accuracy` and `test_accuracy`. \n",
    "\n",
    "4.1. initialize the optimizer state using the `opt_init` function. \n",
    "\n",
    "4.2. loop over the epochs. \n",
    "\n",
    "4.2.1. For each epoch, loop over all minibatches and use the `update()` function to compute the gradients of the `params` and update the model. Updating the model happens automatically upon calling `update()`. How does `update` know about the current value of `params`? Check if `params` is changing after each call of `update`. \n",
    "\n",
    "4.2.2. Compute the mean accuracy of the test and traing data, and store it in `train_accuracy` and `test_accuracy`. Print these values for reference.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Starting training...\n",
      "\n",
      "Epoch 0 in 0.52 sec\n",
      "Training set accuracy 0.8258500099182129\n",
      "Test set accuracy 0.8352000117301941\n",
      "\n",
      "Epoch 10 in 0.10 sec\n",
      "Training set accuracy 0.8929499983787537\n",
      "Test set accuracy 0.9003000259399414\n",
      "\n",
      "Epoch 20 in 0.12 sec\n",
      "Training set accuracy 0.9025333523750305\n",
      "Test set accuracy 0.9090999960899353\n",
      "\n",
      "Epoch 30 in 0.16 sec\n",
      "Training set accuracy 0.9068499803543091\n",
      "Test set accuracy 0.9121000170707703\n",
      "\n",
      "Epoch 40 in 0.13 sec\n",
      "Training set accuracy 0.9095333218574524\n",
      "Test set accuracy 0.9143999814987183\n",
      "\n",
      "Epoch 50 in 0.14 sec\n",
      "Training set accuracy 0.911466658115387\n",
      "Test set accuracy 0.9151999950408936\n",
      "\n",
      "Epoch 60 in 0.10 sec\n",
      "Training set accuracy 0.912933349609375\n",
      "Test set accuracy 0.9157999753952026\n",
      "\n",
      "Epoch 70 in 0.10 sec\n",
      "Training set accuracy 0.9145166873931885\n",
      "Test set accuracy 0.916100025177002\n",
      "\n",
      "Epoch 80 in 0.13 sec\n",
      "Training set accuracy 0.914900004863739\n",
      "Test set accuracy 0.9158999919891357\n",
      "\n",
      "Epoch 90 in 0.10 sec\n",
      "Training set accuracy 0.9155666828155518\n",
      "Test set accuracy 0.916700005531311\n",
      "\n",
      "Epoch 100 in 0.13 sec\n",
      "Training set accuracy 0.9156666398048401\n",
      "Test set accuracy 0.9176999926567078\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### Train model\n",
    "\n",
    "import time\n",
    "import itertools\n",
    "\n",
    "# define geenrator to count the number of updates\n",
    "itercount = itertools.count()\n",
    "\n",
    "# define number of training epochs\n",
    "num_epochs = 101\n",
    "\n",
    "# define figures of merit\n",
    "train_accuracy=np.zeros(num_epochs)\n",
    "test_accuracy=np.zeros_like(train_accuracy)\n",
    "\n",
    "print(\"\\nStarting training...\\n\")\n",
    "\n",
    "# set the initial model parameters in the optimizer\n",
    "opt_state = opt_init(inital_params)\n",
    "\n",
    "# loop over the number of training epochs\n",
    "for epoch in range(num_epochs): \n",
    "    \n",
    "    ### record time\n",
    "    start_time = time.time()\n",
    "    \n",
    "    ### train in minibatches until the entire dataset is exhausted: \n",
    "    # the entire dataset is divided into _random_ minibatches; \n",
    "    # all minibatches are shown to the model before going to next epoch\n",
    "    for _ in range(num_batches):\n",
    "        # use the data to update the model parameters\n",
    "        opt_state = update(next(itercount), opt_state, next(batches))\n",
    "        \n",
    "    ### record time needed for a single epoch\n",
    "    epoch_time = time.time() - start_time\n",
    "    \n",
    "    ### evaluate performance of the model at each fixed epoch\n",
    "    \n",
    "    # retrieve current model parameters\n",
    "    params = get_params(opt_state)\n",
    "    \n",
    "    # measure the accuracy on the training and test datasets\n",
    "    train_accuracy[epoch] = mean_accuracy(params, (train_images, train_labels))\n",
    "    test_accuracy[epoch] = mean_accuracy(params, (test_images, test_labels))\n",
    "    \n",
    "    # print results every 10 epochs\n",
    "    if epoch % 10 == 0:\n",
    "        print(\"Epoch {} in {:0.2f} sec\".format(epoch, epoch_time))\n",
    "        print(\"Training set accuracy {}\".format(train_accuracy[epoch]))\n",
    "        print(\"Test set accuracy {}\\n\".format(test_accuracy[epoch]))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explore the training properties and behavior\n",
    "\n",
    "Below, we compare the training and test average accuracy curves. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1KUlEQVR4nO3dd3iUVdr48e9N6BBaqAYkoShWQIoUcbECK4qyKOqiru+uvO5PVHatqFjXXtaGIvZdeXVtKCgioIwuomAIQaRo6IRuaAkkpJ3fH/ckmSSTMAkzmWTm/lzXXGSeOc8z5wQ493PqI845jDHGmNLqhDsDxhhjaiYLEMYYY/yyAGGMMcYvCxDGGGP8sgBhjDHGr7rhzkAwtW7d2iUkJFTp3IMHD9KkSZPgZqiGszJHvmgrL1iZK2vp0qW/Oefa+PssogJEQkICSUlJVTrX4/EwdOjQ4GaohrMyR75oKy9YmStLRDaV91lIu5hEZLiI/CIia0XkTj+ftxSRGSLyk4gsEZGTvcc7icgCEVktIitF5OZQ5tMYY0xZIQsQIhIDTAFGACcCV4jIiaWS3QWkOOdOBa4GnvMezwNucc6dAAwAbvBzrjHGmBAKZQuiP7DWObfeOZcDvAeMKpXmROArAOfcGiBBRNo557Y755K9xzOA1UB8CPNqjDGmlFCOQcQDW3zepwGnl0qzHBgNLBSR/kBnoCOwszCBiCQAvYHF/r5ERMYD4wHatWuHx+OpUmYzMzOrfG5tZWWOfNFWXrAyB1MoA4T4OVZ646fHgOdEJAVYASxDu5f0AiJNgY+Aic65A/6+xDk3DZgG0LdvX1fVgRob2IoO0VbmaCsvWJmDKZQBIg3o5PO+I7DNN4G30r8WQEQE2OB9ISL10OAw3Tn3cQjzaYwxxo9QjkH8CHQXkUQRqQ9cDsz0TSAiLbyfAfwF+NY5d8AbLF4HVjvnnglhHo0xxpQjZAHCOZcHTAC+RAeZ33fOrRSR60Xkem+yE4CVIrIGne1UOJ11MHAVcLaIpHhfvw9VXo0xplbavRumT6fTu++G5PIhXSjnnJsNzC51bKrPz98D3f2ctxD/YxjGGFOzOQepqdCyJbRpU3xsyRL44AOoVw/699dXfAWTM3NyYONGOHQIWrWCuDjYuxcWL9ZrLVgASUngHPFt2sCUKVA3uFV6RK2kNsYEKD1dKx0p5z7MOfB4oE8faNYsNHlYvhwyM+H004srtl27tBJdvhz27NFXenrxzy1awPjx+urQAbKzISUFDh+GM88sWZ6lS2HaNPj1V1i7FtLS/Oejbl3o2VMr7BNOgNWrtQJesQIaNy6unE85RfPasyds2KBpUlI0T927w7HH6vs5c4q/KzERTjtNj69bB/XrQ0EB5Hnn4jRqpNdv1Up/Bv3d//YbbNqkaf2pXx/69oUHHoDhw/nhwAGGBjk4gAUIY2o25yA/P7h3hk8+CbffrpXhH/8IV16pFZnvd952Gzz9tN7hvvQSXHRRyWts3Qpffql3sfHxMHw4DBqkFZdzetfbsCHExJT9/h9+0Iptzhx937w5nHuunjN3rpa3bVu9+27VChISNFC1agUrV8L998PDDxdX5rm5ep0zz4RnnqFOVhbccgs8+yw0bQonnwxnnw2dOvn/PR48CMuWwfTpcOCABsR+/eDGG/Xa6emwcyd88gm88UbxeQ0bwqmnwo4d8PnnesffrBmcdx7cc49ea8kSDVRdu+qxSy6BBg00YCxZAlu2FAfBw4eLr92tG4wbp4GncWNtOezZoz/3769BqkGD4vQhmtZrAcKY6pSerneFbfzujVZSdrZWEvPnw803w8SJ2m0RiMxMWLqUtvPmaWVSeN6jj8Jdd8GwYVox3nOPvq66Ch55RCv7W26Bf/5Tv3v5chg1Siu2rl31TnzNGn2BVuR79sDjj2tl3LSpvs/J0WDRpYtWdvXq6fFdu7RSj4vTvHTvroFmzhyoUwduvVWD1imnlF+21FQNWqtWwe9/rxXmjh1w333Qrx8DWrTQCvV//xcee0zv8ANRUADbt2vLpI6f4VnnYP16+OknDVonn6zlAg1qO3bo76PwmB/79kFMLsQOGAADBpT5PC9P4+fGjforzDkITYDEHhrD4+L0q/IPQ95Bb5oc2LGjYWBlrCSJpGdS9+3b19lmfYGzMleTzEy9+5w+HebN04rmnHO0Ihw9GmJjy55z4IBWzN98o3fG33yj6f7yF/jd77RS7NCh5Dn5+XqH++KL8PPPxd0TjRtrZd+iBTzxhLYY3n5b76Y3bYKXX9a77Tp14IwzNI833aTH8vK0xfHQQ3qtrl31NXgwjBihlWRGBnz9tZ6Xm6t3+oWV9Nq1+iooKO5KGTwYrr9eg0kQOAdZWdA4dz888gh7vv6aVs88A0OGBOX6peXkaANq82atyFeu1N6o1av18+bN9dWpk8a/xET45RdtHCUna37btdO42bGj3iu0aaMxd84c/bVVVqtWh0lPb3DkhH6IyFLnXF+/n1mAUFZZRoeQljknRyvdwrvPvDx47TW4916dbdK5s1bOderAu+/q3Wh8PMyaBb17F19nxw69M16xAv71L7jiCr1rfeghDTSF/dedO2tLYPhw7e644w49p18/rbz79yd540ZOK+w+KWyRvPVW2a6fjRvhzjvhP/+Bv/1Nu5d8+/MPH9Y7Y3931mGycqUW6913NfvdumlPVKNGG+jWTbvMDh/W3qEdO7TibdBAu/obN4bWrbVijosr/nXk5hZX/lu3auDJydHrHDgA+/drT5iv+vW1t+ukk/Q6+/drS2HTJr2Oc/rPYsAA7Ulr0EBjZmqqNlh279b0bdvqX/sFFxT3INWvr9+7YYP+c9m3T78jJkavWZhm06aVPPDASVX6PVYUIKyLyUSudevguee0YmvVCtq3J6b0Xbc/BQUwY4Z2Y7Rvr5Xteefp/84lS3TmyNatxQOnha+DB/XOuV8/ralmzdJa7Mwz4cMP9e68sIJ96CH49lvt2hkyRGu5YcP07v/BB7WmmjVLK3/Qvu4PPtAaa9kyzce33+p506ZpmsRETfOHPxRV7gc8HrjhBu0C+v57LYu/cYGEBHjvPf3+1q3Lft4gsLvTggJYuFB/fU2aaENp4EA9fft2rRh37NBKtLAiLeyCz8jQ4mVlaSXerZu+jjmmuCLctQv++1/9jo0btSjnnQdXX62Npu+/h82bi8dTRDQItG+vvWwHD+r4b2am/rlvX9ky1K+vd/8dO2qlXb++vmJjtWXQooXG9WOP1RidkFB+r9Lhw5rPDh0qHuvPzdWy+Iu/7dvDccdV/Hv3eHZXnKCKLECYyOMcvPmmdpPk5emt1sGDAPRr2xZefx1GjtSa6bnn4NVXtc3fv7/+T3zrLa1tunTRO/L/+7+S12/SRGuGVq20lujdW39u2VJnryxerN0ynTvDRx9p/33p2UIi2lW0eLEOAI8apTXSli1aiT/9tN6WltaokQ4GDxqkYxK5ubBokQas0aO1JeFPXJyW+Uj8BIeDB7VYnTrpnbfv8e++07vb7ds1zRdfwLZtWqHn5upYcsOGWvGVvvMGPV7Y89SsmRaveXMNFjNnakAorV07jam33QZjxmgl7mv+/G/43e9+V3R9f/GwUG6utiwKO1JiYjQvwWooNWgAxx9/5HQVDFuElQUIU/M5V/50zMOHtVvkiy+04m7VSjtzZ82CoUO1i6ZTJ023ZAn5V18NF16os1p+/FFropEji6+zfz/06KF9F2PH6vcuW6Z97HFxxVMhK6p1QLtz6tc/ck3ToYOOL1x3nQ66TptW3GoIRL16GmiqyDn9FaxZo1//yy96h797t1bOmzYVV9IxMdr10bevzhz97rviCUSgsWXQILj8cv0V5+fr3f6CBfp5YYugQ4fifvqmTSv+FR04oF1EubnFk4QSE8v/5wBQt64LuMKtV69sgDHFLECYmis/X++S335b+ymGD9eO3EOH9O4/JUXv/nfv1n4I0L4KER2MveWW4tqnQQMYMoSkadP43ZIlOktn+HCYPLl4xkxBgd6JH3NMyQDQp4++KqO8O3l/GjfWgBQkzumvZ8cOWLy4FT/8oL+qNWu0ks3P1wq3sE+9cEgDiivMwlmmvXpphRwfr0Fh0SLt1eraVf9qzj1X+97Lm7wzcmRgDZfyNGsWumUY5sgsQJjwck4r+LVr9ecBA7RyPnxYB1Q//FC7YFJT4e9/L3muiN6q3nijdnYX3lbm55d7h+/q1YO779ZXaXXqaGujBnFOK3XfIYCMDJ0ZmpysRY6J0QbLr79qC2DdOt87+1MBreRPPFFjUeEAZ7NmxX3qxx2nn3fpEvTFuKYWs38Kpvrk5upUyO++0wq/cApkRkZxmvh47aNISYGvvtK++MLAsHGjjgm0aFE06ExcXNnvOVL3Tw2Sl6d9+ZmZJddJpaZqH/zMmdqoOfbY4r5sj0eDRmEx8/P17r1bN+39uvBC/TW2bw87dy7j6qt7B7wUwBhfFiBMcDmnfRe+WySkp+u0k/ff16kjdevq1I9u3XROfPfu+vOBAzog/Nxzep2339bpKYUSEvRVi+Tna7FWrdIZNt9/r9MVC3813rFzvxo3hvPPh2uv1VbBL7/oDJ8JE3RMe9Cg4rv98oZpPJ79FhxMlVmAMEdvwwbtmH73XV0tlJ9fNk2jRtpV9Mc/aq1X3rTJyy/X2jM9/chz+8IsO1uzuWaNNnhSUnQmT+Gs1337tGXgq0sX7crp2VMbQc2b6/TJ2Fgd0y6s5Nu00TH2wu15jqSiQVtjqsoChKmcvXvhqae0NtyzR1sEa9fqZ2ecoYu14uKKp30W/ty5s84yCkRcnP+uo2q0e7fucJGTozNtGjXSPv7Fi4uXQWRllTwnPl4DQOH+bC1bFs/WSUzU4ZV27cJTHmOqwgKEKd+332otWLha6bXXdKO0vXt1ekvr1to5/uc/651/Lev+KW3LFp3p+umnOlvH30aanTrpOriLLy6Ogd26Ff86jIkkFiBMWVu3akf3J5+U/ezss+GZZ7SPpJbJz4d165qwcqXOAGrUSLtyGjXSzTi//VbT9e6ts19HjtQgkJmpYwUJCWW3PzImklmAMMXS0nQg+YEHtG/l8cd1/UFqqo6sDhigG8XUog7v3bt1A7TPPtPN0vbt6wdoYMjLK94YrUcP3f3iyiu1m8gYYwEi+hw+DM8/D088wUDQVU7HHqurhX/+WdOccw688oquhoKQ7YoZDDt2aBz74AOd/dq+vfb5p6Xp2Plu7xY17dvrThTt26/muutOoHNnjXOFC8YqenaOMdHKAkQ0mTFD99tfvx7OP5+9IrTPyND1BiecANdco6uLTzqpxtSWe/fq1NB9+4p30qxXT2f8/PorTJ2qjZ1RozT9jh26F1B8vI4TdOum8a53b10H5/HsJCGheI+jevXCPh5uTI1lASIaZGfrmMLrr+v+/V9+CeefzxqPh/Y1bLvvwmUU33+v++198okGAH/q1NHF1pMnayAwxgSXBYhIt3Gjbv+cnKzbS9x/f43YSyErS7dR8nh0IXVmps6a3bq1ePFYXJw+V+YPf9Dpoc2b6+KxvLziB5a1ahXWYhgT0cJfU5jg2rFDVyCvXq3rE1JSNCDMnKl7MIRRdraOCXz4oY4b7Nypa+Hi4nShWKdO+sCU+HjdVmLYMA0CxpjwsAARKbZt0x1MX3lFa+L4eN3CYtw4HXcIUx9MSop2AXk8JVcVn322rjk4ip2qjTEhZgEiEvzrX/qA9txc3bvorrvC2imfmal76r3wgu6+0bKljn8fc4xOL+3ZUx+rYIyp2SxA1GYFBXp7/sgjekv+6qvVMok/LU3XFCxcqBuzHjhQvKXEb7/pJCnQ8YK779YGjG0YZ0ztYwGiNjl4UKeqZmfr+9mz9f1118GUKSF/bmFmpsaip5/WQeK4ON2MtW3b4mcMd+4Mf/qTPoNn8GBtMRhjaicLELVFRoaO4C5cWHysTh3dOO/vfw/puoWcHH2e/V136Syjq66CO+/UpRM1ZLmEMSYELEDUBgcO6IPsFy+Gf/9b94EG3R21ZcuQfe2mTfqI5Nde0+cS9+6tO3EMGhSyrzTG1CAWIGoy5/RJM9ddBz/+qNN+/vCHkH5lQYGuo3v5Zd3ADnTTur/+VR/jUNED5o0xkcUCRE2UlqY7x33xhe5BXa+eBofRo4P+VXv3wksv6TZMmzfr0oldu3RcYdIkjU2dOwf9a40xtYAFiJrm0CG9Zf/1V90XafJk7V7q2DGoX5OTAx9+2JHRo3Wfoy5ddM++ESP0a0ePtkVqxkQ7CxA1iXO6nuGnn7R/Z8SIoF4+P18fhDNzpo4lbN7cjfPOgyefrJWPdzDGhJgFiJpkyhR45x148MGgB4cFC/Rx0Nu3a4/V2WfDDTf8xO23nxrU7zHGRA4bcqwpvvoK/vY33S/p7ruDeumXX9YB5hYt9LkJv/2mD9Hp339PUL/HGBNZQhogRGS4iPwiImtF5E4/n7cUkRki8pOILBGRkwM9N6K89Za2GI47TrfNOMqpQqmp8PXXOq79P/8D/+//6cZ3P/wAY8ZAs2bBybYxJrKFrItJRGKAKcB5QBrwo4jMdM6t8kl2F5DinLtERHp4058T4Lm1X0GBthYeewzOPbf4sWhVtHSpLmabO7fk8dtug0cfhZiYo8uuMSa6hHIMoj+w1jm3HkBE3gNGAb6V/InAowDOuTUikiAi7YAuAZxbu+3bpzvYzZypA9MvvFDlrTJ27oSbbtKB57g43Uq7f3/d5qJ9e3timjGmakIZIOKBLT7v04DTS6VZDowGFopIf6Az0DHAcwEQkfHAeIB27drh8XiqlNnMzMwqn1tZTdat4+R776XBzp2smzCBraNH6653VfDrr02ZPPlk9u+vx9VXb+Gyy7bQpEk+oM9eKHwmsz/VWeaaItrKHG3lBStzUDnnQvICLgVe83l/FfBCqTTNgDeBFODfwI9Az0DO9ffq06ePq6oFCxZU+dxK+fxz5xo1cu6YY5xbuPCoLvV//+dcw4bOderkXHJy5c+vtjLXINFW5mgrr3NW5soCklw5dWooWxBpQCef9x2Bbb4JnHMHgGsBRESADd5X4yOdWysdPAjjx+uzGubO1f6fKli0CB5+WDdzHTJEn9DWtm2Q82qMiXqhnMX0I9BdRBJFpD5wOTDTN4GItPB+BvAX4Ftv0DjiubXSU0/pdqhTplQpOKxZo+sXBg+GJUt06+358y04GGNCI2QtCOdcnohMAL4EYoA3nHMrReR67+dTgROAf4lIPjoA/eeKzg1VXqvF1q36SNAxY/S2v5LmzIHLL9dx7Gee0YZIkyYhyKcxxniFdCW1c242MLvUsak+P38PdA/03FrtrrsgL0+nGFWCc/DPf+pU1VNOgU8/tc3zjDHVw1ZSV4ekJF0AN3FipR4JeuiQPmL6llvg4ov1WUEWHIwx1cUCRKjl5Og6hzZttBURoNRUGDAApk/XrZk++ACaNg1hPo0xphTbrC/U7r0XkpPhk0+gefOATvnsM91Yr25dfSTEsGGhzaIxxvhjLYhQ8nh0YPq662DUqCMmd06TX3SRzoRNTrbgYIwJH2tBhMrevTqA0K2bjjIfQXa2zkz697/hssvgzTehceNqyKcxxpTDAkSo3HqrPnxh0aIjzkd1Dq69Ft57T580evfdIFJN+TTGmHJYgAiFVat0C++JE6FfvyMmf+klDQ4PP1ypcWxjjAkpG4MIhXvv1f6hSZOOmHTJEn1O0AUXwJ2R/dQLY0wtYwEi2JKS4KOPdPFC69YVJt2zBy69FI45JijPCTLGmKCyLqZgu+ceaNUK/v73CpOtWQN/+APs2KEL4Fq1qqb8GWNMgOyeNZi++Qa+/FK7lip4rudHH+nQxK5duiNrAMMUxhhT7SxABNPjj0OHDnDDDeUmefJJ3a/vpJN0ncM551Rj/owxphIsQATLnj0wbx5cdRU0auQ3yYcfwu23w9ix2tjo1MlvMmOMqREsQATLzJm6W+ull/r9OClJ180NGqQzYBs0qN7sGWNMZVmACJYPPtCtVvv0KfNRWppun9G2LcyYAQ0bhiF/xhhTSRYggmHfPu1eGjOmzBJo5+CaayAjA2bNsqe/GWNqD5vmGgyzZkFurgaIUt55B77+GqZO1Qf+GGNMbWEtiGD48EMdcT799BKH9+zR9XIDBuiGrsYYU5tYC+JoHTigax/++tcy3Ut33qlBYv58WyVtjKl9rNo6Wp99BocPl+le+u47ePVV3Wfp1FPDlDdjjDkKFiCO1nvv6WZKAwcWHSoogBtv1F6n++4LY96MMeYoWBfT0Vi/XlsQd9xRog/p3Xdh2TIdoLbnSBtjaitrQRyN55+HmBiYMKHoUHa2PvCnd2+44oow5s0YY46StSCqav9+eP113TcjPr7o8JQpsGmTfmQD08aY2syqsKp6/XXIzNRRaK89e+Af/4Dhw20TPmNM7WcBoiry8rR7aciQEltrPPaYNiyeeCKMeTPGmCCxAFEVn3yi/Ug+rYf0dH229JVX2oppY0xksABRFc8+C1266A58Xi++CAcPBvQYamOMqRUsQFTW0qW6Cu7GG3UGEzoU8fzzGi9OOinM+TPGmCCxAFFZL7wATZrAtdcWHXr1VR2gttaDMSaSWICojF27dBXcNddA8+aA7rLx9NPwu9/ppnzGGBMpbB1EZbz6KuTklFgYN306bN2qs16NMSaSWAsiULm5Ok3p/PPhhBOKDk+ZAj176mFjjIkkFiAC9fHHsG0b3HRT0aGNGyE5GcaNK7PTtzHG1HohDRAiMlxEfhGRtSJyp5/Pm4vILBFZLiIrReRan8/+5j32s4i8KyLhfZLz1KnQtSuMGFF0aMYM/fOSS8KUJ2OMCaGAAoSIfCQiF4hIwAFFRGKAKcAI4ETgChE5sVSyG4BVzrmewFDgaRGpLyLxwE1AX+fcyUAMcHmg3x10ubnw/fcwalSJDZY+/li7l7p2DVvOjDEmZAKt8F8GrgRSReQxEekRwDn9gbXOufXOuRzgPWBUqTQOiBURAZoCe4A872d1gUYiUhdoDGwLMK/Bt3KlTlfq16/o0I4duhxi9Oiw5coYY0IqoFlMzrn5wHwRaQ5cAcwTkS3Aq8A7zrlcP6fFA1t83qcBp5dK8yIwE638Y4GxzrkCYKuIPAVsBrKAuc65uf7yJiLjgfEA7dq1w+PxBFKkMjIzM8s9t8Pnn3M8sDg/nyxvmpkzO+Dc8XTq9CMez8EqfWe4VVTmSBVtZY628oKVOaiccwG9gDjgZiAJrdTHAi8AnnLSXwq85vP+KuCFUmnGAP8EBOgGbACaAS2Br4E2QD3gE2DckfLYp08fV1ULFiwo/8P//V/nmjd3rqCg6ND55zvXrVuJQ7VOhWWOUNFW5mgrr3NW5soCklw5dWqgYxAfA/9Fu3oudM5d5Jz7j3PuRrRryJ80oJPP+46U7Sa6FvjYm8+13gDRAzgX2OCc2+20dfIxMCiQvIbE0qW6a6t3qtLevfD119q9ZLOXjDGRKtAxiBedcyc65x51zm33/cA517ecc34EuotIoojURweZZ5ZKsxk4B0BE2gHHA+u9xweISGPv+MQ5wOoA8xpchw/D8uXQt7iYn32mO37b+IMxJpIFGiBOEJEWhW9EpKWI/L+KTnDO5QETgC/Ryv1959xKEbleRK73JnsIGCQiK4CvgDucc7855xYDHwLJwApvPqdVolzB8/PPOovJJ0DMmKEPkfMZszbGmIgT6FYb1znnphS+cc7tFZHrgJcqOsk5NxuYXerYVJ+ftwF+1yA75+4D7gswf6GTlKR/eh8MlJMD8+bp4jh7pKgxJpIFWsXV8Xb1AEVrHOqHJks1zNKl0LIlJCYCuhwiMxOGDQtzvowxJsQCbUF8CbwvIlPRtQvXA3NClquaJClJu5e88fHLL/UxEGedFeZ8GWNMiAXagrgDnXb6V3T181fA7aHKVI2RnQ0rVpR47vTcuTBwYNFu38YYE7ECXShXgK6mfjm02alhVqzQ6UreAerdu3VzvgcfDHO+jDGmGgQUIESkO/AouqdS0aZ5zrkuIcpXzVA4QO0NEPPmgXM2/mCMiQ6BdjG9ibYe8oCzgH8B/w5VpmqMpCRo3RqOPRbQ8YdWreC008KcL2OMqQaBBohGzrmvAHHObXLO3Q+cHbps1RA+K6id0/GH887TQWpjjIl0gc5iyvZu9Z0qIhOArUDb0GWrBnAOfv0VzjkH0OGIHTuse8kYEz0CbUFMRPdhugnoA4wDrglRnmqGnTshKwu66DDLl1/qYXu0qDEmWhyxBeFdFHeZc+42IBPdYC/yrV+vf3oXyM2dCyedpFtsGGNMNDhiC8I5lw/08V1JHRUKA0SXLhQUwOLFcOaZ4c2SMcZUp0DHIJYBn4rIB0DR03Gccx+HJFc1QWGASEhg/XrIyCixXs4YYyJeoAGiFZBOyZlLDn1OQ2Rav177kxo2JDlZD9n0VmNMNAl0JXV0jDv42rChaIA6ORnq1dMxCGOMiRaBrqR+E20xlOCc+5+g56imWL++aIprcjKcfDLUj479a40xBgi8i+kzn58bApdQ9vGhkSM7G7ZuhcREnINly2DUqHBnyhhjqlegXUwf+b4XkXeB+SHJUU2waZMulOvShbQ0+O03G38wxkSfqj4TrTtwbDAzUqP4THG1AWpjTLQKdAwig5JjEDvQZ0REJp8AsWyePlr01FPDmyVjjKlugXYxxYY6IzXKhg3QsCG0b09yMvToAY0bhztTxhhTvQLqYhKRS0Skuc/7FiJycchyFW7r1+sUVxGSk617yRgTnQIdg7jPObe/8I1zbh9wX0hyVBOsXw+JiezapZOZLEAYY6JRoAHCX7pAp8jWLs4VtSCWLdNDvXuHN0vGGBMOgQaIJBF5RkS6ikgXEfknsDSUGQub9HTdeMlnBlOvXmHNkTHGhEWgAeJGIAf4D/A+kAXcEKpMhVWpKa5du0KLFmHNkTHGhEWgs5gOAneGOC81w4YN+meXLixfbq0HY0z0CnQW0zwRaeHzvqWIfBmyXIWTtwVRcGwCmzZB9+5hzo8xxoRJoF1Mrb0zlwBwzu0lUp9JvX49tG3LzoNNycmBYyN3vbgxxlQo0ABRICJFVaWIJOBnd9eI4J3BtHmzvrUAYYyJVoFOVb0bWCgi33jfnwmMD02Wwmz9ehg40AKEMSbqBTpIPUdE+qJBIQX4FJ3JFHl++w3atbMAYYyJeoFu1vcX4GagIxogBgDfU/IRpLVfQQEcPAixsWzaBM2aQfPmRz7NGGMiUaBjEDcD/YBNzrmzgN7A7pDlKlwOHdKV1LGxbN5srQdjTHQLNEBkO+eyAUSkgXNuDXB86LIVJhkZ+mfTpmzeDJ07hzc7xhgTToEGiDTvOohPgHki8ikBPHJURIaLyC8islZEyiy0E5HmIjJLRJaLyEoRudbnsxYi8qGIrBGR1SIyMMC8Vl1hgLAWhDHGBDxIfYn3x/tFZAHQHJhT0TkiEgNMAc4D0oAfRWSmc26VT7IbgFXOuQtFpA3wi4hMd87lAM8Bc5xzY0SkPhD6JzJ4A0R2vVjS0y1AGGOiW6V3ZHXOfXPkVAD0B9Y659YDiMh7wCjAN0A4IFZEBGgK7AHyRKQZOpX2T97vzEH3ggqtzEwAdmXp85EsQBhjolkot+yOB7b4vE8DTi+V5kVgJtpdFQuMdc4ViEgXdBD8TRHpie4ce7N3T6gSRGQ83jUZ7dq1w+PxVCmzmZmZrFixglMAz9LtAKSnL8Pj2V/xibVYZmZmlX9ftVW0lTnaygtW5mAKZYAQP8dKr74ehk6bPRvoio5v/Nebr9OAG51zi0XkOXSzwMllLujcNGAaQN++fd3QoUOrlFmPx8MpCQkANGqjTwgaNap3RLciPB4PVf191VbRVuZoKy9YmYMp0EHqqkgDOvm870jZge1rgY+dWgtsAHp4z01zzi32pvsQDRih5R2D2JgeS0wMHHNMyL/RGGNqrFAGiB+B7iKS6B1kvhztTvK1GTgHQETaoVNn1zvndgBbRKRwKu05lBy7CA1vgFi3K5b4eKgbmc/MM8aYgISsCnTO5YnIBOBLIAZ4wzm3UkSu934+FXgIeEtEVqBdUnc4537zXuJGYLo3uKxHWxuh5R2kTt3eNKK7lowxJhAhvUd2zs0GZpc6NtXn523A+eWcmwL0DWX+ysjIgMaN2bA5hoGhX3VhjDE1Wii7mGqfjAxc06akpdkUV2OMsQDhKyOD/Max5OZagDDGGAsQvjIzOVzPFskZYwxYgCgpI4NDMRYgjDEGLECUlJFBBhogbCdXY0y0swDhKyODvfmxNG+uDwsyxphoZkvBfGVksKeerYEwxhiwFkRJmZnszIq1AGGMMVgLophzGiDydZsNY4yJdtaC8IrJzgbn+O1wLC1ahDs3xhgTfhYgvGIOHQJgb34ssbFhzowxxtQAFiC8CgNEJk0tQBhjDBYgisRkZQGQgbUgjDEGLEAUKWxBWIAwxhhlAcKrrk8LwhbJGWOMBYgi1oIwxpiSLEB42SC1McaUZAHCywapjTGmJAsQXoUBwloQxhijLEB41T10iJx6jSkgxgKEMcZgezEViTl0iMP1Y6kvUL9+uHNjjDHhZwHCKyYri6yYpsQ2DHdOjDGmZrAuJq+YrCwOxdgAtTHGFLIA4RVz6BAH69giOWOMKWQBwqvuoUM2xdUYY3xYgPCKycrigLMAYYwxhSxAeMVkZbE/39ZAGGNMIZvF5BVz6BD7xFoQxhhTyFoQAM4Rk5XFnlwLEMYYU8gCBMDBg4hzpOdYgDDGmEIWIAAyMgA4YLOYjDGmiAUIgMxM/cM26jPGmCIWIKCoBWFPkzPGmGIWIKBEgLAWhDHGqJAGCBEZLiK/iMhaEbnTz+fNRWSWiCwXkZUicm2pz2NEZJmIfBbKfFqAMMaYskIWIEQkBpgCjABOBK4QkRNLJbsBWOWc6wkMBZ4WEd/Ntm8GVocqj0UsQBhjTBmhbEH0B9Y659Y753KA94BRpdI4IFZEBGgK7AHyAESkI3AB8FoI86hskNoYY8oI5UrqeGCLz/s04PRSaV4EZgLbgFhgrHOuwPvZs8Dt3uPlEpHxwHiAdu3a4fF4Kp3RjsuW0Q1tQaxYsYjt23MqfY3aKDMzs0q/r9os2socbeUFK3MwhTJAiJ9jrtT7YUAKcDbQFZgnIv8FzgR2OeeWisjQir7EOTcNmAbQt29fN3Rohcn9++YbQFsQw4YNippWhMfjoUq/r1os2socbeUFK3MwhbKLKQ3o5PO+I9pS8HUt8LFTa4ENQA9gMHCRiGxEu6bOFpF3QpbTjAwOxzSigBiaNAnZtxhjTK0SyhbEj0B3EUkEtgKXA1eWSrMZOAf4r4i0A44H1jvnJgGTALwtiFudc+NCltOMDLLqNqVpI6hjE3+NCZrc3FzS0tLIzs6utu9s3rw5q1eHfm5LTRJImRs2bEjHjh2pV69ewNcNWYBwzuWJyATgSyAGeMM5t1JErvd+PhV4CHhLRFagXVJ3OOd+C1WeypWZyaGYJlHTtWRMdUlLSyM2NpaEhAR0LkroZWRkEBtl/5mPVGbnHOnp6aSlpZGYmBjwdUO63bdzbjYwu9SxqT4/bwPOP8I1PIAnBNkrlpHBwTpNbRW1MUGWnZ1drcHB+CcixMXFsXv37kqdZx0qABkZZIpNcTUmFCw41AxV+XuwAAGQkUGGa2YBwhhjfFiAAMjMtOdRGxOB9u3bx0svvVSlc3//+9+zb9++CtPce++9zJ8/v0rXr8hbb73FhAkTKkzj8XhYtGhR0L/blwUIgIwMDhRYF5MxkaaiAJGfn1/hubNnz6ZFixYVpnnwwQc599xzq5q9o1IdAcKeSQ2QkcG+fOtiMiaUJk6ElJTgXrNXL3j22fI/v/POO1m3bh29evXivPPO44ILLuCBBx6gQ4cOpKSksGrVKi6++GK2bNlCdnY2N998M+PHjwcgISGBpKQkMjMzGTFiBGeccQaLFi0iPj6eTz/9lEaNGvGnP/2JkSNHMmbMGBISErjmmmuYNWsWubm5fPDBB/To0YPdu3dz5ZVXkp6eTr9+/ZgzZw5Lly6ldevWJfL65ptv8uijj9KhQweOO+44GjRoAMCsWbP4xz/+QU5ODnFxcUyfPp2srCymTp1KTEwM77zzDo8//jg5OTll0rVr1+6ofr/WggC48Ubm5p9rAcKYCPPYY4/RtWtXUlJSePLJJwFYsmQJDz/8MKtWrQLgjTfeYOnSpSQlJfH888+Tnp5e5jqpqanccMMNrFy5khYtWvDRRx/5/b7WrVuTnJzMX//6V5566ikAHnjgAc4++2ySk5O55JJL2Lx5c5nztm/fzn333cd3333HvHnzivIGcMYZZ/DDDz+wbNkyLr/8cp544gkSEhK4/vrr+dvf/kZKSgqDBg3ym+5oWQsCyL3/YT59BB60AGFMyFR0p1+d+vfvX2ItwPPPP8+MGTMA2LJlC6mpqcTFxZU4JzExkV69egHQp08fNm7c6Pfao0ePLkrz8ccfA7Bw4cKi6w8fPpyWLVuWOW/x4sUMHTqUNm3aADB27Fh+/fVXQNeSjB07lu3bt5OTk1PuOoZA01WGtSAo2u3bWhDGRIEmPvvpeDwe5s+fz/fff8/y5cvp3bu331Xfhd09ADExMeTl5fm9dmE63zTOld6Czr/ypqHeeOONTJgwgRUrVvDKK6+Uuyo90HSVYQGC4gBhC+WMiSyxsbFkFP4H92P//v20bNmSxo0bs2bNGn744Yeg5+GMM87g/fffB2Du3Lns3bu3TJrTTz8dj8dDenp60fiFbx7j4+MBePvtt4uOly5beemOhgUIrAVhTKSKi4tj8ODBnHzyydx2221lPh8+fDh5eXmceuqpTJ48mQEDBgQ9D/fddx9z587ltNNO44svvqBDhw5ltsXo0KED999/PwMHDuTcc8/ltNNOK/rs/vvv59JLL2XIkCElBrYvvPBCZsyYQa9evVi0aFG56Y6Kcy5iXn369HFV8d13zoFzX3xRpdNrrQULFoQ7C9Uu2soc7vKuWrWq2r/zwIED1f6dFcnOzna5ubnOOecWLVrkevbsGfTvCLTM/v4+gCRXTp1qg9RYC8IYEzqbN2/msssuo6CggPr16/Pqq6+GO0sBswCBBQhjTOh0796dZcuWhTsbVWJjEFiAMMYYfyxAYAHCGGP8sQCBBQhjjPHHAgQaIOrWLcBnLYwxxkQ9CxBogGjcuOKdHY0xtc/RbPcN8Oyzz3Lo0KEjpvN4PIwcObLCNCkpKcyePbvCNDWNBQgKA4T/pfPGmNqrugJEIGpjgLBprsCBA9aCMCbkwrDfd+ntvp988kmefPJJ3n//fQ4fPswll1zCAw88wMGDB7nssstIS0sjPz+fyZMns3PnTrZt28ZZZ51F69atWbBgQYlrz5kzh4kTJ9K6desSK5+XLFnCxIkTycrKolGjRrz55pskJiZy7733kpWVxcKFC5k0aRKJiYll0h1//PHB/f0cJQsQaAuiUSMLEMZEmscee4yff/6ZFG9gmjt3LqmpqSxZsgTnHBdddBHffvstu3fv5phjjuHzzz8HdF+j5s2b88wzz7BgwYIyW1dkZ2dz3XXX8fXXX9OtWzfGjh1b9FmPHj349ttvqVu3LvPnz+euu+7io48+4sEHHyQpKYkXX3wRgAMHDvhNV5NYgMDGIIypFjVgv++5c+cyd+5cevfuDUBmZiapqakMGTKEW2+9lTvuuIORI0cyZMiQCq+zZs0aEhMT6d69OwDjxo1j2rRpgAaXa665htTUVESE3Nxcv9cINF042RgENgZhTLRwzjFp0iRSUlJISUlh7dq1/PnPf+a4445j6dKlnHLKKUyaNIkHH3zwiNcqb3vuyZMnc9ZZZ/Hzzz8za9ascrfdDjRdOFmAwLqYjIlUpbfEHjZsGG+88QaZmZkAbN26lV27drFt2zYaN27MuHHjuPXWW0lOTvZ7fqEePXqwYcMG1q1bB8C7775b9JnvtttvvfVWuXkpL11NYgEC62IyJlKV3u77/PPP58orr2TgwIGccsopjBkzhoyMDFasWEH//v3p1asXDz/8MPfccw8A48ePZ8SIEZx11lklrtuwYUOmTZvGBRdcwBlnnEHnzp2LPrv99tuZNGkSgwcPJj+/uF4566yzWLVqFb169eI///lPuelqEnEBPu2oNujbt69LSkqq9HnjxsGxx67mkUdOCEGuai6Px8PQoUPDnY1qFW1lDnd5V69ezQknVO//q4yMjDLPW4h0gZbZ39+HiCx1zvX1l95aEMA778D55+8MdzaMMaZGsQBhjDHGLwsQxpiQiqRu7NqsKn8PFiCMMSHTsGFD0tPTLUiEmXOO9PR0GjZsWKnzbKGcMSZkOnbsSFpaGrt3766278zOzq50RVjbBVLmhg0b0rFjx0pd1wKEMSZk6tWrR2JiYrV+p8fjKVopHS1CVWbrYjLGGOOXBQhjjDF+WYAwxhjjV0StpBaR3cCmKp7eGvgtiNmpDazMkS/aygtW5srq7Jxr4++DiAoQR0NEkspbbh6prMyRL9rKC1bmYLIuJmOMMX5ZgDDGGOOXBYhi08KdgTCwMke+aCsvWJmDxsYgjDHG+GUtCGOMMX5ZgDDGGONX1AcIERkuIr+IyFoRuTPc+QkFEekkIgtEZLWIrBSRm73HW4nIPBFJ9f7ZMtx5DTYRiRGRZSLymfd9RJdZRFqIyIcissb79z0wCsr8N++/659F5F0RaRhpZRaRN0Rkl4j87HOs3DKKyCRvnfaLiAyr6vdGdYAQkRhgCjACOBG4QkRODG+uQiIPuMU5dwIwALjBW847ga+cc92Br7zvI83NwGqf95Fe5ueAOc65HkBPtOwRW2YRiQduAvo6504GYoDLibwyvwUML3XMbxm9/7cvB07ynvOSt66rtKgOEEB/YK1zbr1zLgd4DxgV5jwFnXNuu3Mu2ftzBlppxKNlfdub7G3g4rBkMEREpCNwAfCaz+GILbOINAPOBF4HcM7lOOf2EcFl9qoLNBKRukBjYBsRVmbn3LfAnlKHyyvjKOA959xh59wGYC1a11VatAeIeGCLz/s077GIJSIJQG9gMdDOObcdNIgAbcOYtVB4FrgdKPA5Fsll7gLsBt70dqu9JiJNiOAyO+e2Ak8Bm4HtwH7n3FwiuMw+yitj0Oq1aA8Q4udYxM77FZGmwEfAROfcgXDnJ5REZCSwyzm3NNx5qUZ1gdOAl51zvYGD1P6ulQp5+91HAYnAMUATERkX3lyFXdDqtWgPEGlAJ5/3HdHmacQRkXpocJjunPvYe3iniHTwft4B2BWu/IXAYOAiEdmIdh2eLSLvENllTgPSnHOLve8/RANGJJf5XGCDc263cy4X+BgYRGSXuVB5ZQxavRbtAeJHoLuIJIpIfXRgZ2aY8xR0IiJov/Rq59wzPh/NBK7x/nwN8Gl15y1UnHOTnHMdnXMJ6N/r1865cUR2mXcAW0TkeO+hc4BVRHCZ0a6lASLS2Pvv/Bx0jC2Sy1yovDLOBC4XkQYikgh0B5ZU6Rucc1H9An4P/AqsA+4Od35CVMYz0CbmT0CK9/V7IA6d/ZDq/bNVuPMaovIPBT7z/hzRZQZ6AUnev+tPgJZRUOYHgDXAz8C/gQaRVmbgXXSMJRdtIfy5ojICd3vrtF+AEVX9XttqwxhjjF/R3sVkjDGmHBYgjDHG+GUBwhhjjF8WIIwxxvhlAcIYY4xfFiCMqQFEZGjhjrPG1BQWIIwxxvhlAcKYShCRcSKyRERSROQV7/MmMkXkaRFJFpGvRKSNN20vEflBRH4SkRmF+/WLSDcRmS8iy73ndPVevqnPsxyme1cGGxM2FiCMCZCInACMBQY753oB+cAfgSZAsnPuNOAb4D7vKf8C7nDOnQqs8Dk+HZjinOuJ7hu03Xu8NzARfTZJF3Q/KWPCpm64M2BMLXIO0Af40Xtz3wjdIK0A+I83zTvAxyLSHGjhnPvGe/xt4AMRiQXinXMzAJxz2QDe6y1xzqV536cACcDCkJfKmHJYgDAmcAK87ZybVOKgyORS6Srav6aibqPDPj/nY/8/TZhZF5MxgfsKGCMibaHomcCd0f9HY7xprgQWOuf2A3tFZIj3+FXAN06fw5EmIhd7r9FARBpXZyGMCZTdoRgTIOfcKhG5B5grInXQnTVvQB/Mc5KILAX2o+MUoFswT/UGgPXAtd7jVwGviMiD3mtcWo3FMCZgtpurMUdJRDKdc03DnQ9jgs26mIwxxvhlLQhjjDF+WQvCGGOMXxYgjDHG+GUBwhhjjF8WIIwxxvhlAcIYY4xf/x/UlNjeXUqQEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### plot and examine learning curves\n",
    "\n",
    "epochs=list(range(num_epochs))\n",
    "\n",
    "plt.plot(epochs, train_accuracy, '-b', label='training data' )\n",
    "plt.plot(epochs, test_accuracy, '-r', label='test data' )\n",
    "\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('accuracy')\n",
    "\n",
    "plt.grid()\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine the trained Weights\n",
    "\n",
    "Let's examine the trained weights and check how they look like. This is a first attempt to answer the question what the ML model learns and how it does the classification. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAG8CAYAAADdHDSmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABs30lEQVR4nO3de9Bl11nf+d96W/JN9763Wupu2ZJtiMGyAykmFyCEZGaSCgmETAiTEKYgUxS3AQZj5zaVzMRFGCZDSEgmKUiAoYY7CROSTJyagAOhCMnEOGBjHN8ktaRW33WzfO13zx/v20lrr+8jraP9vn1are+nSlXW8jlnr73Ws9beW+95ntOmaYokSZIk6YXbWHcHJEmSJOnFzgcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSVdNa+2B1toX79Jnv7e19oVX/PvrWmu/1lp7qrX2zfP/f7eOey1ZpW+7OTcvFfOY24XP/6HW2l+94t9Xmd8didOrHSettfe11t54tY4nSUvcsO4OSNJOmKbpt82aviPJO6dpetP2v//NpcdorT2Q5Gunafp/n+O414yd7Bud+zpda/3ZNo+5XbXK/F752mt07NA0TZ+x7j5I0ij/YiXpenU8yXvX3Qlde1pru/UfFV9QzO1ifyRJV5EPVpJ2XGvt7tbaP2ytnW2tnW+tfR+85m2ttQ9tf23qN1trXzr7/9/aWntk+/9/f2vt9z1P+3/6ilJr7eeT/N4k39dae7q19trZ/1/2r+pXa+1HkhxL8nPbn/kdcNzPaK29s7X2+PZXr75kdk4PtNa+vbX26621J1prP9FaewWMzX/XWvu5K/79g621n7zi30+21u7f/t93ttZ+ZvtcPnLlV9BmfXvzFV9T+6ntY//VPNv91Dc692oeYI5/etb2va21v3nFvz9X/3Geiv6MjP1bW2u/nuSj84eZ7c+5e/t/f11rbWqtHdr+97e01n5gfn6z91PMlX16vv5sv+ZNrbV3bY/xTyR5xez/H57fy6+t4nj2uc+7frdft6NrGD7/v22tvaMac0m65kzT5D/+4z/+s2P/JNmT5D8k+Z4kN2XrZvB3b/9/DyT54u3//ceT3Jmt/8DzJ5J8NMmR7f/vdUlOJrlz+99PJHlN1T7/7O1/f2e2vu6UK///5+rfQL+edYzZ596Y5INJ/nySlyX5oiRPJXnd7LX/dvvz9yZ5X5KvgzF8dZLHt/twJMmDSR654v+7uP3/bST590n+p+1jvjrJh5P8l7O+vWz7M/6H7X5+WZJPJvmro32bzV05D7PzOJ7kmSS3XhEbp5J83va/l/0fmKcr+zM69u9OcneSV0JfTyb5zCQtyW8k+UCS12//+weTvHEg9t+Z7Zh7vj4N9OfynH3r9md9eZJPwZytMr9fPP/fq6zf3V7D0JfvTPLX172n+Y//+I//jP7jX6wk7bTfka2brbdM0/TRaZo+Pk3Tv56/aJqmn5qm6dFpmjanafqJbN3I/o7t//tSkpcn+czW2o3TND0wTdOHnqN9x/r3PP16Lp+X5OYkf22apk9O0/TzSf5Jkj85e93f3P78C0l+Lsn98w+apunD2boJvz/JFyR5R5JHWmuv3/73X5qmaTPJ5yY5ME3T/7x9zA8n+f4kXwF9u2H72J+apukfZushau55+7ZtaB6maXowybuS/NHtpi9K8sw0Tf9m+9+fq/9DcXTF+Y2O/clpmj4Gn/H49mf8gWw9EL0nye1J/qtsPdT+hyRprX1ta+0NRT9W7dNz9efzsvWQ9De25+ynk/y75zjWyPyOGB73q7CG35Cth1xJelHwwUrSTrs7yYPTNH36uV7UWvuq1tq7t78m9Xi2bqL2J8k0TR9M8i1J/nKSM621H2+t3Vm172T/nqtfz+POJCe3H3guezDJ0dnrHrvifz+TrZtv8q+SfGGSz9/+3+/M1kPVF2z/e7L1F6E7L/d1u79/Pskh6Nsj0zRNV7SdhGMO9W3FefjR/OeHia/c/vfLnqv/Q3G0bXTs6Zwvu5it8/2WJN+b5MkkdyT5+lxR+GSaph+Ypuk9O9Sn5+oPzdmDK7z2uT77uQyP+1VYw2/I1gOuJL0o+GAlaaedTHKMckYua60dz9ZfJr4xyb5pmm7P1g1Uu/yaaZp+dJqm352tm+8pyXc9V/tO9G+gX9P8PVd4NMndrbUr99VjSR5ZsX+XXX6w+j3b//tfpX+wOpnkI9M03X7FP7dM0/QHZ591KsnR1lq7ou3uFfvzrHNfYR5+KskXttbuSvKlefaD1XP1//ni6Mr+jI79c83f40k+J1tfZXtnth6s3pitm/ufvfyi1to7n+MzrjTSp+fqD83ZsRVe+1zz+1zHfd71m+z+Gm6t3bx9Dr/5XP2QpGuJD1aSdtq/zdaN3l9rrd3UWntFa+13zV5zU7ZuqM4mW8UasnUDm+1/f11r7Ytaay9P8vEkH0tyqWrfwf49Z7+SnM5WHhD51WzlmHxHa+3GtvWbQX84yY+v2L/L/lW2iiG8cpqmh5P8Ura+lrYvya9dcS5PbhcDeGVrbU9r7Q2ttc+dfdavZGucvrG1dkNr7Y9k7OuNV/pP577KPEzTdDZbf237wWw9RL3viv/7ufr/fHF05VzsxNhfzFY+0+W/Tj2ZrZylvzdN06Xt874lW1/RHLG0T7+S5NNJvnl7zr4s9ZytOr/PFccj6zfZ/TX827IVL888x3lI0jXFBytJO2r7JvQPJ7k3yUNJHs5WYvuVr/nNJH89WzeEp5N8VpJfvuIlL0/y15Kcy9bX0w5m6ytiVfuO9G+gX9+Z5C9uf/Xp22ef+8kkX5Lkv97u399J8lXTNP3WKv274vP+Y5Kns/VAlWmansxWYYdfvnyjf8W53J/kI9vH/YEkt0HfvizJ12TrLzN/Klv5Pp9YoUv/6dyzNV6rzMOPZqvIwpV/rXrO/g/E0ZX9+eYsH/uL2cpTutzHJ7OVY/X9V7xm+KtpS+Phijn76u2+/Ykk//B5Xjs6v88Vx8+7frdft9tr2K8BSnrRac/+SrYk6aWgtfarSf7uNE0/uO6+vFi01v77JE9P0/Sjz/viNXuxz29r7XuTXJim6a+suy+SNMq/WEnSS0Br7Qtaa4e3vyr2Z5J8dpJ/vu5+vch8Vq7RKnXX0/xu51f9oSS/sO6+SNIq/LV3SXppeF2Sn8xW5bsPJfnyaZpOrbdLLzqfleQFfbXzKrgu5nc7F+3/SvLD2f4arCS9WPhVQEmSnkdr7WeTvHuapr+85q5Ikq5RPlhJkiRJ0kLmWEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI3rPLim266adq7d++z2jY2+mezaZrw/VX7iNbaC35vdWzqOx3n0qVLQ++tzo9eSzY3N1/we6v3U9sNN4xNO53PKnM77/v58+fz9NNPL5vIFd18883Tvn37ntVGfa3ii8Z/NB5o7Pfs2VP2deQ4o0b7s8p5f+pTn+ra6HxofKvzpnOkPo3G4irrcmRPWVfMzvdZUvV/dB8ZHedVLNnrRi3t46il1xyyyt4z+n7aZ5966qlrMmZX2f8+/elPd21L1nx13VuyzxKaz6sVS9X6W3JvQeOzyrHp3Kk/J0+ePDdN04GhTu2A0ZhdZUxH1/dozC65dlXHGd17Vzn2kuOsMr6j16zR8anW/ug5PvjggxizKz1Y7d27N9/yLd/yrLZXvvKV3euqCfn4xz/etY1e8GlDXuWhg24KX/GKV3RtN954Y9f21FNPdW0ve9nLurZqkl71qld1bTRGn/jEJ7o2Gt/qODS+1DZ/0Eg4iOnCRuNYtd98883P+ve3v/3t+N7dtG/fvrztbW97Vhud18tf/nJ8P7VTPNA8PfPMM13bbbfd1rVVG9DTTz/dtY1e0Cg+KRYo3hNeG2fPnu3a5nOc8PnQ65LkySef7NroBohinmKO5oHmOxl7KPyu7/oufO9u2rt3b77jO77jWW2je2LCcUd7EL2fxnSVfXb0ekB9HL1ornIxHH0Io/dWN+KjNxGjeyodpxrzkb1rHfvsaMzeeuut+H4aq4sXL3Zto9dxGqfqJpr2WUKxRP2mvXdpLBEai5tuuglfS+dIezKdI733k5/8ZNdGaz/ha8zHPvaxru2bvumbHsQP2CV79+7NW9/61me10flX9wY0BtRG508xS/NO94QJ79203mhORu/Dq3s9ei3F3ei6qmL2ox/9aNdGYzn6H3JpfKs+0hqmOfvar/1ajFm/CihJkiRJC/lgJUmSJEkLrfRVwKT/M+Do13YS/vPa6J9J6U/HdJzqaz/050Z6Lf05lfpDf/Ktvu5Ex6E/x47msNCxE/6zL503fcWC3kt/Aq/m9sCB/qvR9JW5q6211p0Hjf0tt9yC76evqhH6ahN9JsUCxXbCc0+vpXVFMbLK107OnTvXtY1+1YriuDpHirvHH3986HXVVzTmqu9bU5+qr7NcTa21bq5G8xSS8f1q9GuTNJ8Uc9VraZ4oFunrivR1kOorKnTeNG50zaGv3FRfIxzNyRk9b5qb6hir5ChdTa21LiZW+WrmE0880bXRtWv0q4A0frSnJTymo7mf9NVGur5URr9eSDFLY1Gd42jMj3ylP+GYrb62Rqo1fDVN09T1mWKpQmNKsUR7Jd0vkOorpLR30/WM5onW1fnz54ePTfEwel+ySp43raPRr8mOfgWyOkfau1fJk/QvVpIkSZK0kA9WkiRJkrSQD1aSJEmStJAPVpIkSZK00ErFK1prXcIeJeFVCWGUGEgFDqi2PL13NEEt4WQ0Sq4bTaJdBSXC0hiNJp1XCZZ07vRaShKnsVjlB+vOnDnTtS0dt50yT3ikWKiKK9BrKaGekpjpd59GE+wTTo6lmB8tVEFrtZqjqkDKHPWRYrtal9S+03FTFZUZSVq/Wj9Ge6WNjY1uDC5cuNC9jn6PLlmtEMMczd3oj0AnvI+M/lbZaBGBqsDIaCEVSoCmmKuK79Bxbr/99q6N9onRH2qtzpHmdr5Wl/z48gu1ubnZnS/tnVXc0D5C1+LRYjcUh6vsdbRPj/6eE6n2P9qbqgJcczRm1bWE2keLq1Ac0+etUtBmlSIRu6W11u1DdG9UzQeNP8UIvY7GeZViYXQcKpY1+vuFo4U4Eh4POjatdVoHVSEP6hOdD43F6G85VjFL94OrFKXxL1aSJEmStJAPVpIkSZK0kA9WkiRJkrSQD1aSJEmStJAPVpIkSZK00EpVAZO+GsxoNZCEqyxRNbXRakBUbaaq0kPtVHWEKgdVFZrmqioqt91221B/6HyOHDkyfByqvkWvXVKtqhpfqlA2r9ayjgprly5d6saFKhJVlbRoTkZjnqr80HxUVeuosg1V5KGYpX5TVRyqMpZwNSJaq1QNjdZ5VcGOKu1QjNEaovGh86nOcaQ6XLXWdtPm5mbXZ4o5Wu+X3z9H8UAxSzFSVW0ijzzySNdGsURzTLFA817tI6NVVekz9+7d27VVcz9aaZA+k6pN0fhSpauEz3H+mddKJUva/6q1SPsa7Ys09lShjq5dVXW30Up4dOxTp04NHae6vlD7aKXAVe696BpBn0lrlfaEixcvdm1VhTVaRzSWV9s0Td340/lX1UFHq9fSWNE6pn2A7lETvhbTa+k4oxUzq2OPVsejsaRjU1XP6rW0L95xxx1Dr1sl5mhdrrKv+hcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSZIkSVpopeIVGxsbXULZKgUkRhPQKWmOPnM0uTUZTyilZL/z5893bTfc0A8dFd1IOAmPEkIpYZYSgClZN+GkQnotfSaNz2jid8IJmnPrSKres2dPl+hJiY007wknW1PRAErUpKTd0SIsCScnjxZTOHv2bNdGifPVnFCfKEZGiyKcOXMGj0PjS+uFxpzGkvYEmpuE1+D8teuI2dZat25p7qp9gGKZ2iiWzp0717XR+FXHprij/Xxkv0h4T6yKCdHeRH2nz6QE9SoZnwq2EOontVWFDUbNz3tdBVfm40oxUiW+07jQnFAsjRaxqZLkR2OE9j8a69ECQ8n4tZjuN6itKjhAxYNozGks6L6GzqfaE2gPra63V9OePXu66wWdK10nEl63dM2mcRk9/2qvoc8cLZJG+xoVcKnmk66xtAbp3mD0fiHhawStDbo2jhboWKWwVfVcQ/yLlSRJkiQt5IOVJEmSJC3kg5UkSZIkLeSDlSRJkiQttFLxis3NzS6Bi5Ivq2RcSjyjxEB6/+ivhFPyYMKJmpSgSsm1lKBKyfj0edX7KWGPiiocOXKka6MxT5K9e/d2bZRUSO+nhERKHq6ScOkcR5KZr4Z5YibFHBVHSDh5lM6Dxop+HZ1iu0pQpTmhQiqUsEzJqKOJ0gmf44EDB7o2SqCmPlYJqrRmaB3Q+qXPpNfRWCSc9DqP43UUrxjdZ6uiJ5RkS8nSo8m4NE5UpCLhuKG1RcnFFMd0jlXCMe3JtJ/fcccdXRud49GjR/E4hNY1xQ7tCTS3VcEVMu/70mIYL8SePXu65Hla21Xf6Hxpb6J9lgoOUCxVhTPotaN7N+3RpCooQudIcUPXdtp7q4Ir9JkUd7QnHD9+vGv7yEc+0rVVRR5ofmi9rcN8v6Ixra6RdK9Jex3dM9Hr6Nir3DPRHFNbFSNz1T0RXZ8vXLjQtVHMjxaaqd4/en2htU730lUxNvrMVQqu+BcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSZIkSVpopeIVSZ9kSkliVXIcJaiOJqhRgim9d5XkWErUpM+kRMvRX91OONmP+kNJ9lQIoEqYpcReQknFx44d69poHquEdxqPeSGBqojAbtrc3Ozmj5J2qehBwudLSaY0JzQmlCy5yq+rUwEJSgg9dOhQ10aFAKpYooRbSoSlPtIaevzxx/E41KfRX6qneKLPq+KO4nu+htZVCGC+l1CCb9W30bmjvYUStWn/quKGjkP7IsU8xTHNUVW0g64RNEb0mbQnVMn4NEZUEIM+c7SQR1Vogdrn+9E6igRN0zRUDKUqBkPvpXOlGKHX0XGq5HO6HtIeP1pAhvpIsZnwXkmfOVp8Zt++fXgc2j9ojCiO6Xxoj6HXJXyO1fX2amqtdfNC+xoVZkh43Y4WZ6B9iV5X7XVk9N6C+k3XyBMnTuBxKG7o3pPucamQyblz5/A49FoaI7o/Hy0EV+2ztIeuErP+xUqSJEmSFvLBSpIkSZIW8sFKkiRJkhbywUqSJEmSFlq5eMU8qWuVAgeU9EZJnZRIR8mbo5+XcKI2oYQ5SlobTQCs3k9Jc6NJtFWSKCUbUmIfFRI4efJk10YFLSqU0D0/9joKAdBxq19SJ6O/bk9tlOBLqiR5Sv6kz6S4oc+kJFFKLE44UZsSOim+KAH49OnTeBwqxkHrmvpDybo0D1XhjJEk83UUArh06VKXDEz9qBKbaW+hBGyKbXrvaBwnPCe03ijuDh8+3LVdvHixa7v11lvx2LTPUixSgj7t+9V+RdcY6ieN5etf//qujeKdilwkfB2c930d++zm5mZ3vqOFEBLeR2j/o5in49A40eclyf79+7u20cJYdD4Uc1VhKZp7KjxE76dCFVUhABoPOh/qO61/2o9Wue+jIgTrMO8brdnqGkDnS9cpijsq6LPK3lsVShvpz+g1l/byhGOJ9quq6MfIsRPeZ8+cOTP0frr/oXms5na0MEnFv1hJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kIrFa9orXXJcPQLx1ViHSXxUQIlvX80Sb5K6KbXUqInvY6S1ihRmsYiGf9laEomp+S6qtgBJelTEi4lH1LyNf1SdXVsSuSskoWvptZaWaBh/jpChSHovKhwCSXEUzJq9evfo79uT2uI1gEleT700EN4bBoPSqqm/tD4VImw1E5JqxR3VBRhlcIkNB7zpNWqIM21oFqL1Gfa12hfGhmT6nUJ72GnTp3q2qgABcUxJV9X++xocjElWlMBgwcffBDfT6+lvlNRBRpz2p/o+pJwEab59XJkv9tpGxsbXZzQNaHqG712dF+j/YLinWIz4fVCsXTw4MGujWKJ9vOqWAOdD8UI9Z2u99W1hM6HjkPuvffero3is7r3ovmpCl1cTZcuXequ0XReVdxQfFJ80z0EoWthtcePvp/ige716J67mk+6f6TX0nnTeqnikMZ3tDjSaGGRqtAP7QmrFAW6du8aJEmSJOlFwgcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSZIkSVpopaqASV+Bg6qWVBVfqNIYVVF56qmnujaqWkJVcarqY2fPnu3aqNoVVQ6hijpU8a2qhkhjRJVmRiusUWWoJDl8+PDQ+0cry1E1r7vvvhuPTZV/roWqgBsbG111ParQVMUsVfaisaI5pviiCjhV5TmqvEavpXigOb5w4ULX9qEPfQiPTRUzX/3qV3dtVCnw6NGjXVtV5YzWG8USof2E1nlV8ZHa5/tbVf1uN7XWunke3TsTjjGKWVoHt99+e9dG1SmrmKW9m6qp0f5Hn0kVqKiyVPV+ei2N26OPPtq10fpNuAoVxSydI40PrYEqZkeq4q0jZqdp6vZ7Ooeqb6OVDGk/rq67cwcOHMB2qlBJn0nVIKlC5QMPPNC1nT59Go9N+xVVcqN1OVpFszo+xedopdXRCpwJj1FVce5qmqap6weNSXVvQGue2mju6Ho4WgUzqatMztH+STFHezTFe8L3vu9///u7NrovoUqB1Z5Aa53Gg+43aB7pfE6ePInHJlV1SOJfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpoZWKV2xubnZJc5QcVyU2U+IZJUZTMhsldFJyLCUHJ5x4RsemREs6DiUf0lgknDR3/Pjxru1973tf10aJgpRsm/C4jSajUnIrJfBWyZQ0vpSMuQ7z+aPEz6roCRV8oKT20aRXKnJBRViqPo0WBKFYeOihh7q2X/mVX8H3UwEEijuKbVr/VfEKOh96/2gRF0oervYEis9VElR3C+2zo8nrCe9XdF4333xz10Yxd88993RttIdUx6G9m2KpKtgwd++992I7FaCgZOmq73PVfk5ovdH80HlTH2n9JTxG8/dfK/suFVKoEtWpyMi+ffu6ttHiAHfddVfXVu3xtJ+PrrfRe5Dqukl7HbXRNYsKftBaS/h86Po+eh2ja1YVd9S+jgIrc621oTVORRQSPofRglWExrmKG7r20b0ixSfFEu3xVfEeKlRB9650zaY1WD0vEJov2ieo7xSz1X0JvXaVfvoXK0mSJElayAcrSZIkSVrIBytJkiRJWsgHK0mSJElaaKVM7dZal9w4mnCccKIYJefR6y5evDh0jCopkhJpKVlwNImWkvCOHDmCxz527FjXRgnUlHB35syZro2KECTj50NJhTRulORYJTTS/CyJlZ1y6dKlLhGREnSrohCULEljQGNFsU3FK6qEY0qapTGkIiz0Okq+fPDBB/HY9FoaiyohfK4qGEBjRAVkKI4pKZrWQJWgTK+dJ56PFgvZSXv27OkSyylZmRLSE44HirH77ruva6N98vDhw11bNaY0J7Sf0xqiwgT0usceewyPTWuL9iVaq5T4XSWd07lT3+n9NBZU7IA+r3r/M888g6+9mjY2NrpxocI/NEcJ7yO0T49eiw8cONC1VUWfaG1QQRG6FlPBlFOnTnVtVQEdmk8qKkMxu0oyPe0JJ06cGHovjQ/1h46R8Hq5FopXbGxsdPsL7X907Uk4lmlOKI5H7xdOnjyJxx69HtLeQHsqFUKh63DC++9HPvKRro2Ko9A1q7qWUJ/oM+kc6X6DYo7WecL3T6vcC/gXK0mSJElayAcrSZIkSVrIBytJkiRJWsgHK0mSJElaaOXiFfPkPkrWqxI1KbmREtfoV48peZuSVqtEOOoT9Z0S5ui99EvVVKQi4YQ7Sl48evRo10bJ15SYm3CyNCXxUULkaOJjlThOc7uOxP85KrhCybhVojrNHc3JaPL12bNnu7Yq6ZeSR6mgxcGDB4eOQ2vo9OnTeOzP+IzP6Noo7ighlJJwKb4SngtqGy32ssrcUoLqfHxpDq+G+bjSuVLxjYT7vHfv3q6N9koqaLFKwRWKByrEQDFC76XjVLFE51MVLBg5NhW0SDhuaP3TuqaxeOSRR7q2qlgTXZ9WKWKwWy5dutRdJ6mvVQEjWt9USIBikQpaUFJ6tQ/Q/QYVpaD4ev/73z/0edUcUfEKim9K+qfxpetDwmNE9ysUd7RWaXyr+z5aW+vaV+fmY0jnSnFYof2K7oOosAvtA1UBCdr7qbgK3WceOnRoqD9VzH7gAx8Yej+tVVrnVQEJOsdz5851bVURpzmKzyoOKWZXKRK0/h1ZkiRJkl7kfLCSJEmSpIV8sJIkSZKkhXywkiRJkqSFVs4gnCefUWLeKr+kPPqr3pTQSYUUKBk04URiStSkYgV0PpQAWP2qPCWeUxIvJUlSv6tiB6NjSQnU9DpK9qvmlhIN54nj60iybq3h+c5VvwRPyaw333zz0LHpfKkvVf8onihRk/pDifcUH9WxKcboOLTeaG1Uic20f1CCK7VRLFIbrb+Ek2bn66AqlLCbNjc3u36MrK/L6Lxono8cOdK1UcxS0n81n6N7EO29FEuUkF0lK1M7HZtiiV5HBT8S3rspyZzijo5NMUbXyoTXy3xuqwIRu2ljY6Nbe6NFSxJOIqe1TPFFYz/63oTXFhUjoXVF76X1Qp+XJHfddVfXRkny1Ebrn4pUJMnhw4e7Nho32uOpkAcdmwoLJHxfQ+vgapumqZs/ug+orvc0z7R/UmGH0eJS1T5A8UD9oTm5++67uzbaZ6s5onVA9ypUGIv22argCp0j3VvQfjd6f10VgFoan/7FSpIkSZIW8sFKkiRJkhbywUqSJEmSFvLBSpIkSZIWWql4RWutSwCrEkIJJelSsh8lVY4m61WJu6O/hk59vO+++7o2StarfsWZEhApaY7OkZJ9q0ILlBBOyc6UeEpJhQcPHuzaqmQ/Shaez886CgFM09SNAcXNKr+uTomNNMeU5EnFUapxoT5VBVLmHn744aG2/fv34/spCZqKAxw/frxro1ioktYpwZ/2FDpvikVa59V6oTlbR4zOTdPUjQH1lfbJhNc87YsUn1T0hBKbV9mDaJ+l41BBjCeffLJrO3bsGB6b4pPWEI0lxWEVC1TYhcaD9glKJqf9iMYx4SII87VxrRQJojGlmEu4z9W1Zo6uXTQf1b3KSEGQ6jPpmk97Z1WEgN5Pyfx03RiNr4TPcbR4BaF7lVWKal0LNjc3uzGg869iltYtzQldd6mgBe2J1fWe7uFojmlO6JpPc1QVKKJ9kfYr+ky6DlEsJVxciV5Lewft+1QcqTpH2ruq1xL/YiVJkiRJC/lgJUmSJEkL+WAlSZIkSQv5YCVJkiRJC/lgJUmSJEkLrVQVMKmr7l2pqhhFVUuo+gZVA6KqYtSXqvoYHZtQxakDBw50bVRtpargQtVRqI3Om6oTnT17Fo9D50jViKgS0WhVpqryDxmt6rSbNjc3u+o9VHWpquhIlWBo7mj8Dh8+3LXRfFZVh0ar3lF1otEqmFVlude+9rVdG1W7oipKtAarqotUxYo+k+aB9o69e/d2bTQWVZ9oLK+2G264oavcRHtqFTdUtemxxx7r2s6fP9+1jVa8rOKGKi/RcWgfof2L1lp13jRGdI2gfZqqrlXXElrD1E+qjknVvB566KHhY9O4z8dyXZUt53FCFR1p7Oi9Ca95qopI805t1bWL9huq0Eb7J80xxSfNe8J7EF2LRq85tNYS3mdpPKq4m6NraFXdjfpeXW+vptZat19RHFZVNmm/ov2T9ht6HX1eNaa0Duj6PHqfSv05efIkHpsqAFLlWKoqTeugum+mKq90PjRntC5HnzUS3kMp5iv+xUqSJEmSFvLBSpIkSZIW8sFKkiRJkhbywUqSJEmSFlo5g3Ce1EXJqFWBC0oyo0Q4SuKjZFRKuKsSDQklrlFSJSWYUnJclfhJSbiUHEsJgPS6KnF8tPgFJZjTmFMhgKowCR1nniy4ytzslNZa17dPfOIT3esolhIeF0rApHOjBEh6LyWiVu+neKA1RAUb6LxPnDiBx7777ru7NuonJZ5S0jqt/YTHjc6R1iDNGa1BSlqtzN9fxftuooIrlMRcnRcli9OY0n5B8U5zXBUhoLmnPfXUqVNdGyXYU3xQvCe8J9NxKAmZ9mjaJxMuEEAFhWgs6dpI7632ypFk9HXss5cuXer2HFqz1ZjSfkVrj6591Eb7bHVfQvE9mqhO65I+79ChQ/h+ilnyyCOPdG10v1AVr6A1SEU2aE+h99JYUhGQhO8jqnuYq22+Rql4UTVHNAZHjx7t2iiOaW2MXveqPlGRNSrKQ3s07b0PPvggHvs973nPUD+p+ATtTVVhF7q+07WN1hZdn2geqmMTun+q+BcrSZIkSVrIBytJkiRJWsgHK0mSJElayAcrSZIkSVpopeIVm5ubXQIXJehWSYyUwE7JkpQQSu+tEuIJJb2NJsdSkiJ9XpVoSInRH/nIR7o2+qXr0V+VrlAyP80Z/Uo2JT5SQYWEE2HnCYTrKAQwTVM3BjSfVWIztdP401hRYiQlx1aFAGisH3300aE+jsxHkhw/fhyPTec4ej7U71V+4bwqAjNHibAU29SW8Nrav3//s/6dErd32+bmZjd/VDiE9smEz5cSo2kPIxQ31d5La/zChQtd22gRgvl8JPX1hQog0PWFCijQsas4pPGl99PaoNdRMnl1LaG5mPdnHfvsxsZGV7SBks+rBHC6RlJ8U4L+Y4891rXRvkLXuITjifY/ikXqD62Nao+nAhQf+MAHuja6X6AYqQqXUDEOKppCfR8t1lSha1F1H3E1TdPUrR0a0+regArJ0No7fPhw10Zjf88993RttG8nPJ+0r1Ec0x5E8fXQQw/hsfft29e10VhQH2kN0rpKeM3Q9Xi0kNwqhUmoCMwq993+xUqSJEmSFvLBSpIkSZIW8sFKkiRJkhbywUqSJEmSFlo5M3uenEdJiFWCKiUSUxv9YjMlwlKiYJVUTf2kpER6PyXWUlJhlehORSkoYZneT+ND/Uk4gZASuin5cPSX2atfqqZE4/mYryNh9YYbbuh++X20GEnCCaE095SUSfNB41wVIaC4oURrSt6mz7zrrru6NkowTZIjR450bRRLo2NJyaAJr2H6zNHYocICVTL/SPGadcTsxsZGFzuUjFsl01KhEEpYphihOD516lTXNl9Tl40WLKCxnxc/SDjhmBL+k+TOO+/s2mivpPVC+2yVoE/ttJ/TtYT6Tp9HRRESXq/XQsxO09TFHcVsVcSBxo/mfnRvGC20k/B1jmKRjkOJ96sUQvmt3/qtro2KV9C6ovVbFeqh99OY095BhXNorVFBhoTHl66rV1trrbse01qs5o5eS9d3iiUqVEHjXB2b7i1oz6AYoTbaV6qiHVSoi87x/vvv79poDVXXsdF7ftq7ae+lz6N7moT3j1WKAvkXK0mSJElayAcrSZIkSVrIBytJkiRJWsgHK0mSJElaaOXiFfOENkqmrxJUKcms+mXpOUq4owS1KhGO3k/JrFQwgI5DiZ9Vst+FCxe6NhoLQsmoVcEBOh86DiXxUTIktVWFFl5MKOaqmCWUqElJ8pSoScehZOdKVRhm7tChQ10bxWf1q+dUsICOPZrsS0nRyXjBFoptGt+Xv/zlQ/1J+HxWiYOrieauKpZD65b2DErGpcIjNM6UkJ7wXknjT210PrSfV4UzKMZG44GOXRVCorVOx7l48eLQcWhuqiIEdI7Vdedq2rNnT3ceFF90LaxeS+d17ty5ro32CypQQvORJMePH+/a6BpBBRdo3uneoLrPofVGxxktHETXpoTXKxU7oPV2+PDhro3Op7r3okIAq1zzdss0Td01gM6hKgZTjTUdZ+S9VLyC2hKOO7qvoxih+aTCI1REpfpMuubQ/kmF06pYoHOn4j+0f9J9Kq2B6j6c+lTtycS/WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCK1cFnKOqNtSWcHUUqhhDlVmoigpV+agquFClHar8QdXQ6HyoOhFVYku4ohlVPKEqKvSZVRUVqnBCn0ljQXNDx6kqUFGVmnl/qILcbrt06VJXxWs0FhKOJxpnqlRGVahonKnKWDJeeZIq2VEFG6oGRP2pPpMqU1Fs02fefvvteByqckafSfFFY0FjRntMwpWDbrnllmf9+zpitrXWjT9VU6uqg45WS6RKTnS+tM9WFfNor6TKsTR3tC5p7qrKjXTs+XxWaN+vKrlRNUmKJaoAOFo9i9Zq9dp5HFTVInfTNE3dWqa+VuuJ5pQqldE+QmtjdN9OuNLYvffe27UdOXKka6PrBu1fjz76KB6b9koat9HKeqtUk6R7qqNHj+L756iyZnXsJetyN+3Zs6e7D6Oxr86Lrtt0H0DjTPsAzTG9Lhnf4+maTXNH/b7vvvvw2KMVes+fPz/0Xtr3k/FrOd2LEzrH6r2rxDfxL1aSJEmStJAPVpIkSZK0kA9WkiRJkrSQD1aSJEmStNBKWa4bGxtdAigldFGyc8JJapS0SsUrKBGOCkBUhTMIJX9Tsh8lEFJCIiXmJZzYR4mP1J9VCgFQsjXNDyWO05gTSoJNOKF73rZK8t9Oaa11CdOf/vSnu9dR4mfCieB0rh/+8Ie7Nkpop2NXxRVonuj9lHBcxcgcJYgnvN5Gz4fGrCp6QkmilOBO80PriuKT+li9dp4wu46YpUIAq+x1NC6099KY0h5y9913Yx8JzV1VNGCO1gEluVdrdf/+/V3b6dOnuza6PlEsVAVtqE80vrRPLCkikvBYzvtZFXDaTa21bt3THlIVHqG5pzmhvYVeR+O3yrEpvqnACc07xVe1BmhPpf5QIj8VhanihtYG3W/QZ1JBh+oej9AYrWNfJfPrEhWcqgpI0HWFCqHQtXj0voziI+F5Hi0gQfFFx6lilgo+0L5GfaS2xx57DI9D96R0faKiFDS+tE9UhTOovSoyRvyLlSRJkiQt5IOVJEmSJC3kg5UkSZIkLeSDlSRJkiQttPJPtM8TGSmBr0r2o4TQ0WR+SjyjZL0qcXc0cY2S3ighkYpcUBJdwudD/aTiF6PJrQkn544m+I8mPlYJwKNFHq62zc3Nrh+UaFkV76CkYUJjRWNKa4DGLuFk1le/+tVd22hiMhWKoGTdpJ7nudH4qmKBxp2Spen9tH6pP1WRh0OHDnVt8wTVqujGbtrc3Oz2AlrzVdzQPNOY0lhR8jrtQdWxCc0xvZ+KQlB/qqRqShKnuaexoKRo6k/1flrX1M8DBw50bVQIidqqz1ylYNNuoX2W9pCq6AmNKcUNJb/T62icqrih/YoKVdD7qajM6FpL6vEY+Uy6jlWxQP08ePBg10Zrne5L6L6tilm6f6J1fbXRPkvjVBWQoPOlGKE5pvvM0WI3VZ+ooAgdm675o8UeKnTNoXunhx9+eOi9CccIrXUqMkTrja6hVREW2rtWueb5FytJkiRJWsgHK0mSJElayAcrSZIkSVrIBytJkiRJWmil4hWbm5tdohgl61WJmpScR0mZo8UrKIGvSqCkpFdKqqREOkrso9dViXDUp3379g31kQoTVL+uTgnY1WvnaHxXSeSkJMl54uU6CgG01rpExNEkz4TnjuaeEi3Pnj3btVGBk+rXv0eLnlBSJR2b4riKDzpHSlim9UvnWBXDoMRoKhpABXEoOZbmkfpYHXu+Dqq9bDdtbGx0Sbq0d1bFFSi+ab+hOaF9gBLiq0IAtMZpDVEs0XtpD6qSiCm+qaAFrVXqDyVFV2g86LwpPinGqD/VcebvHy2IsJM2Nze79bhKQSM6L3o/xci5c+eGPo/WQIWKBtC1j/pDcbhKwRVK0KeCA9R27NgxPE51/LnRdUltVSEkWsM0Rlfbnj17uj10dI4T3hfpukmxNFpkqSpGQte00fVG/aH4qO716LpJ6Ni0N1X7FR2HxpcKAtH50H5e3ZfQvfQqBVf8i5UkSZIkLeSDlSRJkiQt5IOVJEmSJC3kg5UkSZIkLbRS8YqNjY0u0X40GTfhBHR6LSVBUuIYJeFVhQAoofvEiRNDr6OEOUrMqxJE6bwp8ZzeP/oL3QknG1JyHo35aDJplexH758n4Y4W0thp8/Ol5ODqF84JJY5SfFIyKRV2oCIVCcfDaDI/jTWt1Sohk9YRJdJSQigVQKh+XZ36+dRTT3VtFNsUczQW1X5ExQXmr11XzC4p9DKa2EyJ6hRfR48e7dqqfXY0sZnWIBUhoPiqikpQLI/u3XTetG8nvAfSceh1tAapP1XMUszP43gdMdta6+Z0NJE/4b1uNKGexoQK21RordEeRrFNxSdWKR5CBTVobVEfX/Oa13Rt1T5L50NxR7FDewftMbSmE16vVeGdq2lzc7MbF9pD6HqUjMc3jRWtb4qbVe5naZ5obdC+f8cddwwdI0kOHz7ctT3++ONdG60NKgpRFcmg/ZPWAcUXjQXNTRWzdK9TFRQi/sVKkiRJkhbywUqSJEmSFvLBSpIkSZIW8sFKkiRJkhZaqXjFNE1dAhcluFHSWsJJapSgSklilIxKiWyrJKpT4hoVEhhNBq8SR6mfo+e4yq/XU98p4ZaOTW00N5QEm3DS6zxxcpWk3p2yZ8+eblxonKsEytFiCDTHlJhMn1cdm8aaEjAp7qg/o8n9VT9Hi2RQLNH6q147et6j8VkVlRkpQLOkiMQS8z2H5qmKm9FCNJSgS/M5OscJr3FKQqYCRaMFDGg/Tvh8aP5GiztQoZnqOLT3UlI1xRzFdlWgg+Z2fg0enf/dRgUKqmsXofGnsTpy5EjXduHCha6NEucT3s9H26iwAcUCFUdJeE8dLexAa43OO+GxpP1j9HpH67IqXLOuAkAj5vsYFWGgManaae1R3I0WKKmKV9A9Nu2ztDZo7miOq2s2XYuowBFdi48fP961VXsdHef06dP42jk6H4rDar3Q+K6yr167ES9JkiRJLxI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQitXBZxX27jpppu611XVYagyDlUJouodVDFllcp8VMmJKhpS1aHRalPVsekzqaIPnQ+9jqraJTzuo1XNqAoKVQ2qqi5Spa51VAGc29zc7OaPquNV1Q5pDKjaH40zxTbNUVW1juKTKtOMVrA6depU11bFB8XiaDVJioVVqi5SNSLqJ30mxVwVs/SZ8ypf66iwNk1TF7M0plV1PDpfGufRSnaj1VwT3gNpPmltjFadrKqMje43o32kCmEJxzxdI2h+KOaorarIRXM738+quNht8/OgPXWVqr0UI7R30+uoCt6hQ4fw2PR+Whu0DugcqapfdW8wWhWZKr5Rf6jSccKV6Wi9URyPHrs6R3r/tVApcJqmbn+n+az6SvNEc0/rcXSOaT4SrhY4WpGQ+kPHrioS0vvPnTvXtVFs0/Wlup8lFHe0J9BxaB7p+SXhWK7uEcn6o1uSJEmSXuR8sJIkSZKkhXywkiRJkqSFfLCSJEmSpIVWLl4xT+CiwgxV0tvFixe7NkpaHS2EQAlmlDyYcIIwFQegNupPlVxMKOGOPpOS/SixjxINEx4jei0l9tF50/hWiZz02nlS4WghjZ3UWuvGmgpIVEUKKMGX3k8xT2NKKPky4fmcF1eoUCzR51EsVH2i5M3R11WFBSiJl8ac0OtGi+EknEi/rsT/uZGYrQoB0JhSfNOapSI2qxTFoWNTYjPt06PJwVVxJIpFOsfRfZbOJUnuuOOOoeNQGxUXoPOuir1QUYVroRBA0q9xWvNVsji9lmKM5p7mneauKq4w+n7ab2gNjhYdSjgeaD6p7zRmVcyOFqqgsaDzGS2Gk3B8r6MoEJnHGBU4qfabUXTNpn2WXrdKQTRaL/Q6+kyK4+o+c0mhMnovFdhIuNgdnc9oARlSrRc6TlWwiVwbO7IkSZIkvYj5YCVJkiRJC/lgJUmSJEkL+WAlSZIkSQu10aSzJGmtnU3y4O51R9e549M0HbiaBzRmtZAxqxcbY1YvRlc1bo1Z7QCM2ZUerCRJkiRJPb8KKEmSJEkL+WAlSZIkSQv5YCVJkiRJC/lgta21dri19uOttQ+11n6ztfbPWmuvba2daK29Z5eO+fLW2k+01j7YWvvV1tqJ3TiOrk9ritnPb629q7X26dbal+/GMXT9WlPMftv2sX69tfYvW2vHd+M4uj6tKWa/rrX2G621d7fW/nVr7TN34zi6fq0jbq849pe31qbW2ufs5nGuVT5YJWmttST/KMk7p2l6zTRNn5nkzyc5tMuH/pokF6dpujfJ9yT5rl0+nq4Ta4zZh5J8dZIf3eXj6Dqzxpj9tSSfM03TZyf56ST/6y4fT9eJNcbsj07T9FnTNN2frXj933f5eLqOrDFu01q7Jck3J/nV3T7WtcoHqy2/N8mnpmn6u5cbpml69zRNv3Tli7af9H9p+7/Yv6u19ju324+01n5x+78uvae19ntaa3taaz+0/e+/0Vr7VjjuH0nyw9v/+6eT/L7tBSE9n7XE7DRND0zT9OtJNnf7BHXdWVfM/sI0Tc9s/+u/SXLXLp6jri/ritknr/jXm5JYvlmrWNc9bZL8L9n6jwEf362Tu9bdsO4OXCPekOTfD7zuTJLfP03Tx1tr9yX5sSSfk+Qrk7xjmqa3t9b2JHlVkvuTHJ2m6Q1J0lq7HT7vaJKTSTJN06dba08k2Zfk3LLT0UvAumJWeqGuhZj9miT/zwvrvl6C1hazrbVvSPJtSV6W5IsWnodeWtYSt621NyW5e5qmf9Ja+/YdOZMXIR+sVnNjku9rrd2f5FKS1263/7sk/6C1dmOSn52m6d2ttQ8neXVr7W8l+adJ/gV8Hv11yv8ypZ200zEr7bZdidnW2p/K1k3DF+xm5/WStOMxO03T307yt1trX5nkLyb5M7t8Dnrp2bG4ba1tZCul5auvUt+vWX4VcMt7k/z2gdd9a5LTSd6YrQv0y5JkmqZfTPL5SR5J8iOtta+apuni9uvemeQbkvwAfN7DSe5OktbaDUluS3JhyYnoJWNdMSu9UGuL2dbaFyf5C0m+ZJqmTyw7Db2EXAv77I8n+aMvoO966VpH3N6Srb+UvbO19kCSz0vyj9tLsICFD1Zbfj7Jy1trf/ZyQ2vtc1tr8/+yeVuSU9M0bSb500n2bL/2eJIz0zR9f5K/n+TNrbX9STamafqZJH8pyZvhuP84//m/Qn15kp+fpsm/WGnEumJWeqHWErPbX0/5e9l6qDqzC+el69e6Yva+K/71DyX5wA6ek65/Vz1up2l6Ypqm/dM0nZim6US28lm/ZJqm/293TvHa5VcBk0zTNLXWvjTJ32itvS1bSXcPJPmW2Uv/TpKfaa398SS/kOSj2+1fmOQtrbVPJXk6yVdlK3/qB7f/PJokfw4O/fez9V8DPpitv1R9xU6dk65v64rZ1trnZqva0B1J/nBr7a9M0/TbdvDUdJ1a4z773UluTvJTbas20EPTNH3JDp2WrmNrjNlv3P4r66eSXIxfA9QK1hi3StL8A4kkSZIkLeNXASVJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpoRtWefHNN9887d2793lft7HBz2uXLl3q2lprQ8emz6T3bm5u4vup/YYb+tOnPlbnM3rsPXv2DB2HzoeO/elPfxqPM9pPOs40TV0bnU91jOrcr3ThwoV89KMfHZvwHXLzzTdP+/bte1YbnWtldEwJHWe0LRmPT3o/9ZvmaHT9LVUdZzQWR+eM1lplZK2fP38+Tz311FWP2fk+OzpOldF4GLVK3NBraewp3lc5x9H9anQdVOc4eo2gNnovWbrPPv3002vfZ1eJWWq/WnsDjSm10ftHY7s69uh9wOhYVHGzZNyW7j2jY/nQQw+dm6bpwPAHL3TTTTcN7bNLr12r3MPN0Z5YHWen7+uqOR69lx5dV0uvJaN9X+V6N/qZVcyu9GC1d+/evOUtb3lWG53oq171Knz/xYsXu7aXv/zlQ8d+xSte0bXRZH7sYx/D93/iE5/o2u64446u7emnn+7aXvayl410sTz2rbfe2rV99KMf7dpuvPHGro3O+8yZM3gcGnean9Eb9meeeaZre+UrX4nH/vjHP961zQPxe77ne/C9u2nfvn1529ve9qw2WmDVDc/omNKi++QnP9m10bE/9alP4bEpPp988smh91O/aQ1QzFVWuZjOVRd8Wlt00aGxpM+8+eabu7bqpobGct6ft7/97fje3bR379689a1vfVYbxVx1cabzpXGmNTv6HxKquKF1RMemsd+/f3/XRrFdXSBp/6X9k2KJxrfa9+kaQeuNjk3nTetqyT773d/93fje3UT77CoPynReNH60h1GMjO4N1bEplm677baujeKGru10D5AkTzzxRNdGcTd6c37TTTfhcSjm6f2jDxaj7014fGkuvv7rv/5B/IBdsnfv3nzbt33bs9poX6uuHxTfNC60ls+fPz90nPl/rLiMxpT6TmuDYpvihtZa9X7qJ/VxtN8JX0votbT30jzQ/Wy1H9F40BqqYtavAkqSJEnSQj5YSZIkSdJCPlhJkiRJ0kIr5VhtbGzg957n6HvDCedTjeaCPPXUUwM9rL8PS9/NpH7S+VG/6Xv+1XdS6bud9H1N+i4ufSe/+q44nTt915Ta6HxuueWWro36nYx/33gd5t/9pnGqcktoTpck49P4VQmqozkjNM6juSnVd/JPnz7dtVGOwWjMVfkq1E/6Hja9n8aSxqzKYaPzqeL7amqtdfFI57BKzgjtYTR39F6Kz2qvI9R32lvOnTvXtdEcVd/Jp76PFpqg8aG9N+F8GYo72vfpekfrcpVcpNH37qbNzc0uJmhMqlyj0cIQo7lYNJ9VDhDF9+2339610d4wWoykuhZSLFMb5fXRWD7++ON4nGrc50bXBp035ZYlfI2prnnrRntV1dfRIiU09xSzFJ+0J1bo2NR3eh3tGVVsHzx48AX3h2K7WpcUN9RPmjNaq6N7TMJ7d3WvRPyLlSRJkiQt5IOVJEmSJC3kg5UkSZIkLeSDlSRJkiQttFIG4TRNXSIeJY5VBS4oiW800ZqS8Cihszr2aIIwJahREjMlVVcJ+nQ+lLxNyZ+UcFclFY7+MOjoL29TgnqVPE3jNh+jdSSstta6caHzqooW0DyNovmg9VIlUFJxldEfcKTj0HmfPXsWj01xQ+uNklFXKQRAyfx79+7t2mit0ntpzKoiD7QfzdfWOgoBtNa6cR0tNJFwzNL4015JYzo671U/aZwpFql4D63LqhgJrYPRHzymfldrf7SozOiPb4+utYTnZ76vjp7zTpuvFVp3VXEFGgNae6MFZ0aLliQ8n9R3ii+KWTpOlfg+WtiKik+MFvJI+BwplkaL/1Bsr3Jfci0UCdrY2Ojmhca0+kFyOt/RH7WmNpqPakwpPkd/IJhiabTQTsJ70+gc02dWRU/o3GncRous0XurdUlzPlI46DL/YiVJkiRJC/lgJUmSJEkL+WAlSZIkSQv5YCVJkiRJC/lgJUmSJEkLrVymbV79gypljFZDSsYr01BFHqrAUlX+oQojVB1ltFoLVUapjk0VuaiSyVNPPdW1UUWtqpIJVUSjqjD79+/v2ui8Ryu+JXVllytV1XV20zRN3dxTZamqOh5VpqJ5pnOjmKWKd2fOnMFj01hT9Z5RVJWx+jxqp/Oh8aGxoDWd8NqiWKSqQ6PVNqlyV5KcP3++a5vvKeuoCkjHpapp1Vp87LHHujYaZ9q76TgU79WYVhUuRz6T9mj6vOr6Mnps2quqsRw9zpIKWLTHVxVmR6rWriNmNzY2hsaQrrkJj98q4zJHe10VN7Qn0zzRuNL50DhUFcXoHoauT4T26KoqII0v7dPURvsxHfv222/HY9Pev457AerD/D5ulf2G5onut2ivo3mi+Koq2o5WxKXrIR2H2qr1vNMVPKvxpbGk685o1crRfifj93MV/2IlSZIkSQv5YCVJkiRJC/lgJUmSJEkL+WAlSZIkSQutXLxinmBMSXiU7JhwIh4lNlLS6r59+7o2SgilRNSEk9Eo6Y2S5EcT4ilpvDrOhQsXujYqcjE6ZgkXpbh48WLXdurUqa6NEsdpbg8ePIjHpnOcJ/tdK4UAaEwOHDiA76UiJTRPFHf0XkqqpETrhMef4pMS2mmsRxM/E064ffTRR7u2xx9/fOi9FUoSH032pddRcmyVhEvvn683Gu/dNk1TtwdS4mw1zhSLowm+qyQxE+onJfhT3+k4q8Rs1T5H64WuJVUhALqWUJzQWNCx6XpXJXTT3jM/zipJ1jtlmqYudlYplkNzT9ckGhda8xQLVQEJ+kwqxEDzRHsixUJVHInigZLs6f6HVOdIfaLj0OtoHmi+qgJWdL9SFb+5mjY2Nrp+0P5XFUyh147GHRV2oPsFKlKR8NqifZ/2Wbqfpfiqrn20n9M5UjENiptqn6W+0xqkNurPKsVeaH5GCrRd5l+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGmhlYtXzBPaRpMik/Gkf0qWpERBSsKjRLbq2FRkg45Dn0mJcFVxBkqEowRoSuwb/fXpJPngBz841CdK9qUxpwTJKtmP+nQtFAJorXXnMZq8nvD403xSjND4UaI0xXGFEkdpTkbPsUr4pxh57Wtf27VRIZTTp093bVUSLhVcoTih9ULnTXFYFdOhtTFPtKbj7rZpmrrjPvXUU93raI4TTvqlWKTPrPbuuWq9UHzSGFJxAOojzWe1/9F8jhY4GU2KTjiJmT6T2iiRn9ZgVaCI+jSPg3UUryBVoQpC15/Rz6T9kz6vKvZC+w1dix955JGu7fz5810bxXG1Xuh8aL1QkQy6DlWFZmhdU2EsGkvaZ0bXQJLccccdXds67gXmpmnq4oTGie4dE77+0Pqk9UjXJBqnaq+j6ykdh4qm0PWZYvbYsWN4bJrnQ4cOdW1Hjx7t2qhoW3XfPHqPTbFIc0PxTgVDkjqWR/kXK0mSJElayAcrSZIkSVrIBytJkiRJWsgHK0mSJElaaOXiFSPJ3FXSLyVGUsLd6K8zU3Lb448/jsem44wmb9Pr6Jeqq4Rj6jslb1IiPyUfVgl3VBhh9LzpM6mtKgRwzz33PO+x11EI4NKlS935UrJ4lTxNc0eJjaPJwfTeqngF9fPw4cNdG80xvZcKxVRzUsXyHCW4UpELSvKujk9J5nQ+lMhK81AlotKx56+tEmt32/y4tLYp4TjhRHkaAzp/aqPk4GpcaP+lfp48ebJrO3LkSNdG8U79qY5DieeUjE9xc/HiRTwOxSft07T+6dpG+2yV3D+y1tcRs9M0df2ga2R1fabxp5insaL9k/avalwooZ6KBtCc0OuosEBVaObhhx/u2qgADMX2gQMHuraq0ALdk1F80+to3Gg/rgrf0H1aVUjkapuvJ+pXtRZpXGjuaVyoeA+9rjo23a/Q2qKxp7VBBS1++Zd/GY9NRVNo3Ghd0v5355134nFozdDaoH2G5oYKbVXjS2tjtLBT4l+sJEmSJGkxH6wkSZIkaSEfrCRJkiRpIR+sJEmSJGmhlYtXzFHyV1UIgBIj77rrrq6NEnQpKXO0CEPCyZZ0HHo/Jf1TEYcqOZbe/8QTT3RtZ86c6dooyZFel/BcUPIiJcJS8vWDDz7YtVVJuDQX8+TjdSRV79mzp0u2pET+KmZprChuaJ4o3qmtSuSlpEyKTxrX0UIx1bEpcZzmfrSoAhVmSbjQAiXXUlIwJZjSuqqKV9AYVcnfV9t8TlfpKxVcofVJ8/TQQw91bTR+1VoeLc5Ax6Z9jRKOKRYSjmUqmkJJ3lRogsYx4Zin91ORDSp2QMdZpdDPfCzXURigtdaNC51/VXCF9l+KG5p7ihvaV6rCGVQEgvaWO+64o2ujPZr2eIqP6jNp3Og+6dy5c10bxVeFxpf2TzoOFRGozpGuO1UcXE179uzp9lUqZFL1dXQfoBih+1G6vq5yP0vXA4p5KlRB+/6HPvQhPDbFA8Xx/fff37WdOHGia6uK3dG40b44Gouj9xDJeDGdin+xkiRJkqSFfLCSJEmSpIV8sJIkSZKkhXywkiRJkqSFVs4gnCfGUpIoJdYlnIBJyZaULD2a9EvJh1WfRpMqKRmYkvWqX2ampEQ6DiV+U7+rQguUvEgFLWjOaNweeeSRru3o0aN47Hvvvbdrm58jjffVME94pDGpYpZikeaOEo5pPilmq2RzmqeDBw8O9ZHOh+KzShylYgkUx5QQSmuDiqgkfI7URuNL5zP6K+xVn+b7zLoKAcxjp0piJg8//HDXRom3dP40VrTfVGuZYpFinuaTzpHWEBW0SMYLj9Cc0nvp2AkXO6CkaopPGvM777yza6sKLYwUEllHkaDWWhcTNO9UTCRJDh8+3LXRPk1Fo6gwy+h+kXDhJkL7PsUN7YmrFK+gcaM9cXQsqs+kIkEUXxTbq+wJdO7U96ttmqbu+kfXOBr7hOOJ9gwqCEL3szR+VXEFmhNaL1TYgYqp0Xura9+b3/zmro3uF+hekT6T+pPwvQmta7rm0z5L8V6tS7ofropgEf9iJUmSJEkL+WAlSZIkSQv5YCVJkiRJC/lgJUmSJEkL+WAlSZIkSQutVBVwmqaumssqVbOoUtm5c+fwOHNU5YMq8lQV8wh9JlVhoQo2VCHo+PHjeByq/EPjRlVLqKJMdY6j1a4IjcXo+CRcMWXez3VUq7p06VJXbYzm4/bbb8f309xTRR6qBkRVbWgMqsp8VF2K1gtVJxod66riEY0HjRtVHaL4pDGrjk/jQWNB76XKcrRWE65kVFUvvJo2Nze76kejVU0TrjhFVeYoRqhCE+1LdIxkvDoezRPtVTQfVRU32oOoShpVSKuqQxHqE80PrSGKY+pjtV5obczjYB377ObmZrmXXKmKmzNnznRtFN90/rRH0+uqql4Ui1QdjsaV5on2RKokmXCM0DX/5MmTXRvFUnV9pvikaxbtExTvhw4d6tqoOluSPProo0Pvv9ouXbrUnS+N6W233Ybvp3invYXijiqbPvDAA11btcePrLWE553uE48cOTL0uoTPke59ab1Qv6u9jtrp2BSzVOWQ9ni6n0rG788r/sVKkiRJkhbywUqSJEmSFvLBSpIkSZIW8sFKkiRJkhZaqXjFnj17usRMSo6jtiRdQnb1Wkp2piQzSvKkRLbqOJSMRgl3p0+f7tooOZb6kySve93rujZKZqVEVkowPXjwIB7nscce69ookZbOmxKAKQHw6NGjeGwqEHDjjTc+69+rRMzd1FrrzoOScasEShp/Gpf5uVZtFEtVUjX1idqoCAOhBPEqiZgSdmltUMzTPFdFIejcqYgBJZhTHFMc0n6SJBcuXOja5uO7jkIAGxsbXdxR0n+VwEzjR22034wm41drmWKe9kpKJKZ97bd+67e6tqoAAq0tSpynRGuK96qoDBUIoDGimKVj05hX11Daj+Zrq9rLdlNrrVtnFAtVAjglpVPMUtzR61Yp9kLofoOK94wWkKC9N+EiGbSH0Xqhex2Kj4QLClF80jzQeqM9tYrZw4cPDx37atvY2Ojmr7oWE7ruUmEd2qcfeeSRrm10jhOeZ+o73ddRkZETJ050bTRvCc/z6P0wxSE9F1THobgZvZdeJeZozvbv3z/8fv9iJUmSJEkL+WAlSZIkSQv5YCVJkiRJC/lgJUmSJEkLrVS84tKlS12iKCVvUtJqhZLDKfmT2qrkYkLJfpQsSL9+TUmBdGxK/Ew4CY+SCinpmBJZq19xp2RD+gVpSsy75557urbP/uzP7tooQTypf3V93ajgCiUxUhxffv8cJVtSPFDc0LGrX3anmKcYo1iihG46R0qeTjg5mZJR6RwpjqsEVWqnNTiaMEvrnH5VPhkruFL9+vxu2tzc7MaV4pDmOBkvCERtNH6rJP1TkRLqD8Ui9YeK5VRJyBRLNG50bCqqQIUmEk7AHi08ROMzWvAj4bU1v47ROe+21lq3nmhvqArJ0DWJxu+pp57q2ugeoiqSQWhMaU+mvYD2aCoi9dBDD+GxaYxov6I+UlGJap+l44yeD80ZzUN1DaXr0zr21blpmrq1TPeE1Xqi86Jxpnum8+fPd20Us9X9Fl236f1UVIfuE1e5vtB9AL324Ycf7tpor6sKrlABGbr3pXmgz6TXVTG7d+/erq0qFkXWH92SJEmS9CLng5UkSZIkLeSDlSRJkiQt5IOVJEmSJC20UvGKjY2NLnl+lQRVSoKmhDJKjKTkTUrgq45NSf8f/OAHuzZKaKdkaUqYrX6pmhLp7rzzzq6NkuOoP9WvcVMCNSULUtvrXve6ro3G7F3vehcem+Zx/v5VkuB3yubmJiaMz1H/E05Up8RGihFKvlwl0ZoSSinp98yZM10bxRytl2pOKNmazpGSnSkhlM67ej+Nx6lTp7o2Ggs672pd0vjOE3Np/ndba63rG51rVXiEikXQPj0aX7T3VsUVaM94zWte07VREjIVqiAXLlzA9ne84x1dG+3xhMZy//79+FqKT3o/xTwVF6CE+Wo/ovier8t1xOw0TV3cjSbYJxx3tDeNFnugcaJiDwlfd6lwCc0JxRcVHLj77rvx2PRaKg5wyy23dG20fqvCYVRcgPZeikVa66PFhBKes2rvWjdas9V6ovGnaz7dr9GYkKqwFa0NWkPHjh3r2uh6SIUmPvCBD+CxR+/Z6X6B7s+p2EsyXryG7nsfffTRro32iepemvaEav8g/sVKkiRJkhbywUqSJEmSFvLBSpIkSZIW8sFKkiRJkhZaqXjFNE1dch8lslXJfpTUToUFRn8JnX69uipeQUmz1E96HSXHUfJl9QvSlCw4mphHSXhVIQAqdEFJ4vRr3JSsRwmqlACYcDLmfH7WVQhgpOAKxVzCybwUI7QOKMGU1kCV0E0o0ZPmjhJZKQGaEkyT8bijdUAJ1JQonXDCM72W2igh+9WvfnXXRusqGSskQOO426Zp6saF1k6VAE7nS3E8mhBP7rjjDmynveWuu+7q2g4dOtS1UYI/raFqrd53331dG+1hFMc0z/Te6v3Uz9HiShTH1dzSep3P2TqKBLXWuuvP6LU94bihWKT9Yt++fV0bxXaVfE59ove/733v69roHCk+qI8JF+UZPR/aE6r9igou0bWcPpPOZ5UCRdSnVa55u4WKsY0U4Xqu147eG9C9Gl03q7VMsUx7PL2OrqVUrKoqoEP3G7SH0Vqlc6zum2kPpOsG7R1UHI7WefW8QH2virMQ/2IlSZIkSQv5YCVJkiRJC/lgJUmSJEkL+WAlSZIkSQutXLxinrhGiYlVMholf9FrKbGPkswoaY0KWiT8i9iUbEmJ1pS8SMmg1JZwEiD94jONz2iyc8JJ65Swd+bMma6NEgUpebtKjqWEyPm4rasQACVbzlUJqpScTInE9LrR5O3qV9gpPkeLsNC6onM8efIkHvvs2bNdG503FTGg864KAVTrdY7Oh9Yb9bFKlKaiNPNYqRKyd9v8uLTXVfsAreXRohRUJIj2Y+pPkuzfv79roxih19FY095ZJVVTO80xFWa5cOFC11YlNtM6or17tHjNwYMHu7ZqvVAy+rygxTpidpqmrs+UqE7nn/AeSPsaJa/TPkCJ/NSfhOPm9OnTXRtds2k/p72K7j8SjqWq0MUcrcHqHoQ+k+KECvrQ6yg+qbBSwvvRuvbVK126dKm7b6FiX9U1ivYW2itHC6lQHFIcJxxjNKb0fjoOxQddCxJeG7R/0nnTmNH+l/D6p8+89957uzbau6nwT1W8i95f7R/Ev1hJkiRJ0kI+WEmSJEnSQj5YSZIkSdJCPlhJkiRJ0kI+WEmSJEnSQitVBWytdVVsqDIVVfNIuPIIVVOiyh/URhVsqkpOVDGFqp5QJSeqakPViarqJlRNhCqzUKWdhx9+uGurqutRhSGqZET9oaouo69LuNrTvDocVVDaba21roIOVUhbpcIaVfShqlZUoYkqN1YVwKjyD1VfpPOhdUVz98QTT+Cxaa6o4hGtK1pDVWXG0fdX1a7maA1UlZVo/5if97VQvSrhNU+xmfBY09xRG40JjX1VMYrGn/Zk2pfofKhqE/Wx+kxal7SG6DhVFShar9Q2eh2j+aLqWQmf+zy+11F9Nen3IboPqPbZ0erAdJ2hmKP9q6omSX2i9x8+fLhro+qpFDfVtY/WEb2W4mZ0T0x43CjuqO9UVZXisKpuS+dTXfOuptZatzfRedHYV6+law3NMY0pvbeqaEtzT/2hPZHW2j333NO1VTFLfRqtQnz06NGujdZVwvvYkSNHurbRaoq0H1UxS5WNRysYJ/7FSpIkSZIW88FKkiRJkhbywUqSJEmSFvLBSpIkSZIWWrl4xTyhlJIiqyRRSpSnQgCU7ExJZsePH+/aKGk/Sd7whjd0baOFIeh89u/f37VREYGEk+boOGfOnOnaKKG7SgCm44wWEqBkUpqb6hzpM+cJjetIqr506VI3XpSgW/WNzovGhcaPxooSP1dJoKTX0mfee++9XRvFcbVeKDn22LFjXRsluFKRjCoJlxKDKWYJJceuErNVcYIX0pedtLGx0e2rFJ+rJMTTnkGvoyRkKuxQFfWg4iw0hnfccUfX9sADD3Rt586d69qqJGKKbxoj6vudd97ZtVVzT3FHcUzjSwUZKA5pHKvXzq+X69hnNzY2utih+ajWHMUd3VvQ+qY9jJL2T548icceRXNC11zq9913342fSTEyeh2n41Qo5ilO6N6LiiLQnkrrIuH5qYqMXU0bGxtdPNE4VYU2Vhn/OdovaJyqfYDmhPZKWi90T0nrcpXrJhXboTVN8U7XgoTjk8ac+kltqxQOqwrwjfIvVpIkSZK0kA9WkiRJkrSQD1aSJEmStJAPVpIkSZK00ErFKzY3N7ukOUr+qn41npJZqe39739/10bHGf119IST3l73utd1bZQUSAmA1G8qdJBwUQpKNKTkulUSPynJnJIFKSmQ3kvjS5+XcMGCeQJglfC+myipmgpAUFJ5kuzbt69ro+RPmhOaO0papaIuCccTjTPFJ80nJXRWyeSUiExrkAoJUJJ4lVBP6/LWW2/t2mhtUB8pybsqQkC/zj5f/+soXjFNUzfWtBarZPHRgg2jBQNoDVRobVGMPfTQQ10bxTa9rlovNFcUd2984xu7Nuo3rdVV0PgSmq9Vkqfnc7uOfXaapu48aL+p+jZacIP2RPrMxx57rGu7ePEifibtYbSnUtzRZ1KhrOqeiK6ndBw6R7oHofuXpE7Sn6N5oM+kmKUxS7hwTrV3XU2bm5tdUSW69lRrkfYMGoMDBw50bTQfdJ9Y7UH0fhpTihvqN8VcVYSO9umjR492bRTzhw4d6tqqwlZ0fR4tEkb7BMVste/QWFaFRIh/sZIkSZKkhXywkiRJkqSFfLCSJEmSpIV8sJIkSZKkhVYqXjFNU5fQtn///u51VaIkJaONFnF49NFHuzZKrqOCFAknzVGiNiW4UbIeFSZ4+OGH8dijhQBGkyGr8R197WiyNBVAqJKPqfjDvKDDugoBzOeKkumrRHNK3KWEY/pM+kVxSpSuCmdQ+969e7s2ik8qAPOmN72pa6uKkVDcvPe97+3aKG6eeeaZrq0qgECxSOuaknhpzkYTWRM+x9GCA7uptdbFU1VkhFAhFUrKpv2PCrPQOFFsJzz+NJ8077TfjCbdJxzLR44c6dpon6X9vEpspn2W1gGNOe0nlLRezffIuK1S+GKnbGxsdNc5mvfqGkB9pusmrWWaTzo27UsJJ89T3FEfT5w40bXdeeedXRsl9yfJ448/3rXRPRVdC+h+g+I4GS8ydvbs2a6NYo72mCpmaU9ZR4GVudZadx5UaKMqykH7J40/jT2N37Fjx7o2GvuE54liieaY4pPWS3U/e/fdd3dtNMej1+Jqj6f1T2uQ9mla6/TeanxpLKt7JeJfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaSEfrCRJkiRpoZWKV+zZs6dLhlvlV+MpcY1eS4mN9LrTp093bVWCKiWEUhslwlJyK51Llfg+eo6U7EyFPCgpMOHEaHotJZlSAiH1p/rFdEqupbFch/kY0HxUfaUkU0qyp6ISVHDhrrvu6tooeT0ZLzxC80nrgNZLVdhhtHAJxdfoL8AndYGAOUp6p4RZmscqZukz5+NRJbfuJiq4QknRFJuX3z/SRudPbRTH1XzS+FMbxSfts9V+TqiflLxN50h7d7U26LWj10EqXkPzWBUhuFb21LnNzc1uvOjaVRU9oUIANH6nTp3q2kaT1+kYCY817alUXIDeS6+rChTR3I/GDa2XqtDCE0880bVRzNI1h9b6KgUpaB3R+6+2aZq68x0topLw3FOBA3rd6H5TrRcqykMFKCiOaZ4uXrzYtVEcJxw3tPcS2nupEEfCBSRovdD6p32S3kvrqnrtKnuvf7GSJEmSpIV8sJIkSZKkhXywkiRJkqSFfLCSJEmSpIV8sJIkSZKkhVaqCtha66plUOWfqjoMVUepquXMUXUUqvJz4cIFfD9VxaGqI1QVhyqwULWpqnoMoQojVKHk5ptv7tpozBOuSEOVYqg6HFV1ocooVSUyqlA0r7BTxcVu2tjY6OZllYqOVAGHKi9RRUaKWYq5qkISzQnFPFXaoXVA1QepylbCcUfxRVUBqT9V9T8ad6qwSOdN76VjV5X9RioIjlYt3EmttW5fpL2zqhxHa5TmkypgURUpivdqD6L3j1YvG63oWO1BFA80f/R+WoM0Zsl4pUjaz0creFZzS2twfs2qqsbups3NzS4mqK+0TyY890ePHu3aaG+g6xkdh6oAP1f73L59+7o2ul+geK+uLzT3Bw4c6NpordJx6PqS8Dqg6wHFDs0jXcupPwnvyTRu6zAfl9EqwAlfk2j8KWbpvpdeV90b0PvpnpTmk8Z+leqrFEtVBdU5WgfVfkVxQ/fxNGd0faJ+V+uSxreqIEj8i5UkSZIkLeSDlSRJkiQt5IOVJEmSJC3kg5UkSZIkLbRS8YpLly7lySeffFYbJX9RImrCCY+UlHnixImu7ezZs10bJahSAl/CyXWUnEzFHigpk5LbqqRmOjYl0lESHyXmVec4moxKfafXjSY+JpyMPk/AXkchgM3NzW78aT6qJFE633PnznVtBw8e7Noo3ikxuUrepH7S2njssce6NopjKmhRJehT4ijNMfWd5pkKcSTJoUOHujaKT0oKpvVP80UFAxLu5zoKrFAf5uexStLv6L5GBRIuXrzYtdG+VhVXoERv6ufJkye7NkoGHy3Ik/BeOb9eJVxUhq5Z1fjSa6lPo4UNKOaq8aXjXCtFguZ76Ghhquq1tF+NFnGhuasS1WlOKEZoP6b5oCT3qmjHaDET2qtoj672OrrPomsenc9o8Qk6RlLfD67bxsZGt2fQOVTnNXqtoT2RXkdzXN1vUZ8oxuiaTe+lvbPa/2i9UZEhuubTWquKvtF6o72NClqM7rPVPTvtR9X+QfyLlSRJkiQt5IOVJEmSJC3kg5UkSZIkLeSDlSRJkiQttFLxiqRPAKNfOKZEuGQ80Z3aKOmffjGdfoW96tNoci0lH1ICX1WcgY5NSaKUxEeFEqqkQno/JSrSWFKCKp1jlexHyZjz86ZjXA3zmKUkeSqOknASM40prQNKPL311lu7tn379uGx6ZfYKUH19a9/fdf26KOPdm00/tUaoFimcSOUFF0l1FOfKNmZEkdpfOl8qmNT+7zYQZW4fLWNnn/C+xWhWKKE4WptECo2QXsGrQNKOKaYq/Y/KhpA+xLtk/SZ1J+EE8JpLmiPp9imfbYqQkDvX0dRoLnWWrff03nR2CW8Z9B1czTJngo5VUVPKMme1hAlyVNRHdozqr2T1gHt+7QuRwsgJDwXtFfS+qU5o7Gka2XCRUhWKWyyW6hIEI1pVXyD9geK49H7HtqPaeyS+h57jtYG3VPSvXTVbzo2xR2ND8VNVYztwQcf7NpojKiftFZpbqp1SetllaJA69+RJUmSJOlFzgcrSZIkSVrIBytJkiRJWsgHK0mSJElaaKUMwtbaUNLhKomJN998c9dGiaOUVEmvq34dmZLRKCGUPpOS4yiRrUocpWRWSiqk5DpKKpwn2D9XP+kzKRmT+kOJglWiIfVp/pnrKAQwTVM395RMX50XzSnN/cWLF7s2SvqlOKRiIgn388iRI13bM88807Xde++9XRvNe5XQTWuYklEpkZVeR7FU9YneT+ua4onOp0pQpfGdFzZYR8GVjY2NLnYo5lbpG40LJd7T2FNhlmPHjuFxKBZp7uh8KJGf1lUVS3SOVBiGChvQZ1aFFmgNU/GK0WIcFMdVwjyN73yPX0cxi2mauj7T9bW6PlOyOV37qLADrQOKpcceewyPTfNJ6HwoPqnYy8GDB/EzqdgWjdHhw4e7Nop3Gp+E72votXQfQDFH65f2iYSLxVRr+Gra2Njo7j9pjukeNeG4oTimaySNyejnJXxfR3vQaAGYVQrNjBZxoHigve6hhx7C91PcUTEPGgs6DhWpqe5/6P00ZxX/YiVJkiRJC/lgJUmSJEkL+WAlSZIkSQv5YCVJkiRJCy3++etVEmUp6W002ZwSVCn5nBLZEk7epAQ1KmJACXOUmFclk9NnUnItjcVoMngynhBJx6HiINRHGsfq2PNE2HUkVW9sbHTxRGNaJYvTWNMY0K/OU+Ip/bL7Aw88gMemuKG+U5IpJc6TVX55nMaIxoISoKsEcYpFSsKltUpjSWuQEswTTt6eH2cdMXvp0qVuXGmvq/pGMUvjTAnk1Ebvff/734/HpsRomifapymJma4Z1VqlY9M+TQUtCCWyJzy+o+uF5ozivUqqHplbipXdtrm52cUOFZ+o9iUaFxo/KkBB+yS1VWNKexjFDRUZor3uxIkTXVsVS3SOVDiIxodioRpfStyn/Y/6Sdd2ug7ROk+4SA6N29V26dKlrugW9auaO7rWUBtdn8+fP9+10X1mtcdT3NA+QvcltD9QH6vCGXRsum7QOdJ+Tq9Lxou+0d5L65/6TdeMhON7Ff7FSpIkSZIW8sFKkiRJkhbywUqSJEmSFvLBSpIkSZIWWql4xebmZpfURcnvVbIfJWVSst/or4RTsmSV7DeaxEfHpmRnSoSrkqIpEY4SDSmxj371m4pKJNx3SnCl/owWIah+hXzkl9gpGXG3TdPUxSMlLFZFTyjhmV67pKhElVRNa4NinhK6R61y3pQkSmudCqFQMnjCewL16dy5c13b6Jo+dOgQHpvOcT6+VUGa3bRnz54u2Zz2C0oYrl5LKCmd9k+KOUq0TnhMaR85c+ZM10Z7Ku371XnTcSiWqJjJsWPHurYqsXm0OAj1hz5zdO+oXjuPUdqLd9uePXswYXyO1nvCMTZacGq0SBDFe8JzX+1Xc2984xu7NoqvqhBAVQxqjvZZKqZRJd3TuI0WQKA2em9VVIbW4DpidK611u131Nd5gYsr3z9H+ydd8ykeaEyo8Ef1fooR2qepP9Tvav+jvY6OTX1f5XmB9lmaH9oraSxHi40kPB6rFFzxL1aSJEmStJAPVpIkSZK0kA9WkiRJkrSQD1aSJEmStJAPVpIkSZK00EpVAadp6qpuUcWYqvLPaNUmqj5GFWeoOl5VrWr0OFQlhKqjjFaLq9qpahFVu6JjV5WEqBIKHYcqs4xWCKoq/1BllnlFryoudlNrrRsDqkJTVYehmKX3V+MyV1WtJDReVHGK5pNiZF5prnpdwlWxqEoP9ZHeW1WBouPTa0erqdH4VpWVaG7nMU9VBnfbNE3d/kJVvWhtJ+OV/ajCJx2H9oGqAirtqVTBarSq6qOPPtq1UbwnXPWN+kPzTjFSjS+NG60tukZQRS6Ksaq6G53PfO9dRyXLzc3Nbl+kc60qjRHaW+gaSa8bqVJ7GVV9o/ikuKO4oeqIVSxRZdPRcaPzrqpJ0niMrg3aE0Yr3iZ8vaRrydXWWuv6QXtV1dfReya6/6Nxpv2rihu6z6VYqvaRkddV80lVf+nYtJ+vch0bHSOan9E4riro0tyusnf5FytJkiRJWsgHK0mSJElayAcrSZIkSVrIBytJkiRJWmilagI33HBDV5CAkklXSUykJDxKEhtN0K+OTQndlOBGBTEouZg+r0oapqQ5SqSl86b3VkmFdO40RpQASKg/VTI/jcc8aXUdhQAIzXuVmEjzREnDFDeUFE1zREnu1XFo7igBk86HEpirY48m4xM6NiVKJzwelBhNyaTUH5qHKoGXkmbna6squrGbpmnq1v0qyeKj8UCJ9zTvo3OU8HrZv38/vnaOYuTo0aNdGxWPSHivpD2Z4oaOTUnaCa/Bai5e6OuqPZrOZ7421hGzrbVuX6UiDlR8J+EYoza6htD5UhGC6vo8ulfSHkTXvXPnznVtVYI+xQP1k9ro+kLXtmS80MX58+eHXrdKEQLaU0aLPe2maZq6ftA5VPcttD/Q3BOaJ4q5qlAP9ZPaqHAGxdIqRegoHqjv9Dq6PlT7OV2f6HxG79HovKv1QuNO663iX6wkSZIkaSEfrCRJkiRpIR+sJEmSJGkhH6wkSZIkaaG2SqJra+1skgd3rzu6zh2fpunA1TygMauFjFm92BizejG6qnFrzGoHYMyu9GAlSZIkSer5VUBJkiRJWsgHK0mSJElayAcrSZIkSVrIB6ttrbXDrbUfb619qLX2m621f9Zae21r7URr7T27dMyvbq2dba29e/ufr92N4+j6tI6Y3T7uf7N9vPe21n50t46j68+a9tnvuWKP/Y+ttcd34zi6Pq0pZo+11n6htfZrrbVfb639wd04jq5fa4rb4621f7kds+9srd21G8e51t2w7g5cC1prLck/SvLD0zR9xXbb/UkOJTm5y4f/iWmavnGXj6HrzLpitrV2X5I/l+R3TdN0sbV2cLeOpevLumJ2mqZvvaIP35TkTbt1LF1f1nhv8BeT/OQ0Tf9Ha+0zk/yzJCd28Xi6jqwxbv+3JP/nNE0/3Fr7oiTfmeRP7+Lxrkn+xWrL703yqWma/u7lhmma3j1N0y9d+aLtJ/1faq29a/uf37ndfqS19ovb/0X0Pa2139Na29Na+6Htf/+N1tq3Rto564rZP5vkb0/TdHH7mGd28Rx1fbkW9tk/meTHdvzMdL1aV8xOSW7d/t+3JXl0l85P16d1xe1nJvmX2//7F5L8kV06v2uaf7Ha8oYk/37gdWeS/P5pmj6+/V/ufyzJ5yT5yiTvmKbp7a21PUleleT+JEenaXpDkrTWbi8+84+11j4/yX9M8q3TNO32X8h0fVhXzL52+//75SR7kvzlaZr++cJz0UvDOvfZtNaOJ7knyc8vOQm9pKwrZv9ykn+x/RfWm5J88cLz0EvLuuL2PyT5Y0m+N8mXJrmltbZvmqbzC8/nRcUHq9XcmOT7tv+keinbN5lJ/l2Sf9BauzHJz07T9O7W2oeTvLq19reS/NMk/wI+7+eS/Ng0TZ9orX1dkh9O8kW7fRJ6SdnpmL0hyX1JvjDJXUl+qbX2hmmaHt/Vs9BLyU7H7GVfkeSnp2m6tHtd10vUTsfsn0zyQ9M0/fXW2n+R5Ee299nNXT8TvZTsdNx++/bnfXWSX0zySJJP7+4pXHv8KuCW9yb57QOv+9Ykp5O8MVtP9S9LkmmafjHJ52criH6ktfZV21+VemOSdyb5hiQ/MP+waZrOT9P0ie1//f7BPkjJmmI2ycNJ/u9pmj41TdNHkrw/Ww9a0vNZV8xe9hXxa4Bazbpi9muS/OT2Z/xKklck2b/kRPSSsq572kenafqyaZrelOQvbLc9sfhsXmR8sNry80le3lr7s5cbWmuf21r7gtnrbktyavu/Gv3pbH0V6vJXTM5M0/T9Sf5+kje31vYn2Zim6WeS/KUkb54ftLV25Ip//ZIk79vBc9L1bS0xm+Rns/X97Wy//rVJPryTJ6br1rpiNq211yW5I8mv7PA56fq2rph9KMnv2/6Mz8jWg9XZHT0zXc/WdU+7v7V2+bnizyX5Bzt8Xi8KfhUwyTRNU2vtS5P8jdba25J8PMkDSb5l9tK/k+RnWmt/PFuJeR/dbv/CJG9prX0qydNJvirJ0SQ/OAuyuW9urX1Jtv5UeiHJV+/QKek6t8aYfUeSP9Ba+81sfXXgLS+170/rhVljzCZbX6368Wmapp05G70UrDFm/8ck379dIGBK8tXGrkatMW6/MMl3ttambH0V8Bt26JReVJprVZIkSZKW8auAkiRJkrSQD1aSJEmStJAPVpIkSZK0kA9WkiRJkrSQD1aSJEmStJAPVpIkSZK0kA9WkiRJkrTQ/w9PaOdYZ7tatAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x504 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "### plot weights vs the pixel position (works only for a single layer)\n",
    "### code works only SoftMax regression (!)\n",
    "\n",
    "from jax. tree_util import tree_flatten # jax params are stored as nested tuples; use this to manipulate tuples\n",
    "\n",
    "# extract weights and biases using tree_flatten\n",
    "params = get_params(opt_state)\n",
    "weights, biases  = tree_flatten(params)[0]\n",
    "\n",
    "# print the weights (biases are not so interesting)\n",
    "plt.figure(figsize=(15, 7)) # figure size \n",
    "\n",
    "scale = np.abs(weights).max() # define overall scale\n",
    "\n",
    "for i in range(10): # loop over the number of weights\n",
    "    \n",
    "    plot = plt.subplot(2, 5, i + 1)\n",
    "    plot.imshow(weights[:,i].reshape(28, 28), interpolation='nearest', cmap=plt.cm.Greys, vmin=-scale, vmax=scale)\n",
    "    plot.set_xticks(())\n",
    "    plot.set_yticks(())\n",
    "    plot.set_xlabel('Class %i' % i)\n",
    "    \n",
    "plt.suptitle('classification weights vector $w_j$ for digit class $j$')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DNN model\n",
    "\n",
    "We now want to train a DNN. To do that, all we need to do is add more layers to the model in the `stax.serial()` function, and re-run the training loop. Build a fully-connected DNN model with layer sizes (784, 512, 256, 10), where 784 corresponds to the input layer, and 10 -- to the output layer; i.e. we have two hidden layers. Use a `Relu` nonlinearity after each layer, except for the output where you need the `LogSoftMax`. \n",
    "\n",
    "\n",
    "### CNN model\n",
    "\n",
    "The CNN model is basically the same as the DNN model, except is also uses convolutional layers. Convolutional layers know about the dimensionality of the input. In our case, the data is two-domensional, and thus needs to be reshaped accordingly. Suppose the data had color: in color images each pixel has three values in between [0,255] -- one for each of the red, green, and blue channels. Hence, the dataset is a four-dimensiona array with shape `(N_points, N_Channels, Height, Width)`. For black and white images, we just set `N_Channels=1`.\n",
    "\n",
    "C.1 reshape the `train_images` and `test_images` to 4-dimensional array. We use the convention `dim_numbers='NCHW'`: (N data points, Channels, Height, Width). \n",
    "\n",
    "C.2 use `GeneralConv(dim_numbers, output_channels, filter_size, strides)` layers to add conv layers to the neural net, followed by `Relu` nonlinearities. Add two layers with:\n",
    "* `output_channels=16, filter_size=(4,4), strides=(4,4)`\n",
    "* `output_channels=32, filter_size=(3,3), strides=(1,1)`\n",
    "\n",
    "C.3. Next, we want to attach two dense layers. To be able to do that, we take the output of the last conv layer which has the shape `(N, C, H, W)`, and flatten it to a 1-dimensional array of size `(N, C*H*W)`. Then, we can stack the dense layers, followed by `Relu` nonlinearities each:\n",
    "* `256` hidden neurals\n",
    "* `10` output neurons, corresponding to the 10 degit categories.\n",
    "\n",
    "C.4. Finally, we add the `LogSoftMax` layer. \n",
    "\n",
    "C.5. Play with the output of the CNN on the toy dataset to convince yourself you have implemented everything properly. \n",
    "\n",
    "\n",
    "The rest of the code we constructed above can be applied without further modification. Why is that so? Do you appreciated now the usefulness of Deep ML packages?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "output shape of the model is (-1, 10).\n",
      "\n",
      "actual output shape is: (3, 10)\n",
      "log(softmax) values: [-2.264744  -2.4081538 -2.2366066 -2.302778  -2.4543526 -2.3551548\n",
      " -2.2510576 -2.1797056 -2.2681572 -2.336163 ]\n",
      "conservation of probability [0.9999999 1.        1.0000001]\n"
     ]
    }
   ],
   "source": [
    "### Convolutional Neural network\n",
    "\n",
    "from jax.experimental.stax import GeneralConv, Flatten # neural network layers\n",
    "\n",
    "\n",
    "# cast data into 2D image format\n",
    "train_images = train_images.reshape(-1,1,28,28) # -1: number of data points, 1: input channels, (28,28) = (height, width) dimensions of image\n",
    "test_images = test_images.reshape(-1,1,28,28)\n",
    "\n",
    "# conv net convention\n",
    "dim_nums=('NCHW', 'OIHW', 'NCHW') # default for (input, filters, output)\n",
    "\n",
    "# define functions which initialize the parameters and evaluate the model\n",
    "initialize_params, predict = stax.serial(    \n",
    "                                            ### convolutional NN (CNN)\n",
    "                                            GeneralConv(dim_nums, 16, (4,4), strides=(4,4) ), # 16 output channels, (4,4) filter\n",
    "                                            Relu,\n",
    "                                            GeneralConv(dim_nums, 32, (3,3), strides=(1,1) ), # 32 output channels, (3,3) filter\n",
    "                                            Relu,\n",
    "                                            Flatten, # flatten output\n",
    "                                            Dense(256), # 256 hidden neurons\n",
    "                                            Relu,\n",
    "                                            Dense(10), # 10 output neurons\n",
    "                                            LogSoftmax # NB: computes the log-probability\n",
    "                                        )\n",
    "\n",
    "# initialize the model parameters\n",
    "output_shape, inital_params = initialize_params(rng, (-1, 1, 28, 28)) # conv layer, 1 input channel, 28x28 pixes in each image\n",
    "\n",
    "print('\\noutput shape of the model is {}.\\n'.format(output_shape))\n",
    "\n",
    "# check how network works on 3 examples\n",
    "predictions = predict(inital_params, test_images[0:3])\n",
    "\n",
    "# print shape of output\n",
    "print(\"actual output shape is:\", predictions.shape)\n",
    "\n",
    "# check if probability is conserved\n",
    "print('log(softmax) values:', predictions[0])\n",
    "print('conservation of probability', np.sum(jnp.exp(predictions), axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further questions\n",
    "\n",
    "* do you see an advantage of JAX when compared to the other ML packages: TensorFlow or PyTorch? \n",
    "* compare the weights plots for the SoftMax regression in the presence of an L2 regularizer added to the cost function. You can use a reguarization strength of `lmbda=0.001`; what happens to the score if you increase/decrease this number? What if you implement an L1 regularizatin instead?\n",
    "* try and look for the patterns of the imprinted digits in the output weights of the DNN and CNN layers. Explain your findings, and compare them to the SoftMax regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "RL_class",
   "language": "python",
   "name": "rl_class"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "latex_metadata": {
   "affiliation": "Faculty of Physics, Sofia University, 5 James Bourchier Blvd., 1164 Sofia, Bulgaria",
   "author": "Marin Bukov",
   "title": "Reinforcement Learning Course: WiSe 2020/21"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}