Soft_Actor_Critic_mine.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r9HxtwHbk6Pc"
      },
      "source": [
        "# Soft Actor Critic Agent(115 Points)\n",
        "This is my implementation of SAC in off-policy and offline RL for RL course of Sharif University of Technology (Winter 2023)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1PJUxRV4MIod"
      },
      "source": [
        "In this notebook, we are going to implement **Soft Actor Critic (SAC)**\n",
        "on the **CartPole** environment in online and offline settings. In this framework, the actor aims to maximize the expected reward while also maximizing **entropy**. That is, to succeed at the task while acting as randomly as possible. This method seeks a high entropy in the policy to explicitly encourage exploration. For the offline setting, you are going to make SAC conservative using CQL method.\n",
        "\n",
        "* SAC is an off-policy algorithm.\n",
        "* The version of SAC implemented here can only be used for environments with discrete action spaces.\n",
        "* An alternate version of SAC, which slightly changes the policy update  rule, can be implemented to handle continuous action spaces.\n",
        "* Complete the **TODO** parts in the code accordingly.\n",
        "* Remember to answer the conceptual questions.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "UC-BecdPmdb3"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch.distributions import Normal\n",
        "import torch.optim as optim\n",
        "import numpy as np\n",
        "import random\n",
        "import gym\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "seed = 42\n",
        "random.seed(seed)\n",
        "np.random.seed(seed)\n",
        "torch.manual_seed(seed)\n",
        "torch.cuda.manual_seed(seed)\n",
        "torch.backends.cudnn.deterministic = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_whWTLZejkm3"
      },
      "source": [
        "## Network Structure (8 points)\n",
        "For constructing SAC agent, we use objects of feedforward neural networks with 3 layers. Complete the code below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4pxOyt0xh4nN",
        "outputId": "9880787a-94ab-4d52-9ca7-3d35aa30560b"
      },
      "outputs": [],
      "source": [
        "class Network(torch.nn.Module):\n",
        "\n",
        "    def __init__(self, input_dimension, output_dimension, output_activation=torch.nn.Identity()):\n",
        "        super(Network, self).__init__()\n",
        "        ##########################################################\n",
        "        # TODO (4 points):\n",
        "        # Define your network layers.\n",
        "        ##########################################################\n",
        "        self.input_dimension = input_dimension\n",
        "        self.output_dimension = output_dimension\n",
        "        self.output_activation = output_activation\n",
        "\n",
        "        self.fc1 = nn.Linear(self.input_dimension, self.input_dimension * 4)\n",
        "        self.fc2 = nn.Linear(self.input_dimension * 4, self.input_dimension * 32)\n",
        "        self.fc3 = nn.Linear(self.input_dimension * 32, self.output_dimension)\n",
        "        ##########################################################\n",
        "\n",
        "    def forward(self, inpt):\n",
        "        output = None\n",
        "        ##########################################################\n",
        "        # TODO (4 points):\n",
        "        # Use relu and the output activation functions to calculate the output\n",
        "        ##########################################################\n",
        "        x = torch.relu(self.fc1(inpt))\n",
        "        x = torch.relu(self.fc2(x))\n",
        "        output = self.output_activation(self.fc3(x))\n",
        "        return output\n",
        "        ##########################################################"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DToSGdmDka1u"
      },
      "source": [
        "## Replay Buffer\n",
        "\n",
        "A SAC agent needs a replay buffer, from which previously visited states can be sampled. You can use the implemented code below. You are going to use the replay buffer of a pretrained online agent to train the offline model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "UC7jTwJXh8wl"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "\n",
        "class ReplayBuffer:\n",
        "\n",
        "    def __init__(self, environment, capacity=500000): #maybe 10^6 is better\n",
        "        transition_type_str = self.get_transition_type_str(environment)\n",
        "        self.buffer = np.zeros(capacity, dtype=transition_type_str)\n",
        "        self.weights = np.zeros(capacity)\n",
        "        self.head_idx = 0\n",
        "        self.count = 0\n",
        "        self.capacity = capacity\n",
        "        self.max_weight = 10**-2\n",
        "        self.delta = 10**-4\n",
        "        self.indices = None\n",
        "        self.mirror_index = np.random.permutation(range(self.buffer.shape[0]))\n",
        "\n",
        "    def get_transition_type_str(self, environment):\n",
        "        state_dim = environment.observation_space.shape[0]\n",
        "        state_dim_str = '' if state_dim == () else str(state_dim)\n",
        "        state_type_str = environment.observation_space.sample().dtype.name\n",
        "        action_dim = environment.action_space.shape\n",
        "        action_dim_str = '' if action_dim == () else str(action_dim)\n",
        "        action_type_str = environment.action_space.sample().__class__.__name__\n",
        "\n",
        "        # type str for transition = 'state type, action type, reward type, state type'\n",
        "        transition_type_str = '{0}{1}, {2}{3}, float32, {0}{1}, bool'.format(state_dim_str, state_type_str,\n",
        "                                                                             action_dim_str, action_type_str)\n",
        "\n",
        "        return transition_type_str\n",
        "\n",
        "    def add_transition(self, transition):\n",
        "        self.buffer[self.head_idx] = transition\n",
        "        self.weights[self.head_idx] = self.max_weight\n",
        "\n",
        "        self.head_idx = (self.head_idx + 1) % self.capacity\n",
        "        self.count = min(self.count + 1, self.capacity)\n",
        "\n",
        "    def sample_minibatch(self, size=100, batch_deterministic_start=None):\n",
        "        set_weights = self.weights[:self.count] + self.delta\n",
        "        probabilities = set_weights / sum(set_weights)\n",
        "        if batch_deterministic_start is None:\n",
        "            self.indices = np.random.choice(range(self.count), size, p=probabilities, replace=False)\n",
        "        else:\n",
        "            self.indices = self.mirror_index[batch_deterministic_start:batch_deterministic_start+size]\n",
        "        return self.buffer[self.indices]\n",
        "\n",
        "    def update_weights(self, prediction_errors):\n",
        "        max_error = max(prediction_errors)\n",
        "        self.max_weight = max(self.max_weight, max_error)\n",
        "        self.weights[self.indices] = prediction_errors\n",
        "\n",
        "    def get_size(self):\n",
        "        return self.count"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "fB2m89BjlNXY"
      },
      "source": [
        "## Questions (18 points)\n",
        "\n",
        "❓ We know that standard RL maximizes the expected sum of rewards. What is the objective function of SAC algorithm? Compare it to the standard RL loss.\n",
        "\\\n",
        "✅ expected sum of rewards plus relative importance multiplied by entropy of current policy.\n",
        "\n",
        "\n",
        "❓ Write down the actor cost function.\n",
        "\\\n",
        "✅ <img src=\"actor_cost_function.png\" alt=\"alt text\" width=\"500\"/>\n",
        "\n",
        "❓ Write down the critic cost function.\n",
        "\\\n",
        "✅ <img src=\"critic_cost_function.png\" alt=\"alt text\" width=\"800\"/>\n",
        "\n",
        "❓ Elaborate on the reason why most implementations of SAC use two critics (one local and one target).\n",
        "\\\n",
        "✅ This is the problem that arrises in Q-Learning using value function approximation (VFA). Because of Non-stationary targets, Q-Learning with VFA can diverge and the solution to this is to have two Q networks and update the target norwotk weights using Polyak Averaging.\n",
        "\n",
        "❓ What is the difference between training samples in offline and online settings?\n",
        "\\\n",
        "✅ In online setting our samples are more fresh (the ones we have seen recenelty and put in replay buffer) but in offline setting the samples are the samples of another policy (agent or exepert and in this exercise another agent) and we are trying to learning from them by stiching.\n",
        "\n",
        "❓ How does adding CQL on top of SAC change the objective function?\n",
        "\\\n",
        "✅ <img src=\"cql_objective.png\" alt=\"alt text\" width=\"800\"/>\n",
        "\\\n",
        "The min max and first 3 terms are the the difference (the third term is optional).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wOrt_EmfFFD"
      },
      "source": [
        "## SAC Agent (50 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ScEk94Ubb01W"
      },
      "source": [
        "Now complete the following class. You can use the auxiliary methods provided in the class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "A8bEgUc2hmet"
      },
      "outputs": [],
      "source": [
        "class SACAgent:\n",
        "\n",
        "    ALPHA_INITIAL = 1.\n",
        "    REPLAY_BUFFER_BATCH_SIZE = 100\n",
        "    DISCOUNT_RATE = 0.99\n",
        "    LEARNING_RATE = 10 ** -4\n",
        "    SOFT_UPDATE_INTERPOLATION_FACTOR = 0.01\n",
        "    TRADEOFF_FACTOR = 1 # trade-off factor in the CQL\n",
        "\n",
        "    def __init__(self, environment, replay_buffer=None, offline=False, use_cql=False):\n",
        "\n",
        "        #deleted assert use_cql and not offline because it was erroneous\n",
        "        #deleted assert offline and replay_buffer is None because it was erroneous\n",
        "\n",
        "        self.environment = environment\n",
        "        self.state_dim = self.environment.observation_space.shape[0]\n",
        "        self.action_dim = self.environment.action_space.n\n",
        "\n",
        "        \n",
        "        self.replay_buffer = ReplayBuffer(self.environment) if replay_buffer is None else replay_buffer\n",
        "        self.offline = offline\n",
        "        self.use_cql = use_cql\n",
        "\n",
        "        ##########################################################\n",
        "        # TODO (6 points):\n",
        "        # Define critics usig your implemented feed forward netwrok.\n",
        "        ##########################################################\n",
        "        self.critic_local = Network(input_dimension=self.state_dim, output_dimension=self.action_dim)\n",
        "        self.critic_local2 = Network(input_dimension=self.state_dim, output_dimension=self.action_dim)\n",
        "        self.critic_optimiser = optim.Adam(self.critic_local.parameters(), lr=self.LEARNING_RATE)\n",
        "        self.critic_optimiser2 = optim.Adam(self.critic_local2.parameters(), lr=self.LEARNING_RATE)\n",
        "        self.critic_target = Network(input_dimension=self.state_dim, output_dimension=self.action_dim)\n",
        "        self.critic_target2 = Network(input_dimension=self.state_dim, output_dimension=self.action_dim)\n",
        "        ##########################################################\n",
        "\n",
        "        self.soft_update_target_networks(tau=1.)\n",
        "\n",
        "        ##########################################################\n",
        "        # TODO (2 points):\n",
        "        # Define the actor using your implemented feed forward netwrok.\n",
        "        # Define the actor optimizer using torch.Adam.\n",
        "        ##########################################################\n",
        "        self.actor_local = Network(input_dimension=self.state_dim, output_dimension=self.action_dim, output_activation=torch.nn.Softmax(dim=1))\n",
        "        self.actor_optimiser  = optim.Adam(self.actor_local.parameters(), lr=self.LEARNING_RATE)\n",
        "        ##########################################################\n",
        "        self.target_entropy = 0.98 * -np.log(1 / self.environment.action_space.n)\n",
        "        self.log_alpha = torch.tensor(np.log(self.ALPHA_INITIAL), requires_grad=True)\n",
        "        self.alpha = self.log_alpha\n",
        "        self.alpha_optimiser = torch.optim.Adam([self.log_alpha], lr=self.LEARNING_RATE)\n",
        "\n",
        "    def get_next_action(self, state, evaluation_episode=False):\n",
        "        if evaluation_episode:\n",
        "            discrete_action = self.get_action_deterministically(state)\n",
        "        else:\n",
        "            discrete_action = self.get_action_nondeterministically(state)\n",
        "        return discrete_action\n",
        "\n",
        "    def get_action_nondeterministically(self, state):\n",
        "        action_probabilities = self.get_action_probabilities(state)\n",
        "        discrete_action = np.random.choice(range(self.action_dim), p=action_probabilities)\n",
        "        return discrete_action\n",
        "\n",
        "    def get_action_deterministically(self, state):\n",
        "        action_probabilities = self.get_action_probabilities(state)\n",
        "        discrete_action = np.argmax(action_probabilities)\n",
        "        return discrete_action\n",
        "\n",
        "    def critic_loss(self, states_tensor, actions_tensor, rewards_tensor,\n",
        "                    next_states_tensor, done_tensor):\n",
        "        ##########################################################\n",
        "        # TODO (12 points):\n",
        "        # You are going to calculate critic losses in this method.\n",
        "        # Also you should implement the CQL loss if the corresponding\n",
        "        # flag is set.\n",
        "        ##########################################################\n",
        "        critic_loss, critic2_loss = 0, 0\n",
        "        with torch.no_grad():\n",
        "            action_probabilities, log_action_probabilities = self.get_action_info(next_states_tensor)\n",
        "            next_q_values_target = self.critic_target.forward(next_states_tensor)\n",
        "            next_q_values_target2 = self.critic_target2.forward(next_states_tensor)\n",
        "            soft_state_values = (action_probabilities * (torch.min(next_q_values_target, next_q_values_target2) - self.alpha * log_action_probabilities)).sum(dim=1)\n",
        "\n",
        "            next_q_values = rewards_tensor + ~done_tensor * self.DISCOUNT_RATE * soft_state_values\n",
        "\n",
        "        soft_q_values = self.critic_local(states_tensor).gather(1, actions_tensor.unsqueeze(-1)).squeeze(-1)\n",
        "        soft_q_values2 = self.critic_local2(states_tensor).gather(1, actions_tensor.unsqueeze(-1)).squeeze(-1)\n",
        "        critic_square_error = torch.nn.MSELoss(reduction=\"none\")(soft_q_values, next_q_values)\n",
        "        critic2_square_error = torch.nn.MSELoss(reduction=\"none\")(soft_q_values2, next_q_values)\n",
        "        weight_update = [min(l1.item(), l2.item()) for l1, l2 in zip(critic_square_error, critic2_square_error)]\n",
        "        if not self.offline:\n",
        "            self.replay_buffer.update_weights(weight_update)\n",
        "        critic_loss = critic_square_error.mean()\n",
        "        critic2_loss = critic2_square_error.mean()\n",
        "        \n",
        "        #CQL\n",
        "        if self.offline and self.use_cql:\n",
        "            cql_temp = 1.0\n",
        "            cql_clip_diff_min = -np.inf\n",
        "            cql_clip_diff_max = np.inf\n",
        "            action_dim = action_probabilities.shape\n",
        "            cql_random_actions = torch.FloatTensor(action_dim).uniform_(-1, 1)\n",
        "            cql_current_actions, cql_current_log_pis = self.get_action_info(states_tensor)\n",
        "            cql_next_actions, cql_next_log_pis = self.get_action_info(next_states_tensor)\n",
        "            cql_current_actions, cql_current_log_pis = cql_current_actions.detach(), cql_current_log_pis.detach()\n",
        "            cql_next_actions, cql_next_log_pis = cql_next_actions.detach(), cql_next_log_pis.detach()\n",
        "            cql_q1_rand = self.critic_target(states_tensor)\n",
        "            cql_q2_rand = self.critic_target2(states_tensor)\n",
        "            cql_q1_current_actions = self.critic_target(states_tensor)\n",
        "            cql_q2_current_actions = self.critic_target2(states_tensor)\n",
        "            cql_q1_next_actions = self.critic_target(states_tensor)\n",
        "            cql_q2_next_actions = self.critic_target2(states_tensor)\n",
        "\n",
        "            cql_cat_q1 = torch.cat(\n",
        "                [cql_q1_rand, torch.unsqueeze(next_q_values, 1), cql_q1_next_actions, cql_q1_current_actions], dim=1\n",
        "            )\n",
        "            cql_cat_q2 = torch.cat(\n",
        "                [cql_q2_rand, torch.unsqueeze(next_q_values, 1), cql_q2_next_actions, cql_q2_current_actions], dim=1\n",
        "            )\n",
        "            random_density = np.log(0.5 ** action_dim[0])\n",
        "            cql_cat_q1 = torch.cat(\n",
        "                [cql_q1_rand - random_density,\n",
        "                    cql_q1_next_actions - cql_next_log_pis.detach(),\n",
        "                    cql_q1_current_actions - cql_current_log_pis.detach()],\n",
        "                dim=1\n",
        "            )\n",
        "            cql_cat_q2 = torch.cat(\n",
        "                [cql_q2_rand - random_density,\n",
        "                    cql_q2_next_actions - cql_next_log_pis.detach(),\n",
        "                    cql_q2_current_actions - cql_current_log_pis.detach()],\n",
        "                dim=1\n",
        "            )\n",
        "            cql_qf1_ood = torch.logsumexp(cql_cat_q1 / cql_temp, dim=1) * cql_temp\n",
        "            cql_qf2_ood = torch.logsumexp(cql_cat_q2 / cql_temp, dim=1) * cql_temp\n",
        "            \"\"\"Subtract the log likelihood of data\"\"\"\n",
        "            cql_qf1_diff = torch.clamp(\n",
        "                cql_qf1_ood - next_q_values,\n",
        "                cql_clip_diff_min,\n",
        "                cql_clip_diff_max,\n",
        "            ).mean()\n",
        "            cql_qf2_diff = torch.clamp(\n",
        "                cql_qf2_ood - next_q_values,\n",
        "                cql_clip_diff_min,\n",
        "                cql_clip_diff_max,\n",
        "            ).mean()\n",
        "            cql_min_qf1_loss = cql_qf1_diff * float(self.TRADEOFF_FACTOR)\n",
        "            cql_min_qf2_loss = cql_qf2_diff * float(self.TRADEOFF_FACTOR)\n",
        "            critic_loss += cql_min_qf1_loss\n",
        "            critic2_loss += cql_min_qf2_loss\n",
        "        return critic_loss, critic2_loss\n",
        "        ##########################################################\n",
        "\n",
        "    def actor_loss(self, states_tensor, actions_tensor):\n",
        "        ##########################################################\n",
        "        # TODO (8 points):\n",
        "        # Now implement the actor loss.\n",
        "        ##########################################################\n",
        "        actor_loss, log_action_probabilities = 0, 0\n",
        "        action_probabilities, log_action_probabilities = self.get_action_info(states_tensor)\n",
        "        q_values_local = self.critic_local(states_tensor)\n",
        "        q_values_local2 = self.critic_local2(states_tensor)\n",
        "        inside_term = self.alpha * log_action_probabilities - torch.min(q_values_local, q_values_local2)\n",
        "        actor_loss = (action_probabilities * inside_term).sum(dim=1).mean()\n",
        "        return actor_loss, log_action_probabilities\n",
        "        ##########################################################\n",
        "\n",
        "    def train_on_transition(self, state, discrete_action, next_state, reward, done):\n",
        "        transition = (state, discrete_action, reward, next_state, done)\n",
        "        self.train_networks(transition)\n",
        "\n",
        "    def train_networks(self, transition=None, batch_deterministic_start=None):\n",
        "        ##########################################################\n",
        "        # TODO (6 points):\n",
        "        # Set all the gradients stored in the optimizers to zero.\n",
        "        # Add the new transition to the replay buffer for online case.\n",
        "        ##########################################################\n",
        "        self.critic_optimiser.zero_grad()\n",
        "        self.critic_optimiser2.zero_grad()\n",
        "        self.actor_optimiser.zero_grad()\n",
        "        self.alpha_optimiser.zero_grad()\n",
        "\n",
        "        if not self.offline:\n",
        "            self.replay_buffer.add_transition(transition)\n",
        "\n",
        "\n",
        "        if self.replay_buffer.get_size() >= self.REPLAY_BUFFER_BATCH_SIZE:\n",
        "            minibatch = self.replay_buffer.sample_minibatch(self.REPLAY_BUFFER_BATCH_SIZE, batch_deterministic_start=batch_deterministic_start)\n",
        "            minibatch_separated = list(map(list, zip(*minibatch)))\n",
        "\n",
        "            states_tensor = torch.tensor(np.array(minibatch_separated[0]))\n",
        "            actions_tensor = torch.tensor(np.array(minibatch_separated[1]))\n",
        "            rewards_tensor = torch.tensor(np.array(minibatch_separated[2])).float()\n",
        "            next_states_tensor = torch.tensor(np.array(minibatch_separated[3]))\n",
        "            done_tensor = torch.tensor(np.array(minibatch_separated[4]))\n",
        "\n",
        "            ##########################################################\n",
        "            # TODO (16 points):\n",
        "            # Here, you should compute the gradients based on this loss, i.e. the gradients\n",
        "            # of the loss with respect to the Q-network parameters.\n",
        "            # Given a minibatch of 100 transitions from replay buffer,\n",
        "            # compute the critic loss and perform the backpropagation,\n",
        "            # then compute the actor loss and backpropagate the gradient on it.\n",
        "            # You also need to update \\alpha.\n",
        "            ##########################################################\n",
        "            critic_loss, critic2_loss = self.critic_loss(states_tensor, actions_tensor, rewards_tensor, next_states_tensor, done_tensor)\n",
        "            critic_loss.backward()\n",
        "            critic2_loss.backward()\n",
        "            self.critic_optimiser.step()\n",
        "            self.critic_optimiser2.step()\n",
        "            actor_loss, log_action_probabilities = self.actor_loss(states_tensor, actions_tensor)\n",
        "            actor_loss.backward()\n",
        "            self.actor_optimiser.step()\n",
        "            alpha_loss = self.temperature_loss(log_action_probabilities)\n",
        "            alpha_loss.backward()\n",
        "            self.alpha_optimiser.step()\n",
        "            self.alpha = self.log_alpha.exp()\n",
        "            self.soft_update_target_networks()\n",
        "            ##########################################################\n",
        "            self.soft_update_target_networks()\n",
        "\n",
        "    def temperature_loss(self, log_action_probabilities):\n",
        "        alpha_loss = -(self.log_alpha * (log_action_probabilities + self.target_entropy).detach()).mean()\n",
        "        return alpha_loss\n",
        "\n",
        "    def get_action_info(self, states_tensor):\n",
        "        action_probabilities = self.actor_local.forward(states_tensor)\n",
        "        z = action_probabilities == 0.0\n",
        "        z = z.float() * 1e-8\n",
        "        log_action_probabilities = torch.log(action_probabilities + z)\n",
        "        return action_probabilities, log_action_probabilities\n",
        "\n",
        "    def get_action_probabilities(self, state):\n",
        "        state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n",
        "        action_probabilities = self.actor_local.forward(state_tensor)\n",
        "        return action_probabilities.squeeze(0).detach().numpy()\n",
        "\n",
        "    def soft_update_target_networks(self, tau=SOFT_UPDATE_INTERPOLATION_FACTOR):\n",
        "        self.soft_update(self.critic_target, self.critic_local, tau)\n",
        "        self.soft_update(self.critic_target2, self.critic_local2, tau)\n",
        "\n",
        "    def soft_update(self, target_model, origin_model, tau):\n",
        "        for target_param, local_param in zip(target_model.parameters(), origin_model.parameters()):\n",
        "            target_param.data.copy_(tau * local_param.data + (1 - tau) * target_param.data)\n",
        "\n",
        "    def predict_q_values(self, state, action):\n",
        "        q_values = self.critic_local(state)\n",
        "        q_values2 = self.critic_local2(state)\n",
        "        return torch.min(q_values, q_values2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7Tx4pS_kdo-"
      },
      "source": [
        "## Online SAC training loop (10 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jUm9_qfAlvf_"
      },
      "source": [
        "Now evaluate your model in CartPole environment in the online setting. After each 4 episodes, you should evaluate your model on a separate test environment. Run your model 4 times separately and plot the mean and deviation of the evaluation curves.\n",
        "\n",
        "**NOTE:** Since you are going to use the replay buffer of this agent as the offline dataset, you may want to save it for later use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c6quXtBzZ6nZ",
        "outputId": "fc91b79a-bc5f-478a-d79f-71f05f022863"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/husseinjafarinia/venvs/ML_For_Bio/lib/python3.10/site-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/Users/husseinjafarinia/venvs/ML_For_Bio/lib/python3.10/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Episode: 1000/1000 "
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABL2ElEQVR4nO2deZhcZZXwf6eWruq90+nOnpAASVhDgMgqyCKCjsq4IzrCjH4MijrO+OmIzriNs7l+44yDguIysqnouCEuiAZFlgAhBEIWSCB7d2fpvfb3++Pe99at6qru6qSr63b3+T1PPV313nurTlVXvec9y3uOGGNQFEVRFIBQrQVQFEVRgoMqBUVRFMVDlYKiKIrioUpBURRF8VCloCiKonioUlAURVE8VCkoShlEZIeIvNy9/1ER+XqtZVKUaqNKQZm2iMi1IvKUiAyJyD4RuUlE2o7kuYwx/2KMedcEiwh4Cme7iAyIyC4RuavEOdeKiBGRt5Q4tkJEvi8iPSLSKyIbROTvRCRcDXmV6Y0qBWVaIiIfBP4d+BDQCpwDHAP8WkTqaimbHxG5BvgL4OXGmCZgDXBfiVOvAQ4C7yi6/jjgYWAncKoxphV4k/s8zVUUXZmmqFJQph0i0gJ8CnifMeZeY0zaGLMDeDOwFHi7e94nReR7IvIdEekXkadFZE2Z5/ykiHzXvb/UXbVfIyIvuiv0j/nODYnIR0TkORE54L5GexlxXwL80hjzHIAxZp8x5uai1z4GeBlwHXC5iMzzHf4U8KAx5u+MMXvd59hsjLnaGHN4XB+coqBKQZmenAfEgR/6B40xA8A9wGW+4dcCdwJtwE+A/xrH67wUWAlcCnxcRE50x98H/DnORL4AOAR8pcxzPAS8Q0Q+JCJryrh83gGsM8bcDWwC3uY79nLgB+OQWVFGRZWCMh3pAHqMMZkSx/a6xy1/MMbcY4zJAv8DnDaO1/mUMWbYGPMk8KTv2uuBjxljdhljksAngTeKSKT4CYwx38VRIpcDvwe6ROTvi057B3C7e/92Cl1Is933pCgTgioFZTrSA3SUmoSB+e5xyz7f/SEgXua6UhRf2+TePwb4kYgcFpHDOKv7LDC31JMYY24zxrwcx1q5HvgnEbkcQETOB5bhWDPgKIVTRWS1+/iA+54UZUJQpaBMR/4EJIHX+wdFpAl4JaUDuRPJTuCVxpg23y1ujNk92kVu7OP7wAbgFHf4GkCA9SKyDyeobMcBfgO8YeLfgjJTUaWgTDuMMb04Adj/FJErRCQqIkuB7wG7cNxE1eSrwD+7AWJEpFNErix1optq+mci0uwGqF8JnAw8LCJxnOD4dcBq3+19wNWuRfMJ4DwR+ZwNQIvI8SLy3SNNv1VmNqoUlGmJMeazwEeBzwN95NM2L3X9/NXkP3CC1r8SkX6cYPLZZc7tc+V8ETgMfBZ4tzHmDzjB6mHgO25W0j5jzD7gViACXOFmLZ2Lk1X1tIj0AncD64D+qrw7ZVoj2mRHURRFsailoCiKonioUlAURVE8VCkoiqIoHqoUFEVRFI9KN+kEko6ODrN06dJai6EoijKleOyxx3qMMZ2ljk1ppbB06VLWrVtXazEURVGmFCLyQrlj6j5SFEVRPFQpKIqiKB6qFBRFURSPKR1TKEU6nWbXrl0kEolaixJI4vE4ixYtIhqN1loURVECyLRTCrt27aK5uZmlS5ciIrUWJ1AYYzhw4AC7du1i2bJltRZHUZQAUjX3kYgsFpH7ReQZt83h37jj7SLyaxHZ6v6d5Y6LiHxZRLa5jcfPOJLXTSQSzJ49WxVCCUSE2bNnqxWlKEpZqhlTyAAfNMachNM0/QYROQn4CHCfMWY5Tl37j7jnvxJY7t6uA2460hdWhVAe/WwURRmNqrmP3CbitpF4v4hsAhYCVwIXuad9G/gd8Pfu+HeMU7b1IRFpE5H5thm5MjkkM1l+sn4Pbzxz0agKZPfhYTbv6+OSE/LNxDbt7eMXT+X/XSLCG89cxOL2hqrKfCTcv7mLJ144xOolbVy0Yg4/eHwXV65eQCxS2CL5J0/u4cLlHbQ11FX0vA9s7WZuS5wVc5sLxp/ceZicMayY28y3HtxBMp0d9XkuWNHJyQta+NaDO8hkDVefvYSegST3bBj5c7johDmsnNvMbQ+/wEAi34F09ZI2XrZiDt/843b6htMVya9MHJFwiLeetYTO5ljJ4zsPDvGDx3ZxpJWqV8xr5tWrFhyNiCWZlJiC2+DkdJya9nN9E/0+8i0KF+LUu7fscscKfgUich2OJcGSJUuqJ/RREA6HOfXUU8lkMixbtoz/+Z//oa2tbdLluOiii/j85z/PmjVrKr5m7ZYePvSDDZy8oJWTFrSUPe+2h17g6w9sZ/NnrvCUx1fu38bPNuzF6hJjIJPL8aHLTziq91ENPvHjp3nx4BAdTTE+fMVKPvyDDRwcTHH9y47zzunqT/D+O57g01eezDvOXTrmc/Yl0vzFNx6hoynGun94ecGxK7/yRwD+/Q2n8rlfbgagnM41Bh56/iDvumAZn73XObcxFuGJFw8VfL723G/+cQcXnzCHnzy5p+CzX9Aa56a3n8lnfr5p1NdTJh47zzfHI/zl+aXjd9996AW+tvb5I/6/vHrVgqmpFNwWiHcDHzDG9PlXn8YYIyLjUpPGmJuBmwHWrFkTyGYQ9fX1rF+/HoBrrrmGr3zlK3zsYx+r6mtmMhkikaP/dybcFezwGCvZVCZHKpsjmckRjzqr68NDac5Y0sYP33M+ACd9/F6S6dxRy1QN7PvrGUjy8PMHAQgX/Tp7h5zVdb9v9T0aP16/B4C6cPlf+a+e3s+shiiP/+NlZS2x//Oddew8OFTwuj0DSXoGkrxk6Sy+f/153vjz3QNc/v/W8pMn9/COc4/h01c6XTz/9Reb+OYfdtDd7/QT+vEN53Pa4raK3ody9AwmM5z8iV+SypT//u/pTXDM7AZ+/6GLJ1GysanqPgURieIohNuMMT90h/eLyHz3+Hygyx3fDSz2Xb7IHZvSnHvuueze7byN5557jiuuuIIzzzyTCy64gGeffZZsNsuyZcswxnD48GHC4TBr164F4MILL2Tr1q088sgjnHvuuZx++umcd955bN7srB6/9a1v8drXvpZLLrmESy+9lOHhYa666ipOPPFEXve61zE8PDxueTM550s82pfZOc/Rx/6Jq3c4TUt9PtU1EhLvvKCRTGdZ0BoHYO3WbgDmtBSa+X0JRykMp0ZXkJY7H3kRgOVFriM/9z3bxZql7aO65hrrwgymMgymnM82HBIODCQ5MJBidmOhjMd2NvH+S5azpL2Bv7tshTfe0Rgjlc2x48AgALObKnN/KRNDxF0YpLPlf0f7eoeZ1xKfLJEqpmqWgjjf+m8Am4wxX/Qd+glO0/F/c//+2Df+XhG5E6d1Ye/RxhM+9dOneWZP39E8xQhOWtDCJ15zckXnZrNZ7rvvPt75zncCcN111/HVr36V5cuX8/DDD/Oe97yH3/72t6xcuZJnnnmG7du3c8YZZ/DAAw9w9tlns3PnTpYvX05fXx8PPPAAkUiE3/zmN3z0ox/l7rvvBuDxxx9nw4YNtLe388UvfpGGhgY2bdrEhg0bOOOM8SdwZbLOJJ4a5csMkHUn+4FkxvOZ9iXSHNvZ6J0TDYdG/VHUkmQmx5nHNLOnN+GtpnNFvt2+YWdSHstqAmcl/7T7XUtlciTSWTbt7eP0JbMAOLajked7nAn6rKXtoz5XYyzCUDLLYNJ53SXtDRwYSHFwMMVLlo2c3N936XJuuPh4QqG8orFKYOv+AedxY2m/tlIdoiFnvZ3Oll8U7e1NcOYxsyZLpIqppvvofOAvgKdEZL079lEcZfA9EXkn8AJOY3KAe4BXAduAIeAvqyhbVRkeHmb16tXs3r2bE088kcsuu4yBgQEefPBB3vSmN3nnJZPOZHTBBRewdu1atm/fzo033sgtt9zCy172Ml7ykpcA0NvbyzXXXMPWrVsREdLpfNDwsssuo73dmWTWrl3L+9//fgBWrVrFqlWrxi27XdmnK7QUBooshVa/pRAWT8kECWMMyUyOFXOb+d2Wbs//WyyrtRSGKrAUDg/l/yfJTJYfr9/NjT98iif+8RW0NkQL/MZrlo4+ETTGIgwkMwwmM4QEFrbV0z2Q5OBQio7G0it+v0IAaHfP29LVT0NdmPq6cKnLlCoRCgkhyVvexeRyhq6+JPNaZ5Cl4DYeL2cjX1rifAPcMJEyVLqin2hsTGFoaIjLL7+cr3zlK1x77bW0tbV5sQY/F154ITfddBN79uzh05/+NJ/73Of43e9+xwUXXADAP/7jP3LxxRfzox/9iB07dnDRRRd51zY2No54vqMh467sx7IUcj5LwT7uK1IK0XCIdJkfRS2x762lPsrCtnp2HXLcbNlcsaVg3UdjxxSsAgmHhGQmx+GhNDkDQ+kMrUQ9JTq7sY5TFraO+lyNdRGSmRx9iTSNdRE6mupYv/MwxsDspspW/B3ueVv3D6jrqEZEwqGyi6KDQylS2Vwg3Uda+6iKNDQ08OUvf5kvfOELNDQ0sGzZMr7//e8Dzmr1ySefBOCss87iwQcfJBQKEY/HWb16NV/72te48MILAcdSWLhwIeDEEcpx4YUXcvvttwOwceNGNmzYMG6Z7eRVaUzBKoWBVIacYaRSCKClkHTfWywSYllHXqmmi5WCawVVYilYBdLZFCOZyZFwA+zpTN7yev3pC7n/QxcRDY/+s2uMOav67v4kjbEI7Y0x73NuL2MpFGMVwUAyQ7u6jmpCNCRlv//7ep0NpPMDaCmoUqgyp59+OqtWreKOO+7gtttu4xvf+AannXYaJ598Mj/+sRNOicViLF68mHPOOQdw3En9/f2ceuqpAHz4wx/mxhtv5PTTTyeTKb9qffe7383AwAAnnngiH//4xznzzDPHLa8XUxhDKWRdC2Ag6UyGNlOnJV4UaA5gTMFmRMUiIY71KYVskayepTBKTCGZydIzkPQUSGdzjGQmSyLjXGOtknTOEIuGCj6fcjTGHAN+f1+Chli4YKVf6arfrzzKuZyU6hKNhMq6j6xSmBtAS2Ha1T4KAgMDAwWPf/rTn3r377333pLXPPDAA979q6++mquvvtp7fO6557Jlyxbv8Wc+8xkArr32Wq699lpvvL6+njvvvPOoZLfunuRYgWZ3ATTgBkN73Qm0IPsosJaCI3MsEubE+fm9GMWZUpVkH33gzvX8YuM+/unPnVTQzuYY+/sSXmqvnRQy2dyYFoLFKoWu/iTtjXXM9k3qlQaMY5EwzbEI/clMxdaFMrFEQuW///v6rKVQP5kiVYRaCkoBfkshkc6WzR7yLIVEhv5E2ltVF7qPpOxKqZZYKygWDfHGMxdxz/ud2M0IpTA8tvvo3qf3AbDnsBOXKOs+yhoioQqVghsU7upP0lAXLogjjCc+YM+tNA6hTCzRcHlLeV9vgpBARwDjPaoUlAL8MYWrb3mIz/9qc+nzXOWxt3eYs//lPm5zc/SLYwpBzD7yxxQi4RDHz2kCSgSaE2O7j2a55S+2dw9SFw7RUh8hmcl61ojnPsrmiEYq27pqLYVUJkdTLOJN7iL516sEayEEceKZCUTC5ffp7OtLMKc5TqRC63EyCZ5EE8CR1hKZCYz12XjZR5kcOw8Ne77PYuwE+uy+foZSWR7Y4mwAa20ojCmMlcVUC/JKwVmRR9x0zpEpqdZSKB/HaXPf744Dg7TUR4hHw6QyOS9ukfErhQothaZY3qvbGIt47qNZDXWEQ5XXRLAWgrqPakM0FCr7/d/fl2BuSzAtuGmnFOLxOAcOHFDFUALbTyEeLx/c8iyFbJZEOjti9WzJup/v893Ohiw7gbbE8xOaYykEUCmkbUzB+fqHQoKUyCnvHx57n0JbfV4pNMejxCIhcgb63WyhdNaQyxlyJr/LdSwafHsKGuoi3uQ+e5yTe4e6j2pKZBT3USKdpaEumCHdYEp1FCxatIhdu3bR3d1da1ECie28Vg67Wk5nDcl0rrxScMd7BpLeWDgkBavc0cznWpL0xRQs0VCobKA5UYH7KJHO0RKPeNaHDbynszkveF9poNn/GTbFwjTWhamLhMa94rfnj1eZKBNDJFTefZrKGurrgrkmn3ZKIRqNalexo8CulhPpLKlseaVQ6sveEo8U1PQJ/j6F/Io8HJKC92qM8QLN6axx3D8lJnW/u6ylPuopmj6/UnA/g2illoJPKTTUOZ/pnOZY2RLM5ZjT7FiE471OmRiiYRmx98WSyuSoC2A8AaahUlCODjuB2UJ3xfWALKWUhT/IDO6PIojuo0yh+wicuIJf1qRbBXZ2Yx0HBlMMpbK01o/8EUd8Pv4W130EfkvBeC6ESi2FhmheWVmr4UtvWT3uFf/rzljInOZYIHPhZwKjuU9TmSx1FSYeTDbBVFVKzbBfYrvSLef+yfqUhTUOWoqUgmM+B1AppEdaCpFwoaVg37+dUIdTWR7dcZA//8ofPaUC4H97LfV595HfUrDBxkozTUIh8dJSbSbSS5a2c2xnU+VvEkdJvfLU+eO6Rpk4Rqv9lc6awFoKwZRKqRl2YrSWQln3kW98pVsquthSiITLb/OvJaViCuGimIKNJ9iCZUOpDE/v7mX9zsPe7m0otKT8lkI+YJ/zJoboODKHrAvJlrxQph6j1f5KZXLURYI5/QZTKqVmWB+onRTLu4/yX/YT5jXTWh8dYSk4wdsAWgpl3Ed+q6bXjSd4lkI6m68g61MefqXpjylYMlmTVwrjWBlat1FjQDNUlLFxvlPlLIXKd7hPNvqNUwoY4T4q86X2j89tifPpK09m0azCXszRSDBLZ5cLNJeyFGwu+XAqn57rVx6FlkJkRI/nQvfROCwF133UoJbClCUySj+RIFsKqhSUAjKepTB6oNk/3tkc48rVC0ecExll804tsTEF/48yWhRTsHsZbHB3KOWzFHyKrkAp1EcLrA/n3JxnLY3Hh2xjCf70VGVqMVqiRSob3OyjYEql1Ay7CralmssFmv3jc8pkt0QD2mQnmckSDUvB7uBwkalvJ37rEhvyWwo+l1iB+ygeHbH6S2eNV/9oPCUNPPeRKoUpS6TE3hdw0p1T2eBaCsGUSqkZxV/i3Cib1+ykOqdMHnwkHNSYQm6EmydSFP+w95vdHdrD6Yz32fiVR7nsI4t/89qRuI80pjB1KVf7K5MzGDO+GNNkUs0ezbcCrwa6jDGnuGN3ASvdU9qAw8aY1SKyFNgE2OprDxljrq+WbEp5is3dbBn3USZrOO+42XQ0xVi1qHQnMdtkxBgzaqP6ySaZyY5YpRWnpHqWgtv/YDiV84LrfsWZM4Z5LXHOPW42J85voasviZ+0L/toPO6CJs0+mvKUcx/ZsaBaCtVchnwL+C/gO3bAGPMWe19EvgD0+s5/zhizuoryKBVQnIJazv2TM4b5rXE++8bTyj6XXQllc2Zcq+Rqk0znRvj+I0VdsuznkHcf+S2FQvfR3JYYX3rLagBi0Xy6Kji+YzsJRMaTkupaCEGtj6OMTbkyL7Z0e1BjCtXs0bzWtQBGIM6y8c3AJdV6feXIKN5XUC7QnMkZwmNU/bQ+9HTWEAnQgtdxHxXKXlzmwk78nvsolSXrqwtlyRlDyDfZF7uPMm6JDHA6cVXKxSd0MpzOjKsqqhIsnCY7Iy2F1BF8HyaTWi1DLgD2G2O2+saWicgTQB/wD8aYB0pdKCLXAdcBLFmypOqCzjSKYwCjFcQba6Fja/2kcznqCY5WSGayI2MKRfEPO/HXR8NEQsKQb59CcaA5LH6lMDL7yKt9VGHpbIALlndywfLOis9Xgke5RAuvyVNALYVaSfVW4A7f473AEmPM6cDfAbeLSEupC40xNxtj1hhj1nR26o9moin+Eo+mFMbqJFauT0GtGE5l+cYftjOUyo7YZBYpthS84HCI+rpw4T6Fos1rhZZCiZTUI9inoEx9yiVaeIsErX3kICIR4PXAXXbMGJM0xhxw7z8GPAesmGzZlJHZR+UCzf7so3JY91FQ6h/9fks3//SzZ3hk+8GS7qN0iZTUSEhocJVCqeyjnCm0FCLhUMHnks4abwd0ULNNlOrgT7Twk48pBMd69lOLb+nLgWeNMbvsgIh0ikjYvX8ssBx4vgayzXiKJ/ByGaWZXG7MwKkNpAVlA9ugu/eiVEpqNBwqiinkJ/J4NEwik81nHxUFmouVo1/hpLM50hlbJTWYK0OlOthFQPFCKxXw70PVlIKI3AH8CVgpIrtE5J3uoasodB0BXAhsEJH1wA+A640xB6slm1Ke4kBzuX0GxW6TUlh3SVDcR/62mqUshUyR+0jEGbc1bErWPjKM+ByKlUJmnE12lOlB3lIuUgozNSXVGPPWMuPXlhi7G7i7WrIolVMcQyi3yHdiChW6jwKygW3Q11azVEzBbwGks/n3F3Vr2OSVnK/2Uc5QvOBzrJA09dEw6awhZV1RAV0ZKtWhXKKF5z4KqFIIplRKzSiewEulpNqew2PFFGyp6KCUz/b3Wi6VfZQtCCDnvEC63diWKRNoHuE+chVOczxSEGgeT/aRMvUpl2jhbV4LqOUYTKmUmpHOGvybj0sFiW3wOTzGLuVoGfN5LHI5w2MvTLz3cChZ3n0UKXIfOXsrnPcXDoVI54y3T6E40ByS0u6jvFKw2Sb6c5tJlEu0UEtBmVJkc6agHWSpjFS7og6P4Q6xk+p4A80PbOvhDTf9ia37+8d13VgUuI9KxRR8cmZy+Xr3UfdY2X0KI2IKzufXFI+Szph86WzdiDajyLuPSlsKQY0xBVMqpWaksznqfaUVSsUDrFIYa5KLHmFKak+/Uz+oZyA1ruvGYtgfaI4WZx8VBZp9MQVbrsBmHxWUwzAjA+6epRCLkM7lOJImO8rUx7ofR1gKAQ80B1MqpWZkcqagCFupGLGdPMcsc2F9qmU2wJXDlu0e9Ll7JoKxLIXignh2Ere9pkvVPsoV7WgGJ6ZQF3Fuabf2UUjGjsEo0wvrLiyOqSUDXvsomFIpNcEYQzZnqPetokttXvPcR2PMcfnaR+OzFDylkJpYpTA8ilIorlOTyeWzjfKWQolAsyntPopFQk6VzIwhncuNq5eCMj3IJ1oUfv+DXiU1mFIpNcFOdv7GLtncyB2Z1qUUHmOiqysTaB5MZkhlcuRyht7h9Ijr+t2ubwMTbilkaHWrno7sp1BcEM/nPgqFSPv2KRQEmnOUDDTHo2GvcXsmawK7KlSqR9l9CmopKFMF++W1DV4sxd4f61Iae59C6ZXS277+MJ+991nu2biX8/71vhGTv3UbDSQmVikMJbOsnNtMOCS0NUQLjoWLYwr+QHNYyOZyZTuvFf+2ZzXWMbuxjrpw3n2kexRmHhHfPgU/R1I1dzLRYu2Kh/3yFnf7Ks6w8SyFMQPNpbMvdh0aYn5rnM7mGIOpLD39yYJexNWKKQylM5y8oIVP//nJLOtoLDhWnJKayebfc7h4R3N2dPfRh16xkoFkhv/87VbHfeSLTygzh2hoaloKqhQUj2wZS2HkLufK9imUy74YTGYZTjs3YIQLKe8+yjKRDCWz1NeFOWHeyAK8kVDIc5WJCOmc8cx/6wYqVfsolxvZVW5WYx2zGuu8ndDpbM7zLyszh1I74AFvh/uMq32kTD2spVDvKgW7kikONnspqRXuUyjsaWwchZAqrxQGks7jic8+yhTESwpkdSftrC/DKOrFFIRs1njvY0SguYxytEohk9VA80yknKWcyuSoC4cC1aLWj35TFQ876dmJs8FNTS1rKRxBlVSrCBLpLIlUOaXgWgoTmH2UzRkS6dwIK8hiN+L5g8n57CN3R3OFm9csTo9ep3R2UFeFSvUou08hkwts5hGoUlB82EnPTpw2tjCib3OFm9dKbfO3pSZGcx8Num6jibQU7GuVUwrFeyrSvkCzLZaXLZF9ZMzI7COL5z7K5DSmMAOJeinZI3c0B3mRoN9UxcNmRdigr+1PXM5SKDcZWiJFq2/IWwFDqaxXoK4vUSamMIHZR7ZsdkNdOfeR6yrz1Tcq3tFcMtA8SlvSaDhEJmfcSUB/ajONqPf9V0tBmaLYSW9ea5wvvOk0Xrt6AVBeKYwVU7DZF+ms4cFtPVz2xd+z89Aw4LqPxogpTOQ+haHkGJZCUfpg2hcHiIZDZLJl3EclylxY7A9/KJXVlNQZSLnNm0FfJARXMmXSsV/eSCjEG85cREdjDBgZaK64zIU7ESbSWa7++sNs7RrgyZ2HAQoCzX0+pZDJ5kikHTkmckfz4BiWQrgo0Jz1xQGcBjz5Zjl+y6dUmQuLtTQS6ayWzZ6BRLwdzUVlLrJqKShTBDsh2snQroBzR5yS6hy/89EXvbF9fQnA8fEPlQg0D/rSUAcnMCXVlrjw13Xy4+WU+0pZWKVne+162UfZQkuhfKA5bykEtUm7Uj3KlY5Pu9lHQaWa7ThvFZEuEdnoG/ukiOwWkfXu7VW+YzeKyDYR2Swil1dLLqU8XrN69wtrv7fFBe0q3bwmIkTDwv6+pDe297DjPsqZvIXgVwr9ruuovbFuQt1Hthhe2ewjryFK3n3kpaQWZVHZH7kxZvRAs999pJbCjCNSLqYwgy2FbwFXlBj/kjFmtXu7B0BETsLp3Xyye81/i0jpX69SNTKe+yjfXAZGxhS8MhcV+MntZLiwrR6Avb0J79ihoZFKwSqCuS1xUpmct/vzaLFZT2UDzaOmpDp/k65bK+1zMUF55WiVylAqE2gfslId/DE1P6mZaikYY9YClbbPuhK40xiTNMZsB7YBZ1VLNqU0xX0SrHuouCVnpZYC5CfUlfOagUKlcHjI6ZfQN5y3CGzG0bwWJ54xUWmpQ2NYCpEiBZjxVTa1n0feUnD+eh3oKnEfaaB5xmFdhsX7FDTQPJL3isgG1700yx1bCOz0nbPLHRuBiFwnIutEZF13d3e1ZZ1RpHPF7qPSPWYrjSlAfgPbgrY4zbFIgVVgDZBSlsK81njB4yMllzP89++2ccsDzwNjB5rte01njW9Hc+HPxFoT1mIay32UzOiO5plIpChOZdGU1EJuAo4DVgN7gS+M9wmMMTcbY9YYY9Z0dnZOsHgzm5Huo3KWQmU7mv3nzm+tp6U+WvKcvkTaC2b73Udw9BlI//f7T/LZezfz7D6ntefYm9fy1kA+JbXwfY60FEq/dp3vOrUUZh5emYsStY/UUnAxxuw3xmSNMTngFvIuot3AYt+pi9wxZRLJB5qtUnDGR8YUKtunAHkrYG5LvKRSaIlHMAb6i8ple0ohmeGP23r4/rqdI64di2zO8MMndvOa0xZ4Y/XR0fcp5Hc0G9/nUNpSGGsTn9/C0JTUmYeIeBV2/aQy2RFNnoLEpFZJFZH5xpi97sPXATYz6SfA7SLyRWABsBx4ZDJlU/wpqdZ9VNr8rbTMhZ/5rXFa60d+3ea2xOlLDNA3nKa1Ppp3H7VY91GWa251vgpvWrN4xPWjYRXM6sVtvOKkufxm0/6yG82KYwrZnPEm8mLlZ5VnbqxAs++HrympM5NISEpsXgt2LayqKQURuQO4COgQkV3AJ4CLRGQ1YIAdwF8DGGOeFpHvAc8AGeAGY8zE1k1WxsS6TsYKNFda5sLP3Ja41/XMz7zWOFu7BugdTrOYfImLOW6geXv3wPjehA9bPqM5HuE1py0osBiKCYfypr5tS2rHxnYflS+IZ9GU1JmJU/9qasUUqqYUjDFvLTH8jVHO/2fgn6sljzI2nvvIncDsPFYu0DyeiW5ea14pNNaFvX0D1k10YDBFOpvj50/t5diORlrizrk/fnKP9xy5XPmSEqWwriv7XKNhJ/Bsznifgx0rfp/ZXKGlMFpBvOLnV2YWTt0s3aegTFFsE5lI0WRYzlIIj2Oia4pFPKXQ0Rzzxk9Z0EJjXZiP3L2Bv71rPdu6BrjxVSfS2RxjTnOMJ1487J07lB6f8WgthZYSbqtivOyjnMlbTEUpqRZbH6nSlFSgZGMfZfpTylIIetXc4EqmTDqVBpqPJKYA+RX77MY6b2xxewPfu/5cYpEQv9i4j9efsZCXnziHeDTMD64/j9OXtHkb3/qLqqmOhd3/UImlEPG1TsxbTIU7mi2ZrGHtlm72HHb2XJRLzV3QFmdeS5wPX7GSN5y5aFyyK9ODqFt23U8ikyUWCe7eXG3HqXjYL68NsFq3yMgqqbmC46Nx/vGzSWec61sbXKXQlLcU6qNhTl7Qyu8+dPGIa5fMbuBH7zmfnz65h/fd8YQTOG6t/P1YJVIqllFMxHMf5fKfQ7h8oPld31nH5SfPAyjr0prTHOehj15aucDKtCPilk+3JDNZ0lnjlaUPIsGVTJl0MkVuoeKMHEvxzufRuO1d53j37eTstxTiZfYN+Glyf0D949zI1pcYj6Xgdx8VWkz+dNK6SIj+RJpUJufttg6wJ0CpMQ11YS95AvIZcU1l2sIGgeBKpkw6djKMFgWay5bOHmfw1O5TaIpFiEVCJDO5svsG/DS7P6D+cTbdsQX3mipYldm4wPbuQWa7JcPt5+CPGcQiIU+OZMaJcYwnC0uZWSxsq2fXoSHvsU25DrJS0DWO4uHtaPb1EYC8ZfD4i4fI+XoVV1Lmwo+1FBpiEepdC6EipeCu9O0qa8v+/oriC32JNM2xSEU7r62r6Au/3sKH734S8FkKPuUX98lr+z6oUlDKsbi9gV2HhjHuwsouKCpZqNQKVQqKx4gAq08pPLWrl9f/94Os3do9rjIXfqwbp7Eu7CmD+vG4jxJpjDG8/r8f5JYHto95Xd9wpmxpjWL872V/b7JgzB9ojkfz923nuPF+DsrMYUl7AwPJjFcReFAtBWUqYTdsibvyDfk2r23e79QO2nloOF/mYpyT4ZyWGPXRMAtn1Y9LKdig3EAyw2Aqy0Aywx63L8No9CXSFQf0/O8lVRxo9h2LR/yWgrqPlNFZ3N4AwIsHHReSuo+UKUUyky2o8+6vHLq9x9lZ3N2XOCpL4cGPXMKrTpnvuWEqcR811uVjCnZD2sHB1JjX9Q2nK7YUSlUxzaeklnYfJTOVlxBXZiZLXKWws1gpqPtImQocGEjR7ssM8mIKxrC9ZxCA/X1JsjlDSPAsivEwq7GOUEhoqAsTCUlFm3jCIaHRzeLodc3wAwPJMa5yso8qyTyyr1FM3lLIy+gvZGZjCpp9pJRj0Sxnj421FGxMoVktBWUq0D2QLNht7JXOzhm29zhf6q5+x1I42lo+9b64QiU0x6MMJNOepXCgyFLwl9+29CfSFe1mhtKusLECzUl1Hylj0BiL0NFU52Ug2ZhCoyoFZSrQM5Cis8lnKbiTXTpn2OFaCl39SXKjNKuvlHg0XNEeBUtTPEJ/IuOVrjgwkFcKhwZTnPMv9/Hzp/YWXNM3nK7YUihVBjxSIiXVH2hW95FSCYvbGwpiCiLl+3oEAVUKikfPQJLOEpbCnsPDDKedlpJd/Umnf/FRToSzGqK0VejvByfYPJDMxxSG01mG3AY8T+46zFAqy9aufEXVXM7Qn8zQUnGgeeRPIepZCj73kc9SsAHp8abmKjOLxbMaCtxHTbHIEbleJ4vg2jDKpJLNGQ4MJOloGqkUtu53JttVi9p44sVDpLLZcVUrLcX/vXxlQW/msWiKuZaCr3XngYEUDe0Rnt7TBzhKzTKQymAMFQeaS72dUgXx4iVq1hztZ6FMb+a3xbl3YxJjDAPJTKAzj0AtBcXl0FCKnKGkUrCZR2ctaydnoKsvedSWwpzmOMfPaar4/OZ4hP5EuqCfs40rPLWrF4Ce/iSf/+Vmrrr5T57yqNR9VGrlVqogXiw68iej7iNlNOa3xEllcxwcTDE4BZRCsKVTJg27yi7lPjrsZvysnNsMwJ7e4UmfCJtj0QL3EeQzkDbucZXCQJL+RIaHtx9k9yFnH0OlgWaAO687BwHecvNDgL+E+BiWQoBdAUrtmdfq9AzZ15dwLIUAp6OCKgXFpaffWXX7LQU72dlCdEtmOznXew8nJr3HrBdoHk4TCQmZnOHAYIrDQyl2uQqgxw0+GwN3Per0dB5PH4Nzjp1NV3/Cexwp0Y4zrpaCMk7mtTppqft6E/QnMoGukApVdB+JyK0i0iUiG31jnxORZ0Vkg4j8SETa3PGlIjIsIuvd21erJZdSmu4BZzLs8GUf2RWybR84z9clbbL96M3xCEOpLAeH0p5yumXt85z7r78F4IR5zXT1J9jb6yiIn27YQ0dTjGPccyvF726KlqiSGi+RRquBZmU07O/GsxQC7j6q5nLvW8AVRWO/Bk4xxqwCtgA3+o49Z4xZ7d6ur6JcSgk8S6GE+wicncd+19LRxhTGi/0h7To0xPzWOPFoiK1dAyycVc8HL1vBq1fNJ5HOefWb0lnDWctmjTvLIxYJ5dtwhm21WME+TSlLQdsvK6PR2RwjHBL29SamREyhal9nY8xa4GDR2K+MMTbl5CFA21EFhJ6BJLFIqGCnpUh+MqyPhomGQ14XtMl2mSyd3QjA9p5BWuujXnnr911yPO+7dLlnokN+1/GaY9rH/Toi4lVljfreo7UWSnXMUveRMhrhkNDZFGNfb4KBRPBjCrVc4/wV8Avf42Ui8oSI/F5ELih3kYhcJyLrRGRdd3d39aWcIXT3O+moxStraxHYwnUr5ja545P71TllodNyzRinBHdHUx2t9VGv+5nf7XXB8k7AyZY6EqzP1591ZOMKJWMK6j5SxmBea5y9vQkGUplAl7iAGgWaReRjQAa4zR3aCywxxhwQkTOB/xWRk40xfcXXGmNuBm4GWLNmjSk+rhwZxSUuLE6w2XglKVbMbeb+zZOvjOe2xOhoqqNnIEVLPMp7Lj6eXM54Pn5/gPxvL1vOyQtaOGl+5UFmP3mlkJ/srXK0r2eD3aD7FJSxmdcSZ8OuwxgT7BIXUAOlICLXAq8GLjVu5wljTBJIuvcfE5HngBXAusmWb6bSM5BiYVt8xHi4yFJY7qal+rtJTQYiwikLW/nd5m5afBaCxcY7mmMRTl7QyskLxtHMuYjmmHUf+S2FQvdRW0Odl8arloIyFvNa49z7tJPMoe4jHyJyBfBh4LXGmCHfeKeIhN37xwLLgecnU7aZTnGJC4unFNwVst2rMJjKTp5wLqe4E31riV3K7Y11iMCCtvoRx8aLtRT8sYK8pRByXy8vg8YUlLGwexUg2L0UoIqWgojcAVwEdIjILuATONlGMeDXru/6ITfT6ELg0yKSBnLA9caYgyWfWJlwSpW4sISL3Cbj2YU80Zyy0HEHlVIK0XCIWQ11LChh7YwXWxrDXx3V1j86aX4LbzhjEa31Uba45T/UfaSMxcUr5/CHrT2EQsKZx8yqtTijMqpSEJEzRjtujHl8lGNvLTH8jTLn3g3cPdprKdWjVIkLi3WNeD2Va1jd8bzjO/izU+eXDSBfe95Sjus8eqXVHI8UdKCDfHyhMRbhC28+ja8/kDdk1X2kjMXKec18911n11qMihjLUviC+zcOrAGeBARYhePvP7d6oimThfWNj2Yp1Puybj58xUoaxtELYaJoiUf5ytvKr1Pef+nyCXmdS0+YS9qtgGqxn4P966+cqvsUlOnEqErBGHMxgIj8EDjDGPOU+/gU4JNVl06ZFOzGtUpiCgDvuej4yRGsRrx0eQcvXd5RMGaDzpESSkEtBWU6UekaZ6VVCADGmI3AidURSZlsSpW4sNj6R+NpiDMdse4jqyT96apaEE+ZTlQaaH5KRL4OfNd9/DZgQ3VEUiabUiUuLHbyG0/rzOlIJCQFcYa6AveRKgVl+lCpUrgWeDfwN+7jtcBN1RBImXx6BpLUFZW4sHiB5pmuFMKhgtTTAveRKgVlGjGmUnD3D/zCjS98qfoiKZNN90CSzhIlLiC/Cq5l1lEQiISkoAigP11VYwrKdGLMmIIxJgvkROTIt4gqgaa7v3SJC/DVPprhlkK02FKIaPaRMj2p1H00gBNX+DUwaAeNMe+vilTKpFKuxAXkg6gz3VIIF1kKdZp9pExTKlUKP3RvyjSkZyDJ6sWlDUENNDtEw0LYXwvJpyA0pqBMJypSCsaYb1dbEKV2HBpMMathZDoq+CyFGa4UIqFQYUzBdR+JMO5GPooSZCpSCiKyHPhX4CSc3c0AGGOOrZJcyiSRzRkyOVOyeQz4ah/NcPdRJCwFFoF1H6nrSJluVBoi+yZOCmoGuBj4Dvk9C8oUxpZzqIuU/iqU2tE8E5nTHC/Y8R31tepUlOlEpTGFemPMfSIixpgXgE+KyGPAx6som1IlhlNZUpkcrQ1RkhlHKfhTLP3oPgWHD1+xkpSvHpL9vNRSUKYblVoKSREJAVtF5L0i8jqgdjWUlaPi336xiXd88xEgbynExrIUZrj7KB4N0xLPl+y2loIGmZXpRqVK4W+ABuD9wJnA24FrqiWUUl26+pN09Tn1jlKepTC6UojPcEuhGM99pDpBmWZU6j46aIwZwNmv8JdVlEeZBNLZnOc20pjCkREtKpCnKNOFSpXCrSKyCHgUeABY66+aqkwtUllDMu2007SWQjmlEBIn66ZczGGmYlNSVSko042K3EfGmJfhlMr+T6AN+LmIjNkuU0RuFZEuEdnoG2sXkV+LyFb37yx3XETkyyKyTUQ2jNX1TTly0pm8pWCDp+XcR5GQ0BANay5+EXWe+0g/F2V6UZFSEJGXAh8EPgb8GfAz4IYKLv0WcEXR2EeA+4wxy4H73McArwSWu7fr0CqsVSOdzZHJGTLZ3JiWQjgkM36PQik00KxMVyp1H/0OeAxnA9s9xphUJRcZY9aKyNKi4SuBi9z733af++/d8e8YYwzwkIi0ich8Y8zeCmVUKsTGEZIZn1IoYynEIiGaSpTUnumEQ0JI1FJQph+V/to7gPOBC4H3i0gO+JMx5h+P4DXn+ib6fcBc9/5CYKfvvF3uWIFSEJHrcCwJlixZcgQvr6SyBnCUQtq9X85SuOGS4zk4WNEaYMZR3GNBUaYDlcYUDgPPA9txJunjcBTEUeFaBWac19xsjFljjFnT2dl5tCLMSPKWQpZU1gk4l4spHNfZxEuWtk+abFOJOlUKyjSk0tpHzwPPAn/A8fX/ZaUupBLst24hEZkPdLnju4HFvvMWuWPKBOMphXSOVMa1FMooBaU80bDoPgVl2lGp++h4Y0xu7NMq4ic4G9/+zf37Y9/4e0XkTuBsoFfjCdUhnfHFFLx9Cjq7jZfixjuKMh2odHl4vIjcZ1NLRWSViPzDWBeJyB3An4CVIrJLRN6JowwuE5GtwMvdxwD34LiotgG3AO8Z31tRKiUfU8h6CqIurBlG4yUaDmmgWZl2VGop3AJ8CPgagDFmg4jcDnxmtIuMMW8tc+jSEucaKktzVY4S6z5KpP2WgrqPxktdRJWCMv2odCZoMMY8UjSWmWhhlMmhINA8RpVUpTyRkKj7SJl2VKoUekTkONxMIRF5I0WposrUwR9oHqv2kVKeaDik/RSUaUel7qMbgJuBE0RkN05q6tuqJpVSNYwx3t6EpK/cRbmUVKU80UgINbCU6UalPZqfB14uIo041sUQcBXwQhVlU6qAVQjgBpqzo+9oVspTpxpBmYaMqhREpAXHSliIkzr6G/fxB4ENwG3VFlCZWNK+7mG2zEUkJOoGOQJeecr88e28VJQpwFiWwv8Ah3DSSv8PTkE8AV5njFlfXdGUalCgFNKOpaDxhCPjr166rNYiKMqEM5ZSONYYcyqAiHwdJ7i8xBiTqLpkSll29AyycU8vr161YNzX+vsMJ1xLQeMJiqJYxpoN0vaOMSYL7FKFUHvueORF/vau9ThbO+APW3vYur+/omttCiq4ZS7UUlAUxcdYs8FpItLn3vqBVfa+iPRNhoDKSIbTWdJZw7DbPe3GH23gpt8/V9G1xYHmVMZokFlRFI9R3UfGGK19EECSaWe135/I0FAXYTiVTy0dixGBZrUUFEXxobPBFMTGBfqGHe9eKpMlk61MKRS4j9zaR2opKIpi0dlgCpLMOG6jvoRTaSSVzRW4hUajMPvIsRSiWiFVURQXVQpTkLz7yLEU0llTMNmPhl95JDJOmQu1FBRFsehsMAWx8YP+RIZszpDNjUcpFO5TSGpKqqIoPnQ2mIKkfErB3q/UfZQqsaNZA82Kolh0NpiC2JhCfyLtKYVKA822qU5jXdirfaTuI0VRLJVWSZ0wRGQlcJdv6Fjg40AbTimNbnf8o8aYeyZXuqmB332UzDoKIlVxoNk5rykeUUtBUZQRTLpSMMZsBlYDiEgY2A38CPhL4EvGmM9PtkxTjbxSSPvcRzlyOcNAKkNLPFr2WhtTaI5HvX4KqhQURbHUeja4FHjOGKMluMdBMp1PSfW7j+7ZuJfz/vW3DKXKN8WzMYWmWISE23lNA82KolhqPRtcBdzhe/xeEdkgIreKyKxaCRV07MTen0h799NZw97DCQaSGfoT5ZVC3lKIuPsUjFoKiqJ41Gw2EJE64LXA992hm4DjcFxLe4EvlLnuOhFZJyLruru7S50y7bH7FPoKso9ynoJIuJZEKWygudmLKWQ10KwoikctZ4NXAo8bY/YDGGP2G2OyxpgccAtwVqmLjDE3G2PWGGPWdHZ2TqK4wcEfaE5n80rBjo9WB8kLNMciTkE8jSkoiuKjlrPBW/G5jkRkvu/Y64CNky7RFCCXMwXuo6QXUzCe1WAtiVLkYwpRkhmnPEZU20oqiuIy6dlHAG6v58uAv/YNf1ZEVgMG2FF0THHxbz7zb15LZXN5pZAZxX3kKYWwd35dWIvhKoriUBOlYIwZBGYXjf1FLWSZaljLoD4aZiCZ8R47MYVswTmlSGednszxurwiUPeRoigWnQ2mGNYK6GiuI5sz9A45RfFyBhLpsQPNNgU1HskrBXUfKYpiUaUwxbDxgo6mGAA9g0nvmN2fMFagORoWlnU2emMxtRQURXHR2WCKYWMKnlLoT3nHBpLWfTSKpeBmG52xJL8NRDevKYpi0dlgimEthfaGOgAOD+WVwlAyU3BOKdKu+6i1PkqDG1fQmIKiKBadDaYY1gqY1egqBbclJ8BAcqT76MFtPXz6p894j9PZfFmL0xa1ARASjSkoiuKgSmGKYSf82a5SOOSzFAa9mELeffSrZ/bzzQe3k8s5m9b8+xJOX9IGwJ7e4arLrSjK1KAmKanKkWP3FniWwlDeUhhyYwoJn/toIJnBGOhPZmitjzo9mV1L4d0XHUdXf5I3r1k8WeIrihJwVClMMaylMKvBKY89lqUw6LqU+obTtNZHC0plN8ejfP5Np02K3IqiTA3UfTTFKI4p9PpiCtZCSBZZCgB9Cec8f0xBURSlGJ0dphh2wm+rdywFU6Lhmj/QnLcUnL/pjNY6UhSlPKoUphh2n0JDXYT6aOmaRYXuo3w/Z3u9WgqKopRDZ4cphu26VhcJ0RgrrRSKA83g9F4Ax32kO5gVRSmHzg5TDOsaikVCNNSVzhMosBRS+UAzaExBUZTR0dlhilGoFMq5j5xzjDH5mIIXaDaqFBRFKYvODlOMVCZHOCREwiEaY46lEAkVBo5tMNo20YF8oLlvOF3W7aQoiqJKYYqR9PVUtkrB/vWfA/nMI3Ashd7hNAcGUyyd3YiiKEopVClMMZKZHLGoqxRc91FTkVKwgWabeQSOhbCjZxCApR2qFBRFKU3NdjSLyA6gH8gCGWPMGhFpB+4CluK05HyzMeZQrWQMIsl0PnvIBpqLYwvWUhgoshS2u0rhWFUKiqKUodaWwsXGmNXGmDXu448A9xljlgP3uY8VH6lsjpjbNc3GBka6j1xLwc08ikVC9A1n2N4ziAgsmd0wiRIrijKVqLVSKOZK4Nvu/W8Df147UYJJMpP1ahdZS8EfOI6GxVMK1lKY3xr3LIVFs+o9paIoilJMLZWCAX4lIo+JyHXu2FxjzF73/j5gbvFFInKdiKwTkXXd3d2TJWtg8LuPbEzBv1+hOR71NrjZQPOCtnr6hh2lsKyjaZIlVhRlKlFLpfBSY8wZwCuBG0TkQv9BY4zBURwUjd9sjFljjFnT2dk5SaIGh2TGF1Nw3Ub+QHNTLELCuo88S6Ge/qTjPlqmriNFUUahZkrBGLPb/dsF/Ag4C9gvIvMB3L9dtZIvqCQz2XxMwbMU8u6g5niEVCaHMcbr2bygLY4xjjvp+DlqKSiKUp6aKAURaRSRZnsfeAWwEfgJcI172jXAj2shX5DpT2S8wLK1FOqjYez+tZa4Uz01mckx4NY7mtca966/8vSFkyitoihTjVqlpM4FfiROb+AIcLsx5l4ReRT4noi8E3gBeHON5AssA8kMLXE3wOxaCHWREJFwiFQmR7N7LJnJMZjKEI+GaHXLbH/o8pWe0lAURSlFTZSCMeZ5YETLL2PMAeDSyZdo6tCfyNAUL9zJXBcJUecpBWspZBlIZmiKRbj85Hnc/q6zOfe42TWTW1GUqYG245xCOHGCjGcNNLpZR9FwiIjbOMezFNI5BpOOqykaDnHe8R21EVpRlClF0PYpKKOQSOfI5gxNMccaaHD3J8QiIa/yaYvnPso6SqFMeW1FUZRSqFIIMLmc4fO/3Mwln/8dXX0Jr3uadR81u+6jeDTsFcmz7qNEOkfvcHpEXSRFUZTRUKUQYL7w68381/3beL5nkH/7xbP0u/sOrDUwpyXOf1y1mtesWjDCfXR4KM2Tu3o5dVFrbYRXFGVKosvIgGKM4YeP7+bSE+awcl4z//275zj72HagcLPalaudFNNokaXw22e7SGVyXHLCnEmWXFGUqYxaCgFl095+9vYmuPzkebzxzEUAPLrDKRjbXCKt1Dbaaal3FMbPNuyhKRbhJUvbJ0liRVGmA6oUAspvn90PwEUndNLRHAPwSl+XihPYInmrF7exenEbXf1JLlje4Y0riqJUgrqPAspvn+3itEWtzGmOY4whFgl5SsHGDfxY91FjXYQ7rzuHr/7+OS4/ed6kyqwoytRHl5EBJJszbNzTx1nLHNePiNDRFOPgYAoorRQiISESEkIhIR4N84GXr+DE+S2TKreiKFMfVQoB5IUDg6QyOZbPbfbGrAsJRjbVAXdXs7qKFEU5SnQWCSBb9g8AsMKnFDqb6gCn+J11FfmJhlUpKIpy9OgsEkC27u8HYLmvzHWnayk0lXAdgeM+qiuhLBRFUcaDziIBZEvXAItm1Re4iTqaHKXQXGaHclTdR4qiTACafRRAtuzrL3AdgU8plLEULlrRyaJZ9VWXTVGU6Y0qhQDwh609ALx0eQfpbI7newa46ITCVqNWKZRzH71pzeLqCqkoyoxA/Q0B4F/u2cQH7lpPOptjw65e0lnDKQsKaxbZmEJzTJvkKIpSPVQpHAHDqSx3PvIi6WzuqJ/LGMPOg0P0DCS5b1MX9z/bRTgkXLC8sP9Bh5t9VM5SUBRFmQgmXSmIyGIRuV9EnhGRp0Xkb9zxT4rIbhFZ795eNdmyVco9T+3lIz98im/9ccdRP9fhobRX/fS2h1/gvme7OHPJLNoa6grOs/sUtBS2oijVpBaWQgb4oDHmJOAc4AYROck99iVjzGr3dk8NZKuILW7K6H/ct5WuvsSI43967gA9A0kAEuksdzzyIt996AX63H4IfnYeGgLg9CVtPLC1h017+7jkxJGVTZtjEU6Y18xJC3SXsqIo1WPSlYIxZq8x5nH3fj+wCVg42XIcDVv299PRVMdAMsPPn9pbcKwvkebt33iY/7xvK+BYFTf+8Cn+4X838r1Hd454rhcPOkrhn//8VN53yfG0xCNcUaJmkYhw7wcu5M0aUFYUpYrUNKYgIkuB04GH3aH3isgGEblVRGaVueY6EVknIuu6u7snS9QCtuwf4LzjOohHQ+w+NFxw7PEXDpHNGR7efhCAp3b3Uh8NM7uxjs37+kc8l1UKS2Y38MFXrGT9x1/B0o7G6r8JRVGUEtRMKYhIE3A38AFjTB9wE3AcsBrYC3yh1HXGmJuNMWuMMWs6OztLnVJVBpIZdh8eZuW8Zha01bOnt1ApPLrDUQab9/fTO5xm4+5eTlrQwsp5zWzpGhjxfDsPDtPeWOfFCkJuXwRFUZRaUBOlICJRHIVwmzHmhwDGmP3GmKwxJgfcApxVC9nGwl+CYmFbPbsPF8YUHt1xiHg0hDGwbsdBnt7TxykLWlgxt5mt+/s5MJDk/me72Li7F4CdB4dY3N4w6e9DURSlFJOeyiIiAnwD2GSM+aJvfL4xxjroXwdsnGzZxqJnIMmvn3Ga36yc18yC1nqe3dflHU9msqzfeZg3nbmIux7dyffW7WQoleWUha2ksjmGUlmuvuVhNu/vJyTwpxsvZeehIVYtaqvRO1IURSmkFvmN5wN/ATwlIuvdsY8CbxWR1YABdgB/XQPZRuV9tz/Bn54/QEs8wqJZDSycVU93f5JEOks8Gmbzvn5SmRwvPb6DLfv7+ZWrQE5Z2MqAm3a6eX8/L1vRye+3dPO7zV3sPjTMa1YtqOXbUhRF8Zh0pWCM+QNQynEe2BRUy7P7+njFSXP5hz87iXBIWNDm1Bra15tgaUcjew478YXF7Q187o2n8VffepSegSTHz2liKJkFQAT+6cpTuOI/1vJf928jkzOcd/zsmr0nRVEUP7oTqkIOD6U4NJTmJUvbWTLbiQEsaIsDsOfwMEs7GtnX68QX5rfGmd0U42fvfymHhtJEwyFaG0IsmlXPirnNLJndwBlLZvGHbT00xyK8ZGl7zd6XoiiKH1UKFWL7Iy/zpYsudC2F3a6FsLcvQV04RHujsxu5oS5CQ13+I77tXWfTEndqF61Z6iiFC1d0lmyaoyiKUgtm/GxkjKnoPKsU/HsI5rXGEYE9bgbS/t4Ec1tjOLH0kRwzu5FZrsI4e5njMrq0xO5lRVGUWjHjLYX33v4EdZEQX3rL6lHP294zSEhgiS99NBYJ09kUY5dbqmJvb4J5LfGKXvecY9v59l+dxUuP7xj7ZEVRlEliRisFYwxrt3ZXVGRue88gi9sbRnQ3O7azkW3dzqa0/X0JTlnYWuryEYgIL1sx+ZvvFEVRRmNGu49ePDhEfyLD3t5EyWJ1frb3DBbEEyzOprQBcjnD3t4E81srsxQURVGCyIxWCk+5u4ohv1O5FE/v6WVb10BJpbB8bjMDyQyb9vWRzOSYW6H7SFEUJYjMaKWwcXefd3/L/pF1icBRFm/66p9ob6zjL845ZsTxFXOaAHjAbak5v1X7JCuKMnWZ4Uqhl5Pmt1AfDXs9EvwYY/jET54mGg7xo/ecz7GdTSPOWTG3GYC1W5yKrfNaY9UVWlEUpYrMyEDzpr19XPc/69h7OMGb1iwiEhZ+9fR+1u88zA0XHc9d63aydHYDc1viPPjcAT595cnMKxMrmNVYR2dzjHU7DgGo+0hRlCnNjFQKjXURXnJMO7JUePs5x3DrH3awYdcu9vcleNd31hEJCZmcs3/h5SfO4eqzloz6fCvmNvHHbQd4xUlzvQ1tiqIoUxGpdPNWEFmzZo1Zt27dUT/Pxt293Lepi6vOWsxXf/8crz1tATsODLLncILrX3Yc4TF6HDy4rYdn9vbxl+cvG/NcRVGUWiMijxlj1pQ8pkpBURRlZjGaUpjRgWZFURSlEFUKiqIoiocqBUVRFMVDlYKiKIriETilICJXiMhmEdkmIh+ptTyKoigziUApBREJA18BXgmchNO3+aTaSqUoijJzCJRSAM4CthljnjfGpIA7gStrLJOiKMqMIWhKYSGw0/d4lzvmISLXicg6EVnX3d09qcIpiqJMd6ZcmQtjzM3AzQAi0i0iLxzF03UAPRMiWHUIunygMk4UQZcx6PKByjgeRpZ8dgmaUtgNLPY9XuSOlcQYc1Sty0RkXbldfUEg6PKByjhRBF3GoMsHKuNEETT30aPAchFZJiJ1wFXAT2osk6IoyowhUJaCMSYjIu8FfgmEgVuNMU/XWCxFUZQZQ6CUAoAx5h7gnkl6uZsn6XWOlKDLByrjRBF0GYMuH6iME8KUrpKqKIqiTCxBiykoiqIoNUSVgqIoiuIxI5VCUOoricitItIlIht9Y+0i8msR2er+neWOi4h82ZV5g4icMQnyLRaR+0XkGRF5WkT+JoAyxkXkERF50pXxU+74MhF52JXlLjebDRGJuY+3uceXVltGn6xhEXlCRH4WRBlFZIeIPCUi60VknTsWpP91m4j8QESeFZFNInJuwORb6X529tYnIh8IkowVYYyZUTecrKbngGOBOuBJ4KQayXIhcAaw0Tf2WeAj7v2PAP/u3n8V8AtAgHOAhydBvvnAGe79ZmALTk2qIMkoQJN7Pwo87L7294Cr3PGvAu92778H+Kp7/yrgrkn8f/8dcDvwM/dxoGQEdgAdRWNB+l9/G3iXe78OaAuSfEWyhoF9OJvEAiljWdlrLcCkv2E4F/il7/GNwI01lGdpkVLYDMx3788HNrv3vwa8tdR5kyjrj4HLgioj0AA8DpyNs2s0Uvw/x0l3Pte9H3HPk0mQbRFwH3AJ8DN3IgiajKWUQiD+10ArsL34cwiKfCXkfQXwxyDLWO42E91HY9ZXqjFzjTF73fv7gLnu/ZrK7bowTsdZiQdKRtctsx7oAn6NYwkeNsZkSsjhyege7wVmV1tG4P8BHwZy7uPZAZTRAL8SkcdE5Dp3LCj/62VAN/BN1wX3dRFpDJB8xVwF3OHeD6qMJZmJSmHKYJzlQ81zhkWkCbgb+IAxps9/LAgyGmOyxpjVOKvxs4ATailPMSLyaqDLGPNYrWUZg5caY87AKV1/g4hc6D9Y4/91BMfVepMx5nRgEMcV4xGE7yKAGxt6LfD94mNBkXE0ZqJSGFd9pRqwX0TmA7h/u9zxmsgtIlEchXCbMeaHQZTRYow5DNyP44ppExG7OdMvhyeje7wVOFBl0c4HXisiO3DKwV8C/EfAZMQYs9v92wX8CEfBBuV/vQvYZYx52H38AxwlERT5/LwSeNwYs999HEQZyzITlULQ6yv9BLjGvX8Njh/fjr/DzVg4B+j1maRVQUQE+AawyRjzxYDK2Ckibe79epyYxyYc5fDGMjJa2d8I/NZdvVUNY8yNxphFxpilON+33xpj3hYkGUWkUUSa7X0cn/hGAvK/NsbsA3aKyEp36FLgmaDIV8RbybuOrCxBk7E8tQ5q1OKGE/XfguN7/lgN5bgD2AukcVZC78TxHd8HbAV+A7S75wpOV7rngKeANZMg30txTN0NwHr39qqAybgKeMKVcSPwcXf8WOARYBuOGR9zx+Pu423u8WMn+X9+Efnso8DI6MrypHt72v4uAva/Xg2sc//X/wvMCpJ87us24lh1rb6xQMk41k3LXCiKoigeM9F9pCiKopRBlYKiKIrioUpBURRF8VCloCiKonioUlAURVE8VCkoig8RyRZVuhy1iq6IXC8i75iA190hIh1H+zyKcrRoSqqi+BCRAWNMUw1edwdOnnrPZL+2ovhRS0FRKsBdyX9WnH4Dj4jI8e74J0Xk/7r33y9O74kNInKnO9YuIv/rjj0kIqvc8dki8itxekB8HWcjk32tt7uvsV5EviYi4Rq8ZWWGokpBUQqpL3IfvcV3rNcYcyrwXzhVT4v5CHC6MWYVcL079ingCXfso8B33PFPAH8wxpyMU2doCYCInAi8BTjfOEX+ssDbJvINKspoRMY+RVFmFMPuZFyKO3x/v1Ti+AbgNhH5X5wyDOCUCnkDgDHmt66F0ILTYOn17vjPReSQe/6lwJnAo07pKerJF1BTlKqjSkFRKseUuW/5M5zJ/jXAx0Tk1CN4DQG+bYy58QiuVZSjRt1HilI5b/H9/ZP/gIiEgMXGmPuBv8cpd90EPIDr/hGRi4Ae4/SkWAtc7Y6/Eqe4GziF094oInPcY+0ickz13pKiFKKWgqIUUu92cbPca4yxaamzRGQDkMQpj+wnDHxXRFpxVvtfNsYcFpFPAre61w2RL6H8KeAOEXkaeBB4EcAY84yI/ANOB7QQTgXdG4AXJvh9KkpJNCVVUSpAU0aVmYK6jxRFURQPtRQURVEUD7UUFEVRFA9VCoqiKIqHKgVFURTFQ5WCoiiK4qFKQVEURfH4/1hPClqvnsA0AAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "TRAINING_EVALUATION_RATIO = 4\n",
        "EPISODES_PER_RUN = 1000\n",
        "STEPS_PER_EPISODE = 200\n",
        "env = gym.make(\"CartPole-v1\")\n",
        "\n",
        "##########################################################\n",
        "# TODO (10 points):\n",
        "# Implement the training loop for the online SAC.\n",
        "# 1) You need to initialize an agent with the\n",
        "#    `replay_buffer` set to None. Also, leave the\n",
        "#    `use_cql` and `offline` flags to remain False.\n",
        "# 2) After each epoch, run `EPISODES_PER_RUN` validation\n",
        "#    episodes and plot the mean return over these\n",
        "#    episodes in the end.\n",
        "# 3) Plot the learning curves.\n",
        "##########################################################\n",
        "agent_results = []\n",
        "online_sac_agent = SACAgent(env, replay_buffer=None, offline=False, use_cql=False)\n",
        "run_results = []\n",
        "for episode_number in range(EPISODES_PER_RUN):\n",
        "    print('\\r', f'Episode: {episode_number + 1}/{EPISODES_PER_RUN}', end=' ')\n",
        "    evaluation_episode = episode_number % TRAINING_EVALUATION_RATIO == 0\n",
        "    episode_reward = 0\n",
        "    state = env.reset()\n",
        "    done = False\n",
        "    i = 0\n",
        "    while not done and i < STEPS_PER_EPISODE:\n",
        "        i += 1\n",
        "        action = online_sac_agent.get_next_action(state, evaluation_episode=evaluation_episode)\n",
        "        next_state, reward, done, info = env.step(action)\n",
        "        if not evaluation_episode:\n",
        "            online_sac_agent.train_on_transition(state, action, next_state, reward, done)\n",
        "        else:\n",
        "            episode_reward += reward\n",
        "        state = next_state\n",
        "    if evaluation_episode:\n",
        "        run_results.append(episode_reward)\n",
        "\n",
        "env.close()\n",
        "\n",
        "x_axis = list(range(len(run_results)))\n",
        "x_axis = [x * (TRAINING_EVALUATION_RATIO - 1) for x in list(range(len(x_axis)))]\n",
        "\n",
        "plt.plot(x_axis, run_results, label='Reward')\n",
        "plt.xlabel('Episode')\n",
        "plt.ylabel('Reward')\n",
        "plt.title('Online SAC')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WxqbDN6DU0AY"
      },
      "source": [
        "## Offline SAC training loop (10 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRwciG6WU6ES"
      },
      "source": [
        "In this part you are going to train an SAC agent using the replay buffer from the online agent. During training you sample from this replay buffer and train the offline agent **without adding transitions to the replay buffer**. The loss function and every thing else is the same as the online setting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "AcArEy_EU5H0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Epoch: 200/200 "
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvy0lEQVR4nO3deXxcdb3/8dcnS5OmSZu2SRe6pRulLV2JZZGWsgkigrgBF7EoVy4qV71wXYB7vbhcf1cB8aIIlguKCggICCoim1AQC5TSlRZKF9qkaZMmzb40y+f3xzkJ05K0WWbJ8n4+HvOYme85M/OZk8l85ruc79fcHREREYCkRAcgIiK9h5KCiIi0UVIQEZE2SgoiItJGSUFERNooKYiISBslBRmwLPBLM9tvZq+GZV80s71mVm1mI83MzWxauO0OM/vPxEYtEltKCtJvmdllZrbezGrNbI+Z3W5m2RG7nAycCYx390Vmlgr8GPiQu2e6e2nk87n7le7+vRjEOcjMbjazgjAZ7TCzn7Sz3w1hkjq+nW2LzOwJMys3szIze9XMPhftWKX/U1KQfsnMrgF+CHwdGAacAEwCnjazQeFuk4Ad7l4T3h8NpAMb4xzutUA+sAjIApYCqyN3MDMDPguUhdeR204EngNeAKYBI4EvAh+OcdzSH7m7Lrr0qwswFKgGPn1IeSZQAnweuByoB5rDfe8HagAP7z8XPsaBaeHtXwHfD28vBQqAa4BioAj4XMRrpQE3ATuBvcAdwOAO4v0T8LUjvKclQB1wCVAKDIrY9hJwW6KPuy7946KagvRHJxH84n8kstDdq4EngDPd/S7gSuAfHjQVXQzMDnfNdvfTOvE6YwhqIeMIksxtZjY83PY/wNHAfIJf7+OAb3fwPCuBq83sS2Y2J6wVHGoZ8EfgwfD+RwHMLAM4Efh9J+IVOSIlBemPcoB97t7UzraicHs0NALfdfdGd3+CoIYxI/xSvwL4N3cvc/cq4AfARR08z/8jaOq6BFgFFJrZstaN4Rf/p4D73L2RIAG0NiENJ/g/LorSe5IBLiXRAYjEwD4gx8xS2kkMY8Pt0VB6yPPXEjRR5QIZwOsRP/oNSG7vSdy9GbiNoKYxmKB5624ze9XdNwEXAE0EtRyAe4FnzCwX2A+0hO9rc5TelwxgqilIf/QPoAH4eGShmWUSdL4+G+PX30fQ/j/b3bPDyzB3zzzSA929zt1vI/iynxUWLyNINjvNbA/wEJAK/JO71xK830/E4o3IwKOkIP2Ou1cA3wF+amZnm1mqmeURtMcXAL+J8eu3AHcCt5jZKAAzG2dmZ7W3v5l9zcyWmtlgM0sJm46ygDfMbBxwOnAuQf/EfGAeQXNTaxPSN4DLzOzrZjYyfM55Zva7mL1J6beUFKRfcvcfAdcRjACqBF4BdgGnu3tDHEL4JvAOsNLMKoFngBkd7FsL3AzsIahlfBn4hLtvAy4F1rj7U+6+p/UC3ArMNbNj3f1l4LTwss3MyoDlvNfcJNJp5q5FdkREJKCagoiItFFSEBGRNkoKIiLSRklBRETa9OmT13JycjwvLy/RYYiI9Cmvv/76PnfPbW9bn04KeXl5rFq1KtFhiIj0KWb2bkfb1HwkIiJtlBRERKSNkoKIiLTp030K7WlsbKSgoID6+vpEh9IrpaenM378eFJTUxMdioj0Qv0uKRQUFJCVlUVeXh7tr1UycLk7paWlFBQUMHny5ESHIyK9UMyaj8xsgpn9zczeNLONZvbVsHyEmT1tZlvC6+FhuZnZrWb2jpmtM7OF3Xnd+vp6Ro4cqYTQDjNj5MiRqkWJSIdi2afQBFzj7rMIFk3/spnNAr4FPOvu0wnmtf9WuP+Hgenh5Qrg9u6+sBJCx3RsRORwYtZ85O5FhEsEunuVmW0iWKf2fIJFzwHuAZ4nmGb4fODXHkzbutLMss1sbPg8In3S394q5o139yc6DOmHjh6Txblzj4r688alTyFc4GQBwZz2oyO+6PcAo8Pb4wjmu29VEJYdlBTM7AqCmgQTJ06MXdA9kJyczJw5c2hqamLy5Mn85je/ITs7O+5xLF26lJtuuon8/Py4v/ZA19DUzDd/v44/rNkNgCpoEm3nzj2qbyaFcAnEh4GvuXtlZPOFu7uZdWlBB3dfTrCACPn5+b1yMYjBgwezZs0aAJYtW8Ztt93G9ddfH9PXbGpqIiWl340b6LP+vK6IP6zZzVWnTuOrZ0wnNVmjv6VviOkn1cxSCRLCve7+SFi818zGhtvHAsVheSEwIeLh48OyPu3EE0+ksDB4G1u3buXss8/muOOOY/HixWzevJnm5mYmT56Mu1NeXk5ycjIrVqwAYMmSJWzZsoVXX32VE088kQULFnDSSSfx1ltvAfCrX/2K8847j9NOO43TTz+duro6LrroImbOnMkFF1xAXV1dwt73QPfc5mJyMtO4+syjlRCkT4nZT0sLqgR3AZvc/ccRmx4nWIj8f8LrxyLKrwrXlT0eqOhpf8J3/riRN3dX9uQp3mfWUUP5r4/O7tS+zc3NPPvss1x++eUAXHHFFdxxxx1Mnz6dV155hS996Us899xzzJgxgzfffJPt27ezcOFCXnzxRY4//nh27drF9OnTqays5MUXXyQlJYVnnnmG6667jocffhiA1atXs27dOkaMGMGPf/xjMjIy2LRpE+vWrWPhwm4N4JIeamxu4YW3Szh79hiSktRuJH1LLNsbPkiwvux6M1sTll1HkAweNLPLgXeBT4fbngDOIVjXthb4XAxji6m6ujrmz59PYWEhM2fO5Mwzz6S6upqXX36ZT33qU237NTQESwUvXryYFStWsH37dq699lruvPNOTjnlFD7wgQ8AUFFRwbJly9iyZQtmRmNjY9tznHnmmYwYMQKAFStW8JWvfAWAuXPnMnfu3Hi9ZYnw+rv7qapv4vSZoxIdikiXxXL00UtARz+TTm9nfydYsDxqOvuLPtpa+xRqa2s566yzuO2227jsssvIzs5u62uItGTJEm6//XZ2797Nd7/7XW688Uaef/55Fi9eDMB//ud/cuqpp/Loo4+yY8cOli5d2vbYIUOGxOldSWf9bXMxqcnGydPbnZlYpFdTY2cMZWRkcOutt3LzzTeTkZHB5MmTeeihh4Dg7OK1a9cCsGjRIl5++WWSkpJIT09n/vz5/OIXv2DJkiVAUFMYN24cEPQjdGTJkiXcd999AGzYsIF169bF8N1JR1ZuL2PBxOFkpqnjX/oeJYUYW7BgAXPnzuX+++/n3nvv5a677mLevHnMnj2bxx4LulPS0tKYMGECJ5xwAhA0J1VVVTFnzhwAvvGNb3DttdeyYMECmpqaOnytL37xi1RXVzNz5ky+/e1vc9xxx8X+Dcr7lNU0MC57cKLDEOkWC1pt+qb8/Hw/dJGdTZs2MXPmzARF1DfoGMXWnBv+yicWjueG8xLTfClyJGb2uru3ewKTagoiUdTU3EJVfRPDBmsWWumblBREoqiyPmjey85QUpC+qV8mhb7cJBZrOjaxVV57AFBSkL6r3yWF9PR0SktL9eXXjtb1FNLT0xMdSr9VXhecQ5I9eFCCIxHpnn43Zm78+PEUFBRQUlKS6FB6pdaV1yQ2KsKkMEw1Bemj+l1SSE1N1apikjAVta01BSUF6Zv6XfORSCK916eg5iPpm5QURKKotU9haHq/q4TLAKGkIBJF5bWNZKWnkKLpsqWP0idXJIoq6ho1HFX6NCUFkSgqrz2g4ajSpykpiERRuWoK0scpKYhEUUVto+Y9kj5NSUEkisrrlBSkb4tZUjCzu82s2Mw2RJQ9YGZrwsuO1mU6zSzPzOoitt0Rq7hEYqWlxYM+BTUfSR8Wy8HUvwJ+Bvy6tcDdL2y9bWY3AxUR+2919/kxjEckZtydqoYmWlzzHknfFss1mleYWV5728zMgE8Dp8Xq9UXi6dK7XqUlnIRR8x5JX5aoPoXFwF533xJRNtnM3jCzF8xscUcPNLMrzGyVma3SpHfSW6wrKOflraWA5j2Svi1RSeFi4P6I+0XARHdfAFwN3GdmQ9t7oLsvd/d8d8/Pzc2NQ6gih1ff2Ny2uA5o3iPp2+KeFMwsBfg48EBrmbs3uHtpePt1YCtwdLxjE+mO4soGAM6ePYas9BQmjcxIcEQi3ZeIWbvOADa7e0FrgZnlAmXu3mxmU4DpwLYExCbSZcVV9QBcfPxEfn7JQpKSLMERiXRfLIek3g/8A5hhZgVmdnm46SIObjoCWAKsC4eo/h640t3LYhWbSDTtDWsKo7LSlBCkz4vl6KOLOyi/rJ2yh4GHYxWLSCy11hRGZaUlOBKRntMZzSI9tLeygdRkY7g6mKUfUFIQ6aHiqnpyM9V0JP2DkoJID5VUNTBqaHqiwxCJCiUFkR7aW1mv/gTpN5QURHqouKqBUUOVFKR/UFIQ6YH6xmbKaxsZnaXmI+kflBREeqCkKjxHQTUF6SeUFER6oLgtKaimIP2DkoJID+yp0Ilr0r8oKYh0U31jMz99bgvZGankjRyS6HBEokJJQaQLVu/cT1nNAQC++6c32bynilsunM+QtETMLSkSfUoKIp1UVd/Ihb/4Bzf+dTOV9Y088NouPnviJE6dMSrRoYlEjZKCSCet3FZGY7Pz7KZiXtqyj+YW59y5RyU6LJGoUlIQ6aSXtgTLvxZXNXD781vJSk9h4cTsxAYlEmVKCiKd9OI7+5g7fhhmsL6wgpOn5ZCSrH8h6V/0iRbphN3ldWwrqeG8eUcxb3w2AKccrTXCpf/RkAmRTvjrxj0AnDw9hwPNLawvrOCUGUoK0v/EcjnOu82s2Mw2RJTdYGaFZrYmvJwTse1aM3vHzN4ys7NiFZdIVz38egHf//Mmjps0nBmjs/jnk6fwl68uZuywwYkOTSTqYtl89Cvg7HbKb3H3+eHlCQAzm0WwdvPs8DE/N7PkGMYm0ikPrtrFNQ+t5YQpI7jn84swMwalJHH06KxEhyYSEzFLCu6+Aijr5O7nA79z9wZ33w68AyyKVWwinfG3zcV88+F1LJ6ew13LPkCmTlCTASARHc1Xmdm6sHlpeFg2DtgVsU9BWPY+ZnaFma0ys1UlJSWxjlUGsD+tK2JExiCWX5pPeqoqrjIwxDsp3A5MBeYDRcDNXX0Cd1/u7vnunp+bq44+iZ1d+2uZmpvJ4EFKCDJwxDUpuPted2929xbgTt5rIioEJkTsOj4sE0mYgrJaxo9QZ7IMLHFNCmY2NuLuBUDryKTHgYvMLM3MJgPTgVfjGZtIpIamZooq65kwPCPRoYjEVcx6zszsfmApkGNmBcB/AUvNbD7gwA7gXwDcfaOZPQi8CTQBX3b35ljFJnIku8vrcYcJI5QUZGCJWVJw94vbKb7rMPv/N/DfsYpHpCt2ldUCMFFJQQYYTXMh0o5d+4OkMEF9CjLAKCmItGNnWS2DkpMYnaW1l2VgUVIQaUdBWR3jhg8mKckSHYpIXCkpiLRj1/5adTLLgKSkINKOnWW1TBiu/gQZeJQURA6xv+YA5bWNqinIgKSkIHKIJzYUAXDS1JEJjkQk/pQURELuDsBDqwqYMTqLOeOGJTgikfjTXMAiBCernfezlzh5ei5rdpXzHx+ZiZlGHsnAo5qCDCi7ympZdverlFQ1HFT+u9d2Ul7XyB/X7iY5yTh/frszt4v0e6opyIBy81Nv8cLbJax4u4RPHDcegKbmFh5aVcBpM0bxmRMnUV57gNystARHKpIYqinIgPFOcRWPrd0NwPrCirby5zYXU1zVwEWLJnLqjFFcsGB8okIUSTjVFGRAaG5xfvDEZjJSk5kwIoO1BeVt23732i5GZaVx6gwt2iSimoL0ey0tzvWPrue5zcVc86EZnDwthzd3V9LY3EJRRR3Pv1XMp/LHk5KsfwcR/RdIv/fc5mJ+99ourjp1Gp8/eTJzxg+joamFLXureWhVAS0OF+ZPTHSYIr2CkoL0e2t2lZOcZFx12jQA5o3Pbit/4LVdnDwth4kjdfayCCgpyACwqaiSabmZpKcmAzBpZAZD01P47p82UlhexyXHq5Yg0ipmScHM7jazYjPbEFF2o5ltNrN1ZvaomWWH5XlmVmdma8LLHbGKSwaeN4sqmTk2q+2+mfHBaTkMTk3mxk/O5exjxyQwOpHeJZajj34F/Az4dUTZ08C17t5kZj8ErgW+GW7b6u7zYxiPDED7aw5QVFHPrKOGHlR+y4XzAdpqDyISiFlNwd1XAGWHlD3l7k3h3ZWABoRLTG0qqgRg1tiD5zFKT01WQhBpRyL7FD4P/CXi/mQze8PMXjCzxR09yMyuMLNVZraqpKQk9lFKn/ZmmBQim49EpGMJSQpmdj3QBNwbFhUBE919AXA1cJ+ZDW3vse6+3N3z3T0/N1cnG8nhvVlUyeihaYzM1LQVIp0R96RgZpcB5wKXeDhXsbs3uHtpePt1YCtwdLxjk/6juKqeqx9cw5Mb9jBrbLu/L0SkHXFNCmZ2NvAN4Dx3r40ozzWz5PD2FGA6sC2esUn/8teNe3lkdSGnHTOKaz40I9HhiPQZMRt9ZGb3A0uBHDMrAP6LYLRRGvB0OFf9Sne/ElgCfNfMGoEW4Ep3L2v3iUU6oaSqATP4yYXzNX2FSBccNimY2cLDbXf31YfZdnE7xXd1sO/DwMOHey2RrthX3cCIjEFKCCJddKSaws3hdTqQD6wFDJgLrAJOjF1oIt23r6qBHHUui3TZYX9Gufup7n4qweigheGon+OABUBhPAIU6Y591Q3kZA1KdBgifU5n69Yz3H196x133wDMjE1IIj23r/qAagoi3dDZjub1ZvZ/wG/D+5cA62ITkkjP7atW85FId3Q2KVwGfBH4anh/BXB7LAIS6anaA03UHmhWUhDphiMmhfD8gb+EfQu3xD4kkZ7ZV3UAgJxM9SmIdNUR+xTcvRloMbNhR9pXpDcoqa4HIDdLNQWRrups81E1Qb/C00BNa6G7fyUmUYn0QElbTUFJQaSrOpsUHgkvIr3evuoGQDUFke7oVFJw93tiHYhItLQmhRFD1Kcg0lWdSgpmNh34f8AsgrObAXD3KTGKS6Tb9lU3MDwjlVRNcSHSZZ39r/klwRDUJuBUgiU2f3vYR4gkyL4qnbgm0l2dTQqD3f1ZwNz9XXe/AfhI7MIS6T6duCbSfZ3taG4wsyRgi5ldRTDvUWbswhLpvn3VDcwZn53oMET6pM7WFL4KZABfAY4DPgMsi1VQIj0RzHukTmaR7uhsTaHM3asJzlf4XAzjEemR+sZmqhua1Hwk0k2dTQp3m9l44DXgRWBF5KypIr1FSVV4joKSgki3dKr5yN1PIZgq+6dANvBnMzvicplmdreZFZvZhoiyEWb2tJltCa+Hh+VmZrea2Ttmtu5Iq76JtKckPEdBaymIdE+nkoKZnQxcA1xPMOroT8CXO/HQXwFnH1L2LeBZd58OPBveB/gwMD28XIFmYZVu2BfWFNR8JNI9nW0+eh54neAEtifc/UBnHuTuK8ws75Di84Gl4e17wuf+Zlj+a3d3YKWZZZvZWHcv6mSMIuyr1rxHIj3R2dFHOcB3CdZkftLMnjGz73XzNUdHfNHvAUaHt8cBuyL2KwjLDmJmV5jZKjNbVVJS0s0QpL9qneJipEYfiXRLZ/sUyoFtwHaC9ZqnAkt6+uJhrcC7+Jjl4VrR+bm5uT0NQfqZfdUNDBucSlpKcqJDEemTOtunsA24GRhB0NY/I+x87o69ZjY2fN6xQHFYXghMiNhvfFgmckT7aw7Q2NwSns2sWoJId3W2T2Gau7dE6TUfJzjx7X/C68ciyq8ys98BxwMV6k+QzvrQT1bwmeMnad4jkR7qbJ/CNDN7tnVoqZnNNbP/ONKDzOx+4B/ADDMrMLPLCZLBmWa2BTgjvA/wBEET1TvAncCXuvZWZKCqb2ympKqBV7aXBjUFraMg0m2drSncCXwd+AWAu68zs/uA7x/uQe5+cQebTm9nX6dzw1xFDlJZ1wjA+sIKcFiimoJIt3U2KWS4+6tmFlnWFIN4RLqssj5IClX1wUdSfQoi3dfZ5qN9ZjaVcKSQmX2SYBSSSMJVhDWFVupTEOm+ztYUvgwsB44xs0KCoamXxCwqkS6orDu40qqkINJ9nV2jeRtwhpkNIahd1AIXAe/GMDaRTmmtKWRnpFJe26iOZpEeOGzzkZkNNbNrzexnZnYmQTJYRjBC6NPxCFDkSFr7FD44NQdQn4JITxyppvAbYD/BsNIvEEyIZ8AF7r4mtqGJdE7r6KNLjp9I7YEmxgxNT3BEIn3XkZLCFHefA2Bm/0fQuTzR3etjHplIJ1XUNTI4NZmTpuVw0rScRIcj0qcdafRR27AOd28GCpQQpLeprGti6ODOjpkQkcM50n/SPDOrDG8bMDi8bwTnmw2NaXQinVBR18iwwamJDkOkXzhsUnB3TTUpvV5lfSND05UURKKhsyevifRaqimIRI+SgvR5lfWNDFVSEIkKJQXp8ypqVVMQiRYlBenTWlqcqoYmhqZr9JFINCgpSJ9W1dCEO2o+EokSJQXp01rPZlZSEIkOJQXp01onw1Ofgkh0xL0h1sxmAA9EFE0Bvg1kE8yvVBKWX+fuT8Q3OulrWifD03kKItER96Tg7m8B8wHMLBkoBB4FPgfc4u43xTsm6bta11JQTUEkOhLdfHQ6sNXdtS6DdMt7fQoafSQSDYlOChcB90fcv8rM1pnZ3WY2vL0HmNkVZrbKzFaVlJS0t4sMIK3NR6opiERHwpKCmQ0CzgMeCotuB6YSNC0VATe39zh3X+7u+e6en5ubG49QpZdqaGrmj2t3k52RypBBqimIREMiawofBla7+14Ad9/r7s3u3gLcCSxKYGzSB3z/T5tYW1DBDz8xl6QkS3Q4Iv1CIpPCxUQ0HZnZ2IhtFwAb4h6R9BmF5XX8ZuW7XHZSHmfNHpPocET6jYTUuc1sCHAm8C8RxT8ys/mAAzsO2SZykGfe3AvApSdOSnAkIv1LQpKCu9cAIw8puzQRsUjf9MymvUzJHcLU3MxEhyLSryR69JFIl1XWN7JyWylnzhqd6FBE+h0lBelzXnirhMZm58yZSgoi0aakIH1KS4tz99+3M3poGgsmtnsqi4j0gJKC9CmPvlHIGzvL+fpZx5CsYagiUaczfqRPqGlo4pHVBfzkmS3Mm5DNxxeMS3RIIv2SkoL0etUNTVy8fCXrCyuYNXYoP9LJaiIxo6Qgvdb9r+7kj2t3U1HXyOY9VdzxmeM4a/ZozJQQRGJFSUF6rd+ufJd3S2vJTEvhpk/N5exjdeaySKwpKUiv1NDUzNt7q7j85Cl868PHJDockQFDo4+kV3prTxWNzc6cccMSHYrIgKKkMICV1RzgneKqRIfRrvWFFQBKCiJxpuajAaipuYUv37eap97ciwFPX31K3OcQuu+VnYwZlsZpx7R/VvKGwkqGDU5lwojBcY1LZKBTTWEAemDVLv66cS+fPWESKclJ/PLv26P+GpX1jby0ZR/1jc3v21ZV38gNj2/kK/evYXd5XbuP31BYwbHjhmqkkUicKSkMMFX1jdzy9NssyhvBDefN5mPzj+L3rxewv+YAu8vreHFLCcVV9T1+nTue38pn7nqF4773NA+u2gVAY3ML7s5zm4s50NxCXWMz1z26HncHwN3532e2cPWDa3hrTxXHqulIJO7UfDTA3P/qTvZVH+CuZTMxMy4/eQoPripg0Q+eobE5+HI+/ZhR3HXZB3r0Om/sLGdyzhBys9K4/tH11Dc2c+uzW1gyPZfqhiZGD03jn0+ewn8/sYk3dpWzcOJwfvn3HdzyzNukJhuNzc788dlReMci0hVKCgPM2l0VTByRwbwJ2QDMGJPFN88+hr2V9UwblcnKbaU8tXEv1Q1NZKa99/HYV91AS4szamg6W0uqeXtPFdNGZTJ9dNb7XqOlxdlQWMH5C47imjNncO5PX+Lbj20kKz2FR94oxAw+e8IkPnnceH7wl028+PY+UpOS+N6f3+Ss2aP50SfnsWpHGUtnjIrXYRGRkJLCALB6535e37GfLyyZwqY9lRwz5uAv8i8undp2e/qoTP60rogVb5dwzpxghdTH1+7m+kfXc6CphTNmjeapjXvaahXXnHk0/3r69IOeb0dpDVUNTcwdl83wIYP4v2X5PPpGIV9aOpUrf/s6K7eV8eE5Yxk+ZBBzxg3jpXdKKK1pYFByEjd+ah5D01M5XdNiiyREwpKCme0AqoBmoMnd881sBPAAkEewJOen3X1/omLsL37xwlb+unEv584by459NZw796gO9z1u0nBGDBnEUxv3cM6csTyyuoCrH1zLgonZTByRwWNrdvOx+Uex7KQ87nppOzc//TYzxw7ljFmjcXcamlrahpPOnRD0CcwcO5SZY4cC8LN/WsiTG/awKG8EAIun53DHC9vYUlzNGbNGMzQ9NcZHQ0QOJ9E1hVPdfV/E/W8Bz7r7/5jZt8L730xMaJ3X1NzC37eWcsrRuYkO5X3cndd2BHn1tyvfpcVh5pj3N/m0SklO4rRjRvHXjXv4x9ZSvv3YRhbljeC+LxxPSnIS3//YsWSFX9w3fWoo75bW8m8PruHJry3hxic389I7pZw0dSTpqUlMa2eYa05mGp854b11lU+elsttf9tKeW0j58/rOFmJSHz0ttFH5wP3hLfvAT6WuFA677nNxSy7+9VeeSLY1pJqymoOAHD/q8EooGPCX+0d+cTC8dQ0NHHxnSsx4McXziMlOfioZEX8kk9PTea2f1pIS4vziZ+/zB/W7GZfdQOPr93N7KOGtT3mcBZOyiZjUDLDBqeqD0GkF0hkTcGBp8zMgV+4+3JgtLsXhdv3AO9rWDazK4ArACZOnBivWA9rf+2B8LoxwZG83yvbywDIG5nBjtJaBqcmM3FExmEfc+LUkTz/76eyYksJM8ZkMX54x/tPHJnBd84/ln9/aC2nHJ3LB6eN5AdPbO70mchpKclcecpUstJTGJTS236jiAw8iUwKJ7t7oZmNAp42s82RG93dw4TBIeXLgeUA+fn579ueCDUNwQla1Q1NCY7k/V7bXsaorDQ+edx4bnrqbY4ek9WpFcsmjszgMyMnHXE/gE8sHMeYoenMn5jN4NRkahqaOXfu2E7H+JVDOqpFJHESlhTcvTC8LjazR4FFwF4zG+vuRWY2FihOVHxdURMmg9qG95+9m0juzivby/jA5BGcOHUkAMe0M4S0p8yMk6fntN3/tzOPjvpriEh8JKS+bmZDzCyr9TbwIWAD8DiwLNxtGfBYIuLrquoDQVKo6WU1ha0l1RRV1HPC5BHMGZfNwonZnDFLQz1FpGOJqimMBh4N57VJAe5z9yfN7DXgQTO7HHgX+HSC4uuS1hpCzYHelRQeWV1IcpJx1rFjGJSSxCNf+mCiQxKRXi4hScHdtwHz2ikvBU6Pf0Q901pD6E01hZYW5w9vFLJkeg6jstITHY6I9BEa7hEFrTWE6gT2KRxoauHJDXt4+Z3gtI+V20vZXVHPBQvHJywmEel7En3yWr/QOvqoNgHNRw+8tpP7X93F9n01VNQ1kpmWwsrrTueB13aRmZbCh9SHICJdoJpCFLxXU4h/Urj9+a0UV9Zz9uwx/MdHZlLd0MStz27hT+uKuOgDE0hPTY57TCLSd6mmEAWH9ilU1DYyLCP2c/jsrznAjtJavnH2DL60dBruzuNrd7N8xTZSk43LF0+OeQwi0r+ophAF7zUfNbOhsIL533uKLXtjP+XF2oJyAOaH02CbGZ89MQ+Ajy8Yz9hhWspSRLpGNYUoiGw+2r6vBnfYUVrb7loD0bRmVzlmBy9u/9FwJtRLT+zc2cgiIpGUFKIgsvnovXmQDsT8ddfsKmf6qMyDJqlLS0nm38+aEfPXFpH+Sc1HPXSgqaVtwZmahua2GUkrYjw5nruzdld5W9ORiEg0KCn0UGstwSxoRtpfE5+awrulteyvbWT+hOExfR0RGViUFHqotT9h5JBB1DY0UxbWEGI9jfbmPUFH9uyjDr82gohIVygp9FDryKPcrHQONLewt7IegIq62NYUdpbVAJA3ckhMX0dEBhYlhR5qPWFt9NA0AAr31wGwvya2NYWdZbUMG5wal/MhRGTgUFLoodapLUZlBUmhqCJMCjHuU9hZVnfEFdRERLpKSaGHWjuaW2cibQnXgquoi3FNobSGiSOVFEQkupQUeqi1T2FU2HwEkGSxrSk0tzgF+1VTEJHoU1LooZpDmo8AJozIoL6xhfrG2Eylvbu8jqYWZ5KSgohEWdyTgplNMLO/mdmbZrbRzL4alt9gZoVmtia8nBPv2LojcvRRq8k5wYig8hgNS91VVgugmoKIRF0iagpNwDXuPgs4Afiymc0Kt93i7vPDyxMJiK3LahqaSDLIyRzUVjYlJxOIXRPSu61JQX0KIhJlcZ/7yN2LgKLwdpWZbQLGxTuOaKk50MSQQSlkpr13KCfnBjWFWCWFnWW1pCabZkEVkahLaJ+CmeUBC4BXwqKrzGydmd1tZn1i/oaahiaGpKUwJCIpTAmbj2I1/9HO0lrGD88gOcli8vwiMnAlLCmYWSbwMPA1d68EbgemAvMJahI3d/C4K8xslZmtKikpiVe4HappaCYjLZm0lCSSkwwzmBQ268Riqoum5hbWFpSTp6YjEYmBhCQFM0slSAj3uvsjAO6+192b3b0FuBNY1N5j3X25u+e7e35ubm78gu5AzYEmMtNSMDMyBiWTPTiVkUOCkUixaD56ZHUhBfvr+KfjtV6CiERfIkYfGXAXsMndfxxRPjZitwuADfGOrTtqGprIGBSsg5yZlsLwIYMYPCioOUT7BLaGpmb+99ktzBs/jDNmjorqc4uIQGIW2fkgcCmw3szWhGXXAReb2XzAgR3AvyQgti6raWjmqOxgOOqQtBSyBwdzEQ3PGNQ2jXa03LliG4Xldfzg43MIcquISHQlYvTRS0B732h9YgjqoSrqGjl6dDAEdc64YYwYEgxNzc5IpTyKNYUNhRX85JktfGTuWJZMz4na84qIRNJynD3Q2NzCnsp6xg8POn1vuXB+27bcrDTe2lNFY3MLqck9a6XbU1HPv97/BsOHDOL75x+rWoKIxIymueiBwv11NLd4uyeRffbEPHaW1XLPyzu6/fzPbtrLjX/dzCfveJmSqgZ+fslChg8ZdOQHioh0k2oKPbAzPLO4vTmIzpg5ilOOzuV/n9nCxxaMIycz7X37HM6TG4q48rerSU4yxg8fzH1fOJ6547OjEbaISIdUU+iB1ukmJrWz+pmZ8R8fmUlVQxN/eKOw08/Z3OL8ZX0RVz+4lvkTstn4nbN44eunKiGISFyoptADO0trSEtJOmiG1EjTR2cxa+xQnlhfxD8vngLAym2lrN65n89/cDLpqcm4Ow+9XkBzi1NZ18hvVr5Lwf468kZm8ItLjyM9NTmeb0lEBjglhR54t7SWCSMySDrMdBPnzBnDTU+9TVFFHaOz0rnukfVs21fD71cVcMelx/FuaS3f+P26tv0XTR7B9efM5MxZo0npYQe1iEhXKSn0wM6y2iOuafDhOWO56am3eXLDHqbmZrJtXw2XnZTHn9cXceldr5CSlMT0UZn837J83CEv5/1NUSIi8aKfohHcnVU7yjq1OI67s7Os9ojTV0/NzWTG6Cx++fcd/OSZt8nJHMS15xzDby5fRO2BZgrL6/jO+bOZNHKIEoKIJNyArinsLq/jb28Vs7O0lrSUJF58Zx9v7CxnUd4I7v7cBw6aDvtQ+6oPUHuguVOrn/3XR2fxtQfWsHpnOf962jTSUpI5ZsxQ7v/CCWzeU8VJU3Uymoj0DgM2Kby0ZR9fuvd1KuubGJScRGNLC7mZaXxh8WTu/vsOLrlzJTd/ej7TRmUe9Dh3p+ZAM2/tqQLaH3l0qJOm5fDcvy/liXVFfGTue1M8HTtuGMeOGxbdNyYi0gMDMin8Y2spy375KtNyM3noygVMj/jiT0oy8vNG8I3fr+Oc/32REUMG0djcQmZ6CvWNzeyvaeRAc0vb/pM6OYV1ZloKn/7AhKi/FxGRaBqQSeG4ScP58tKpfGHJFLLSU9+3/azZY1g4cTg/f/4dahqaSElOorq+icGpyQwfMojhGakkmTF0cErbeswiIv2BuXuiY+i2/Px8X7VqVaLDEBHpU8zsdXfPb2+bRh+JiEgbJQUREWmjpCAiIm2UFEREpE2vSwpmdraZvWVm75jZtxIdj4jIQNKrkoKZJQO3AR8GZhGs2zwrsVGJiAwcvSopAIuAd9x9m7sfAH4HnJ/gmEREBozelhTGAbsi7heEZW3M7AozW2Vmq0pKSuIanIhIf9fnzmh29+XAcgAzKzGzd3vwdDnAvqgEFl2Kq2sUV9f11tgUV9d0N65JHW3obUmhEIicIGh8WNYud8/tyYuZ2aqOzupLJMXVNYqr63prbIqra2IRV29rPnoNmG5mk81sEHAR8HiCYxIRGTB6VU3B3ZvM7Crgr0AycLe7b0xwWCIiA0avSgoA7v4E8EScXm55nF6nqxRX1yiuruutsSmurol6XH16llQREYmu3tanICIiCaSkICIibQZkUugt8yuZ2QQz+5uZvWlmG83sq2H5DWZWaGZrwss5CYpvh5mtD2NYFZaNMLOnzWxLeD08zjHNiDgua8ys0sy+lohjZmZ3m1mxmW2IKGv3+Fjg1vAzt87MFsY5rhvNbHP42o+aWXZYnmdmdRHH7Y5YxXWY2Dr825nZteExe8vMzopzXA9ExLTDzNaE5XE7Zof5jojd58zdB9SFYFTTVmAKMAhYC8xKUCxjgYXh7SzgbYI5n24A/r0XHKsdQM4hZT8CvhXe/hbwwwT/LfcQnIgT92MGLAEWAhuOdHyAc4C/AAacALwS57g+BKSEt38YEVde5H4JOmbt/u3C/4W1QBowOfy/TY5XXIdsvxn4dryP2WG+I2L2ORuINYVeM7+Suxe5++rwdhWwiUOm9eiFzgfuCW/fA3wscaFwOrDV3XtyVnu3ufsKoOyQ4o6Oz/nArz2wEsg2s7Hxisvdn3L3pvDuSoITQ+Oug2PWkfOB37l7g7tvB94h+P+Na1xmZsCngftj8dqHc5jviJh9zgZiUjji/EqJYGZ5wALglbDoqrD6d3e8m2giOPCUmb1uZleEZaPdvSi8vQcYnZjQgODkxsh/1N5wzDo6Pr3pc/d5gl+TrSab2Rtm9oKZLU5QTO397XrLMVsM7HX3LRFlcT9mh3xHxOxzNhCTQq9jZpnAw8DX3L0SuB2YCswHigiqrolwsrsvJJjK/MtmtiRyowf11YSMabbgjPfzgIfCot5yzNok8vh0xMyuB5qAe8OiImCiuy8ArgbuM7OhcQ6r1/3tDnExB//4iPsxa+c7ok20P2cDMSl0aX6lWDOzVII/9r3u/giAu+9192Z3bwHuJEZV5iNx98Lwuhh4NIxjb2t1NLwuTkRsBIlqtbvvDWPsFceMjo9Pwj93ZnYZcC5wSfhFQtg0Uxrefp2g3f7oeMZ1mL9dbzhmKcDHgQday+J9zNr7jiCGn7OBmBR6zfxKYVvlXcAmd/9xRHlkG+AFwIZDHxuH2IaYWVbrbYKOyg0Ex2pZuNsy4LF4xxY66NdbbzhmoY6Oz+PAZ8PRIScAFRHV/5gzs7OBbwDnuXttRHmuBYtbYWZTgOnAtnjFFb5uR3+7x4GLzCzNzCaHsb0az9iAM4DN7l7QWhDPY9bRdwSx/JzFowe9t10IeujfJsjw1ycwjpMJqn3rgDXh5RzgN8D6sPxxYGwCYptCMPJjLbCx9TgBI4FngS3AM8CIBMQ2BCgFhkWUxf2YESSlIqCRoO328o6OD8FokNvCz9x6ID/Ocb1D0Nbc+jm7I9z3E+Hfdw2wGvhoAo5Zh3874PrwmL0FfDiecYXlvwKuPGTfuB2zw3xHxOxzpmkuRESkzUBsPhIRkQ4oKYiISBslBRERaaOkICIibZQURESkjZKCyBGYWbMdPDNr1GbWDWfcTNQ5FSLv0+uW4xTphercfX6igxCJB9UURLopnGP/RxasOfGqmU0Ly/PM7LlwgrdnzWxiWD7agrUM1oaXk8KnSjazO8P58p8ys8EJe1My4CkpiBzZ4EOajy6M2Fbh7nOAnwE/Cct+Ctzj7nMJJp67NSy/FXjB3ecRzN2/MSyfDtzm7rOBcoIzZkUSQmc0ixyBmVW7e2Y75TuA09x9Wzhp2R53H2lm+wimamgMy4vcPcfMSoDx7t4Q8Rx5wNPuPj28/00g1d2/H4e3JvI+qimI9Ix3cLsrGiJuN6O+PkkgJQWRnrkw4vof4e2XCWbfBbgEeDG8/SzwRQAzSzazYfEKUqSz9ItE5MgGW7hoe+hJd28dljrczNYR/Nq/OCz7V+CXZvZ1oAT4XFj+VWC5mV1OUCP4IsHMnCK9hvoURLop7FPId/d9iY5FJFrUfCQiIm1UUxARkTaqKYiISBslBRERaaOkICIibZQURESkjZKCiIi0+f9iT3Q1MSfrGAAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "RUNS = 1\n",
        "NUM_EPOCHS = 200\n",
        "EPISODES_PER_RUN = 100\n",
        "\n",
        "env = gym.make(\"CartPole-v1\")\n",
        "\n",
        "##########################################################\n",
        "# TODO (10 points):\n",
        "# Implement the training loop for the offline SAC.\n",
        "# 1) You need to initialize an agent with the\n",
        "#    `replay_buffer` of the pretrained online agent. Set\n",
        "#     the `offline` flag and leave the `use_cql` flag to\n",
        "#     remain False.\n",
        "# 2) You can use `batch_deterministic_start` in the\n",
        "#    `train_networks` method to select all minibatches\n",
        "#    of the data to train in an offline manner.\n",
        "# 3) After each epoch, run `EPISODES_PER_RUN` validation\n",
        "#    episodes and plot the mean return over these\n",
        "#    episodes in the end.\n",
        "##########################################################\n",
        "agent_results = []\n",
        "offline_sac_agent = SACAgent(env, replay_buffer=online_sac_agent.replay_buffer, offline=True, use_cql=False)\n",
        "average_run_results = []\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "    print('\\r', f'Epoch: {epoch + 1}/{NUM_EPOCHS}', end=' ')\n",
        "    #train\n",
        "    minibatch_size = 1000\n",
        "    batch_deterministic_start = 0\n",
        "    while batch_deterministic_start < offline_sac_agent.replay_buffer.count:\n",
        "        minibatch = offline_sac_agent.replay_buffer.sample_minibatch(minibatch_size, batch_deterministic_start=batch_deterministic_start)\n",
        "        minibatch_separated = list(map(list, zip(*minibatch)))\n",
        "        \n",
        "        states_tensor = torch.tensor(np.array(minibatch_separated[0]))\n",
        "        actions_tensor = torch.tensor(np.array(minibatch_separated[1]))\n",
        "        rewards_tensor = torch.tensor(np.array(minibatch_separated[2])).float()\n",
        "        next_states_tensor = torch.tensor(np.array(minibatch_separated[3]))\n",
        "        done_tensor = torch.tensor(np.array(minibatch_separated[4]))\n",
        "        \n",
        "        batch_deterministic_start += minibatch_size\n",
        "        offline_sac_agent.train_on_transition(states_tensor, actions_tensor, next_states_tensor, rewards_tensor, done_tensor)\n",
        "    \n",
        "    \n",
        "    #validation\n",
        "    run_results = []\n",
        "    for episode_number in range(EPISODES_PER_RUN):\n",
        "        episode_reward = 0\n",
        "        state = env.reset()\n",
        "        done = False\n",
        "        i = 0\n",
        "        while not done and i < STEPS_PER_EPISODE:\n",
        "            i += 1\n",
        "            action = offline_sac_agent.get_next_action(state, evaluation_episode=True)\n",
        "            next_state, reward, done, info = env.step(action)\n",
        "            episode_reward += reward\n",
        "            state = next_state\n",
        "        run_results.append(episode_reward)\n",
        "    average_run_results.append(np.mean(run_results))\n",
        "    \n",
        "\n",
        "env.close()\n",
        "\n",
        "x_axis = list(range(len(average_run_results)))\n",
        "\n",
        "plt.plot(x_axis, average_run_results, label='Reward')\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Reward')\n",
        "plt.title('Offline SAC')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJ8HlBr0kkZv"
      },
      "source": [
        "## Conservative SAC training loop (5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VOIESLFwXs7Q"
      },
      "source": [
        "Similar to the previous part, you are going to train another offline agent. In this part, you are going to use the conservative version of SAC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "vP_L7h0bYD4V"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Epoch: 200/200 "
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwiUlEQVR4nO3deXxddZ3/8dcnSZM0bdqkaVq6py1tKdDSlrIUaQEBWcYBwQ3GnxZl7KAy6ujogIyO+nPm5wb+figDwoCgAiqiA46A7BZksy3d6EL3Nmmardn35fP745yEELK19N5zk/t+Ph73kXu/59x7Pjm5uZ/7Xc73a+6OiIgIQErUAYiISOJQUhARkS5KCiIi0kVJQUREuigpiIhIFyUFERHpoqQgIiJdlBRkyLPAz8ys0sxeC8s+Y2YlZlZnZnlm5mZ2fLjtDjP7erRRiyQmJQVJeGZ2jZltMrMGMztkZrebWU63Xc4GLgSmuvvpZjYCuAV4n7uPdveK7q/n7te5+/+OUayTzOxuMys2s1oz22Zm3zKzUeF2M7OvmNkOM2s0s/1m9h9mlt7tNe41s+8M8njXhseoDZPgY2aW3WOfc8Ok+C9HGq8kHyUFSWhm9mXge8BXgLHAmcAM4KluH6QzgL3uXh8+nghkAm/EOdZxwMvASGCZu2cTJKscYHa4263AKuATQDZwCXAB8KujON45wH8AV4fHmg/8upddVwKHw2MeabySbNxdN90S8gaMAeqAj/QoHw2UAZ8CrgWagPZw3weBesDDx8+Gz3Hg+PD+vcB3wvvnAoXAl4FSoBj4ZLdjZQA/BPYDJcAdwMg+4v0OsAlI6WP7nDDO03uUTwOagXN6xjfA+fln4L8H2GcUUAtcBbQASwcbr27JeVNNQRLZWQTf+H/XvdDd64DHgAvd/W7gOuBlD5qKrgZOCnfNcff3DuI4xxHUQqYQJJnbzCw33PZdYC6wCDg+3OcbfbzOBcDv3L2jj+3nA4Xu/lqP3+cA8ArwvkHE2t2rwEVhc897zCyjl32uJEiODwF/Iqg1DDZeSUJKCpLIxgPl7t7Wy7bicPux0Ap8291b3f0xgg/ReWZmBE09/+Tuh929lqC55qo+XicvjKsv4/vZXgzkH0nQ7v4CwYf+EuCPQIWZ3WJmqd12Wwn82t3bgQeAq8I+l8HEK0lISUESWTkw3szSetk2Kdx+LFT0SDwNBE1U+UAWsNbMqsysCniCvj+8K8K4+lLez/aj+n3c/XF3/1tgHHA5cA3w9wBmNg04D7g/3P0RgprX3wwyXklCSgqSyF4maGu/snuhmY0m6KB9JsbHLwcagZPcPSe8jXX30X3s/zRwhZn19X/1LDDNzE7vXhh+eJ8JPH+0gbp7h7s/Ex7j5LD44wT/438ws0PAboKk0NmENFC8koT0ZpCE5e7VwLeAH5vZxWY2wswKgN8QdA7/IsbH7wDuAn5kZhMAzGyKmV3Ux1NuIegcv8/MZnTb/xYzW+jubxJ0VN9vZmeaWaqZnQQ8DLxE8CHdKdXMMrvd0nsezMwuN7OrzCw3HOp6OnAOQf8EBB/+3yLoD+m8fRC41MzyBor3aM6ZDH1KCpLQ3P37wNcIRgDVEHSuHgDOd/fmOITwL8BO4BUzqyH44J7XR6yHCTrHW4FXzayWoDZTHb4GwPXAfwG/JGim2gzsAz7Qo8P3BoJaSuft2V4OWQl8GthBcG5+CfzA3e83s86hu7e5+6Fut0fDWK4eZLySZMxdK6+JRMXMvgVcAaxw96qIwxFRUhCJmpldD+x09yeijkVESUFERLqoT0FERLr0Nv57yBg/frwXFBREHYaIyJCydu3acnfv9XqbIZ0UCgoKWLNmTdRhiIgMKWa2r69taj4SEZEuSgoiItJFSUFERLoM6T6F3rS2tlJYWEhTU1PUoSSkzMxMpk6dyogRIwbeWUSSzrBLCoWFhWRnZ1NQUEAw87F0cncqKiooLCxk5syZUYcjIgkoZs1HZjbNzJ4zsy1m9oaZfSEsH2dmT4Vr1D7VuZhJOKHXrWa208w2mtmSozluU1MTeXl5Sgi9MDPy8vJUixKRPsWyT6EN+LK7n0gwLfDnzOxEgom+nnH3OQSTb90Q7n8JwXKFcwgWNrn9aA+shNA3nRsR6U/Mmo/cvZhwVSd3rzWzrQRLGV5OsC4uwH0Ec8j/S1j+cw/m3XjFzHLMbFL4OiKRampt5/ntZewur6OppT3qcESYe1w27184+Zi/blz6FMI58BcTTHs8sdsH/SFgYnh/CsGUyJ0Kw7K3JQUzW0VQk2D69OmxC/pdSE1NZcGCBbS1tTFz5kx+8YtfkJOTE/c4zj33XH74wx+ydOnSuB97uLntuZ38+NlgNmlVtiQRvH/h5KGZFMJVsh4GvujuNd2bL9zdzeyIZuRz9zuBOwGWLl2akLP5jRw5kvXr1wOwcuVKbrvtNm666aaYHrOtrY20tGE3biBhvLiznFOmjuX+T5/J6AydZxm+YnqdQrhA+MPA/e7+u7C4xMwmhdsnAaVheREwrdvTp4ZlQ9qyZcsoKgp+jV27dnHxxRdz6qmnsnz5crZt20Z7ezszZ87E3amqqiI1NZXVq1cDsGLFCnbs2MFrr73GsmXLWLx4MWeddRbbt28H4N577+Wyyy7jve99L+effz6NjY1cddVVzJ8/nyuuuILGxsbIfu/hpL65jY2F1Zw9Z7wSggx7MXuHW1AluBvY6u63dNv0KMEygd8Nfz7Srfx6M/sVcAZQ/W77E771hzfYcrDm3bzEO5w4eQz/9rcnDWrf9vZ2nnnmGa699loAVq1axR133MGcOXN49dVX+exnP8uzzz7LvHnz2LJlC3v27GHJkiW88MILnHHGGRw4cIA5c+ZQU1PDCy+8QFpaGk8//TRf+9rXePjhhwFYt24dGzduZNy4cdxyyy1kZWWxdetWNm7cyJIlRzWAS3r4697DtHc4Z87KizoUkZiL5dee9xAsHL7JzNaHZV8jSAa/MbNrCZYh/Ei47THgUoJlABuAT8YwtphqbGxk0aJFFBUVMX/+fC688ELq6up46aWX+PCHP9y1X3NzsJrk8uXLWb16NXv27OHGG2/krrvu4pxzzuG0004DoLq6mpUrV7Jjxw7MjNbW1q7XuPDCCxk3bhwAq1ev5vOf/zwACxcuZOFCLbN7LLyy+zAjUo1TZ+RGHYpIzMVy9NGLQF9dcuf3sr8DnzuWMQz2G/2x1tmn0NDQwEUXXcRtt93GNddcQ05OTldfQ3crVqzg9ttv5+DBg3z729/mBz/4Ac8//zzLly8H4Otf/zrnnXcev//979m7dy/nnntu13NHjRoVp98qeVQ3tPKhO17ia5fO57wTJvDy7gpOmZpDVrqajmT409xHMZSVlcWtt97KzTffTFZWFjNnzuShhx4CgquLN2zYAMDpp5/OSy+9REpKCpmZmSxatIif/vSnrFixAghqClOmTAGCfoS+rFixggceeACAzZs3s3Hjxhj+dsPXoxuK2FFaxz1/2UNpTRObi6pZNltNR5IclBRibPHixSxcuJAHH3yQ+++/n7vvvptTTjmFk046iUceCbpTMjIymDZtGmeeeSYQNCfV1tayYMECAL761a9y4403snjxYtra2vo81mc+8xnq6uqYP38+3/jGNzj11FNj/wsOQw+tLQTgLzvL+f6fttPhzgeXTI04KpH4GNJrNC9dutR7LrKzdetW5s+fH1FEQ4POUd+2Ftdwyf97gZXLZnDfy8E6JBedNJGfflzXesjwYWZr3b3XN7VqCpL0HttUzL1/2QPA79YVkp6awhcvmMvi6TkAfHr5rAijE4kv9ZxJ0rv5ye2U1DTz8WUFvLrnMKfOyCV3VDr//L55vLCjXKOOJKkMy6Tg7pr4rQ9DubkwFg4cbmBXWT0AGwqr2HKwhlUrgprBe44fz3uOHx9leCJxN+yajzIzM6moqNCHXy8611PIzMyMOpSEsXpHWdf9+17aS1uHs3i6agaSvIZdTWHq1KkUFhZSVlY28M5JqHPlNQk8v72MKTkjaWnv4I8bgwvoO/sSRJLRsEsKI0aM0KpiMigtbR28tLOcDyyewuH6Fh7ffIjp47IYPzoj6tBEIjPsmo9EBusvO8upb2nn3HkTujqTVUuQZDfsagoig/Xga/vJG5XOOXPzeeNgOgBL1J8gSU5JQZJSaU0Tz2wr5e+XzyQ9LYVF03L4f1ct4sITJw78ZJFhTElBktJDawtp73CuOi1Yvc/MuHzRlIijEome+hQkKT29tYQl03OYOV6zzIp0p6QgScfd2VFSx8lTxkYdikjCUVKQpHOwuom65jbmTMyOOhSRhKOkIEnnzZJaAOZOGB1xJCKJJ2ZJwczuMbNSM9vcrezXZrY+vO3tXKbTzArMrLHbtjtiFZfIjs6koJqCyDvEcvTRvcBPgJ93Frj7Rzvvm9nNQHW3/Xe5+6IYxiPDzIYDVcyeMJrRGUf2Nt5RUsf40RnkjkqPUWQiQ1fMagruvho43Ns2C6Yw/QjwYKyOL8NbfXMbH7z9JX75yr4jfu6bpXXMnaimI5HeRNWnsBwocfcd3cpmmtnrZvZnM1ve1xPNbJWZrTGzNZr0LnkdqGygrcM5WNV4RM9zd3aW1KrpSKQPUSWFq3l7LaEYmO7ui4EvAQ+Y2Zjenujud7r7Undfmp+fH4dQJREVHg6SQXldMwC/f72Q3WV1ve7r7uwpD9ZMKKpqpL6lnTmqKYj0Ku5JwczSgCuBX3eWuXuzu1eE99cCu4C58Y5Nho7CygYAymqbaWnr4Mu/2cAv+mhK+j+Pb+O8Hz7P5qJqNhfVAHDCcaopiPQmimkuLgC2uXthZ4GZ5QOH3b3dzGYBc4DdEcQmQ0RhZWdNoYXS2iY6PLjf06//up87Vwdvpee2lXKopolR6aksmJITz3BFhoyYJQUzexA4FxhvZoXAv7n73cBVvLODeQXwbTNrBTqA69y9105qEXgrKZTVNlNc3QRAeW3z2/bZcKCKf/3vzSyfM56KuhZW7yijpKaZZbPzSE/TJToivYlZUnD3q/sov6aXsoeBh2MViww/hVVB81Fdcxt7wjWWK+rfSgrVDa189v51TMjO5MdXL+a/XtjDbc/vxB2uPVuLMIn0RV+XZEgqrGwkKz0VgE1FweUu3ZuPfvd6IUVVjdx69WJystJZMTefzmW7V8zVAAWRvigpyJBT29RKVUMrC6cGE9p1JoXKhhba2jsAeH1/FceNyXzbimrZGWlMzR1JQV5WNIGLDAFaT0GGnKLw2oTF03N5ZfdhthQHI4rc4XBDCxOyM9lQWMWiaTldzxmRmsI/XzSP0RlpBNdOikhvlBRkyOm8RmFx+KHf0tbRta2iroW0lBT2VTR0LaDTaeVZBfEKUWTIUvORDCmPbjjI45sPAXDKtBw6v/R3NgmV1zWzobAK4G01BREZHNUUZMgorW3i8w++DsCYzDQmZGeQm5XO4foWTpoylr0VDVTUtbCnvJ4Uo6vPQUQGTzUFGTIO1weji7584Vx+c90yzIz80RkALAhXUSuva2b9gSrmTMhm1BHOnioiSgoyhFTWtwJwakEuJxwXTI01PjuY/nrexGzSU1Moqw2SgpqORI6OkoIMGdWNQU0hZ+Rb6yB01hQm5WSSNzqdF3eWU93YyhmzxkUSo8hQp6QgQ0ZlQ1BTyMka0VU2vjMpjBnJ+NEZvHEwGJ66bHZe/AMUGQbU6CpDRlWYFHKz3qopXLLgOFrbOxgzMo280UF5QV4Wk8aOjCRGkaFOSUGGjKrGFtLTUsgc8VYF99QZ4zh1RtBU1FlrUC1B5Oip+UiGjKr6VnKzRvR5RXJnTeHMWUoKIkdLSUGGjKrGlrd1Mvc0LTeLEanGMiUFkaOm5iMZMiobWhnbrZO5pw8vncpZs/OYMCYzjlGJDC+qKciQUd0QNB/1JSMtlVn5WntZ5N1QUpAho7Kh/+YjEXn3YpYUzOweMys1s83dyr5pZkVmtj68Xdpt241mttPMtpvZRbGKS4Ymd6eqsfVt1yiIyLEXy5rCvcDFvZT/yN0XhbfHAMzsRIK1m08Kn/OfZpYaw9hkiGlq7aClrYOcLNUURGIpZknB3VcDhwe5++XAr9y92d33ADuB02MVmww9lQ3hFBeqKYjEVBR9Cteb2caweSk3LJsCHOi2T2FY9g5mtsrM1pjZmrKysljHKgnirauZlRREYineSeF2YDawCCgGbj7SF3D3O919qbsvzc/XAuzD2Q/+tI0Tvv44Z3/vWf78ZvAFYKw6mkViKq5Jwd1L3L3d3TuAu3iriagImNZt16lhmSSxF3eUk5+dweH6Fu5+cQ+g5iORWItrUjCzSd0eXgF0jkx6FLjKzDLMbCYwB3gtnrFJtNraO3h8UzEdHQ4Eo432lNdzztx83nvCBMrrmoG3T4YnIsdeLIekPgi8DMwzs0Izuxb4vpltMrONwHnAPwG4+xvAb4AtwBPA59y9PVaxSeL51V8P8Jn71/GXXeVA0IdQ09RGQd4oLjn5re8SqimIxFbMprlw96t7Kb67n/3/Hfj3WMUjicvd+fnLewHYUVLH8jn57K2oB6AgbxTLZueRkRZ8f8kcoZHKIrGkK5olcq/sPsybJXUA7CoLfu6raACgYHwWozLSOH/+BI4bqzmNRGJNE+JJ5H756j5ys0YwOWckO0uDpLC3oh4zmJqbBcC/f2ABVY2tUYYpkhSUFCRymwqrOXtOPiNHpPDstmDo6d7yeiaPHdnVXJQ7Kp3cUepkFok1NR9JpDo6nEPVTUzOyWR2/mjK65qpbmhlb0UDBeOzog5PJOkoKUikyuubaWnvYErOSI6fEEx7vbOsjn0V9czIGxVxdCLJR0lBIlVc1QTApLEjmR2uhfD6/koqG1opyFNNQSTelBQkUsXVjQBMGpvJ1NyRpKem8LO/7AXoqjmISPwoKUikisKawpSckaSlpjBz/CiKqhq5cvEUzpk7IeLoRJKPRh9JpIqrGskckdJ1pfLHl82gpKaJf7pgLikpFnF0IslHSUEidbC6kcljR2IWJID/deaMiCMSSW5qPpJIHaxqYnLOyKjDEJGQkoJEqri6kUmavkIkYSgpSGRa2joorW1mkmoKIglDSUEiU1LThDtMyVFNQSRRqKNZIvFmSS2PrA8W15s0VjUFkUShpCCRuO4Xa9ldXk9WeipzJ2ZHHY6IhJQUJO46OpwDlQ1cc1YBX7t0PulpasUUSRSxXI7zHjMrNbPN3cp+YGbbzGyjmf3ezHLC8gIzazSz9eHtjljFJdErr2umtd2ZlT9KCUEkwcTyP/Je4OIeZU8BJ7v7QuBN4MZu23a5+6Lwdl0M45KIHax+axI8EUksMUsK7r4aONyj7El3bwsfvgJMjdXxJXEVV701CZ6IJJYo6+6fAh7v9nimmb1uZn82s+V9PcnMVpnZGjNbU1ZWFvso5ZjrrCnoSmaRxBNJUjCzm4A24P6wqBiY7u6LgS8BD5jZmN6e6+53uvtSd1+an58fn4DlmCquaiQjLYXccBI8EUkccU8KZnYN8H7gY+7uAO7e7O4V4f21wC5gbrxjk/gorg7mO+qcBE9EEkdck4KZXQx8FbjM3Ru6leebWWp4fxYwB9gdz9gkfg5qviORhBXLIakPAi8D88ys0MyuBX4CZANP9Rh6ugLYaGbrgd8C17n74d5eV4a+4qomjTwSSVD9XrxmZkv62+7u6/rZdnUvxXf3se/DwMP9HUuGh9b2Dkpqm5is+Y5EEtJAVzTfHP7MBJYCGwADFgJrgGWxC02Go85J8FRTEElM/TYfuft57n4eweigJeGon1OBxUBRPAKU4aW488I11RREEtJg+xTmufumzgfuvhmYH5uQZDg7GF64Nlk1BZGENNgJ8TaZ2X8BvwwffwzYGJuQZDhbf6CKjLQUCsZnRR2KiPRisEnhGuAzwBfCx6uB22MRkAxvr+w+zNKCXDLSUqMORUR6MWBSCK8feDzsW/hR7EOS4aqqoYVth2r40gW6LlEkUQ3Yp+Du7UCHmY2NQzwyjL265zDucObsvKhDEZE+DLb5qI6gX+EpoL6z0N0/H5OoZFh6ZXcFmSNSWDhV3y9EEtVgk8LvwpvIUXtl92GWzhin/gSRBDaopODu98U6EBneOjqcnaW1nHP2rKhDEZF+DCopmNkc4P8AJxJc3QyAu+s/XAaltDZYgnNqrq5PEElkg7147WcEQ1DbgPOAn/PWNQsiAyqsDCbFVVIQSWyDTQoj3f0ZwNx9n7t/E/ib2IUlw01hZXAl89RcXbQmksgG29HcbGYpwA4zu55g3qPRsQtLhhvVFESGhsHWFL4AZAGfB04F/hewMlZByfBTVNXI+NEZZI7QyCORRDbYmsJhd68juF7hkzGMR4apwspGpqiWIJLwBpsU7jGzqcBfgReA1d1nTRUZSGFlIydOHhN1GCIygEE1H7n7OQRTZf8YyAH+aGYDLpdpZveYWamZbe5WNs7MnjKzHeHP3LDczOxWM9tpZhsHWvVNho6ODqeoslH9CSJDwKCSgpmdDXwZuIlg1NH/AJ8bxFPvBS7uUXYD8Iy7zwGeCR8DXALMCW+r0Cysw0ZZXTMt7R0aeSQyBAy2+eh5YC3BBWyPuXvLYJ7k7qvNrKBH8eXAueH9+8LX/pew/Ofu7sArZpZjZpPcvXiQMUqC0sgjkaFjsKOPxgPfJliT+Qkze9rM/vdRHnNitw/6Q8DE8P4U4EC3/QrDsrcxs1VmtsbM1pSVlR1lCBJPndcoTFNSEEl4g+1TqAJ2A3sI1mueDax4twcPawV+hM+5M1wreml+fv67DUFirLW9g/te2kt2Rpqaj0SGgMHOfbQb2Aa8SNDW/8nBNiH1oqSzWcjMJgGlYXkRMK3bflPDMhnCbn7yTdbtr+LHVy/WNQoiQ8Bg+xSOd/eOY3TMRwkufPtu+PORbuXXm9mvgDOAavUnDG2t7R3c/eJuPrBoMn97yuSowxGRQRhsn8LxZvZM59BSM1toZv860JPM7EHgZWCemRWa2bUEyeBCM9sBXBA+BniMoIlqJ3AX8Nkj+1Uk0RysaqS13Tlr9vioQxGRQRpsTeEu4CvATwHcfaOZPQB8p78nufvVfWw6v5d9ncENc5UhYl9FMOpoep76EkSGisHWFLLc/bUeZW3HOhgZeppa26luaO11277DQVKYoaQgMmQMNimUm9lswpFCZvYhglFIkuR+8KftfOA//0JQ0Xu7/RX1pKelMDE7s5dnikgiGmzz0eeAO4ETzKyIYGjqx2IWlQwZ+yoa2FNez4HDje9oJtpX0cD0cVmkpFhE0YnIkRrsdQq73f0CIB84ATgHODuWgcnQUNkQjEz+6953ToW1/3ADM8ap6UhkKOk3KZjZGDO70cx+YmYXAg0Ew0h3Ah+JR4CS2PpKCu7O/sMN6mQWGWIGaj76BVBJMKz00wQT4hlwhbuvj21oMhRUhZ3Mr/VICmV1zTS0tKumIDLEDJQUZrn7AgAz+y+CzuXp7t4U88gk4XV0OFUNLYxKT2V3WT3ldc2MH50BwP6KzpFHo6IMUUSO0EB9Cl1jDd29HShUQpBONU2tdDice8IEANZ0qy3oGgWRoWmgpHCKmdWEt1pgYed9M6uJR4CSuA7XB/0J58zJJz0thbX7Kru2vVlSS1qKabpskSGm3+Yjd9cMZtKnyrA/IX9MBgumjOX1/VVd257bXsppBePISNNbSGQoGezFayLvUBWOPMrNSmfxtBw2FVXT0tbBgcMNvFlSx/nzJ0QcoYgcKSUFOWqdzUfjstJZPD2X5rYOthbX8PTWEgAumD+xv6eLSAJSUpCj1jkcNWfUCJbMyAHg9f2VPLO1lNn5oygYr5FHIkPNYKe5EOny/PZSyutaqGxoIS3FyM5IY0zmCI4bk8kvX93PnvJ6/n75zKjDFJGjoJqCHLHbn9/Fv/9xC5UNLeRkpWMWzG20ZEYOO0vrOHV6Lp85Z3bEUYrI0VBNQY7Y7vJ6KhtaeeNgDblZI7rKrztnNvOPG8Oqc2Zp1JHIEKWkIEekpqmVstpmADYVVXPajHFd2xZOzWHh1JyIIhORYyHuScHM5gG/7lY0C/gGkEMwv1JZWP41d38svtHJQHaX1Xfdd4fcUSP62VtEhpq4JwV33w4sAjCzVKAI+D3wSeBH7v7DeMckg7e7rA6A7Iw0apvbyM1KjzgiETmWou5oPh/Y5e77Io5DBml3WT1pKdZ1YVqOkoLIsBJ1UrgKeLDb4+vNbKOZ3WNmub09wcxWmdkaM1tTVlbW2y5yFDo6vGtm0/7sKqtj+rgsFk8P/jzj1HwkMqxElhTMLB24DHgoLLodmE3QtFQM3Nzb89z9Tndf6u5L8/Pz4xFqUvjlq/s47+bnKazsPzHsLqtnVv4oTp4yFkDNRyLDTJQ1hUuAde5eAuDuJe7e7u4dwF3A6RHGlhTaO5zi6kbcnQde3U97h/e6rGb3/fdU1DMrfzSLp+XwrctO4qKTj4tjxCISa1EOSb2abk1HZjbJ3YvDh1cAmyOJKkm0tXfw2fvX8fTWEm645AS2HaoFYN2+Kq5YPPUd+3/rD2/whw0HaWnrYHb+KFJSjJVnFcQ5ahGJtUiSgpmNAi4E/qFb8ffNbBHgwN4e2+QY+/ojb/DklhLyRqXzH49tIyMthXnHZbNuf2Wv+z+x+RBZ6WksnzOG5XPUbCcyXEWSFNy9HsjrUfbxKGJJRoWVDTz42n4+9Z6Z/N0Z07jsJ3/hkpMnMTknk/98fhcNLW1kpafxjw++zvjR6Vx/3vEUVzfxr38zn79fPivq8EUkhnRFcxJ6YvMhAFaeNYMZeaP481fOY8zINF7aWUF7h7PhQDUnHJfNHzceJCcrnXPmBjWDEyePiTJsEYkDJYUk9MTmQ8yfNIYZecHU1vnZGQAsnp4DwLr9lRRXN9LhwZoJv1tXBMBJk8ZGEq+IxE/U1ylInJXUNLF2fyWX9DJqKCcrnROOy+bhdYU8tqmY7IzgO8MfNxUzNXckY7N0TYLIcKekkCTcnW8++gZ/d9cruNNrUgC44ZIT2F1Wz9NbS3n/KZOZkZdFe4dz8mTVEkSSgZJCkqhpbOPel/aSlpLC58+fw5yJ2b3ud+68CVyxeAoAF8yfwFmzg/EAJ6k/QSQpqE8hSVQ2BOspf3rFLD506juvQ+jum5edxOLpOZw7bwL1Le08+NqBriuYRWR4U1JIElWNwXrKuYPoFxg7cgSfWFYAwKUnH0fHRxexYq6uTRBJBkoKSaKzpnCks5qmpabwgbA5SUSGP/UpJInqhqCmkKMRRCLSDyWFJNFZU9CspiLSHyWFJFEV1hTGZKrFUET6pqSQJKoaWhiTmUZaqv7kItI3fUIkiarGVi2dKSIDUlJIEpUNrYMajioiyU1JIUlUN7QwVjUFERmAkkKSUE1BRAZDSSFJVDW0kDNSSUFE+hfZ+EQz2wvUAu1Am7svNbNxwK+BAoIlOT/i7r2vDymD1tbeQU1TmzqaRWRAUdcUznP3Re6+NHx8A/CMu88Bngkfy7tU09QGDG7eIxFJblEnhZ4uB+4L798HfCC6UIaPo533SESST5RJwYEnzWytma0Kyya6e3F4/xAwseeTzGyVma0xszVlZWXxinVIq+pKCqopiEj/opzz4Gx3LzKzCcBTZrat+0Z3dzPznk9y9zuBOwGWLl36ju3yTlVdk+GppiAi/YuspuDuReHPUuD3wOlAiZlNAgh/lkYV33BS2TD4tRREJLlFkhTMbJSZZXfeB94HbAYeBVaGu60EHokivu5qmlrZXFQddRhH7I8bi/nu40Hlq6v5aKRqCiLSv6hqChOBF81sA/Aa8Ed3fwL4LnChme0ALggfR+oXL+/jyttfoqWtI+pQjsgDr+3jjj/v4lB1E1UNraQYZGuGVBEZQCSfEu6+Gzill/IK4Pz4R9S3yvoWWto6KK9rZnLOyKjDGRR3Z2txLQB/euMQL+0qp2D8KFJSLOLIRCTRJdqQ1ITT2NoOQGltc8SR9G1veT2Lvv0kGw5UAUGsh+uDJqM7/ryLdfur+MSZMyKMUESGCiWFATS2hEmhpiniSPr2+OZDVDW08ocNBwHYUlwDwJmzxlFc3UR2RhofWjotyhBFZIhQUhhAZ02hJIFrCs9tL33bz61hUvjShfMAuOr0aYzOUH+CiAxMnxQD6EwKZQlaU6hubGXtvkryszPYVVbP/ooGthbXMjV3JKfPHMd9nzqd0wpyow5TRIYI1RQG0NCS2H0KL+4op73D+cpFQa3g+TdL2XKwmvmTxgBwztx8stKV+0VkcJQUBtDU2XyUoDWF57aXMiYzjSsXT6EgL4u7X9zDnvL6rqQgInIklBQGkOg1hXX7KjljVh5pqSl89tzjGZGaQlZ6GmcfPz7q0ERkCFK7wgAaEzgp1Da1sru8ng8sngLAR06bxkdO0ygjETl6qikMoLOjuaKumbb2xLqqecvBYJTRgiljI45ERIYLJYUBNLa0k5WeSodDRXhBWKLYHCaFk6ao/0BEjg0lhX64O42t7czIGwVAaU1iNSFtLqpm4pgMJmRnRh2KiAwTSgr9aGoNmosK8rIAKK1NrBFIm4uqOXmymo5E5NhRUuhHZ39CV00hgTqbG1ra2FVWx8nqTxCRY0hJoR8NLcGC99PHBTWFRLpWYWtxLR2OkoKIHFNKCv3ovHAtOzON48ZkUljZGHFEb9l+KJga+4TjsiOORESGEyWFfnReuDZyRCoz8rLYW14fcURv2VFaS1Z6KlOGyBoPIjI0xD0pmNk0M3vOzLaY2Rtm9oWw/JtmVmRm68PbpfGOrafOC9ey0lMpyBvF3oqGiCN6y46SOuZMGK2Fc0TkmIriiuY24Mvuvi5cp3mtmT0VbvuRu/8wgph61RA2H2WmpzJjfBbldc3UNbclxDTUb5bUsmJuftRhiMgwE/eagrsXu/u68H4tsBWYEu84BqOpR00BYF9F9E1I1Q2tlNY2M2fC6KhDEZFhJtI+BTMrABYDr4ZF15vZRjO7x8wiXwSgc0hqZ58CwL4EaELaURp0Ms+dqE5mETm2IksKZjYaeBj4orvXALcDs4FFQDFwcx/PW2Vma8xsTVlZWUxj7OpoTk/tulZhbwLUFN4sqQNgzkTVFETk2IokKZjZCIKEcL+7/w7A3Uvcvd3dO4C7gNN7e6673+nuS919aX5+bNvUm7rVFEZnpDF+dAb7yqOvKbxZEow8mjxWI49E5NiKYvSRAXcDW939lm7lk7rtdgWwOd6x9dR9SCoE011EXVNobmtnzb7DGnkkIjERxTCa9wAfBzaZ2fqw7GvA1Wa2CHBgL/APEcT2No2t7aSnppCWGuTOgvGjeHFHedyOX93QSnVjK42t7WwqqqaqoYWnt5awuaiG7165IG5xiEjyiHtScPcXgd6+4j4W71gG0tjSTuaItypTBXlZ/HZtE9UNrYzNGhGz45bWNPGt/9nCnzYfoq3D37YtPS2FWz5yClcumRqz44tI8op+wH0CC9ZSeOsUrZibzw+ffJP/2XSQj50xI2bH/eGT23lqSwkrzypg/qQxpKUYJ08ZQ352JhlpKWSGzVkiIseakkI/GlrbGZn+1gfwgiljmTtxNL9dWxizpFDT1MofNhRz5eIpfP39J8bkGCIifdHcR/1obGnv6mQGMDM+fOo0Xt9fxc7wWoFj7b9fL6KxtT2mNRERkb4oKfSjqUdNAeDyxZNJTTH+8cH1PLTmAO092vzfjdf3V3L3i3tYMGUsC6ZqSmwRiT81H/WjoaXtbX0KABOyM/nBhxZy23M7+cpvN/LQ2kL+70cXMfkoZittam3nuW2lrNtfyZ/fLOPNkjrGjhzBty8/+Vj9CiIiR0RJoR+NrR3kjX5np+6VS6ZyxeIp/HZtId/6wxau+dlrPHr92UfUAezuXPfLtTy/vYz01BSWFuTyd6dP50NLpyXEhHsikpz06dOPxpa2t/UpdGdmfHjpNMZnZ/DJn/2Vm5/czk1/M3DHcF1zG3vL61m3v5Lnt5dxwyUn8Mn3FJCRphFFIhI9JYV+NLa2k5Xe/4f1efMm8HdnTOeuF/bwPxuLuXLJFL54wVxGpL6zu2Z3WR0rf/YaBw4HK7gtm5XHquWzdGWyiCQMJYV+NLS0D6pJ6BvvP5ETjsvmhR3l3PbcLl7cWcFZs/Ooa2rjQGUDJ08eS2t7B7/66wFSU4zvfXABFfUtfGjJVCUEEUkoSgr9aBpETQEgc0Qqn1hWwCeWFfDI+iK+/8R27lq9m5EjUpmcM5IXdpRjwHknTOCmS+dTMH5U7IMXETkKSZkUthys4R9+uYaODuhwD2/BthEpRlpqCqkpRmu799mn0JfLF03h8kVTcA9e0MyoaWrFO4jp1BgiIsdCUiaF0RlpnDZjHGZGikFqihFM3uq0tjtt7R10OCydkculCycN+Hq9CV4vMCZTyUBEhoakTArT87K45aOLog5DRCTh6IpmERHpoqQgIiJdlBRERKSLkoKIiHRJuKRgZheb2XYz22lmN0Qdj4hIMkmopGBmqcBtwCXAiQTrNmulGRGROEmopACcDux0993u3gL8Crg84phERJJGoiWFKcCBbo8LwzIREYmDIXfxmpmtAlaFD+vMbPu7eLnxQPm7j+qYU1xHRnEduUSNTXEdmaONq8/1fhMtKRQB07o9nhqWdXH3O4E7j8XBzGyNuy89Fq91LCmuI6O4jlyixqa4jkws4kq05qO/AnPMbKaZpQNXAY9GHJOISNJIqJqCu7eZ2fXAn4BU4B53fyPisEREkkZCJQUAd38MeCxOhzsmzVAxoLiOjOI6cokam+I6Msc8Luuc919ERCTR+hRERCRCSgoiItIlKZNCosyvZGbTzOw5M9tiZm+Y2RfC8m+aWZGZrQ9vl0YU314z2xTGsCYsG2dmT5nZjvBnbpxjmtftvKw3sxoz+2IU58zM7jGzUjPb3K2s1/NjgVvD99xGM1sS57h+YGbbwmP/3sxywvICM2vsdt7uiFVc/cTW59/OzG4Mz9l2M7soznH9ultMe81sfVget3PWz2dE7N5n7p5UN4JRTbuAWUA6sAE4MaJYJgFLwvvZwJsEcz59E/jnBDhXe4HxPcq+D9wQ3r8B+F7Ef8tDBBfixP2cASuAJcDmgc4PcCnwOGDAmcCrcY7rfUBaeP973eIq6L5fROes179d+L+wAcgAZob/t6nxiqvH9puBb8T7nPXzGRGz91ky1hQSZn4ldy9293Xh/VpgK4k/rcflwH3h/fuAD0QXCucDu9x9XxQHd/fVwOEexX2dn8uBn3vgFSDHzI5uAfCjiMvdn3T3tvDhKwQXhsZdH+esL5cDv3L3ZnffA+wk+P+Na1xmZsBHgAdjcez+9PMZEbP3WTImhYScX8nMCoDFwKth0fVh9e+eeDfRdOPAk2a21oLpRQAmuntxeP8QMDGa0IDg4sbu/6iJcM76Oj+J9L77FMG3yU4zzex1M/uzmS2PKKbe/naJcs6WAyXuvqNbWdzPWY/PiJi9z5IxKSQcMxsNPAx80d1rgNuB2cAioJig6hqFs919CcFU5p8zsxXdN3pQX41kTLMFV7xfBjwUFiXKOesS5fnpi5ndBLQB94dFxcB0d18MfAl4wMzGxDmshPvb9XA1b//yEfdz1stnRJdj/T5LxqQw4PxK8WRmIwj+2Pe7++8A3L3E3dvdvQO4ixhVmQfi7kXhz1Lg92EcJZ3V0fBnaRSxESSqde5eEsaYEOeMvs9P5O87M7sGeD/wsfCDhLBppiK8v5ag3X5uPOPq52+XCOcsDbgS+HVnWbzPWW+fEcTwfZaMSSFh5lcK2yrvBra6+y3dyru3AV4BbO753DjENsrMsjvvE3RUbiY4VyvD3VYCj8Q7ttDbvr0lwjkL9XV+HgU+EY4OOROo7lb9jzkzuxj4KnCZuzd0K8+3YHErzGwWMAfYHa+4wuP29bd7FLjKzDLMbGYY22vxjA24ANjm7oWdBfE8Z319RhDL91k8etAT7UbQQ/8mQYa/KcI4ziao9m0E1oe3S4FfAJvC8keBSRHENotg5McG4I3O8wTkAc8AO4CngXERxDYKqADGdiuL+zkjSErFQCtB2+21fZ0fgtEgt4XvuU3A0jjHtZOgrbnzfXZHuO8Hw7/vemAd8LcRnLM+/3bATeE52w5cEs+4wvJ7get67Bu3c9bPZ0TM3mea5kJERLokY/ORiIj0QUlBRES6KCmIiEgXJQUREemipCAiIl2UFEQGYGbt9vaZWY/ZzLrhjJtRXVMh8g4JtxynSAJqdPdFUQchEg+qKYgcpXCO/e9bsObEa2Z2fFheYGbPhhO8PWNm08PyiRasZbAhvJ0VvlSqmd0Vzpf/pJmNjOyXkqSnpCAysJE9mo8+2m1btbsvAH4C/N+w7MfAfe6+kGDiuVvD8luBP7v7KQRz978Rls8BbnP3k4AqgitmRSKhK5pFBmBmde4+upfyvcB73X13OGnZIXfPM7NygqkaWsPyYncfb2ZlwFR3b+72GgXAU+4+J3z8L8AId/9OHH41kXdQTUHk3fE+7h+J5m7321Ffn0RISUHk3flot58vh/dfIph9F+BjwAvh/WeAzwCYWaqZjY1XkCKDpW8kIgMbaeGi7aEn3L1zWGqumW0k+LZ/dVj2j8DPzOwrQBnwybD8C8CdZnYtQY3gMwQzc4okDPUpiBylsE9hqbuXRx2LyLGi5iMREemimoKIiHRRTUFERLooKYiISBclBRER6aKkICIiXZQURESky/8HDsjO9AJ0MZQAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "RUNS = 1\n",
        "NUM_EPOCHS = 200\n",
        "EPISODES_PER_RUN = 100\n",
        "\n",
        "env = gym.make(\"CartPole-v1\")\n",
        "\n",
        "##########################################################\n",
        "# TODO (5 points):\n",
        "# Implement the training loop for the conservative SAC.\n",
        "# 1) You need to initialize an agent with the current\n",
        "#    `replay_buffer` of the online agent. Set the `offline`\n",
        "#     and `use_cql` flags.\n",
        "# 2) You can use `batch_deterministic_start` in the\n",
        "#    `train_networks` method to select all minibatches\n",
        "#    of the data to train in an offline manner.\n",
        "# 3) After each epoch, run `EPISODES_PER_RUN` validation\n",
        "#    episodes and plot the mean return over these\n",
        "#    episodes in the end.\n",
        "##########################################################\n",
        "agent_results = []\n",
        "offline_cql_sac_agent = SACAgent(env, replay_buffer=online_sac_agent.replay_buffer, offline=True, use_cql=True)\n",
        "average_run_results = []\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "    print('\\r', f'Epoch: {epoch + 1}/{NUM_EPOCHS}', end=' ')\n",
        "    #train\n",
        "    minibatch_size = 1000\n",
        "    batch_deterministic_start = 0\n",
        "    while batch_deterministic_start < offline_cql_sac_agent.replay_buffer.count:\n",
        "        minibatch = offline_cql_sac_agent.replay_buffer.sample_minibatch(minibatch_size, batch_deterministic_start=batch_deterministic_start)\n",
        "        minibatch_separated = list(map(list, zip(*minibatch)))\n",
        "        \n",
        "        states_tensor = torch.tensor(np.array(minibatch_separated[0]))\n",
        "        actions_tensor = torch.tensor(np.array(minibatch_separated[1]))\n",
        "        rewards_tensor = torch.tensor(np.array(minibatch_separated[2])).float()\n",
        "        next_states_tensor = torch.tensor(np.array(minibatch_separated[3]))\n",
        "        done_tensor = torch.tensor(np.array(minibatch_separated[4]))\n",
        "        \n",
        "        batch_deterministic_start += minibatch_size\n",
        "    \n",
        "        offline_cql_sac_agent.train_on_transition(states_tensor, actions_tensor, next_states_tensor, rewards_tensor, done_tensor)\n",
        "    \n",
        "    \n",
        "    #validation\n",
        "    run_results = []\n",
        "    for episode_number in range(EPISODES_PER_RUN):\n",
        "        episode_reward = 0\n",
        "        state = env.reset()\n",
        "        done = False\n",
        "        i = 0\n",
        "        while not done and i < STEPS_PER_EPISODE:\n",
        "            i += 1\n",
        "            action = offline_cql_sac_agent.get_next_action(state, evaluation_episode=True)\n",
        "            next_state, reward, done, info = env.step(action)\n",
        "            episode_reward += reward\n",
        "            state = next_state\n",
        "        run_results.append(episode_reward)\n",
        "    average_run_results.append(np.mean(run_results))\n",
        "    \n",
        "\n",
        "env.close()\n",
        "\n",
        "x_axis = list(range(len(average_run_results)))\n",
        "\n",
        "plt.plot(x_axis, average_run_results, label='Reward')\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Reward')\n",
        "plt.title('Offline CQL SAC')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "3q3LIScuYTeX"
      },
      "source": [
        "## Comparisons (14 points)\n",
        "Now, analyze your results and justify the trends you see. Then answer the following questions.\n",
        "\n",
        "❓ What is the reason for the difference between online and offline performance of the agent?\n",
        "\\\n",
        "✅ In offline setting we have distribution shift and OOD generalization problem (for the Q function part we are getting max of Q function and max acts as an adverserial example in for our estimator and also we maybe do actions that we haven't seen in dataset).\n",
        "\n",
        "❓ Which one is better: offline SAC or conservative SAC?\n",
        "\\\n",
        "✅ Conservative SAC. Because it somewhat solves the problem of distribution shift and ood generalization specially the Q function part my git min of it while getting max of policy.\n",
        "\n",
        "❓ What is the effect of `TRADEOFF_FACTOR` in the offline setting? How does changing its value affect the results?\n",
        "\\\n",
        "✅ There is a tradeoff for it. The more it is the more conservative the algorithm is and the slower the convergence and the less it is the faster but noisier the convergence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#References:\n",
        "#https://arxiv.org/abs/1812.05905\n",
        "#https://arxiv.org/abs/1910.07207\n",
        "#https://github.com/Felhof/DiscreteSAC\n",
        "#https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch/blob/master/agents/actor_critic_agents\n",
        "#https://towardsdatascience.com/adapting-soft-actor-critic-for-discrete-action-spaces-a20614d4a50a\n",
        "#https://arxiv.org/pdf/2006.04779.pdf\n",
        "#https://github.com/young-geng/CQL"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}