{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Hackathon 10\n", "\n", "Topics:\n", "- Deep Learning frameworks\n", "- OpenAI Gym\n", "- Deep Q-Learning (DQN)\n", "\n", "DQN presentation based on the implementation in the [PyTorch documentation](http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html). Thanks to author [Adam Paszke](https://github.com/apaszke).\n", "\n", "This is all setup in a IPython notebook so you can run any code you want to experiment with. Feel free to edit any cell, or add some to run your own code." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deep Learning Frameworks\n", "\n", "**TensorFlow**: [TensorFlow](http://tensorflow.com/) is an open source software library from Google Brain for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) that flow between them. Typically used declaratively, it recently added Eager execution in update 1.7.\n", "\n", "**Sonnet**: [Sonnet](https://github.com/deepmind/sonnet), based on TensorFlow, uses an object-oriented approach, similar to Torch/NN, allowing modules to be created which define the forward pass of some computation. Modules are ‘called’ with some input Tensors, which adds ops to the Graph and returns output Tensors. One of the design choices was to make sure the variable sharing is handled transparently by automatically reusing variables on subsequent calls to the same module. [DeepMind Blog Post](https://deepmind.com/blog/open-sourcing-sonnet/)\n", "\n", "**Keras**: [Keras](https://keras.io/) is a high-level neural networks API by [François Chollet](https://github.com/fchollet), written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. It's hard to do unusual things with, but should handle most use cases easily.\n", "\n", "**PyTorch**: [PyTorch](http://pytorch.org/tutorials/index.html) is based on Torch, primarily developed by Facebook's artificial-intelligence research group, and Uber's \"Pyro\" software for probabilistic programming is built on it. Uses an object oriented, eager approach to specifying/running neural networks. We'll be using this framework today.\n", "\n", "**and more...**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tutorial shows how to use PyTorch to train a Deep Q Learning (DQN) agent on the Breakout-v0 task from the OpenAI Gym.\n", "\n", "Gym is a toolkit for developing and comparing reinforcement learning algorithms. It makes no assumptions about the structure of your agent, and is compatible with any numerical computation library, such as TensorFlow or Theano. The gym library is a collection of test problems — environments — that you can use to work out your reinforcement learning algorithms. These environments have a shared interface, allowing you to write general algorithms.\n", "\n", "**Packages**\n", "\n", "\n", "First, let's import needed packages. Firstly, we need [gym](https://gym.openai.com/docs) for the environment. It's included in the `Python (pytorch)` kernel we installed with the script distributed with this hackathon. If you haven't done that yet, you should do it now.\n", "\n", "We'll also use the following from PyTorch:\n", "\n", "- torch (``torch``)\n", "\n", "A PyTorch Tensor is conceptually identical to a numpy array: a Tensor is an n-dimensional array, and PyTorch provides many functions for operating on these Tensors. Like numpy arrays, PyTorch Tensors do not know anything about deep learning or computational graphs or gradients; they are a generic tool for scientific computing. However unlike numpy, PyTorch Tensors can utilize GPUs to accelerate their numeric computations.\n", "\n", "PyTorch autograd looks a lot like TensorFlow: in both frameworks we define a computational graph, and use automatic differentiation to compute gradients. The biggest difference between the two is that TensorFlow’s computational graphs are static and PyTorch uses dynamic computational graphs. In TensorFlow, we define the computational graph once and then execute the same graph over and over again, possibly feeding different input data to the graph. In PyTorch, each forward pass defines a new computational graph.\n", "\n", "- neural networks (``torch.nn``)\n", "\n", "In TensorFlow, packages like Keras provide higher-level abstractions over raw computational graphs that are useful for building neural networks. In PyTorch, the `nn` package serves this same purpose. The `nn` package defines a set of Modules, which are roughly equivalent to neural network layers. A Module receives input Variables and computes output Variables, but may also hold internal state such as Variables containing learnable parameters. The nn package also defines a set of useful loss functions that are commonly used when training neural networks.\n", "\n", "- optimization (``torch.optim``)\n", "\n", "Up to this point we have updated the weights of our models by manually mutating the .data member for Variables holding learnable parameters. This is not a huge burden for simple optimization algorithms like stochastic gradient descent, but in practice we often train neural networks using more sophisticated optimizers like AdaGrad, RMSProp, Adam, etc. The `optim` package in PyTorch abstracts the idea of an optimization algorithm and provides implementations of commonly used optimization algorithms.\n", "\n", "- automatic differentiation (``torch.autograd``)\n", "\n", "We can use automatic differentiation to automate the computation of backward passes in neural networks. The autograd package in PyTorch provides exactly this functionality. When using autograd, the forward pass of your network will define a computational graph; nodes in the graph will be Tensors, and edges will be functions that produce output Tensors from input Tensors. Backpropagating through this graph then allows you to easily compute gradients.\n", "\n", "This sounds complicated, but it’s pretty simple to use in practice. We wrap our PyTorch Tensors in Variable objects; a Variable represents a node in a computational graph. If `x` is a Variable then `x.data` is a Tensor, and `x.grad` is another Variable holding the gradient of `x` with respect to some scalar value.\n", "\n", "PyTorch Variables have the same API as PyTorch Tensors: (almost) any operation that you can perform on a Tensor also works on Variables; the difference is that using Variables defines a computational graph, allowing you to automatically compute gradients." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import gym\n", "import math\n", "import random\n", "import numpy as np\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "from collections import namedtuple\n", "from itertools import count\n", "from PIL import Image\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torch.optim as optim\n", "import torch.nn.functional as F\n", "from torch.autograd import Variable\n", "\n", "# set up matplotlib\n", "is_ipython = 'inline' in matplotlib.get_backend()\n", "if is_ipython:\n", " from IPython import display\n", "\n", "plt.ion()\n", "\n", "# if gpu is to be used\n", "use_cuda = torch.cuda.is_available()\n", "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n", "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n", "ByteTensor = torch.cuda.ByteTensor if use_cuda else torch.ByteTensor\n", "Tensor = FloatTensor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll start with the Breakout environment." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "env = gym.make('Breakout-v0').unwrapped" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Replay Memory\n", "-------------\n", "\n", "We'll be using experience replay memory for training our DQN. It stores\n", "the transitions that the agent observes, allowing us to reuse this data\n", "later. By sampling from it randomly, the transitions that build up a\n", "batch are decorrelated. It has been shown that this greatly stabilizes\n", "and improves the DQN training procedure.\n", "\n", "For this, we're going to need two classses:\n", "\n", "- ``Transition`` - a named tuple representing a single transition in\n", " our environment\n", "- ``ReplayMemory`` - a cyclic buffer of bounded size that holds the\n", " transitions observed recently. It also implements a ``.sample()``\n", " method for selecting a random batch of transitions for training." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "Transition = namedtuple('Transition',\n", " ('state', 'action', 'next_state', 'reward'))\n", "\n", "\n", "class ReplayMemory(object):\n", "\n", " def __init__(self, capacity):\n", " self.capacity = capacity\n", " self.memory = []\n", " self.position = 0\n", "\n", " def push(self, *args):\n", " \"\"\"Saves a transition.\"\"\"\n", " if len(self.memory) < self.capacity:\n", " self.memory.append(None)\n", " self.memory[self.position] = Transition(*args)\n", " self.position = (self.position + 1) % self.capacity\n", "\n", " def sample(self, batch_size):\n", " return random.sample(self.memory, batch_size)\n", "\n", " def __len__(self):\n", " return len(self.memory)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's define our model. But first, let quickly recap what a DQN is.\n", "\n", "### DQN algorithm\n", "\n", "Our environment is deterministic, so all equations presented here are\n", "also formulated deterministically for the sake of simplicity. In the\n", "reinforcement learning literature, they would also contain expectations\n", "over stochastic transitions in the environment.\n", "\n", "Our aim will be to train a policy that tries to maximize the discounted,\n", "cumulative reward\n", "$R_{t_0} = \\sum_{t=t_0}^{\\infty} \\gamma^{t - t_0} r_t$, where\n", "$R_{t_0}$ is also known as the *return*. The discount,\n", "$\\gamma$, should be a constant between $0$ and $1$\n", "that ensures the sum converges. It makes rewards from the uncertain far\n", "future less important for our agent than the ones in the near future\n", "that it can be fairly confident about.\n", "\n", "The main idea behind Q-learning is that if we had a function\n", "$Q^*: State \\times Action \\rightarrow \\mathbb{R}$, that could tell\n", "us what our return would be, if we were to take an action in a given\n", "state, then we could easily construct a policy that maximizes our\n", "rewards:\n", "\n", "\\begin{align}\\pi^*(s) = \\arg\\!\\max_a \\ Q^*(s, a)\\end{align}\n", "\n", "However, we don't know everything about the world, so we don't have\n", "access to $Q^*$. But, since neural networks are universal function\n", "approximators, we can simply create one and train it to resemble\n", "$Q^*$.\n", "\n", "For our training update rule, we'll use a fact that every $Q$\n", "function for some policy obeys the Bellman equation:\n", "\n", "\\begin{align}Q^{\\pi}(s, a) = r + \\gamma Q^{\\pi}(s', \\pi(s'))\\end{align}\n", "\n", "The difference between the two sides of the equality is known as the\n", "temporal difference error, $\\delta$:\n", "\n", "\\begin{align}\\delta = Q(s, a) - (r + \\gamma \\max_a Q(s', a))\\end{align}\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class DQN(nn.Module):\n", "\n", " def __init__(self):\n", " super(DQN, self).__init__()\n", " self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)\n", " self.bn1 = nn.BatchNorm2d(16)\n", " self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)\n", " self.bn2 = nn.BatchNorm2d(32)\n", " self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)\n", " self.bn3 = nn.BatchNorm2d(32)\n", " self.conv4 = nn.Conv2d(32, 32, kernel_size=5, stride=2)\n", " self.bn4 = nn.BatchNorm2d(32)\n", " self.head = nn.Linear(1120, env.action_space.n)\n", "\n", " def forward(self, x):\n", " x = F.relu(self.bn1(self.conv1(x)))\n", " x = F.relu(self.bn2(self.conv2(x)))\n", " x = F.relu(self.bn3(self.conv3(x)))\n", " x = F.relu(self.bn4(self.conv4(x)))\n", " return self.head(x.view(x.size(0), -1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code below are utilities for extracting and processing rendered\n", "images from the environment. Once you run the cell it will\n", "display an example patch that it extracted." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def clean_screen(observation):\n", " screen = observation.transpose(\n", " (2, 0, 1)) # transpose into torch order (CHW)\n", " # Strip off the top and bottom of the screen\n", " screen = screen[:, 30:195, 10:150]\n", " # Convert to float, rescare, convert to torch tensor\n", " # (this doesn't require a copy)\n", " screen = np.ascontiguousarray(screen, dtype=np.float32) / 255\n", " screen = torch.from_numpy(screen)\n", " # Resize, and add a batch dimension (BCHW)\n", " return screen.unsqueeze(0).type(Tensor)\n", "\n", "\n", "env.reset()\n", "observation, reward, done, info = env.step(0)\n", "plt.figure()\n", "plt.imshow(observation)\n", "plt.title('Original screen')\n", "plt.show()\n", "plt.figure()\n", "plt.imshow(clean_screen(observation).cpu().squeeze(0).permute(1, 2, 0).numpy(),\n", " interpolation='none')\n", "plt.title('Cleaned screen')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This next cell instantiates our model and its optimizer, and defines some\n", "utilities:\n", "\n", "- ``Variable`` - this is a simple wrapper around\n", " ``torch.autograd.Variable`` that will automatically send the data to\n", " the GPU every time we construct a Variable.\n", "- ``select_action`` - will select an action accordingly to an epsilon\n", " greedy policy. Simply put, we'll sometimes use our model for choosing\n", " the action, and sometimes we'll just sample one uniformly. The\n", " probability of choosing a random action will start at ``EPS_START``\n", " and will decay exponentially towards ``EPS_END``. ``EPS_DECAY``\n", " controls the rate of the decay.\n", "- ``plot_durations`` - a helper for plotting the durations of episodes,\n", " along with an average over the last 100 episodes (the measure used in\n", " the official evaluations). The plot will be underneath the cell\n", " containing the main training loop, and will update after every\n", " episode." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "BATCH_SIZE = 128\n", "GAMMA = 0.999\n", "EPS_START = 0.9\n", "EPS_END = 0.05\n", "EPS_DECAY = 200\n", "TARGET_UPDATE = 10\n", "\n", "policy_net = DQN()\n", "target_net = DQN()\n", "target_net.load_state_dict(policy_net.state_dict())\n", "target_net.eval()\n", "\n", "if use_cuda:\n", " policy_net.cuda()\n", " target_net.cuda()\n", "\n", "optimizer = optim.RMSprop(policy_net.parameters())\n", "memory = ReplayMemory(10000)\n", "\n", "\n", "steps_done = 0\n", "\n", "\n", "def select_action(state):\n", " global steps_done\n", " sample = random.random()\n", " eps_threshold = EPS_END + (EPS_START - EPS_END) * \\\n", " math.exp(-1. * steps_done / EPS_DECAY)\n", " steps_done += 1\n", " if sample > eps_threshold:\n", " return policy_net(\n", " Variable(state, volatile=True).type(FloatTensor)).data.max(1)[1].view(1, 1)\n", " else:\n", " return LongTensor([[random.randrange(env.action_space.n)]])\n", "\n", "\n", "episode_durations = []\n", "\n", "\n", "def plot_durations():\n", " plt.figure(2)\n", " plt.clf()\n", " durations_t = torch.FloatTensor(episode_durations)\n", " plt.title('Training...')\n", " plt.xlabel('Episode')\n", " plt.ylabel('Duration')\n", " plt.plot(durations_t.numpy())\n", " # Take 100 episode averages and plot them too\n", " if len(durations_t) >= 100:\n", " means = durations_t.unfold(0, 100, 1).mean(1).view(-1)\n", " means = torch.cat((torch.zeros(99), means))\n", " plt.plot(means.numpy())\n", "\n", " plt.pause(0.001) # pause a bit so that plots are updated\n", " if is_ipython:\n", " display.clear_output(wait=True)\n", " display.display(plt.gcf())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To minimise the error, we will use the [Huber\n", "loss](https://en.wikipedia.org/wiki/Huber_loss). The Huber loss acts\n", "like the mean squared error when the error is small, but like the mean\n", "absolute error when the error is large - this makes it more robust to\n", "outliers when the estimates of $Q$ are very noisy. We calculate\n", "this over a batch of transitions, $B$, sampled from the replay\n", "memory:\n", "\n", "\\begin{align}\\mathcal{L} = \\frac{1}{|B|}\\sum_{(s, a, s', r) \\ \\in \\ B} \\mathcal{L}(\\delta)\\end{align}\n", "\n", "\\begin{align}\\text{where} \\quad \\mathcal{L}(\\delta) = \\begin{cases}\n", " \\frac{1}{2}{\\delta^2} & \\text{for } |\\delta| \\le 1, \\\\\n", " |\\delta| - \\frac{1}{2} & \\text{otherwise.}\n", " \\end{cases}\\end{align}\n", " \n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def optimize_model():\n", " if len(memory) < BATCH_SIZE:\n", " return\n", " transitions = memory.sample(BATCH_SIZE)\n", " # Transpose the batch (see http://stackoverflow.com/a/19343/3343043 for\n", " # detailed explanation).\n", " batch = Transition(*zip(*transitions))\n", "\n", " # Compute a mask of non-final states and concatenate the batch elements\n", " non_final_mask = ByteTensor(tuple(map(lambda s: s is not None,\n", " batch.next_state)))\n", " non_final_next_states = Variable(torch.cat([s for s in batch.next_state\n", " if s is not None]),\n", " volatile=True)\n", " state_batch = Variable(torch.cat(batch.state))\n", " action_batch = Variable(torch.cat(batch.action))\n", " reward_batch = Variable(torch.cat(batch.reward))\n", "\n", " # Compute Q(s_t, a) - the model computes Q(s_t), then we select the\n", " # columns of actions taken\n", " state_action_values = policy_net(state_batch).gather(1, action_batch)\n", "\n", " # Compute V(s_{t+1}) for all next states.\n", " next_state_values = Variable(torch.zeros(BATCH_SIZE).type(Tensor))\n", " next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0]\n", " # Compute the expected Q values\n", " expected_state_action_values = (next_state_values * GAMMA) + reward_batch\n", " # Undo volatility (which was used to prevent unnecessary gradients)\n", " expected_state_action_values = Variable(expected_state_action_values.data)\n", "\n", " # Compute Huber loss\n", " loss = F.smooth_l1_loss(state_action_values, expected_state_action_values)\n", "\n", " # Optimize the model\n", " optimizer.zero_grad()\n", " loss.backward()\n", " for param in policy_net.parameters():\n", " param.grad.data.clamp_(-1, 1)\n", " optimizer.step()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below, you can find the main training loop. At the beginning we reset\n", "the environment and initialize the ``state`` variable. Then, we sample\n", "an action, execute it, observe the next screen and the reward,\n", "and optimize our model once. When the episode ends (our model\n", "fails), we restart the loop.\n", "\n", "Below, `num_episodes` is set small. You should download\n", "the notebook and run lot more epsiodes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DEBUG = True\n", "num_episodes = 1\n", "for i_episode in range(num_episodes):\n", " # Initialize the environment and state\n", " env.reset()\n", " observation, _, _, _ = env.step(0)\n", " last_screen = clean_screen(observation)\n", " current_screen = clean_screen(observation)\n", " state = current_screen - last_screen\n", " for t in count():\n", " # Select and perform an action\n", " action = select_action(state)\n", " observation, reward, done, _ = env.step(action[0, 0])\n", " reward = Tensor([reward])\n", " \n", " if DEBUG:\n", " print(\"Action: \" + str(action))\n", " plt.figure()\n", " plt.imshow(observation)\n", " plt.title('Observation')\n", " plt.show()\n", "\n", " # Observe new state\n", " last_screen = current_screen\n", " current_screen = clean_screen(observation)\n", " if not done:\n", " next_state = current_screen - last_screen\n", " else:\n", " next_state = None\n", "\n", " # Store the transition in memory\n", " memory.push(state, action, next_state, reward)\n", "\n", " # Move to the next state\n", " state = next_state\n", "\n", " # Perform one step of the optimization (on the target network)\n", " optimize_model()\n", " if done:\n", " episode_durations.append(t + 1)\n", " plot_durations()\n", " break\n", " # Update the target network\n", " if i_episode % TARGET_UPDATE == 0:\n", " target_net.load_state_dict(policy_net.state_dict())\n", "\n", "print('Complete')\n", "env.close()\n", "plt.ioff()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# No exercise this week\n", "## Project update due on Sunday, the 8th" ] } ], "metadata": { "kernelspec": { "display_name": "Python (pytorch)", "language": "python", "name": "pytorch" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }