{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Hackathon #2\n", "\n", "Topics: \n", "- Linear and Non-Linear Transformations\n", "- Gradient Descent Optimization\n", "- Training by minibatch/gradient step and epoch\n", "- TensorBoard\n", "\n", "This is all setup in a IPython notebook so you can run any code you want to experiment with. Feel free to edit any cell, or add some to run your own code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# we'll start with our library imports...\n", "# tensorflow to specify and run computation graphs\n", "# numpy to run any numerical operations that need to take place outside of the TF graph\n", "import tensorflow as tf\n", "import numpy as np\n", "# this one lets us draw plots in our notebook\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A First Attempt at Classifying MNIST\n", "\n", "Let's have a look at our data first. We'll use the `next_batch` function to get subsets of the data in parallel arrays, which returns a tuple of images and labels. The images (shape `[28,28]`) are initially flat when we get them (shape `[784]`), but we can use [np.reshape](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.reshape.html) to correct that and allow us to plot the image with `matplotlib`. The labels are \"one-hot\", or arrays of length 10 (the number of classes) with a single index set to 1 and the rest to 0, indicating the integer value.\n", "\n", "MNIST is a dataset of handwritten digits, and we'll use it for a 10-class classification problem." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# extract our dataset, MNIST\n", "dir_prefix = '/work/cse496dl/shared/hackathon/02/mnist/'\n", "train_images = np.load(dir_prefix + 'mnist_train_images.npy')\n", "print(\"Train image tensor shape: \" + str(train_images.shape))\n", "train_labels = np.load(dir_prefix + 'mnist_train_labels.npy')\n", "print(\"Train image tensor shape: \" + str(train_labels.shape))\n", "test_images = np.load(dir_prefix + 'mnist_test_images.npy')\n", "print(\"Train image tensor shape: \" + str(test_images.shape))\n", "test_labels = np.load(dir_prefix + 'mnist_test_labels.npy')\n", "print(\"Train image tensor shape: \" + str(test_labels.shape))\n", "# visualize some of the data\n", "print(\"A label looks like this: \" + str(train_labels[0]))\n", "print(\"And an image looks like this:\")\n", "imgplot = plt.imshow(train_images[0].reshape((28,28)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll start specifying a simple neural network in this hackathon. The most basic building block is the linear transformation (AKA dense layer, linear layer, or fully connected layer), so we'll declare a function that creates the layer. Each is composed of two variables, the weight matrix and the bias vector, to calculate the function `f(x) = a(Wx + b)` where `a` is the activation function." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def linear_layer(x, output_size, activation=tf.identity, name='linear'):\n", " \"\"\"\n", " Args:\n", " - x: a rank two tensor, [batch_size, data_size]\n", " - output_size: (int) number of neurons\n", " - activation: function applied to the output\n", " - name: TensorFlow name scope for variable\n", " \"\"\"\n", " with tf.name_scope(name) as scope:\n", " (_, data_size) = x.get_shape().as_list()\n", " W = tf.Variable(tf.truncated_normal([data_size, output_size]), name='weights')\n", " b = tf.Variable(tf.truncated_normal([output_size]), name='bias')\n", " return activation(tf.matmul(x, W) + b)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first dimension of the input is usually reserved to be the \"batch\" dimension, which allows us to run many data through the model simultaneously. Then, each column of `W` corresponds to the weights of one linear unit of the layer. After adding the bias vector, we activate with a non-linearity or the identity, if we just want to use a linear transformation. [tf.name_scope](https://www.tensorflow.org/api_docs/python/tf/name_scope) is used to group the laher parameters, and its effects can be seen in TensorBoard.\n", "\n", "TensorFlow variables, which host the model parameters persistently in the graph, are declared with [tf.Variable](https://www.tensorflow.org/api_docs/python/tf/Variable), needing only an initial value to be created. You can also name the variable or set it to be untrainable, but both are optional and `Variable`s are trainable by default. We use [tf.truncated_normal](https://www.tensorflow.org/api_docs/python/tf/truncated_normal) to provide our initial values here, but there are much better initialization schemes (just make sure you never use a constant, e.g., [tf.zeros](https://www.tensorflow.org/api_docs/python/tf/zeros)).\n", "\n", "Then, we'll define a simple, two layer network with this function. We activate its output with [tf.nn.softmax](https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/softmax) so that we can interpret its output as the parameters of a discrete probability distribution." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "x = tf.placeholder(tf.float32, [None, 784], name='data')\n", "# use a single name scope for the model\n", "with tf.name_scope('linear_model') as scope:\n", " hidden = linear_layer(x, 200, name='hidden_layer')\n", " output = linear_layer(hidden, 10, activation=tf.nn.softmax, name='output_layer')\n", "tf.summary.FileWriter(\"logs\", tf.get_default_graph()).close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the last line, we log what we just defined with TensorBoard for visualization. We use [tf.get_default_graph](https://www.tensorflow.org/api_docs/python/tf/get_default_graph) to retrieve a handle to the TF graph that we're working in (the default because we haven't specified a particular graph), and then we write a summary of the graph with [tf.summary.FileWriter](https://www.tensorflow.org/api_docs/python/tf/summary/FileWriter). This puts an events file in the \"logs\" directory which maybe opened with TensorBoard. Access by running `tensorboard --logdir=logs` and pointing a browser at http://localhost:6006. We can't do this on Crane, but you could download the log file and perform this process locally.\n", "\n", "Note that, though the number of units in the hidden layer may be chosen freely as a hyperparameter, the number of units in the output unit must equal the number of classes in any classification problem.\n", "\n", "To summarize how the model is performing in its classification task, let's add a placeholder for the correct output and calculate the [cross-entropy](https://stackoverflow.com/a/41990932) loss between the estimated and correct discrete distributions (i.e., between the model's softmaxed distribution and a distribution with a probability of 1 in the correct class, a one hot vector). We use `EPSILON` to avoid trying to calculate `log(0)` which is undefined." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "y = tf.placeholder(tf.float32, [None, 10], name='label')\n", "EPSILON = 1e-10\n", "with tf.name_scope('cross_entropy') as scope:\n", " cross_entropy = -tf.reduce_sum(output * tf.log(y + EPSILON), axis=1)\n", "tf.summary.FileWriter(\"logs\", tf.get_default_graph()).close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we've constructed the TF graph, we'll create a session, initialize all variables, and run through an epoch of the test set to plot a histogram of the cross-entropy loss. (One epoch is an iteration of minibatches such that each datum is seen once. Minibatches are also frequently referred to as \"batches\", as below)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "# calculate values of interest to minibatching and epoch calculation\n", "# batch_size is adjustable\n", "train_num_examples = train_images.shape[0]\n", "test_num_examples = test_images.shape[0]\n", "batch_size = 32\n", "\n", "# finalize the graph\n", "with tf.Session() as session:\n", " session.run(tf.global_variables_initializer())\n", " loss_vals = []\n", " # loop through each test datum once, saving the cross entropy\n", " for i in range(test_num_examples // batch_size):\n", " batch_xs = test_images[i*batch_size:(i+1)*batch_size, :]\n", " batch_ys = test_labels[i*batch_size:(i+1)*batch_size, :]\n", " ce_val = session.run(cross_entropy, {x: batch_xs, y: batch_ys})\n", " loss_vals.append(ce_val)\n", "# now plot per-datum losses\n", "loss_vals = np.concatenate(loss_vals)\n", "hist, bin_edges = np.histogram(loss_vals)\n", "plt.bar(bin_edges[:-1], hist, width = 1)\n", "plt.xlim(min(bin_edges), max(bin_edges))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "It looks like the model guesses the correct class (corresponding to low loss and the smaller bar on the right) on about 10% of the data, which we would have anticipated for a naive model in a 10 class problem. We can improve this model. Let's try this again, with an optimizer and a non-linearity.\n", "\n", "### A Second Attempt...\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Clear the graph so we can re-use it. If this is omitted, we get an error.\n", "tf.reset_default_graph()\n", "x = tf.placeholder(tf.float32, [None, 784], name='data')\n", "# use a single name scope for the model\n", "with tf.name_scope('linear_model') as scope:\n", " hidden = tf.layers.dense(x, 200, activation=tf.nn.relu, name='hidden_layer')\n", " output = tf.layers.dense(hidden, 10, name='output_layer')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This time, we're using [tf.layers.dense](https://www.tensorflow.org/api_docs/python/tf/layers/dense), which has a very similar API to our custom function, but much more functionality. Also, note that we're using [tf.nn.relu](https://www.tensorflow.org/api_docs/python/tf/nn/relu) as the activation of our hidden layer.\n", "\n", "Further, instead of using a custom cross-entropy function, which lots of potential for numerical instability, we'll use TF's built in function, [tf.nn.softmax_cross_entropy_with_logit](https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits). It combines the `softmax` activation of the final layer with the cross-entropy calculation." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# define classification loss\n", "y = tf.placeholder(tf.float32, [None, 10], name='label')\n", "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, using the loss tensor, we'll define an optimizer that uses backpropagation to update the values of each layer's variables." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# setup optimizer and training operation\n", "optimizer = tf.train.AdamOptimizer()\n", "train_op = optimizer.minimize(cross_entropy)\n", "tf.summary.FileWriter(\"logs\", tf.get_default_graph()).close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll use the [Adam](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer#minimize) optimizer (usually a safe first choice), and use the [minimize](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer#minimize) function which is defined for every TensorFlow optimizer. It returns an operation that automatically calculates the gradient of the provided function, and updates all variables marked trainable. We'll pass it to `sess.run` to train for one epoch and then check the test loss values again." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "batch_size = 32\n", "session = tf.Session()\n", "session.run(tf.global_variables_initializer())\n", "\n", "# train for one epoch\n", "for i in range(train_num_examples // batch_size):\n", " batch_xs = train_images[i*batch_size:(i+1)*batch_size, :]\n", " batch_ys = train_labels[i*batch_size:(i+1)*batch_size, :] \n", " session.run(train_op, {x: batch_xs, y: batch_ys})\n", "\n", "# loop through each test datum once\n", "loss_vals = []\n", "for i in range(test_num_examples // batch_size):\n", " batch_xs = test_images[i*batch_size:(i+1)*batch_size, :]\n", " batch_ys = test_labels[i*batch_size:(i+1)*batch_size, :]\n", " ce_val = session.run(cross_entropy, {x: batch_xs, y: batch_ys})\n", " loss_vals.append(ce_val)\n", "\n", "# now plot per-datum losses\n", "loss_vals = np.concatenate(loss_vals)\n", "hist, bin_edges = np.histogram(loss_vals)\n", "plt.bar(bin_edges[:-1], hist, width = 1)\n", "plt.xlim(min(bin_edges), max(bin_edges))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just one training epoch has dramatically improved the model's performance. But on how many of the instances is it actually guessing the correct label?\n", "\n", "## Hackathon 2 Exercise 1\n", "\n", "Write code to calculate the maximum aposteriori (MAP) estimate of the model on the test data, and compare to the true labels to calculate a confusion matrix with [tf.confusion_matrix](https://www.tensorflow.org/api_docs/python/tf/confusion_matrix). (For the inexperienced, [what is a confusion matrix?](https://en.wikipedia.org/wiki/Confusion_matrix))\n", "\n", "(Hint #0: Re-use and modify my code from above. Try not to reinvent the wheel, but always remember to cite borrowed code.)\n", "\n", "(Hint #1: The MAP estimate is just the class whose probability is greatest. I reccomend using [tf.argmax](https://www.tensorflow.org/versions/master/api_docs/python/tf/argmax) with the correct `axis` argument to find this to find the max over the correct dimension of the output.)\n", "\n", "(Hint #2: tf.confusion_matrix is a function that needs be run in a `session.run` call that returns matrices. Store the resulting matrices in a list and then sum to get the matrix for the full test dataset. Remember to specify the `num_classes` argument.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Coda\n", "\n", "### Saving and Loading TF models\n", "\n", "https://www.tensorflow.org/programmers_guide/saved_model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Numpy Broadcasting\n", "\n", "TensorFlow uses [Numpy broadcasting](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html) when doing arithmetic with arrays of different shapes.\n", "\n", "E.g.," ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"scalar-matrix addition\")\n", "print(np.ones([2,3]) + 1)\n", "print(\"a sample vector\")\n", "print(np.arange(3))\n", "print(\"matrix-vector addition\")\n", "print(np.ones([2,3]) + np.arange(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The importance of non-linearities" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from IPython.display import HTML\n", "# From Colah's Blog, linearly separating spirals with linear transforms and non-linearities\n", "HTML('')" ] } ], "metadata": { "kernelspec": { "display_name": "TensorFlow 1.4 (py36)", "language": "python", "name": "tensorflow-1.4.0-py36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }