{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Hackathon #1\n",
    "\n",
    "Topics: \n",
    "- TF Graph\n",
    "- Tensors\n",
    "- Sessions\n",
    "- Operations\n",
    "- Placeholders\n",
    "- Variables\n",
    "- Initialization\n",
    "\n",
    "Some material adapted from the TensorFlow documention: https://www.tensorflow.org\n",
    "\n",
    "This is all setup in a IPython notebook so you can run any code you want to experiment with. Feel free to edit any cell, or add some to run your own code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll start with our library imports...\n",
    "# tensorflow to specify and run computation graphs\n",
    "# numpy to run any numerical operations that need to take place outside of the TF graph\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "TensorFlow programs typically consist of two sections:\n",
    "1. Building the computation graph\n",
    "2. Running the computation graph\n",
    "\n",
    "A computational graph is a series of TensorFlow operations arranged into a graph of nodes. Each node takes zero or more tensors as inputs and produces a tensor as an output. This is called the tensor-in tensor-out (TITO) model, where we may think of tensors as the edges between node operations.\n",
    "\n",
    "The basic unit of data in TensorFlow is the tensor. A tensor consists of a set of primitive values (think `float` or `int`) shaped into an array of any number of dimensions. A tensor's rank is its number of dimensions. Here are some examples of tensors:\n",
    "```\n",
    "3 # a rank 0 tensor; a scalar with shape []\n",
    "[1., 2., 3.] # a rank 1 tensor; a vector with shape [3]\n",
    "[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]\n",
    "[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]\n",
    "```\n",
    "TF docs for `tf.Tensor`: https://www.tensorflow.org/api_docs/python/tf/Tensor\n",
    "\n",
    "One type of graph node is a constant. TF docs for `constant`: https://www.tensorflow.org/api_docs/python/tf/constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Const:0\", shape=(), dtype=float32) Tensor(\"Const_1:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "node1 = tf.constant(3.0, dtype=tf.float32)\n",
    "node2 = tf.constant(4.0) # also tf.float32 implicitly\n",
    "print(node1, node2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that printing the nodes does not output the values `3.0` and `4.0` as you might expect.\n",
    "We can see that each `Tensor` has a name, a shape, and a data type.\n",
    "This is all information available statically about the `Tensor`s, before initializing the graph or running anything.\n",
    "\n",
    "Now, we have to create a TensorFlow session in which to run our graph. TF docs for `Session`: https://www.tensorflow.org/api_docs/python/tf/Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.0, 4.0]\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "# This prints what we expect\n",
    "print(sess.run([node1, node2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll add an operation to the graph. TF docs for `Operation`: https://www.tensorflow.org/api_docs/python/tf/Operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "node3: Tensor(\"Add:0\", shape=(), dtype=float32)\n",
      "sess.run(node3): 7.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "# tf.add sums the two tensors provided\n",
    "node3 = tf.add(node1, node2)\n",
    "print(\"node3:\", node3)\n",
    "print(\"sess.run(node3):\", sess.run(node3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operation is in the TF graph, so we have to run it in a sesssion to get the output. TF docs for `Session.run` https://www.tensorflow.org/versions/r0.12/api_docs/python/client/session_management#Session.run\n",
    "\n",
    "If you visualize this subgraph in TensorBoard, it looks like this:\n",
    "![img_1](https://www.tensorflow.org/images/getting_started_add.png)\n",
    "\n",
    "Notice that when we print `node3`, it's a `Tensor`. This is following with the TITO nature of Tensorflow operations.\n",
    "\n",
    "A graph can be parameterized to accept external inputs, known as placeholders. TF docs for `placeholder`: https://www.tensorflow.org/api_docs/python/tf/placeholder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: Tensor(\"Placeholder:0\", dtype=float32)\n",
      "b: Tensor(\"Placeholder_1:0\", dtype=float32)\n",
      "adder_node: Tensor(\"add:0\", dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "adder_node = a + b  # + provides a shortcut for tf.add(a, b)\n",
    "print(\"a:\", a)\n",
    "print(\"b:\", b)\n",
    "print(\"adder_node:\", adder_node)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many python operators are overloaded by TensorFlow to be their pointwise equivalents using numpy broadcasting. Much of the TensorFlow API closely resembles that of Numpy. Numpy docs on broadcasting: https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html\n",
    "\n",
    "Now, the graph looks like this:\n",
    "![img_2](https://www.tensorflow.org/images/getting_started_adder.png)\n",
    "\n",
    "In order to use operations that rely directly or indirectly on placeholders we must provide a `feed_dict` to the `Session.run` method. This is a dictionary with `placeholder` as keys and the value that they should use as the corresponding values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.5\n",
      "[ 3.  7.]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(adder_node, {a: 3, b: 4.5}))\n",
    "print(sess.run(adder_node, feed_dict={a: [1, 3], b: [2, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that, in the second `print` we feed lists for the variables. Python lists and numpy arrays can both be converted into Tensors.\n",
    "\n",
    "We can make the computational graph more complex by adding another operation. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24.0\n"
     ]
    }
   ],
   "source": [
    "add_and_triple = adder_node * 3.\n",
    "print(sess.run(add_and_triple, {a: 3, b:5}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph finally looks like this:\n",
    "![img_3](https://www.tensorflow.org/images/getting_started_triple.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hackathon 1 Exercise 1\n",
    "\n",
    "Write code to evaluate the function `f(x, y) = 7xy^2*cos(3x) + sqrt(5)*xy + exp(2y)` in the cell below using placeholders for `x` and `y`, and feeding values of your choice to evaluate the function in a session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "537.82666"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "x = tf.placeholder(tf.float32)\n",
    "y = tf.placeholder(tf.float32)\n",
    "first_term = 7. * x * tf.square(y) * tf.cos(3. * x)\n",
    "second_term = tf.sqrt(5.) * x * y\n",
    "third_term = tf.exp(2 * y)\n",
    "fn = first_term + second_term + third_term\n",
    "sess.run(fn, {x: 2., y: 3.})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the model trainable, we need to be able to modify the graph to get new outputs with the same input\n",
    "Variables allow us to add trainable parameters to a graph\n",
    "They are constructed with a type and initial value\n",
    "TF docs: https://www.tensorflow.org/api_docs/python/tf/Variable\n",
    "High-level Variable How-To: https://www.tensorflow.org/programmers_guide/variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W: <tf.Variable 'Variable:0' shape=(1,) dtype=float32_ref>\n",
      "b: <tf.Variable 'Variable_1:0' shape=(1,) dtype=float32_ref>\n",
      "linear_model: Tensor(\"add_3:0\", dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "W = tf.Variable([.3], dtype=tf.float32)\n",
    "b = tf.Variable([-.2], dtype=tf.float32)\n",
    "x = tf.placeholder(tf.float32)\n",
    "linear_model = W*x + b\n",
    "print(\"W:\", W)\n",
    "print(\"b:\", b)\n",
    "print(\"linear_model:\", linear_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constants are initialized when you call `tf.constant`, and their value can never change.\n",
    "By contrast, variables are not initialized when you call `tf.Variable`.\n",
    "To initialize all the variables in a TensorFlow program, you must explicitly call a special operation as follows.\n",
    "TF docs: https://www.tensorflow.org/api_docs/python/tf/global_variables_initializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([ 0.30000001], dtype=float32), array([-0.2], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "# Now we can do things with the values of the Variables we defined\n",
    "print(sess.run([W, b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since `x` is a placeholder, we can evaluate `linear_model` for several values of `x` simultaneously as follows.\n",
    "This is also how you run more than one input at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.10000001  0.40000004  0.70000005  1.        ]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(linear_model, {x: [1, 2, 3, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've created a model, but we don't know how good it is yet.\n",
    "To evaluate the model on training data, we need a `y` placeholder to provide the desired values, and we need to write a loss function.\n",
    "\n",
    "A loss function measures how far apart the current estimated output is from the provided output.\n",
    "We'll use a standard loss model for linear regression, which sums the squares of the deltas between the current model and the provided data.\n",
    "`linear_model - y` creates a vector where each element is the corresponding example's error delta.\n",
    "We call `tf.square` to square that error.\n",
    "Then, we sum all the squared errors to create a single scalar that abstracts the error of all examples using `tf.reduce_sum`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25.26\n"
     ]
    }
   ],
   "source": [
    "y = tf.placeholder(tf.float32)\n",
    "squared_deltas = tf.square(linear_model - y)\n",
    "loss = tf.reduce_sum(squared_deltas)\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could improve this manually by reassigning the values of `W` and `b` to the perfect values of `-1` and `1`.\n",
    "A variable is initialized to the value provided to `tf.Variable` but can be changed using operations like `tf.assign`.\n",
    "For example, `W=-1` and `b=1` are the optimal parameters for our model. We can change `W` and `b` accordingly. TF docs for `tf.assign`: https://www.tensorflow.org/api_docs/python/tf/assign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n"
     ]
    }
   ],
   "source": [
    "fixW = tf.assign(W, [-1.])\n",
    "fixb = tf.assign(b, [1.])\n",
    "sess.run([fixW, fixb])\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We guessed the \"perfect\" values of `W` and `b`, but the whole point of machine learning is to find the correct model parameters automatically. We will show how to accomplish this in the next Hackathon. [Click here](https://www.youtube.com/watch?v=dQw4w9WgXcQ) to see whether we're having class next Monday."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TensorFlow 1.4 (py36)",
   "language": "python",
   "name": "tensorflow-1.4.0-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}