Large Scale Data Analysis Using Deep Learning (Prof. U Kang)
Introduction to TensorFlow
2017.04.17 Beunguk Ahn
(
beunguk.ahn@gmail.com)
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
You already know how to use You already installed
Prerequisites
Python
TensorFlow
TensorFlow
Open source so ware library for numerical computation using data flow graphs.
Simply, it is one of widely used frameworks for deep
learning programs.
TensorFlow
Open source so ware library for numerical computation using data flow graphs.
Simply it is one of widely used frameworks for deep
learning programs.
Role of TensorFlow
Machine learning flow in general
Why use TensorFlow?
Large community engagement (Google)
Multiple GPU Support
Data and Model Parallelism Python + Numpy
Tensorboard
However, you don't have to use TensorFlow.
Other frameworks like Theano, Caffe, Torch, etc are also good options.
Tensor
Tensor is simply multi dimension array
1-D Tensor : vector 2-D Tensor : matrix 3-D Tensor : tensor 4-D Tensor : tensor ...
1Dtensor = [ 1, 2] # vector 2Dtensor = [[ 1, 2], # matrix [ 3 ,4]]
3Dtensor = [[[ 1, 2], # tensor [ 3, 4]],
[[ 5, 6], [ 7, 8]], [[ 9,10], [11,12]]]
Shape of Tensor
Shape of tensor is expressed as tuple
import tensorflow as tf
scalar = tf.constant(1) # () zero = tf.constant([]) # (0,) vector = tf.constant([1,2,3,4]) # (4,) matrix = tf.constant([[1,2],[3,4]]) # (2, 2) tensor = tf.constant([[[1],[ 3]], # (3, 2, 1) [[5],[ 7]],
[[9],[11]]]) print '\n'.join(
map(str, [scalar, zero, vector, matrix, tensor]))
>>> Tensor("Const:0", shape=(1,), dtype=int32)
>>> Tensor("Const_1:0", shape=(0,), dtype=float32)
>>> Tensor("Const_2:0", shape=(4,), dtype=int32)
>>> Tensor("Const_3:0", shape=(2, 2), dtype=int32)
Data Type of Tensor
What kind of data contained in tensor
Also float(32,64), [u]int(8,16,32,64), bool, complex(64,128) are also available
import tensorflow as tfint32 = tf.constant([[1,2],[3,4]]) # int32 float32 = tf.constant([[1.0, 2], [3, 4.0]]) # float32 float64 = tf.constant([[1.0, 2], [3, 4.0]], # float64 dtype=tf.float64)
_string = tf.constant([["A", "B"], ["C", "D"]], # string dtype=tf.string)
print '\n'.join(
map(str, [int32, float32, float64, _string]))
>>> Tensor("Const:0", shape=(2, 2), dtype=int32)
>>> Tensor("Const_1:0", shape=(2, 2), dtype=float32)
>>> Tensor("Const_2:0", shape=(2, 2), dtype=float64)
>>> Tensor("Const_3:0", shape=(2, 2), dtype=string)
TensorFlow program in two parts
Construction Phase Build a graph
Declare Variables, Placeholders, Operations, Optimizers
Execution Phase
Initialize model and run
Start session and iterate through the graph
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
Variable
Stores a scalar or tensor
Used as an input for Operations
Requires initial value for future use
At constuction phase, variables are "just" defined.
Value will be assigned a er Session initialized
You will get an idea through slides
Variable
import tensorflow as tf
# Define variable, yet initialized
w = tf.Variable(tf.random_normal([1, 3]), name="weight")
with tf.Session() as sess:
w_value = sess.run(w) # get value for var 'w'
print '\n'.join(map(str, ([w, w_value])))
"tensorflow.python.framework.errors_impl.
FailedPreconditionError: Attempting to use uninitialized value weight"
Variable
import tensorflow as tf
# Define variable, yet initialized
w = tf.Variable(tf.random_normal([1, 3]), name="weight")
# Define init operation
init_op = tf.global_variables_initializer() with tf.Session() as sess:
sess.run(init_op) # initialize all declared variables w_value = sess.run(w) # get value for var 'w'
print '\n'.join(map(str, ([w, w_value])))
>>> Tensor("weight/read:0", shape=(2, 3), dtype=float32)
>>> [[-0.50282216 -1.94712341 -0.47356459]]
Variable
import tensorflow as tf
# Define variable, yet initialized
w = tf.Variable(tf.random_normal([1, 3]), name="weight")
# Define init operation
init_op = tf.global_variables_initializer() with tf.Session() as sess:
sess.run(init_op) # initialize all declared variables w_value = sess.run(w); x_value = sess.run(w)
sess.run(init_op) # re-initialize y_value = sess.run(w)
print '\n'.join(map(str, ([w_value, x_value, y_value])))
>>> [[-0.50282216 -1.94712341 -0.47356459]]
>>> [[-0.50282216 -1.94712341 -0.47356459]] # same
>>> [[ 1.1028192 0.25910807 2.37004709]] # changed
Placeholder
Placeholder for tensor
Must be fed with data on execution for operation
that takes the placeholder as operand
Placeholder
import tensorflow as tf x = [[1, 2]]
y = [[1], [3]]
pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)
with tf.Session() as sess:
A = sess.run(matmul)
print '\n'.join(map(str, ([pla, A])))
>>> "You must feed a value for placeholder tensor 'Placeholder' with dtype int32 and shape [1,2]"
Placeholder
import tensorflow as tf x = [[1, 2]]
y = [[1], [3]]
pla = tf.placeholder(tf.int32, (1,2)) matmul = tf.matmul(pla, y)
with tf.Session() as sess:
A = sess.run(matmul, feed_dict={pla : x}) print '\n'.join(map(str, ([pla, A])))
>>> Tensor("Placeholder:0", shape=(2, 3), dtype=int32)
>>> [[7]]
Shape of Placeholder
Shape could be partially known
import tensorflow as tf
p1 = tf.placeholder(tf.float32, [1, None]) p2 = tf.placeholder(tf.float32, [None, 2])
p3 = tf.placeholder(tf.float32, [None, None]) ops = [tf.multiply(x, 10) for x in [p1, p2, p3]]
mat1 = [[1, 2]]
mat2 = [[3, 4], [5, 6]]
mat3 = [[1], [2]]
with tf.Session() as sess:
print '\n'.join(map(str, sess.run(ops, feed_dict={
p1:mat1, p2:mat2, p3:mat3})))
>>> [[ 10. 20.]]
>>> [[ 30. 40.] [ 50. 60.]]
>>> [[ 10.] [ 20.]]
Shape of Placeholder
Useful when you feed input/output to your model without giving the size of batch
import tensorflow as tf
# Placeholder MNIST
# image size 28x28
x = tf.placeholder(tf.float32, [None, 784])
# 10 classes [0..9]
y_ = tf.placeholder(tf.float32, [None, 10])
# Softmax regression Wx = b
W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b) ...
Operation
Operation is graph node that performs computation on tensors
Computation should be run on Session
It takes inputs and generates outputs(optional)
Usually, it ends up with prediction, cost function, or updating variables
We have already seen some operations through the slides
Operation
Linear Regression (Mean square error)
xW + b = y, L = 2m 1
′∑ m i=1
′( W − x i y i ) 2
import tensorflow as tf, numpy as np
# tf Graph Input
X = tf.placeholder("float") Y = tf.placeholder("float")
# Set model weights
W = tf.Variable(np.random.randn(), name="weight") b = tf.Variable(np.random.randn(), name="bias")
# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)
# Mean square error
Operation
It is better to read through documentation so that you don't have to reinvent the wheel
Math matmul, reduce_sum, argmax, norm
Assign ...
assign, assign_add, assign_sub
Tensor transform ...
slice, split, stack, concat
...
Optimizer
Provides methods to compute gradients for a loss and apply gradients to variables
Getting cost function as input and
Calcualte gradients
Process gradients Apply gradients
"minimize" method on Optmizers do all 3 steps Many optimizers are alreafy implemented
Gradient Descent Optimizer, Adagrad Optimizer, RMSProp Optmizer
...
Optimizer
import tensorflow as tf, numpy as np ...
# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)
# Mean square error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
# Gradient descent optimizer = tf.train\
.GradientDescentOptimizer(learning_rate)\
.minimize(cost) with tf.Session() as sess:
for epoch in range(training_epochs):
for (x, y) in zip(train_X, train_Y): # SGD m'=1 sess.run(optimizer,
feed_dict={X: x, Y: y}) # update weights
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
Session
Session launchs the graph, initalize them, and run operations as you defined
We have seen sessions through the slide, and I hope
you got the idea of what it is
Session
Your data will be lost when you close the session
import tensorflow as tf
var = tf.Variable(tf.random_normal((1, 3))) init = tf.global_variables_initializer() with tf.Session() as sess:
sess.run(init)
print sess.run(var) # Fetch train data # Calcuate cost # Optimize
# Do Test
# Display or Save print var.eval()
"No default session is registered.
Use `with sess.as_default()`
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
MNIST
Famous example for deep learning Data: Hand-writing digit[0-9] image data (28*28)
Each image will be treated as 784 length of vector
Task: Classify what digit in each image
10 classes
Model: So max regression model
Also use 3 hidden layers
MNIST
x = set of images y = set of labels laye = r 1 W 1 x + b 1
laye = r 2 W 2 laye + r 1 b 2 pred = W 3 laye + r 2 b 3
cost = 1 ∑ m
′y ∗ log(softmax(pre − )) d y
Hyper parameters
Reference:
# Parameters
learning_rate = 0.001 training_epochs = 15 batch_size = 100
display_step = 1
# Network Parameters
n_hidden_1 = 256 # 1st layer number of features n_hidden_2 = 256 # 2nd layer number of features
n_input = 784 # MNIST data input (img shape: 28*28) n_classes = 10 # MNIST total classes (0-9 digits)
TensorFlow Exmaples
Placeholder & Variables
import tensorflow as tf
# tf Graph input
x = tf.placeholder("float", [None, n_input]) # input y = tf.placeholder("float", [None, n_classes]) # output
# Store layers weight & bias
h1 = tf.Variable(tf.random_normal([n_input, n_hidden_1])) h2 = tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])) out = tf.Variable(tf.random_normal([n_hidden_2, n_classes])) b1 = tf.Variable(tf.random_normal([n_hidden_1]))
b2 = tf.Variable(tf.random_normal([n_hidden_2])) b3 = tf.Variable(tf.random_normal([n_classes]))
Build a model
# Hidden layer 1 with RELU activation layer_1 = tf.add(tf.matmul(x, h1), b1) layer_1 = tf.nn.relu(layer_1)
# Hidden layer 2 with RELU activation
layer_2 = tf.add(tf.matmul(layer_1, h2), b2) layer_2 = tf.nn.relu(layer_2)
# Output layer with linear activation pred = tf.matmul(layer_2, out) + b3
Optimizer
# Define loss and optimizer
y_hat_softmax = tf.nn.softmax(pred) cost = tf.reduce_mean(
-tf.reduce_sum(y * tf.log(y_hat_softmax), [1])) optimizer = tf.train.AdamOptimizer(
learning_rate=learning_rate)\
.minimize(cost)
# Initializing the variables
init = tf.global_variables_initializer()
Run & Learn
# Launch the graph
with tf.Session() as sess:
sess.run(init) # Training cycle
for epoch in range(training_epochs):
avg_cost = 0.
total_batch = int(mnist.train.num_examples/batch_size) # Loop over all batches
for i in range(total_batch):
batch_x, batch_y = mnist.train.next_batch(batch_size)
# Run optimization op (backprop) and cost op (to get loss value) _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,
y: batch_y}) # Compute average loss
avg_cost += c / total_batch # Display logs per epoch step if epoch % display_step == 0:
print "Epoch:", '%04d' % (epoch+1), "cost=", \ "{:.9f}".format(avg_cost)
print "Optimization Finished!"
# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1)) # Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})
Result
>>> ...
>>> Epoch: 0010 cost= 3.325720582
>>> Epoch: 0011 cost= 2.454958990
>>> Epoch: 0012 cost= 1.861004718
>>> Epoch: 0013 cost= 1.412745127
>>> Epoch: 0014 cost= 1.158697252
>>> Epoch: 0015 cost= 0.943136121
>>> Optimization Finished!
>>> Accuracy: 0.9485
What is TensorFlow?
Consturction Phase Execution Phase
Examples
Gotchas and Tips
Contents
Gotchas & Tips
Don't forget to put value to feed_dict
import tensorflow as tf
pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]])
y = tf.Variable([[3,4],[1,2]])
init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)
with tf.Session() as sess:
sess.run(init)
f_dict = { pla: x } # x is actually tensor "object"
result = sess.run(matmul, feed_dict=f_dict) print result
"ValueError: setting an array element with a sequence."
Gotchas & Tips
Don't forget to put value to feed_dict
import tensorflow as tf
pla = tf.placeholder(tf.int32, [2,2]) x = tf.Variable([[1,2],[3,4]])
y = tf.Variable([[3,4],[1,2]])
init = tf.global_variables_initializer() matmul = tf.matmul(pla, y)
with tf.Session() as sess:
sess.run(init) x_v = sess.run(x)
f_dict = { pla: x_v } # x_v is now "array(2,2)"
result = sess.run(matmul, feed_dict=f_dict) print result
>>> [[ 5 8]
Gotchas & Tips
Better not to make Variable/Placeholder/Operation inside epoch
import tensorflow as tf with tf.Session() as sess:
for epoch in range(100):
# It will make session overflow!
v = tf.Variable([[1,2]]) op = tf.add(1,2)
pla = tf.placeholder(tf.int32, [1,2]) print '\n'.join(map(str, [v, op, pla]))
>>> ...
>>> Tensor("Variable_99/read:0", shape=(1, 2), dtype=int32)
>>> Tensor("Add_99:0", shape=(), dtype=int32)
