`tape` is required when a `Tensor` loss is passed - tensorflow

Some question about tf.
import numpy as np
import tensorflow as tf
from tensorflow import keras
x_train = [1,2,3]
y_train = [1,2,3]
W = tf.Variable(tf.random.normal([1]), name = 'weight')
b = tf.Variable(tf.random.normal([1]), name = 'bias')
hypothesis = W*x_train+b
optimizer = tf.optimizers.SGD (learning_rate=0.01)
train = tf.keras.optimizers.Adam().minimize(cost, var_list=[W, b])
As I start the last line of my code, below error comes out.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-52-cd6e22f66d09> in <module>()
----> 1 train = tf.keras.optimizers.Adam().minimize(cost, var_list=[W, b])
1 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py in _compute_gradients(self, loss, var_list, grad_loss, tape)
530 # TODO(josh11b): Test that we handle weight decay in a reasonable way.
531 if not callable(loss) and tape is None:
--> 532 raise ValueError("`tape` is required when a `Tensor` loss is passed.")
533 tape = tape if tape is not None else backprop.GradientTape()
534
ValueError: `tape` is required when a `Tensor` loss is passed.
I know it is related with tensorflow version 2, but don't want to modify to version 1.
Want a solution with tensorflow ver2. Thx.

Since you did not provided the cost function, I added one. Here is the code
import numpy as np
import tensorflow as tf
from tensorflow import keras
x_train = [1,2,3]
y_train = [1,2,3]
W = tf.Variable(tf.random.normal([1]), name = 'weight')
b = tf.Variable(tf.random.normal([1]), name = 'bias')
hypothesis = W*x_train+b
#tf.function
def cost():
y_model = W*x_train+b
error = tf.reduce_mean(tf.square(y_train- y_model))
return error
optimizer = tf.optimizers.SGD (learning_rate=0.01)
train = tf.keras.optimizers.Adam().minimize(cost, var_list=[W, b])
tf.print(W)
tf.print(b)

Related

ValueError: tape is required when a Tensor loss is passed

As said in https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer?hl=en#minimize, the first parameter of minmize should satisfy the requirement,
Tensor or callable. If a callable, loss should take no arguments and return the value to minimize. If a Tensor, the tape argument must be passed.
The first piece of code takes tensor as the input of minimize(), and it requires the gradient tape, but I don't know how.
The second piece of code takes callable function as the input of minimize(), which is easy
import numpy as np
import tensorflow as tf
from tensorflow import keras
x_train = [1, 2, 3]
y_train = [1, 2, 3]
W = tf.Variable(tf.random.normal([1]), name='weight')
b = tf.Variable(tf.random.normal([1]), name='bias')
hypothesis = W * x_train + b
#tf.function
def cost():
y_model = W * x_train + b
error = tf.reduce_mean(tf.square(y_train - y_model))
return error
optimizer = tf.optimizers.SGD(learning_rate=0.01)
cost_value = cost()
train = tf.keras.optimizers.Adam().minimize(cost_value, var_list=[W, b])
tf.print(W)
tf.print(b)
How to add the gradient tape, I know the following code certainly works.
import numpy as np
import tensorflow as tf
from tensorflow import keras
x_train = [1, 2, 3]
y_train = [1, 2, 3]
W = tf.Variable(tf.random.normal([1]), name='weight')
b = tf.Variable(tf.random.normal([1]), name='bias')
hypothesis = W * x_train + b
#tf.function
def cost():
y_model = W * x_train + b
error = tf.reduce_mean(tf.square(y_train - y_model))
return error
optimizer = tf.optimizers.SGD(learning_rate=0.01)
cost_value = cost()
train = tf.keras.optimizers.Adam().minimize(cost, var_list=[W, b])
tf.print(W)
tf.print(b)
Please help me revise the first piece of code and let it run, thanks!
This occurs because .minimize() expects a function. While cost_value&cost(), is a tf.Tensor object, cost is a tf.function. You should directly pass your loss function into the minimize as tf.keras.optimizers.Adam().minimize(cost, var_list=[W, b]).
Changed part for Gradient:
train = tf.keras.optimizers.Adam().minimize(cost(), var_list=[W, b],tape=tf.GradientTape())
This is a late answer (Hakan basically got it for you), but I write this in hopes that it will help people in the future that are stuck and googling this exact question (like I was). This is also an alternate implementation using the tf.GradientTape() directly.
import numpy as np
import tensorflow as tf
from tensorflow import keras
x_train = [1, 2, 3]
y_train = [1, 2, 3]
W = tf.Variable(tf.random.normal([1]), trainable = True, name='weight')
b = tf.Variable(tf.random.normal([1]), trainable = True, name='bias')
#tf.function
def cost(W, b):
y_model = W * x_train + b
error = tf.reduce_mean(tf.square(y_train - y_model))
return error
optimizer = tf.optimizers.SGD(learning_rate=0.01)
trainable_vars = [W,b]
epochs = 100 #(or however many iterations you want it to run)
for _ in range(epochs):
with tf.GradientTape() as tp:
#your loss/cost function must always be contained within the gradient tape instantiation
cost_fn = cost(W, b)
gradients = tp.gradient(cost_fn, trainable_vars)
optimizer.apply_gradients(zip(gradients, trainable_vars))
tf.print(W)
tf.print(b)
This should give you the value of your weights and biases after the number of epochs you ran.
You must compute the loss function everytime a new gradient tape is invoked. Then you get the gradient of your loss function, and then call optimizer.apply_gradient to do your minimization according to what tensorflow documentation says here: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer#apply_gradients.

Using GradientTape with a SavedModel to do a gradient descent on the input

I am trying to make an adversarial attack for a model I loaded using the SavedModel API. I want to do a gradient descent of my input with respect to the loss of the model given a target. The code is a little bit long but it's the bare minimum for illustrating the problem.
from __future__ import absolute_import, division, print_function, unicode_literals
from tensorflow import keras
from tensorflow.keras import layers, models
import tensorflow as tf
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# creating the dataset
X, y = make_classification(n_samples=10000, n_informative=10)
X, X_test, y, y_test = train_test_split(X, y)
# training the model
model = models.Sequential()
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(2, activation='softmax', name="output"))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(X, y,
epochs=10,
batch_size=32,
verbose=0)
print(f"final accuracy is {model.evaluate(X_test, y_test, verbose=0)[1]}")
# saving and loading it
tf.saved_model.save(model, "/content/demo")
loaded_model = tf.saved_model.load("/content/demo")
inference_func = loaded_model.signatures["serving_default"]
# making the adversarial example
x = tf.random.normal([1, 20])
x = tf.Variable(x)
target = tf.convert_to_tensor([0, 1], dtype=tf.float32)
cce = tf.keras.losses.CategoricalCrossentropy()
with tf.GradientTape() as t:
t.watch(x)
y = inference_func(x)["output"]
loss = cce(target, y)
dl_dx = t.gradient(loss, x)
x.assign_sub(learning_rate * dl_dx)
print(x.numpy())
And I get the following error:
final accuracy is 0.8899999856948853
INFO:tensorflow:Assets written to: /content/demo/assets
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-31-1b61c316b9dc> in <module>()
40 with tf.GradientTape() as t:
41 t.watch(x)
---> 42 y = inference_func(x)["output"]
43 loss = cce(target, y)
44 dl_dx = t.gradient(loss, x)
6 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
InvalidArgumentError: cannot compute __forward_signature_wrapper_147414 as input #0(zero-based) was expected to be a float tensor but is a resource tensor [Op:__forward_signature_wrapper_147414]
I quite new to low level tensorflow and I don't really understand how it works yet. I believe the problem is linked to my inference function not being a real #tf.function since it's type is tensorflow.python.saved_model.load._WrapperFunction. But how can I retrieve the real function then?
I figured it out ! So the function I was looking for was in loaded_model.__call__. I don't know why tensorflow doc does not explain that clearly.

Having AttributeError problem with __enter__ while using tensorflow

import tensorflow as tf
import numpy as np
tf.enable_eager_execution()
x_data = [[1,2,1,1],[2,1,3,2],[3,1,3,4],[4,1,5,5],[1,7,5,5],[1,2,5,6],[1,6,6,6],[1,7,7,7]]
y_data = [[0,0,1],[0,0,1],[0,0,1],[0,1,0],[0,1,0],[0,1,0],[1,0,0],[1,0,0]]
x_data = np.asarray(x_data, dtype=np.float32)
y_data = np.asarray(y_data, dtype=np.float32)
nb_classes = 3
W = tf.Variable(tf.random_normal([4, nb_classes]), name = 'weight')
b = tf.Variable(tf.random_normal([nb_classes]), name = 'bias')
variables = [W,b]
def hypothesis(X):
hypo = tf.nn.softmax(tf.matmul(X,W) + b)
return hypo
def cost_fn(X,Y):
logits = hypothesis(X)
cost = -tf.reduce_sum(Y * tf.log(logits), axis = 1)
cost_mean = tf.reduce_mean(cost)
return cost_mean
def grad_fn(X,Y):
with tf.GradientTape as tape:
cost = cost_fn(X,Y)
grads = tape.gradient(cost, variables)
return grads
So i was trying about classifications and was making a gradient function for gradient descent optimizer.
and the error occured at the last part of the code
with tf.GradientTape as tape:
AttributeError : enter occured and i don't understand why. Can i get a reason for the error or way to solve it?
You are missing the parenthesis in your GradientTape. It should be as follows.
def grad_fn(X,Y):
with tf.GradientTape() as tape:
cost = cost_fn(X,Y)
grads = tape.gradient(cost, variables)
return grads

Reshaping arrays in Time-Series RNN Beginners Block

New to python and deep learning. I was trying to build an RNN with some data and I don't know where am I going wrong.
This is my code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
raw = pd.read_excel('Online Retail.xlsx',index_col='InvoiceDate')
sales = raw.drop(['InvoiceNo','StockCode','Country','Description'],axis=1)
sales.head()
sales.index = pd.to_datetime(sales.index)
sales.info()
train_set = sales.head(50000)
test_set = sales.tail(41909)
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
training = np.nan_to_num(train_set)
testing = np.nan_to_num(test_set)
train_scaled = scaler.fit_transform(training)
test_scaled = scaler.fit_transform(testing)
def next_batch(training_data,batch_size,steps):
rand_start = np.random.randint(0,len(training_data)-steps)
y_batch =
np.array(training_data[rand_start:rand_start+steps+1].reshape(26,steps+1))
return
y_batch[:,:-1].reshape(-1,steps,1),y_batch[:,1:].reshape(-1,steps,1)
import tensorflow as tf
num_inputs = 1
num_time_steps = 10
num_neurons = 100
num_outputs = 1
learning_rate = 0.03
num_train_iterations = 4000
batch_size = 1
X = tf.placeholder(tf.float32,[None,num_time_steps,num_inputs])
y = tf.placeholder(tf.float32,[None,num_time_steps,num_outputs])
cell = tf.contrib.rnn.OutputProjectionWrapper(
tf.contrib.rnn.BasicLSTMCell(num_units=num_neurons,activation=tf.nn.relu),output_size=num_outputs)
outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)
loss = tf.reduce_mean(tf.square(outputs - y)) # MSE
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
train = optimizer.minimize(loss)
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session(config=tf.ConfigProto()) as sess:
sess.run(init)
for iteration in range(num_train_iterations):
X_batch, y_batch = next_batch(train_scaled,batch_size,num_time_steps)
sess.run(train, feed_dict={X: X_batch, y: y_batch})
if iteration % 100 == 0:
mse = loss.eval(feed_dict={X: X_batch, y: y_batch})
print(iteration, "\tMSE:", mse)
# Save Model for Later
saver.save(sess, "./ex_time_series_model")
The output:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-36-f2f7c66a33df> in <module>()
4 for iteration in range(num_train_iterations):
5
----> 6 X_batch, y_batch = next_batch(train_scaled,batch_size,num_time_steps)
7 sess.run(train, feed_dict={X: X_batch, y: y_batch})
8
<ipython-input-26-f673a469c67d> in next_batch(training_data, batch_size, steps)
1 def next_batch(training_data,batch_size,steps):
2 rand_start = np.random.randint(0,len(training_data)-steps)
----> 3 y_batch = np.array(training_data[rand_start:rand_start+steps+1].reshape(26,steps+1))
4 return y_batch[:,:-1].reshape(-1,steps,1),y_batch[:,1:].reshape(-1,steps,1)
ValueError: cannot reshape array of size 33 into shape (26,11)
In [ ]:
I'm not sure where the number 26 came from, but it doesn't match with your data dimensions. After you dropped four columns, the training_data array is (50000, 3), of which you take (11, 3) batches. This array obviously can't reshape to (26, 11).
What you probably meant is this (in next_batch function):
y_batch = np.array(training_data[rand_start:rand_start+steps+1].reshape(3,steps+1))
The error says that you trying to reshape a tensor with size 33 into a tensor with size 26x11, which you can't. You should reshape a tensor with size 286 into 26x11.
Try to debug the next_batch function by printing the y_batch shape in each step using print (y_batch.get_shape()) and check it, if it has shape 286.
I didn't catch this point, why you fetch each batch randomly? why didn't you read input data normally?
It would be good if you fix the indents when you posting your code, it is hard to track.

How should I fix that after I use a slice of a tensor, tensorflow optimizer will break?

Here is a sample:
import numpy as np
import tensorflow as tf
from tensorflow.python.ops import rnn, rnn_cell
if __name__ == '__main__':
embs = tf.Variable(np.random.random((40,5)),dtype=tf.float32)
X = np.array(np.array(range(1,25)).reshape(4, 6))
x0 = tf.placeholder(tf.int32, [None, None])
x1 = tf.nn.embedding_lookup(embs, x0)
lstm = tf.nn.rnn_cell.BasicLSTMCell(5,state_is_tuple=True)
outputs, states = tf.nn.dynamic_rnn(lstm, x1, dtype=tf.float32,time_major = True)
cost = tf.reduce_mean(outputs[:,-1,:])
optimizer = tf.train.AdagradOptimizer(learning_rate=0.12).minimize(cost)
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
result3, opt = sess.run([outputs, optimizer],{x0:X})
I use just one slice of outputs which is outputs[:,-1,:] to get a cost function. When I run the code, I got the result
F ./tensorflow/core/framework/tensor.h:581] Check failed: new_num_elements == NumElements() (0 vs. 20)
How to fix this? It's just a sample. I met this problem when I implement a hierarchical LSTM in which the representations of sentences computed by a LSTM is feed into another LSTM.
I confirmed that this is a bug in TensorFlow 0.10. Upgrading to TensorFlow 0.11 will fix the problem.