I want to understand the error message of the following code
M = K.eye(2)
K.assign(M[0,1],1.0)
The message I got is "Tried to convert 'input' to a tensor and failed. Error: None values not supported."
You can assign an element of a variable in tensorflow. Here is an example. (I didn't find K.assign this operation in my installed tensorflow version, btw)
import tensorflow as tf
import keras.backend as K
M = tf.Variable(K.eye(2), tf.float32)
assign_op = tf.assign(M[0,1], 1.0)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run(M))
sess.run(assign_op)
print(sess.run(M))
#[[1. 0.]
# [0. 1.]]
#[[1. 1.]
# [0. 1.]]
Related
This is probably a very simple question, however I am fairly new to tensorflow and have been stuck at this issue. I use tensorflow 1.12 and python 3.
My question is, what is the proper way to set the shape of a tensor object that is returned by the iterator?
With placeholders I can make make something like this code work, but I would like to make this work without a placeholder and using tensorflow datasets.
I cannot figure out how to align the shape of a tensor with a matrix in order to use tf.matmul.
The error I receive is: ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_19' (op: 'MatMul') with input shapes: [2], [2,1].
The dataset of the iterator is specified as: TensorSliceDataset shapes: (2,), types: tf.float32>.
Thanks in advance!
import tensorflow as tf
import numpy as np
batch_size = 200
# this simulates a dataset read from a csv.....
x=np.array([[0., 0.], [1., 0.], [0., 1.], [1., 1.]],dtype="float32")
y=np.array([0, 0, 0, 1],dtype="float32")
dataset = tf.data.Dataset.from_tensor_slices((x))
print(dataset) # <TensorSliceDataset shapes: (2,), types: tf.float32>
dataset = dataset.repeat(10000)
print('repeat ds ', dataset) # repeat ds <RepeatDataset shapes: (2,), types: tf.float32>
iter = dataset.make_initializable_iterator()
print('iterator ', iter) # iterator <tensorflow.python.data.ops.iterator_ops.Iterator object at 0x0000028589C62550>
sess = tf.Session()
sess.run(iter.initializer)
next_elt= iter.get_next()
print('shape of dataset ', dataset , '[iterator] elt ', next_elt) # shape of dataset <RepeatDataset shapes: (2,), types: tf.float32> [iterator] elt Tensor("IteratorGetNext_105:0", shape=(2,), dtype=float32)
print('shape of it ', next_elt.shape) #s hape of it (2,)
for i in range(4):
print(sess.run(next_elt))
''' outputs:
[0. 0.]
[1. 0.]
[0. 1.]
[1. 1.]
'''
w = tf.Variable(tf.random_uniform([2,1], -1, 1, seed = 1234),name="weights_layer_1")
# this is where the error is because of shape mismatch of iterator and w variable.
# How od I make the shape of the iterator (2,1) so that matmul can be used?
# What is the proper way of aligning a tensor shape with inut data
# The output of the error:
# ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_19' (op: 'MatMul') with input shapes: [2], [2,1].
H = tf.matmul( sess.run(next_elt) , w)
You can use tf.reshape. Just add tf.reshape(next_elt, [1,2]) prior to matmul op
More on reshape https://www.tensorflow.org/api_docs/python/tf/reshape
Assuming training is finished: what values does Keras use for the 0th cell state and hidden states at inference (in LSTM and RNN layers)? I could think of at least three scenarios, and could not find any conclusive answer in the documentation:
(a) The initial states are learned and then used for all predictions
(b) or the initial states are always set at zero
(c) the initial states are always random (let's hope not...?)
If using LSTM(stateful=True), hidden states are initialized to zero, change with fit or predict, and are kept at whatever they are until .reset_states() is called. If LSTM(stateful=False), states are reset after fitting/predicting/etc each batch.
This can be verified from the .reset_states() source code, and by direct inspection; both for stateful=True below. For more info on how states are passed, see this answer.
Direct inspection:
batch_shape = (2, 10, 4)
model = make_model(batch_shape)
X = np.random.randn(*batch_shape)
y = np.random.randint(0, 2, (batch_shape[0], 1))
show_lstm_states("STATES INITIALIZED")
model.train_on_batch(X, y)
show_lstm_states("STATES AFTER TRAIN")
model.reset_states()
show_lstm_states("STATES AFTER RESET")
model.predict(X)
show_lstm_states("STATES AFTER PREDICT")
Output:
STATES INITIALIZED
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
STATES AFTER TRAIN
[[0.12061571 0.03639204 0.20810013 0.05309075]
[0.01832913 0.00062357 0.10566339 0.60108346]]
[[0.21241754 0.0773523 0.37392718 0.15590034]
[0.08496398 0.00112716 0.23814857 0.95995367]]
STATES AFTER RESET
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]]
STATES AFTER PREDICT
[[0.12162527 0.03720453 0.20628096 0.05421837]
[0.01849432 0.00064993 0.1045063 0.6097021 ]]
[[0.21398112 0.07894284 0.3709934 0.15928769]
[0.08605779 0.00117485 0.23606434 0.97212094]]
Functions / imports used:
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.layers import Input, Dense, LSTM
from tensorflow.keras.models import Model
import numpy as np
def make_model(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = LSTM(4, stateful=True, activation='relu')(ipt)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile('adam', 'binary_crossentropy')
return model
def show_lstm_states(txt=''):
print('\n' + txt)
states = model.layers[1].states
for state in states:
if tf.__version__[0] == '2':
print(state.numpy())
else:
print(K.get_value(state))
Inspect source code:
from inspect import getsource
print(getsource(model.layers[1].reset_states))
My understanding from this is that they are initialized to zero in most cases.
I am currently using tensorflow 1.7
For some reason the optimizer gradients are not being applied. Here is my simple code:
import tensorflow as tf
import numpy as np
print("tensorflow version={}".format(tf.__version__))
a= np.zeros((2,2), dtype="float32")
b= np.array([[6,7],[8,9]], dtype="float32")
t1= tf.placeholder(tf.float32, shape=(2,2))
label_t = tf.placeholder(tf.float32, shape=(2,2))
t2 = tf.layers.dense(t1,2, activation=tf.nn.relu)
loss = tf.reduce_sum(tf.square(t2 - label_t))
optimizer = tf.train.AdamOptimizer(0.1)
train_op = optimizer.minimize(loss)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
steps=2
for i in range(steps):
print("trainop", train_op)
pred, loss_val, _= sess.run([t2,loss, train_op], feed_dict={t1:a, label_t:b})
print("pred={}, loss={}".format(pred, loss_val))
The output is:
trainop name: "Adam"
op: "NoOp"
input: "^Adam/update_dense/kernel/ApplyAdam"
input: "^Adam/update_dense/bias/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
pred=[[0. 0.]
[0. 0.]], loss=230.0
trainop name: "Adam"
op: "NoOp"
input: "^Adam/update_dense/kernel/ApplyAdam"
input: "^Adam/update_dense/bias/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
pred=[[0. 0.]
[0. 0.]], loss=230.0
I have tried separating the minimize() into compute_gradient and apply_gradient. Even so, the gradients do not seem to be applied(loss value remains the same and the trainable variables stay the same).
Can anyone give me a hint on what I am doing wrong?
as #BugKiller helped me out in the comments, the cause was my bad choice of activation function. I can see the loss diminishing if I remove the explicit activation assignment. And as #BugKiller has kindly explained in the comments, it was my bad from the beginning to blindly use the tf.nn.relu function as the activation function.
So I am trying to initialize my variable with some specific weights:
W = tf.Variable(np.eye(19) , name = 'Diag')
But if i now run this code:
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(W.eval())
I end up with a zeros matrix:
[[ 0. 0. 0.]
[ 0. 0. 0.]
[ 0. 0. 0.]]
I do not understand what is going on since e.g.
W = tf.Variable([1,2,3] , name = 'Diag')
preserves the values. What should I do?
I guess the tf.initialize_all_variables() overwrites my values but without it I get the FailedPreconditionError that complains about unitialized variables.
I'm using a softmax function for a binary classification task.
My test label is a one hot list and looks like:
test_y = [[1. 0.] [1. 0.]…]
And the predicted label is a probability list:
test_y_pred = [[ 4.39091297e-09 1.00000000e+00]
[ 1.75207238e-10 1.00000000e+00] …]
When I try to use the f1_score, I get an error :
ValueError: Can't handle mix of binary and continuous
How can I handle this issue?
Thanks
f1_score will not classify the results for you.
Change your predictions to vectors of classes, for example:
import numpy as np
test_y = [np.argmax(prediction) for prediction in test_y]
test_y_pred= [np.argmax(prediction) for prediction in test_y_pred]