I am very new to tensorflow. Now I am implementing a pairwise cnn model. While I got the error:
tensorflow.python.pywrap_tensorflow.StatusNotOK: Not found:
FeedInputs: unable to find feed output Placeholder:0
The error is attributed to the code:
_,train_loss,train_feature1,train_feature2=sess.run([model.train_op,model.cost,model.feature1,model.feature2],feed)
Specifically in sess.run function, feed is a dictionary containing input data and corresponding labels. feature1 and feature2 are the output of the same neural networks and finally I would like to compare feature1 and feature2.
The gist codes are:
with self.graph.device('/gpu:1'):
self.inputs1=tf.placeholder(tf.float32,[self.batchSize,self.inputH,self.inputW,hp.channel])
self.inputs2=tf.placeholder(tf.float32,[self.batchSize,self.inputH,self.inputW,hp.channel])
self.labels=tf.placeholder(tf.float32,[self.batchSize])
self.feature1=forward(self.inputs1)
self.feature2=forward(self.inputs2)
self.cost=tf.reduce_sum( tf.mul( -1.0*self.labels, tf.sub( self.feature1,self.feature2 ) ) )
self.tvars=tf.trainable_variables()
grads=tf.gradients(self.cost,self.tvars)
optimizer=tf.train.MomentumOptimizer(learning_rate=self.learning_rate,momentum=self.momentum)
self.train_op=optimizer.apply_gradients(zip(grads,self.tvars))
Here I create three placeholders, and do some operations like train_op. Forward is the neural networks used in my implementation. Then in the training part, I load text data and labels with data_loader.next_batch(), finally run the optimization operation with sess.run(). The codes is available below.
for b in xrange(data_loader.batch_num):
pbar.update(int((b/data_loader.batch_num)*100) )
time.sleep(0.01)
maps1,maps2,labels=data_loader.next_batch()
feed={model.labels:labels,model.inputs1:maps1,model.inputs2:maps2}
_,train_loss,train_feature1,train_feature2=sess.run([model.train_op,model.cost,model.feature1,model.feature2],feed)
NEW ERROR
I tried to change the maps1 as Tensors instead of numpy ndarray, the error above disappeared. Unfortunately new errors raised.
File "Train.py", line 55, in train _,train_loss,train_feature1,train_feature2=sess.run([model.train_op,model.cost,model.feature1,model.feature2],feed)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 334, in run
np_val = np.array(subfeed_val, dtype=subfeed_t.dtype.as_numpy_dtype)
ValueError: setting an array element with a sequence.
Related
I am running into an error after upgrading to TFF 0.17.0. The same code works perfectly in TFF 0.16.1. The training works just fine in both versions however when I try to copy weights from the FL state to model to evaluate it on test dataset, I get the following error:
File "fl/main_fl.py", line 166, in keras_evaluate
loss, accuracy = self.model.evaluate(test_dataset, verbose=0)
File "/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_v1.py", line 905, in evaluate
self._assert_built_as_v1()
File "/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 852, i$ _assert_built_as_v1
(type(self),))
ValueError: Your Layer or Model is in an invalid state. This can happen for the following cases:
1. You might be interleaving estimator/non-estimator models or interleaving models/layers made in tf.compat.v1.Graph.as_default() with model$/layers created outside of it. Converting a model to an estimator (via model_to_estimator) invalidates all models/layers made before the conv$rsion (even if they were not the model converted to an estimator). Similarly, making a layer or a model inside a a tf.compat.v1.Graph invalid$tes all layers/models you previously made outside of the graph.
2. You might be using a custom keras layer implementation with custom __init__ which didn't call super().__init__. Please check the impleme$tation of <class 'tensorflow.python.keras.engine.functional.Functional'> and its bases.
Below is my keras_evaluate method:
def keras_evaluate(self, test_dataset, mode='test', step=0):
self.state.model.assign_weights_to(self.model)
loss, accuracy = self.model.evaluate(test_dataset, verbose=0)
print('Mode={}, Loss={}, Accuracy={}'.format(mode, loss, accuracy))
self.state is the state returned by tff.learning.build_federated_averaging_process i.e tff.templates.IterativeProcess, test_dataset is of type tf.data.Dataset and self.model is tf.keras.Model type i.e keras functional model. I have one custom layer however it does have super() method so point 2 in the error is misleading me.
Any help will be appreciated.
The code for this problem is quite complex because I'm trying to implement fractalNet but changing the convolution base block to just a dense layer. I'm trying to separately build two fractalNets (one after the other so I don't think they should be interfering). One for the policy and one for the value function.
There are also a number of issues I have seen so far that may or may not be related. One is that I can't import numpy as np and use np which is why I've been forced to use numpy(). The other is that my code seems to trying to be working on tensors tf.Tensor[stuff] as well as Tensor[stuff] in different sections at the same time. The build_model function below outputs Tensor[stuff] from the Input call whereas the neural network builder code uses tf.Tensor[stuff]. I tried but to no avail to stick to type.
Here is the complete error that keeps killing the code:
/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py:190: UserWarning: Model inputs must come from `keras.layers.Input` (thus holding past layer metadata), they cannot be the output of a previous non-Input layer. Here, a tensor specified as input to your model was not an Input tensor, it was generated by layer activation_1.
Note that input tensors are instantiated via `tensor = keras.layers.Input(shape)`.
The tensor that caused the issue was: activation_1/Relu:0
str(x.name))
Traceback (most recent call last):
File "train.py", line 355, in <module>
main(**vars(args))
File "train.py", line 302, in main
val_func = NNValueFunction(bl,c,layersizes,dropout,deepest,obs_dim) # Initialize the value function
File "/home/ryan/trpo_fractalNN/trpo/value.py", line 37, in __init__
self.model = self._build_model()
File "/home/ryan/trpo_fractalNN/trpo/value.py", line 56, in _build_model
model = Model(inputs=obs_input, outputs=outputs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 94, in __init__
self._init_graph_network(*args, **kwargs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 241, in _init_graph_network
self.inputs, self.outputs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 1511, in _map_graph_network
str(layers_with_complete_input))
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 29), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
So here is the part of the code that I'm suspicious of at the moment because of the fact that somehow it is breaking at the very beginning on the value function's neural net.
def _build_model(self):
""" Construct TensorFlow graph, including loss function, init op and train op """
# hid1 layer size is 10x obs_dim, hid3 size is 10, and hid2 is geometric mean
# hid3_units = 5 # 5 chosen empirically on 'Hopper-v1'
# hid2_units = int(np.sqrt(hid1_units * hid3_units))
# heuristic to set learning rate based on NN size (tuned on 'Hopper-v1')
obs = keras.layers.Input(shape=(self.obs_dim,))
# I'm not sure why it won't work with np??????????????????????????????????????????????????????????????????????????????????
obs_input = Dense(int(self.layersizes[0][0].numpy()))(obs) # Initial fully-connected layer that brings obs number up to a len that will work with fractal architecture
obs_input = Activation('relu')(obs_input)
self.lr = 1e-2 / np.sqrt(self.layersizes[2][0]) # 1e-2 empirically determined
print('Value Params -- lr: {:.3g}'
.format(self.lr))
outputs = fractal_net(self,bl=self.bl,c=self.c,layersizes=self.layersizes,
drop_path=0.15,dropout=self.dropout,
deepest=self.deepest)(obs_input)
model = Model(inputs=obs_input, outputs=outputs)
optimizer = Adam(self.lr)
model.compile(optimizer=optimizer, loss='mse')
return model
I found out the issue. The problem was that since I was trying to combine multiple files, I had a 'Dense' call to bring the obs_len to the desired size and then took that and plugged it into the fractalNet code. However, I didn't realize that this would break things. I solved the issue by removing the initial Dense call and placing it inside the fractalNet code itself.
So moral of the story, don't try to break up different parts of the NN layers into separate files. Just as a side comment, In the current fractalNN code, it calls fractal_net and then a Dense layer afterwards and apparently this still works. But I think it breaks things to try to reverse this order. I hope this helps someone else.
Update: As rvinas pointed out, I had forgotten to add inputs_aux as second input in Model. Fixed now, and it works. So ConditionalRNN can readily be used to do what I want.
I'd like to treat time-series together with non-time-series characteristics in extended LSTM cells (a requirement also discussed here). ConditionalRNN (cond-rnn) for Tensorflow in Python seems to allow this.
Can it be used in Keras Functional API (without eager execution)?
That is, does anyone have a clue how to fix my failed approach below, or a different example where ConditionalRNN (or alternatives) are used to readily combine TS and non-TS data in LSTM-style cells or any equivalent?
I've seen the eager execution-bare tf example on Pilippe Remy's ConditionalRNN github page, but I did not manage to extend it to a readily fittable version in Keras Functional API.
My code looks as follows; it works if, instead of the ConditionalRNN, I use a standard LSTM cell (and adjust the model 'x' input correspondingly). With ConditionalRNN, I did not get it to execute; I receive either the must feed a value for placeholder tensor 'in_aux' error (cf. below), or instead some different type of input size complaints when I change the code, despite trying to be careful about data dimensions compatibility.
(Using Python 3.6, Tensorflow 2.1, cond-rnn 2.1, on Ubuntu 16.04)
import numpy as np
from tensorflow.keras.models import Model
from tensorflow.keras.layers import LSTM, Dense, Input
from cond_rnn import ConditionalRNN
inputs = Input(name='in',shape=(5,5)) # Each observation has 5 dimensions à 5 time-steps each
x = Dense(64)(inputs)
inputs_aux = Input(name='in_aux', shape=[5]) # For each of the 5 dimensions, a non-time-series observation too
x = ConditionalRNN(7, cell='LSTM')([x,inputs_aux]) # Updated Syntax for cond_rnn v2.1
# x = ConditionalRNN(7, cell='LSTM', cond=inputs_aux)(x) # Syntax for cond_rnn in some version before v2.1
predictions = Dense(1)(x)
model = Model(inputs=[inputs, inputs_aux], outputs=predictions) # With this fix, [inputs, inputs_aux], it now works, solving the issue
#model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics=['mse'])
data = np.random.standard_normal([100,5,5]) # Sample of 100 observations with 5 dimensions à 5 time-steps each
data_aux = np.random.standard_normal([100,5]) # Sample of 100 observations with 5 dimensions à only 1 non-time-series value each
labels = np.random.standard_normal(size=[100]) # For each of the 100 obs., a corresponding (single) outcome variable
model.fit([data,data_aux], labels)
The error I get is
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'in_aux' with dtype float and shape [?,5]
[[{{node in_aux}}]]
and the traceback is
Traceback (most recent call last):
File "/home/florian/temp_nonclear/playground/test/est1ls_bare.py", line 20, in <module>
model.fit({'in': data, 'in_aux': data_aux}, labels) #model.fit([data,data_aux], labels) # Also crashes when using model.fit({'in': data, 'in_aux': data_aux}, labels)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py", line 643, in fit
use_multiprocessing=use_multiprocessing)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 664, in fit
steps_name='steps_per_epoch')
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 383, in model_iteration
batch_outs = f(ins_batch)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/backend.py", line 3353, in __call__
run_metadata=self.run_metadata)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
run_metadata_ptr)
I noticed that you are not passing inputs_aux as input to your model. TF is complaining because this tensor is required to compute your output predictions and it is not being fed with any value. Defining your model as follows should solve the problem:
model = Model(inputs=[inputs, inputs_aux], outputs=predictions)
TF1.4 made Keras an integral part.
When trying to create Estimators from Keras models with propratery input function (I.e., not using the tf.estimator.inputs.numpy_input_fn) things are not working as Tensorflow can not fuse the model with the Input function.
I am using tf.keras.estimator.model_to_estimator
keras_estimator = tf.keras.estimator.model_to_estimator(
keras_model = keras_model,
config = run_config)
train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn,
max_steps=self.train_steps)
eval_spec = tf.estimator.EvalSpec(input_fn=eval_input_fn,
steps=None)
tf.estimator.train_and_evaluate(keras_estimator, train_spec, eval_spec)
and I get the following error message:
Cannot find %s with name "%s" in Keras Model. It needs to match '
'one of the following:
I found some reference for this topic here (strangely enough its hidden in the TF docs in the master branch - compare to this)
If you have the same issue - see my answer below. Might save you several hours.
So here is the deal. You must make sure that your custom Input Function returns a dictionary of {inputs} and a dictionary of {outputs}.
The dictionary keys must match your Keras input/output layers name.
From TF docs:
First, recover the input name(s) of Keras model, so we can use them as the
feature column name(s) of the Estimator input function
This is correct.
Here is how I did this:
# Get inputs and outout Keras model name to fuse them into the infrastructure.
keras_input_names_list = keras_model.input_names
keras_target_names_list = keras_model.output_names
Now, that you have the names, you need to go to your own input function and change it so it will deliver two dictionaries with the corresponding input and output names.
In my example, before the change, the input function returned [image_batch],[label_batch]. This is basically a bug because it is stated that the inputfn returns a dictionary and not a list.
To solve this, we need to wrap it up into a dict:
image_batch_dict = dict(zip(keras_input_names_list , [image_batch]))
label_batch_dict = dict(zip(keras_target_names_list , [label_batch]))
Only now, TF will be able to connect the input function to the Keras input layers.
I am trying to train a model using Tensorflow's SVM canned estimator. My input data is a queue sourced from Big Query using Tensorflow's BigQueryReader which then feeds a shuffle_batch queue. My features are Rank 2 and my example keys are Rank 1. The actual shape of the key and feature tensors aren't fully defined since the number of training/eval examples can vary. I am getting the following error during training:
File "/root/.local/lib/python2.7/site-
packages/tensorflow/contrib/linear_optimizer/python/ops/sdca_ops.py",
line 313, in minimize example_state_data =self._hashtable.lookup(example_ids_hashed)
File "/root/.local/lib/python2.7/site-packages/tensorflow/contrib/linear_optimizer/python/ops/sharded_mutable_dense_hashtable.py",
line 108, in lookup self._check_keys(keys)
File "/root/.local/lib/python2.7/site-packages/tensorflow/contrib/linear_optimizer/python/ops/sharded_mutable_dense_hashtable.py",
line 99, in _check_keys keys.get_shape())
ValueError: Key shape must be fully defined, got (?, 2)
The issue seems to be tied specifically to the processing of the example key tensor. Has anyone else run into this issue? I have tried very similar code with the other canned estimators and haven't gotten a complaint about the shape not being fully defined. However, this is the first estimator that required keys. When the data is parsed by parse_example after being read from BigQuery, the output are Rank-2 tensors with shape (?,1). In order to create a rank-1 tensor for the keys I am using the reshape function which results in shape (?,). I'm not sure if this could be contributing to my problem:
features = tf.parse_example(examples_serialized, features=FEATURES)
features[key_column] = tf.reshape(features[key_column],[-1])