Assuming the two models has been established in tensorflow,the model1 followed by model2.
The condition is that the output's type of model1 is a "tensor" and
the input type of model2 is requiring "ndarray" in creating structure of graph's model.(the data don't flow the graph) If we haven't build two or more Session, how we can combine model1 with model2.
(In fact, The library fuction requiring the input's type is "ndarray" can be call in model2. I don't want to code this process)
The sample is following
import tensorflow as tf
import cv2
img = cv2.read("star_sky.jpg")#assumpting shape of image is (256,256,3)
x_input = tf.placeholder(shape=(1,256,256,3),dtype=tf.float32)
W = tf.Variable(tf.random_normal([3,3,3,3]),dtype = tf.float32)
x_output_temp = tf.nn.conv2d(x_input,W,[1,1,1,1],padding="SAME")
#the other model want to use x_output to get Canny edge of image
x_output_ = x_output_temp[0]
x_output = cv2.Canny(x_output_,100,200)#number is parameter of threshold
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
img = [img]
x_output.eval({x_input:img})
If you want to use cv2 to process a tensorflow Tensor you need to do it inside a tf.py_func (which will convert the tensor to an ndarray at graph execution time and run the python code you pass on that array)
Related
The following code (based on https://keras.io/getting-started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer) allows us to print the output of an intermediate layer of a model, given some input, which we need to provide. Specifically, in this example, I am printing the output of the layer dense, given the input to the layer input_1, which also happens to be the input layer of my model (but this does not have to be like that).
import numpy as np
import tensorflow as tf
def get_model():
inp = tf.keras.layers.Input(shape=(1,))
x = tf.keras.layers.Dense(8)(inp)
x = tf.keras.layers.Dense(16)(x)
out = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs=inp, outputs=out)
model.summary()
return model
def train():
my_model = get_model()
my_model.compile(optimizer="adam", loss="mse")
get_layer_output = tf.keras.backend.function([my_model.get_layer("input_1").input],
[my_model.get_layer("dense").output])
data_x = np.array([[1], [2], [3], [4]])
layer_output = get_layer_output(data_x)[0]
print(layer_output)
if __name__ == '__main__':
train()
However, I would like to print the output of each layer, given the output of the corresponding previous layer (as defined by the model), while training the model, i.e. after each mini-batch. I tried to use a callback, which calls tf.print for printing the output of a model, but I am getting an error, which is described in this Github issue (i.e. there is a bug in TensorFlow 2.0, which is the version I am using and that I want to use).
To be clearer, I would like to debug the output of each layer while I am training the model, so that I can understand how the inputs flow throughout each layer and if the activations are not too high (explode) or too small (vanish). I could iteratively provide a batch of data to get_layer_output, but I would like to be able to print the activations of each layer while training the model with fit or fit_generator. Furthermore, I would like to understand how the values of the activations evolve from the input layer to the output layer and not just print the activations of one layer given the input to another layer.
Here's a Github issue that asks a similar thing.
I want to change the shape and the content of the tensor in a keras model. Tensor is the output of a layer and has
shape1=(batch_size, max_sentences_in_doc, max_tokens_in_doc, embedding_size)
and I want to convert to
shape2=(batch_size, max_documents_length, embedding_size)
suitable as input of the next layer. Here sentences are made of tokens, and are zero-padded so every sentence has length=max_tokens_in_sentence.
In detail:
I wanto to concatenate all the sentences of a batch taking only the nonzero part of the sentences;
then I zero-pad this concatenation to a length=max_document_length.
So passing from shape1 to shape2 is not only a reshape as mathematical operations are involved.
I created the function embedding_to_docs(x) that iterates on the tensor of shape1 to transform it into shape2. I call the function using a Lambda layer in the model, it works in debug with fictious data, but when I try to call it during the build of the model an error is raised:
Tensor objects are only iterable when eager execution is enabled. To iterate over this tensor use tf.map_fn.
def embedding_to_docs(x):
new_output = []
for doc in x:
document = []
for sentence in doc:
non_zero_indexes = np.nonzero(sentence[:, 0])
max_index = max(non_zero_indexes[0])
if max_index > 0:
document.extend(sentence[0:max_index])
if MAX_DOCUMENT_LENGTH-len(document) > 0:
a = np.zeros((MAX_DOCUMENT_LENGTH-len(document), 1024))
document.extend(a)
else:
document = document[0:MAX_DOCUMENT_LENGTH]
new_output.append(document)
return np.asarray(new_output)
...
# in the model:
tensor_of_shape2 = Lambda(embedding_to_docs)(tensor_of_shape1)
How to fix this?
You can use py_function, which allows you to switch from the graph mode (used by Keras) to the eager mode (where it is possible to iterate over tensors like in your function).
def to_docs(x):
return tf.py_function(embedding_to_docs, [x], tf.float32)
tensor_of_shape2 = Lambda(to_docs)(tensor_of_shape1)
Note that the code run within your embedding_to_docs must be written in tensorflow eager instead of numpy. This means that you'd need to replace some of the numpy calls with tensorflow. You'd surely need to replace the return line with:
return tf.convert_to_tensor(new_output)
Using numpy arrays will stop the gradient computation, but you are likely not interested in gradient flowing through the input data anyway.
Following the upgrade to Keras 2.0.9, I have been using the multi_gpu_model utility but I can't save my models or best weights using
model.save('path')
The error I get is
TypeError: can’t pickle module objects
I suspect there is some problem gaining access to the model object. Is there a work around this issue?
To be honest, the easiest approach to this is to actually examine the multi gpu parallel model using
parallel_model.summary()
(The parallel model is simply the model after applying the multi_gpu function). This clearly highlights the actual model (in I think the penultimate layer - I am not at my computer right now). Then you can use the name of this layer to save the model.
model = parallel_model.get_layer('sequential_1)
Often its called sequential_1 but if you are using a published architecture, it may be 'googlenet' or 'alexnet'. You will see the name of the layer from the summary.
Then its simple to just save
model.save()
Maxims approach works, but its overkill I think.
Rem: you will need to compile both the model, and the parallel model.
Workaround
Here's a patched version that doesn't fail while saving:
from keras.layers import Lambda, concatenate
from keras import Model
import tensorflow as tf
def multi_gpu_model(model, gpus):
if isinstance(gpus, (list, tuple)):
num_gpus = len(gpus)
target_gpu_ids = gpus
else:
num_gpus = gpus
target_gpu_ids = range(num_gpus)
def get_slice(data, i, parts):
shape = tf.shape(data)
batch_size = shape[:1]
input_shape = shape[1:]
step = batch_size // parts
if i == num_gpus - 1:
size = batch_size - step * i
else:
size = step
size = tf.concat([size, input_shape], axis=0)
stride = tf.concat([step, input_shape * 0], axis=0)
start = stride * i
return tf.slice(data, start, size)
all_outputs = []
for i in range(len(model.outputs)):
all_outputs.append([])
# Place a copy of the model on each GPU,
# each getting a slice of the inputs.
for i, gpu_id in enumerate(target_gpu_ids):
with tf.device('/gpu:%d' % gpu_id):
with tf.name_scope('replica_%d' % gpu_id):
inputs = []
# Retrieve a slice of the input.
for x in model.inputs:
input_shape = tuple(x.get_shape().as_list())[1:]
slice_i = Lambda(get_slice,
output_shape=input_shape,
arguments={'i': i,
'parts': num_gpus})(x)
inputs.append(slice_i)
# Apply model on slice
# (creating a model replica on the target device).
outputs = model(inputs)
if not isinstance(outputs, list):
outputs = [outputs]
# Save the outputs for merging back together later.
for o in range(len(outputs)):
all_outputs[o].append(outputs[o])
# Merge outputs on CPU.
with tf.device('/cpu:0'):
merged = []
for name, outputs in zip(model.output_names, all_outputs):
merged.append(concatenate(outputs,
axis=0, name=name))
return Model(model.inputs, merged)
You can use this multi_gpu_model function, until the bug is fixed in keras. Also, when loading the model, it's important to provide the tensorflow module object:
model = load_model('multi_gpu_model.h5', {'tf': tf})
How it works
The problem is with import tensorflow line in the middle of multi_gpu_model:
def multi_gpu_model(model, gpus):
...
import tensorflow as tf
...
This creates a closure for the get_slice lambda function, which includes the number of gpus (that's ok) and tensorflow module (not ok). Model save tries to serialize all layers, including the ones that call get_slice and fails exactly because tf is in the closure.
The solution is to move import out of multi_gpu_model, so that tf becomes a global object, though still needed for get_slice to work. This fixes the problem of saving, but in loading one has to provide tf explicitly.
It's something that need a little work around by loading the multi_gpu_model weight to the regular model weight.
e.g.
#1, instantiate your base model on a cpu
with tf.device("/cpu:0"):
model = create_model()
#2, put your model to multiple gpus, say 2
multi_model = multi_gpu_model(model, 2)
#3, compile both models
model.compile(loss=your_loss, optimizer=your_optimizer(lr))
multi_model.compile(loss=your_loss, optimizer=your_optimizer(lr))
#4, train the multi gpu model
# multi_model.fit() or multi_model.fit_generator()
#5, save weights
model.set_weights(multi_model.get_weights())
model.save(filepath=filepath)
`
refrence: https://github.com/fchollet/keras/issues/8123
I have a placeholder variable that expects a batch of input images:
input_placeholder = tf.placeholder(tf.float32, [None] + image_shape, name='input_images')
Now I have 2 sources for the input data:
1) a tensor and
2) some numpy data.
For the numpy input data, I know how to feed data to the placeholder variable:
sess = tf.Session()
mLoss, = sess.run([loss], feed_dict = {input_placeholder: myNumpyData})
How can I feed a tensor to that placeholder variable?
mLoss, = sess.run([loss], feed_dict = {input_placeholder: myInputTensor})
gives me an error:
TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.
I don't want to convert the tensor into a numpy array using .eval(), since that would slow my program down, is there any other way?
This has been discussed on GitHub in 2016, and please check here. Here is the key point by concretevitamin:
One key thing to note is that Tensor is simply a symbolic object. The values of your feed_dict are the actual values, e.g. a Numpy ndarry.
The tensor as a symbolic object is flowing in the graph while the actual values are outside of it, then we can only pass the actual values into the graph and the symbolic object can not exist outside the graph.
You can use feed_dict to feed data into non-placeholders. So, first, wire up your dataflow graph directly to your myInputTensor tensor data source (i.e. don't use a placeholder). Then when you want to run with your numpy data you can effectively mask myImportTensor with myNumpyData, like this:
mLoss, = sess.run([loss], feed_dict={myImportTensor: myNumpyData})
[I'm still trying to figure out how to do this with multiple tensor data sources however.]
One way of solving the problem is to actually remove the Placeholder tensor and replace it by your "myInputTensor".
You will use the myInputTensor as the source for the other operations in the graph and when you want to infer the graph with your np array as input data, you will feed a value to this tensor directly.
Here is a quick example:
import tensorflow as tf
import numpy as np
# Input Tensor
myInputTensor = tf.ones(dtype=tf.float32, shape=1) # In your case, this would be the results of some ops
output = myInputTensor * 5.0
with tf.Session() as sess:
print(sess.run(output)) # == 5.0, using the Tensor value
myNumpyData = np.zeros(1)
print(sess.run(output, {myInputTensor: myNumpyData}) # == 0.0 * 5.0 = 0.0, using the np value
This works for me in latest version...maybe you have older version of TF?
a = tf.Variable(1)
sess.run(2*a, feed_dict={a:5}) # prints 10
I recently have started using Keras to build neural networks. I built a simple CNN to classify MNIST dataset. Before learning the model I used K.set_image_dim_ordering('th') in order to plot a convolutional layer weights. Right now I am trying to visualize convolutional layer output with K.function method, but I keep getting error.
Here is what I want to do for now:
input_image = X_train[2:3,:,:,:]
output_layer = model.layers[1].output
input_layer = model.layers[0].input
output_fn = K.function(input_layer, output_layer)
output_image = output_fn.predict(input_image)
print(output_image.shape)
output_image = np.rollaxis(np.rollaxis(output_image, 3, 1), 3, 1)
print(output_image.shape)
fig = plt.figure()
for i in range(32):
ax = fig.add_subplot(4,8,i+1)
im = ax.imshow(output_image[0,:,:,i], cmap="Greys")
plt.xticks(np.array([]))
plt.yticks(np.array([]))
fig.subplots_adjust(right=0.8)
cbar_ax = fig.add_axes([1, 0.1, 0.05 ,0.8])
fig.colorbar(im, cax = cbar_ax)
plt.tight_layout()
plt.show()
And this is what I get:
File "/home/kinshiryuu/anaconda3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1621, in function
return Function(inputs, outputs, updates=updates)
File "/home/kinshiryuu/anaconda3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1569, in __init__
raise TypeError('`inputs` to a TensorFlow backend function '
TypeError: `inputs` to a TensorFlow backend function should be a list or tuple.
You should do the following changes:
output_fn = K.function([input_layer], [output_layer])
output_image = output_fn([input_image])
K.function takes the input and output tensors as list so that you can create a function from many input to many output. In your case one input to one output.. but you need to pass them as a list none the less.
Next K.function returns a tensor function and not a model object where you can use predict(). The correct way of using is just to call as a function
I think you can also use K.function to get gradients.
self.action_gradients = K.gradients(Q_values, actions)
self.get_action_gradients=K.function[*self.model.input, K.learning_phase()], outputs=action_gradients)
which basically runs the graph to obtain the Q-value to calculate the gradient of the Q-value w.r.t. action vector in DDPG. Source code here (lines 64 to 70): https://github.com/nyck33/autonomous_quadcopter/blob/master/criticSolution.py#L65
In light of the accepted answer and this usage here (originally from project 5 autonomous quadcopter in the Udacity Deep Learning nanodegree), a question remains in my mind, ie. is K.function() something that can be used fairly flexibly to run the graph and to designate as outputs of K.function() for example outputs of a particular layer, gradients or even weights themselves?
Lines 64 to 67 here: https://github.com/nyck33/autonomous_quadcopter/blob/master/actorSolution.py
It is being used as a custom training function for the actor network in DDPG:
#caller
self.actor_local.train_fn([states, action_gradients, 1])
#called
self.train_fn = K.function(inputs=[self.model.input, action_gradients, K.learning_phase()], \
outputs=[], updates=updates_op)
outputs is given a value of an empty list because we merely want to train the actor network with the action_gradients from the critic network.