In TensorFlow.js, I have a very simple tf.Sequential model created like this:
let model = tf.sequential();
model.add(tf.layers.dense({inputShape: [784], units: 128, activation: 'relu'}));
model.add(tf.layers.dense({units: 10}));
model.add(tf.layers.softmax());
During prediction time, how can I get the activation of the second tf.layers.dense layer?
Can I just delete model.layers[2] and use model.predict() as normal?
(I know I can do this in advance by defining two model outputs with the functional API, but let's say I have a pre-made tf.Sequential model that I want to inspect the logits of.)
For more complex models, there's an easier way. If model is the original model, you can create a copy using tf.model({inputs:model.inputs, outputs: model.layers[2].output}), thereby only needing to provide the first and last layer
I figured out how to do this.
Deleting model.layers[2] doesn't work, since apparently model.predict() doesn't depend on that property.
One way to do this is to create a duplicate tf.Sequential model, copying over all the layers (except the last) from the original.
let m2 = tf.sequential();
m2.add(model.layers[0]);
m2.add(model.layers[1]);
Then m2.predict() will output the logits.
Related
I am having a lot of trouble understanding this basic idea of the bottleneck block of the mobilenetv2 architecture. For stride = 1, where you add the input layer to the last layer (AKA a skip connection), these are virtually never the same size! How are you supposed to add to tensors that are almost never the same size.
I've tried to think about it in many ways and I can't see what is being done for stride=1.
you can add the layer using model.add
import tensorflow as tf
base_model = tf.keras.applications.MobileNetV2(input_shape=(96,96,3),
include_top=False,
weights='imagenet')
model = tf.keras.Sequential()
model.add(base_model)
model.add(tf.keras.Input(shape=( )))
Thank You.
I am implementing a model in Tensorflow 2, and I want to apply a penalization on a tensor (multiplication from two layers' outputs) in my model.
I am used to use regularization on layers (kernel, bias or activity regularization).
I could build a custom layer that only has an activity regularization, but I am hopping that there is a simpler solution to add regularization to a tensor.
I saw this code in Tensorflow:
regularizer = tf.keras.regularizers.L2(2.)
tensor = tf.ones(shape=(5, 5))
regularizer(tensor)
Which outputs:
<tf.Tensor: shape=(), dtype=float32, numpy=50.0>
But does this only compute the regularization value or it also add it to my model's loss?
Or would adding self.add_loss(tf.keras.regularizers.L2(2.)(tensor)) in my call function work?
How would you add a penalty on a tensor?
It is my first question on stackoverflow so sorry if I didn't ask in the good place.
No, the regularizer classes pretty much only handle the computation of the penalty itself. E.g. the source code for the L2 regularizer has
def __call__(self, x):
return self.l2 * tf.reduce_sum(tf.square(x))
and there is also nothing in __init__ that points at any "housekeeping" work. However, activity_regularizer is actually an argument for the base Layer class so any subclassed layer should be able to handle it by default, meaning it should be easy to write a custom layer (you essentially only have to write the call method which sounds like a one-liner in your case).
You may even be able to use a Lambda layer to avoid having to write any sub-classing code, since these also inherit from Layer. However, the docs mention that there are some issues with Lambda regarding saving and loading models...
I'm dealing with Keras functional API.
Specifically for my experiments, I'm using Keras resnet50 model obtained with:
model = resnet50.ResNet50(weights='imagenet')
Obviously, to get the final output of the network we need to feed a value to the placeholder input_1.
My question is, can I somehow start inferencing this graph from the relu layer which is depicted at the bottom of the picture below, provided that I feed a value of the appropriate dimensions into it?
I tried to achieve this with Keras functions. Something like:
self.inp = model.input
self.outputs = [layer.output for layer in model.layers]
self.functor = K.function([self.inp, K.learning_phase()], [self.outputs[6], self.outputs[17]])
But this approach will not work, because again to inference any output I need to feed value into tensor.
Is recreating graph from scratch my best option here?
Thanks
If I got you right, you can just specify input and output nodes
base_model = tf.keras.applications.ResNet50(weights='imagenet')
inference_model = tf.keras.Model(inputs=base_model.input, outputs=base_model.get_layer('any_layer_name').output)
You can set the output to any layer name
Let's say I have a model with one input and two outputs. And I want the output of the third layer of my model to be the y_true in my cost function for my second output.
I've tried this:
model.fit(x, [y, model.layers[3].output], ...)
But got the error:
'Tensor' object has no attribute 'ndim'
Which I believe is referring to the second y_true I gave the fit method.
Is it possible to do something like this in Keras? If so, how?
I managed to this by changing only the cost function, like:
def custom_euclidean_distance_loss(layer_output):
from keras import backend as K
def wrap(y_true, y_pred):
return K.mean(K.square(y_pred - layer_output))
return wrap
And since I do not use any previously known y_true I just fed a dummy one to fit. Note that the printed metrics from Keras won't be correct this way but the model will train with no problem.
If you do know of a better way (like actually feeding the layer output to fit) please let me know
So, I trained a tensorflow model with a few layers, more or less like this:
with tf.variable_scope('model1') as scope:
inputs = tf.placeholder(tf.int32, [None, num_time_steps])
embeddings = tf.get_variable('embeddings', (vocab_size, embedding_size))
lstm = tf.nn.rnn_cell.LSTMCell(lstm_units)
embedded = tf.nn.embedding_lookup(embeddings, inputs)
_, state = tf.nn.dynamic_rnn(lstm, embedded, dtype=tf.float32, scope=scope)
# more stuff on the state
Now, I wanted to reuse the embedding matrix and the lstm weights in another model, which is very different from this one except for these two components.
As far as I know, if I load them with a tf.Saver object, it will look for
variables with the exact same names, but I'm using different variable_scopes in the two graphs.
In this answer, it is suggested to create the graph where the LSTM is trained as a superset of the other one, but I don't think it is possible in my case, given the differences in the two models. Anyway, I don't think it is a good idea to make one graph dependent on the other, if they do independent things.
I thought about changing the variable scope of the LSTM weights and embeddings in the serialized graph. I mean, where it originally read model1/Weights:0 or something, it would be another_scope/Weights:0. Is it possible and feasible?
Of course, if there is a better solution, it is also welcome.
I found out that the Saver can be initialized with a dictionary mapping variable names (without the trailing :0) in the serialized file to the variable objects I want to restore in the graph. For example:
varmap = {'model1/some_scope/weights': variable_in_model2,
'model1/another_scope/weights': another_variable_in_model2}
saver = tf.train.Saver(varmap)
saver.restore(sess, path_to_saved_file)