I know that it is possible to get the current learning rate simply by doing self.optimizer.lr when you are in your custom model, but I need to do something similar when implementing my own layer.
For now I have solved the issue by creating a function in my custom layer that accepts it as a parameter and is called by my custom model, but I was wondering if there is another way since there are many layers in my architecture and this way is pretty awful to see. I leave some code to be clearer.
For my purpose, it would be enough even just to get the current epoch or current step from inside the layer.
I am working in tensorflow 2.8.2. Thank you.
My layer structure is as follows
class my_layer(tf.keras.layers.Layer):
#constructor, build, call methods etc..
def function_to_get_lr(self,lr):
#do sth with the lr
and in my custom model I do something like this
class my_model(tf.keras.Model):
#other functions, constructor etc..
def call(self, inputs, training=False):
if training:
for layer in self.layers:
if "my_layer" in layer.name:
layer.function_to_get_lr(self.optimizer.lr)
I'm pretty sure that there is no "nice way" to do this, but you can do something like this:
def CustomLayer(Layer):
def __init__(optimizer, ...):
self.optimizer = optimizer
def call(...):
lr = self.optimizer.lr
class M(Model):
def build(...):
this.custom_layer = CustomLayer(self.optimizer, ...)
...
Related
I'm currently asking myself how to build a model with a couple of extra functions.
I got an entity of custom functions, and I want to embed them as layers into my model (NN).
For that I'm using TF 2.0. but I'm currently struggling to do that.
All I find is answers about activation functions, but that's not what I'm looking for.
A custom function returns something like a+b or any other algorithm (matrix multiplication etc.)
What we can say is, I have one layer to another one, and want to embed my custom function in between those two layers like so:
I'm going to say that the activation function from one layer to another is the custom function. But what if my custom function takes two inputs? Or I have two functions I want to process my input in before I pass it to the next function?
Another way to solve that problem:
Let's say I got my custom functions cm*, and my layers l*;
what I do is build a model for each layer I want to put in between two custom functions
cm1 -> model(l1) -> cm2 -> model(l2,l3) -> cm3 -> cm4 -> model(l4) -> ....
but wouldn't it be stupid to build a model for each of those trajectories?
And what about the loss? The back propagation of residual connected layers is something else than having a lot of models and functions layered together.
Or am I wrong?
I'm not sure about TF 2.0, but in Keras you can build your own custom layers that can receive multiple inputs by overriding the Layer class. See https://keras.io/guides/making_new_layers_and_models_via_subclassing/ for more details. The link doesn't explain how to pass in multiple inputs to a layer, but all you have to do is to call the layer with a list of inputs and unpack them inside the call function, something like this:
class MyCustomLayer(tf.keras.Layer):
def __init__(self):
# your code here
pass
def call(self, inputs): # example call: MyCustomLayer()([1, 2])
x, y = inputs
# your code here
output = x + y # placeholder
return output
i am search a solution to reuse pre-trained Resnet50 model as a customized sub-layers inside a big customized model.
In my scenario, I use Resnet50 as FCN backbone, and then I will use FCN output result as input to another network layers.(its a pipeline)
you can see my code:
https://github.com/piginzoo/textscanner/blob/master/network/layers/fcn_layer.py
https://github.com/piginzoo/textscanner/blob/master/network/model.py
my question is, i cannot find an example code, to integrate the Resnet50 layers as partial of customized layer in TF 2.0, i search out stackoverflow, TF2.0 offical website and blogs, cannot find a feasible code snippet to demo how to implement such requirement.
but I believe this is a very common paradigm, does anyone ever faces on such problems, and any advices or code demo will be very appreciated.
Finally i found a feasible way, like:
as the code shown, I created a new model who use pre-trained Resnet50's input as his input, and use the 3 "conv_block_out"s as new model's outputs, and in call method, i just call the model as "a layer" style.
for detail, you can refer to my Github for completed code.
Dummy code:
class FCNLayer(Layer):
def __init__(self, name, resnet50_model):
super().__init__(name=name)
resnet50_model.layers.pop()
resnet50_model.summary()
self.resnet50_model = resnet50_model
def build(self, input_image, FILTER_NUM=4):
layer_names = [
"conv3_block4_out", # 1/8
"conv4_block6_out", # 1/16
"conv5_block3_out", # 1/32
]
layers = [self.resnet50_model.get_layer(name).output for name in layer_names]
self.FCN_left = Model(inputs=self.resnet50_model.input, outputs=layers)
......
def call(self, input_image, training=True):
pool3, pool4, pool5 = self.FCN_left(input_image)
.......
The goal is to create a block of layers, using Keras' functional API, which is usable (also syntax-wise) like a 'normal' Keras layer.
Here is a toy example
from tensorflow.keras import layers as kl
def layer_block(prev_layer, args):
# some code using 'args'
layer = kl.Dense(units=prev_layer.shape[1])(prev_layer)
layer = kl.Dense(units=5)(layer)
layer = kl.Dense(units=prev_layer.shape[1])(layer)
return layer
This block is called using layer_block(prev_layer, args) which is in contradiction to Keras' functional API's syntax. It should rather look like layer_block(args)(prev_layer).
The approach so far is to wrap this block by another block:
def outer_block(args):
def layer_block(prev_layer, args):
# some code using 'args'
layer = kl.Dense(units=prev_layer.shape[1])(prev_layer)
layer = kl.Dense(units=5)(layer)
layer = kl.Dense(units=prev_layer.shape[1])(layer)
return layer
return lambda prev_layer: layer_block(prev_layer, args)
Now two questions arise:
Is there an easier way to achieve this?
Is it effective this way or does it have negative impact on performance?
Thank you in advance!
What you're doing doesn't affect performance, you're creating layers perfectly fine.
There is no problem in any of your two approaches, but if you do want to make it work as an actual layer, transform it into a model.
This may not work in every keras version:
class LayerBlock(tensorflow.keras.Model): #not sure if it works in normal keras (without tf)
def __init__(self):
super(LayerBlock, self).__init__(outer_units)
self.layer1 = kl.Dense(units=outer_units)
self.layer2 = kl.Dense(units=5)
self.layer3 = kl.Dense(units=outer_units)
def call(self, inputs):
x = self.layer1(inputs)
x = self.layer2(x)
x = self.layer3(x)
return x
This tutorial seems to suggest that you can use tf.keras.Layer instead of tf.keras.Model, but that sounds strange to me. It may work with eager mode on, but it lacks the build method with a self.built=True statement.
I've created a keras subclass model just like this:
class SubModel(tf.Keras.Model):
def __init__(self, features, **kwargs):
"""Init function of Model.
Args:
features: A list of SparseFeature and DenseFeature.
"""
assert len(features) > 0
super(SubModel, self).__init__(name='SubModel', **kwargs)
self.features = features
Note that there is a features arrtibute in __init__ function which will be used in call method of this model. Everything works well when I train and evaluate the model with keras style.
But, now I want to convert this model to estimator using tf.keras.model_to_estimator function. It raise an error: AttributeError: '_ListWrapper' object has no attribute 'get_config'.
According to my debug, it's the features attribute which is added to the model cause this error. When convertint to estimator, it regard features as a layer of the model, and try to call the get_config function when cloning the model. It seems that all the attributes added to the model will be treated as layer when cloning the model.
But I really want to use features as a part of model, so that it can be accessed through other function of this model like call. Is there other ways to solve this?
I think tf.keras.model_to_estimator is compatible with Sequential or Functional API Keras model perfectly but poorly with Subclass model especially when implementing complicated operation in subclass.
So, If you have defined a subclass keras model, and want to covert it to estimator, the best way is define model_fn function, and put the keras model in it like code below:
def model_fn(features, labels, mode):
model = SubModel()
outputs = model(features)
loss = tf.keras.losses.xx(labels, outputs)
return tf.estimator.EstimatorSpec(...)
I'm trying to implement my own optimization algorithm for MxNet (Imperative / Gluon) that does not use gradients. My question is pretty simple is there a simple way to create new nn.Dense(...) layer initialized with parameters (i.e. Biases and Weights) represented by two nd.array() instances?
Thank you in advance!
You can create a custom block with parameters that set differentiable=False, and provide the data for initialization through the init argument. See the scales parameter in the example below taken from this tutorial. You can also see an example of FullyConnected which you'll want to use for your dense layer too. F is used to denote a generic backend, typically this would be mx.ndarray, but after hybridization this is set to mx.symbol.
class NormalizationHybridLayer(gluon.HybridBlock):
def __init__(self, hidden_units, scales):
super(NormalizationHybridLayer, self).__init__()
with self.name_scope():
self.weights = self.params.get('weights',
shape=(hidden_units, 0),
allow_deferred_init=True)
self.scales = self.params.get('scales',
shape=scales.shape,
init=mx.init.Constant(scales.asnumpy().tolist()), # Convert to regular list to make this object serializable
differentiable=False)
def hybrid_forward(self, F, x, weights, scales):
normalized_data = F.broadcast_div(F.broadcast_sub(x, F.min(x)), (F.broadcast_sub(F.max(x), F.min(x))))
weighted_data = F.FullyConnected(normalized_data, weights, num_hidden=self.weights.shape[0], no_bias=True)
scaled_data = F.broadcast_mul(scales, weighted_data)
return scaled_data