i recently made a simple custom layer but i have a doubt on how correctly define trainable distributions. Here i had to add .sample() to make it work, but is it the correct way? to me it seems that by using sample() tensorflow substitute a number to the distribution instead of optimizing the variable. what the correct way? if i don't add .sample() i get errors
class MyDenseLayer(tf.keras.layers.Layer):
def __init__(self):
super(MyDenseLayer, self).__init__()
def build(self, input_shape):
self.omegam = tfd.Normal(loc=tf.Variable(0.3, name='omegam'), scale=0.1).sample()
self.omegaz = tfd.Normal(loc=tf.Variable(-1., name='omegaz'), scale=0.1).sample()
self.omegau = tfd.Normal(loc=tf.Variable(0., name='omegau'), scale=0.1).sample()
def call(self, inputs):
inverse=1/tf.math.sqrt((self.omegam*(1+inputs)**3+ (1-self.omegam)*(1+inputs)*tf.math.exp(self.omegau*inputs+(1+self.omegaz -self.omegau))))
return inverse
Related
I am following this tutorial to use custom layers for pre-processing.
def pre_process(file_path):
# loading file from disk and transforming into [90,13,1]
class PreProcessBlock(layers.Layer):
def __init__(self):
super(PreProcessBlock,self).__init__()
def call(self, inputs):
return pre_process(inputs.numpy())
def compute_output_shape(self, input_shape):
return input_shape
preprocess = tf.keras.Sequential([
PreProcessBlock()
])
model = keras.Sequential(
[
preprocess,
layers.Dense(256, activation = "relu"),
layers.Dropout(.5),
layers.Dense(len(LABELS))]
I am creating my dataset as
files = ['file1,'file2`]
labels = [0,1]
def get_data_set(files, labels, is_training=False):
dataset = tf.data.Dataset.from_tensor_slices((files, labels))
if is_training:
dataset = dataset.shuffle(SHUFFLE_BUFFER_SIZE, reshuffle_each_iteration = True)
dataset = dataset.batch(BATCH_SIZE)
dataset = dataset.prefetch(AUTOTUNE)
return dataset
train_dataset = get_data_set(files, labels, is_training=True)
val_dataset = get_data_set(files, labels)
Model fitting fails with error
model.fit(train_dataset, epochs=1, verbose=1,validation_data=val_dataset)
Error
AttributeError: in user code:
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function *
return step_function(self, iterator)
<ipython-input-158-3f6d9dd39f2f>:6 call *
return pre_process(inputs.numpy())
AttributeError: 'Tensor' object has no attribute 'numpy'
My question
Is this a valid way of implementing the model pipeline?
The structure of your layers and organizing all preprocessing layers into one sequential layer is great. You should not load any learning examples in layers (it is not part of model and it makes it less portable).
Two issues:
You don't have numpy() method because of this. I recommend sticking to static graphs and do not try to convert anything to numpy in your keras graph, unless it is absolutely necessary - performance issue. Most operations on tensors can be done using tf.
Your custom preprocessing layer should inherit from tensorflow.keras.layers.experimental.preprocessing.PreprocessingLayer (all layers from tf.keras.layers.experimental.preprocessing inherit from it directly or via CombinerPreprocessingLayer from the same package). There isn't much going on in the source code of PreprocessingLayer class, but all of it is important:
PreprocessingLayer provides interface for adapt method:
adapt(self, data, reset_state=True). Please see "pure" keras docs why and when we need this.
PreprocessingLayer class have the flag _must_restore_from_config = True which from the Layer documentation we read:
When loading from a SavedModel, Layers typically can be revived into a
generic Layer wrapper. Sometimes, however, layers may implement
methods that go beyond this wrapper, as in the case of
PreprocessingLayers' adapt method. When this is the case, layer
implementers can override must_restore_from_config to return
True; layers with this property must be restored into their actual
objects (and will fail if the object is not available to the
restoration code).
Let's take for example Resizing layer code (comment were ommited for readability):
class Resizing(PreprocessingLayer):
def __init__(self,
height,
width,
interpolation='bilinear',
name=None,
**kwargs):
self.target_height = height
self.target_width = width
self.interpolation = interpolation
self._interpolation_method = get_interpolation(interpolation)
self.input_spec = InputSpec(ndim=4)
super(Resizing, self).__init__(name=name, **kwargs)
base_preprocessing_layer._kpl_gauge.get_cell('V2').set('Resizing')
def call(self, inputs):
outputs = image_ops.resize_images_v2(
images=inputs,
size=[self.target_height, self.target_width],
method=self._interpolation_method)
return outputs
def compute_output_shape(self, input_shape):
input_shape = tensor_shape.TensorShape(input_shape).as_list()
return tensor_shape.TensorShape(
[input_shape[0], self.target_height, self.target_width, input_shape[3]])
def get_config(self):
config = {
'height': self.target_height,
'width': self.target_width,
'interpolation': self.interpolation,
}
base_config = super(Resizing, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
This is a pretty generic layer. It can resize images to some width and height. However, the point of preprocessing layers is to have entire end-to-end pipeline saved in one model. So in your pipline you would have specific width and height and you do not want to be bothered with instantiating the layer with proper arguments when doing inference- it should be the same as in training (applies to any preprocessing method, really). So in the get_config() method, apart from basic config , both height and width is saved and it can be easily read when resoring model later on. Please note that this layer does not override adapt method as it is invariant to data.
I got a problem with my custom layer
class L2Layer(tf.keras.layers.Layer):
def __init__(self, **kwargs):
super(L2Layer, self).__init__(**kwargs)
def build(self, input_shape):
# Create a trainable weight variable for this layer.
self._x = self.add_weight(name='alpha_l2',
shape=(1,),
initializer='ones',
trainable=True)
super(L2Layer, self).build(input_shape) # Be sure to call this at the end
def call(self, x):
return self._x * tf.divide(x, tf.norm(x, ord='euclidean'))
def compute_output_shape(self, input_shape):
return input_shape[0]
which is saved in a different .py file, for instance: models.py.
When I am trying to load the model via
loaded_model = tf.keras.models.load_model('outputs/test.hdf5', custom_objects={'L2Layer': L2Layer})
I got the error:
NameError: name 'L2Layer' is not defined
Questions
How can I fix this problem?
You should define a get_config in class L2Layer .
Read this for more detail .
From tensorflow keras example. I can create a custom layer which contains Linear layer recursively
class MLPBlock(layers.Layer):
def __init__(self):
super(MLPBlock, self).__init__()
self.linear_1 = Linear(32)
self.linear_2 = Linear(32)
self.linear_3 = Linear(1)
def call(self, inputs):
x = self.linear_1(inputs)
x = tf.nn.relu(x)
x = self.linear_2(x)
x = tf.nn.relu(x)
return self.linear_3(x)
how do i access all the component layers of a custom layer, I want to access weight and biases of all the component layers.
ex:
MLPBlock(Parent Layer):
linear_1
linear_2
linear_3
I have looked into tensorflow keras api version r 1.14 https://www.tensorflow.org/guide/keras
but could not find any way to do this.
I assume that you are following this tutorial. Based on that, here is how you can access the weights:
class MLPBlock(tf.keras.Model):
def __init__(self):
super(MLPBlock, self).__init__()
self.linear_1 = tf.keras.layers.Dense(32)
self.linear_2 = tf.keras.layers.Dense(32)
self.linear_3 = tf.keras.layers.Dense(1)
def call(self, inputs):
x = self.linear_1(inputs)
x = tf.nn.relu(x)
x = self.linear_2(x)
x = tf.nn.relu(x)
return self.linear_3(x)
mlp_block = MLPBlock()
y = mlp_block(tf.ones(shape=(3, 64)))
for layer in mlp_block.layers:
weights, biases = layer.get_weights()
Please note that I slightly modified the example so that you can access the layer's weights and biases. Namely, what I did is instead of subclassing the class with tf.keras.layers.Layer, I subclassed with tf.keras.Model so that the stack of layers can be treated as a model, and then you can access the layers of that model. Then, instead of using the custom Linear layer, I used the tf.keras.layers.Dense for simplicity, however, using the custom layer should not make a difference.
I was trying to implement the Google Bert model in tensorflow-keras using tensorflow hub. For this I designed a custom keras layer "Bertlayer" . Now the problem is when I am compiling the keras model it keeps showing that
AttributeError: 'Bertlayer' object has no attribute '_keras_style'
I don't know where I am wrong and what _keras_style attribute is.Please help to find the error in the code.
This is the github link to the full code: https://github.com/PradyumnaGupta/BERT/blob/master/Untitled21.ipynb
class BertLayer(tf.layers.Layer):
def __init__(self, n_fine_tune_layers=10, **kwargs):
self.n_fine_tune_layers = n_fine_tune_layers
self.trainable = True
self.output_size = 768
super(BertLayer, self).__init__(**kwargs)
def build(self, input_shape):
self.bert = hub.Module(
bert_path,
trainable=self.trainable,
name="{}_module".format(self.name)
)
trainable_vars = self.bert.variables
# Remove unused layers
trainable_vars = [var for var in trainable_vars if not "/cls/" in var.name]
# Select how many layers to fine tune
trainable_vars = trainable_vars[-self.n_fine_tune_layers :]
# Add to trainable weights
for var in trainable_vars:
self._trainable_weights.append(var)
for var in self.bert.variables:
if var not in self._trainable_weights:
self._non_trainable_weights.append(var)
super(BertLayer, self).build(input_shape)
def call(self, inputs):
inputs = [K.cast(x, dtype="int32") for x in inputs]
input_ids, input_mask, segment_ids = inputs
bert_inputs = dict(
input_ids=input_ids, input_mask=input_mask, segment_ids=segment_ids
)
result = self.bert(inputs=bert_inputs, signature="tokens", as_dict=True)[
"pooled_output"
]
return result
def compute_output_shape(self, input_shape):
return (input_shape[0], self.output_size)
So, tensorflow version 1.* is a bit misleading. It actually has 2 bases classes called Layer. One - the one that you are using. It is intended to implement shortcut wrappers over regular TF operations. The other from tensorflow.keras.layers import Layer is for Keras-like models and sequencies.
Judging by your error, you are using keras/models to train further.
You probably should start form derivering your layer from keras.layers.Layer instead of tf.layers.Layer.
I'm building a NN that supports complex numbers. Currently working on complex activation. According to a Benjio paper, this is a good one:
Where b is a trainable parameter to be learnt. So I'm building a special layer to do this activation. I'm new to Keras and stuck already. I created this code below, but it gives an error with the build function. I have no idea what's happening, I just tried to copy the template. Please help.
class modrelu(Layer):
def __init__(self, **kwargs):
super(modrelu, self).__init__(**kwargs)
def build(self):
self.b= K.variable(value=np.random.rand()-0.5, dtype='float64')
super(modrelu, self).build() # Be sure to call this at the end
def call(self, x):
assert isinstance(x, list)
ip_r, ip_i = x
comp= tf.complex(ip_r, ip_i )
ABS= tf.math.abs(comp)
ANG= tf.math.angle(comp)
ABS= K.relu( self.b + ABS)
op_r= ABS * K.sin(angle) #K.dot ??
op_i= ABS * K.cos(angle)
return [op_r, op_i]
def compute_output_shape(self, input_shape):
assert isinstance(input_shape, list)
shape_a, shape_b = input_shape
return [shape_a, shape_b]
Comments on my code:
In the init I didn't add anything, cause it is an activation layer that takes no input when instantiated.
In the build method, I tried to add the b's. Not sure if I should use the self.add_weight method. Ideally, I want to have as many b's as the dimension of input.
In the call method, this one, I'm pretty sure what I'm doing. It is easy, I just implemented the function.
The last one, compute_output_shape, I just copied-pasted the template. The output should be the same as the input, cause it is just an activation layer.
Finally, the error for what its worth, I know it is nonsense
TypeError Traceback (most recent call last)
<ipython-input-5-3101a9226da5> in <module>
1 a=K.variable(np.array([1,2]))
2 b=K.variable(np.array([3,4]))
----> 3 act([a,b])
~\AppData\Local\conda\conda\envs\python36\lib\site-packages\keras\engine\base_layer.py in __call__(self, inputs, **kwargs)
429 'You can build it manually via: '
430 '`layer.build(batch_input_shape)`')
--> 431 self.build(unpack_singleton(input_shapes))
432 self.built = True
433
TypeError: build() takes 1 positional argument but 2 were given
There are several issues with your code.
First of all I should address the error you get from interpreter:
TypeError: build() takes 1 positional argument but 2 were given
The build method should take input_shape argument. Therefore you should declare build method as build(self, input_shape)
The second issue is undefined shape of the variables in the build method. You should explicitly declare shape of the variables. In your case the np.random.rand array should be of input_shape shape.
Another issue is that you are trying to return 2 results ([op_r, op_i]) in the call method. I'm not specialist in Keras but as far as I know you can't do it. Every Keras layer should have one and only one output. See here for the details: https://github.com/keras-team/keras/issues/3061
However if you use tensorflow backend you may use complex numbers (tf.complex) to return both real (op_r) and imagenary (op_i) parts of the complex number.
Here is the working implementation of modrelu layer with simple usage example. It is writtern for TensorFlow 1.12.0 which is distributed with it's own implementation of Keras API but I think you can easily adopt it for original Keras:
import tensorflow as tf
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.engine import Layer
import numpy as np
class modrelu(Layer):
def __init__(self, **kwargs):
super(modrelu, self).__init__(**kwargs)
# provide input_shape argument in the build method
def build(self, input_shape):
# You should pass shape for your variable
self.b= K.variable(value=np.random.rand(*input_shape)-0.5,
dtype='float32')
super(modrelu, self).build(input_shape) # Be sure to call this at the end
def call(self, inputs, **kwargs):
assert inputs.dtype == tf.complex64
ip_r = tf.math.real(inputs)
ip_i = tf.math.imag(inputs)
comp = tf.complex(ip_r, ip_i )
ABS = tf.math.abs(comp)
ANG = tf.math.angle(comp)
ABS = K.relu(self.b + ABS)
op_r = ABS * K.sin(ANG) #K.dot ??
op_i = ABS * K.cos(ANG)
# return single tensor in the call method
return tf.complex(op_r, op_i)
real = tf.constant([2.25, 3.25])
imag = tf.constant([4.75, 5.75])
x = tf.complex(real, imag)
y = modrelu()(x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run(y))
P.S.: I didn't check the math so you should check it by yourself.
You are not coding the layer correctly, the build function takes a input_shape parameter, which you can use to initialize the weights/parameters of your layer.
You can see an example in Keras' source code.