From what I read, tf.keras.initializers.ones() is one of the initializers that initializes weights.
The 3 lines of code below generate the same tensors:
>>> a = tf.keras.initializers.ones()((1, 2))
>>> b = tf.ones((1, 2))
>>> c = tf.constant([[1., 1.]])
>>> a
<tf.Tensor: id=27, shape=(1, 2), dtype=float32, numpy=array([[1., 1.]], dtype=float32)>
>>> b
<tf.Tensor: id=30, shape=(1, 2), dtype=float32, numpy=array([[1., 1.]], dtype=float32)>
>>> c
<tf.Tensor: id=35, shape=(1, 2), dtype=float32, numpy=array([[1., 1.]], dtype=float32)>
Looks like I can use tf.ones((1, 2)) to get the same weight values. But I should still use the ones initializer to initialize the weights as good practice.
Want to make sure I didn't miss anything important here with the initializers.
Here is the source of tf.keras.initializers.ones. It's mainly a call to array_ops.ones, which is the same as tf.ones. It also checks that the type is numeric or boolean.
Even though they seem equivalent, I would suggest using tf.keras.initializers.ones because it makes it that much clearer that you are using those values to initialize something.
Related
Consider the following tensor...
tot_fam=rand(1230)
TK=tf.math.floor(tf.size(tot_fam)/52)
The output of this snippet is
<tf.Tensor: shape=(), dtype=float64, numpy=23.0>
How can I extract the length, i.e., 23.0 ?
Let's say I have a list of coefficients [1,2,3]. How do I convert this to x^3 + 2x^2 + 3 or something similar in NumPy? Is it even possible?
As described in the docs, which I recommend reading:
>>> p1 = np.polynomial.Polynomial([3, 2, 1])
>>> p1
Polynomial([3., 2., 1.], domain=[-1, 1], window=[-1, 1])
>>> p1(0)
3.0
Note that the order of coefficients is reversed.
Basically this:
Is it possible to enforce mathematical constraints between tensorflow neural network output nodes in the last layer?
For example, monotonicity between nodes, such as output node 1 being larger than node 2, which in turn is larger than node 3, and so forth.
In general -- not really, not directly, at least. Keras layers support arguments for constraints on the weights, and you may be able to translate a desired output constraint into a weight constraint instead -- but otherwise, you will need to think about how to set up the structure of your network such that the constraints are fulfilled.
Here is a sketch for how a monotonicity constraint might be possible. Actually including this in a model likely requires creating a custom Layer subclass or perhaps using the functional API.
First, let's create some dummy data. This could be the output of a standard Dense layer (4 is batch size, 5 the number of outputs).
raw_outputs = tf.random.normal([4, 5])
>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[ 0.3989258 , -1.7693167 , 0.13419539, 1.1059834 , 0.3271042 ],
[ 0.6493515 , -1.4397207 , 0.05153034, -0.2730962 , -1.1569825 ],
[-1.3043666 , 0.20206456, -0.3841469 , 1.8338723 , 1.2728293 ],
[-0.3725195 , 1.1708363 , -0.01634515, -0.01382025, 1.2707714 ]],
dtype=float32)>
Next, make all outputs be positive using softplus. Think of this as the output activation function. Any function that returns values >= 0 will do. For example, you could use tf.exp but the exponential growth might lead to numerical issues. I would not recommend relu since the hard 0s prevent gradients from flowing -- usually a bad idea in the output layer.
positive_outputs = tf.nn.softplus(raw_outputs)
>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[0.9123723 , 0.15738781, 0.7624942 , 1.3918277 , 0.8700147 ],
[1.0696293 , 0.21268418, 0.71924424, 0.56589293, 0.2734058 ],
[0.24007489, 0.7992745 , 0.5194075 , 1.9821143 , 1.5197192 ],
[0.5241344 , 1.4409455 , 0.685008 , 0.68626094, 1.5181118 ]],
dtype=float32)>
Finally, use cumsum to add up the values:
constrained = tf.cumsum(positive_outputs, reverse=True, axis=-1)
>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[4.0940967, 3.1817245, 3.0243368, 2.2618425, 0.8700147],
[2.8408566, 1.7712271, 1.558543 , 0.8392987, 0.2734058],
[5.0605907, 4.8205156, 4.021241 , 3.5018334, 1.5197192],
[4.8544607, 4.3303266, 2.889381 , 2.204373 , 1.5181118]],
dtype=float32)>
As we can see, the outputs for each batch element are monotonically decreasing! This is because each of our original outputs (positive_outputs) basically just encodes how much is added at each unit, and because we forced them to be positive, the numbers can only get larger (or smaller in this case because of reverse=True in cumsum).
There are many ways one is neurons learning and the second is mathematics Fn or both by scores weights or rewards or learning from other neurons.
This is one way I teach the networks to lean, you may study from the online sample.
X = tf.compat.v1.placeholder(tf.float32, shape=( (10, 88, 80, 4)))
y = tf.compat.v1.placeholder(tf.float32, shape=(1, 1))
X_action = tf.compat.v1.get_variable('X_action', dtype = tf.float32, initializer = tf.random.normal((1, 1))) # X_var
in_training_mode = tf.compat.v1.get_variable('in_training_mode', dtype = tf.float32, initializer = tf.random.normal((1, 1))) # X_var
loss = tf.reduce_mean(input_tensor=tf.square((X * y) - (X * X_action))) *
optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
This is another way me for the similar tasks with simple configuration just select the correct optimizer and loss Fn that performs faster learning, the previous sample is just Error roots mean sequare.
optimizer = tf.keras.optimizers.Nadam(
learning_rate=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,
name='Nadam'
)
lossfn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=lossfn, metrics=['accuracy'])current version.
I am trying to create a custom model using TensorFlow. I am using tf.TensorArray() function with dynamic_size=True.
I am facing issue when I want to delete a particular element based on a if condition.
for example,
t_a = tf.TensorArray(dtype=tf.float32,size=0,dynamic_size=True)
t_a.write(0,[0,23])
t_a.write(1,[1,67])
t_a.write(2,[3,0])
t_a.write(3,[4,9])
Output:
t_a.read(2)
<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[3., 0.]], dtype=float32)>
Now, my application would require me to read the value of first element in each row, if true, I need to delete that element from the TensorArray.
To access the individual element I use t_a.gather([indices]).
Any suggestions or work around for this is really appreciated.
You can clear the value by making clear_after_read as True. After reading element, clear that value.
Sample working code
import tensorflow as tf
ta = tf.TensorArray(tf.float32, size=0, dynamic_size=True, clear_after_read=True)
ta = ta.write(0, 10)
ta = ta.write(1, 20)
ta = ta.write(2, 30)
indices = 1
ta.read(indices)
ta.gather([indices])
Output
<tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
I am using the TensorFlow backend.
I am applying a convolution, max-pooling, flatten and a dense layer sequentially. The convolution requires a 3D input (height, width, color_channels_depth).
After the convolution, this becomes (height, width, Number_of_filters).
After applying max-pooling height and width changes. But, after applying the flatten layer, what happens exactly? For example, if the input before flatten is (24, 24, 32), then how it flattens it out?
Is it sequential like (24 * 24) for height, weight for each filter number sequentially, or in some other way? An example would be appreciated with actual values.
The Flatten() operator unrolls the values beginning at the last dimension (at least for Theano, which is "channels first", not "channels last" like TF. I can't run TensorFlow in my environment). This is equivalent to numpy.reshape with 'C' ordering:
‘C’ means to read / write the elements using C-like index order, with
the last axis index changing fastest, back to the first axis index
changing slowest.
Here is a standalone example illustrating Flatten operator with the Keras Functional API. You should be able to easily adapt for your environment.
import numpy as np
from keras.layers import Input, Flatten
from keras.models import Model
inputs = Input(shape=(3,2,4))
# Define a model consisting only of the Flatten operation
prediction = Flatten()(inputs)
model = Model(inputs=inputs, outputs=prediction)
X = np.arange(0,24).reshape(1,3,2,4)
print(X)
#[[[[ 0 1 2 3]
# [ 4 5 6 7]]
#
# [[ 8 9 10 11]
# [12 13 14 15]]
#
# [[16 17 18 19]
# [20 21 22 23]]]]
model.predict(X)
#array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.,
# 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.,
# 22., 23.]], dtype=float32)
Flattening a tensor means to remove all of the dimensions except for one.
A Flatten layer in Keras reshapes the tensor to have a shape that is equal to the number of elements contained in the tensor.
This is the same thing as making a 1d-array of elements.
For example in the VGG16 model you may find it easy to understand:
>>> model.summary()
Layer (type) Output Shape Param #
================================================================
vgg16 (Model) (None, 4, 4, 512) 14714688
________________________________________________________________
flatten_1 (Flatten) (None, 8192) 0
________________________________________________________________
dense_1 (Dense) (None, 256) 2097408
________________________________________________________________
dense_2 (Dense) (None, 1) 257
===============================================================
Note how flatten_1 layer shape is (None, 8192), where 8192 is actually 4*4*512.
PS, None means any dimension (or dynamic dimension), but you can typically read it as 1. You can find more details in here.
It is sequential like 24*24*32 and reshape it as shown in following code.
def batch_flatten(x):
"""Turn a nD tensor into a 2D tensor with same 0th dimension.
In other words, it flattens each data samples of a batch.
# Arguments
x: A tensor or variable.
# Returns
A tensor.
"""
x = tf.reshape(x, tf.stack([-1, prod(shape(x)[1:])]))
return x