There's an argument in that function seed=something. Even when I set its value, the shuffle give random results. I want the same results.
tf.random.suffle(tf.range(5), seed=5)
If you want to reproduce shuffle results, use (on TF 2.0 beta) the following
tf.random.set_seed(5)
tf.random.shuffle(tf.range(5))
<tf.Tensor: id=35, shape=(5,), dtype=int32, numpy=array([0, 4, 1, 3, 2], dtype=int32)>
tf.random.set_seed(5)
tf.random.shuffle(tf.range(5))
<tf.Tensor: id=41, shape=(5,), dtype=int32, numpy=array([0, 4, 1, 3, 2], dtype=int32)>
tf.random.set_seed(5)
tf.random.shuffle(tf.range(5))
<tf.Tensor: id=47, shape=(5,), dtype=int32, numpy=array([0, 4, 1, 3, 2], dtype=int32)>
About the seed you have used, it indeed fails to reproduce results, tested in TF 2.0 beta
In TF 1.x I believe the right functions is tf.random.set_random_seed
From the docs, I can see there are op level seed, and graph level seed. You are setting the op level, which is not enough - setting the graph level seed with the function in the code above solves this behavior.
Came across this question. It seems like there is now a tf.random.experimental.stateless_shuffle that should give reproducible results.
To get random seeds, you can use tf.random.Generator.
Related
I am currently trying to perform a TensorFlow slice assignment similar to this PyTorch code.
input_seq[1:, :] = torch.from_numpy(stroke[:-1, :])
A plain item assignment like the above does not work for TensorFlow and gives the following error.
TypeError: 'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment.
Previous solutions for the same problem are quite dated using older versions of TensorFlow. I would greatly appreciate any help regarding how to tackle the same.
Here's an example. You can't do numpy like slice assignements. But can do the following (Tested tensorflow==2.9).
tensorflow as tf
import numpy as np
a = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
b = np.array([[6, 5, 4], [9, 8, 7]])
c = tf.tensor_scatter_nd_update(a, [[1], [2]], b)
All reproducible code below is run at Google Colab with TF 2.2.0-rc2.
Adapting the simple example from the documentation for creating a dataset from a simple Python list:
import numpy as np
import tensorflow as tf
tf.__version__
# '2.2.0-rc2'
np.version.version
# '1.18.2'
dataset1 = tf.data.Dataset.from_tensor_slices([1, 2, 3])
for element in dataset1:
print(element)
print(type(element.numpy()))
we get the result
tf.Tensor(1, shape=(), dtype=int32)
<class 'numpy.int32'>
tf.Tensor(2, shape=(), dtype=int32)
<class 'numpy.int32'>
tf.Tensor(3, shape=(), dtype=int32)
<class 'numpy.int32'>
where all data types are int32, as expected.
But changing this simple example to feed a list of strings instead of integers:
dataset2 = tf.data.Dataset.from_tensor_slices(['1', '2', '3'])
for element in dataset2:
print(element)
print(type(element.numpy()))
gives the result
tf.Tensor(b'1', shape=(), dtype=string)
<class 'bytes'>
tf.Tensor(b'2', shape=(), dtype=string)
<class 'bytes'>
tf.Tensor(b'3', shape=(), dtype=string)
<class 'bytes'>
where, surprisingly, and despite the tensors themselves being of dtype=string, their evaluations are of type bytes.
This behavior is not confined to the .from_tensor_slices method; here is the situation with .list_files (the following snippet runs straightforward in a fresh Colab notebook):
disc_data = tf.data.Dataset.list_files('sample_data/*.csv') # 4 csv files
for element in disc_data:
print(element)
print(type(element.numpy()))
the result being:
tf.Tensor(b'sample_data/california_housing_test.csv', shape=(), dtype=string)
<class 'bytes'>
tf.Tensor(b'sample_data/mnist_train_small.csv', shape=(), dtype=string)
<class 'bytes'>
tf.Tensor(b'sample_data/california_housing_train.csv', shape=(), dtype=string)
<class 'bytes'>
tf.Tensor(b'sample_data/mnist_test.csv', shape=(), dtype=string)
<class 'bytes'>
where again, the file names in the evaluated tensors are returned as bytes, instead of string, despite that the tensors themselves are of dtype=string.
Similar behavior is observed also with the .from_generator method (not shown here).
A final demonstration: as shown in the .as_numpy_iterator method documentation, the following equality condition is evaluated as True:
dataset3 = tf.data.Dataset.from_tensor_slices({'a': ([1, 2], [3, 4]),
'b': [5, 6]})
list(dataset3.as_numpy_iterator()) == [{'a': (1, 3), 'b': 5},
{'a': (2, 4), 'b': 6}]
# True
but if we change the elements of b to be strings, the equality condition is now surprisingly evaluated as False!
dataset4 = tf.data.Dataset.from_tensor_slices({'a': ([1, 2], [3, 4]),
'b': ['5', '6']}) # change elements of b to strings
list(dataset4.as_numpy_iterator()) == [{'a': (1, 3), 'b': '5'}, # here
{'a': (2, 4), 'b': '6'}] # also
# False
probably due to the different data types, since the values themselves are evidently identical.
I didn't stumble upon this behavior by academic experimentation; I am trying to pass my data to TF Datasets using custom functions that read pairs of files from the disk of the form
f = ['filename1', 'filename2']
which custom functions work perfectly well on their own, but mapped through TF Datasets give
RuntimeError: not a string
which, after this digging, seems at least not unexplained, if the returned data types are indeed bytes and not string.
So, is this a bug (as it seems), or am I missing something here?
This is a known behavior:
From: https://github.com/tensorflow/tensorflow/issues/5552#issuecomment-260455136
TensorFlow converts str to bytes in most places, including sess.run, and this is unlikely to change. The user is free to convert back, but unfortunately it's too large a change to add a unicode dtype to the core. Closing as won't fix for now.
I guess nothing changed with TensorFlow 2.x - there are still places in which strings are converted to bytes and you have to take care of this manually.
From the issue you have opened yourself, it would seem that they treat the subject as a problem of Numpy, and not of Tensorflow itself.
I was using Tensorflow 2.0 to build a super resolution model. During pre-processing, I wanted to crop both the low and high resolution images by a given patch size. In order to do so, I wanted to get the height and width of the low and high resolution images. But tf.shape(image) is returning None.
Is there a better approach?
Currently I am just resizing every image to some size before using tf.shape, but since not all images have equal size, it is affecting the quality of the imaged. Looking forward to your suggestions.
Edited part:
Here is some parts of the code
low_r = tf.io.decode_jpeg(lr_filename, channels=3)
low_r = tf.cast(low_r, dtype=tf.float32)
print(low_r.shape)
The print statement prints (None, None, 3)
What I wanted was to get the height and weight, like (240,360,3)
I'm not sure if this is also your case, but in my TensorFlow (v2.4.0rc2), my_tensor.shape() also returns TensorShape([None, None, None, None]). It is connected to the fact that the TensorShape tensor is generated during the build and not during the execution.
Using tf.shape() (mentioned in your question, but not used in your code snippet actually) solves it for me.
> my_tensor.shape()
TensorShape([None, None, None, None])
> tf.shape(my_tensor)
[10 512 512 8]
I'm unable to repeat your issue, but this should give you a way to test out your Tensorflow 2.0 install and compare with the results you're currently getting.
Create a tensor and check it's shape:
import tensorflow as tf
t = tf.constant([[[1, 1, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]])
tf.shape(t) # [2, 2, 3]
Out[1]: <tf.Tensor: id=1, shape=(3,), dtype=int32, numpy=array([2, 2, 3])>
Next, checking what the function return when called:
tf_shape_var = tf.shape(t)
print(tf_shape_var)
Output:
tf.Tensor([2 2 3], shape=(3,), dtype=int32)
Finally, calling it on an int and string to get back a valid return:
tf.shape(1)
Out[10]: <tf.Tensor: id=12, shape=(0,), dtype=int32, numpy=array([], dtype=int32)>
tf.shape('asd')
Out[11]: <tf.Tensor: id=15, shape=(0,), dtype=int32, numpy=array([], dtype=int32)>
And the print statements:
print(tf.shape(1))
print(tf.shape('asd'))
Output:
tf.Tensor([], shape=(0,), dtype=int32)
tf.Tensor([], shape=(0,), dtype=int32)
Link for tf.shape() https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/shape
My question is: Are the tf.nn.dynamic_rnn and keras.layers.RNN(cell) truly identical as stated in docs?
I am planning on building an RNN, however, it seems that tf.nn.dynamic_rnn is depricated in favour of Keras.
In particular, it states that:
Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future
version. Instructions for updating: Please use keras.layers.RNN(cell),
which is equivalent to this API
But I don't see how the APIs are equivalent, in the case of variable sequence lengths!
In raw TF, we can specify a tensor of shape (batch_size, seq_lengths). This way, if our sequence is [0, 1, 2, 3, 4] and the longest sequence in the batch is of size 10, we can pad it with 0s and [0, 1, 2, 3, 4, 0, 0, 0, 0, 0], we can say seq_length=5 to process [0, 1, 2, 3, 4].
However, in Keras, this is not how it works! What we can do, is specify the mask_zero=True in previous Layers, e.g. the Embedding Layer. This will also mask the 1st zero!
I can go around it by adding ones to the whole vector, but then thats extra preprocessing that I need to do after processing using tft.compute_vocabulary(), which maps vocabulary words to 0 indexed vector.
No, but they are (or can be made to be) not so different either.
TL;DR
tf.nn.dynamic_rnn replaces elements after the sequence end with 0s. This cannot be replicated with tf.keras.layers.* as far as I know, but you can get a similar behaviour with RNN(Masking(...) approach: it simply stops the computation and carries the last outputs and states forward. You will get the same (non-padding) outputs as those obtained from tf.nn.dynamic_rnn.
Experiment
Here is a minimal working example demonstrating the differences between tf.nn.dynamic_rnn and tf.keras.layers.GRU with and without the use of tf.keras.layers.Masking layer.
import numpy as np
import tensorflow as tf
test_input = np.array([
[1, 2, 1, 0, 0],
[0, 1, 2, 1, 0]
], dtype=int)
seq_length = tf.constant(np.array([3, 4], dtype=int))
emb_weights = (np.ones(shape=(3, 2)) * np.transpose([[0.37, 1, 2]])).astype(np.float32)
emb = tf.keras.layers.Embedding(
*emb_weights.shape,
weights=[emb_weights],
trainable=False
)
mask = tf.keras.layers.Masking(mask_value=0.37)
rnn = tf.keras.layers.GRU(
1,
return_sequences=True,
activation=None,
recurrent_activation=None,
kernel_initializer='ones',
recurrent_initializer='zeros',
use_bias=True,
bias_initializer='ones'
)
def old_rnn(inputs):
rnn_outputs, rnn_states = tf.nn.dynamic_rnn(
rnn.cell,
inputs,
dtype=tf.float32,
sequence_length=seq_length
)
return rnn_outputs
x = tf.keras.layers.Input(shape=test_input.shape[1:])
m0 = tf.keras.Model(inputs=x, outputs=emb(x))
m1 = tf.keras.Model(inputs=x, outputs=rnn(emb(x)))
m2 = tf.keras.Model(inputs=x, outputs=rnn(mask(emb(x))))
print(m0.predict(test_input).squeeze())
print(m1.predict(test_input).squeeze())
print(m2.predict(test_input).squeeze())
sess = tf.keras.backend.get_session()
print(sess.run(old_rnn(mask(emb(x))), feed_dict={x: test_input}).squeeze())
The outputs from m0 are there to show the result of applying the embedding layer.
Note that there are no zero entries at all:
[[[1. 1. ] [[0.37 0.37]
[2. 2. ] [1. 1. ]
[1. 1. ] [2. 2. ]
[0.37 0.37] [1. 1. ]
[0.37 0.37]] [0.37 0.37]]]
Now here are the actual outputs from the m1, m2 and old_rnn architectures:
m1: [[ -6. -50. -156. -272.7276 -475.83362]
[ -1.2876 -9.862801 -69.314 -213.94202 -373.54672 ]]
m2: [[ -6. -50. -156. -156. -156.]
[ 0. -6. -50. -156. -156.]]
old [[ -6. -50. -156. 0. 0.]
[ 0. -6. -50. -156. 0.]]
Summary
The old tf.nn.dynamic_rnn used to mask padding elements with zeros.
The new RNN layers without masking run over the padding elements as if they were data.
The new rnn(mask(...)) approach simply stops the computation and carries the last outputs and states forward. Note that the (non-padding) outputs that I obtained for this approach are exactly the same as those from tf.nn.dynamic_rnn.
Anyway, I cannot cover all possible edge cases, but I hope that you can use this script to figure things out further.
I'm trying to use the tf.image_summary function in tensorflow. I'm trying to visualize a convolutional layer's filter. The filter is defined as tf.Variable(tf.constant(0.1, shape=[5, 5, 16, 32])).
But here, since I only want to see the final filters, I want to find a way to get a filter of size [5, 5, 32] by just taking the first index of the dimension that was 16. If I use [:, :, 0, :] then I assume I would get a [5, 5, 1, 32] filter instead of the [5, 5, 32] I want.
What should I do?
So tf.image_summary takes in a batch as input however it expects 1,3,or 4 in terms of color channels.
so you'd have to pass in to tf.image_summary something like this:
for i in range(int(math.floor(filter.get_shape()[4]/3))):
tf.image_summary(filter[:,:,:,i:i+3])