How to add random op in tensorflow dataset api? - tensorflow

Question

I want to generate random data in a py_func op, but when I run this code, I get different random data which is expected as the same in this code.
I think the problem come with the tf.data.Dataset.map api. When I get the values of d5, it will run random function twice which d3 and d4 will get different values in this case.
Thanks.
import tensorflow as tf
import numpy as np
def random(x):
def random_(x):
x = np.float32(np.random.rand())
print("run random_")
return x,x
return tf.compat.v1.py_func(
random_,
[x],
[tf.float32, tf.float32],
name="generate_random")
d1 = tf.data.Dataset.from_tensor_slices(tf.zeros(10))
d2 = d1.map(random)
d3 = d2.map(lambda x1,x2: x1)
d4 = d2.map(lambda x1,x2: x2)
d5 = tf.data.Dataset.zip( (d3,d4) )
it = d5.make_one_shot_iterator()
fetch = it.get_next()
with tf.Session() as sess:
while True:
X = sess.run(fetch)
print(X)

Answer 1

As per the documentation -
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset,
and returns a new dataset containing the transformed elements, in the
same order as they appeared in the input. map_func can be used to
change both the values and the structure of a dataset's elements. For
example, adding 1 to each element, or projecting a subset of element
components.
I think it’s best to see the map function as part of the pipeline. Every time you run map, nothing happens with the data yet. Instead, a chain of operations is made in the background. Note that these transformations are executed lazy. Only when you actually request data from the dataset (like with a for loop and batch), the source data from your dataset are fed through the pipeline. At that time, each map is applied on the image or on the batch.
As an alternative, you can fulfill your requirement as below -
Code -
%tensorflow_version 1.x
import tensorflow as tf
import numpy as np
def random(x):
x = np.float32(np.random.rand())
print("run random_")
return x
a = []
for i in range(10):
a.append(random(i))
d1 = tf.data.Dataset.from_tensor_slices(a)
d5 = tf.data.Dataset.zip( (d1,d1) )
it = d5.make_one_shot_iterator()
fetch = it.get_next()
with tf.Session() as sess:
while True:
X = sess.run(fetch)
print(X)
Output -
run random_
run random_
run random_
run random_
run random_
run random_
run random_
run random_
run random_
run random_
WARNING:tensorflow:From <ipython-input-2-ee4ae8ca7ce4>:18: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
(0.6963448, 0.6963448)
(0.92889917, 0.92889917)
(0.054273304, 0.054273304)
(0.3719434, 0.3719434)
(0.33791015, 0.33791015)
(0.873847, 0.873847)
(0.66726416, 0.66726416)
(0.77356577, 0.77356577)
(0.4188753, 0.4188753)
(0.8886825, 0.8886825)
Hope this answers your question. Happy Learning.