dataset = tf.data.Dataset.range(1, 6)
def aug(y):
x = np.random.uniform(0,1)
if x > 0.5:
y = 100
return y
dataset = dataset.map(aug)
print(list(dataset))
Run this code, then all the elements in the dataset are as they were, or all equal to 100. How do I make it so each element is individually transformed?
My more specific question below is basically asking this
I create my segmentation training set by:
dataset = tf.data.Dataset.from_tensor_slices((image_paths, mask_paths))
I then apply my augmentation function to the dataset:
def augment(image_path, mask_path)):
//use tf.io.read_file and tf.io.decode_jpeg to convert paths to tensors
x = np.random.choice([0,1])
if x == 1:
image = tf.image.flip_up_down(image)
mask = tf.image.flip_up_down(mask)
return image, mask
training_dataset = dataset.map(augment)
BATCH_SIZE=2
training_dataset = training_dataset.shuffle(100, reshuffle_each_iteration=True)
training_dataset = training_dataset.batch(BATCH_SIZE)
training_dataset = training_dataset.repeat()
training_dataset = training_dataset.prefetch(-1)
However when I visualise my training dataset, all the images have same flip applied- the are all either flipped upside down or not flipped. Where as I'm expecting them to have different flips- some upside down and some not.
Why is this happening?
You need to use tensorflow operations (not numpy or normal python) because tf.data.Dataset.map() executes the mapped function as a graph. When converting a function to a graph, numpy and base python are converted to constants. The augmentation function is only running np.random.uniform(0,1) once and storing it as a constant.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph.
The source for the above is here.
One solution is to use tensorflow operations. I have included an example below. Note that the y value in the if has to be cast to the same dtype as the input.
dataset = tf.data.Dataset.range(1, 6)
def aug(y):
x = tf.random.uniform([], 0, 1)
if x > 0.5:
y = tf.cast(100, y.dtype)
return y
dataset = dataset.map(aug)
print(list(dataset))
You can use a uniform random function or other probability distribution
tf.random.uniform(
shape, minval=0, maxval=None, dtype=tf.dtypes.float32, seed=None, name=None
)
even you can use prebuild method in TensorFlow or Keras for fliping
tf.keras.layers.experimental.preprocessing.RandomFlip(
mode=HORIZONTAL_AND_VERTICAL, seed=None, name=None, **kwargs
)
I'm trying to extract indices of 'True' in tensorflow's tensor based on specific axis.
expected value of
[ [True,False,True,False],
[False,False,True,False] ]
is
[[0,2],
[2]]
I found out that
'tf.where'
is similar to what I expected, but
when I use this funciton, the result was
[[0, 0],
[0, 2],
[1, 2]],
is there any way to get indices of 'True' value according to specific axis?
This should give you what you're looking for -
import tensorflow as tf
b = tf.constant([[True, False, True, False],[ False, False, True, False]])
x=tf.unstack(b)
c = [tf.squeeze(tf.where(e)) for e in x]
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init_op)
print(sess.run([c]))
This edit is inspired by chrert's comment on the previous version of my answer. The solution doesn't actually require one to specify the axis, and I have modified it to reflect this(It does however take the axis=0, which may be considered restrictive). He is right that tf.while_loop along with TensorArrays would allow you to loop over a tensor of any given shape without knowing its shape beforehand; and its good to know how to work with dynamically shaped tensors as well ! However, for bj1123's particular usecase this could fail when trying to stack the results. This is because each row or slice can (and in all probability will!!) have different counts of True and False values. This will raise the error "InvalidArgumentError (see above for traceback): TensorArray has inconsistent shapes." To quantify what I'm trying to say -
import tensorflow as tf
import numpy as np
inputs = tf.placeholder(dtype=tf.bool, shape=(2,4))
time_steps = tf.shape(inputs)[0]
initial_outputs = tf.TensorArray(dtype=tf.int32, size=time_steps)
initial_t = tf.placeholder(dtype='int32')
def cond(t, *args):
return t < time_steps
def body(t, outputs_):
sub = tf.gather(inputs, t)
cur = tf.squeeze(tf.cast(tf.where(sub), tf.int32))
outputs_ = outputs_.write(t, cur)
return t + 1, outputs_
t, outputs = tf.while_loop(cond, body,[initial_t, initial_outputs])
outputs = outputs.stack()
with tf.Session() as sess:
init = tf.global_variables_initializer()
sess.run([init])
print(outputs)
print(sess.run([outputs], feed_dict={inputs: np.asarray([[True,False,True,False],[True, True, False, False]]), initial_t:0}))
The above code will run fine, now try replacing the placeholder with
np.asarray([[True,True,True,False],[True, True, False, False]])
and now you can see the error. There doesn't seem to be any direct way of stacking irregularly shaped tensors into a single tensor. The only way is to have a list of tensors as shown in my first version.
I have also edited the
I am new to PyMC3 and I have been attempting to create a mixture of independent Poisson's using the following code:
import pymc3 as pm
import numpy as np
from pymc3.distributions.continuous import Uniform
from pymc3.distributions.discrete import Poisson
from pymc3.distributions.mixture import Mixture
from pymc3 import Deterministic
# Bayesian non-parametric histogram comparison
def bayesian_nphist(bins_1, bins_2):
# n equals m anyway
m = len(bins_1)
n = len(bins_2)
with pm.Model():
beta = Uniform('beta', lower=0, upper=np.mean(np.append(bins_1, bins_2)) * 2, shape=m)
eta = Uniform('eta', lower=0, upper=np.mean(bins_1) * 2, shape=m)
gamma = Uniform('gamma', lower=0, upper=np.mean(bins_2) * 2, shape=n)
pi_1 = Uniform('pi_1', lower=0, upper=1)
pi_2 = Deterministic('pi_2', 1 - pi_1)
m1 = Poisson.dist(mu=beta, shape=[m, 1])
n1 = Poisson.dist(mu=beta, shape=[n, 1])
m2 = Poisson.dist(mu=eta, shape=[m, 1])
n2 = Poisson.dist(mu=gamma, shape=[n, 1])
Mixture('mixture_M', [pi_1, pi_2], [m1, m2], observed=bins_1)
Mixture('mixture_N', [pi_1, pi_2], [n1, n2], observed=bins_2)
start = pm.find_MAP()
step = pm.NUTS(state=start)
trace = pm.sample(5000, step=step, progressbar=True)
pm.traceplot(trace)
Basically, this is a non-parametric way of comparing histograms, using the fact that a multinomial distribution is a non-parametric model for a histogram, and that multinomial can be modelled as a bunch of independent Poisson variables.
Somehow I managed to get this error:
ValueError: Input dimension mis-match. (input[0].shape[1] = 2, input[1].shape[1] = 14)
on line:
Mixture('mixture_M', [pi_1, pi_2], [m1, m2], observed=bins_1)
when running with bins_1 and bins_2 as (14, 1) integer arrays, so n=m=14. I printed out the shapes of different variables and I believe they match. Have I misunderstood how Mixture in PYMC3 works? Thanks in advance!
I want to compare the predicted values yp from my neural network in a pairwise fashion, and so I was using (back in my old numpy implementation):
idx = np.repeat(np.arange(len(yp)), len(yp))
jdx = np.tile(np.arange(len(yp)), len(yp))
s = yp[[idx]] - yp[[jdx]]
This basically create a indexing mesh which I then use. idx=[0,0,0,1,1,1,...] while jdx=[0,1,2,0,1,2...]. I do not know if there is a simpler manner of doing it...
Anyhow, TensorFlow has a tf.tile(), but it seems to be lacking a tf.repeat().
idx = np.repeat(np.arange(n), n)
v2 = v[idx]
And I get the error:
TypeError: Bad slice index [ 0 0 0 ..., 215 215 215] of type <type 'numpy.ndarray'>
It also does not work to use a TensorFlow constant for the indexing:
idx = tf.constant(np.repeat(np.arange(n), n))
v2 = v[idx]
-
TypeError: Bad slice index Tensor("Const:0", shape=TensorShape([Dimension(46656)]), dtype=int64) of type <class 'tensorflow.python.framework.ops.Tensor'>
The idea is to convert my RankNet implementation to TensorFlow.
You can achieve the effect of np.repeat() using a combination of tf.tile() and tf.reshape():
idx = tf.range(len(yp))
idx = tf.reshape(idx, [-1, 1]) # Convert to a len(yp) x 1 matrix.
idx = tf.tile(idx, [1, len(yp)]) # Create multiple columns.
idx = tf.reshape(idx, [-1]) # Convert back to a vector.
You can simply compute jdx using tf.tile():
jdx = tf.range(len(yp))
jdx = tf.tile(jdx, [len(yp)])
For the indexing, you could try using tf.gather() to extract non-contiguous slices from the yp tensor:
s = tf.gather(yp, idx) - tf.gather(yp, jdx)
According to tf api document, tf.keras.backend.repeat_elements() does the same work with np.repeat() . For example,
x = tf.constant([1, 3, 3, 1], dtype=tf.float32)
rep_x = tf.keras.backend.repeat_elements(x, 5, axis=0)
# result: [1. 1. 1. 1. 1. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 1. 1. 1. 1. 1.]
Just for 1-d tensors, I've made this function
def tf_repeat(y,repeat_num):
return tf.reshape(tf.tile(tf.expand_dims(y,axis=-1),[1,repeat_num]),[-1])
It looks like your question is so popular that people refer it on TF tracker. Sadly the same function is not still implemented in TF.
You can implement it by combining tf.tile, tf.reshape, tf.squeeze. Here is a way to convert examples from np.repeat:
import numpy as np
import tensorflow as tf
x = [[1,2],[3,4]]
print np.repeat(3, 4)
print np.repeat(x, 2)
print np.repeat(x, 3, axis=1)
x = tf.constant([[1,2],[3,4]])
with tf.Session() as sess:
print sess.run(tf.tile([3], [4]))
print sess.run(tf.squeeze(tf.reshape(tf.tile(tf.reshape(x, (-1, 1)), (1, 2)), (1, -1))))
print sess.run(tf.reshape(tf.tile(tf.reshape(x, (-1, 1)), (1, 3)), (2, -1)))
In the last case where repeats are different for each element you most probably will need loops.
Just in case anybody is interested for a 2D method to copy the matrices. I think this could work:
TF_obj = tf.zeros([128, 128])
tf.tile(tf.expand_dims(TF_obj, 2), [1, 1, 2])
import numpy as np
import tensorflow as tf
import itertools
x = np.arange(6).reshape(3,2)
x = tf.convert_to_tensor(x)
N = 3 # number of repetition
K = x.shape[0] # for here 3
order = list(range(0, N*K, K))
order = [[x+i for x in order] for i in range(K)]
order = list(itertools.chain.from_iterable(order))
x_rep = tf.gather(tf.tile(x, [N, 1]), order)
Results from:
[0, 1],
[2, 3],
[4, 5]]
To:
[[0, 1],
[0, 1],
[0, 1],
[2, 3],
[2, 3],
[2, 3],
[4, 5],
[4, 5],
[4, 5]]
If you want:
[[0, 1],
[2, 3],
[4, 5],
[0, 1],
[2, 3],
[4, 5],
[0, 1],
[2, 3],
[4, 5]]
Simply use tf.tile(x, [N, 1])
So I have found that tensorflow has one such method to repeat the elements of an array. The method tf.keras.backend.repeat_elements is what you are looking for. Anyone who comes at a later point of time can save lot of their efforts. This link offers an explanation to the method and specifically says
Repeats the elements of a tensor along an axis, like np.repeat
I have included a very short example which proves that the elements are copied in the exact way as np.repeat would do.
import numpy as np
import tensorflow as tf
x = np.random.rand(2,2)
# print(x) # uncomment this line to see the array's elements
y = tf.convert_to_tensor(x)
y = tf.keras.backend.repeat_elements(x, rep=3, axis=0)
# print(y) # uncomment this line to see the results
You can simulate missing tf.repeat by tf.stacking the value with itself:
value = np.arange(len(yp)) # what to repeat
repeat_count = len(yp) # how many times
repeated = tf.stack ([value for i in range(repeat_count)], axis=1)
I advice using this only on small repeat counts.
Though many clean and working solutions have been given, they seem to all be based on producing the set of indices from scratch each iteration.
While the cost to produce these node's isn't typically significant during training, it may be significant if using your model for inference.
Repeating tf.range (like your example) has come up a few times so I built the following function creator. Given the maximum number of times something will be repeated and the maximum number of things that will need repeating, it returns a function which produces the same values as np.repeat(np.arange(len(multiples)), multiples).
import tensorflow as tf
import numpy as np
def numpy_style_repeat_1d_creator(max_multiple=100, max_to_repeat=10000):
board_num_lookup_ary = np.repeat(
np.arange(max_to_repeat),
np.full([max_to_repeat], max_multiple))
board_num_lookup_ary = board_num_lookup_ary.reshape(max_to_repeat, max_multiple)
def fn_to_return(multiples):
board_num_lookup_tensor = tf.constant(board_num_lookup_ary, dtype=tf.int32)
casted_multiples = tf.cast(multiples, dtype=tf.int32)
padded_multiples = tf.pad(
casted_multiples,
[[0, max_to_repeat - tf.shape(multiples)[0]]])
return tf.boolean_mask(
board_num_lookup_tensor,
tf.sequence_mask(padded_multiples, maxlen=max_multiple))
return fn_to_return
#Here's an example of how it can be used
with tf.Session() as sess:
repeater = numpy_style_repeat_1d_creator(5,4)
multiples = tf.constant([4,1,3])
repeated_values = repeater(multiples)
print(sess.run(repeated_values))
The general idea is to store a repeated tensor and then mask it, but it may help to see it visually (this is for the example given above):
In the example above the following Tensor is produced:
[[0,0,0,0,0],
[1,1,1,1,1],
[2,2,2,2,2],
[3,3,3,3,3]]
For multiples [4,1,3] it will collect the non-X values:
[[0,0,0,0,X],
[1,X,X,X,X],
[2,2,2,X,X],
[X,X,X,X,X]]
resulting in:
[0,0,0,0,1,2,2,2]
tl;dr: To avoid producing the indices each time (can be costly), pre-repeat everything and then mask that tensor each time
A relatively fast implementation was recently added with RaggedTensor utilities from 1.13, but it's not a part of the officially exported API. You can still use it, but there's a chance it might disappear.
from tensorflow.python.ops.ragged.ragged_util import repeat
From the source code:
# This op is intended to exactly match the semantics of numpy.repeat, with
# one exception: numpy.repeat has special (and somewhat non-intuitive) behavior
# when axis is not specified. Rather than implement that special behavior, we
# simply make `axis` be a required argument.
Tensorflow 2.10 has implemented np.repeat feature.
tf.repeat([1, 2, 3], repeats=[3, 1, 2], axis=0)
<tf.Tensor: shape=(6,), dtype=int32, numpy=array([1, 1, 1, 2, 3, 3], dtype=int32)>