I need to fetch the intermediate value of a tensor in tf.while_loop(), however, it only gives me the last returned value.
For example, I have a variable x, which has 3 pages and its dimension is 3*2*4. Now I want to fetch each page one time and calculate the total sum, the page sum, the mean, max and min value of each page. Then I define the condition and body function and want to use tf.while_loop() to calculate the needed results. The source code is as bellow.
import tensorflow as tf
x = tf.constant([[[41, 8, 48, 82],
[9, 56, 67, 23]],
[[95, 89, 44, 54],
[11, 33, 29, 1]],
[[34, 9, 5, 70],
[14, 35, 18, 17]]], dtype=tf.int32)
def cond(out, count, x):
return count < 3
def body(out, count, x):
outTemp = tf.slice(x, [count, 0, 0], [1, -1, -1])
count += 1
outPack = tf.unpack(out)
outPack[0] += tf.reduce_sum(outTemp)
outPack[1] = tf.reduce_sum(outTemp)
outPack[2] = tf.reduce_mean(outTemp)
outPack[3] = tf.reduce_max(outTemp)
outPack[4] = tf.reduce_min(outTemp)
out = tf.pack(outPack)
return out, count, x
out = tf.Variable(tf.constant([0, 0, 0, 0, 0])) # total sum, page sum, mean, max, min
count = tf.Variable(tf.constant(0))
result = tf.while_loop(cond, body, [out, count, x])
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
print(sess.run(x))
print(sess.run(result)[0])
When I run the program, it only gives me the returned value of the last time and I can only get the results of the last page.
So the question is, How can I get the results of each page and How can I get the intermediate value from tf.while_loop()?
Thank you.
To get the "intermediate value" of any variable, you can simply make use of the tf.Print op which really is an identity operation with the side effect of printing a relevant message when evaluating the aforementioned variable.
As an example,
x = tf.Print(x, [x], "Value of x is: ")
Can be placed in any line where you want the value to be reported.
Related
I have two tensors as follows:
a = tf.constant([1, 0, 0, 1, 0, 1, 1])
b = tf.constant(0) # see below
I want to add one to b each time a[i] = 1
So I did this:
i = tf.constant(0)
def condition(i):
return i < 7
def f1(): return 1
def f2(): return 0
def body(i):
b = tf.constant(0)
b += tf.cond(tf.equal(a[i], 1), f1, f2)
return i + 1
r = tf.while_loop(condition, body, [i])
tf.print(r)
But I got the following error:
TypeError: Cannot iterate over a scalar-tensor.
I don't understand why after many tries to figure out and correct.
Can someone help me fix this? Also, I wanted to retrieve the value of b at the end of the loop, how to do this?
Thank you in advance.
You can't have border effects in your loop. It means that your condition function and your body function must take as argument all the variable that you are using in your loop.
You should declare your functions the following way:
def condition(i,a,b):
return i < 7
def f1(): return 1
def f2(): return 0
def body(i,a,b):
b += tf.cond(tf.equal(a[i], 1), f1, f2)
return i + 1, a, b
Notice that the body function returns all the loop variables, in the same order that they were passed to the function.
and call the while loop passing the tuple (i,a,b) as your loop variables:
>>> tf.while_loop(condition, body, (i,a,b))
(<tf.Tensor: shape=(), dtype=int32, numpy=7>,
<tf.Tensor: shape=(7,), dtype=int32, numpy=array([1, 0, 0, 1, 0, 1, 1], dtype=int32)>,
<tf.Tensor: shape=(), dtype=int32, numpy=4>)
The loop returns a tuple containing the values of (i,a,b) after the execution of the loop. The last element is b, which is equal to 4, corresponding to the number of 1 in a.
Side note: I assume that your example is a simplified example, but if that's not the case, you should replace that while_loop with a call to tf.math.reduce_sum.
I am trying to build a windowed dataset from a univariate time series.
The idea is if the series looks like [1, 2, 3, 4, 5, 6] and the window length was 2, then
I'd take windows of length 3 to account for 2 X features and Y target output, so
[[1, 2, 3], [2, 3, 4], [3, 4, 5], [4, 5, 6]] then I'll shuffle them up to avoid bias from that, and split out the input features from target output for each window: [[[1, 2], [3]], [[2, 3], [4]], [[3, 4], [5]], [[4, 5], [6]]]
def windowed_dataset(series):
# Initially the data is (N,) expand dims to (N, 1)
series = tf.expand_dims(series, axis=-1)
# Tensorflow Dataset from the array
ds = tf.data.Dataset.from_tensor_slices(series)
# Create the windows that will serve as input features and label (hence +1)
ds = ds.window(window_len + 1, shift=1, drop_remainder=True)
ds = ds.flat_map(lambda w: w.batch(window_len + 1))
# randomize order
ds = ds.shuffle(shuffle_buffer)
# Separate the inputs and the target output(label)
ds = ds.map(lambda w: (w[:-1], w[-1]))
return ds.batch(batch_size).prefetch(1)
However I'd like to add some normalization. For example if my window is w=[1, 2, 3] then I'd like to normalize according to [p/w[0] - 1 for p in w]
I thought I could achieve this with ds.map and
def normalize_window(w):
return [((i/w[0]) -1) for i in w]
ds = ds.map(normalize_window)
because map is supposed to apply the function to each window in the dataset, but this didn't work. All the example in tensorflow dataset docs use map with lambda functions but I presume it works with regular functions too
Does anyone know how it should be done?
EDIT
The traceback I get is
<ipython-input-39-929295e1b775> in <module>()
----> 1 dataset = model_forecast_datasets(btc_model, np_data[:6])
11 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
263 except Exception as e: # pylint:disable=broad-except
264 if hasattr(e, 'ag_error_metadata'):
--> 265 raise e.ag_error_metadata.to_exception(e)
266 else:
267 raise
OperatorNotAllowedInGraphError: in user code:
<ipython-input-38-b3d0f7e17689>:12 normalize_window *
return [(i/w[0] -1) for i in w]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:561 __iter__
self._disallow_iteration()
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:557 _disallow_iteration
self._disallow_in_graph_mode("iterating over `tf.Tensor`")
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:537 _disallow_in_graph_mode
" this function with #tf.function.".format(task))
OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with #tf.function.
You would need a function that vectorizes the calculation, something like
def normalize(data):
mean = tf.math.reduce_mean(data)
std = tf.math.reduce_std(data)
data = tf.subtract(data, mean)
data = tf.divide(data, std)
return data
ds = ds.map(normalize)
Edit: For your specific normalization this may work:
def normalize(data):
data1 = tf.subtract(data, tf.constant(1))
data1 = tf.divide(data1, data[0])
return data1
(this would have to go after batching ds = ds.flat_map(...)
I have a dataset (call it Data) with ~25000 instances that I want to split into a train set, development set, and test set. I want it to be such that,
train set = 0.7*Data
development set = 0.1*Data
test set = 0.2*Data
When making the split, I want the instances to be randomly sampled and NOT REPEATED between the 3 sets. This is why I can't use something like,
train_set = Data.sample(frac=0.7)
dev_set = Data.sample(frac=0.1)
train_set = Data.sample(frac=0.2)
where instances from Data may be repeated in the sets. Is there a build in function that I am missing or could you help me write a function for doing this?
I will use an array to demonstrate an example of what I am looking for.
A = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
splits = [0.7, 0.1, 0.2]
def splitFunction(data, array_of_splits):
// I need your help here
splits = splitFunction(A, splits)
#output
[[1, 3, 8, 9, 6, 7, 2], [4], [5, 0]]
Thank you in advance!
from random import shuffle
def splitFunction(data, array_of_splits):
data_copy = data[:] # copy data if don't want to change original array
shuffle(data_copy) # randomizes data
splits = []
startIndex = 0
for val in array_of_splits:
split = data_copy[startIndex:startIndex + val*len(data)]
startIndex = startIndex + val*len(data)
splits.append(split)
return splits
I have an array of points along a line:
a = np.array([18, 56, 32, 75, 55, 55])
I have another array that corresponds to the indices I want to use to access the information in a (they will always have equal lengths). Neither array a nor array b are sorted.
b = np.array([0, 2, 3, 2, 2, 2])
I want to group a into multiple sub-arrays such that the following would be possible:
c[0] -> array([18])
c[2] -> array([56, 75, 55, 55])
c[3] -> array([32])
Although the above example is simple, I will be dealing with millions of points, so efficient methods are preferred. It is also essential later that any sub-array of points can be accessed in this fashion later in the program by automated methods.
Here's one approach -
def groupby(a, b):
# Get argsort indices, to be used to sort a and b in the next steps
sidx = b.argsort(kind='mergesort')
a_sorted = a[sidx]
b_sorted = b[sidx]
# Get the group limit indices (start, stop of groups)
cut_idx = np.flatnonzero(np.r_[True,b_sorted[1:] != b_sorted[:-1],True])
# Split input array with those start, stop ones
out = [a_sorted[i:j] for i,j in zip(cut_idx[:-1],cut_idx[1:])]
return out
A simpler, but lesser efficient approach would be to use np.split to replace the last few lines and get the output, like so -
out = np.split(a_sorted, np.flatnonzero(b_sorted[1:] != b_sorted[:-1])+1 )
Sample run -
In [38]: a
Out[38]: array([18, 56, 32, 75, 55, 55])
In [39]: b
Out[39]: array([0, 2, 3, 2, 2, 2])
In [40]: groupby(a, b)
Out[40]: [array([18]), array([56, 75, 55, 55]), array([32])]
To get sub-arrays covering the entire range of IDs in b -
def groupby_perID(a, b):
# Get argsort indices, to be used to sort a and b in the next steps
sidx = b.argsort(kind='mergesort')
a_sorted = a[sidx]
b_sorted = b[sidx]
# Get the group limit indices (start, stop of groups)
cut_idx = np.flatnonzero(np.r_[True,b_sorted[1:] != b_sorted[:-1],True])
# Create cut indices for all unique IDs in b
n = b_sorted[-1]+2
cut_idxe = np.full(n, cut_idx[-1], dtype=int)
insert_idx = b_sorted[cut_idx[:-1]]
cut_idxe[insert_idx] = cut_idx[:-1]
cut_idxe = np.minimum.accumulate(cut_idxe[::-1])[::-1]
# Split input array with those start, stop ones
out = [a_sorted[i:j] for i,j in zip(cut_idxe[:-1],cut_idxe[1:])]
return out
Sample run -
In [241]: a
Out[241]: array([18, 56, 32, 75, 55, 55])
In [242]: b
Out[242]: array([0, 2, 3, 2, 2, 2])
In [243]: groupby_perID(a, b)
Out[243]: [array([18]), array([], dtype=int64),
array([56, 75, 55, 55]), array([32])]
I am looking for a TensorFlow way of implementing something similar to Python's list.index() function.
Given a matrix and a value to find, I want to know the first occurrence of the value in each row of the matrix.
For example,
m is a <batch_size, 100> matrix of integers
val = 23
result = [0] * batch_size
for i, row_elems in enumerate(m):
result[i] = row_elems.index(val)
I cannot assume that 'val' appears only once in each row, otherwise I would have implemented it using tf.argmax(m == val). In my case, it is important to get the index of the first occurrence of 'val' and not any.
It seems that tf.argmax works like np.argmax (according to the test), which will return the first index when there are multiple occurrences of the max value.
You can use tf.argmax(tf.cast(tf.equal(m, val), tf.int32), axis=1) to get what you want. However, currently the behavior of tf.argmax is undefined in case of multiple occurrences of the max value.
If you are worried about undefined behavior, you can apply tf.argmin on the return value of tf.where as #Igor Tsvetkov suggested.
For example,
# test with tensorflow r1.0
import tensorflow as tf
val = 3
m = tf.placeholder(tf.int32)
m_feed = [[0 , 0, val, 0, val],
[val, 0, val, val, 0],
[0 , val, 0, 0, 0]]
tmp_indices = tf.where(tf.equal(m, val))
result = tf.segment_min(tmp_indices[:, 1], tmp_indices[:, 0])
with tf.Session() as sess:
print(sess.run(result, feed_dict={m: m_feed})) # [2, 0, 1]
Note that tf.segment_min will raise InvalidArgumentError when there is some row containing no val. In your code row_elems.index(val) will raise exception too when row_elems don't contain val.
Looks a little ugly but works (assuming m and val are both tensors):
idx = list()
for t in tf.unpack(m, axis=0):
idx.append(tf.reduce_min(tf.where(tf.equal(t, val))))
idx = tf.pack(idx, axis=0)
EDIT:
As Yaroslav Bulatov mentioned, you could achieve the same result with tf.map_fn:
def index1d(t):
return tf.reduce_min(tf.where(tf.equal(t, val)))
idx = tf.map_fn(index1d, m, dtype=tf.int64)
Here is another solution to the problem, assuming there is a hit on every row.
import tensorflow as tf
val = 3
m = tf.constant([
[0 , 0, val, 0, val],
[val, 0, val, val, 0],
[0 , val, 0, 0, 0]])
# replace all entries in the matrix either with its column index, or out-of-index-number
match_indices = tf.where( # [[5, 5, 2, 5, 4],
tf.equal(val, m), # [0, 5, 2, 3, 5],
x=tf.range(tf.shape(m)[1]) * tf.ones_like(m), # [5, 1, 5, 5, 5]]
y=(tf.shape(m)[1])*tf.ones_like(m))
result = tf.reduce_min(match_indices, axis=1)
with tf.Session() as sess:
print(sess.run(result)) # [2, 0, 1]
Here is a solution which also considers the case the element is not included by the matrix (solution from github repository of DeepMind)
def get_first_occurrence_indices(sequence, eos_idx):
'''
args:
sequence: [batch, length]
eos_idx: scalar
'''
batch_size, maxlen = sequence.get_shape().as_list()
eos_idx = tf.convert_to_tensor(eos_idx)
tensor = tf.concat(
[sequence, tf.tile(eos_idx[None, None], [batch_size, 1])], axis = -1)
index_all_occurrences = tf.where(tf.equal(tensor, eos_idx))
index_all_occurrences = tf.cast(index_all_occurrences, tf.int32)
index_first_occurrences = tf.segment_min(index_all_occurrences[:, 1],
index_all_occurrences[:, 0])
index_first_occurrences.set_shape([batch_size])
index_first_occurrences = tf.minimum(index_first_occurrences + 1, maxlen)
return index_first_occurrences
And:
import tensorflow as tf
mat = tf.Variable([[1,2,3,4,5], [2,3,4,5,6], [3,4,5,6,7], [0,0,0,0,0]], dtype = tf.int32)
idx = 3
first_occurrences = get_first_occurrence_indices(mat, idx)
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
sess.run(first_occurrence) # [3, 2, 1, 5]