I need calculate cosine_distance repeatedly, and tf.losses.cosine_distance returns a scalar Tensor, so I did it like this:
x # a tensor list
y # a tensor list
for i in x:
for j in y:
distance = tf.losses.cosine_distance(i, j, dim=0)
This approach makes graph too big and loading of the program too slow. How can I optimize it?
Loops are no good in tensorflow.
I am assuming all the vectors in the tensor lists are of equal length
Try this:
x_t = tf.stack(x)
y_t = tf.stack(y)
prod = tf.matmul(x_t, y_t, transpose_b=True)
x_len = tf.sqrt(tf.reduce_sum(tf.matmul(x_t, x_t), axis=0))
y_len = tf.sqrt(tf.reduce_sum(tf.matmul(y_t, y_t), axis=0))
cosine_dist = prod/tf.matmul(x_len, y_len, transpose_b=True)
Related
The following code has the irritating trait of making every row of "out" the same. I am trying to classify k time series in Xtrain as [1,0,0,0], [0,1,0,0], [0,0,1,0], or [0,0,0,1], according to the way they were generated (by one of four random algorithms). Anyone know why? Thanks!
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import copy
n = 100
m = 10
k = 1000
hidden_layers = 50
learning_rate = .01
training_epochs = 10000
Xtrain = []
Xtest = []
Ytrain = []
Ytest = []
# ... fill variables with data ..
x = tf.placeholder(tf.float64,shape = (k,1,n,1))
y = tf.placeholder(tf.float64,shape = (k,1,4))
conv1_weights = 0.1*tf.Variable(tf.truncated_normal([1,m,1,hidden_layers],dtype = tf.float64))
conv1_biases = tf.Variable(tf.zeros([hidden_layers],tf.float64))
conv = tf.nn.conv2d(x,conv1_weights,strides = [1,1,1,1],padding = 'VALID')
sigmoid1 = tf.nn.sigmoid(conv + conv1_biases)
s = sigmoid1.get_shape()
sigmoid1_reshape = tf.reshape(sigmoid1,(s[0],s[1]*s[2]*s[3]))
sigmoid2 = tf.nn.sigmoid(tf.layers.dense(sigmoid1_reshape,hidden_layers))
sigmoid3 = tf.nn.sigmoid(tf.layers.dense(sigmoid2,4))
penalty = tf.reduce_sum((sigmoid3 - y)**2)
train_op = tf.train.AdamOptimizer(learning_rate).minimize(penalty)
model = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(model)
for i in range(0,training_epochs):
sess.run(train_op,{x: Xtrain,y: Ytrain})
out = sigmoid3.eval(feed_dict = {x: Xtest})
Likely because your loss function is mean squared error. If you're doing classification you should be using cross-entropy loss
Your loss is penalty = tf.reduce_sum((sigmoid3 - y)**2) that's the squared difference elementwise between a batch of predictions and a batch of values.
Your network output (sigmoid3) is a tensor with shape [?, 4] and y (I guess) is a tensor with shape [?, 4] too.
The squared difference has thus shape [?, 4].
This means that the tf.reduce_sum is computing in order:
The sum over the second dimension of the squared difference, producing a tensor with shape [?]
The sum over the first dimension (the batch size, here indicated with ?) producing a scalar value (shape ()) that's your loss value.
Probably you don't want this behavior (the sum over the batch dimension) and you're looking for the mean squared error over the batch:
penalty = tf.reduce_mean(tf.squared_difference(sigmoid3, y))
I am trying to generate a random boolean mask sampled according to a predefined probability distribution. The probability distribution is stored in a tensor of the same shape as the resulting mask. Each entry contains the probability that the mask will be true at that particular location.
In short I am looking for a function that takes 4 inputs:
pdf: A tensor to use as a PDF
s: The number of samples per mask
n: The total number of masks to generate
replace: A boolean indicating if sampling should be done with replacement
and returns n boolean masks
A simplified way to do this using numpy would look like this:
def sample_mask(pdf, s, replace):
hight, width = pdf.shape
# Flatten to 1 dimension
pdf = np.resize(pdf, (hight*width))
# Sample according to pdf, the result is an array of indices
samples=np.random.choice(np.arange(hight*width),
size=s, replace=replace, p=pdf)
mask = np.zeros(hight*width)
# Apply indices to mask
for s in samples:
mask[s]=1
# Resize back to the original shape
mask = np.resize(mask, (hight, width))
return mask
I already figured out that the sampling part, without the replace parameter, can be done like this:
samples = tf.multinomial(tf.log(pdf_tensor), n)
But I am stuck when it comes to transforming the samples to a mask.
I must have been sleeping, here is how I solved it:
def sample_mask(pdf, s, n, replace):
"""Initialize the model.
Args:
pdf: A 3D Tensor of shape (batch_size, hight, width, channels=1) to use as a PDF
s: The number of samples per mask. This value should be less than hight*width
n: The total number of masks to generate
replace: A boolean indicating if sampling should be done with replacement
Returns:
A Tensor of shape (batch_size, hight, width, channels=1, n) containing
values 1 or 0.
"""
batch_size, hight, width, channels = pdf.shape
# Flatten pdf
pdf = tf.reshape(pdf, (batch_size, hight*width))
if replace:
# Sample with replacement. Output is a tensor of shape (batch_size, n)
sample_fun = lambda: tf.multinomial(tf.log(pdf), s)
else:
# Sample without replacement. Output is a tensor of shape (batch_size, n).
# Cast the output to 'int64' to match the type needed for SparseTensor's indices
sample_fun = lambda: tf.cast(sample_without_replacement(tf.log(pdf), s), dtype='int64')
# Create batch indices
idx = tf.range(batch_size, dtype='int64')
idx = tf.expand_dims(idx, 1)
# Transform idx to a 2D tensor of shape (batch_size, samples_per_batch)
# Example: [[0 0 0 0 0],[1 1 1 1 1],[2 2 2 2 2]]
idx = tf.tile(idx, [1, s])
mask_list = []
for i in range(n):
# Generate samples
samples = sample_fun()
# Combine batch indices and samples
samples = tf.stack([idx,samples])
# Transform samples to a list of indicies: (batch_index, sample_index)
sample_indices = tf.transpose(tf.reshape(samples, [2, -1]))
# Create the mask as a sparse tensor and set sampled indices to 1
mask = tf.SparseTensor(indices=sample_indices, values=tf.ones(s*batch_size), dense_shape=[batch_size, hight*width])
# Convert mask to a dense tensor. Non-sampled values are set to 0.
# Don't validate the indices, since this requires indices to be ordered
# and unique.
mask = tf.sparse.to_dense(mask, default_value=0,validate_indices=False)
# Reshape to input shape and append to list of tensors
mask_list.append(tf.reshape(mask, [batch_size, hight, width, channels]))
# Combine all masks into a tensor of shape:
# (batch_size, hight, width, channels=1, number_of_masks)
return tf.stack(mask_list, axis=-1)
Function for sampling without replacement as proposed here: https://github.com/tensorflow/tensorflow/issues/9260#issuecomment-437875125
It uses the Gumble-max trick: https://timvieira.github.io/blog/post/2014/07/31/gumbel-max-trick/
def sample_without_replacement(logits, K):
z = -tf.log(-tf.log(tf.random_uniform(tf.shape(logits),0,1)))
_, indices = tf.nn.top_k(logits + z, K)
return indices
I'm trying to obtain a matrix, where each element is calculated as follows:
X = torch.ones(batch_size, dim)
X_ = torch.ones(batch_size, dim)
Y = torch.ones(batch_size, dim)
M = torch.zeros(batch_size, batch_size)
for i in range(batch_size):
for j in range(batch_size):
M[i, j] = ((X[i] - X_[i] * Y[j])**2).sum()
It's very slow to calculate M element-wise, is there any suggestion about how to use matrix multiplication to replace the for loops?
Thanks.
If you want to sum() over dim, you can "lift" your 2D problem to 3D and sum there:
M = ((X[:, None, :] - X_[:, None, :] * Y[None, ...])**2).sum(dim=2)
How it works:
X[:, None, :] and X_[:, None, :] are 3D of size (batch_size, 1, dim), and Y[None, ...] is of size (1, batch_size, dim).
When multiplying X_[:, None, :] * Y[None, ...] pytorch broadcasts the dimensions of size 1 to the appropriate dimension to get a result of size (batch_size, batch_size, dim).
Finally, you sum() only over the last dimension (dim=2) to get an output M of size (batch_size, batch_size).
The trick here is done by taking advantage of broadcasting.
I want to create a tensor whose spectral norm is less than or equal to one. I wrote some code to do this but I'm not sure if this is actually correct because I'm not sure whether numpy's L2 norm actually calculates the spectral norm. My code:
def make_tensor(shape):
Y = np.zeros(shape)
mat = []
for i in range(3):
matrix = np.random.random_integers(0, 255, (shape[0], shape[1]))
matrix = matrix/(LA.norm(matrix,2)+np.random.randint(0,255))
mat.append(matrix)
Y[:, :, i] = mat[i]
return Y
Will Y be tensor whose spectral norm is less than or equal to one?
I'm writing a tensorflow code where I want to normalize the variable vector after each update. The following code works well:
sess = tf.InteractiveSession()
y = tf.Variable(tf.random_uniform([2], -0.5, 0.5))
init = tf.initialize_all_variables()
sess.run(init)
a = [2, 3]
loss = tf.reduce_sum(tf.square(a - y))
y = y / tf.sqrt(tf.reduce_sum(tf.square(y)))
optimizer = tf.train.GradientDescentOptimizer(0.05)
train = optimizer.minimize(loss)
for step in range(100):
sess.run(train)
temp2= sess.run(y)
print(temp2)
and gives the desired answer which is [ 0.55469805 0.83205169], a normalized vector in the direction of [2,3]
However, if I change
y = tf.Variable(tf.random_uniform([2], -0.5, 0.5))
to
y = tf.Variable(tf.random_uniform([2,2], -0.5, 0.5))
and
y = y / tf.sqrt(tf.reduce_sum(tf.square(y)))
to
y[0] = y[0] / tf.sqrt(tf.reduce_sum(tf.square(y[0])))
Then I get the error which says "'Variable' object does not support item assignment". I also changed the loss function to
loss = tf.reduce_sum(tf.square(a - y[0]))
Can anybody help how I can normalize a vector column like y[0] of variable type in tensorflow?
As y is a tensor object , you cannot assign the value to the tensor as you do. Hence, you should works on the array of the tensor, and after that assign the value into like the following:
yArray = y.eval()
a = [2, 3]
loss = tf.reduce_sum(tf.square(a - y.eval()[0][:]))
yArray[0][:] = yArray[0][:] / tf.sqrt(tf.reduce_sum(tf.square(yArray[0][:]))).eval()
y.assign(yArray)
In the above, get the array of the tensor using eval function. Then, compute the loss function, and yArray normalization. Finally, assign the value of the yArray into the y.