Let's say I have a permutation index:
pa = [2,0,4,3,1,5] # node permutation index
pw = [0,3,4,1,2,5] # weight permutation index
a = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]
w = np.array([0.7, 0.6, 0.5, 0.4, 0.3, 0.2]
I want semantics that resemble this:
ap = a[pa] # [0.3, 0.1, 0.5, 0.4, 0.2, 0.6]
except that instead of a copy, I want a view. That is, I want:
ap[0] = 0.123
assert(a[2] == 0.123)
I don't think numpy has this concept. But wondering if there is a way to make this happen in TensorFlow.
I want this concept in TensorFlow for arbitrary weight sharing. I need the weight sharing to be pre-defined by an arbitrary index because each target node will have a different ordering of the same set of weights, and apply these weights to an arbitrary subset of the input layer. The weight must be referenced through the permutation so that back-propagation will modify all instances of the same weight.
Related
I am using a MLP model for classification.
When I predict for new data, I want to keep only those predictions whose probability of prediction is larger than 0.5, and change all other predictions into class 0.
How can I do it in keras ?
I'm using using last layer as follows
model.add(layers.Dense(7 , activation='softmax'))
Is it meaningful to get predictions with probability larger than 0.5 using the softmax?
newdata = (nsamples, nfeatures)
predictions = model.predict (newdata)
print (predictions.shape)
(500, 7)
you can do something like:
preds=model.predict etc
index=np.argmax(preds)
probability= preds(index)
if probability >=.75:
print (' class is ', index,' with high confidence')
elif probability >=.5:
print (' class is ', index,' with medium confidence')
else:
print (' class is ', index,' with low confidence')
Softmax function outputs probabilities. So in your case you will have 7 classes and their probability sum will be equal to 1.
Now consider a case [0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.3] which is the output of the softmax. Appyling a threshold in that case would not make sense as you can see.
Threshold 0.5 has nothing to do with n-classed predictions. It is something special for binary classification.
For to get classes, you should use argmax.
Edit: If you want to drop your predictions if they are under a certain threshold, you can use, but that's not a correct way to deal with multi-class predictions:
labels = []
threshold = 0.5
for probs_thresholded in out:
labels.append([])
for i in range(len(probs_thresholded)):
if probs_thresholded[i] >= threshold:
labels[-1].append(1)
else:
labels[-1].append(0)
I was wondering how to penalize less represented classes more then other classes when dealing with a really imbalanced dataset (10 classes over about 20000 samples but here is th number of occurence for each class : [10868 26 4797 26 8320 26 5278 9412 4485 16172 ]).
I read about the Tensorflow function : weighted_cross_entropy_with_logits (https://www.tensorflow.org/api_docs/python/tf/nn/weighted_cross_entropy_with_logits) but I am not sure I can use it for a multi label problem.
I found a post that sum up perfectly the problem I have (Neural Network for Imbalanced Multi-Class Multi-Label Classification) and that propose an idea but it had no answers and I thought the idea might be good :)
Thank you for your ideas and answers !
First of all, there is my suggestion you can modify your cost function to use in a multi-label way. There is code which show how to use Softmax Cross Entropy in Tensorflow for multilabel image task.
With that code, you can multiple weights in each row of loss calculation. Here is the example code in case you have multi-label task: (i.e, each image can have two labels)
logits_split = tf.split( axis=1, num_or_size_splits=2, value= logits )
labels_split = tf.split( axis=1, num_or_size_splits=2, value= labels )
weights_split = tf.split( axis=1, num_or_size_splits=2, value= weights )
total = 0.0
for i in range ( len(logits_split) ):
temp = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( logits=logits_split[i] , labels=labels_split[i] ))
total += temp * tf.reshape(weights_split[i],[-1])
I think you can just use tf.nn.weighted_cross_entropy_with_logits for multiclass classification.
For example, for 4 classes, where the ratios to the class with the largest number of members are [0.8, 0.5, 0.6, 1], You would just give it a weight vector in the following way:
cross_entropy = tf.nn.weighted_cross_entropy_with_logits(
targets=ground_truth_input, logits=logits,
pos_weight = tf.constant([0.8,0.5,0.6,1]))
So I am not entirely sure that I understand your problem given what you have written. The post you link to writes about multi-label AND multi-class, but that doesn't really make sense given what is written there either. So I will approach this as a multi-class problem where for each sample, you have a single label.
In order to penalize the classes, I implemented a weight Tensor based on the labels in the current batch. For a 3-class problem, you could eg. define the weights as the inverse frequency of the classes, such that if the proportions are [0.1, 0.7, 0.2] for class 1, 2 and 3, respectively, the weights will be [10, 1.43, 5]. Defining a weight tensor based on the current batch is then
weight_per_class = tf.constant([10, 1.43, 5]) # shape (, num_classes)
onehot_labels = tf.one_hot(labels, depth=3) # shape (batch_size, num_classes)
weights = tf.reduce_sum(
tf.multiply(onehot_labels, weight_per_class), axis=1) # shape (batch_size, num_classes)
reduction = tf.losses.Reduction.MEAN # this ensures that we get a weighted mean
loss = tf.losses.softmax_cross_entropy(
onehot_labels=onehot_labels, logits=logits, weights=weights, reduction=reduction)
Using softmax ensures that the classification problem is not 3 independent classifications.
I try to implement roulette wheel selection in Tensorflow. So I started with this:
x = tf.random_uniform([tf.shape(probabilities)[0]])
cumsum = tf.cumsum(probabilities, axis=1) # cumulative sum
b = tf.greater_equal(x, cumsum) # Boolean values now
...
indices = tf.where(b) # this given indices for all the True values, I need only the first one per row
indices = indices[:,1] # we only need column index
Any suggestions for this? Or a better procedure to do the roulette wheel selection?
So a small example to make it more clear
probabilities = [[0.2 0.3 0.5],
[0.1 0.6 0.3],
[0.5 0.4 0.1]]
x = [0.27, 0.86, 0.73] # drawn randomly
Then I want as output [1, 2, 1]
As far as I understand, you want to draw the samples from multinomial distribution. To do that, it is easiest to simply use tf.multinomial:
samples = tf.multinomial(tf.log(probabilities), 1)
Possibly followed by reshaping:
samples_vector = tf.reshape(samples, [-1])
I have a matrix L of shape (2,5,2). The values along the last axis form a probability distribution. I want to sample another matrix S of shape (2, 5) where each entry is one of the following integers: 0, 1.
For example,
L = [[[0.1, 0.9],[0.2, 0.8],[0.3, 0.7],[0.5, 0.5],[0.6, 0.4]],
[[0.5, 0.5],[0.9, 0.1],[0.7, 0.3],[0.9, 0.1],[0.1, 0.9]]]
One of the samples could be,
S = [[1, 1, 1, 0, 1],
[1, 1, 1, 0, 1]]
The distributions are binomial in the above example. However, in general, the last dimension of L can be any positive integer, so the distributions can be multinomial.
The samples need to be generated efficiently within Tensorflow computation graph. I know how to do this using numpy using the functions apply_along_axis and numpy.random.multinomial.
You can use tf.multinomial() here.
You will first need to reshape your input tensor to shape [-1, N] (where N is the last dimension of L):
# L has shape [2, 5, 2]
L = tf.constant([[[0.1, 0.9],[0.2, 0.8],[0.3, 0.7],[0.5, 0.5],[0.6, 0.4]],
[[0.5, 0.5],[0.9, 0.1],[0.7, 0.3],[0.9, 0.1],[0.1, 0.9]]])
dims = L.get_shape().as_list()
N = dims[-1] # here N = 2
logits = tf.reshape(L, [-1, N]) # shape [10, 2]
Now we can apply the function tf.multinomial() to logits:
samples = tf.multinomial(logits, 1)
# We reshape to match the initial shape minus the last dimension
res = tf.reshape(samples, dims[:-1])
Be cautious when using tf.multinomial(). The inputs to the function should be logits and not probability distributions.
However, in your example, the last axis is a probability distribution.
I use numpy.polyfit to get a linear regression: coeffs = np.polyfit(x, y, 1).
What is the best way to calculate the error of the fit's slope using numpy?
As already mentioned by #ebarr in the comments, you can use np.polyfit to return the residuals by using the keyword argument full=True.
Example:
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
z, residuals, rank, singular_values, rcond = np.polyfit(x, y, 3, full=True)
residuals then is the sum of least squares.
Alternatively, you can use the keyword argument cov=True to get the covariance matrix.
Example:
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
z, cov = np.polyfit(x, y, 3, cov=True)
Then, the diagonal elements of cov are the variances of the coefficients in z, i.e. np.sqrt(np.diag(cov)) gives you the standard deviations of the coefficients. You can use the standard deviations to estimate the probability that the absolute error exceeds a certain value, e.g. by inserting the standard deviations in the uncertainty propagation calculation. If you use e.g. 3*standard deviations in the uncertainty propagation, you calculate the error which will not be exceeded in 99.7% of the cases.
One last hint: you have to choose whether you choose full=True or cov=True. cov=True only works when full=False (default) or vice versa.