I'm trying to plot a graph for precision vs recall this is my classification report. i don't know how to plot a graph displaying these.
this is my code for classification report
from sklearn.metrics import classification_report
print("")
print("Confusion Matrix")
print(confusion_matrix(Y_test, predictions))
print("")
print("Classification Report XGBOOST")
print(classification_report(predictions,Y_test))
output:
Confusion Matrix
[[1163 55]
[ 46 665]]
Classification Report xgboost
precision recall f1-score support
0 0.95 0.96 0.96 1209
1 0.94 0.92 0.93 720
accuracy 0.95 1929
macro avg 0.95 0.94 0.94 1929
weighted avg 0.95 0.95 0.95 1929
i'm trying to do something like this:
visulise my precision and recall using a graph.
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(Y_test,predictions)
plt.step(recall, precision, color='b', alpha=0.2,
where='post')
plt.fill_between(recall, precision, alpha=0.2, color='b')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
Try this.
Related
I am training an electra model with tensorflow on a multi label task. The ROC performance of each individual label is
AUROC per tag
morality_binary: 0.8840802907943726
emotion_binary: 0.8690611124038696
positive_binary: 0.9115268588066101
negative_binary: 0.9200447201728821
care_binary: 0.9266915321350098
fairness_binary: 0.8638730645179749
authority_binary: 0.8471786379814148
sanctity_binary: 0.9040042757987976
harm_binary: 0.9046630859375
injustice_binary: 0.8968375325202942
betrayal_binary: 0.846387505531311
subversion_binary: 0.7741811871528625
degradation_binary: 0.9601025581359863
But when I run the the sklearn classification report:
THRESHOLD = 0.5
y_pred = predictions.numpy()
y_true = labels.numpy()
upper, lower = 1, 0
y_pred = np.where(y_pred > THRESHOLD, upper, lower)
print(classification_report(
y_true,
y_pred,
target_names=LABEL_COLUMNS,
zero_division=0
))
... five of the labels turns out with an f-score of 0:
precision recall f1-score support
morality_binary 0.72 0.73 0.73 347
emotion_binary 0.66 0.73 0.69 303
positive_binary 0.71 0.76 0.73 242
negative_binary 0.70 0.62 0.65 141
care_binary 0.67 0.60 0.63 141
fairness_binary 0.55 0.53 0.54 166
authority_binary 0.00 0.00 0.00 49
sanctity_binary 0.00 0.00 0.00 23
harm_binary 0.48 0.32 0.39 50
injustice_binary 0.62 0.56 0.59 97
betrayal_binary 0.00 0.00 0.00 30
subversion_binary 0.00 0.00 0.00 8
degradation_binary 0.00 0.00 0.00 10
Can someone explain to me how this is possible? I can understand a low f-score, but 0?
I assume 0 is negative and 1 is positive.
AUROC calculates the area under the ROC curve as a measure of how well a classifier performs (0.5 score is a random, coin-flip model). To draw the ROC curve, you need to calculate two values at different threshold values to distinguish positive from negative examples.
y-axis: True positive rate (TPR) - How many of the positive examples did the model predict as negative.
x-axis: False positive rate (FPR) - How many of the negative examples did the model predict as positive.
TPR is also called recall. We calculate this using the following formula:
TPR = True positives / (True positives + False Negatives)
= True positives / All positives
So the only way TPR can be 0 is because TP is also 0. This means that precision will also be 0 as we calculate precision using the following formula:
Precision = True positives / (True positives + False positives)
Which will also result in 0 if and only if TP is equal to 0.
Now given the ROC curve (Roc curve shown here), if FPR is 0, the area under the curve will also be equal to 0. This is because you have picked a single threshold value (0.5) in your code to predict 0 or 1. This is not a representation of the ROC curve and AUROC measure.
I suggest you take a look at the ROC curve and try different values for you classification threshold. The resulting AUROC values suggest that your model performs better than a random one in general, so you should find a good threshold.
I have a DataFrame like this:
biases = ['+5V']*3 + ['-5V']*8
df = pd.DataFrame({'Bias': biases,
'Ea': [1.059, 0.287, 0.451, 0.533, 0.587, 0.466, 0.805, 0.499, 0.473, 0.721, 0.461],
'dEa': [0.072,0.005,0.004,0.018,0.036,0.032,0.009,0.015,0.011,0.044,0.004]})
print(df.head())
Bias Ea dEa
3 -5V 0.533 0.018
4 -5V 0.587 0.036
5 -5V 0.466 0.032
6 -5V 0.805 0.009
7 -5V 0.499 0.015
I would like to make a scatterplot with Bias on the x axis, Ea on the y axis, the y errors should be given by dEa. I can do it easily with:
df.hvplot.scatter(x='Bias', y='Ea')*\
df.hvplot.errorbars(x='Bias', y='Ea', yerr1='dEa')
The problem is that I want to add a jitter to this plot, but only the hv.scatter function has the jitter option. Therefore, by calling df.hvplot.scatter(x='Bias', y='Ea').opts(jitter=0.1) I get the jitter only on the scatter plot and not on the errors:
Do you have any idea on how can I get the errors to follow the scatter points?
I am working on an image-segmentation application where the loss function is Dice loss. The issue is the the loss function becomes NAN after some epochs. I am doing 5-fold cross validation and checking validation and training losses for each fold. For some folds, the loss quickly becomes NAN and for some folds, it takes a while to reach it to NAN. I have inserted a constant in loss function formulation to avoid over/under-flow but still it the same problem occurs. My inputs are scaled within range [-1, 1]. I have seen people suggested using regularizers and different optimizers but I dont understand why the loss gets to NAN at first place. I have pasted the loss function, and training and validation losses for some epochs below. Initially only the validation loss and dice score for validation loss becomes NAN, but later all metrics becomes NAN.
def dice_loss(y_true, y_pred): #y_true--> ground-truth, y_pred-->predictions
smooth=1.
y_true_f = tf.keras.backend.flatten(y_true)
y_pred_f = tf.keras.backend.flatten(y_pred)
intersection = tf.keras.backend.sum(y_true_f * y_pred_f)
return 1-(2. * intersection +smooth) / (tf.keras.backend.sum(y_true_f) +
tf.keras.backend.sum(y_pred_f) +smooth)
epoch train_dice_score train_loss val_dice_score val_loss
0 0.42387727 0.423877264 0.35388064 0.353880603
1 0.23064087 0.230640889 0.21502239 0.215022382
2 0.17881058 0.178810576 0.1767999 0.176799848
3 0.15746565 0.157465705 0.16138957 0.161389555
4 0.13828343 0.138283484 0.12770002 0.127699989
5 0.10434002 0.104340041 0.0981831 0.098183098
6 0.08013707 0.080137035 0.08188484 0.081884826
7 0.07081806 0.070818066 0.070421465 0.070421467
8 0.058371827 0.058371854 0.060712796 0.060712777
9 0.06381426 0.063814262 nan nan
10 0.105625264 0.105625251 nan nan
11 0.10790708 0.107907102 nan nan
12 0.10719114 0.10719115 nan nan
I was getting same problem with my segmentation model too. I got that problem when I use both of dice loss and weighted cross entropy loss. I found a solution if somebody still has a same problem.
I was focusing my custom loss but then I figure out nan value came from inside of model when calculation time. Because of relu, inner values becomes to high then become nan.
To solve this I use batch normalization after every convolution with relu and it worked for me.
i started writing Neuronal Networks with tensorflow and there is one Problem i seem to face in each of my example Projects.
My loss allways starts at something like 50 or higher and does not decrease or if it does, it does so slowly that after all my epochs i do not even get near an acceptable loss-rate.
Things it already tried (and did not affect the result very much)
tested on overfitting, but in the following example
you can see that i have 15000 training and 15000 testing-datasets and
something like 900 neurons
tested different optimizers and optimizer-values
tried increasing the traingdata by using the testdata as
trainingdata aswell
tried increasing and decreasing the batchsize
I created the network on knowledge of https://youtu.be/vq2nnJ4g6N0
But let us have a look on one of my testprojects:
I have a list of names and wanted to assume the gender so my raw data looks like this:
names=["Maria","Paul","Emilia",...]
genders=["f","m","f",...]
For feeding it into the network i transform the names into an array of charCodes (expecting a maxlength of 30) and the gender into a bit array
names=[[77.,97. ,114.,105.,97. ,0. ,0.,...]
[80.,97. ,117.,108.,0. ,0. ,0.,...]
[69.,109.,105.,108.,105.,97.,0.,...]]
genders=[[1.,0.]
[0.,1.]
[1.,0.]]
I built the network with 3 hidden layers [30,20],[20,10],[10,10] and [10,2] for the output layer. All hidden layers have a ReLU as activation function. The output layer has a softmax.
# Input Layer
x = tf.placeholder(tf.float32, shape=[None, 30])
y_ = tf.placeholder(tf.float32, shape=[None, 2])
# Hidden Layers
# H1
W1 = tf.Variable(tf.truncated_normal([30, 20], stddev=0.1))
b1 = tf.Variable(tf.zeros([20]))
y1 = tf.nn.relu(tf.matmul(x, W1) + b1)
# H2
W2 = tf.Variable(tf.truncated_normal([20, 10], stddev=0.1))
b2 = tf.Variable(tf.zeros([10]))
y2 = tf.nn.relu(tf.matmul(y1, W2) + b2)
# H3
W3 = tf.Variable(tf.truncated_normal([10, 10], stddev=0.1))
b3 = tf.Variable(tf.zeros([10]))
y3 = tf.nn.relu(tf.matmul(y2, W3) + b3)
# Output Layer
W = tf.Variable(tf.truncated_normal([10, 2], stddev=0.1))
b = tf.Variable(tf.zeros([2]))
y = tf.nn.softmax(tf.matmul(y3, W) + b)
Now the calculation for the loss, accuracy and the training operation:
# Loss
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
# Accuracy
is_correct = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
# Training
train_operation = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)
I train the network in batches of 100
sess = tf.Session()
sess.run(tf.global_variables_initializer())
for i in range(150):
bs = 100
index = i*bs
inputBatch = inputData[index:index+bs]
outputBatch = outputData[index:index+bs]
sess.run(train_operation, feed_dict={x: inputBatch, y_: outputBatch})
accuracyTrain, lossTrain = sess.run([accuracy, cross_entropy], feed_dict={x: inputBatch, y_: outputBatch})
if i%(bs/10) == 0:
print("step %d loss %.2f accuracy %.2f" % (i, lossTrain, accuracyTrain))
And i get the following result:
step 0 loss 68.96 accuracy 0.55
step 10 loss 69.32 accuracy 0.50
step 20 loss 69.31 accuracy 0.50
step 30 loss 69.31 accuracy 0.50
step 40 loss 69.29 accuracy 0.51
step 50 loss 69.90 accuracy 0.53
step 60 loss 68.92 accuracy 0.55
step 70 loss 68.99 accuracy 0.55
step 80 loss 69.49 accuracy 0.49
step 90 loss 69.25 accuracy 0.52
step 100 loss 69.39 accuracy 0.49
step 110 loss 69.32 accuracy 0.47
step 120 loss 67.17 accuracy 0.61
step 130 loss 69.34 accuracy 0.50
step 140 loss 69.33 accuracy 0.47
What am i doing wrong?
Why does it start at ~69 in my Project and not lower?
Thank you very much guys!
There's nothing wrong with 0.69 nats of entropy per samples, as a starting point for a binary classification.
If you convert to base 2, 0.69/log(2), you'll see that it's almost exactly 1 bit per sample which is exactly what you would expect if you're unsure about a binary classification.
I usually use the mean loss instead of the sum so things are less sensitive to batch size.
You should also not calculate the entropy directly yourself, because that method breaks easily. you probably want tf.nn.sigmoid_cross_entropy_with_logits.
I also like starting with the Adam Optimizer instead of pure gradient descent.
Here are two reasons you might be having some trouble with this problem:
1) Character codes are ordered, but the order doesn't mean anything. Your inputs would be easier for the network to take as input if they were input as one-hot vectors. So your input would be a 26x30 = 780 element vector. Without that the network has to waste a bunch of capacity learning the boundaries between letters.
2) You've only got fully connected layers. This makes it impossible for it to learn a fact independent of it's absolute position in the name. 6 of the top 10 girls names in 2015 ended in 'a', while 0 of the top 10 boys names did. As currently written your network needs to re-learn "Usually it's a girl's name if it ends in 'a'" independently for each name length. Using some convolution layers would allow it to learn facts once across all name lengths.
I have a file of data consisting of dates in column one and a series of measurements in columns 2 thru n. I like that Pandas understands dates but I can't figure out how to do simple best fit line. Using np.polyfit is easy but it doesn't understand dates. A sample of my attempt follows.
from datetime import datetime
from StringIO import StringIO
import pandas as pd
zdata = '2013-01-01, 5.00, 100.0 \n 2013-01-02, 7.05, 98.2 \n 2013-01-03, 8.90, 128.0 \n 2013-01-04, 11.11, 127.2 \n 2013-01-05 13.08, 140.0'
unames = ['date', 'm1', 'm2']
df = pd.read_table(StringIO(zdata), sep="[ ,]*", header=None, names=unames, \
parse_dates=True, index_col=0)
Y = pd.Series(df['m1'])
model = pd.ols(y=Y, x=df, intercept=True)
In [232]: model.beta['m1']
Out[232]: 0.99999999999999822
In [233]: model.beta['intercept']
Out[233]: -7.1054273576010019e-15
How do I interpret those numbers? If I use 1,2..5 instead of dates np.polyfit gives [ 2.024, 2.958]
which are slope and intercept I expect.
I looked for simple examples but didn't find any.
I believe you're doing multiple linear regression with the code you provided:
-------------------------Summary of Regression Analysis-------------------------
Formula: Y ~ <m1> + <m2> + <intercept>
Number of Observations: 5
Number of Degrees of Freedom: 3
R-squared: 1.0000
Adj R-squared: 1.0000
Rmse: 0.0000
F-stat (2, 2): inf, p-value: 0.0000
Degrees of Freedom: model 2, resid 2
-----------------------Summary of Estimated Coefficients------------------------
Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5%
--------------------------------------------------------------------------------
m1 1.0000 0.0000 271549416425785.53 0.0000 1.0000 1.0000
m2 -0.0000 0.0000 -0.09 0.9382 -0.0000 0.0000
intercept -0.0000 0.0000 -0.02 0.9865 -0.0000 0.0000
---------------------------------End of Summary---------------------------------
Note the formula for regression: Y ~ <m1> + <m2> + <intercept>. If you want a simple linear regression for m1 and m2 separately, then you should create Xs:
X = pd.Series(range(1, len(df) + 1), index=df.index)
And make the regression:
model = pd.ols(y=Y, x=X, intercept=True)
Result:
-------------------------Summary of Regression Analysis-------------------------
Formula: Y ~ <x> + <intercept>
Number of Observations: 5
Number of Degrees of Freedom: 2
R-squared: 0.9995
Adj R-squared: 0.9993
Rmse: 0.0861
F-stat (1, 3): 5515.0414, p-value: 0.0000
Degrees of Freedom: model 1, resid 3
-----------------------Summary of Estimated Coefficients------------------------
Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5%
--------------------------------------------------------------------------------
x 2.0220 0.0272 74.26 0.0000 1.9686 2.0754
intercept 2.9620 0.0903 32.80 0.0001 2.7850 3.1390
---------------------------------End of Summary---------------------------------
It's a bit weird that you got slightly different numbers when using np.polyfit. Here's my output:
[ 2.022 2.962]
Which is the same as pandas' ols output. I checked this with scipy's linregress and got the same result.