I'm using default pipeline configuration (ssd_inception_v2_pets.config) and pretrained inception v2 COCO model. In TensorBoard, the loss continues decreasing, but the average precision isn't getting any better. Has anyone done similar experiment using inception v2 for SSD? What's your experience?
The reason of the low mAP is due to extreme low score threshold at non maximum suppression step. The result of such low threshold is that almost every image would produces more than 70 detections, while there's only one ground truth in each image. Changing this threshold to a more reasonable value -- 0.1 -- produces much better mAP plot.
You may have been bitten by this issue: https://github.com/tensorflow/models/issues/2749
as I was.
Try updating & grabbing one of the new starting pretrained files, and seeing if your problem is resolved.
Related
I am using YOLOv4-tiny for a custom dataset of 26 classes that I collected from Open Images Dataset. The dataset is almost balanced(850 images per class but different number of bounding boxes). When I used YOLOv4-tiny to train on just 3 classes the loss was near 0.5, it was fairly accurate. But for 26 classes as soon as the loss goes below 2 the model starts to overfit. The prediction are also very inaccurate.
I have tried to change the parameters like the learning rate, the momentum and the size but whatever I do the models becomes worse then before. Using regular YOLOv4 model rather then YOLO-tiny does not help either. How can I bring the loss further down?
Have you tried training with mAP? You can take a subset of your training set and make it the validation set. This can be done in the same way you made your training and test set. Then, you can run darknet.exe detector train data/obj.data yolo-obj.cfg yolov4.conv.137 -map. This will keep track of the loss in your validation set. When the error in the validation say goes up, this is the time to stop training and prevent overfitting (this is called: early stopping).
You need to run the training for (classes*2000)iterations. However, for the best scores, you need to train your model for at least 6000 iterations (also known as max_batches). Also please remember if you are using a b&w image, change the channels=3 to channels=1. You can stop your training once the avg loss becomes something like this: 0.XXXX.
Here's my mAP graph for 6000 iterations that ran for 6.2 hours:
avg loss with 6000 max_batches.
Moreover, you can follow this FAQ documentation here by Stéphane Charette.
i am working on a dataset of 368609 samples and 34 features, i wanted to use a neural network to predict latency (real value) using keras, the model has 3 hidden layers, each layer has 1024 neurons, i have used drop_out (50 %) and l2 regularization (0.001) for each hidden layer. The problem is i am getting a test mean absolute error of 3.5505 ms and train mean absolute error of 3.4528. Here, the train error is smaller than test error by a small gap, does this mean that we have an overfitting problem here ?
Not really, yet it is still always a good idea to see how your model is generalizing to new data.
Keep something between 10%-20% of your original dataset as a test set and try to predict the output for each record in the test set.
Sometimes when we deal with the same validation set for many attempts of improving our model, we tend to overfit the evaluation dataset as well.
Having 3 different datasets for training, evaluation and testing usually provides a whole solution to overfitting.
If you get a high accuracy on your training set and a low accuracy on you test set it often means that your are overfitting. So in you case - no you are probably not overfitting.
Normally you would also have a validation set, so you don't fit your data to the testset.
I am training a neural network using Tensorflow's object detetction API to detect cars. I used the following youtube video to learn and execute the process.
https://www.youtube.com/watch?v=srPndLNMMpk&t=65s
Part 1 to 6 of his series.
Now in his video, he has mentioned to stop the training when the loss value reaches ~1 or below on an average and that it would take about 10000'ish' steps.
In my case, it is 7500 steps right now and the loss values keep fluctuating from 0.6 to 1.3.
Alot of people complained in the comment section about false positives on this series but I think this happened because of the unnecessary prolonged process of training (because they din't know maybe when to stop ?) which caused overfitting!
I would like to avoid this problem. I would like to have not the most optimum weights but fairly optimum weights while avoiding false detection or overfitting. I am also observing 'Total Loss' section of Tensorboard. It fluctuates between 0.8 to 1.2. When do I stop the training process?
I would also like to know in general, which factors does the 'stopping of training' depend on? is it always about the average loss of 1 or less?
Additional information:
My training data has ~300 images
Test data ~ 20 images
Since I am using the concept of transfer learning, I chose ssd_mobilenet_v1.model.
Tensorflow version 1.9 (on CPU)
Python version 3.6
Thank you!
You should use a validation test, different from the training set and the test set.
At each epoch, you compute the loss of both training and validation set.
If the validation loss begin to increase, stop your training. You can now test your model on your test set.
The Validation set size is usually the same as the test one. For example, training set is 70% and both validation and test set are 15% each.
Also, please note that 300 images in your dataset seems not enough. You should increase it.
For your other question :
The loss is the sum of your errors, and thus, depends on the problem, and your data. A loss of 1 does not mean much in this regard. Never rely on it to stop your training.
When I train the model by xgboost and I find "eval-merror" is increasing and "train-merror" is decreasing as below;is something in error?
enter image description here
You are likely overfitting. Have you tried setting early_stopping_rounds? This will terminate the training once xgboost detects that the validation error is increasing.
If this behavior occurs from the very first training step on, you might want to try a smaller learning rate (called eta).
You can find more information on the just mentioned parameters in the api reference: http://xgboost.readthedocs.io/en/latest/python/python_api.html
I have a dataset for classification which is composed of a training of size 8000x(32x32x3 images) and of a test of size 2000x(same size images).
I am doing a very simple task of distinguishing vehicules and background. I am using the cross_entropy as a cost function.
The net I am using is almost the same as the one used in DeepMNIST except the first filter has size 3x... instead of 1x... because it is a colour image and the output has size two because there are only two classes : vehicules or not vehicules.
Seeing the results of this relatively straight forward task has led me to ask myself several interrogations :
-First if I do not use a large enough batch size (>200) I get stuck almost every time at accuracy 62% (in a local optima) over the two sets which is not sufficient for my need
-Secondly whenever I use the right optimizer Adam with the right batch size and learning rate I go up to 92% however the outputs are always very disturbingly good like [0.999999999 0.000000000001].
This should not happen as the task is difficult.
Therefore when I go fully convolutional to create a heatmap I got 1.000001 almost everywhere due to the saturation.
What am I doing wrong ? Do you think whitening would solve the problem ? Batch normalization ? Something else ? What am I facing ?
That's a sign of overfitting. If you train on small dataset long enough with a large enough model, eventually your confidences get saturated to 0's and 1's. Hence, same techniques that prevent overfitting (regularization penalties, dropout, early stopping, data augmentation) will help there.
My first step for a tiny dataset like this, would be to augment the dataset with noise corrupted examples. IE, for your example I would add 800k noise corrupted examples with original labels, and train on those.