I have trained an object-detection model in Tensorflow 2 with EfficientDet. And now I am trying to evaluate the model performance on test dataset. I ran below command to evaluate model -
python model_main_tf2.py \
--model_dir=path/to/model_dir \
--pipeline_config_path=path/to/model_dir/pipeline.config \
--checkpoint_dir=path/to/checkpoint_dir
And I can see below results -
Here I can see Average-Precision for IoU=0.50 (second line) and IoU=0.75 (third line).
Now, is there any way that I can see Average-Precision for IoU=0.8 and IoU=0.9 too?
Also, can I see Average-Recall for IoU=0.8 and IoU=0.9 too?
Related
I am trying to do the evaluation of a trained SSD_Mobilenetv2 320x320 fpnlite on tensorflow. I ran training and evaluation parallel in two different colabs account. But I am always getting the metrics values as -1(evaluation result) after each checkpoint. The lose is also increasing after evaluating each checkpoint.
Below is the command used to run the evaluation:
!python /model_main_tf2.py \
--pipeline_config_path /pipeline.config \
--model_dir /model_ \
--checkpoint_dir /model_
I am training MobileNet_v1_1.0_224 using TensorFlow. I am using the python scripts present in the TensorFlow-Slim image classification model library for training. My dataset distribution with 4 classes is as follows:
normal_faces: 42070
oncall_faces: 13563 (People faces with mobile in the image when they're on call)
smoking_faces: 5949
yawning_faces: 1630
All images in the dataset are square images and larger than 224x224
I am using train_image_classifier.py to train the model with following arguments,
python train_image_classifier.py \
--train_dir=${TRAIN_DIR} \
--dataset_name=custom \
--dataset_split_name=train \
--dataset_dir=${DATASET_DIR} \
--model_name=mobilenet_v1 \
--batch_size=32\
--max_number_of_steps=25000
After training the model, eval_image_classifier.py shows an accuracy greater than 95% on Validation set but when I exported the frozen graph and used it for predictions, it performs very poorly.
I have also tried this notebook but this also produced similar results.
Log: Training Log
Plots: Loss and Accuracy
What is the reason for this? How do I fix this issue?
I have seen similar issues on SO but nothing related to MobileNets specifically.
Did you use a validation set? If so what was the validation accuracy?
If you used a validation set a good way to check if you are doing predictions properly is to run model.predict on the validation set.
When model of faster_rcnn_resnet101 trains, losses are shown on terminal each step.
I want to know which data is input each step. when loss increases, i don't know why loss increases.
Is there someone knowing how to see input data each step?
You can't check each step result ,but in your object_detection/training directory trained checkpoint prefix is created which updating on after specific no of step.
you can check object detection using current trained model.
Eg:
python3 export_inference_graph.py \
--input_type image_tensor \
--pipeline_config_path training/ssd_mobilenet_v1_pets.config \
--trained_checkpoint_prefix training/model.ckpt-25000 \
--output_directory latest_dataset
here model.ckpt-25000 is the no of steps(25000) trained till now.
I used the ssd_mobilenet_v1_coco from detection model zoo in tensorflow object detection. I am currently training the model by running
python legacy/train.py --logtostderr --train_dir=trainingmobile/ --pipeline_config_path=trainingmobile/pipeline.config
I want to run an evaluation job by running eval.py to get other metrics like IOU and PR Curve but I don't know how to do that. I am able to run the command
python legacy/eval.py \
--logtostderr \
--checkpoint_dir= path/to/checkpoint \
--eval_dir= path/to/eval \
--pipeline_config_path= path/to/config
then I ran the command
tensorboard --logdir=path/to/eval
The tensorboard shows only the test image output. How can i get other metrics like IOU and PR Curve?
First of all, I'd highly recommend you to use the newer model_main.py script for training and evaluation combined. You can use it as shown below:
python object_detection/model_main.py \
--pipeline_config_path=path/to/config \
--model_dir=path/to/train_dir \
--num_train_steps=NUM_TRAIN_STEPS \
--num_eval_steps=NUM_EVAL_STEPS \
--alsologtostderr
It combines training and evaluation and you can enter tensorboard with
tensorboard -logdir=path/to/train_dir
Tensorboard will not only disply the training process, it will also show your progress over your validation set. They use the COCO metric as default metric!
To your original problem: Maybe you should change the eval settings in your config file to larger numbers:
eval_config: {
num_examples: 8000
# Note: The below line limits the evaluation process to 10 evaluations.
# Remove the below line to evaluate indefinitely.
max_evals: 10}
If you'll use the model_main.py script, the number of evaluation will be set by the flags.
Good to know: The info output of tnesorflow is disabled in the newer model_main.py script. You can enable it by adding
tf.logging.set_verbosity(tf.logging.INFO)
after the import section.
Is it possible to train the current deeplab model in TensorFlow to reasonable accuracy using 4 GPUs with 11GB? I seem to be able to fit 2 batches per GPU, so am running a total batch size of 8 across 4 clones.
Following the instructions included with the model, I get a mean IoU of < 30% after 90,000 iterations.
PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim python deeplab/train.py \
--logtostderr --training_number_of_steps=90000 \
--train_split="train" --model_variant="xception_65" \
--atrous_rates=6 --atrous_rates=12 --atrous_rates=18 \
--output_stride=16 --decoder_output_stride=4 --train_crop_size=769 \
--train_crop_size=769 --train_batch_size=8 --num_clones=4 \
--dataset="cityscapes" \
--tf_initial_checkpoint=deeplab/models/xception/model.ckpt \
--train_logdir=$LOGDIR \
--dataset_dir=deeplab/datasets/cityscapes/tfrecord
I have tried with batch norm both enabled and disabled without much difference in outcome.
Thanks!
It seems I needed a much larger step length than the default. 1e-2 gives results closer to the published results, with batch size 15 and a smaller crop window size.
if you check this link https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
It has links to pretrained models for MobileNet v2 and DeepLab trained on Cityscapes. You can modify the existing shell scripts present here to train on cityscapes.