Tensorflow: training and evaluation process - tensorflow

Question

I am using Tensorflow to train my data-set (with Object-detection API) locally with 1080 Nvidia 8GB,
I use create_pet_tf_record.py to generate TFRecords files. I don't train from scratch I use mask_rcnn_inception_v2_coco_2018_01_28/model.ckpt as a fine_tune_checkpoint.
When I run python object_detection/train.py and /eval.py, I check the training and evaluation process thru Tensorboard. Initially, everything seems correct like this pic1 with zero step.
The training checkpoint interval takes long time to be saved. After more than 5,000 training steps, the evaluation moved from /model.ckpt-0 to /model.ckpt-3642 and the whole process will be NOT okay at this moment as shown in this pic2.
This is my file mask_rcnn_inception_v2.config
model {
faster_rcnn {
num_classes: 1
image_resizer {
fixed_shape_resizer {
height: 375
width: 500
}
}
number_of_stages: 3
feature_extractor {
type: 'faster_rcnn_inception_v2'
first_stage_features_stride: 16
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5, 1.0, 2.0]
aspect_ratios: [0.5, 1.0, 2.0]
height_stride: 16
width_stride: 16
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.7
first_stage_max_proposals: 300
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
use_dropout: false
dropout_keep_probability: 1.0
predict_instance_masks: true
mask_height: 15
mask_width: 15
mask_prediction_conv_depth: 0
mask_prediction_num_conv_layers: 2
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.0
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 300
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
second_stage_mask_prediction_loss_weight: 4.0
}
}
train_config: {
batch_size: 1
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0002
schedule {
step: 900000
learning_rate: .00002
}
schedule {
step: 1200000
learning_rate: .000002
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
gradient_clipping_by_norm: 10.0
fine_tune_checkpoint: "/home/jesse/gpu-py3/models/research/object_detection/models/model/mask_rcnn_inception_v2_coco_train/mask_rcnn_inception_v2_coco_2018_01_28/model.ckpt"
from_detection_checkpoint: true
# Note: The below line limits the training process to 200K steps, which we
# empirically found to be sufficient enough to train the pets dataset. This
# effectively bypasses the learning rate schedule (the learning rate will
# never decay). Remove the below line to train indefinitely.
num_steps: 200000
data_augmentation_options {
random_horizontal_flip {
}
}
}
train_input_reader: {
tf_record_input_reader {
input_path: "/home/jesse/gpu-py3/models/research/ttt/pet_train.record"
}
label_map_path: "/home/jesse/gpu-py3/models/research/object_detection/data/pet_label_map.pbtxt"
load_instance_masks: true
mask_type: PNG_MASKS
}
eval_config: {
num_examples: 8000
# Note: The below line limits the evaluation process to 10 evaluations.
# Remove the below line to evaluate indefinitely.
max_evals: 10
}
eval_input_reader: {
tf_record_input_reader {
input_path: "/home/jesse/gpu-py3/models/research/ttt/pet_val.record"
}
label_map_path: "/home/jesse/gpu-py3/models/research/object_detection/data/pet_label_map.pbtxt"
load_instance_masks: true
mask_type: PNG_MASKS
shuffle: false
num_readers: 1
}
I don't know where I am mistaken, I feel like I should run evaluation more often, and the training checkpoint should be saved every 2000 steps for example. Or I may need to edit the pipeline file mask_rcnn_inception_v2.config. I don't know why the training result is very disappointed after 3642 steps as seen in pic2.
Any help is highly appreciated

Answer 1

My 2 cents on this, assuming you have not modified the important config parameters much, your training data is very diverse and as more iterations go through its generalising. Try with more accurate labelling of images, even if it means fewer images.