Training my own dataset on deeplab but inference all black - tensorflow

Question

I try to train my own dataset on deeplab model in TensorFlow model garden, I could get a decreasing loss result through time, I using pre-train model provided by official repo.
But when I try to vis with latest checkpoint or try to freeze the model to .pb and do inference, outcome nothing but the black image( I check these images with NumPy all pixels are 0).
My training script like this:
python deeplab/train.py \
--logtostderr \
--num_clones=1 \
--training_number_of_steps=500000 \
--train_split="train" \
--model_variant="xception_65" \
--atrous_rates=6 \
--atrous_rates=12 \
--atrous_rates=18 \
--output_stride=16 \
--decoder_output_stride=4 \
--train_crop_size="513,513" \
--train_batch_size=2 \
--dataset={$OWN_DATASET} \
--train_logdir={$TRAIN_LOGDIR} \
--dataset_dir={$DATASET_DIR} \
--tf_initial_checkpoint={$INITIAL_CHECKPOINT}
did anyone happen before?

Answer 1

This is an old thread and I don't know if you still need help but you haven't provided much information regarding your dataset. Here are some general pointers:
Try setting these flags in train.py
--fine_tune_batch_norm=False \
--initialize_last_layer=False \
--last_layers_contain_logits_only=True \
Make sure your SegmentationClassRaw folder label masks are 0, 1, 2, 3... where 0 is the background and 1, 2, 3, ... are individual classes. Run "asarray(image)" to see these pixels and make sure the label is correct.
If you have an unbalanced dataset, you can try setting the weights for different labels in through train.py.
--label_weights=1 \ # Weight for label 0 (Background)
--label_weights=10 \ #Weight for label 1 (Object class 1)
--label_weights=15 \ #Weight for label 2 (Object class 2)
If all fails, try a larger dataset. A dataset size of 225 images and 2000 steps (with an initial mobilenetv2 checkpoint) yielded results for me, although the accuracy/performance was not very good since the dataset size was small. Just as a reference point, the loss of this small dataset was around 0.05-0.06 after 2000 steps.