Training an object detector on dataset layers of increasing ambiguity - tensorflow

Question

I have a number of Raspberry Pi cameras focused on bird feeders,
continually running a TensorFlow Object Detection graph (SSD MNet2) to detect birds.
Over time I've built a dataset of +10k image over 11 species, retraining the graph frequently.
I intend to cap the number of items in the dataset to 10k items (perhaps arbitrarily).
There is a flow of data through the dataset so that it continually improves.
New candidate detections are triaged by a judge (me) as follows:
Add as new training/evaluation item.
The detection is judged representative of a category.
After adjustment, the image and detection can be added to the ground truth
Add as counter example item.
The detection is false, but can be converted to an unclassified counter example.
After adjustment, the image and detection can be added to the ground truth
Discard item
Not useful for training.
Also note that some existing data is retired when sufficient better data is available.
To date, all the items in the ground truth are delivered to training with a weight of 1.0.
See: https://github.com/tensorflow/models/blob/master/research/object_detection/data_decoders/tf_example_decoder.py
def default_groundtruth_weights():
return tf.ones(
[tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0]],
dtype=tf.float32)
But this is obviously not very true.
I know by inspection that some of the items are not so good, but at any one time they're the best examples available.
Over time, eventually, bad items get replaced with better items.
Ranked training records
I have wondered about the impact on training and whether the situation would be improved by ranking the dataset, by some value of ideality,
and then training in successive ranks, so that the model initialises on the most ideal data and subsequently learns less and less ideal data.
What I'm imagining trying to avoid is the model paying too much attention to bad data and not enough to good data, especially during the initial epochs of training.
Where bad and good data mean how well the data items contribute to the veracity and visualization (via Lucid) of the trained models.
Weighted Dataset
Setting a weight (between 0 and 1) on an item means the loss calculated for that item is reduced (by the weight factor);
I assume it means "pay less attention to this item by this much".
See: Class weights for balancing data in TensorFlow Object Detection API
I've visited every item in my dataset to retrospectively set a weight.
I did this by running some recent models over the dataset images (admittedly, from which the models were trained) and then matching detections.
The weight given to each item was calculated by averaging scores from the model detections (and rounding to one decimal place to make bands).
The entire dataset was then reviewed to increase or reduce weights as judged necessary.
The results are shown in the following table:
| Class\Weight Bin | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | Total|
| blackbird | | | 34 | 84 | 212 | 305 | 115 | 750 |
| blue tit | | | 47 | 94 | 211 | 435 | 241 | 1028 |
| collared dove | | | 17 | 52 | 236 | 302 | 101 | 708 |
| dunnock | | | 50 | 140 | 260 | 236 | 228 | 914 |
| goldfinch | | | 60 | 103 | 220 | 392 | 164 | 939 |
| great tit | | 35 | 42 | 71 | 234 | 384 | 201 | 967 |
| mouse | 40 | 29 | 35 | 50 | 87 | 142 | | 383 |
| robin | | 43 | 44 | 97 | 175 | 207 | 52 | 618 |
| sparrow | | 31 | 51 | 75 | 278 | 475 | 220 | 1130 |
| starling | | 19 | 28 | 39 | 97 | 227 | 73 | 483 |
| wood pigeon | | | 10 | 34 | 82 | 265 | 560 | 951 |
| | | | | | | | | 8871 |
The first training results look promising, in that the model is training well.
But I haven't reviewed the visualizations yet.
Is setting an appropriate weight on each dataset item equivalent to layering the delivery of ranked training records?

Answer 1

First, try removing ambiguous data from the dataset and train the model, and compare its results with previous model.
If that does not help, then go with class weights for balancing data.