DataFolder class not detecting training samples - python-bob

Question

I am trying to reproduce the experiments in the paper Cross Modal Focal Loss for RGBD Face Anti-Spoofing (https://arxiv.org/pdf/2103.00948.pdf) . I've pointed my preprocessed directory to point to the mc-pixbis-224 preprocessed data in order to train the RGBDMH - CMFL model .I've selected to train the grandtest protocol and pointed the annotations directory to PROTOCOL-grand_test-curated.csv file . However , my DataFolder class fails to load any training samples as the length of dataset when printed is 0 .
Traceback (most recent call last):
File "bin/train_generic.py", line 22, in <module>
sys.exit(bob.learn.pytorch.scripts.train_generic.main())
File "/home/hazeeq/anaconda3/envs/bob.paper.cross_modal_focal_loss_cvpr2021/lib/python3.7/site-packages/bob/learn/pytorch/scripts/train_generic.py", line 150, in main
shuffle=True,
File "/home/hazeeq/anaconda3/envs/bob.paper.cross_modal_focal_loss_cvpr2021/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 262, in __init__
sampler = RandomSampler(dataset, generator=generator) # type: ignore
File "/home/hazeeq/anaconda3/envs/bob.paper.cross_modal_focal_loss_cvpr2021/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 104, in __init__
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
(bob.paper.cross_modal_focal_loss_cvpr2021) hazeeq#hazeeq-U3033:~/test/bob.paper.cross_modal_focal_loss_cvpr2021$
Line 152 of train_generic.py refers to this section of the code where dataloader["train"] fails to be loaded with the proper DataLoader object within the 'else' statement:
# Which device to use is figured out at this point, no need to use `use-gpu` flag anymore
# get data
if hasattr(configuration, "dataset"):
dataloader = {}
if not do_crossvalidation:
logger.info(
"There are {} training samples".format(
len(configuration.dataset["train"])
)
)
dataloader["train"] = torch.utils.data.DataLoader(
configuration.dataset["train"],
batch_size=batch_size,
num_workers=num_workers,
shuffle=True,
)
else:
dataloader["train"] = torch.utils.data.DataLoader(
configuration.dataset["train"],
batch_size=batch_size,
num_workers=num_workers,
shuffle=True,
)
dataloader["val"] = torch.utils.data.DataLoader(
configuration.dataset["val"],
batch_size=batch_size,
num_workers=num_workers,
shuffle=True,
)
logger.info(
"There are {} training samples".format(
len(configuration.dataset["train"])
)
)
logger.info(
"There are {} validation samples".format(
len(configuration.dataset["val"])
)
)
else:
logger.error("Please provide a dataset in your configuration file !")
sys.exit()
assert hasattr(configuration, "optimizer")
# train the network
if hasattr(configuration, "network"):
trainer = GenericTrainer(
configuration.network,
configuration.optimizer,
configuration.compute_loss,
learning_rate=learning_rate,
device=device,
verbosity_level=verbosity_level,
tf_logdir=output_dir + "/tf_logs",
do_crossvalidation=do_crossvalidation,
save_interval=save_interval,
)
trainer.train(dataloader, n_epochs=epochs, output_dir=output_dir, model=model)
else:
logger.error("Please provide a network in your configuration file !")
sys.exit()
The code also reports multiple missing files , so I am not sure if there are any missing files that should be part of the MC-pixbis-224 preprocessed data . Here I have attached some of the missing file prompts , there are more missing files than the ones showed below .
...............................HLDI self.annotation_directory ./hqwmca-protocols-csv/PROTOCOL-grand_test-curated.csv
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/02.04.19/1_03_0064_0000_06_01_013-e3a1456b.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/01.04.19/1_03_0001_0000_07_00_001-c8bd4c01.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/02.04.19/1_03_0001_0000_06_01_001-48c7d79c.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/11.03.19/1_03_0523_0018_08_00_004-315ad7b2.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/02.04.19/1_03_0002_0000_06_01_002-173e70ed.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/11.10.19/1_01_0002_0000_00_00_000-51e86383.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/11.10.19/1_01_0002_0000_00_00_000-7517b634.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/07.10.19/1_01_0077_0000_00_00_000-9f7b92f8.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/07.10.19/1_01_0077_0000_00_00_000-d416451d.hdf5
Missing file: /home/Dataset/FaceAntiSpoofing/HQ-WMCA/MC-PixBiS-224/preprocessed/face-station/11.10.19/1_01_0084_0000_00_00_000-305a3a31.hdf5

Answer 1

The CSV files are not annotations, they are just showing the distribution of files in different folds in each of the protocols if you want to use the datasets outside bob. Moreover, the MC-pixbis-224 preprocessed files do not correspond to the RGB-D data, it corresponds to an earlier paper. For RGB-D data you have to access the RAW data, and preprocess the data using the documentation.
Regards,
Anjith