CoreML and YOLOv3 performance issue - objective-c

Question

currently I am facing issue with performance of YOLOv3 implemented in objective-c/C++ XCode project for MacOS, however the performance is too slow. I do not have much experience with MacOS and XCode so I followed this tutorial. The execution time is around ~0.25 second.
Setup:
I run it on MacBook Pro Intel Core i5 3.1 GHz and graphic Intel Iris Plus Graphic 650 1536MB and the performance is around 4 fps. That's understandable, the GPU is not powerful one and it uses mostly CPU. Accually, it is impresive because it is faster than Pytorch implementation running on CPU. However, I run this example on MacBook pro Intel i7 2.7GHz and AMD Radeon Pro 460 and the performance is only 6 fps.
By this website the performance should be much better. Can you please let me know where I am doing mistake or it the best performance I can get with this setup? Please note that I've checked system monitor and GPU is fully used in both cases.
This is my initialisation:
//loading model
MLModel *model_ml = [[[YOLOv3 alloc] init] model];
float confidencerThreshold = 0.8;
NSMutableArray<Prediction*> *predictions = [[NSMutableArray alloc] init];
VNCoreMLModel *model = [VNCoreMLModel modelForMLModel:model_ml error:nil];
VNCoreMLRequest *request = [[VNCoreMLRequest alloc] initWithModel:model completionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error){
for(VNRecognizedObjectObservation *observation in request.results)
{
if(observation.confidence > confidencerThreshold){
CGRect rect = observation.boundingBox;
Prediction* prediction = [[Prediction alloc] initWithValues: 0 Confidence: observation.confidence BBox: rect];
[predictions addObject:prediction];
}
}
}];
request.imageCropAndScaleOption = VNImageCropAndScaleOptionScaleFill;
float ratio = height/CGFloat(width);
And my loop implementation
cv::Mat frame;
int i = 0;
while(1){
cap>>frame;
if(frame.empty()){
break;
}
image = CGImageFromCVMat(frame.clone());
VNImageRequestHandler *imageHandler = [[VNImageRequestHandler alloc] initWithCGImage:image options:nil];
NSDate *methodStart = [NSDate date]; //Measuring performance here
NSError *error = nil;
[imageHandler performRequests:#[request] error:&error]; //Call request
if(error){
NSLog(#"%#",error.localizedDescription);
}
NSDate *methodFinish = [NSDate date];
NSTimeInterval executionTime = [methodFinish timeIntervalSinceDate:methodStart]; //get execution time
// draw bounding boxes
for(Prediction *prediction in predictions){
CGRect rect = [prediction getBBox];
cv::rectangle(frame,cv::Point(rect.origin.x * width,(1 - rect.origin.y) * height),
cv::Point((rect.origin.x + rect.size.width) * width, (1 - (rect.origin.y + rect.size.height)) * height),
cv::Scalar(0,255,0), 1,8,0);
}
std::cout<<"Execution time "<<executionTime<<" sec"<<" Frame id: "<<i<<" with size "<<frame.size()<<std::endl;
[predictions removeAllObjects];
}
cap.release();
Thank you.

Answer 1

Set a breakpoint on the line that calls [imageHandler performRequests] and run the app with optimizations disabled. Use the "Step Into" button from the debugger a number of times. Look in the stacktrace for "Espresso".
Does this show something like Espresso::BNNSEngine? Then the model runs on the CPU, not the GPU.
Does the stacktrace show something like Espresso::MPSEngine? Then you're running on the GPU.
My guess is Core ML runs your model on the CPU, not on the GPU.