An instance of AVCaptureVideoPreviewLayer continues to update its contents from the video capture stream even while the main thread is blocked. Is it possible to generally replicate this behaviour with a custom subclass of CALayer? In other words, given raw image data, can we update what is displayed on-screen while the main thread is blocked?
You can't update anything in the view when the main thread is blocked. The whole of UIKit is single-threaded and runs on the main event loop. Video capture is a special case because it draws directly to the screen buffer, but you won't be able to replicate it yourself.
Furthermore, if you do a long-running task on the main thread, iOS will assume your app has crashed and kill it after a few seconds anyway.
Why not perform your other task on a background thread instead? That's the standard practice.
I've found a way to update the UI on non-UI-Thread.
We can excute the code in any thread, and it actually changes the layer's transform even when the main thread is sleeping.
self.labelLayer.transform = CATransform3DMakeScale(1.2, 1.2, 1.0);
So if anyone can explain this, please feel free to concat me!
Related
NOTE: Updated below...
I have a cocoa desktop application which consists of a series of controls around a custom NSView. I am using displayLink to drive the updates.
When a user clicks on an NSControl (a slider, a button, a checkbox, a radio button) the application appears to freeze until the mouse is released. I can confirm in fact that the displayLink callback (getFrameForTime) is NOT firing during the time. If I create a timer, that also does not fire, both remain paused until the user releases the mouse, at which point the application resumes updating.
The control is bound, and if I update that value from another thread (for example, via a callback from a MIDI interface) the slider behaves as expected: it moves, the value updates and the application does not pause.
I feel like this should be a fairly obvious fix, but I'm stumped.
Checking "continuous" in IB does as advertised: sends the values continuously, but still exhibits this behavior (preventing the UI update) until the mouse is released.
This seems to be related specifically to mouseDown on NSControl? Why would this block, and do I really need to subclass all my UI elements to change this behavior (seems extreme)
DisplayLink is in its own thread, so why mouseDown on the main thread block it? If this is the case, given the injunction on updating the Cocoa UI from other than the main thread, how do I deal with it?
Any help much appreciated.
Update
Per #Nikolai's comments below, I can confirm that using an NSTimer and adding it to NSEventTrackingRunLoopMode does NOT block. However, I would really like to use CVDisplayLink which (according to the documentation) runs in it's own thread and should not be blocked in this way. Unlike CADisplayLink, I cannot find a way to explicitly assign a runloop to CVDisplayLink (it seems it doesn't work that way), so perhaps the new question should be:
Why does CVDisplayLink block on NSEventTrackingRunLoopMode?
When clicking on an NSControl the runloop mode goes from NSDefaultRunLoopMode to NSEventTrackingRunLoopMode, as long as the mouse is down. That means that only run loop sources (display link) and timers fire that have been added to this mode.
You can add timers to any mode by using -[NSRunLoop addTimer:forMode:]. For a display link the equivalent method is -[CADisplayLink addToRunLoop:forMode:].
To make your animation continue during event tracking you would do something like:
[myDisplayLink addToRunLoop:[NSRunLoop currentRunLoop]
forMode:NSEventTrackingRunLoopMode];
Your test project shows that you are calling a view's display method from within the display link's callback.
When commenting the display message out, the display link is called continuously even while moving the slider.
So what goes wrong is that when the runloop goes into event tracking mode, the call to display on the display link's thread blocks until the mouse is released and the run loop goes back to default mode. You can easily confirm this by putting a log statement before the call and one after it.
Why exactly that happens is not clear to me. What is clear is that it's illegal to call a view's methods from a background thread. You have to trigger the view's display by dispatching a setNeedsDisplay: on the main thread:
static CVReturn MyDisplayLinkCallback(CVDisplayLinkRef displayLink, const CVTimeStamp* now, const CVTimeStamp* outputTime, CVOptionFlags flagsIn, CVOptionFlags* flagsOut, void* displayLinkContext)
{
dispatch_async(dispatch_get_main_queue(), ^{
[(__bridge MyCustomView*)displayLinkContext setNeedsDisplay:YES];
});
return kCVReturnSuccess;
}
I have been working on an opencv project for iOS. I was given a simple project to start developing with that captured and displayed frames. I never payed much attention to how it worked until I started having memory issues and traced them back to the original project setup. I now plan to re-write the capturing/displaying code but I don't understand why it worked in the first place. There was a play/pause button which called the method
- (IBAction)play_pause:(id)sender
{
play = !play;
while(play)
{
if (_videoCapture && _videoCapture->grab())
{
(*_videoCapture) >> _display_frame;
//process frame
self.imageView.image = [UIImage imageWithCVMat:_display_frame];
}
}
}
play is just a global bool that signifies whether the application is playing or paused. The strange thing is that the processing should be taking place inside an infinite loop, there is no way out. play is never modified within the loop. In spite of that, when the application is running the play/pause button remains responsive and is capable of flipping the play bool and pausing the execution. Not only that, other bools (use_greyscale for instance) can be flipped by other buttons and their values change inside the loop. I would have expected the application to freeze and never even draw new frames to the screen. The application should stay trapped inside that function for most of its lifetime, unable to perform other tasks such as drawing and UIControl. It seems as though the only way this is possible is if the IBAction call is running on its own thread. I cannot find any evidence of threading in the source code. Could someone explain how apple handles threading in its UI? I was under the impression that there was one main runloop thread and that extra threads were not created automatically. If that is true, how can this behavior be explained?
side note-
What finally made me investigate this was that [UIImage imageWithCVMat:_display_frame] returns an auto-released object. Since all this takes place inside a loop, the objects could not be released without the execution being paused which was causing crashes.
The reason is worked is because the implementation of the cv::VideoCapture::grab() method runs the current run loop to pause the thread until it gets a frame.
When you launch your application, the main function executes a function named UIApplicationMain, which executes CFRunLoopRun. When CFRunLoopRun is executed on the main thread, it runs the main run loop, which is the run loop that processes all the UI events received from the system and refresh the user interface. For information on run loops, you may read Apple Threading Programming Guide.
So, when you execute an infinite loop, your code never returns to the run loop and the waiting events cannot be processed. But in your case, the grab method runs the run loop again with an expiration delay. So the run loop may process incoming events (which may invoke your code again) until the delay expires, then return to your code that will run the run loop again.
If you look at the callstack when you touch the button to pause, you will see this:
main function → run loop → event handling → your code → OpenCV → run loop → event handling → your code
The run loop is running inside itself, which is perfectly fine because run loops are reentrant. Scroll views actually use that behavior: When you scroll a UIScrollView, it runs the run loop again in a different mode in order to ignore some events until you end scrolling.
But I'm not sure the developers of OpenCV had this in mind when they wrote their code. So I think it would be better to load your frames in a background thread/queue.
You are correct, there is no "automatic threading" in iOS. Grand Central Dispatch (GCD) certainly makes threading much easier, but it does not happen automatically.
You can write some debug code and test for [NSThread isMainThread] in the while loop to see if play_pause is indeed being run on the main UI thread, which I suspect it is not.
I have an app where I want to have an image animation while I am reading some info from a database and building object. I have used UIImageView and set up and array of images, but if I start the animation and then do my DB processing, the animation does not play.
Is there another way to start the animation, or for me to do processing during the animation?
Thanks
It sounds like you're doing your processing on the main thread, which is preventing your animations from running.
Animations run on the main thread, so to avoid blocking this thread processing should be scheduled on a different thread.
You can achieve this using blocks like so:
dispatch_queue_t queue;
queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
// Your processing to be performed on this thread.
});
Or on earlier iOS versions, like so:
[self performSelectorInBackground:#selector(yourProcessing) withObject:nil];
- (void)yourProcessing {
// Your processing to be performed on this thread.
}
I highly recommend a read through of the Threading Programming Guide, followed by watching the WWDC sessions covering Blocks and Grand Central Dispatch (WWDC 2009).
You can run your DB processing on the background thread after you begin the animation. This will allow the two to happen simultaneously.
Sounds like you’re trying to do your database processing on the main thread, which, yes, will block your UI so it can’t animate (and so the user can’t interact with anything). Take a look at the Concurrency Programming Guide.
There is a NSCollectionView in my Mac application, and it contains some icons. Meanwhile, I am using NSOperationQueue as a task scheduler to queue some operations. In one NSOperation, I modify the Content of NSCollectionView, the view should update at once when Content changes.
If I update it in the main thread, then it works perfectly well. If I put the same source code at the end of the NSOperation::main(), then the NSCollectionView would be completely blank.
Is this some kind of bug or I do it in the wrong way?
Thanks!
All interaction with UI elements must be performed on the main thread.
You can use the performSelectorOnMainThread:withObject:waitUntilDone: method to help you update UI elements from background threads.
I have started to implement cvDisplayLink to drive the render loop instead of nstimer, as detailed in this technical note https://developer.apple.com/library/archive/qa/qa1385/_index.html
Is it better to do the actual rendering within the displaylink callback itself, or should I instead call setNeedsDisplay on the view and let the main thread do the rendering?
In order to render directly in the callback, I need to lock the opengl context, which I was hoping to avoid.
Is it possible to use cvDisplayLink for timing and still render on the main thread?
Thanks in advance for any feedback....
There’s no intrinsic problem with locking the context; it should be uncontended and unlock quickly. Redrawing on the main thread, via -setNeedsDisplay:, has a chance of missing the VBL window just because of the runloop round-trip.