iOS automatic threading? - objective-c

I have been working on an opencv project for iOS. I was given a simple project to start developing with that captured and displayed frames. I never payed much attention to how it worked until I started having memory issues and traced them back to the original project setup. I now plan to re-write the capturing/displaying code but I don't understand why it worked in the first place. There was a play/pause button which called the method
- (IBAction)play_pause:(id)sender
{
play = !play;
while(play)
{
if (_videoCapture && _videoCapture->grab())
{
(*_videoCapture) >> _display_frame;
//process frame
self.imageView.image = [UIImage imageWithCVMat:_display_frame];
}
}
}
play is just a global bool that signifies whether the application is playing or paused. The strange thing is that the processing should be taking place inside an infinite loop, there is no way out. play is never modified within the loop. In spite of that, when the application is running the play/pause button remains responsive and is capable of flipping the play bool and pausing the execution. Not only that, other bools (use_greyscale for instance) can be flipped by other buttons and their values change inside the loop. I would have expected the application to freeze and never even draw new frames to the screen. The application should stay trapped inside that function for most of its lifetime, unable to perform other tasks such as drawing and UIControl. It seems as though the only way this is possible is if the IBAction call is running on its own thread. I cannot find any evidence of threading in the source code. Could someone explain how apple handles threading in its UI? I was under the impression that there was one main runloop thread and that extra threads were not created automatically. If that is true, how can this behavior be explained?
side note-
What finally made me investigate this was that [UIImage imageWithCVMat:_display_frame] returns an auto-released object. Since all this takes place inside a loop, the objects could not be released without the execution being paused which was causing crashes.

The reason is worked is because the implementation of the cv::VideoCapture::grab() method runs the current run loop to pause the thread until it gets a frame.
When you launch your application, the main function executes a function named UIApplicationMain, which executes CFRunLoopRun. When CFRunLoopRun is executed on the main thread, it runs the main run loop, which is the run loop that processes all the UI events received from the system and refresh the user interface. For information on run loops, you may read Apple Threading Programming Guide.
So, when you execute an infinite loop, your code never returns to the run loop and the waiting events cannot be processed. But in your case, the grab method runs the run loop again with an expiration delay. So the run loop may process incoming events (which may invoke your code again) until the delay expires, then return to your code that will run the run loop again.
If you look at the callstack when you touch the button to pause, you will see this:
main function → run loop → event handling → your code → OpenCV → run loop → event handling → your code
The run loop is running inside itself, which is perfectly fine because run loops are reentrant. Scroll views actually use that behavior: When you scroll a UIScrollView, it runs the run loop again in a different mode in order to ignore some events until you end scrolling.
But I'm not sure the developers of OpenCV had this in mind when they wrote their code. So I think it would be better to load your frames in a background thread/queue.

You are correct, there is no "automatic threading" in iOS. Grand Central Dispatch (GCD) certainly makes threading much easier, but it does not happen automatically.
You can write some debug code and test for [NSThread isMainThread] in the while loop to see if play_pause is indeed being run on the main UI thread, which I suspect it is not.

Related

OSX 10.10 App doesn't get focus or event loop

Restarting app development after a 15 year hiatus. Current project is conversion of old windows-type command line utility into interactive OS X windowed app.
I created a view delegate inside main window and can draw and update NSTable view.
The updates are generated in the App's main loop which takes a UDP/TCP stream, parses and updates view via appropriate delegation.
Here's the problem: When I run the app, the main window does not apparently get
focus (window control buttons on upper left are grey), the Menu created from my .xib is inert, and the window itself does not respond to resizing or to mouse hits inside the table view scroll bars. Also, the mouse pointer is the spinning beachball when over the App's window.
My guess is that I am not providing time to the Objective C run loop for event processing. I do send a "display" to my window on every iteration of my app loop, but I guess it is not sufficient (Apple is not very clear about what objects get what messages when sending this kind of update message). Am I on the right track?
Is there a way to let the system Event loop run an iteration each time through my app main loop?
Thanks!
Update: I tried explicitly providing event loop time in my app's main loop with:
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.1]];
There was no apparent change in app behavior.
My guess is that I am not providing time to the Objective C run loop for event processing.
In this case, I recommend you read about Grand Central Dispatch, which provides concurrency, allowing the GUI to remain responsive.
There's a good explanation of GCD here and whilst it looks like a large and complicated subject, you'll probably only need a few lines of code to make use of GCD. For example: -
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
// do some work
});
Despite comparing the nibs and connections between my code and a freshly created .xib prototype, i could find no reason for my code to not get focus.
The fix was to move my code guts into the project with a new .xib.
This works but is unsatisfying

How can I allow a user adjust an NSSlider without pausing the application update loop?

NOTE: Updated below...
I have a cocoa desktop application which consists of a series of controls around a custom NSView. I am using displayLink to drive the updates.
When a user clicks on an NSControl (a slider, a button, a checkbox, a radio button) the application appears to freeze until the mouse is released. I can confirm in fact that the displayLink callback (getFrameForTime) is NOT firing during the time. If I create a timer, that also does not fire, both remain paused until the user releases the mouse, at which point the application resumes updating.
The control is bound, and if I update that value from another thread (for example, via a callback from a MIDI interface) the slider behaves as expected: it moves, the value updates and the application does not pause.
I feel like this should be a fairly obvious fix, but I'm stumped.
Checking "continuous" in IB does as advertised: sends the values continuously, but still exhibits this behavior (preventing the UI update) until the mouse is released.
This seems to be related specifically to mouseDown on NSControl? Why would this block, and do I really need to subclass all my UI elements to change this behavior (seems extreme)
DisplayLink is in its own thread, so why mouseDown on the main thread block it? If this is the case, given the injunction on updating the Cocoa UI from other than the main thread, how do I deal with it?
Any help much appreciated.
Update
Per #Nikolai's comments below, I can confirm that using an NSTimer and adding it to NSEventTrackingRunLoopMode does NOT block. However, I would really like to use CVDisplayLink which (according to the documentation) runs in it's own thread and should not be blocked in this way. Unlike CADisplayLink, I cannot find a way to explicitly assign a runloop to CVDisplayLink (it seems it doesn't work that way), so perhaps the new question should be:
Why does CVDisplayLink block on NSEventTrackingRunLoopMode?
When clicking on an NSControl the runloop mode goes from NSDefaultRunLoopMode to NSEventTrackingRunLoopMode, as long as the mouse is down. That means that only run loop sources (display link) and timers fire that have been added to this mode.
You can add timers to any mode by using -[NSRunLoop addTimer:forMode:]. For a display link the equivalent method is -[CADisplayLink addToRunLoop:forMode:].
To make your animation continue during event tracking you would do something like:
[myDisplayLink addToRunLoop:[NSRunLoop currentRunLoop]
forMode:NSEventTrackingRunLoopMode];
Your test project shows that you are calling a view's display method from within the display link's callback.
When commenting the display message out, the display link is called continuously even while moving the slider.
So what goes wrong is that when the runloop goes into event tracking mode, the call to display on the display link's thread blocks until the mouse is released and the run loop goes back to default mode. You can easily confirm this by putting a log statement before the call and one after it.
Why exactly that happens is not clear to me. What is clear is that it's illegal to call a view's methods from a background thread. You have to trigger the view's display by dispatching a setNeedsDisplay: on the main thread:
static CVReturn MyDisplayLinkCallback(CVDisplayLinkRef displayLink, const CVTimeStamp* now, const CVTimeStamp* outputTime, CVOptionFlags flagsIn, CVOptionFlags* flagsOut, void* displayLinkContext)
{
dispatch_async(dispatch_get_main_queue(), ^{
[(__bridge MyCustomView*)displayLinkContext setNeedsDisplay:YES];
});
return kCVReturnSuccess;
}

How can I update the contents of a CALayer while the main thread is blocked?

An instance of AVCaptureVideoPreviewLayer continues to update its contents from the video capture stream even while the main thread is blocked. Is it possible to generally replicate this behaviour with a custom subclass of CALayer? In other words, given raw image data, can we update what is displayed on-screen while the main thread is blocked?
You can't update anything in the view when the main thread is blocked. The whole of UIKit is single-threaded and runs on the main event loop. Video capture is a special case because it draws directly to the screen buffer, but you won't be able to replicate it yourself.
Furthermore, if you do a long-running task on the main thread, iOS will assume your app has crashed and kill it after a few seconds anyway.
Why not perform your other task on a background thread instead? That's the standard practice.
I've found a way to update the UI on non-UI-Thread.
We can excute the code in any thread, and it actually changes the layer's transform even when the main thread is sleeping.
self.labelLayer.transform = CATransform3DMakeScale(1.2, 1.2, 1.0);
So if anyone can explain this, please feel free to concat me!

Interrupting a loop using NSNotification

I have a class containing a method with a loop. I need to be able to break the loop if a certain event (e.g. button press) occurs.
I am using the NSNotificationCenter to notify the class containing the loop when the button is pressed.
However, If I press the button while the loop is being executed, the notification occurs after the loop is complete instead of interrupting the loop.
I'm guessing this is because it is operating in the same thread.
So how do I get the NSNotificationCenter operating in a background / different thread? Is this possible? Or is there a better way to do it?
It's not just the notification center.
I have a class containing a method with a loop. I need to be able to break the loop if a certain event (e.g. button press) occurs.
The events for that button press come in on the main thread. If your loop is running on the main thread, then the button press itself does not get processed until your loop is finished. The notification is posted immediately, relative to the button press actually getting processed by your application.
Or, in list form:
The user presses the button.
Your loop runs out of things to do and returns.
The button press arrives in your application and is turned by the button into an action message.
You post the notification.
You receive the notification.
The delay that you're seeing is between steps 1 and 2; step 4 happens immediately after step 3.
Notifications on a local (not distributed) NSNotificationCenter are dispatched on the thread you post them from, so posting it from your action method means that it will be dispatched on the main thread. This is normal and OK.
Move the loop, not the notification, to a background thread, dispatch queue, or operation queue. If you use an operation queue, you may not need the notification at all, as you can tell an operation queue to cancel all pending operations. (Your operations will need to check at any appropriate time(s) whether they have been canceled; for reasons previously discussed, killing a thread/operation at a random time is a Bad Idea.)
Background threads, blocks, and operations can communicate back to the main thread when needed (e.g., to update the UI). To send a message through the main thread's run loop, use performSelectorOnMainThread:withObject:waitUntilDone:. To dispatch a block on the main thread, use dispatch_async and dispatch_get_main_queue. To schedule an operation on the main thread, add it to [NSOperationQueue mainQueue].
For more info, read the Concurrency Programming Guide and the Notification Programming Topics.
I would run your loop in a separate thread, and have an instance variable BOOL abort;, when your button press notification comes in, set abort = TRUE; then in the loop check this value and exit if it is true.
I would run the loop in a separate thread. Even better, make it an NSOperation so that you can call [.. cancel]. Just make sure to use performSelectorOnMainThread when updating the UI from the NSOperation object. It's not a good idea to have a long running loop on the main thread.
You can't put the notification center on another thread. That object is out of your control. The problem isn't so much that they are on the same thread as that you are not allowing the run loop, which is responsible for handling the button press, to do anything. (There's one and only one run loop per thread.) As has been stated by both edsko and Peter Hosey, the button press itself, and in fact your entire UI, is stopped while your loop is running. It is generally a good idea to put long-running operations onto a background thread, then call back to the main thread to update the UI, performSelectorOnMainThread:withObject:waitUntilDone: being an easy way to do such a call back.
That said, if you were to keep the loop on the main thread, you need to let control return to the run loop periodically so that the button press will be registered. There are two ways I can think of to do this. First, you can explicitly give the run loop control briefly during each iteration of your loop:
while( !buttonWasPressed ){
// Do work...
// Let the run loop do some processing before the next iteration.
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.01]];
}
Or, you can make a single method that consists only of the code from your loop, and use performSelector:withObject:afterDelay: to have the method repeatedly called while still allowing the run loop to work:
- (void) loopTheLoop {
if( buttonWasPressed ) return;
// Do work...
// Run this method again as soon as possible.
[self performSelector:#selector(loopTheLoop)
withObject:nil
afterDelay:0.0];
}

Is there a function for "processing messages", i.e. updating screen, responding to accumulated user input

I have encountered a strange bug in my application, and I am trying to debug it using step execution.
However it seems that things on an iphone often do not happen as synchronously as I would like, for example when I step-over this line
[self.view addSubview:FinndomoEmbeddedMWView.view];
nothing happens in the emulator.
If I just let the program run, the view is added and the screen changes as it should.
So I am guessing, addSubview does not do everything related to adding a view, it just sort of starts the process, and then it is completed later.
I don't know if there are message queues on ios similair to winapi, but there must be something like that, so is there a function for "processing all accumulated messages". I would then like to call this function after my addSubview and actually see the things change in the emulator while I debug, not when the program is running.
And I experience this not only with addSubview, so I want to have a general solution for things like this.
Sat yesterday and found out the answer:
NSDate *dtr = [[NSDate alloc] initWithTimeIntervalSinceNow:0.5];
[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:dtr];
This is sort of what I wanted. As you can see, this particular code may end up waiting for 0.5 seconds if there are no more events, and if there are events pending, it is only processing the first one and then returns (according to documentation).
But that could be avoided of course, this is just an example, if anyone will want the same thing. In my case there was only 1 important event, and so the provided snippet worked.
The short answer is no.
Cocoa is event-driven.
The core of each app is the event loop. On each pass through the event loop, the app handles events added to the event queue. Updating the screen is one such event.
Thus, changes to screen display don't take place until after your code returns, on the next pass through the event loop.
For debugging purposes where you want to figure out what's happening line-by-line, you need to either use the debugger's facilities, or add NSLog statements to your code.