I am just wondering at what exact moment a completionBlock is executed on a NSOperation owned by a NSOperationQueue.
On my newest project, a client for Amazon S3 (https://github.com/StudioIstanbul/SIAFAWSClient), I use a NSOperation with a completionBlock for requests to Amazon REST API. The client is able to monitor the status of all scheduled requests via a property on the main class called isBusy. In my operation's completion block I set the value for this property to NO if there are no other operations scheduled in my NSOperationQueue. I now figured out that in some rare cases my current NSOperation is still included in my NSOperationQueue when the completionBlock gets called. This seems a little strange to me. I ended up checking for existence of the current NSOperation in my queue to fix this, but this seems unnecessary from a design point of view.
__weak AWSOperation* thisOperation = operation;
[operation setCompletionBlock:^{
if (self.operationQueue.operationCount <= 0
|| (self.operationQueue.operationCount == 1
&& [self.operationQueue.operations objectAtIndex:0] == thisOperation)) {
[self willChangeValueForKey:#"isBusy"];
_isBusy = NO;
[self didChangeValueForKey:#"isBusy"];
}
}];
Does anybody have more information on this behavior?
From the docs:
The exact execution context for your completion block is not guaranteed but is typically a secondary thread. Therefore, you should not use this block to do any work that requires a very specific execution context.
So, you can't guarantee that it will be called before or after the operation is removed from the queue. It's sent after the trigger to remove the operation from the queue (because the operation is finished) but there is effectively a race between those 2 things.
Your best option is to push your consideration to the next iteration of the main runloop if you're interrogating the operation queue itself to determine the state.
Related
How do I prevent from a dispatch_group from getting stuck? I have found to be possible to get stuck in the following code (with or without the dispatch_group_wait call) if one of the images I attempt to load is not loaded (e.g. due to bad url). The block in dispatch_group_notify is never called.
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0);
dispatch_group_t group = dispatch_group_create();
for (...) {
if (...) {
dispatch_group_enter(group);
dispatch_async(queue, ^{
[self loadImageWithUrl:url onCompletion:^{
dispatch_group_leave(group);
}];
});
}
}
dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, (int64_t)(2.0 * NSEC_PER_SEC)));
dispatch_group_notify(group, queue, ^{
NSLog(#"load image complete");
});
dispatch_group_notify queues its block when the group is complete. Your group never completes. So don't use dispatch_group_notify. Just use dispatch_group_wait as you are to wait with a timeout, then dispatch your block:
...
dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, (int64_t)(2.0 * NSEC_PER_SEC)));
dispatch_async(queue, ^{
NSLog(#"load image complete");
});
If you want to mimic a dispatch_group_notify with a timeout, just do the above in its own async block:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, (int64_t)(2.0 * NSEC_PER_SEC)));
dispatch_sync(queue, ^{
NSLog(#"load image complete");
});
});
Note that you can use the return value of dispatch_group_wait to determine if everything completed or if it timed-out if that is useful information to you.
Keep in mind that the previous blocks will not be cancelled, so they may eventually run their completion blocks. You may need to add cancellation logic to the system if that's a problem.
I do not think the issue is with your group notify process. For me, the issue that leaps out at me is that, rather than trying to handle the scenario where the completion block is not called, that you change loadImageWithUrl to ensure that it always calls the completion block, whether successful or not. You might even want to add a NSError parameter to the block or something like that, so the caller will be notified if something failed (for example to warn the user, or initiate Reachability process that will wait for the connection to be re-established before attempting a retry, etc).
So, it might look like:
- (void)loadImageWithUrl:(NSURL *)url loadImageWithUrl:(void (^)(NSError *error))block
{
BOOL success;
NSError *error;
// do your download, setting `success` and `error` appropriately
// then, when done, call the completion block, whether successful or not
if (block) {
if (success) {
block(nil);
} else {
block(error);
}
}
}
Clearly, the details of the above are entirely dependent upon how you're doing these requests, but that's the basic idea. Then, you just make sure that your caller is changed to include this extra parameter:
for (...) {
if (...) {
dispatch_group_enter(group);
dispatch_async(queue, ^{
[self loadImageWithUrl:url onCompletion:^(NSError *error){
if (error) {
// handle the error however you want, if you want
}
dispatch_group_leave(group);
}];
});
}
}
I care less about how you choose to handle the error than I do in encouraging you ensure your completion block is called regardless of whether the download was successful or not. This ensures that the number of times you enter the group is perfectly balanced with the number of times you leave the group.
Having said that, when downloading many resources, GCD is ill-suited for this task. The issue is that it's non-trivial to constrain GCD to how many concurrent tasks can be performed at one time. Generally, you want to constrain how many requests that can run concurrently. You do this because (a) there's a limit as to how many NSURLSessionTask or NSURLConnection requests can run concurrently anyway; (b) if you run more than that, on slow connections you run serious risk of requests timing-out unnecessarily; (c) you can reduce your app's peak memory usage; but (d) you still enjoy concurrency, striking a balance between memory usage and optimal network bandwidth optimization.
To accomplish this, a common solution is to use operation queues rather than GCD's dispatch queues. You can then wrap your download requests in NSOperation objects and add these network operation to a NSOperationQueue for which you have set some reasonable maxConcurrentOperationCount (e.g. 4 or 5). And instead of a dispatch group notify, you can add a completion operation which is dependent upon the other operations you've added to your queue.
If you don't want to implement this yourself, you can use AFNetworking or SDWebImage, which can facilitate the downloading of images using operation queues to manage the download process.
And one final thought is that many apps adopt a lazy loading process, where images are seamlessly loaded as they're needed. It avoids consuming too much of the user's data plan performing some bulk download (or risking that the image the user needs first is backlogged behind a bunch of other images they don't immediately need). Both AFNetworking and SDWebImage offer UIImageView categories that offer an incredibly simple lazy loading of images.
Would it be possible to do a synchronous load of the image in the inner blocks? That way you could use dispatch_group_async() instead of the manually keeping track of the enter/leave paradigm.
I suspect the error lies in how the blocks complete and how the context is not that correct, it seems weird to me that you enter a group from outside of the block/context you leave the group from.
Finally, are you sure the completion block of the image loading is always called? Is it possible that when the request fails the completion is not called and thus the group counter is never decremented?
Sorry about my initial answer btw, I misread the question totally.
EDIT: Now that I think about what the goal is (synchronising after all images have loaded), it seems that the approach is not really reasonable. Does the code need to block until all the images are loaded? If not, then assuming all completion blocks are fired on a single thread, I would simply keep track of the number of blocks that have been fired and decrement that count in the completion block. When the last one completes, then the contents of the current dispatch_group_notify() could be executed.
Another, perhaps a bit more futureproof option would be to refactor the image loading code to either offer a synchronous way of fetching an image (meant to be used in cases like this) or offer an async API that is capable taking a dispatch group/queue, this obviously assumes that the internals of the image loader uses GCD.
Finally, you could write a NSOperation subclass, that takes care of a single image loading procedure, then those operations could be used in an NSOperationQueue (offering a bit more abstraction from GCD) that can be easily used to keep track how many operations are ongoing and when they all finish.
The problem is your use of dispatch_group_async(). It should not be used unless you are doing tasks that are synchronous that you want to be done asynchronously. Your loadImageWithUrl() is already asynchronous. This is how you should structure your use of dispatch_group.
dispatch_group_t group = dispatch_group_create();
for (...) {
if (...) {
dispatch_group_enter(group);
[self loadImageWithUrl:url onCompletion:^{
dispatch_group_leave(group);
}];
}
}
dispatch_group_notify(group, queue, ^{
NSLog(#"load image complete");
});
Also dispatch_group_wait is the alternative to using dispatch_group_notify. It should only be used if you want to wait synchronously for the group to finish.
I realize that this is a problem others have had before me but I seem unable to find a good solution. My problem is that I want to loop through a number of operations stored in a NSOperationQueue, and when all operations have finished I would like to post a notification. The notification starts a final process that relies on all of the operations to have finished. However, my notification is sent several times prior to the operations have finished. This is my approach:
self.queueFinished=FALSE;
[self createFileobjectsForDirectories:self.mymode];
This results in an array of operations (self.arrayOfOperations) which is further used:
[self.myFileobjectsQueue addOperations:self.arrayOfOperations waitUntilFinished:NO];
NSInvocationOperation *completionOperation = [[NSInvocationOperation alloc] initWithTarget:self
selector:#selector(runLastOperation:)
object:start];
for (NSOperation *myOp in self.myFileobjectsQueue.operations){
[completionOperation addDependency:myOp];
}
[completionOperation setQueuePriority:NSOperationQueuePriorityVeryLow];
[self.myFileobjectsQueue addOperation:completionOperation];
-(void) runLastOperation:(NSDate*) start {
[[NSNotificationCenter defaultCenter] postNotificationName:kCompareFinished object:nil userInfo:nil];
self.queueFinished=TRUE;
}
I have tried several approaches to observing when my queue has finished as suggested elsewhere, without success. It is almost like the notifications are called when the queue is empty, but prior to operations have finished their tasks. But according to Apple documentation the operation will not be removed from the queue until the isFinished flag is TRUE. I am running concurrent operations using the regular main function in a subclass of NSOperation. Perhaps I am misunderstanding something here, but suggestions for how to make sure that not only the queue is empty, but to also make sure that the operations have finished would be greatly appreciated.
Cheers, Trond
It doesn't seem like you need a notification (at least not to trigger the final processing). Instead, create a new operation, make it dependent upon all of the other operations that you add for your processing and then add it to the queue. Then it will run only once all of the other operations are complete.
You're correct about isFinished determining when the operation is finished. One thing to consider is that any completionBlock added to an operation is run after the operation has completed (not just before it completes).
What happens if you dispatch_async a block of code on a queue that's currently blocked by it's own dispatch_sync operation? Do they lock or will the blocked queue continue after the dispatch_sync operation returns?
I have an object I created that manages access to a backing store (SQLite, in this case). It uses one concurrent GCD queue and any other objects that want to access the information from the store will pass a request to the manager along with a block that will be executed asynchronously. The essence of what happens is this (not actual code):
- (void) executeRequest:(StoreRequest *)request withCompletionBlock:(void(^)(NSInteger result)block{
dispatch_queue_t currentContext = dispatch_get_current_queue();
dispatch_async(_storeQueue, ^{
NSInteger result = [_store executeRequest:request];
if (block){
dispatch_async(currentContext, ^{
block(result);
}
}
});
}
The real code is a bit more complex (I actually queue up and store requests/blocks/contexts to execute at the end of a run loop). I also use dispatch_barrier_async for write requests to prevent concurrent read/writing. This all works fine, but in certain situations I also need to perform a synchronous request on the store. Now this request doesn't need to be performed before any queued up operations, but I do need the requesting queue blocked until the operation is performed. This can be easily done:
- (NSInteger) executeRequest:(StoreRequest *)request{
__block NSInteger result = 0;
dispatch_sync(_storeQueue, ^{
result = [_store executeRequest:request];
});
return result;
}
My question is this: What happens if a pending asynchronous operation placed before the synchronous operation dispatches a block of code asynchronously on the queue that is currently blocked by the synchronous dispatch. In other words, the above operation will dispatch its request at the end of the _store queue and wait. But it's quite possible (even likely) that the operations in front of it include asynchronous dispatches back to the waiting queue (for other operations). Will this lock the threads? Since the queued blocks are dispatched asynchronously the _store queue will never be blocked and therefore will finish, theoretically allowing the queue it's blocking to continue...but I'm not sure what happens with the blocks that were asynchronously dispatched or if dispatching anything to a block thread locks it up. I would assume that the blocked queue will continue, finish it's request and then the process the pending blocks, but I want to make sure.
Actually, now that I've written this all up, I'm pretty sure it'll work just fine, but I'm going to post this question anyway to make sure I'm not missing anything.
dispatch_async never blocks. It's that simple.
The dispatch_async itself never blocks. It appends the block to the end of the queue and returns immediately.
Will the block get executed? It depends. In a sequential queue, if one block is blocked, no other block will execute until that block gets unblocked and finishes. On a background queue, the queue can use multiple threads, so even if some blocks are blocked, it will just start other blocks. I haven't tried if there is a limit to the number of blocked blocks, but there's a good chance that all unblocked blocks will eventually execute and finish, and you are left with the blocked ones.
First I create a serial queue like this
static dispatch_queue_t queue = dispatch_queue_create("myQueue", DISPATCH_QUEUE_SERIAL);
then, at some unknown point in time a task gets added to the queue like this
dispatch_async(queue, ^{
// do something, which takes some time
});
If the first task hasn't finished yet, the new task will wait until the first completes (that's of course what a serial queue is for).
But if I add 5 new tasks to the queue, while the original first one is still running, I don't want to execute new task no.1, then no.2, then no.3 and so on, but want to get rid of tasks 1 to 4 and directly start executing task no.5 after the original first task has finished.
In other words, I want to pop any waiting task (not the one that is currently running) off the queue, if I add a new one.
Is there a build in mechanism for this or do I have to implement this myself? And for the latter, how would I identify single tasks inside a queue and remove them?
Once a block has been submitted to a GCD dispatch queue, it will run. There is no way to cancel it. You can, as you know, implement your own mechanism to "abort" the block execution early.
An easier way to do this would be to use NSOperationQueue, as it already provides an implementation for canceling pending operations (i.e., those not yet running), and you can easily enqueue a block with the new-ish addOperationWithBlock method.
Though NSOperationQueue is implemented using GCD, I find GCD much easier to use in most cases. However, in this case, I would seriously consider using NSOperationQueue because it already handles canceling pending operations.
With Davids answer getting me on track I succeeded in doing this like so
taskCounter++;
dispatch_async(queue, ^{
if (taskCounter > 1) {
taskCounter--;
NSLog(#"%#", #"skip");
return;
}
NSLog(#"%#", #"start");
// do stuff
sleep(3);
taskCounter--;
NSLog(#"%#", #"done");
});
taskCounter has to be either an ivar or a property (initialize it with 0). In that case it doesn't even need the __block attribute.
The way you handle this is to use an ivar that indicates to the queued blocks they should just return:
^{
if(!canceled) {
... do work
}
}
You don't need to use a simple boolean either - you can make this more complex - but the general idea is to use one or more ivars that the block queries before doing anything.
I use this technique (but did not invent it) with great success.
If instead of adding a closure in you add a DispatchWorkItem, you can cancel it as long as it hasn't started executing yet.
In the following code, backgroundWorkItem will never run, because it is cancelled before it starts executing.
let backgroundWorkItem = DispatchWorkItem {
print("Background work item executed")
}
DispatchQueue.main.async(execute: backgroundWorkItem)
backgroundWorkItem.cancel()
Working with some code, I'm coming across run loops, which I'm new to, inside NSOperations.
The NSOperations are busy downloading data - and whilst they are busy, there is code to wait for the downloads to complete, in the form of NSRunLoops and thread sleeping.
This code in particular is of interest to me:
while (aCertainConditionIsTrue && [self isCancelled]==NO) {
if(![[NSRunLoop currentRunLoop] runMode: NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:1.0]]){
[NSThread sleepForTimeInterval:1.0];
}
}
I've read about the run loops, and runMode:beforeDate: will wait for an input source or a timeout. Although I'm not 100% what counts as an input souce.
On the first execution of this it always returns NO and hits the sleepForTimeInterval:. Is this bad?
In a particular utility class, it's hitting the sleepForTimeInterval: a lot - once for each thread - which significantly hurts the performance.
Any better solutions for this, or advice?
Sleeping locks up the thread. Perhaps you change your code to use performSelector:withObject:afterDelay. That way your thread can continue to run.
...
done = NO;
[self checkDoneCondition:nil];
...
- (void)checkDoneCondition:(id)object {
if (aCertainConditionIsTrue && [self isCancelled]==NO) {
if(...) {
[self performSelector:#selector(checkDoneCondition:) withObject:[con error] afterDelay:1.0];
} else {
done = YES;
}
}
}
It looks like you need to use a concurrent NSOperation. Here is the relevant part in the Apple docs:
In contrast to a non-concurrent operation, which runs synchronously, a
concurrent operation runs asynchronously. In other words, when you
call the start method of a concurrent operation, that method could
return before the corresponding task is completed. This might happen
because the operation object created a new thread to execute the task
or because the operation called an asynchronous function. It does not
actually matter if the operation is ongoing when control returns to
the caller, only that it could be ongoing.
(...)
In a concurrent operation, your start method is responsible for
starting the operation in an asynchronous manner. Whether you spawn a
thread or call an asynchronous function, you do it from this method.
Upon starting the operation, your start method should also update the
execution state of the operation as reported by the isExecuting
method. You do this by sending out KVO notifications for the
isExecuting key path, which lets interested clients know that the
operation is now running. Your isExecuting method must also return the
status in a thread-safe manner.
(from https://developer.apple.com/library/mac/#documentation/Cocoa/Reference/NSOperation_class/Reference/Reference.html)
In other words, you can override the -start method in your NSOperation subclass, and have ivar for the executing and finished property. This method will start the download in a separate thread. When the download starts, you set the executing flag and trigger KVO. WHen it is finished in this thread, you do the same with finished and executing. It seems complicated but it's actually quite simple.
See also this question on Stack Overflow with a great explanation: Subclassing NSOperation to be concurrent and cancellable