I am trying to use MKNetworkKit to fetch an array of links from a web service, then parse each response on a background thread, and use the dispatch_group_t of GCD to wait until all threads are finished processing. Where I'm stuck is I can't figure out why my dispatch_group_notify is not waiting for all threads in the group to complete. Running this code will print:
results count: 0
added into results, count: 1
added into results, count: 2
The dispatch group is not waiting on its threads. I have also tried dispatch_group_wait but that gave me a crash. I don't know if MKNetworkKit's use of NSOperation is conflicting with this issue. Thanks for any help!
- (MKNetworkOperation *)getABunchOfMovies:(NSArray *)movies onCompletion:(CastResponseBlock)completionBlock onError:(MKNKErrorBlock)errorBlock
{
MKNetworkOperation *operation;
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_group_t group = dispatch_group_create();
block NSMutableArray *results = [[NSMutableArray alloc] initWithCapacity:[movies count]];
for (NSString *movieTitle in movies) {
operation = [self operationWithPath:REQUEST_URL(API_KEY, [movieTitle urlEncodedString])];
[operation onCompletion:^(MKNetworkOperation *completedOperation) {
dispatch_group_async(group, queue, ^{
NSDictionary *response = [completedOperation responseJSON];
id result = [self processResponse:response withMovieTitle:movieTitle];
#synchronized (results) {
[results addObject:result];
NSLog(#"added into results, count: %d", [results count]);
}
});
}
onError:^(NSError *error) {
errorBlock(error);
}];
[self enqueueOperation:operation];
}
dispatch_group_notify(group, dispatch_get_main_queue(), ^{
NSLog(#"results count: %d", [results count]);
// Return array here
completionBlock(results);
});
dispatch_release(group);
return operation;
}
Edit:
I still can't figure out why, but if I change it to use dispatch_group_enter(group); and match it with a dispatch_group_leave(group); at the end of the completion block, it works. Does anyone have any idea why this is happening?
At the moment MKNetworkKit doesn't support queue completion handlers.
You should consider adding operation dependency instead of this hack.
[lastOperation addDependency:op1];
[lastOperation addDependency:op2];
and assume that when "lastOperation" completes, the queue has indeed completed.
Another way is to KVO the "operationCount" keypath on the engine and check if it reaches zero.
MKNetworkEngine has a code block that does this to show and hide the network activity indicator.
Related
I have an asynchronous dispatch queue which reads data from large files in the background. During the course of that it does a few other things including some NSTask operations. The problem I'm facing is that I populate some variables with the result of those operations and the background queue has already moved on by the time those variables are ready (not NULL).
My code looks similar to the following:
dispatch_queue_t backgroundQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(backgroundQueue, ^{
// reading large files
...
NSTaskClass *operation = [[NSTaskClass alloc] init];
NSString *result = [operation doTask:filePath];
NSLog(#"result: %#" result); // returns NULL since task isn't done yet.
...
// continue large file operations
});
Would would be the best way to handle this? I looked into creating a callback but I couldn't figure it out, and I'm not sure if that's even the right approach. Any advice on best practices is appreciated, thanks.
Just give you an idea, hope to solve your problem.
Use a dispatch_sempahore to handle the operations in your NSTaskClass
Create a handler for doTask method, use - (void)doTask:(NSString *)filePath handler:(void(^)(NSString *result))handler instead of - (NSString *)doTask:(NSString *)filePath
Then, your code should look something like this:
dispatch_queue_t backgroundQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(backgroundQueue, ^{
// reading large files
dispatch_semaphore_t sema = dispatch_semaphore_create(0);
...
NSTaskClass *operation = [[NSTaskClass alloc] init];
[operation doTask:filePath handler:^(NSString * _Nonnull result) {
// Get your result here
result = [NSString stringWithString:result];
...
dispatch_semaphore_signal(sema);
}];
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
NSLog(#"result: %#" result); // Here should be fine (not NULL)
...
// continue large file operations
});
I have an async NSOperation to download data for multiple ImageFile objects. Since this is all happening asynchronously I am using a dispatch group to track the requests and then dispatch_group_notify to complete the operation when they are all done.
The question I have is what happen when the operation is ended prematurely, either by cancelation or by some other error. The dispatch group will be left with unmatched dispatch_group_enter and dispatch_group_leave so dispatch_group_notify will never be called. Is it the block retained somewhere by the system forever waiting, or will it get released when the NSOperation gets released?
Or is my approach not ideal, how else should I do this?
- (void)main
{
if (self.cancelled) {
[self completeOperation];
return;
}
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
context.persistentStoreCoordinator = self.persistentStoreCoordinator;
context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy;
[context performBlock:^{
NSFetchRequest *request = [ImageFile fetchRequest];
request.predicate = ....
request.sortDescriptors = ...
NSError *error;
NSArray *imageFiles = [context executeFetchRequest:request error:&error];
if (!imageFiles) {
// Error handling...
[self completeOperation];
return;
}
dispatch_group_t group = dispatch_group_create();
for (ImageFile *imageFile in imageFiles) {
dispatch_group_enter(group);
#autoreleasepool {
[self.webService requestImageWithId:imageFile.id completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
if (self.cancelled) {
[self completeOperation];
return;
}
[context performBlock:^{
if (data && !error) {
imageFile.data = data;
NSError *error;
if (![context save:&error]) {
// Error handling...
[self completeOperation];
return;
}
}
dispatch_group_leave(group);
}];
}];
}
}
dispatch_group_notify(group, dispatch_get_main_queue(), ^{
[self completeOperation];
});
}];
}
From the docs for dispatch_group_enter():
A call to this function must be balanced with a call to dispatch_group_leave.
From the docs for dispatch_group_t:
The dispatch group keeps track of how many blocks are outstanding, and GCD retains the group until all its associated blocks complete execution.
It talks about outstanding blocks, but what it really means is unmatched calls to dispatch_group_enter().
So, the answer to your question about what happens is that the dispatch group object effectively leaks. The block object passed to dispatch_group_notify() and any objects it has strong references to also leak. In your case, that includes self.
The answer to your question of whether your approach is "ideal" is: no, it's not ideal. It's not even valid by the design contract of GCD. You must balance all calls to dispatch_group_enter() with calls to dispatch_group_leave().
If you want to somehow distinguish between success and failure or normal completion and cancellation, you should set some state that's available to the notify block and then code the notify block to consult that state to decide what to do.
In your case, though, the code paths where you fail to call dispatch_group_leave() just do the same thing the notify block would do. So I'm not even sure why you're not just calling dispatch_group_leave() rather than calling [self completeOperation] in those cases.
I have several methods that have the following structure:
- (void) doSomethingWithCompletion: (void (^)(NSError *error)) completion {
__block NSError *fetchError = nil;
dispatch_group_t dispatchGroup = dispatch_group_create();
for (Item* item in self.items)
{
dispatch_group_enter(dispatchGroup);
// fetchError = fetch online data
}
dispatch_group_notify(dispatchGroup, dispatch_get_main_queue(),^{
if (completion)
completion(fetchError);
});
}
My goal is to run several doSomethings after each other, so I could so something like this:
[self doSomethingAWithCompletion: ^(NSArray *results NSError *error) {
if (error == nil) {
[self doSomethingBWithArray: results withCompletion: ^(NSError *error) {
if (error == nil) {
[self doSomethingCWithCompletion: ^(NSError *error) {
if (error == nil) {
// done!!
}
}];
}];
}];
What I am struggling with is the second code block (no pun); is nesting all the methods the way to go, or are there other solutions?
The important thing is, is that doSomethingBWithCompletion cannot begin before doSomethingAWithCompletion is done, and doSomethingCWithCompletion needs to wait until doSomethingBWithCompletion is complete, etc.
Also, doSomethingBWithCompletion uses data that is generated in doSomethingAWithCompletion, etc.
EDIT: After a lot of thinking, refactoring, and simplifying my code, I was able to end up with only two functions, using the nested approach as I outlined above and with a #property for the results array.
The important thing is, is that doSomethingBWithCompletion cannot begin before doSomethingAWithCompletion is done, and doSomethingCWithCompletion needs to wait until doSomethingBWithCompletion is complete, etc.
According to the comments:
The Results of the block are not depending on the result of the first aren't they?
And
Yes they are. For instance, in the first doSomething I determine which items are outdated, in the second doSomething I download and parse the updated items, and in the third doSomething I save them to the store.
(BTW: You should really add this information to your Q.)
If an action depends on the result (not only execution) of a previous action, you have to nest the blocks. Your code does not look like this, because there is no data passed to the completion blocks.
If you do not have such a dependency, you could use a private serial dispatch queue. However, this is a solution in your case, too, if you have akin of a manager class holding the data passed from block to block. But this seems to be highly anticonceptual.
There may be community attempt to add promises to objective-c, and it would be nice to have, because that's just what's needed here. Without committing to a whole new library, you can handle the nesting (which I agree is a bummer) by doing the async tasks recursively... something like this for your example code:
Start with an operation that takes no params and results in an array...
- (void)firstOpWithCompletion:(void (^)(NSArray *, NSError *))completion {
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
NSArray *components = [#"this is an array of strings from the FIRST op" componentsSeparatedByString:#" "];
if (completion) {
completion(components, nil);
}
});
}
Here are a couple that take an array param and result in an array...
- (void)secondOpWithParam:(NSArray *)array completion:(void (^)(NSArray *, NSError *))completion {
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
if (completion) {
NSArray *components = [#"these strings are from the SECOND op" componentsSeparatedByString:#" "];
NSArray *result = [array arrayByAddingObjectsFromArray:components];
if (completion) {
completion(result, nil);
}
}
});
}
- (void)thirdOpWithParam:(NSArray *)array completion:(void (^)(NSArray *, NSError *))completion {
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(queue, ^{
if (completion) {
NSArray *components = [#"these strings are from the THIRD op" componentsSeparatedByString:#" "];
NSArray *result = [array arrayByAddingObjectsFromArray:components];
if (completion) {
NSLog(#"we did it. returning %#", result);
completion(result, nil);
}
}
});
}
// ...as many as these as you need
Now, as in my answer prior to this edit, we just add a param pass initially and in the intermediate calls...
- (void)doSeveralThingsInSequence:(NSArray *)todo param:(NSArray *)param {
if (todo.count == 0) return;
// you could generalize further here, by passing a "final" block and run that before the return
NSString *nextTodo = todo[0];
SEL sel = NSSelectorFromString(nextTodo);
IMP imp = [self methodForSelector:sel];
void (*func)(id, SEL, NSArray *, void (^)(NSArray *, NSError *)) = (void *)imp;
func(self, sel, param, ^(NSArray *result, NSError *error) {
if (!error) {
NSArray *remainingTodo = [todo subarrayWithRange:NSMakeRange(1, todo.count-1)];
[self doSeveralThingsInSequence:remainingTodo param:result];
}
});
}
Stepping through the code: this method bails if there's nothing to do, otherwise it takes the next selector name from the passed array, gets the C function implementation for it and invokes it, placing a completion block on the call stack that starts the process over for the remaining selectors.
Finally, doEverything calls the first operation to get started, then starts running a list of operations (which can be an arbitrarily long list) passing the array output from one as the array input to the next. (You could generalize this further by passing id's along the chain
- (void)doEverything {
[self firstOpWithCompletion:^(NSArray *array, NSError *error) {
NSArray *todo = #[ #"secondOpWithParam:completion:", #"thirdOpWithParam:completion:" ];
[self doSeveralThingsInSequence:todo param:array];
}];
}
I tested this exactly as posted and saw the expected output:
(
this,
is,
an,
array,
of,
strings,
from,
the,
FIRST,
op,
these,
strings,
are,
from,
the,
SECOND,
op,
these,
strings,
are,
from,
the,
THIRD,
op
)
I have the code below to load a group of images and notify me via completionHandler when they are all done loading. However, I find that certain dispatch_group_leave won't be called at times and my guess is imageLoader is deallocated before the block gets to run. If I put a reference of imageLoader within the loadImageWithURL:completionHandler block, then everything works as intended.
Is my guess of the cause correct? What's the correct fix for this issue? I know ARC does block copy in most cases automatically, should I do a block copy in this case?
- (void)loadGroupImagesAsyncWithCompletion:(void(^)(NSError *))completionHandler {
dispatch_group_t group = dispatch_group_create();
int index = 0;
for (Item *item in items) {
char queueLabel[30] = {0};
sprintf(queueLabel, "loader%d", index);
dispatch_queue_t queue = dispatch_queue_create(queueLabel, NULL);
dispatch_group_enter(group);
dispatch_async(queue, ^{
ImageLoader *imageLoader = [[ImageLoader alloc] init];
[imageLoader loadImageWithURL:url completionHandler:^(UIImage *image, NSError *error) {
if (image) {
item.image = image;
}
//NOTE: if item object is referenced in this block,
//then there is no missed dispatch_group_leave call.
dispatch_group_leave(group);
}];
});
}
// Non-blocking wait
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// shouldn't take more than 5 secs to load all images
dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, (int64_t)(5.0 * NSEC_PER_SEC)));
dispatch_async(dispatch_get_main_queue(), ^{
completionHandler(nil);
});
});
}
Here's my guess. It's only a guess because you haven't posted the code of anything to do with ImageLoader.
If -loadImageWithURL:completionHandler: operates asynchronously i.e. it uses an async dispatch queue itself then you could be right that it is being deallocated before the load finishes. This is because its lifetime is just the scope of the for block it is declared in.
In fact, there is no reason why that method needs to do stuff asynchronously, because you have already got it in an asynchronous block. Just have a synchronous method and call dispatch_group_leave() after the method finishes.
EDIT
Given that you have no control over ImageLoader and -loadImageWithURL:completionHandler: operates asynchronously without any help from you, you should remove the dispatch_async wrapper around the call. You'll still have the problem of the ImageLoader being deallocated, but that can be avoided by putting each ImageLoader in a array when you create it.
The code would look something like this:
- (void)loadGroupImagesAsyncWithCompletion:(void(^)(NSError *))completionHandler
{
NSMutableArray* loaders = [[NSMutableArray alloc] init];
dispatch_group_t group = dispatch_group_create();
int index = 0;
for (Item *item in items) {
char queueLabel[30] = {0};
sprintf(queueLabel, "loader%d", index);
dispatch_queue_t queue = dispatch_queue_create(queueLabel, NULL);
dispatch_group_enter(group);
ImageLoader *imageLoader = [[ImageLoader alloc] init];
[imageLoader loadImageWithURL:url completionHandler:^(UIImage *image, NSError *error) {
if (image) {
item.image = image;
}
//NOTE: if item object is referenced in this block,
//then there is no missed dispatch_group_leave call.
dispatch_group_leave(group);
}];
[loaders addObject: imageLoader];
}
// Non-blocking wait
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// shouldn't take more than 5 secs to load all images
dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, (int64_t)(5.0 * NSEC_PER_SEC)));
dispatch_async(dispatch_get_main_queue(), ^{
completionHandler(nil);
});
[loaders removeAllObjects];
});
}
I am using a NSOperationQueue to queue and call a number of Geocoding location lookups. I want to call a completion method when all asynchronicly running lookups have been finished.
-(void)geocodeAllItems {
NSOperationQueue *geoCodeQueue = [[NSOperationQueue alloc]init];
[geoCodeQueue setName:#"Geocode Queue"];
for (EventItem *item in [[EventItemStore sharedStore] allItems]) {
if (item.eventLocationCLLocation){
NSLog(#"-Location Saved already. Skipping-");
continue;
}
[geoCodeQueue addOperationWithBlock:^{
NSLog(#"-Geocode Item-");
CLGeocoder* geocoder = [[CLGeocoder alloc] init];
[self geocodeItem:item withGeocoder:geocoder];
}];
}
[geoCodeQueue addOperationWithBlock:^{
[[NSOperationQueue mainQueue]addOperationWithBlock:^{
NSLog(#"-End Of Queue Reached!-");
}];
}];
}
- (void)geocodeItem:(EventItem *)item withGeocoder:(CLGeocoder *)thisGeocoder{
NSLog(#"-Called Geocode Item-");
[thisGeocoder geocodeAddressString:item.eventLocationGeoQuery completionHandler:^(NSArray *placemarks, NSError *error) {
if (error) {
NSLog(#"Error: geocoding failed for item %#: %#", item, error);
} else {
if (placemarks.count == 0) {
NSLog(#"Error: geocoding found no placemarks for item %#", item);
} else {
if (placemarks.count > 1) {
NSLog(#"warning: geocoding found %u placemarks for item %#: using the first",placemarks.count,item);
}
NSLog(#"-Found Location. Save it-");
CLPlacemark* placemark = placemarks[0];
item.eventLocationCLLocation = placemark.location;
[[EventItemStore sharedStore] saveItems];
}
}
}];
}
Output
[6880:540b] -Geocode Item-
[6880:110b] -Geocode Item-
[6880:540b] -Called Geocode Item-
[6880:110b] -Called Geocode Item-
[6880:110b] -Geocode Item-
[6880:540b] -Geocode Item-
[6880:110b] -Called Geocode Item-
[6880:540b] -Called Geocode Item-
[6880:110b] -Geocode Item-
[6880:580b] -Geocode Item-
[6880:1603] -Geocode Item-
[6880:110b] -Called Geocode Item-
[6880:1603] -Called Geocode Item-
[6880:580b] -Called Geocode Item-
[6880:907] -End Of Queue Reached!-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
[6880:907] -Found Location. Save it-
As you can see the End of Queue function is called before the actual end of all geocoding processes + saving events. "End of Queue Reached" should only be displayed at the very end when all queued lookups have been processed. How can I get this into the right order?
Several issues are coming up here. For one, geocodeAddressString: is asynchronous, so it is returning immediately and the block operation is ending, allowing the next one to start right away. Second, you should not be making multiple calls to geocodeAddressString: one right after the other. From Apple's docs for this method:
After initiating a forward-geocoding request, do not attempt to
initiate another forward-or reverse-geocoding request.
Third, you haven't set a max number of concurrent operations on your NSOperationQueue, so multiple blocks may be executing at once anyway.
For all of these reasons, you might want to use some GCD tools to track your calls to geocodeAddressString:. You could do this with a dispatch_semaphore (to make sure one finishes before the other starts) and a dispatch_group (to make sure you know when all of them have finished) -- something like the following. Let's assume you've declared these properties:
#property (nonatomic, strong) NSOperationQueue * geocodeQueue;
#property (nonatomic, strong) dispatch_group_t geocodeDispatchGroup;
#property (nonatomic, strong) dispatch_semaphore_t geocodingLock;
and initialized them like this:
self.geocodeQueue = [[NSOperationQueue alloc] init];
[self.geocodeQueue setMaxConcurrentOperationCount: 1];
self.geocodeDispatchGroup = dispatch_group_create();
self.geocodingLock = dispatch_semaphore_create(1);
You could do your geocoding loop like this (I've altered the code a bit to make the key parts more obvious):
-(void) geocodeAllItems: (id) sender
{
for (NSString * addr in #[ #"XXX Address 1 XXX", #"XXX Address 2 XXX", #"XXX Address 3 XXXX"]) {
dispatch_group_enter(self.geocodeDispatchGroup);
[self.geocodeQueue addOperationWithBlock:^{
NSLog(#"-Geocode Item-");
dispatch_semaphore_wait(self.geocodingLock, DISPATCH_TIME_FOREVER);
[self geocodeItem: addr withGeocoder: self.geocoder];
}];
}
dispatch_group_notify(self.geocodeDispatchGroup, dispatch_get_main_queue(), ^{
NSLog(#"- Geocoding done --");
});
}
- (void)geocodeItem:(NSString *) address withGeocoder:(CLGeocoder *)thisGeocoder{
NSLog(#"-Called Geocode Item-");
[thisGeocoder geocodeAddressString: address completionHandler:^(NSArray *placemarks, NSError *error) {
if (error) {
NSLog(#"Error: geocoding failed for item %#: %#", address, error);
} else {
if (placemarks.count == 0) {
NSLog(#"Error: geocoding found no placemarks for item %#", address);
} else {
if (placemarks.count > 1) {
NSLog(#"warning: geocoding found %u placemarks for item %#: using the first",placemarks.count, address);
}
NSLog(#"-Found Location. Save it:");
}
}
dispatch_group_leave(self.geocodeDispatchGroup);
dispatch_semaphore_signal(self.geocodingLock);
}];
}
A good solution is to add all geocoding operations as dependencies of the final cleanup operation:
- (void)geocodeAllItems {
NSOperationQueue *geoCodeQueue = [[NSOperationQueue alloc] init];
NSOperation *finishOperation = [NSBlockOperation blockOperationWithBlock:^{
// ...
}];
for (EventItem *item in [[EventItemStore sharedStore] allItems]) {
// ...
NSOperation *operation = [NSBlockOperation blockOperationWithBlock:^{
// ...
}];
[finishOperation addDependency:operation]
[geoCodeQueue addOperation:operation];
}
[geoCodeQueue addOperation:finishOperation];
}
Another solution would be to make the operation queue serial. The operations are still performed on a background thread, but only one at a time and in the order in which they are added to the queue:
NSOperationQueue *geoCodeQueue = [[NSOperationQueue alloc] init];
[geoCodeQueue setMaxConcurrentOperationCount:1];
CompletionBlocks are built in to NSOperation and NSBlockOperation can handle multiple blocks so it's really easy to just add all the work that you need to run async and set your completion block to be called when it is all finished.
- (void)geocodeAllItems {
NSOperationQueue *geoCodeQueue = [[NSOperationQueue alloc] init];
NSBlockOperation *operation = [[[NSBlockOperation alloc] init] autorelease]
for (EventItem *item in [[EventItemStore sharedStore] allItems]) {
// ...
// NSBlockOperation can handle multiple execution blocks
operation addExecutionBlock:^{
// ... item ...
}];
}
operation addCompletionBlock:^{
// completion code goes here
// make sure it notifies the main thread if need be.
}];
// drop the whole NSBlockOperation you just created onto your queue
[geoCodeQueue addOperation:operation];
}
Note: You can't assume that the operations will be performed in your geoCodeQueue. They will be run concurrently. NSBlockOperation manages this concurrency.
NSOperationQueue doesn't work in the way you think, there is no direct dependence between the execution order and adding order. You can call the function in which you subtract till the number equals to zero and you can call the "callback" function.
NSOperationQueues run multiple operations concurrently by default. In practice of course, that means that operations added to the queue will not necessarily be started or finished in the same order you added them.
You can make the queue run all operations serially by setting the queue's maxConcurrentOperationCount value to 1 after you create it:
NSOperationQueue *geoCodeQueue = [[NSOperationQueue alloc]init];
[geoCodeQueue setName:#"Geocode Queue"];
[geoCodeQueue setMaxConcurrentOperationCount:1];
If you do indeed want operations to run concurrently, but still want to be notified when they've all finished, observe the queue's operations property and wait until it reaches zero, as explained in answer linked to by Srikanth in his comment.
EDIT: Nikolai Ruhe's answer is great too.