My library exposes 2 APIs as follows to:
-(void) createFile{
dispatch_sync(queueSerial, ^{ //B1
[fileObj createFileInfo:file completion:^(NSError *error){
//execute completion block C1
}];
});
}
-(void) readFile:(NSData*)timeStamp{
dispatch_async(queueSerial, ^{ //B2
[fileObj readFileInfo:fileName completion:^(NSError *error) {
dispatch_async(queueSerial2, ^{
//execute completion block C2
});
}]
});
}
Both readFile and createFile are asynchronous methods.
I usually recommend to the people using my library to call createFile before readFile. However, there is no guarantee how the callers will end up implementing this. It usually gets invoked in the following fashion (and I have no control over this)
[fileClass createFile];
[fileClass readFile:timeStamp];
What I want to do is to ensure readFile gets called after the completion block C1 is executed. I also don't want to block the main thread with createFile (but this expectation can be relaxed). So what I want to achieve as end result is :
Caller (that I have no control over) calls createFile and immediately after calls readFile
createFile fully executes, completion block C1 gets fired and after
that, readFile is dispatched to do it's thing.
How can I achieve this?
It feels like you're trying to take an existing completion-handler based API and shoe-horn it into a serial queue pattern. But serial queues only make sense when the dispatched blocks are, themselves, synchronous. (And don't confuse this with dispatch_sync ... we're talking about whether the task inside the block is asynchronous or not, not how you dispatched it to your queue.) But you're dealing with asynchronous methods being called from other asynchronous methods.
The basic pattern when dealing with methods that take completion handler blocks, is to eliminate the serial queue altogether (no GCD queue is needed or is useful when the tasks are already asynchronous) and just use completion handler blocks in your own methods, and make the last call in the most deeply nested block call your own completion handler.
For example:
- (void)foo:(void (^)())completion {
[fileObj setFileInfo:file completion:^(NSError *error) {
// do some other stuff
completion();
}];
}
- (void)bar:(void (^)())completion {
[fileObj getFileInfo:fileName completion:^(NSError *error) {
// do some other stuff
completion();
}];
}
And then you'd call it like so:
[self foo:^{
[self bar:^{
// do whatever you want when its all done
}];
}];
This means that it won't do any of the getFile stuff until the setFile stuff is done.
Having said this, I wonder what added value your setFile and getFile methods add above and beyond the FileObject classes' own methods.
But the key is that when dealing with asynchronous methods, you can't easily coordinate them with a simple serial GCD queue. If you want dispatch queue like behavior for tasks that are, themselves, asynchronous, then you'd generally consider using operation queues instead (or promises/futures or something like that).
Recently I came to a point where I needed some block of code to execute always on the main thread synchronously. This block can be called from any thread. I solved this problem with the code that was already suggested in this SO answer by #Brad Larson
As the comments to this answer it is evident that the deadlock can occur, but I got into the deadlock very very easily. Please have a look at this code.
-(IBAction) buttonClicked
{
// Dispatch on the global concurrent queue async.
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
NSString* data = [self getTheString];
NSLog(#"From Background Thread: %#", data);
};
// Dispatch on the main queue async.
dispatch_async(dispatch_get_main_queue(), ^{
NSString* data = [self getTheString];
NSLog(#"From Main Thread: %#", data);
};
}
// This method can be called from any thread so synchronize it.
// Also the code that sets the string variable based on some logic need to execute on main thread.
-(NSString*) getTheString
{
__block NSString* data = nil;
#synchronized(self)
{
// Have some code here that need to be synchronized between threads.
// .......
//
// Create a block to be executed on the main thread.
void (^blockToBeRunOnMainThread)(void) = ^{
// This is just a sample.
// Determining the actual string value can be more complex.
data = #"Tarun";
};
[self dispatchOnMainThreadSynchronously:blockToBeRunOnMainThread];
}
}
- (void) dispatchOnMainThreadSynchronously:(void(^)(void))block
{
if([NSThread isMainThread])
{
if (block)
{
block();
}
}
else
{
dispatch_sync(dispatch_get_main_queue(), ^{
if (block)
{
block();
}
});
}
}
In this piece of code there are two simultaneous asynchronous requests to function getTheString (Assume you have no control over the buttonClicked method and how it calls getTheString api) . Suppose the request from global queue comes first and it is trying to run the block on the main thread synchronously, till that time background thread in waiting for main thread to execute the block synchronously, at the same time request from main queue comes and trying the acquire the lock from background thread, but as background thread in not complete main thread waiting for background thread to complete. Here we have a deadlock on main thread as main thread waiting for background thread to finish, and background thread is waiting for main thread to execute block.
If I remove the #synchronize statement everything works fine as expected. May be I don't need a #synchronize statement here but in same case you may need to have this. Or it can even happen from some other parts of the code.
I tried to search the whole net for the solution and also tried dispatch_semaphore but couldn't solve the issue. May be I am just not doing things the right way.
I assume this is classic problem of deadlock and faced by developers again and again, and probably have solved it to some extent. Can anyone help with this, or point me to right direction?
I would create a synchronous queue (NSOperationQueue would be simplest) and submit the block to be run on the main thread to that queue. The queue would dispatch the blocks in the order received, maintaining the ordering you desire. At the same time, it disassociates the synchronicity between calling the getTheString method and the dispatch to the main thread.
From the docs:
The completion block you provide is executed when the value returned by the isFinished method changes to YES. Thus, this block is executed by the operation object after the operation’s primary task is finished or cancelled.
I'm using RestKit/AFNetworking, if that matters.
I have multiple dependencies in my NSOperation in a OperationQueue. I use the completion block to set some variables (appending the results to an array) that my child requires.
(task1,...,taskN) -> taskA
taskA addDependency: task1-taskN
Will taskA receive incomplete data since the child can execute before the completion block is fired?
Reference
Do NSOperations and their completionBlocks run concurrently?
I did a simple test by adding a sleep in my completion block and I had a different result. The completion block runs in the main thread. While all the completion block are sleeping, the child task ran.
As I discuss below under "a few observations", you have no assurances that this final dependent operation will not start before your other sundry AFNetworking completion blocks have finished. It strikes me that if this final operation really needs to wait for these completion blocks to finish, then you have a couple of alternatives:
Use semaphores within each of the n the completion blocks to signal when they're done and have the completion operation wait for n signals; or
Don't queue this final operation up front, but rather have your completion blocks for the individual uploads keep track of how many pending uploads are still incomplete, and when it falls to zero, then initiate the final "post" operation.
As you pointed out in your comments, you could wrap your invocation of the AFNetworking operation and its completion handler in your own operation, at which point you can then use the standard addDependency mechanism.
You could abandon the addDependency approach (which adds an observer on the isFinished key of the operation upon which this operation is dependent, and once all those dependencies are resolved, performs the isReady KVN; the problem being that this can theoretically happen before your completion block is done) and replace it with your own isReady logic. For example, imagine you had a post operation which you could add your own key dependencies and remove them manually in your completion block, rather than having them removed automatically upon isFinished. Thus, you custom operation
#interface PostOperation ()
#property (nonatomic, getter = isReady) BOOL ready;
#property (nonatomic, strong) NSMutableArray *keys;
#end
#implementation PostOperation
#synthesize ready = _ready;
- (void)addKeyDependency:(id)key {
if (!self.keys)
self.keys = [NSMutableArray arrayWithObject:key];
else
[self.keys addObject:key];
self.ready = NO;
}
- (void)removeKeyDependency:(id)key {
[self.keys removeObject:key];
if ([self.keys count] == 0)
self.ready = YES;
}
- (void)setReady:(BOOL)ready {
if (ready != _ready) {
[self willChangeValueForKey:#"isReady"];
_ready = ready;
[self didChangeValueForKey:#"isReady"];
}
}
- (void)addDependency:(NSOperation *)operation{
NSAssert(FALSE, #"You should not use addDependency with this custom operation");
}
Then, your app code could do something like, using addKeyDependency rather than addDependency, and explicitly either removeKeyDependency or cancel in the completion blocks:
PostOperation *postOperation = [[PostOperation alloc] init];
for (NSInteger i = 0; i < numberOfImages; i++) {
NSURL *url = ...
NSURLRequest *request = [NSURLRequest requestWithURL:url];
NSString *key = [url absoluteString]; // or you could use whatever unique value you want
AFHTTPRequestOperation *operation = [[AFHTTPRequestOperation alloc] initWithRequest:request];
[operation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
// update your model or do whatever
// now inform the post operation that this operation is done
[postOperation removeKeyDependency:key];
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
// handle the error any way you want
// perhaps you want to cancel the postOperation; you'd either cancel it or remove the dependency
[postOperation cancel];
}];
[postOperation addKeyDependency:key];
[queue addOperation:operation];
}
[queue addOperation:postOperation];
This is using AFHTTPRequestOperation, and you'd obviously replace all of this logic with the appropriate AFNetworking operation for your upload, but hopefully it illustrates the idea.
Original answer:
A few observations:
As I think you concluded, when your operation completes, it (a) initiates its completion block; (b) makes the queue available for other operations (either operations that had not yet started because of maxConcurrentOperationCount, or because of dependencies between the operations). I do not believe that you have any assurances that the completion block will be done before that next operation commences.
Empirically, it looks like the dependent operation does not actually trigger until after the completion blocks are done, but (a) I don't see that documented anywhere and (b) this is moot because if you're using AFNetworking's own setCompletionBlockWithSuccess, it ends up dispatching the block asynchronously to the main queue (or the defined successCallbackQueue), thereby thwarting any (undocumented) assurances of synchrony.
Furthermore, you say that the completion block runs in the main thread. If you're talking about the built in NSOperation completion block, you have no such assurances. In fact, the setCompletionBlock documentation says:
The exact execution context for your completion block is not guaranteed but is typically a secondary thread. Therefore, you should not use this block to do any work that requires a very specific execution context. Instead, you should shunt that work to your application’s main thread or to the specific thread that is capable of doing it. For example, if you have a custom thread for coordinating the completion of the operation, you could use the completion block to ping that thread.
But if you're talking about one of AFNetworking's custom completion blocks, e.g. those that you might set with AFHTTPRequestOperation's setCompletionBlockWithSuccess, then, yes, it's true that those are generally dispatched back to the main queue. But AFNetworking does this using the standard completionBlock mechanism, so the above concerns still apply.
It matters if your NSOperation is a subclass of AFHTTPRequestOperation. AFHTTPRequestOperation uses the NSOperation's property completionBlock for its own purpose in method setCompletionBlockWithSuccess:failure. In that case, don't set the property completionBlock yourself!
It seems, AFHTTPRequestOperation's success and failure handler will run on the main thread.
Otherwise, the execution context of NSOperation's completion block is "undefined". That means, the completion block can execute on any thread/queue. In fact it executes on some private queue.
IMO, this is the preferred approach, unless the execution context shall be explicitly specified by the call-site. Executing completion handlers on threads or queues which instances are accessible (the main thread for example) can easily cause dead locks by an unwary developer.
Edit:
If you want to start a dependent operation after the completion block of the parent operation has been finished, you can solve that by making the completion block content itself a NSBlockOperation (a new parent) and add this operation as a dependency to the children operation and start it in a queue. You may realize, that this quickly becomes unwieldy, though.
Another approach would require an utility class or class library which is especially suited to solve asynchronous problems in a more concise and easy way. ReactiveCocoa would be capable to solve such (an easy) problem. However, it's unduly complex and it actually has a "learning curve" - and a steep one. I wouldn't recommend it, unless you agree to spend a few weeks in learning it and have a lot other asynchronous use cases and even much more complex ones.
A simpler approach would utilize "Promises" which are pretty common in JavaScript, Python, Scala and a few other languages.
Now, please read carefully, the (easy) solution is actually below:
"Promises" (sometimes called Futures or Deferred) represent the eventual result of an asynchronous task. Your fetch request is such asynchronous task. But instead specifying a completion handler, the asynchronous method/task returns a Promise:
-(Promise*) fetchThingsWithURL:(NSURL*)url;
You obtain the result - or the error - with registering a success handler block or a failure handler block like so:
Promise* thingsPromise = [self fetchThingsWithURL:url];
thingsPromise.then(successHandlerBlock, failureHandlerBlock);
or, the blocks inlined:
thingsPromise.then(^id(id things){
// do something with things
return <result of success handler>
}, ^id(NSError* error){
// Ohps, error occurred
return <result of failure handler>
});
And shorter:
[self fetchThingsWithURL:url]
.then(^id(id result){
return [self.parser parseAsync:result];
}, nil);
Here, parseAsync: is an asynchronous method which returns a Promise. (Yes, a Promise).
You might wonder how to get the result from the parser?
[self fetchThingsWithURL:url]
.then(^id(id result){
return [self.parser parseAsync:result];
}, nil)
.then(^id(id parserResult){
NSLog(#"Parser returned: %#", parserResult);
return nil; // result not used
}, nil);
This actually starts async task fetchThingsWithURL:. Then when finished successfully, it starts async task parseAsync:. Then when this finished successfully, it prints the result, otherwise it prints the error.
Invoking several asynchronous tasks sequentially, one after the other, is called "continuation" or "chaining".
Note that the whole statement above is asynchronous! That is, when you wrap the above statement into a method, and execute it, the method returns immediately.
You might wonder how to catch any errors, say fetchThingsWithURL: fails, or parseAsync::
[self fetchThingsWithURL:url]
.then(^id(id result){
return [self.parser parseAsync:result];
}, nil)
.then(^id(id parserResult){
NSLog(#"Parser returned: %#", parserResult);
return nil; // result not used
}, nil)
.then(/*succes handler ignored*/, ^id (NSError* error){
// catch any error
NSLog(#"ERROR: %#", error);
return nil; // result not used
});
Handlers execute after the corresponding task has been finished (of course). If the task succeeds, the success handler will be called (if any). If the tasks fails, the error handler will be called (if any).
Handlers may return a Promise (or any other object). For example, if an asynchronous task finished successfully, its success handler will be invoked which starts another asynchronous task, which returns the promise. And when this is finished, yet another one can be started, and so force. That's "continuation" ;)
You can return anything from a handler:
Promise* finalResult = [self fetchThingsWithURL:url]
.then(^id(id result){
return [self.parser parseAsync:result];
}, nil)
.then(^id(id parserResult){
return #"OK";
}, ^id(NSError* error){
return error;
});
Now, finalResult will either eventually become the value #"OK" or an NSError.
You can save the eventual results into an array:
array = #[
[self task1],
[self task2],
[self task3]
];
and then continue when all tasks have been finished successfully:
[Promise all:array].then(^id(results){
...
}, ^id (NSError* error){
...
});
Setting a promise's value will be called: "resolving". You can resolve a promise only ONCE.
You may wrap any asynchronous method with a completion handler or completion delegates into a method which returns a promise:
- (Promise*) fetchUserWithURL:(NSURL*)url
{
Promise* promise = [Promise new];
HTTPOperation* op = [[HTTPOperation alloc] initWithRequest:request
success:^(NSData* data){
[promise fulfillWithValue:data];
}
failure:^(NSError* error){
[promise rejectWithReason:error];
}];
[op start];
return promise;
}
Upon completion of the task, the promise can be "fulfilled" passing it the result value, or it can be "rejected" passing it the reason (error).
Depending on the actual implementation, a Promise can also be cancelled. Say, you hold a reference to a request operation:
self.fetchUserPromise = [self fetchUsersWithURL:url];
You can cancel the asynchronous task as follows:
- (void) viewWillDisappear:(BOOL)animate {
[super viewWillDisappear:animate];
[self.fetchUserPromise cancel];
self.fetchUserPromise = nil;
}
In order to cancel the associated async task, register a failure handler in the wrapper:
- (Promise*) fetchUserWithURL:(NSURL*)url
{
Promise* promise = [Promise new];
HTTPOperation* op = ...
[op start];
promise.then(nil, ^id(NSError* error){
if (promise.isCancelled) {
[op cancel];
}
return nil; // result unused
});
return promise;
}
Note: you can register success or failure handlers, when, where and as many as you want.
So, you can do a lot with promises - and even more than in this brief introduction. If you read up to here, you might get an idea how to solve your actual problem. It's right there - and it's a few lines of code.
I admit, that this short introduction into promises was quite rough and it's also quite new to Objective-C developers, and may sound uncommon.
You can read a lot about promises in the JS community. There are one or three implementations in Objective-C. The actual implementation won't exceed a few hundred lines of code. It happens, that I'm the author of one of it:
RXPromise.
Take it with a grain of salt, I'm probably totally biased, and apparently all others ever dealt with Promises, too. ;)
I need to protect a critical area of my code, which is multi-threaded. I want to prevent it from being called multiple times before the other thread is finished. This is what I am working with:
- (void) filterAllEventsIntoDictionary{
// start critical area
if (self.sortedKeys.count != 0) {
[self.sortedKeys removeAllObjects];
}
dispatch_async(self.filterMainQueue, ^{
[self internal_filterAllEventsIntoDictionary];
dispatch_sync(dispatch_get_main_queue(), ^{
[self.tableView reloadData];
});
});
}
Since the internal_filterAllEventsIntoDictionary method also accesses self.sortedKeys, if this code is called twice, it crashes because of removeAllObjects at the start.
I still need to call the internal... method in another thread since I don't want to block the UI. So what's the best way to block on the start of this method while the dispatch_async call is still not finished?
While I am far from being a concurrency expert, it sounds to me like you need a lock on your sortedKeys object. If you used a traditional lock, though, you'd end up blocking the main thread.
The recommended replacement for locks in the world of Grand Central Dispatch is to put critical sections of code on a serial queue. See "Eliminating Lock-Based Code" in the Concurrency Programming Guide.
If you put the [self.sortedKeys removeAllObjects]; call onto the same queue that the block with the internal... call is scheduled on, you guarantee that it won't happen until after that block completes:
// start critical area
dispatch_async(self.filterMainQueue, ^{
if (self.sortedKeys.count != 0) {
[self.sortedKeys removeAllObjects];
}
});
This assumes that filterMainQueue is serial. Using dispatch_async for the critical section ensures that the main thread will not be blocked. Also note the warning in "Dispatch Queues and Thread Safety":
Do not call the dispatch_sync function from a task that is executing on the same queue that you pass to your function call. Doing so will deadlock the queue.
Although this will only be an issue if the internal... method does something that causes this method to be called again.
If I have a method called "-uploadToServer:(Object *)objectToUpload", and a mutable array of several Objects, and I want to upload each object one after the other, how could I best handle this?
There are three important considerations:
Don't want NSOperation because I don't want to deal with threading issues
Need to wait for notification of task completion before continuing
Server calls are asynchronous and non-blocking
Here is some code I already have:
for (Object *task in objectsToUpload) {
[self uploadToServer:task];
//need to wait to get notification that upload completed
}
-(void)uploadToServer:(Object *)objectToUpload {
//perform asynchronous server operation here
//either block callback or delegate to notify
//self that upload finished
}
Seeing the above, how do you think I should handle this?
Don't want NSOperation because I don't want to deal with threading issues
Honestly, I think this is your easiest option. The only other way is to do asynchronous IO and use the run loop.
With NSOperation, you'd need two different kinds of operation called e.g. UploadOperation and NotifyOperation: one to upload an object and one to send a notification to the main thread when everything is done.
Then you'd loop through thwe objects putting them all on an NSOperationQueue in an UploadOperation, each one dependent on the previous one (addDependency:). Finally, you'd put the NotifyOperation on the queue dependent on the last UploadOperation.
The NotifyOperation overrides main as follows
-(void) main
{
[someObjectEgViewController performSelectorOnMainThread: #selector(finishedUpload)
withObject: nil
waitUntilDone: NO];
}
objectsToUpload is an NSMutableArray of tasks
-(void)uploadToServer{
//check if there is a task available
if (objectsToUpload.count > 0){
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^(void) {
//get first task
id nextTask = [objectsToUpload objectAtIndex:0];
//do something
//complete async
dispatch_async(dispatch_get_main_queue(), ^(void) {
//remove completed task
[objectsToUpload removeObject:nextTask];
//complete async upload task, check or notify and or start the next task
BOOL shouldDoNextTask = [self check];
if (shouldDoNextTask){
[self uploadToServer];
}
});
});
}
}
I would suggest you do not need to wait for the task to complete. What you need is to respond to the task's completion.
NSURLConnection will provide a delegate with callback methods.