NSURLSessionAuthChallengeRejectProtectionSpace fails - custom certificate validation on OS X iOS - objective-c

Question

I have application with custom certificate validation. Code is quite simple:
- (void) URLSession:(NSURLSession *)session
task:(NSURLSessionTask *)task
didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge
completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition disposition,
NSURLCredential *credential))completionHandler
{
NSString * authenticationMethod = challenge.protectionSpace.authenticationMethod;
CSHTTPDownoadTaskProxy *proxyTask = [self streamedDataTaskForHttpTask: task]; // get custom information about task
if ([authenticationMethod isEqualToString: NSURLAuthenticationMethodServerTrust]) {
if (!proxyTask) {
completionHandler(NSURLSessionAuthChallengeRejectProtectionSpace, nil);
LOG_UNEXPECTED_TASK(task);
return;
}
[self validateCertificate: challenge.protectionSpace withCompletionHandler: completionHandler];
} else if ([authenticationMethod isEqualToString: NSURLAuthenticationMethodHTTPBasic] {
… … …
}
}
- (void) validateCertificate: (NSURLProtectionSpace *)protectionSpace
withCompletionHandler: (void (^)(NSURLSessionAuthChallengeDisposition disposition,
NSURLCredential *credential))completionHandler {
// some logic
… … …
completionHandler(NSURLSessionAuthChallengeRejectProtectionSpace, nil);
}
Now when certificate is rejected usually everything works as expected, certificate is rejected and error is reported for HTTP request.
For some servers even though completionHandler(NSURLSessionAuthChallengeRejectProtectionSpace, nil); is invoked request is processed and completed successfully.
NSURLSessionAuthChallengeRejectProtectionSpace documentation says:
Reject this challenge, and call the authentication delegate method
again with the next authentication protection space. The provided
credential parameter is ignored.
This is a bit confusing, what does it mean "next authentication protection space"?
This API covers also password protection space and I understand that that if service has multiple ways of authentication, for example: Negotiate, Diget, Basic, NSURLSession prepossess best one first it it is rejected it notifies about alternative possibility or fails request.
But my delegate is invoked only once for this authentication method NSURLAuthenticationMethodServerTrust!
For most cases when I'm using this value expected error is reported (invalid certificate):
Error Domain=NSURLErrorDomain Code=-1202 "The certificate for this
server is invalid. You might be connecting to a server that is
pretending to be “10.133.32.55” which could put your confidential
information at risk."
For other cases (logs show that completion handler was called with proper values) HTTP request is processed and completes successfully (it happens for some servers on different client machines).
Did anyone else experience similar problems?
I could do nasty workaround with NSURLSessionAuthChallengeCancelAuthenticationChallenge. This value reports that request was "Canceled" so I have to generate my own error.
I don't like this approach and prefer find reason why NSURLSessionAuthChallengeRejectProtectionSpace doesn't work sometimes.
Update
Ok I have some extra clue. Problem appears only on servers which are demanding client identity certificate.
When client identity is not needed certificate validation is accepted (certificate is rejected when requested), if client identity is demanded and available, server certificate validation result is ignored.
After finding precise answer it looks like whoever configured test servers did that more properly for servers with client identity leading to this side effects.

Answer 1

The purpose of NSURLSessionAuthChallengeRejectProtectionSpace is to tell the OS that you aren't able to do anything with a particular authentication type. For example, if the server says that it is willing to accept either client certificates, basic auth (username/password) or digest auth (essentially signing the request using the username/password as a key) then NSURLSession will ask you for one of those methods at a time.
If the session asks you for client certificates and all you have are a username and password, you would reject the client certificate authentication method by calling the continuation block with NSURLSessionAuthChallengeRejectProtectionSpace, and the session will then ask you for a username and password.
IMO, it doesn't make much sense to use NSURLSessionAuthChallengeRejectProtectionSpace in combination with server trust authentication, because there is no alternative to TLS for evaluating whether the connection is trusted. In practice, I think that this is equivalent to calling NSURLSessionAuthChallengePerformDefaultHandling, which means that if the cert is trusted, it will be accepted, otherwise it will be rejected. But you should be using NSURLSessionAuthChallengePerformDefaultHandling explicitly if that is the desired behavior (or NSURLSessionAuthChallengeCancelAuthenticationChallenge if you want to cancel the request).
It may be that rejecting the request in the presence of a client certificate causes the server's cert to be ignored, in which case this is probably a bug, and you should file it. But the reason for doing so is presumably because accepting the server's cert is required when sending a client cert, and you told it to skip the server validation and send you the next method (client cert). So I'm pretty sure that's the reason things are behaving strangely.
Also, you need a final "else" case there to handle unknown protection spaces. Chances are, the else case should use NSURLSessionAuthChallengePerformDefaultHandling. If you don't do this, then when you get an unknown protection space, the connection will just stay open forever and will never move forward or get canceled, because nothing ever calls the continuation block.