NSURLConnection + NSMutableData for file downloads is really slow - cocoa-touch

Basically what's happening is that I need to download a whole bunch of files in my app and I've set up a queue of sorts that downloads each file with an NSURLConnection and stores the server response incrementally in an NSMutableData until the download is finished and then writes the whole thing to disk.
Here's the relevant parts:
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)_response {
response = [_response retain];
if([response expectedContentLength] < 1) {
data = [[NSMutableData alloc] init];
}
else {
data = [[NSMutableData dataWithCapacity:[response expectedContentLength]] retain];
}
}
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)_data {
[data appendData:_data];
}
- (void)connectionDidFinishLoading:(NSURLConnection *)connection {
NSLog(#"saved: %#", self.savePath);
[data writeToFile:self.savePath atomically:YES];
}
Any insights as to why this would be awfully slow? It's pretty bad with the Simulator and gets even worse on an actual device. My maximum download size is around 2 megabytes, so I figured storing the whole thing in memory until it finishes wouldn't be that bad of an idea. This gets up to about 20KB/s at best (with a direct ad-hoc wifi connection).
Edit: in all my test cases I do get a Content-Length header, so it's not a matter of growing the NSMutableData with each bit of response received.
Edit 2: this is all Shark gives me.
Edit 3: So this is how I set up the connection
NSMutableURLRequest *request = [[NSMutableURLRequest requestWithURL:[NSURL URLWithString:[#"http://xxx.xxx.xxx.xxx/index.php?service=" stringByAppendingString:service]]] retain];
[request setHTTPMethod:#"POST"];
[request setHTTPBody:[[options JSONRepresentation] dataUsingEncoding:NSUTF8StringEncoding]];
NSURLConnection *conn = [NSURLConnection connectionWithRequest:request delegate:self];
[conn start];
Of course I don't actually have a hardcoded url and both request and conn are instance variables of the downloader class. Not that it should matter, but for JSON I'm using http://code.google.com/p/json-framework/. Options and service are method parameters (NSString and NSDictionary), not that those should matter either.

Boy this is embarrassing. Turns out my Content-Length header was inaccurate, which resulted in NSURLConnection needing to wait for some sort of timeout before it would finish, even though it had all the data. Makes sense really. Maybe this will help someone else out.

I would profile to find out where the slow down is occurring and in what pattern. Put a log statement in connection:didReceiveData to see record how often it is called. You're looking for:
The relative elapsed time between calls to the method.
Whether the time between calls increases as the app runs.
If the elapsed time between calls is where the app spends most of its time then the bottleneck is in the request itself. Either the request is misconfigured of the server is not sending quickly.
If the time between calls increases the longer the app runs, then it is probably a memory issue. As the data grows larger and the memory more constrained, the app has to swap more stuff in and out of memory which slows everything down. To test, log the various didReciveMemoryWarning methods in any active objects.
Update:
According to Shark, the problem is in your URL request and not the code you posted. You need to look at how you set up the request.

Related

Calculate md5 with NSURLSessionDownloadTask for very large file

NSURLConnection can be used to calculate the md5 on-the-fly:
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)theData
// theData is a small piece
NSURLSessionDownloadTask is an "upgrade" of NSURLConnection. But how can we check the md5 without read through the whole file again after it is downloaded? Its interface is like:
NSURLSessionDownloadTask *downloadTask = [session downloadTaskWithRequest:request
completionHandler:
^(NSURL *location, NSURLResponse *response, NSError *error) {
// the whole file is downloaded and saved at location.
}];
The key requirement here is low memory footprint and the file has to be downloaded completely.
If you want the data to arrive in small NSData pieces that you can examine and append to a larger NSMutableData bit by bit, as you did with connection:didReceiveData:, ask for a data task instead of a download task.
You call dataTaskWithRequest:, supply a delegate, and start the data task (with resume) - and the delegate receives URLSession:dataTask:didReceiveData:, exactly like in the old NSURLConnection days.
Here's a complete working example (except that I don't tell you what to do with the bits of data as they arrive):
- (NSURLSession*) configureSession {
NSURLSessionConfiguration* config =
[NSURLSessionConfiguration ephemeralSessionConfiguration];
config.allowsCellularAccess = NO;
NSURLSession* session = [NSURLSession sessionWithConfiguration:config delegate:self delegateQueue:[NSOperationQueue mainQueue]];
return session;
}
- (IBAction) doHTTP: (id) sender {
if (!self.session)
self.session = [self configureSession];
NSString* s = // some URL string
NSURL* url = [NSURL URLWithString:s];
NSMutableURLRequest* req = [NSMutableURLRequest requestWithURL:url];
NSURLSessionDataTask* task = [[self session] dataTaskWithRequest:req];
self.task = task;
[task resume];
}
-(void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveData:(NSData *)data {
NSLog(#"received %lu bytes of data", (unsigned long)data.length);
// do something with the data here!
}
-(void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didCompleteWithError:(NSError *)error {
NSLog(#"completed; error: %#", error);
}
As Matt said, you can use a data task, which lets you easily see the data as it is downloaded.
However, if you want to observe a download task, you can accomplish a similar thing if you are willing to take a few small risks.
I'm sure I'll get a million down votes for the following, but just remember... when you need a screwdriver, and all you have is a hammer, you flip the hammer over, and use it as a screwdriver... or bang on the screw so much that it turns into a nail...
First, I think the API is broken. The delegates should provide at least one of these two things. If you agree, file a radar. The delegate should provide the temporary file (much less preferred - I think it should remain opaque) or it should provide the NSData that is being written in URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite: -- this is the right answer.
Anyway, if you are willing to use an undocumented, unofficial approach...
The temporary files are stored in Library/Caches/com.apple.nsnetworkd/ so you can easily look in there and determine which files are being used as the temporary destination.
Or, you can, again unofficially, determine the temporary file by canceling the download with cancelByProducingResumeData: and then unarchiving the resume data blob -- the resume data blob is currently an archived dictionary -- and get the file path from the dictionary. Then, you can resume the download, knowing which temporary file is being used for the download.
Anyway, once you have the file, inside your URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite: you can then just read the most recently written chunk from the file.
Now, having said that, you may want to just use a data task, because it will officially provide you the piece of data it just downloaded... but you can resort to this hack to get at the data that was downloaded to the file, if you must do a background download -- which must be down with a download task.
One problem you may have is that the file IO may be buffered, so what's been actually flushed (and available form a separate file descriptor) may be different from what has been reported in the delegate method. You may just need to keep track of the last byte you read, and inside that delegate, just read from there to the current end of the file...
Your mileage will most certainly vary, but it will give you access to the data as it is being written to the file.
You will have to do the same thing for URLSession:downloadTask:didFinishDownloadingToURL: to get the final piece of data.

Unit Testing an NSURLConnection Delegate: Faking a Connection

I'm writing a weather application and I've created a sort of weather model that does all the calculations, fetching of data, etc. Before I created the ViewControllers, I wanted to write some unit tests for my model to ensure that everything was working properly (the weather is being fetched in the expected format, weather is refreshing correctly, etc.).
Now I would love to unit test with confidence and test for equality like:
STAssertEquals([[testableModel weatherDictionary]objectForKey:#"current_conditions"], #"Sunny", #"The weather should be sunny.");
...but alas, mother nature changes so fast. Also, (and correct me on this), I don't think I can connect to the Internet while I'm unit testing(...?) (Either way, it doesn't particularly matter.)
So, I searched on my most favorite website and I found this question: how to unit test a NSURLConnection Delegate?
It was quite informative, but now I've got a question. When I call a method that invokes an NSURLConnection delegate method, such as:
[myBeautifulWeatherModel getTheWeather];
...how do I feed the model (i.e., myBeautifulWeatherModel, which is implementing the NSURLConnection delegate methods) data? The model is going to do some JSON parsing when it receives its data in this delegate method:
- (void)connectionDidFinishLoading:(NSURLConnection *)connection;
My first thought was to take the JSON data that Wunderground sends back and just change some of the keys so that they match expected data and feed that in. But the question is HOW would I feed in that response?
I understand that NSURLConnection is going to call 3 required delegate methods. I feel like I need to "fake out" the following:
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response;
-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data;
But again, I'm a little confused as to how I feed it in and get this model to think its connected to the Internet.
I think you're trying to test too broadly. What you want to test here are 2 things: 1) when the NSURLConnectionDelegate callbacks are invoked, they correctly write the data somewhere useful, and 2) given some stored data when the connection completes, it gets stored in your model appropriately. So something like this:
-(void)testShouldAppendData {
expect([connectionDelegate data]).to.beNil();
NSString *response = #"1";
NSData *responseData = [input dataUsingEncoding:NSUTF8StringEncoding];
[connectionDelegate connection:connection didReceiveData:responseData];
NSString *stringFromResponseData = [[[NSString alloc] initWithData:[connectionDelegate data] encoding:NSUTF8StringEncoding] autorelease];
expect(stringFromResponseData).to.equal(#"1");
[connectionDelegate connection:connection didReceiveData:responseData];
stringFromResponseData = [[[NSString alloc] initWithData:[connectionDelegate data] encoding:NSUTF8StringEncoding] autorelease];
expect(stringFromResponseData).to.equal(#"11");
}
and for the data format:
-(void)testShouldUpdateWeatherModel {
NSString *response = #"{\"current_conditions\":\"sunny\"}}";
NSMutableData *responseBytes = [NSMutableData dataWithData:[response dataUsingEncoding:NSUTF8StringEncoding]];
[connectionDelegate setData:responseBytes];
[connectionDelegate connectionDidFinishLoading:nil];
expect([[connectionDelegate weatherDictionary] objectForKey:#"current_conditions"]).to.equal(#"sunny");
}

Street address verification

My user enters a recipients address (Street address not email). I need to verify it with the USPS so I know that it is actually an address.
I am digging through their API right now and I think I understand it but I'm not exactly sure how to go about it with objective-c.
So pretty much it works like so:
I have to create an XML request that contains the recipient name, address, and zip code.
I have to post that to their server
They respond with an XML response
Here is an example of what one of their constructed XML request looks like:
http://SERVERNAME/ShippingAPITest.dll?API=Verify&XML=<AddressValidateRequest% 20USERID="xxxxxxx"><Address ID="0"><Address1></Address1>
<Address2>6406 Ivy Lane</Address2><City>Greenbelt</City><State>MD</State> <Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>
A bit garbled but broken down:
http://SERVERNAME/ShippingAPITest.dll?API=Verify&XML=
<AddressValidateRequest% 20USERID="xxxxxxx">
<Address ID="0">
<Address1></Address1>
<Address2>6406 Ivy Lane</Address2>
<City>Greenbelt</City>
<State>MD</State>
<Zip5></Zip5>
<Zip4></Zip4>
</Address>
</AddressValidateRequest>
My first idea seems obvious but there maybe a better way to go about it. Since the XML feed short, should I go about construction by simple doing something along the lines of:
NSString *request = [NSString stringWithFormat:#"......"]
Where it is filled in and formatted along the lines posted above.
The second question is how to go about correctly sending this to the server?
I simply create a NSURL request and with the URL as the constructed XML string?
Here what I have but I keep getting that the URL was constructed wrong:
- (void)verifyAddress:(Recipient*)_recipient {
NSURL *_url = [NSURL URLWithString:#"http://testing.shippingapis.com/ShippingAPITest.dll?API=Verify&XML=<AddressValidateRequest%20USERID=\"********\"><Address ID=\"0\"><Address1></Address1><Address2>6406 Ivy Lane</Address2><City>Greenbelt</City><State>MD</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>"];
// Create the request.
NSURLRequest *theRequest=[NSURLRequest requestWithURL:_url
cachePolicy:NSURLRequestUseProtocolCachePolicy
timeoutInterval:60.0];
// create the connection with the request
// and start loading the data
NSURLConnection *theConnection=[[NSURLConnection alloc] initWithRequest:theRequest delegate:self];
if (theConnection) {
// Create the NSMutableData to hold the received data.
// receivedData is an instance variable declared elsewhere.
receivedData = [NSMutableData data];
NSString* newStr = [[NSString alloc] initWithData:receivedData
encoding:NSUTF8StringEncoding];
NSLog(#"the response '%#'", newStr);
} else {
// Inform the user that the connection failed.
NSLog(#"error");
}
}
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
// This method is called when the server has determined that it
// has enough information to create the NSURLResponse.
// It can be called multiple times, for example in the case of a
// redirect, so each time we reset the data.
// receivedData is an instance variable declared elsewhere.
[receivedData setLength:0];
}
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
// Append the new data to receivedData.
// receivedData is an instance variable declared elsewhere.
[receivedData appendData:data];
}
- (void)connection:(NSURLConnection *)connection
didFailWithError:(NSError *)error
{
// inform the user
NSLog(#"Connection failed! Error - %# %#",
[error localizedDescription],
[[error userInfo] objectForKey:NSURLErrorFailingURLStringErrorKey]);
}
- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
NSString* newStr = [[NSString alloc] initWithData:receivedData
encoding:NSUTF8StringEncoding];
NSLog(#"the response '%#'", newStr);
// do something with the data
// receivedData is declared as a method instance elsewhere
NSLog(#"Succeeded! Received %d bytes of data",[receivedData length]);
}
I get the following error:
Connection failed! Error - bad URL (null)
My only question now is, am I doing everything ok as far as NSURLConnection goes? I can play around with the URL, I just want to make sure my implementation is ok so Im not running around in circles. :P
You have % 20 in your URL. It should be %20 (no space).
There may be other problems, but that was one was easy to spot. If you are getting an error message, you need to edit your question and paste in the exact error message.
Also, you might consider using Apple's NSURLRequest and NSURLConnection classes, because more people are likely to be familiar with them so it may be easier for you to find help.
Cory, I work in the address validation industry (for SmartyStreets, where what you're trying to do is our specialty in fact) and have seen a lot of similar issues to yours.
We actually used to support an XML-endpoint for our address verification API (LiveAddress). Last year we deprecated it and deployed a new JSON format because the XML was clunky to use and had a lot of problems when it's actually just a simple task (for you, the developer).
So a few things to keep in mind... and while Rob's answer is programmatically comprehensive, these are important to consider also:
The USPS is the official source of addresses for the USA, but its core domain is not providing API service. Especially with recent financial troubles, I suspect that support and maintenance of the API will wane over time.
The License Agreement for the API you're using is quite restrictive. For example:
User agrees to use the USPS Web site, APIs and USPS data to facilitate USPS shipping transactions only. [27 Jan 2012]
Meaning, if you're shipping mail or packages via the USPS by using their API, it's fine, but for any other purpose it's not allowed and it violates the TOS.
I see you're developing for iOS. There's a great JSON library for that called TouchJSON that, in my opinion, is easier to use than XML formats in Objective-C.
While the USPS service does work, they CASS-certify private entities to provide their data at a better value (more specialty, experience, features, etc).
These and other maladies can be remedied by service from a third-party vendor. More details and reasons are documented here. Which provider you choose is up to you, but I'll be happy to personally answer any other address-validation-related questions.

NSURLConnection leak?

i have set up a nsurl which grabs the data from http.
when i run instrument, it says i have a leak NSFNetwork object.
and how do i release theConnection in (void)ButtonClicked? or it will be release later on?
- (void)ButtonClicked {
NSURLRequest *theRequest = [NSURLRequest requestWithURL:[NSURL URLWithString:KmlUrl]
cachePolicy:NSURLRequestUseProtocolCachePolicy
timeoutInterval:20.0f];
NSURLConnection *theConnection = [[NSURLConnection alloc] initWithRequest:theRequest delegate:self];
if (theConnection) {
// receivedData is declared as a method instance elsewhere
NSMutableData *receivedData = [[NSMutableData data] retain];
[self setKMLdata:receivedData];
} else {
// inform the user that the download could not be made
}
}
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
// append the new data to the receivedData
// receivedData is declared as a method instance elsewhere
[KMLdata appendData:data];
NSLog(#"didReceiveData");
}
- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
// release the connection, and the data object
[connection release];
[KMLdata release];
}
- (void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
// release the connection, and the data object
[connection release];
// receivedData is declared as a method instance elsewhere
[KMLdata release];
}
I finally found the answer for this.
The error in the above code (which by the way is the near-exact sample from the SDK docs) is not in the memory management code. Autorelease is one option, manual release is another. Regardless of how you handle your NSURLConnection object, you get leaks using NSURLConnection.
First up, here is the solution. Just copy these 3 lines of code directly into connectionDidFinishLoading, didFailWithError and anywhere else you release the NSURLConnection object.
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:nil];
[NSURLCache setSharedURLCache:sharedCache];
[sharedCache release];
Credit to mpramodjain on http://forums.macrumors.com/showthread.php?t=573253 for the code.
The problem seems to be this – the SDK caches the requests and replies on the iPhone. Even it seems if your NSMutableURLRequest cachePolicy is set to not load the reply from the cache.
The silly thing is that it seems to cache a lot of data by default. I'm transmitting a lot of data (split into multiple connections) and started to get memory warnings, and finally my App died.
The docs we need are in NSURLCache (not NSURLConnection), they state:
NSURLCache implements the caching of
responses to URL load requests by
mapping NSURLRequest objects to
NSCachedURLResponse objects. It is a
composite of an in-memory and an
on-disk cache.
Methods are provided to manipulate the
sizes of each of these caches as well
as to control the path on disk to use
for persistent storage of cache data.
Those three lines have the effect of nuking the cache totally. After adding them to my App (GPS Log), my #living object count remains steady.
Hello have you test this delegate method ?
- (NSCachedURLResponse *)connection:(NSURLConnection *)connection willCacheResponse:(NSCachedURLResponse *)cachedResponse
{
return nil;
}
You can manage the cache more precisely.
"reset" NSURLCache *sharedCache can cause problems on other part of your code ?
This is a common question and is solved by the magic of [object autorelease]. In your code this would be as follows:
NSURLConnection *theConnection = [[[NSURLConnection alloc] initWithRequest:theRequest delegate:self] autorelease];
In this way, the object is automatically added to the "autorelease pool" and dealloc'd at the start of the next run loop after it is no longer referenced.
Hope that helps
Edit: Also, I don't see why you're needing to call -retain on your receivedData variable.
I am using the static method/autoreleased approach and it appears to work fine:
[NSURLConnection connectionWithRequest:theRequest delegate:self];
This way you don't even have to worry about releasing in the delegate callbacks. It turns out that the retain count of the connection is actually 2 (not 1) after it is alloc'd in the examples above, which changes the way I thought about this memory "leak."
#rpetrich I actually don't think you need to worry about the delegate being released before the connection is released. The connection retains it's delegate and the connection itself is actually retained by some sort of open connections queue. I wrote a blog post on my experiments with NSURLConnection on my blog:
"Potential leak of object" with NSURLConnection

Is it possible to prevent an NSURLRequest from caching data or remove cached data following a request?

On iPhone, I perform a HTTP request using NSURLRequest for a chunk of data. Object allocation spikes and I assign the data accordingly. When I finish with the data, I free it up accordingly - however instruments doesn't show any data to have been freed!
My theory is that by default HTTP requests are cached, however - I don't want my iPhone app to cache this data.
Is there a way to clear this cache after a request or prevent any data from being cached in the first place?
I've tried using all the cache policies documented a little like below:
NSMutableURLRequest *theRequest = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:url]];
theRequest.cachePolicy = NSURLRequestReloadIgnoringLocalCacheData;
but nothing seems to free up the memory!
Usually it's easier to create the request like this
NSURLRequest *request = [NSURLRequest requestWithURL:url
cachePolicy:NSURLRequestReloadIgnoringCacheData
timeoutInterval:60.0];
Then create the connection
NSURLConnection *conn = [NSURLConnection connectionWithRequest:request
delegate:self];
and implement the connection:willCacheResponse: method on the delegate. Just returning nil should do it.
- (NSCachedURLResponse *)connection:(NSURLConnection *)connection willCacheResponse:(NSCachedURLResponse *)cachedResponse {
return nil;
}
I have the same problem in my app when I requested info from twitter. In my case I didn't need to preserve those credentials, so I simple erase them using the next code:
- (void) eraseCredentials{
NSURLCredentialStorage *credentialsStorage = [NSURLCredentialStorage sharedCredentialStorage];
NSDictionary *allCredentials = [credentialsStorage allCredentials];
//iterate through all credentials to find the twitter host
for (NSURLProtectionSpace *protectionSpace in allCredentials)
if ([[protectionSpace host] isEqualToString:#"twitter.com"]){
//to get the twitter's credentials
NSDictionary *credentials = [credentialsStorage credentialsForProtectionSpace:protectionSpace];
//iterate through twitter's credentials, and erase them all
for (NSString *credentialKey in credentials)
[credentialsStorage removeCredential:[credentials objectForKey:credentialKey] forProtectionSpace:protectionSpace];
}
}
I hope it works for somebody :)
If you use NSURLConnection take a look at the delegate:
- (NSCachedURLResponse *)connection:(NSURLConnection *)connection willCacheResponse:(NSCachedURLResponse *)cachedResponse
Return Value
The actual cached response to store in the cache. The delegate may return cachedResponse unmodified, return a modified cached response, or return nil if no cached response should be stored for the connection.
If you're using NSURLSession, another solution to prevent request and parameters being written to the Cache.db iOS creates within the app's Caches directory, is to set the NSURLCache for the session's configuration to a 0 size memory and 0 size disk cache e.g.
let configuration = URLSessionConfiguration.default
configuration.urlCache = URLCache(memoryCapacity: 0, diskCapacity: 0, diskPath: nil)
let session = URLSession(configuration: configuration)
or as mentioned above set at a global cache level
URLCache.shared = URLCache(memoryCapacity: 0, diskCapacity: 0, diskPath: nil)
Presumably it's the 0 for disk size that stops iOS writing to disk but if you have a policy to reloadIgnoringLocalCacheData then you probably aren't interested in memory caching either.
Note This will prevent any Caches/Cache.db (requests & responses) or Caches/fsCachedData/ folder (response data) being created at all. We've decided to take this approach in an app for security purposes as we don't want our requests to be stored on disk cache ever.
If anyone knows is there's a way to stop only request caching but keep response data caching from the iOS URL Loading mechanism, I'd be interested to know. (there's no API or official documentation about this from what I can tell)
If not specific to a single request(U want disable cache for whole app) below one is the best option.Add this code in app delegate or based on ur need any where
int cacheSizeMemory = 0; // 0MB
int cacheSizeDisk = 0; // 0MB
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:cacheSizeMemory diskCapacity:cacheSizeDisk diskPath:#"nsurlcache"];
[NSURLCache setSharedURLCache:sharedCache];
NSMutableURLRequest* request = [[NSMutableURLRequest alloc] url];
[request setValue:#"no-store" forHTTPHeaderField:#"Cache-Control"];
[request setCachePolicy:NSURLRequestReloadIgnoringLocalCacheData];
Assuming the server is correctly implemented, putting the Cache-Control:no-store header in the request will generate a server response with the same header, thus causing NSURLCache to not store the response data on disk.
Therefore, no need for the shotgun approach of disabling NSURLCache disk caching.
PS: Adding the header should work for all HTTP frameworks, like AFNetworking