we build an iPad app that downloads a bunch of data and PDF documents from a web service (data first, documents later in the background). To do so, we use SOAP via HTTP(S) requests. It works fine and altogether, the app is running well. Problem is, if there are too many documents to download at some point the app crashes. Using Instruments I figured out that it is a memory issue, particularly NSRegularExpression and NSRunLoop. (I'm using ARC)
I could improve my code to optimize the NSRegularExpression creation. But I don't know how to improve the NSRunLoop issue.
I tried both, asynchronous and synchronous HTTP request. Using async, I had to wait for the download to finish and since sleep()/[NSThread sleepForTimeInterval:] aren't an option, I use
while ( _waitToFinish ) {
[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]];
}
Using sync request, Instruments reveals that
[NSURLConnection sendSynchronousRequest:theRequest returningResponse:&urlResponse error:&error];
also "waits" with help of NSRunLoop and also messes up the memory.
Is this a bug in CoreFoundation or ARC?
Is there another way to idle while waiting for the requests to finish?
Thanks in advance.
Edit:
With "memory issue" I meant that the app crashes (or is killed by iOS) because it uses too much memory.
This is what Instruments shows:
The percentage get higher the longer the app is downloading.
Edit:
Going further down revealed that it is NSURLConnection, that is messing up the memory. It seems that I have somehow missed setting connection and receivedData to nil (see URL Loading System Programming Guide). This improved my memory usage again a little.
Now, there are two more big memory allocation operations:
And this is the code I think belongs to what Instruments displays:
-(void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
[_receivedData appendData:data];
}
-(void)connectionDidFinishLoading:(NSURLConnection *)connection {
NSString *responseText = [[NSString alloc] initWithBytes:[_receivedData mutableBytes] length:[_receivedData length] encoding:NSUTF8StringEncoding];
self.lastResponse = responseText;
responseText = nil;
connection = nil;
_receivedData = nil;
_lastResult = TRUE;
_waitToFinish = FALSE;
}
Is there anything I could change to improve the code?
Edit: (Changed title from "NSRunLoop messes up iPad memory")
Edit:
I created a test app to prove that it is the NSURLConnection, that messes up the memory. Then I contacted the Apple Developer Support.
Since I am downloading a lot of PDF in an iteration with NSURLConnection, the solution was to add an #autoreleasepool { .. } in the iteration and another one around the NSRunLoop.
Thanks.
It's not a bug in Core Foundation or ARC. It's a bug in your code.
Don't run the run loop yourself. Just use +[NSURLConnection sendAsynchronousRequest:queue:completionHandler:]. It will call your completionHandler block when the request is complete. That's when you process the response. Meanwhile, just return out of your method and let the system worry about running the run loop.
You say “it's a memory issue” with NSRegularExpression and NSRunLoop, but you don't say what the issue is, or what evidence Instruments is showing you. If you describe the “issue” in more detail, maybe we can help you with that.
Related
I'm trying to figure out why an iPhone running iOS 14 isn't using TLS 1.3 to connect to a compatible web server.
Relevant code is:
- (void) streamOpened:(NSStream *)stream {
NSDictionary *settings = #{
(__bridge NSString *)kCFStreamSSLValidatesCertificateChain: (__bridge NSNumber *)kCFBooleanFalse
};
CFReadStreamSetProperty((CFReadStreamRef)inputStream, kCFStreamPropertySSLSettings, (CFTypeRef)settings);
CFWriteStreamSetProperty((CFWriteStreamRef)outputStream, kCFStreamPropertySSLSettings, (CFTypeRef)settings);
}
The full source can be seen here: https://github.com/tls-inspector/tls-inspector/blob/app-store/CertificateKit/Getters/CKAppleCertificateChainGetter.m
I've tried specifying the SSL Level with kCFStreamSSLLevel set to kCFStreamSocketSecurityLevelTLSv1_3 but that didn't do anything.
If I use OpenSSL to connect it uses TLS 1.3 and I can verify that with a packet capture, but using CFStream it sticks to 1.2.
The short answer is that you end up using a deprecated API that does not support TLS 1.3.
The long answer, which details a potential solution, is given below.
I tried to solve this using CFStream but did not succeed.
It might be possible. The problem is that you end up, as you do, at a low level using SSLConnectionRef and friends and at a higher level using NSInputStream and NSOutputStream and friends and at some point you run into this https://developer.apple.com/documentation/security/secure_transport?language=objc legacy API.
On that page it mentions that that API was replaced by the Network framework and really this is what I am suggesting you should use. I was hoping to also implement a solution quickly but it needs a bit more rework than what SO is for so I leave it at that.
However, herewith some suggestions, basically the ideas I was hoping to implement.
As before, there are two levels.
At the lower level you end up using the nw_ and family but I would say do not go that way. It might be required for some specialised needs and your app might be in that category, but still, just take note that the higher level is built on top of this.
At the higher level, and this is where I think your solution lies, you end up using NSURL and friends. Your goto guy here is probably NSURLSession but the implementation will dictate that. I tried to give an outline and you can look at my code to get more detail but I think you will be in a much better position to implement from here onwards.
In that I was hoping to connect your legacy stream code to NSURLSession but when that failed I stopped. That is perhaps a bit optimistic and I think it requires a more serious rework, but the various delegates (NSURLSessionDelegate, NSURLSessionTaskDelegate, NSURLSessionStreamDelegate and so on) seem to be ready and waiting for your solution and I don't think it is a lot of work actually.
The most relevant snippet of code from my attempt is below. This switches on TLS 1.3 and I tried to implement around this.
// Some configuration
NSURLSessionConfiguration * config = NSURLSessionConfiguration.ephemeralSessionConfiguration;
// Note this one!!!
config.TLSMaximumSupportedProtocol = kTLSProtocolMaxSupported;
NSURLSession * urlSession = [NSURLSession sessionWithConfiguration:config
delegate:self
delegateQueue:NSOperationQueue.currentQueue];
From this point onwards I tried to reuse your stream code but to be honest I think the final solution may not even use streams any more, and in stead just rely on using the correct delegate. Forgive me for getting excited at this point, but I suspect you will be able to greatly simplify your code as a result.
I enjoyed your app - it is pretty polished and I look forward to you solving this problem. I also enjoyed toying with it but for now that is my contribution.
First attempt
I eyeballed your code - it needs some reworking for TLS 1.3. I tried to do it but I am now rewriting that class so I stopped. You can do it, but, I can not guarantee that it will work!
Anyhow, here are some thoughts.
First on the existing code. Just some - ahem - observations, nothing serious ...
Note that your streamOpened will apply the settings no matter which stream was triggered. The delegate will message that twice I believe, once for the input and once for the output stream. Although here it seems it does not matter you should be careful as that could introduce some serious bugs in another situation.
Also, I think you need to configure the streams before they are opened but this did not make any difference. If you configure the streams before they are opened in your performTaskForURL or afterwards in the streamOpened it did not matter.
I played with the configuration a bit. You need not set one on the output stream, only on the input. And the only key that is required is the one you already set. I could not get any difference no matter what I did.
Second, the solution I - ahem - think will work here.
You need to configure the URL session. So what I did was the following
- (void) performTaskForURL:(NSURL *)url{
queryURL = url;
// Some configuration
NSURLSessionConfiguration * config = NSURLSessionConfiguration.ephemeralSessionConfiguration;
// Note this one!!!
config.TLSMaximumSupportedProtocol = kTLSProtocolMaxSupported;
NSURLSession * urlSession = [NSURLSession sessionWithConfiguration:config
delegate:self
delegateQueue:NSOperationQueue.currentQueue];
// Just code to test the idea, not production ready I know
NSURLSessionStreamTask * streamTask = [urlSession streamTaskWithHostName:url.host
port:443];
[streamTask captureStreams];
}
- (void)URLSession:(NSURLSession *)session
streamTask:(NSURLSessionStreamTask *)streamTask
didBecomeInputStream:(NSInputStream *)inputStreamUrl
outputStream:(NSOutputStream *)outputStreamUrl
{
// I was hoping to get away with this,
// just setting your streams equal to the
// URL stream task streams but it did not
// work ... problem is you need more of the
// stream task delegate methods I believe
inputStream = inputStreamUrl;
outputStream = outputStreamUrl;
// This is some left over code from your performTaskForURL message
// Here you can see how I toyed with the stream configuration
// Configure here before it is opened
// Pretty much your current streamOpened message
// Note only input needs be configured (fwiw)
[self configureStream:inputStream];
inputStream.delegate = self;
outputStream.delegate = self;
[outputStream scheduleInRunLoop:[NSRunLoop mainRunLoop] forMode:NSDefaultRunLoopMode];
[inputStream scheduleInRunLoop:[NSRunLoop mainRunLoop] forMode:NSDefaultRunLoopMode];
[outputStream open];
[inputStream open];
// I was hoping this would just work but I think it needs more work
}
The idea here is to configure a URL session with kTLSProtocolMaxSupported and then to create a stream task off that. Now this is where I want to stop and think you are much better at going further. I was hoping I can just get the streams and throw them back to your code, but it did not work and for now I'm not going further.
I could not test my idea as I need to hook up a few more things to the NSURLSessionStreamDelegate that your class now also implements. I think you do the same in the class already, but for streams.
Now, again, I may be wrong, and wanted to test before I posted, but either see if this might work and then I think you need to e.g. implement delegate methods such as URLSession:task:didReceiveChallenge:completionHandler:.
I think you already do that sort of thing in your code, but this is why I am stopping here, as I think you will be a much better judge of this idea.
I've toyed with it some more but no success - but I think you need to use NSURLSession here, maybe even in stead of sockets.
This is the way I am trying to manage my thread
-(void)ExecuteThread {
#autoreleasepool {
bInsideDifferentThread = YES;
//some code...
bInsideDifferentThread = NO;
}
[NSThread exit];
}
-(void)ThreadCallerEvent {
NSThread *myThread = [[NSThread alloc] initWithTarget:self selector:#selector(ExecuteThread) object:nil];
if (!bInsideThread)
[myThread start];
else
{
[myThread cancel];
}
}
I do it this way becuase I don't want the thread to be started until it has finished working. The problem is that this is generating leaks from a non released memory allocated at [NSThread init]
Any ideas of how to fix this problem?
I ran a fragment similar to yours and wasn't able to detect leaks; but the context is almost certainly different. I'm really not sure what ARC is doing with myThread in your example when it goes out of scope. A typical pattern for using NSThread would be:
[NSThread detachNewThreadSelector:#selector(executeThread)
toTarget:self
withObject:nil];
in which case you're not responsible for dealing directly with the thread you are detaching. (Note: I changed your method name to use camel case which is the preferred method and variable naming convention in Cocoa.)
All of the above said, managing threads are no longer the preferred way of achieving concurrent design. It's perfectly acceptable; but Apple is encouraging developers to migrate to GCD. Far better to think in terms of units of work that need to be performed concurrently.
Without understanding your needs more deeply in this case, it's hard to know what advantages, if any, working directly with threads might offer you; but I would consider looking at concurrent queues/GCD more closely. Perhaps you could simply use something like:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// do your background work
});
and achieve both a clearer design for concurrency and avoid whatever memory management issues you are now seeing.
I'm trying to get UIProgressView, NSURLConnection, NSOperation and NSOperationQueue working together.
Here is the code: http://pastie.org/4080576
Problem:
connection:DidReceiveData: never called: execution going immediately to -start(), through -main(), then to -connection:DidReceiveResponse:, back to -main() (sometimes), and finally to -connectionDidFinishDownloading:destinationURL:.
When I trying to download picture from this link:
http://upload.wikimedia.org/wikipedia/commons/4/41/Kharkov_picture_1787.jpg
I get this output:
2012-06-13 19:43:06.189 downloadingFilesInOpQueue[5070:f803] Received
response: 2012-06-13 19:43:06.190
downloadingFilesInOpQueue[5070:f803] Filesize is: 3075638 Suggested
filename is: Kharkov_picture_1787.jpg 2012-06-13 19:43:12.476
downloadingFilesInOpQueue[5070:f803] Finished downloading. Received 0
bytes
Also, I can't figure out where connection:didReceiveResponse: belong:
NSURLConnectionDelegate or NSURLConnectionDataDelegate.
P.S. If there some style issues, I would glad to hear about them. Thx.
I strongly suspect your "busy" while-loops are consuming the underlying run-loop's execution and choking out the NSURLConnection from being able to handle data as it arrives.
Looking at your pastie.org code, here's your primary culprits:
while (![self isFinished]) {
...
while ([self fileSize] == 0) {}
while ([self receivedDataLength] == 0) {}
...
}
Busy loops like this should almost never be used in iOS. iOS programming is largely event-driven, especially NSURLConnection. Instead, update your UIProgressView in response to connection:DidReceiveData: etc.
Honestly, coding your own NSURLConnection handlers is kind of a messy pain. I strongly recommend you look into using an open-source library that handles some of the difficulties. Here's two decent ones to check out:
https://github.com/AFNetworking/AFNetworking
https://github.com/pokeb/asi-http-request
After using Instruments, I have found the a spot in my code that is very long running and blocking my UI: lots of Core Data fetches (it's part of a process of ingesting a large JSON packet and building up managed objects while making sure that objects have not be duplicated).
While my intentions are to break up this request into smaller pieces and process them serially, that only means I'll be spreading out those fetches - I anticipate the effect will be small bursts of jerkiness in the app instead of one long hiccup.
Everything I've read both in Apple's docs and online at various blog posts indicates that Core Data and concurrency is akin to poking a beehive. So, timidly I've sat down to give it the ol' college try. Below is what I've come up with, and I would appreciate someone wiser pointing out any errors I'm sure I've written.
The code posted below works. What I have read has me intimidated that I've surely done something wrong; I feel like if pulled the pin out of a grenade and am just waiting for it to go off unexpectedly!
NSBlockOperation *downloadAllObjectContainers = [NSBlockOperation blockOperationWithBlock:^{
NSArray *containers = [webServiceAPI findAllObjectContainers];
}];
[downloadAllObjectContainers setCompletionBlock:^{
NSManagedObjectContext *backgroundContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
[backgroundContext setPersistentStoreCoordinator:[_managedObjectContext persistentStoreCoordinator]];
[[NSNotificationCenter defaultCenter] addObserverForName:NSManagedObjectContextDidSaveNotification
object:backgroundContext
queue:[NSOperationQueue mainQueue]
usingBlock:^(NSNotification *note) {
[_managedObjectContext mergeChangesFromContextDidSaveNotification:note];
}];
Builder *builder = [[Builder alloc] init];
[builder setManagedObjectContext:backgroundContext];
for (ObjectContainer *objCont in containers) { // This is the long running piece, it's roughly O(N^2) yuck!
[builder buildCoreDataObjectsFromContainer:objCont];
}
NSError *backgroundContextSaveError = nil;
if ([backgroundContext hasChanges]) {
[backgroundContext save:&backgroundContextSaveError];
}
}];
NSOperationQueue *background = [[NSOperationQueue alloc] init];
[background addOperation:downloadAllObjectContainers];
Since you are using NSPrivateQueueConcurrencyType you must be doing it for iOS5, you do not have to go through all the trouble of creating context in a background thread and merging it in the main thread.
All you need is to create a managed object context with concurrency type NSPrivateQueueConcurrencyType in the main thread and do all operations with managed objects inside a block passed in to managedObjectContext:performBlock method.
I recommend you take a look at WWDC2011 session 303 - What's New in Core Data on iOS.
Also, take a look at Core Data Release Notes for iOS5.
Here's a quote from the release notes:
NSManagedObjectContext now provides structured support for concurrent operations. When you create a managed object context using initWithConcurrencyType:, you have three options for its thread (queue) association
Confinement (NSConfinementConcurrencyType).
This is the default. You promise that context will not be used by any thread other than the one on which you created it. (This is exactly the same threading requirement that you've used in previous releases.)
Private queue (NSPrivateQueueConcurrencyType).
The context creates and manages a private queue. Instead of you creating and managing a thread or queue with which a context is associated, here the context owns the queue and manages all the details for you (provided that you use the block-based methods as described below).
Main queue (NSMainQueueConcurrencyType).
The context is associated with the main queue, and as such is tied into the application’s event loop, but it is otherwise similar to a private queue-based context. You use this queue type for contexts linked to controllers and UI objects that are required to be used only on the main thread.
Concurrency
Concurrency is the ability to work with the data on more than one queue at the same time. If you choose to use concurrency with Core Data, you also need to consider the application environment. For the most part, AppKit and UIKit are not thread-safe. In OS X in particular, Cocoa bindings and controllers are not threadsafe Core Data, Multithreading, and the Main Thread
Ever since switching to XCode 4 the leaks tool shows a LOT of leakage, all from JSONKit and ASIHTTPRequest, after a 2 min run I am leaking hundreds of arrays/dictionaries/strings (from jk_create_dictionary, jk_parse_array, HTTPMessage::*, etc.) totaling a few 100s KB. Most of the stack traces don't originate in any of my calls, and the rest are completely innocent.
I am pretty positive it was not the case pre-XCode 4.
I don't know who the culprit is. Any insight would be lovely.
Update:
The JSONKit leaks are probably JSONDecoder caching.
For example:
static JSONDecoder *decoder = nil;
if (!decoder)
decoder=[[JSONDecoder alloc] init];
ASIHTTPRequest *request = [ASIHTTPRequest requestWithURL:[NSURL URLWithString:url]];
[request setCachePolicy:ASIDoNotWriteToCacheCachePolicy];
[request setCompletionBlock:^{
NSData *response = [request responseData];
NSDictionary *json = [decoder objectWithUTF8String:[response bytes] length:[response length]];
// ...
}];
[request setFailedBlock:^{
// ...
}];
[request startAsynchronous];
EDIT : Before you read the rest of this answer:
If you see that kind of memory leaks, don't blame Instruments or JSONKit... Both are reliable!
...Blame yourself, 99.9% chances your code is leaking the data you parsed with JSONKit!
END_OF_EDIT
Not an answer, more a complement, and an attempt to understand what's going on since I'm seeing leaks too with instruments.
I'm using JSONKit that way :
NSArray *lines = [dataString componentsSeparatedByString:#"\n"];
for (NSString *line in lines) { // I know, strange format isn't? :)
NSDictionary *json = [line objectFromJSONStringWithParseOptions:JKParseOptionLooseUnicode];
// use dictionary data...
}
#ssteinberg, is that the kind of leaks you're having? :
Note that I had this after some heavy load testing, 500 requests with huge JSON responses, which explains leaks are in MB (using latest gh version)
Please note that I'm quite new using Instruments, and I'm not sure how to understand these results. According to Frames reported, yes that looks like Caching... but I'd like to be sure...
So I opened an Issue on GH, I hope that #johnezang, or anyone, will enlight us about this.
All my apologies if that's just a misunderstanding with Instruments, which I would prefer :)
According to Apple's WWDC 2010 videos (Advanced Memory Analysis with Instruments), false positive leaks are rare. Sometimes the Leaks tool misses leaks, but it's reliable for the ones it reports. I'm not all that great with statics, but have you checked to make sure you're not alloc'ing the decoder without releasing it? If it's not released and falls out of scope, that would constitute a leak, right?