App I'm working on uses ARC. I wanted it to process large files, so instead of loading files as a whole, I'm loading chunks of data using NSFileHandle readDataOfLength method. It's happening inside a loop which repeats until the whole file is processed:
- (NSString*)doStuff { // called with NSInvocationOperation
// now we open the file itself
NSFileHandle *fileHandle =
[NSFileHandle fileHandleForReadingFromURL:self.path
error:nil];
...
BOOL done = NO;
while(!done) {
NSData *fileData = [fileHandle readDataOfLength: CHUNK_SIZE];
...
if ( [fileData length] == 0 ) done = YES;
...
}
...
}
According to profiler, there are no memory leaks; however, my app eats a LOT of memory while it processes the file. My guess — autorelease comes only after I process the file. Can I fix it without switching to manual memory management?
Wrap the code within that loop with a autorelease pool.
while(!done)
{
#autoreleasepool
{
NSData *fileData = [fileHandle readDataOfLength: CHUNK_SIZE];
...
if ( [fileData length] == 0 )
{
done = YES;
}
...
}
};
readDataOfLength retuns autoreleased data and since you stick inside that loop and therefor its method, that autoreleased data that does not get released until your loop and the encapsulating method is done.
Related
I want to fill my app's database with some data from a CSV file during initial startup (50'000 entries). I however run into performance issues...it is way too slow right now and in the simulator takes like 2-3 minutes. My context hierarchy is as follows:
parentContext (separate thread)
context (main thread)
importContext (separate thread)
The code:
for (NSString *row in rows){
columns = [row componentsSeparatedByString:#";"];
//Step 1
Airport *newAirport = [Airport addAirportWithIcaoCode: [columns[0] stringByTrimmingCharactersInSet:
[NSCharacterSet whitespaceCharacterSet]]
name:columns[1]
latitude:[columns[2] doubleValue]
longitude:[columns[3] doubleValue]
elevation:[columns[4] doubleValue]
continent:columns[5]
country:columns[6]
andIataCode:columns[7]
inManagedObjectContext:self.cdh.importContext];
rowCounter++;
//Increment progress
dispatch_async(dispatch_get_main_queue(), ^{
[progress setProgress:rowCounter/totalRows animated:YES];
[progress setNeedsDisplay];
});
// STEP 2: Save new objects to the parent context (context).
//if(fmodf(rowCounter, batchSize)== 0) {
NSError *error;
if (![self.cdh.importContext save:&error]) {
NSLog(#"error saving %#", error);
}
//}
// STEP 3: Turn objects into faults to save memory
[self.cdh.importContext refreshObject:newAirport mergeChanges:NO];
}
If I activate the if with modulo for batch size 2000 in step 2, then of course only every 2000nd entry gets saved and then the performance is fast. But like this it is super slow and I wonder why? I have my import context in a separate thread and still it lags very much...
Normally, it is enough to issue a single save after processing all your entries. You don't need to save this often. Just do it after the for loop. I see that you try to save memory by turning the objects into faults each time, so this might require a save (did not try this before).
I suggest you use #autoreleasepool instead and let the system decide where to save memory:
for (NSString *row in rows) {
#autoreleasepool {
// Just STEP 1 here
...
}
}
NSError *error;
if (![self.cdh.importContext save:&error]) {
NSLog(#"error saving %#", error);
}
A Command Line Tool project was generated in XCode 5.1 for OSX, with the following simple code only to prove NSData memory leak.
The NSData object is never released. Every new instance in the loop increments memory leak. Autorelease doesn't work. Some of the direct release attempts results to syntax error. Any advice?
Sourrounding the NSData allocation with #autoreleasepool didn't help.
If I change NSData object to an NSString object, it also causes memory leak. So it seems to be a general memory deallocation problem in this Command line tool environment.
Returning from the memoryLeak method to the caller, the memory is still unrealased.
+ (void) memoryLeak {
NSURL *tileURL = [NSURL URLWithString:#"http://c.tile.openstreetmap.org/9/0/0.png"];
for (int i=0; i < 10; i++) {
NSData *tile = [NSData dataWithContentsOfURL:tileURL];
}
}
Try updating your code to:
+ (void) memoryLeak {
#autoreleasepool {
NSURL *tileURL = [NSURL URLWithString:#"http://c.tile.openstreetmap.org/9/0/0.png"];
for (int i=0; i < 10; i++) {
NSData *tile = [NSData dataWithContentsOfURL:tileURL];
}
}
// Memory should be cleaned up here
}
Does it still leak? If so looks like a bug, if not, you were possibly detecting leaks before the autorelease pool had a chance to clean it up.
As you can see, the code below isnt doing much (all commented out) more than enumerating over a set of files, however, my memory usage is growing to over 2 GB after 40 seconds of running the function below which is launched by pressing a button on the UI.
I can run the UI for hours, and before pressing the button, the memory usage does not exceed 8MB.
Given that ARC is turned on, what is holding on to the memory?
removed original code as the edit below made no differance.
EDIT:
Attempted #autoreleasepool{ dispatch_asyny ... } and permutations of that around the while and inside the while loop which had no effect.
Here is the code with autorelasepool added and cleaned up
-(void) search{
self.dict = [[NSMutableDictionary alloc] init];
NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:#"/tmp/SeaWall.log"];
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
NSString *bundleRoot = #"/";
NSFileManager *manager = [NSFileManager defaultManager];
NSDirectoryEnumerator *direnum = [manager enumeratorAtPath:bundleRoot];
NSString *filename;
while ((filename = [NSString stringWithFormat:#"/%#", [direnum nextObject]] ) && !self.exit) {
#autoreleasepool {
NSString *ext = filename.pathExtension;
if ([ext hasSuffix:#"so"] || [ext hasSuffix:#"dylib"] ) {
if (filename == nil || [NSURL URLWithString:filename] == nil) {
continue;
}
NSData *nsData = [NSData dataWithContentsOfFile:filename];
if (nsData != nil){
NSString *str = [nsData MD5];
nsData = nil;
[self writeToLogFile:[NSString stringWithFormat:#"%# - %#", [filename lastPathComponent], str]];
}
}
ext = nil;
} // end autoreleasepool
}
[fileHandle closeFile];
[self ControlButtonAction:nil];
});
}
The memory is not exactly leaked: it is very much ready to be released, but it never has a chance to be.
ARC builds upon the manual memory management rules of Objective-C. The base rule is that "the object/function that calls init owns the new instance", and the owner must release the object when it no longer needs it.
This is a problem for convenience methods that create objects, like [NSData dataWithContentsOfFile:]. The rule means that the NSData class owns the instance, because it called init on it. Once the value will be returned, the class will no longer need the object, and it would need to release it. However, if this happens before the callee gets a chance to retain the instance, it will be gone before anything had a chance to happen.
To solve this problem, Cocoa introduces the autorelease method. This method transfers the ownership of the object to the last autorelease pool that was set up. Autorelease pools are "drained" when you exit their scope.
Cocoa/AppKit/UIKit automatically set up autorelease pools around event handlers, so you generally do not need to worry about that. However, if you have a long-running method, this becomes an issue.
You can declare an autorelease pool using the #autoreleasepool statement:
#autoreleasepool
{
// code here
}
At the closing bracket, the objects collected by the autorelease pool are released (and possibly deallocated, if no one else has a reference to them).
So you would need to wrap the body of your loop in this statement.
Here's an example. This code "leaks" about 10 megabytes every second on my computer, because the execution never leaves the #autoreleasepool scope:
int main(int argc, const char * argv[])
{
#autoreleasepool
{
while (true)
{
NSString* path = [NSString stringWithFormat:#"%s", argv[0]];
[NSData dataWithContentsOfFile:path];
}
}
}
On the other hand, with this, the memory usage stays stable, because execution leaves the #autoreleasepool scope at the end of every loop iteration:
int main(int argc, const char * argv[])
{
while (true)
{
#autoreleasepool
{
NSString* path = [NSString stringWithFormat:#"%s", argv[0]];
[NSData dataWithContentsOfFile:path];
}
}
}
Creating objects in the loop condition is awkward for long loops because these are not picked up by the inner #autoreleasepool. You will need to get these inside the #autoreleasepool scope as well.
Returning
Whenever we return an object (maybe to Swift), we need to register into nearest #autoreleasepool block (by calling autorelease method to prevent memory-leak, according to ownership-rules), but nowadays ARC does that automatically for us;
Whenever ARC disabled; after using alloc and/or init, call autorelease manually, like:
- (NSString *)fullName {
NSString *string = [[[NSString alloc] initWithFormat:#"%# %#",
self.firstName, self.lastName] autorelease];
return string;
}
Memory needs to be released by an autorelease pool.
Otherwise it will be locked up as you are experiencing and it will leak.
In your loop put:
#autoreleasepool { /* BODY */ }
I use the NSTask to run shell command and output the data via NSPipe. At first, I using bellow method to read output data, it is no any problem.
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData = [[NSMutableData alloc] init];
NSData *taskData = nil;
if((taskData = [readHandle availableData]) && [taskData length])
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
The problem about the method is that it only output 4096 bytes data. So I using while loop to get more data, modify the method like this：
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData; //Added.
NSData *taskData = nil;
while ((taskData = [readHandle availableData]) && [taskData length]) {
[allData appendData:taskData];
}
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
Then problem occurs: the program is blocking in the while loop and can not perform the following statements. I ensure that allData is what I wanted, but after appending the last data chunk, it is blocking.
Could you give me some solutions? Thanks.
Your while() loop effectively blocks further notifications, causing the whole program to block waiting for something to flush the buffer.
You should readInBackgroundAndNotify, then pull off availableBytes on each notification, appending it to your NSMutableData (which is likely held in an instance variable). When you handle the notification, don't attempt to wait for more data or do any kind of a while loop. The system will notify you when more data is available.
I.e. the system pushes data to you, you do not pull data from the system.
Ahh... OK. You should still only pull data when there is data available. Your while() loop is doing that. Not enough coffee. My bad.
The final block is most likely because your external process is not closing the pipe; no EOF is received and, thus, the program is waiting forever for more data that never arrives.
Either:
make sure the background task exits
detect when you've received enough data and terminate the process
If you are doing some kind of conversion program (say, tr) where you write data on the processes standard input, then you might need to close the standard input pipe.
I have a situation that I receive a byte data through Web Services request and want to write it to a file on my iOS device. I used to append all data (till end of data) in a memory variable and at the end writing the data using NSStream to a file in my iOS device using method:
stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent
It works fine for small size of data, but the problem is if I am receiving data via web services it could be a big chunk (couple MBs) and I don't want to collect all in memory to write it to the file, to make it efficent I think I have to switch to NSFileHandle to write data in a small chunk size to the same file in several times. Now my question is what is the best approach to do this? I mean how can I do write to the file in BACKGROUND using NSFileHandle? I use code like this:
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
if(![[NSFileManager defaultManager] fileExistsAtPath:newFilePath ]) {
[[NSFileManager defaultManager] createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
}
but with passing a data size of 1-2 Mb to above method, do I need to make it running in background? FYI I'm writing in main thread.
Maybe you can try Grand Central Dispatch.
I spent some time trying it, bellow is my way to do it.
According to Apple's document, if our program need executing only one task at a time, we should create a "Serial Dispatch Queue".So, first declare a queue as iVar.
dispatch_queue_t queue;
create a serial dispatch queue in init or ViewDidLoad using
if(!queue)
{
queue = dispatch_queue_create("yourOwnQueueName", NULL);
}
When data occurs, call your method.
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
NSFileManager *fileManager = [[NSFileManager alloc] init];
if(![fileManager fileExistsAtPath:newFilePath ]) {
[fileManager createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
self.fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
dispatch_async( queue ,
^ {
// execute asynchronously
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
});
}
At last, we need to release the queue in ViewDidUnload or dealloc
if(queue)
{
dispatch_release(queue);
}
I combine these code with ASIHttp, and it works.
Hope it helps.