NSFileHandle & Writing Asynch to a file in iOS - objective-c

I have a situation that I receive a byte data through Web Services request and want to write it to a file on my iOS device. I used to append all data (till end of data) in a memory variable and at the end writing the data using NSStream to a file in my iOS device using method:
stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent
It works fine for small size of data, but the problem is if I am receiving data via web services it could be a big chunk (couple MBs) and I don't want to collect all in memory to write it to the file, to make it efficent I think I have to switch to NSFileHandle to write data in a small chunk size to the same file in several times. Now my question is what is the best approach to do this? I mean how can I do write to the file in BACKGROUND using NSFileHandle? I use code like this:
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
if(![[NSFileManager defaultManager] fileExistsAtPath:newFilePath ]) {
[[NSFileManager defaultManager] createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
}
but with passing a data size of 1-2 Mb to above method, do I need to make it running in background? FYI I'm writing in main thread.

Maybe you can try Grand Central Dispatch.
I spent some time trying it, bellow is my way to do it.
According to Apple's document, if our program need executing only one task at a time, we should create a "Serial Dispatch Queue".So, first declare a queue as iVar.
dispatch_queue_t queue;
create a serial dispatch queue in init or ViewDidLoad using
if(!queue)
{
queue = dispatch_queue_create("yourOwnQueueName", NULL);
}
When data occurs, call your method.
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
NSFileManager *fileManager = [[NSFileManager alloc] init];
if(![fileManager fileExistsAtPath:newFilePath ]) {
[fileManager createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
self.fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
dispatch_async( queue ,
^ {
// execute asynchronously
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
});
}
At last, we need to release the queue in ViewDidUnload or dealloc
if(queue)
{
dispatch_release(queue);
}
I combine these code with ASIHttp, and it works.
Hope it helps.

Related

NSTask: why program is blocking when read from NSPipe?

I use the NSTask to run shell command and output the data via NSPipe. At first, I using bellow method to read output data, it is no any problem.
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData = [[NSMutableData alloc] init];
NSData *taskData = nil;
if((taskData = [readHandle availableData]) && [taskData length])
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
The problem about the method is that it only output 4096 bytes data. So I using while loop to get more data, modify the method like this:
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData; //Added.
NSData *taskData = nil;
while ((taskData = [readHandle availableData]) && [taskData length]) {
[allData appendData:taskData];
}
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
Then problem occurs: the program is blocking in the while loop and can not perform the following statements. I ensure that allData is what I wanted, but after appending the last data chunk, it is blocking.
Could you give me some solutions? Thanks.
Your while() loop effectively blocks further notifications, causing the whole program to block waiting for something to flush the buffer.
You should readInBackgroundAndNotify, then pull off availableBytes on each notification, appending it to your NSMutableData (which is likely held in an instance variable). When you handle the notification, don't attempt to wait for more data or do any kind of a while loop. The system will notify you when more data is available.
I.e. the system pushes data to you, you do not pull data from the system.
Ahh... OK. You should still only pull data when there is data available. Your while() loop is doing that. Not enough coffee. My bad.
The final block is most likely because your external process is not closing the pipe; no EOF is received and, thus, the program is waiting forever for more data that never arrives.
Either:
make sure the background task exits
detect when you've received enough data and terminate the process
If you are doing some kind of conversion program (say, tr) where you write data on the processes standard input, then you might need to close the standard input pipe.

Threads are killing my app

Looking for some advice on stablizing my app. First some requirements - files with PII (personally identifying information) must be encrypted when on disk. Tumbnails and logos are in the custom TableViewCells (if available) and must be decrypted before display.
There are several layers of threading going on. There is a central function getFileData that checks to see if files are on the device or if the files need to be obtained from the network. I desire to keep the UI responsive and (I think) therein lies my problem.
Here is some code:
This is the workhorse method for processing files in my application. It decides where the file is decrypts it and hands it back to a callback:
-(void)fetchFileData:(UserSession *) session
onComplete: (void(^)(NSData* data)) onComplete
{
NSURL *url = [File urlForMail:self.fileId andSession:session];
//NSLog(#"File id: %#", self.fileId);
NSString *encryptionKey = session.encryptionKey;
dispatch_queue_t cryptoQ = dispatch_queue_create(FILE_CRYPTOGRAPHY_QUEUE, NULL);
dispatch_async(cryptoQ, ^(void){
// Get the file and d/encrypt it
NSError *error = nil;
if ([File fileExistsAtUrl:url] == YES) {
NSLog(#"file is on disk.");
NSData *localEncryptedFile = [File getDataForFile:url];
NSData *decryptedFile = [RNDecryptor decryptData:localEncryptedFile
withPassword:encryptionKey
error:&error];
onComplete(decryptedFile);
dispatch_release(cryptoQ);
} else {
//NSLog(#"File is not on disk");
NSDictionary *remoteFile = [session.apiFetcher getFileContent:self.fileId
andToken:session.token];
if (remoteFile && [[remoteFile objectForKey:#"success"] isEqualToString:#"true"]) {
NSData *remoteFileData = [remoteFile objectForKey:#"data"];
NSString *mimeType = [remoteFile objectForKey:#"mimeType"];
self.mimeType = mimeType;
NSData *encryptedData = [RNEncryptor encryptData:remoteFileData
withSettings:kRNCryptorAES256Settings
password:encryptionKey
error:&error];
[encryptedData writeToURL:url atomically:YES];
onComplete(remoteFileData);
dispatch_release(cryptoQ);
}
}
});
Here is an example of a getFileData caller:
+(void)loadThumbnailForMail: (NSNumber*)thumbnailId
session: (UserSession*)session
callback: (void(^)(NSData* data))callback
{
File *file = [File findFile:thumbnailId inContext:session.mailDatabase.managedObjectContext];
dispatch_queue_t fetchQ = dispatch_queue_create(FILE_FETCHER_QUEUE_LABEL, NULL);
dispatch_async(fetchQ, ^(void) {
if (file) {
[file fetchFileData:session onComplete:^(NSData *data) {
if (data && file.mimeType) {
callback(data);
}
}];
}
});
dispatch_release(fetchQ);
}
Here is an example of the TableViewCell that is calling loadThumbnailForMail:
-(void)loadAndShowThumbnailImage:(Mail*) mail
{
UIImage *placeHolder = [UIImage imageNamed:#"thumbnail_placeholder.png"];
[self.thumbnailImageForMail setImage:placeHolder];
dispatch_queue_t loaderQ = dispatch_queue_create(THUMBNAIL_FETCHER, NULL);
dispatch_async(loaderQ, ^ {
[File loadThumbnailForMail: mail.thumbnailId
session: [UserSession instance]
callback: ^(NSData *data) {
dispatch_async(dispatch_get_main_queue(), ^{
UIImage *thumbnailImage = [UIImage imageWithData:data];
[self.thumbnailImageForMail setImage:thumbnailImage];
});
}];
});
dispatch_release(loaderQ);
}
I think that my issue here is the callback in my loadThumbnailImage. If the user scrolls fast enough I suspect that there could be two threads trying to access the same TableViewCell
(MyCell *cell = (MyCell*)[tableView dequeueReusableCellWithIdentifier:CellTableIdentifier];)
It doesn't always happen right away but eventually, after some scrolling the tableView list of cells the app crashes with this: * Terminating app due to uncaught exception 'NSGenericException', reason: '* Collection <__NSCFSet: 0xde6a650> was mutated while being enumerated.'
I need to have the decrypted images in the cells and the first solution (above) does this for me when the images are available but causes the app to crash. I am wondering if some sort of in memory cache would help improve this if I put images in memory there when they were decrypted and checked that cache in loadAndShowThumbnailImage before I kick off all the threads to get and decrypt them.
Thoughts? I have been banging on this for a week now trying different things and would appreciate some fresh perspective.
Thanks.
Based on Justins link and subsequent research I ended up going in this direction: In a UITableView, best method to cancel GCD operations for cells that have gone off screen?

UIManagedDocument insert objects in background thread

This is my first question on Stack Overflow, so please excuse me if I'm breaking any etiquette. I'm also fairly new to Objective-C/app creation.
I have been following the CS193P Stanford course, in particular, the CoreData lectures/demos. In Paul Hegarty's Photomania app, he starts with a table view, and populates the data in the background, without any interruption to the UI flow. I have been creating an application which lists businesses in the local area (from an api that returns JSON data).
I have created the categories as per Paul's photo/photographer classes. The creation of the classes themselves is not an issue, it's where they are being created.
A simplified data structure:
- Section
- Sub-section
- business
- business
- business
- business
- business
- business
My application starts with a UIViewController with several buttons, each of which opens a tableview for the corresponding section (these all work fine, I'm trying to provide enough information so that my question makes sense). I call a helper method to create/open the URL for the UIManagedDocument, which was based on this question. This is called as soon as the application runs, and it loads up quickly.
I have a method very similar to Paul's fetchFlickrDataIntoDocument:
-(void)refreshBusinessesInDocument:(UIManagedDocument *)document
{
dispatch_queue_t refreshBusinessQ = dispatch_queue_create("Refresh Business Listing", NULL);
dispatch_async(refreshBusinessQ, ^{
// Get latest business listing
myFunctions *myFunctions = [[myFunctions alloc] init];
NSArray *businesses = [myFunctions arrayOfBusinesses];
// Run IN document's thread
[document.managedObjectContext performBlock:^{
// Loop through new businesses and insert
for (NSDictionary *businessData in businesses) {
[Business businessWithJSONInfo:businessData inManageObjectContext:document.managedObjectContext];
}
// Explicitly save the document.
[document saveToURL:document.fileURL
forSaveOperation:UIDocumentSaveForOverwriting
completionHandler:^(BOOL success){
if (!success) {
NSLog(#"Document save failed");
}
}];
NSLog(#"Inserted Businesses");
}];
});
dispatch_release(refreshBusinessQ);
}
[myFunctions arrayOfBusinesses] just parses the JSON data and returns an NSArray containing individual businessses.
I have run the code with an NSLog at the start and end of the business creation code. Each business is assigned a section, takes 0.006 seconds to create, and there are several hundred of these. The insert ends up taking about 2 seconds.
The Helper Method is here:
// The following typedef has been defined in the .h file
// typedef void (^completion_block_t)(UIManagedDocument *document);
#implementation ManagedDocumentHelper
+(void)openDocument:(NSString *)documentName UsingBlock:(completion_block_t)completionBlock
{
// Get URL for document -> "<Documents directory>/<documentName>"
NSURL *url = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
url = [url URLByAppendingPathComponent:documentName];
// Attempt retrieval of existing document
UIManagedDocument *doc = [managedDocumentDictionary objectForKey:documentName];
// If no UIManagedDocument, create
if (!doc)
{
// Create with document at URL
doc = [[UIManagedDocument alloc] initWithFileURL:url];
// Save in managedDocumentDictionary
[managedDocumentDictionary setObject:doc forKey:documentName];
}
// If the document exists on disk
if ([[NSFileManager defaultManager] fileExistsAtPath:[url path]])
{
[doc openWithCompletionHandler:^(BOOL success)
{
// Run completion block
completionBlock(doc);
} ];
}
else
{
// Save temporary document to documents directory
[doc saveToURL:url
forSaveOperation:UIDocumentSaveForCreating
completionHandler:^(BOOL success)
{
// Run compeltion block
completionBlock(doc);
}];
}
}
And is called in viewDidLoad:
if (!self.lgtbDatabase) {
[ManagedDocumentHelper openDocument:#"DefaultLGTBDatabase" UsingBlock:^(UIManagedDocument *document){
[self useDocument:document];
}];
}
useDocument just sets self.document to the provided document.
I would like to alter this code to so that the data is inserted in another thread, and the user can still click a button to view a section, without the data import hanging the UI.
Any help would be appreciated I have worked on this issue for a couple of days and not been able to solve it, even with the other similar questions on here. If there's any other information you require, please let me know!
Thank you
EDIT:
So far this question has received one down vote. If there is a way I could improve this question, or someone knows of a question I've not been able to find, could you please comment as to how or where? If there is another reason you are downvoting, please let me know, as I'm not able to understand the negativity, and would love to learn how to contribute better.
There are a couple of ways to this.
Since you are using UIManagedDocument you could take advantage of NSPrivateQueueConcurrencyType for initialize a new NSManagedObjectContext and use performBlock to do your stuff. For example:
// create a context with a private queue so access happens on a separate thread.
NSManagedObjectContext *context = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
// insert this context into the current context hierarchy
context.parentContext = parentContext;
// execute the block on the queue of the context
context.performBlock:^{
// do your stuff (e.g. a long import operation)
// save the context here
// with parent/child contexts saving a context push the changes out of the current context
NSError* error = nil;
[context save:&error];
}];
When you save from the context, data of the private context are pushed to the current context. The saving is only visible in memory, so you need to access the main context (the one linked to the UIDocument) and do a save there (take a look at does-a-core-data-parent-managedobjectcontext-need-to-share-a-concurrency-type-wi).
The other way (my favourite one) is to create a NSOperation subclass and do stuff there. For example, declare a NSOperation subclass like the following:
//.h
#interface MyOperation : NSOperation
- (id)initWithDocument:(UIManagedDocument*)document;
#end
//.m
#interface MyOperation()
#property (nonatomic, weak) UIManagedDocument *document;
#end
- (id)initWithDocument:(UIManagedDocument*)doc;
{
if (!(self = [super init])) return nil;
[self setDocument:doc];
return self;
}
- (void)main
{
NSManagedObjectContext *moc = [[NSManagedObjectContext alloc] init];
[moc setParentContext:[[self document] managedObjectContext]];
// do the long stuff here...
NSError *error = nil;
[moc save:&error];
NSManagedObjectContext *mainMOC = [[self document] managedObjectContext];
[mainMOC performBlock:^{
NSError *error = nil;
[mainMOC save:&error];
}];
// maybe you want to notify the main thread you have finished to import data, if you post a notification remember to deal with it in the main thread...
}
Now in the main thread you can provide that operation to a queue like the following:
MyOperation *op = [[MyOperation alloc] initWithDocument:[self document]];
[[self someQueue] addOperation:op];
P.S. You cannot start an async operation in the main method of a NSOperation. When the main finishes, delegates linked with that operations will not be called. To say the the truth you can but this involves to deal with run loop or concurrent behaviour.
Hope that helps.
Initially I was just going to leave a comment, but I guess I don't have the privileges for it. I just wanted to point out the UIDocument, beyond the change count offers
- (void)autosaveWithCompletionHandler:(void (^)(BOOL success))completionHandler
Which shouldn't have the delay I've experienced with updating the change count as it waits for a "convenient moment".

Huge memory footprint with ARC

App I'm working on uses ARC. I wanted it to process large files, so instead of loading files as a whole, I'm loading chunks of data using NSFileHandle readDataOfLength method. It's happening inside a loop which repeats until the whole file is processed:
- (NSString*)doStuff { // called with NSInvocationOperation
// now we open the file itself
NSFileHandle *fileHandle =
[NSFileHandle fileHandleForReadingFromURL:self.path
error:nil];
...
BOOL done = NO;
while(!done) {
NSData *fileData = [fileHandle readDataOfLength: CHUNK_SIZE];
...
if ( [fileData length] == 0 ) done = YES;
...
}
...
}
According to profiler, there are no memory leaks; however, my app eats a LOT of memory while it processes the file. My guess — autorelease comes only after I process the file. Can I fix it without switching to manual memory management?
Wrap the code within that loop with a autorelease pool.
while(!done)
{
#autoreleasepool
{
NSData *fileData = [fileHandle readDataOfLength: CHUNK_SIZE];
...
if ( [fileData length] == 0 )
{
done = YES;
}
...
}
};
readDataOfLength retuns autoreleased data and since you stick inside that loop and therefor its method, that autoreleased data that does not get released until your loop and the encapsulating method is done.

iTunes File Sharing app: realtime monitoring for incoming datas

I'm working on iOS project that supports iTunes file sharing feature. The goal is realtime tracking incoming/changed data's.
I'm using (kinda modified) DirectoryWatcher class from Apple's sample code
and also tried this source code.
The data is NSBundle (*.bundle) and some bundles are in 100-500 MB ranges, depends on its content, some video/audio stuff. The bundles has xml based descriptor file in it.
The problem is any of these codes above fires notification or whatever else when the data just started copying and but not when the copy/change/remove process finished completely.
Tried next:
checking file attributes:
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
BOOL fileBusy = [[fileAttrs objectForKey:NSFileBusy] boolValue];
looking for the fileSize changes:
dispatch_async(_checkQueue, ^{
for (NSURL *contURL in tempBundleURLs) {
NSInteger lastSize = 0;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
NSInteger fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
do {
lastSize = fileSize;
[NSThread sleepForTimeInterval:1];
fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
NSLog(#"doing job");
} while (lastSize != fileSize);
NSLog(#"next job");
}
);
any other solutions?
The solution above works great for bin files, but not for .bundle (as .bundle files are directory actually). In order to make it work with .bundle, you should iterate each file inside .bundle
You can use GCD's dispatch sources mechanism - using it you can observe particular system events (in your case, this is vnode type events, since you're working with file system).
To setup observer for particular directory, i used code like this:
- (dispatch_source_t) fileSystemDispatchSourceAtPath:(NSString*) path
{
int fileDescr = open([path fileSystemRepresentation], O_EVTONLY);// observe file system events for particular path - you can pass here Documents directory path
//observer queue is my private dispatch_queue_t object
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE, fileDescr, DISPATCH_VNODE_ATTRIB| DISPATCH_VNODE_WRITE|DISPATCH_VNODE_LINK|DISPATCH_VNODE_EXTEND, observerQueue);// create dispatch_source object to observe vnode events
dispatch_source_set_registration_handler(source, ^{
NSLog(#"registered for observation");
//event handler is called each time file system event of selected type (DISPATCH_VNODE_*) has occurred
dispatch_source_set_event_handler(source, ^{
dispatch_source_vnode_flags_t flags = dispatch_source_get_data(source);//obtain flags
NSLog(#"%lu",flags);
if(flags & DISPATCH_VNODE_WRITE)//flag is set to DISPATCH_VNODE_WRITE every time data is appended to file
{
NSLog(#"DISPATCH_VNODE_WRITE");
NSDictionary* dict = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:nil];
float size = [[dict valueForKey:NSFileSize] floatValue];
NSLog(#"%f",size);
}
if(flags & DISPATCH_VNODE_ATTRIB)//this flag is passed when file is completely written.
{
NSLog(#"DISPATCH_VNODE_ATTRIB");
dispatch_source_cancel(source);
}
if(flags & DISPATCH_VNODE_LINK)
{
NSLog(#"DISPATCH_VNODE_LINK");
}
if(flags & DISPATCH_VNODE_EXTEND)
{
NSLog(#"DISPATCH_VNODE_EXTEND");
}
NSLog(#"file = %#",path);
NSLog(#"\n\n");
});
dispatch_source_set_cancel_handler(source, ^{
close(fileDescr);
});
});
//we have to resume dispatch_objects
dispatch_resume(source);
return source;
}
I found two rather reliable (i.e. not 100% reliable but reliable enough for my needs) approaches, which only work in conjunction with polling the contents of the directory:
Check NSURLContentModificationDateKey. While the file is being transferred, this value is set to the current date. After transfer has finished, it is set to the value the original file had: BOOL busy = (-1.0 * [modDate timeintervalSinceNow]) < pollInterval;
Check NSURLThumbnailDictionaryKey. While the file is being transferred, this value is nil, afterwards it cointains a thumbnail, but probably only for file types from which the system can produce a thumbnail. Not a problem for me cause I only care about images and videos, but maybe for you. While this is more reliable than solution 1, it hammers the CPU quite a bit and may even cause your app to get killed if you have a lot of files in the import directory.
Dispatch sources and polling can be combined, i.e. when a dispatch source detects a change, start polling until no busy files are left.