I'm working on iOS project that supports iTunes file sharing feature. The goal is realtime tracking incoming/changed data's.
I'm using (kinda modified) DirectoryWatcher class from Apple's sample code
and also tried this source code.
The data is NSBundle (*.bundle) and some bundles are in 100-500 MB ranges, depends on its content, some video/audio stuff. The bundles has xml based descriptor file in it.
The problem is any of these codes above fires notification or whatever else when the data just started copying and but not when the copy/change/remove process finished completely.
Tried next:
checking file attributes:
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
BOOL fileBusy = [[fileAttrs objectForKey:NSFileBusy] boolValue];
looking for the fileSize changes:
dispatch_async(_checkQueue, ^{
for (NSURL *contURL in tempBundleURLs) {
NSInteger lastSize = 0;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
NSInteger fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
do {
lastSize = fileSize;
[NSThread sleepForTimeInterval:1];
fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
NSLog(#"doing job");
} while (lastSize != fileSize);
NSLog(#"next job");
}
);
any other solutions?
The solution above works great for bin files, but not for .bundle (as .bundle files are directory actually). In order to make it work with .bundle, you should iterate each file inside .bundle
You can use GCD's dispatch sources mechanism - using it you can observe particular system events (in your case, this is vnode type events, since you're working with file system).
To setup observer for particular directory, i used code like this:
- (dispatch_source_t) fileSystemDispatchSourceAtPath:(NSString*) path
{
int fileDescr = open([path fileSystemRepresentation], O_EVTONLY);// observe file system events for particular path - you can pass here Documents directory path
//observer queue is my private dispatch_queue_t object
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE, fileDescr, DISPATCH_VNODE_ATTRIB| DISPATCH_VNODE_WRITE|DISPATCH_VNODE_LINK|DISPATCH_VNODE_EXTEND, observerQueue);// create dispatch_source object to observe vnode events
dispatch_source_set_registration_handler(source, ^{
NSLog(#"registered for observation");
//event handler is called each time file system event of selected type (DISPATCH_VNODE_*) has occurred
dispatch_source_set_event_handler(source, ^{
dispatch_source_vnode_flags_t flags = dispatch_source_get_data(source);//obtain flags
NSLog(#"%lu",flags);
if(flags & DISPATCH_VNODE_WRITE)//flag is set to DISPATCH_VNODE_WRITE every time data is appended to file
{
NSLog(#"DISPATCH_VNODE_WRITE");
NSDictionary* dict = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:nil];
float size = [[dict valueForKey:NSFileSize] floatValue];
NSLog(#"%f",size);
}
if(flags & DISPATCH_VNODE_ATTRIB)//this flag is passed when file is completely written.
{
NSLog(#"DISPATCH_VNODE_ATTRIB");
dispatch_source_cancel(source);
}
if(flags & DISPATCH_VNODE_LINK)
{
NSLog(#"DISPATCH_VNODE_LINK");
}
if(flags & DISPATCH_VNODE_EXTEND)
{
NSLog(#"DISPATCH_VNODE_EXTEND");
}
NSLog(#"file = %#",path);
NSLog(#"\n\n");
});
dispatch_source_set_cancel_handler(source, ^{
close(fileDescr);
});
});
//we have to resume dispatch_objects
dispatch_resume(source);
return source;
}
I found two rather reliable (i.e. not 100% reliable but reliable enough for my needs) approaches, which only work in conjunction with polling the contents of the directory:
Check NSURLContentModificationDateKey. While the file is being transferred, this value is set to the current date. After transfer has finished, it is set to the value the original file had: BOOL busy = (-1.0 * [modDate timeintervalSinceNow]) < pollInterval;
Check NSURLThumbnailDictionaryKey. While the file is being transferred, this value is nil, afterwards it cointains a thumbnail, but probably only for file types from which the system can produce a thumbnail. Not a problem for me cause I only care about images and videos, but maybe for you. While this is more reliable than solution 1, it hammers the CPU quite a bit and may even cause your app to get killed if you have a lot of files in the import directory.
Dispatch sources and polling can be combined, i.e. when a dispatch source detects a change, start polling until no busy files are left.
Related
I know that the file exists, because I can download it, but I need to check to see whether it exists. I have tried using
[NSFileManager contentsOfDirectoryAtPath:error:]
but it gives me null. I don't understand why that is because I can still download the files that I'm looking for. Maybe it's an incorrect URL, but the URL that I'm using is the one that I printed at creation of my UIDocument that I'm looking for. Maybe I'm using the wrong method?
EDIT:
I have also tried using NSMetadataQuery, and I can get it to give back notifications, but it doesn't ever have results even though I can explicitly download the files I'm looking for.
To find files in iCloud, you use NSMetadataQuery. That will find both files that have already been downloaded as well as files that are in the user's account but which haven't been downloaded to the local device yet. Using NSFileManager will, at best, only find files that have already been downloaded.
You set it up with something like this:
NSMetadataQuery *query = [[NSMetadataQuery alloc] init];
[self setMetadataQuery:query];
[query setSearchScopes:#[NSMetadataQueryUbiquitousDataScope]];
[query setPredicate:[NSPredicate predicateWithFormat:#"%K LIKE '*'", NSMetadataItemFSNameKey]];
You'll want to observe NSMetadataQueryDidStartGatheringNotification, NSMetadataQueryDidUpdateNotification, and probably NSMetadataQueryDidFinishGatheringNotification. Then start the query:
[query startQuery];
With that done, you'll get notifications as the query discovers iCloud files. The notifications will include instances of NSMetadataItem, which you can use to get information like file size, download status, etc.
Use a metadata query - here is some sample code
/*! Creates and starts a metadata query for iCloud files
*/
- (void)createFileQuery {
[_query stopQuery];
if (_query) {
[_query startQuery];
}
else {
_query = [[NSMetadataQuery alloc] init];
[_query setSearchScopes:[NSArray arrayWithObjects:NSMetadataQueryUbiquitousDocumentsScope, NSMetadataQueryUbiquitousDataScope, nil]];
// NSString * str = [NSString stringWithFormat:#"*.%#",_fileExtension];
NSString *str = #"*";
[_query setPredicate:[NSPredicate predicateWithFormat:#"%K LIKE %#", NSMetadataItemFSNameKey, str]];
NSNotificationCenter* notificationCenter = [NSNotificationCenter defaultCenter];
[notificationCenter addObserver:self selector:#selector(fileListReceived) name:NSMetadataQueryDidFinishGatheringNotification object:_query];
[notificationCenter addObserver:self selector:#selector(fileListReceived) name:NSMetadataQueryDidUpdateNotification object:_query];
[_query startQuery];
}
}
/*! Gets called by the metadata query any time files change. We need to be able to flag files that
we have created so as to not think it has been deleted from iCloud.
*/
- (void)fileListReceived {
LOG(#"fileListReceived called.");
NSArray* results = [[_query results] sortedArrayUsingComparator:^(NSMetadataItem* firstObject, NSMetadataItem* secondObject) {
NSString* firstFileName = [firstObject valueForAttribute:NSMetadataItemFSNameKey];
NSString* secondFileName = [secondObject valueForAttribute:NSMetadataItemFSNameKey];
NSComparisonResult result = [firstFileName.pathExtension compare:secondFileName.pathExtension];
return result == NSOrderedSame ? [firstFileName compare:secondFileName] : result;
}];
//FLOG(#" results of query are %#", results);
for (NSMetadataItem* result in results) {
NSURL* fileURL = [result valueForAttribute:NSMetadataItemURLKey];
NSString* fileName = [result valueForAttribute:NSMetadataItemDisplayNameKey];
NSNumber* percentDownloaded = [result valueForAttribute:NSMetadataUbiquitousItemPercentDownloadedKey];
NSNumber *isDownloaded = nil;
NSNumber *isDownloading = nil;
NSError *er;
[fileURL getResourceValue: &isDownloaded forKey:NSURLUbiquitousItemIsDownloadedKey error:&er];
[fileURL getResourceValue: &isDownloading forKey:NSURLUbiquitousItemIsDownloadingKey error:&er];
FLOG(#" Found file %#", fileName);
}
}
From the docs:
In iOS, actively download files when required. Items in iCloud but not
yet local are not automatically downloaded by iOS; only their metadata
is automatically downloaded. The initial download of new iCloud-based
documents requires your attention and careful design in your app.
After you explicitly download such an item, the system takes care of
downloading changes arriving from iCloud.
Consider keeping track of
file download status as part of your iOS app’s model layer. Having
this information lets you provide a better user experience: you can
design your app to not surprise users with long delays when they want
to open a document that is not yet local. For each file (or file
package) URL provided by your app’s metadata query, get the value of
the NSURLUbiquitousItemIsDownloadedKeykey by calling the NSURL method
getResourceValue:forKey:error:. Reading a file that has not been
downloaded can take a long time, because the coordinated read
operation blocks until the file finishes downloading (or fails to
download).
For a file (or file package) that is not yet local, you can initiate
download either when, or before, the user requests it. If your app’s
user files are not large or great in number, one strategy to consider
is to actively download all the files indicated by your metadata
query. For each file (or file package) URL provided by the query, make
the corresponding item local by calling the NSFileManager method
startDownloadingUbiquitousItemAtURL:error:. If you pass this method a
URL for an item that is already local, the method performs no work and
returns YES.
Update: iOS7 should use NSURLUbiquitousItemDownloadingStatusKey instead of NSURLUbiquitousItemIsDownloadedKey.
Is there a faster file system API that I can use if I only need to know if a file is a folder/symlink and its size. I'm currently using [NSFileManager attributesOfItemAtPath...] and only NSFileSize and NSFileType.
Are there any bulk filesystem enumeration APIs I should be using? I suspect this could be faster without having to jump in and out of user code.
My goal is to quickly recurse through directories to get a folders true file size and currently calling attributesOfItemAtPath is my 95% bottleneck.
Some of the code I'm currently using:
NSDictionary* properties = [fileManager attributesOfItemAtPath:filePath error:&error];
long long fileSize = [[properties objectForKey:NSFileSize] longLongValue];
NSObject* fileType = [[properties objectForKey:NSFileType isEqual:NSFileTypeDirectory];
If you want to get really hairy, the Mac OS kernel implements a unique getdirentriesattr() system call which will return a list of files and attributes from a specified directory. It's messy to set up and call, and it's not supported on all filesystems (!), but if you can get it to work for you, it can speed things up significantly.
There's also a closely related searchfs() system call which can be used to rapidly search for files. It's subject to most of the same gotchas.
You can use stat and lstat. Take a look at this answer for calculating directory size.
CPU raises with attributesOfItemAtPath:error:
Whether it's faster or not I'm not certain, but NSURL will give you this information via getResourceValue:forKey:error:
NSError * e;
NSNumber * isDirectory;
BOOL success = [URLToFile getResourceValue:&isDirectory
forKey:NSURLIsDirectoryKey
error:&e];
if( !success ){
// error
}
NSNumber * fileSize;
BOOL success = [URLToFile getResourceValue:&fileSize
forKey:NSURLFileSizeKey
error:&e];
You might also find it convenient to wrap this up if you don't really care about the error:
#implementation NSURL (WSSSimpleResourceValueRetrieval)
- (id)WSSResourceValueForKey: (NSString *)key
{
id value = nil;
BOOL success = [self getResourceValue:&value
forKey:key
error:nil];
if( !success ){
value = nil;
}
return value;
}
#end
This is given as the substitute for the deprecated File Manager function FSGetCatalogInfo(), which is used in a solution in an old Cocoa-dev thread that Dave DeLong gives the thumbs up to.
For the enumeration part, the File System Programming Guide has a section "Getting the Contents of a Directory in a Single Batch Operation", which discusses using contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:
i am developing an iPad application witch has a functionality downloading books. The books size are about 180 Mo. The books are in server and the have the extension .zip. I download the book (.zip) then i unzip it and then remove the .zip. I am doing like this :
- (BOOL)downloadBookWithRequest:(BookDownloadRequest*)book
{
if (![book isValid])
{
NSLog(#"Couldn't launch download since request had missing parameter");
return NO;
}
if ([self bookIsCurrrentlyDownloadingWithID:book.ID]) {
NSLog(#"Book already downloaded");
return NO;
}
ASIHTTPRequest *download = [[ASIHTTPRequest alloc] initWithURL:book.URL];
download.userInfo = book.dictionary;
download.downloadDestinationPath = [self downloadPathForBookID:book.ID];
download.downloadProgressDelegate = self.downloadVC.downloadProgress;
download.shouldContinueWhenAppEntersBackground = YES;
[self.downloadQueue addOperation:download];
[download release];
// Update total requests
self.requestsCount++;
[self refreshDownloadsCount];
if(self.downloadQueue.isSuspended)
[self.downloadQueue go];
[self.downloadVC show];
return YES;
}
- (void)requestFinished:(ASIHTTPRequest*)request
{
NSString *bookStoragePath = [[BooksManager booksStoragePath] stringByAppendingPathComponent:request.downloadDestinationPath.lastPathComponent.stringByDeletingPathExtension];
NSString *bookZipPath = request.downloadDestinationPath;
// Tell not to save the zip file into iCloud
[BooksManager addSkipBackupAttributeToItemAtPath:bookZipPath];
NSFileManager *fileManager = [NSFileManager defaultManager];
NSError *removeExistingError = nil;
if ([fileManager fileExistsAtPath:bookStoragePath])
{
[fileManager removeItemAtPath:bookStoragePath error:&removeExistingError];
if (removeExistingError)
{
[self bookDidFailWithRequest:request errorMessageType:DownloadErrorTypeFILE_ERROR];
NSLog(#"ERROR: Couldn't remove existing book to unzip new download (%#)", removeExistingError);
} else
NSLog(#"INFO: Removed existing book to install new download");
}
ZipArchive* zip = [[ZipArchive alloc] init];
if([self isCompatibleWithFileAtPath:bookZipPath] && [zip UnzipOpenFile:bookZipPath])
{
BOOL unzipSucceeded = [zip UnzipFileTo:bookStoragePath overWrite:YES];
if (!unzipSucceeded)
{
[self bookDidFailWithRequest:request errorMessageType:DownloadErrorTypeFILE_ERROR];
NSLog(#"ERROR: Couldn't unzip file %#\n to %#",bookZipPath,bookStoragePath);
} else {
[self bookDidInstallWithRequest:request];
NSLog(#"INFO: Successfully unziped downloaded file");
}
[zip UnzipCloseFile];
}
else
{
[self bookDidFailWithRequest:request errorMessageType:DownloadErrorTypeFILE_ERROR];
NSLog(#"ERROR: Unable to open zip file %#\n",bookZipPath);
}
[self removeZipFileAtPath:bookZipPath];
[zip release];
}
-(BOOL) removeZipFileAtPath:(NSString*) bookZipPath {
NSFileManager *fileManager = [NSFileManager defaultManager];
if ([fileManager fileExistsAtPath:bookZipPath])
{
NSError *removeZipFileError = nil;
[fileManager removeItemAtPath:bookZipPath error:&removeZipFileError];
if (removeZipFileError) {
NSLog(#"ERROR: Couldn't remove existing zip after unzip (%#)", removeZipFileError);
return NO;
}
else {
NSLog(#"INFO: Removed zip downloaded after unzip");
return YES;
}
}
return NO;
}
My Problem is : This code is working fine with iPhone 4/iPhone 4s/ iPad 2G /iPad3G, but it crash with an iPad 1st Generation (when unzipping the book) and the crash reporter says that the are Memory warning.
How question is, how i can optimize this code to avoid the memory warning and avoid the crash ? Thanks for your answers;
Edit : I have found that the problem is caused by this portion of code :
NSData *bookData = [[NSData alloc]initWithContentsOfFile:bookPath];
the bookPath is the path to the .zip ( about 180 Mo) and when i am in iPad 1G this line crash my application i.e. : i receive memory warnings and the system kill the App. Du you know how i can avoid this. I an using this line to calculate the MD5 of the book (.zip)
I have a category in NSData like this :
#import <CommonCrypto/CommonDigest.h>
#implementation NSData(MD5)
- (NSString*)MD5
{
// Create byte array of unsigned chars
unsigned char md5Buffer[CC_MD5_DIGEST_LENGTH];
// Create 16 byte MD5 hash value, store in buffer
CC_MD5(self.bytes, self.length, md5Buffer);
// Convert unsigned char buffer to NSString of hex values
NSMutableString *output = [NSMutableString stringWithCapacity:CC_MD5_DIGEST_LENGTH * 2];
for(int i = 0; i < CC_MD5_DIGEST_LENGTH; i++)
[output appendFormat:#"%02x",md5Buffer[i]];
return output;
}
How i can avoid the crash ? thanks
EDIT:
So, it seems that the culprit is loading into memory the whole file in order to calculate its MD5 hash.
The solution to this would be calculating the MD5 without having to load into memory the whole file. You can give a look at this post explaining how to compute efficiently an MD5 or SHA1 hash, with the relative code. Or if you prefer, you can go directly to github and grab the code.
Hope it helps.
OLD ANSWER:
You should inspect your app, especially the ZipArchive class, for memory leaks or not-released memory. You can use Instruments' Leaks and Memory Allocation tools to profile your app.
The explanation of the different behavior between iPad1 and the rest of devices may lay with their different memory footprint, as well as with different memory occupation states of the devices (say, you iPad 1 has less free memory when you run the app then the iPad 2 because of the state other apps you ran on the iPad 1 left the device in). You might think of rebooting the iPad 1 to inspect its behavior out of a fresh start.
In any case, besides the possible explanation of the different behaviors, the ultimate cause is how your app manages memory and Instruments is the way to go.
I don't agree with Sergio.
You are saying the app crashes when you init the NSData object with a 180mb zip archive.
Well it's natural you run out of memory, since the 1st gen iPad has half the memory of the 2nd gen... (256MB vs 512)
The solution is to split the zip archive in smaller parts and process them one by one.
I have a situation that I receive a byte data through Web Services request and want to write it to a file on my iOS device. I used to append all data (till end of data) in a memory variable and at the end writing the data using NSStream to a file in my iOS device using method:
stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent
It works fine for small size of data, but the problem is if I am receiving data via web services it could be a big chunk (couple MBs) and I don't want to collect all in memory to write it to the file, to make it efficent I think I have to switch to NSFileHandle to write data in a small chunk size to the same file in several times. Now my question is what is the best approach to do this? I mean how can I do write to the file in BACKGROUND using NSFileHandle? I use code like this:
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
if(![[NSFileManager defaultManager] fileExistsAtPath:newFilePath ]) {
[[NSFileManager defaultManager] createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
}
but with passing a data size of 1-2 Mb to above method, do I need to make it running in background? FYI I'm writing in main thread.
Maybe you can try Grand Central Dispatch.
I spent some time trying it, bellow is my way to do it.
According to Apple's document, if our program need executing only one task at a time, we should create a "Serial Dispatch Queue".So, first declare a queue as iVar.
dispatch_queue_t queue;
create a serial dispatch queue in init or ViewDidLoad using
if(!queue)
{
queue = dispatch_queue_create("yourOwnQueueName", NULL);
}
When data occurs, call your method.
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
NSFileManager *fileManager = [[NSFileManager alloc] init];
if(![fileManager fileExistsAtPath:newFilePath ]) {
[fileManager createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
self.fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
dispatch_async( queue ,
^ {
// execute asynchronously
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
});
}
At last, we need to release the queue in ViewDidUnload or dealloc
if(queue)
{
dispatch_release(queue);
}
I combine these code with ASIHttp, and it works.
Hope it helps.
Is there anyway to do Files Handling in Objective-C? I am just trying to do simple read and write and can use 'c' but i am force to use Objective-C classes for that :#. I am looking into NSInputStream, but its going over my head. Is there any tutorial which explains how to use NSInputStream?
I had trouble with basic file i/o when I first hit it in Obj-C as well. I ended up using NSFileHandle to get C style access to my file. Here's a basic example:
// note: myFilename is an NSString containing the full path to the file
// create the file
NSFileManager *fManager = [[NSFileManager alloc] init];
if ([fManager createFileAtPath:myFilename contents:nil attributes:nil] != YES) {
NSLog(#"Failed to create file: %#", myFilename);
}
[fManager release]; fManager = nil;
// open the file for updating
NSFileHandle *myFile = [NSFileHandle fileHandleForUpdatingAtPath:myFilename];
if (myFile == nil) {
NSLog(#"Failed to open file for updating: %#", myFilename);
}
// truncate the file so it is guaranteed to be empty
[myFile truncateFileAtOffset:0];
// note: rawData is an NSData object
// write data to a file
[myFile writeData:rawData];
// close the file handle
[myFile closeFile]; myFile = nil;
If all you need to do is really simple I/O, you can just tell an object to initialize itself from, or write itself to, a filesystem path or URL. This works with several Foundation classes, including NSString, NSData, NSArray, and NSDictionary among others.
Try starting out by looking at the following two NSString methods:
- initWithContentsOfFile:encoding:error:
- writeToFile:atomically:encoding:error:
I find apple's guides short and to the point.
http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Streams/Articles/ReadingInputStreams.html