Is there a faster file system API that I can use if I only need to know if a file is a folder/symlink and its size. I'm currently using [NSFileManager attributesOfItemAtPath...] and only NSFileSize and NSFileType.
Are there any bulk filesystem enumeration APIs I should be using? I suspect this could be faster without having to jump in and out of user code.
My goal is to quickly recurse through directories to get a folders true file size and currently calling attributesOfItemAtPath is my 95% bottleneck.
Some of the code I'm currently using:
NSDictionary* properties = [fileManager attributesOfItemAtPath:filePath error:&error];
long long fileSize = [[properties objectForKey:NSFileSize] longLongValue];
NSObject* fileType = [[properties objectForKey:NSFileType isEqual:NSFileTypeDirectory];
If you want to get really hairy, the Mac OS kernel implements a unique getdirentriesattr() system call which will return a list of files and attributes from a specified directory. It's messy to set up and call, and it's not supported on all filesystems (!), but if you can get it to work for you, it can speed things up significantly.
There's also a closely related searchfs() system call which can be used to rapidly search for files. It's subject to most of the same gotchas.
You can use stat and lstat. Take a look at this answer for calculating directory size.
CPU raises with attributesOfItemAtPath:error:
Whether it's faster or not I'm not certain, but NSURL will give you this information via getResourceValue:forKey:error:
NSError * e;
NSNumber * isDirectory;
BOOL success = [URLToFile getResourceValue:&isDirectory
forKey:NSURLIsDirectoryKey
error:&e];
if( !success ){
// error
}
NSNumber * fileSize;
BOOL success = [URLToFile getResourceValue:&fileSize
forKey:NSURLFileSizeKey
error:&e];
You might also find it convenient to wrap this up if you don't really care about the error:
#implementation NSURL (WSSSimpleResourceValueRetrieval)
- (id)WSSResourceValueForKey: (NSString *)key
{
id value = nil;
BOOL success = [self getResourceValue:&value
forKey:key
error:nil];
if( !success ){
value = nil;
}
return value;
}
#end
This is given as the substitute for the deprecated File Manager function FSGetCatalogInfo(), which is used in a solution in an old Cocoa-dev thread that Dave DeLong gives the thumbs up to.
For the enumeration part, the File System Programming Guide has a section "Getting the Contents of a Directory in a Single Batch Operation", which discusses using contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:
Related
OSX Yosemite introduced a very handy attribute on NSURL: NSURLDocumentIdentifierKey.
Quoting from the documentation:
NSURLDocumentIdentifierKey
The document identifier returned as an NSNumber (read-only).
The document identifier is a value assigned by the kernel to a file or directory. This value is used to identify the document regardless of where it is moved on a volume. The identifier persists across system restarts. It is not transferred when the file is copied, but it survives "safe save” operations. For example, it remains on the path to which it was assigned, even after calling the replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error: method. Document identifiers are only unique within a single volume. This property is not supported by all volumes.
Available in OS X v10.10 and iOS 8.0.
Unfortunately, the value seems to be mostly nil (except rare examples that seem completely disconnected one to the other).
In particular, this code will throw an exception at the last line (tested on Yosemite 10.10.3):
NSFileManager *fileManager = [NSFileManager defaultManager];
NSArray *attributesFlags = #[NSURLNameKey, mNSURLDocumentIdentifierKey];
NSDirectoryEnumerator *en = [fileManager enumeratorAtURL:[NSURL URLWithString:NSHomeDirectory()]
includingPropertiesForKeys:attributesFlags
options:NSDirectoryEnumerationSkipsHiddenFiles
errorHandler:^BOOL(NSURL *url, NSError *error) {
NSAssert(NO, #"An error has occured");
return YES;
}];
for(NSURL *URL in en) {
NSNumber *documentID = nil;
NSError *error = nil;
BOOL result = [URL getResourceValue:&documentID forKey:NSURLDocumentIdentifierKey error:&error]; \
NSAssert(result == YES && error==nil, #"Unable to read property. Error: %#", error); \
NSLog(#"Processing file: %#", URL);
// This will break most of the times
NSAssert(documentID != nil, #"Document ID should not be nil!!");
}
Perhaps I misunderstood the documentation but it seems to me NSURLDocumentIdentifierKey should be available on every file on disk.
I filed a bug with Apple on this issue and got a feedback on my report. As of today, the information on tracking the DocumentIdentifier is not part of the documentation yet, but the ticket is still open.
The missing information is, that the filesystem does not track the DocumentIdentifier by default. You'll have to enable the tracking by setting a flag on each file that you want to track using chflags with the UF_TRACKED flag.
The following script will print the DocumentIdentifier for a file:
https://gist.github.com/cmittendorf/fac92272a941a9cc64d5
And this script will enable tracking the DocumentIdentifier:
https://gist.github.com/cmittendorf/b680d1a03aefa08583d7
Apparently Yosemite assigns a DocumentIdentifier to a file only when it knows something is trying to track its identity (like Versions or iCloud).
I don't see any way to talk to the kernel and tell it to start tracking files you're interested on. I hope this changes in future releases, since the API has been made public on OS X 10.10 and it's mostly useless at this point.
This issue still exists in macOS 10.14. It probably won't change.
The work-around is to get the inode from NSFileManager, like this:
NSFileManager *fmgr = [NSFileManager defaultManager];
NSDictionary *attributes = [fmgr attributesOfItemAtPath:url.path error:nil;
if (attributes != nil) {
NSNumber *inode = [attributes objectForKey:NSFileSystemFileNumber];
...
}
I know that the file exists, because I can download it, but I need to check to see whether it exists. I have tried using
[NSFileManager contentsOfDirectoryAtPath:error:]
but it gives me null. I don't understand why that is because I can still download the files that I'm looking for. Maybe it's an incorrect URL, but the URL that I'm using is the one that I printed at creation of my UIDocument that I'm looking for. Maybe I'm using the wrong method?
EDIT:
I have also tried using NSMetadataQuery, and I can get it to give back notifications, but it doesn't ever have results even though I can explicitly download the files I'm looking for.
To find files in iCloud, you use NSMetadataQuery. That will find both files that have already been downloaded as well as files that are in the user's account but which haven't been downloaded to the local device yet. Using NSFileManager will, at best, only find files that have already been downloaded.
You set it up with something like this:
NSMetadataQuery *query = [[NSMetadataQuery alloc] init];
[self setMetadataQuery:query];
[query setSearchScopes:#[NSMetadataQueryUbiquitousDataScope]];
[query setPredicate:[NSPredicate predicateWithFormat:#"%K LIKE '*'", NSMetadataItemFSNameKey]];
You'll want to observe NSMetadataQueryDidStartGatheringNotification, NSMetadataQueryDidUpdateNotification, and probably NSMetadataQueryDidFinishGatheringNotification. Then start the query:
[query startQuery];
With that done, you'll get notifications as the query discovers iCloud files. The notifications will include instances of NSMetadataItem, which you can use to get information like file size, download status, etc.
Use a metadata query - here is some sample code
/*! Creates and starts a metadata query for iCloud files
*/
- (void)createFileQuery {
[_query stopQuery];
if (_query) {
[_query startQuery];
}
else {
_query = [[NSMetadataQuery alloc] init];
[_query setSearchScopes:[NSArray arrayWithObjects:NSMetadataQueryUbiquitousDocumentsScope, NSMetadataQueryUbiquitousDataScope, nil]];
// NSString * str = [NSString stringWithFormat:#"*.%#",_fileExtension];
NSString *str = #"*";
[_query setPredicate:[NSPredicate predicateWithFormat:#"%K LIKE %#", NSMetadataItemFSNameKey, str]];
NSNotificationCenter* notificationCenter = [NSNotificationCenter defaultCenter];
[notificationCenter addObserver:self selector:#selector(fileListReceived) name:NSMetadataQueryDidFinishGatheringNotification object:_query];
[notificationCenter addObserver:self selector:#selector(fileListReceived) name:NSMetadataQueryDidUpdateNotification object:_query];
[_query startQuery];
}
}
/*! Gets called by the metadata query any time files change. We need to be able to flag files that
we have created so as to not think it has been deleted from iCloud.
*/
- (void)fileListReceived {
LOG(#"fileListReceived called.");
NSArray* results = [[_query results] sortedArrayUsingComparator:^(NSMetadataItem* firstObject, NSMetadataItem* secondObject) {
NSString* firstFileName = [firstObject valueForAttribute:NSMetadataItemFSNameKey];
NSString* secondFileName = [secondObject valueForAttribute:NSMetadataItemFSNameKey];
NSComparisonResult result = [firstFileName.pathExtension compare:secondFileName.pathExtension];
return result == NSOrderedSame ? [firstFileName compare:secondFileName] : result;
}];
//FLOG(#" results of query are %#", results);
for (NSMetadataItem* result in results) {
NSURL* fileURL = [result valueForAttribute:NSMetadataItemURLKey];
NSString* fileName = [result valueForAttribute:NSMetadataItemDisplayNameKey];
NSNumber* percentDownloaded = [result valueForAttribute:NSMetadataUbiquitousItemPercentDownloadedKey];
NSNumber *isDownloaded = nil;
NSNumber *isDownloading = nil;
NSError *er;
[fileURL getResourceValue: &isDownloaded forKey:NSURLUbiquitousItemIsDownloadedKey error:&er];
[fileURL getResourceValue: &isDownloading forKey:NSURLUbiquitousItemIsDownloadingKey error:&er];
FLOG(#" Found file %#", fileName);
}
}
From the docs:
In iOS, actively download files when required. Items in iCloud but not
yet local are not automatically downloaded by iOS; only their metadata
is automatically downloaded. The initial download of new iCloud-based
documents requires your attention and careful design in your app.
After you explicitly download such an item, the system takes care of
downloading changes arriving from iCloud.
Consider keeping track of
file download status as part of your iOS app’s model layer. Having
this information lets you provide a better user experience: you can
design your app to not surprise users with long delays when they want
to open a document that is not yet local. For each file (or file
package) URL provided by your app’s metadata query, get the value of
the NSURLUbiquitousItemIsDownloadedKeykey by calling the NSURL method
getResourceValue:forKey:error:. Reading a file that has not been
downloaded can take a long time, because the coordinated read
operation blocks until the file finishes downloading (or fails to
download).
For a file (or file package) that is not yet local, you can initiate
download either when, or before, the user requests it. If your app’s
user files are not large or great in number, one strategy to consider
is to actively download all the files indicated by your metadata
query. For each file (or file package) URL provided by the query, make
the corresponding item local by calling the NSFileManager method
startDownloadingUbiquitousItemAtURL:error:. If you pass this method a
URL for an item that is already local, the method performs no work and
returns YES.
Update: iOS7 should use NSURLUbiquitousItemDownloadingStatusKey instead of NSURLUbiquitousItemIsDownloadedKey.
I'm working on iOS project that supports iTunes file sharing feature. The goal is realtime tracking incoming/changed data's.
I'm using (kinda modified) DirectoryWatcher class from Apple's sample code
and also tried this source code.
The data is NSBundle (*.bundle) and some bundles are in 100-500 MB ranges, depends on its content, some video/audio stuff. The bundles has xml based descriptor file in it.
The problem is any of these codes above fires notification or whatever else when the data just started copying and but not when the copy/change/remove process finished completely.
Tried next:
checking file attributes:
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
BOOL fileBusy = [[fileAttrs objectForKey:NSFileBusy] boolValue];
looking for the fileSize changes:
dispatch_async(_checkQueue, ^{
for (NSURL *contURL in tempBundleURLs) {
NSInteger lastSize = 0;
NSDictionary *fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
NSInteger fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
do {
lastSize = fileSize;
[NSThread sleepForTimeInterval:1];
fileAttrs = [[NSFileManager defaultManager] attributesOfItemAtPath:[contURL path] error:nil];
fileSize = [[fileAttrs objectForKey:NSFileSize] intValue];
NSLog(#"doing job");
} while (lastSize != fileSize);
NSLog(#"next job");
}
);
any other solutions?
The solution above works great for bin files, but not for .bundle (as .bundle files are directory actually). In order to make it work with .bundle, you should iterate each file inside .bundle
You can use GCD's dispatch sources mechanism - using it you can observe particular system events (in your case, this is vnode type events, since you're working with file system).
To setup observer for particular directory, i used code like this:
- (dispatch_source_t) fileSystemDispatchSourceAtPath:(NSString*) path
{
int fileDescr = open([path fileSystemRepresentation], O_EVTONLY);// observe file system events for particular path - you can pass here Documents directory path
//observer queue is my private dispatch_queue_t object
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_VNODE, fileDescr, DISPATCH_VNODE_ATTRIB| DISPATCH_VNODE_WRITE|DISPATCH_VNODE_LINK|DISPATCH_VNODE_EXTEND, observerQueue);// create dispatch_source object to observe vnode events
dispatch_source_set_registration_handler(source, ^{
NSLog(#"registered for observation");
//event handler is called each time file system event of selected type (DISPATCH_VNODE_*) has occurred
dispatch_source_set_event_handler(source, ^{
dispatch_source_vnode_flags_t flags = dispatch_source_get_data(source);//obtain flags
NSLog(#"%lu",flags);
if(flags & DISPATCH_VNODE_WRITE)//flag is set to DISPATCH_VNODE_WRITE every time data is appended to file
{
NSLog(#"DISPATCH_VNODE_WRITE");
NSDictionary* dict = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:nil];
float size = [[dict valueForKey:NSFileSize] floatValue];
NSLog(#"%f",size);
}
if(flags & DISPATCH_VNODE_ATTRIB)//this flag is passed when file is completely written.
{
NSLog(#"DISPATCH_VNODE_ATTRIB");
dispatch_source_cancel(source);
}
if(flags & DISPATCH_VNODE_LINK)
{
NSLog(#"DISPATCH_VNODE_LINK");
}
if(flags & DISPATCH_VNODE_EXTEND)
{
NSLog(#"DISPATCH_VNODE_EXTEND");
}
NSLog(#"file = %#",path);
NSLog(#"\n\n");
});
dispatch_source_set_cancel_handler(source, ^{
close(fileDescr);
});
});
//we have to resume dispatch_objects
dispatch_resume(source);
return source;
}
I found two rather reliable (i.e. not 100% reliable but reliable enough for my needs) approaches, which only work in conjunction with polling the contents of the directory:
Check NSURLContentModificationDateKey. While the file is being transferred, this value is set to the current date. After transfer has finished, it is set to the value the original file had: BOOL busy = (-1.0 * [modDate timeintervalSinceNow]) < pollInterval;
Check NSURLThumbnailDictionaryKey. While the file is being transferred, this value is nil, afterwards it cointains a thumbnail, but probably only for file types from which the system can produce a thumbnail. Not a problem for me cause I only care about images and videos, but maybe for you. While this is more reliable than solution 1, it hammers the CPU quite a bit and may even cause your app to get killed if you have a lot of files in the import directory.
Dispatch sources and polling can be combined, i.e. when a dispatch source detects a change, start polling until no busy files are left.
We have a POS apartment leasing iPad app that is used to collect a lot of data about a user and their interests (with their knowledge of course).
We use RestKit to sync CoreData with the server, which is totally sweet.
I'm using text files in the cache directory to store a history of their interactions with a guest card, such that it can be submitted in case of error, or sent to us via email, to recreate any guest card in case of some sort of syncing issue.
Although these should be very small text files, probably around 1-3k, I feel as though I should eventually clear these from the cache directory. (As I type this, maybe its so small I shouldn't worry about it).
I was curious if there was any way to clear files from the cache directory after a set amount of time? Say 90 days or so?
Word on the street is that if you use the sanctioned NSCachesDirectory location, if the OS needs that space, it'll delete things that are in that directory. I've never actually seen it happen in practice, but I've heard such things (and it stands to reason, otherwise why bother having special, OS-sanctioned location for cache files.)
That said, this task sounds pretty straightforward. Just fire off a low priority GCD background block to iterate through the files in that directory and delete any whose creation date was > 90 days ago. This is really easy if you only care about how long ago the data was created (as opposed to the last time you accessed the data which is harder to ascertain without keeping track yourself.) NSFileManager is your friend here. Something like this ought to work:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0), ^{
NSFileManager* fm = [NSFileManager defaultManager];
NSMutableArray* urlsToDelete = [NSMutableArray array];
for (NSURL* dirUrl in [fm URLsForDirectory: NSCachesDirectory inDomains:NSUserDomainMask])
{
NSDirectoryEnumerator* dirEnum = [fm enumeratorAtURL: dirUrl
includingPropertiesForKeys: [NSArray arrayWithObject: NSFileModificationDate]
options: 0
errorHandler: ^(NSURL* a, NSError* b){ return (BOOL)YES; }];
NSURL* url = nil;
while ((url = [dirEnum nextObject]))
{
NSDate* modDate = [[dirEnum fileAttributes] objectForKey: NSFileModificationDate];
if (modDate && [[NSDate date] compare: [modDate dateByAddingTimeInterval: 60 * 60 * 24 * 90]] == NSOrderedDescending)
{
[urlsToDelete addObject: url];
}
}
}
for (NSURL* url in urlsToDelete)
{
[fm removeItemAtURL: url error: NULL];
}
});
To clarify, if you're looking for some mechanism by which to tell the OS 'delete this if I don't access it for more than 90 days' and have it keep track of this for you, I don't believe such a mechanism exists.
Is there anyway to do Files Handling in Objective-C? I am just trying to do simple read and write and can use 'c' but i am force to use Objective-C classes for that :#. I am looking into NSInputStream, but its going over my head. Is there any tutorial which explains how to use NSInputStream?
I had trouble with basic file i/o when I first hit it in Obj-C as well. I ended up using NSFileHandle to get C style access to my file. Here's a basic example:
// note: myFilename is an NSString containing the full path to the file
// create the file
NSFileManager *fManager = [[NSFileManager alloc] init];
if ([fManager createFileAtPath:myFilename contents:nil attributes:nil] != YES) {
NSLog(#"Failed to create file: %#", myFilename);
}
[fManager release]; fManager = nil;
// open the file for updating
NSFileHandle *myFile = [NSFileHandle fileHandleForUpdatingAtPath:myFilename];
if (myFile == nil) {
NSLog(#"Failed to open file for updating: %#", myFilename);
}
// truncate the file so it is guaranteed to be empty
[myFile truncateFileAtOffset:0];
// note: rawData is an NSData object
// write data to a file
[myFile writeData:rawData];
// close the file handle
[myFile closeFile]; myFile = nil;
If all you need to do is really simple I/O, you can just tell an object to initialize itself from, or write itself to, a filesystem path or URL. This works with several Foundation classes, including NSString, NSData, NSArray, and NSDictionary among others.
Try starting out by looking at the following two NSString methods:
- initWithContentsOfFile:encoding:error:
- writeToFile:atomically:encoding:error:
I find apple's guides short and to the point.
http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Streams/Articles/ReadingInputStreams.html