I am trying to watch files in a directory to determine when files are opened/accessed. I thought FileSystemWatcher would do the trick using the event Changed.
Problem is that some applications do not create a lock on the file they open/access or change either the date modified or date accessed (even after fsutil behavior set disablelastaccess 0). Notepad for example. Apparently is makes a copy of the file in memory and plays with it there until you save it. Nor does it update the Date Accessed.
How can I monitor a directory of files and be notified when a file is simply opened/accessed by any program (e.g. Notepad)? Files may be opened from another computer, not necessarily on the computer running the "watcher".
I found lots of similar questions but did not see one focusing on file "access".
This is quite normal. Updating an existing file is quite dangerous since it can cause irretrievable data loss. A disk error (like disk full) while writing is very bad news. The common algorithm used:
rename the original file
write a new file using the original name
no error: delete the renamed file
error: delete the new file, rename original file back
Clearly this doesn't cause a Changed event to be raised, no file was changed.
Sorry, I didn't read the question well enough. There is no notification whatsoever for an app just opening a file for reading. FSW can only detect changes to the file system. There is no ready alternative either, this requires a custom file system filter driver that snoops on driver requests. Like the kind that SysInternals' ProcMon utility uses. I'm not aware of such a driver ready for use in a C# program, you can't write them in C# either. This just isn't a common requirement.
Related
I have to read the MFT file of a running Windows (XP or higher) and through it to reach the HD sectors that held the contents ($DATA) of a specific file that exists on the machine.
The problem is that between the time of reading the MFT until the fetching of the relevant sectors and reading them, the file system structure can vary and the locations may not be relevant anymore.
Is there a way to "freeze" the system for a certain time? Perhaps guarantee that there will not be changes for this file? Lock a specific file in order to make it not moving between sectors? (Including due to optimizations and changes in indirect)
Of course I would prefer not to copy the entire hard disk and to work statically since it's a slow operation that would disallow normal use of the system at this time. Needless to say, I don't want to use the API functions of the OS or to write a driver.
I'd simply open the file, requesting read/write access, with read share mode. If you succeed to open the file, you're guaranteed that data will not change until you close the handle. See https://msdn.microsoft.com/en-us/library/windows/desktop/hh449422%28v=vs.85%29.aspx
If you want to achieve that on files that are already opened and locked by different processes, that's entirely different story and I believe you have to write own filter driver.
If the file location in the system varies, it will be accordingly reflected in the MFT. So instead of trying to stop any activity for the file you can simply compare the MFT info before and after reading the file. Unless you are de-fragmenting or deleting contents of the file the file storage structure will not change. Additions to files do not affect the consistency of data that you read. So if this is your scenario, you can just go ahead with the above method.
I am using FSEventsFramework to monitor directories for changes. I was wondering if my logic here is sound, for detecting old file name and new name after a rename.
I think on rename, both events are in the same callback.
Old file name will have event id X-1
New file name will have even id X
Is this true?
Thanks
As far as I experienced it, your assumption is not always true. On FSEventStreamCreate you will pass a latency as well as a flag for the meaning of the latency kFSEventStreamCreateFlagNoDefer. Therefore the events may or may not come in the same callback. Furthermore there are different ways the file can be renamed. Some Filesystem-APIs actually rename the file while keeping the inode like mv, others like NSDocument create a new inode. Sometimes you will receive kFSEventStreamEventFlagItemRenamed in the callback, sometimes not.
EDIT: Alternatives to FSEvents are Kernel Queues and NSFileCoordinator
FSEvents are somehow not fully documented in the API-Docs. Have a look at the header file FSEvent.h, there is more to read there.
Then to easily see what is going on for a rename, run your app and do some renaming with Finder, with terminal mv as well as in a document based app using the small triangle right to the filename.
My app is monitoring a "hot" folder somewhere on the local filesystem for newly added files to push to a network location. I'm running into a problem when very large files are being written into the hot folder: the file system event notifying me of changes in the hot folder will fire well before the file completes writing. When my app tries to upload the file, it mis-reads the file size as the current number of copied bytes, not the eventual total number of bytes.
Things I've tried:
NSURL getResourceValue:forKey:error: to read NSURLAllocatedFileSizeKey (same value as NSURLFileSizeKey while the file is being written).
NSFileManager attributesOfItemAtPath:error: to look at NSFileBusy (always NO).
I can't seem to find any mechanism short of repeatedly polling a file for its size to determine if the file is finished copying and can be uploaded.
There aren't great ways to do this.
If you can be certain that the writer is using NSFileCoordinator, then you can also use that to coordinate your access to the file.
Likewise, if you're sure that the writer has opted in to advisory locking, you could try to open the file for shared access by calling open() with the O_SHLOCK and O_NONBLOCK flags. If you succeed, then there are no other descriptors open for exclusive access. You can either use the file descriptor you've got or close it and then use some other API to access the file.
However, if you can't be sure of any of those, then your best bet may be to set a timer to repeatedly check the file's metadata (size, date modified, etc.). Only when you see that it has stopped changing over a reasonable time interval (2 seconds, maybe) would you attempt to access it (and cancel the timer).
You might want to do all three. Wait for the file's metadata to settle down, then use a NSFileCoordinator to read from the file. When it calls your reader block, use open() with O_SHLOCK | O_NONBLOCK to make sure there are no other processes which have exclusive access to it.
You need some form of coordinated file locking.
fcntl() and flock() are common functions for this.
Read up on it first.
Then see what options you have.
If you can control the code base of those other processes, all the better.
The problem with really large files is that what's changed or changing inside them is opaque and isn't always at the end.
Good processes should generally be doing atomic writes. (Write to a temp file then swap it out) but if these files are actually databases then you will want to look at using the db's server app for this sort of thing.
If the files are wrappers containing other files then it gets extra messy as those contents might have dependencies on one another to be in a usable state.
Imagine there are 3 or more independent locations where a file can be modified. These locations communicate to each other through email or mail (direct flash drive restoration). Though there is a big room for flow - to make simultaneous editing to the file and screw up things, this client won't change too much. He rather call everyone that he is working on the last update or tell the other guys that he is waiting for third guy's last update. Anyway, at some point after several exchanges, due to one of participants unintentional error THE LAST VERSION of the file eventually gets mixed up. From this point everyone searches for the last version BY LOOKING THE CONTENT of the file.
This client wants to have a central location (he has actually, that is his PC's some location) and let everybody (including himself) copy any new or suspected new file to this location but prevent file's last version being copied. From this location he has to easily copy, send or open the file and work.
So, here is my concept (2 steps):
step 1: I made an ad to the main application where this file is created or edited. This ad prompts the user to give a version number to the file with every invoked save command from the editing application. In fact the file can be re-saved multiple times but not considered modified (file attributes creation, save etc. do not have great meaning here). This said the user can cancel my ad-in but have saved the file, not saving a new file version.
step 2: multiple solutions:
solution A: I'm thinking to have a folder/file watch and prevent the last version of the file being overwritten. As you know, FileSystemWatcher will fire the change/delete etc., events AFTER FACT so, I have to back copy overwritten file after the fact (w/ some tricks).
solution B: have a database to store all version of files and built-in some shell extension to extract/view files from the database. Move all copied/pasted files to the database (my program folder) and restore latest file in working folder after watcher fires change/delete event.
solution 3: find out built-in windows tools (API etc.) to greatly rely on it with some programming.
Any ideas?
Thanks in advance.
Source code: vb.net
We are using File.Move() method to move the file from source to destination locaion.
But the destination location is being monitored by one tool, whenever we are moving files to the destination location, it will pick up the file and process it. The issue here when we try to move huge volume file like around 5GB file, the tool is immediately picking up the file and try to process it before the move operation is complete and send failure notice to all the users.. After again successfully moving the file completely, it picks up the same and process it sucessfully this time and send successful notice this time.
We can't have control over the tool which is monitoring the destination folder because it is a third party tool. However we want to find out the alternative option to place a lock over the destination foler like ReadWrite access till the move operation completes so that 3rd party will not be able to pick up or try to access that file.
Pls help us.
Not sure if it works, but you might be able to make the following work with directories as well:
FileOpen(1, "c:\file.ext", OpenMode.Binary)
Lock(1)
'Do something with file here
Unlock(1)
FileClose(1)
Reference and example here
I hope it helps.
First, I agree with #hometoast, sometimes tools like this just look for specific file extensions so you can copy in with a different file extension and then rename.
But barring that, download the file to a temp location and then Move the file into the dir getting watched. A Move does not recopy the file contents but just updates its pointers in the filesystem. Should be atomic.