I am working with multiple processes that write to the same directory.
I have a directory dir1/
My process creates a file a.txt under dir1/. However the other process creates a-temp1.txt and renames it to a.txt. I don't have control over the other process since that code comes from a library. Can I prevent a-temp.txt from being renamed?
There's nothing you can do that the other process can't undo. Your best hope (other than changing your program to work sanely) is that the other process doesn't try too hard to do the rename. That is, it tries the simple approach and gives up if that fails.
In particular, you can set the UF_IMMUTABLE flag on either file and that will prevent one from being renamed to replace the other. You can set the flag using chflags(). Using Cocoa, you could also use [someURL setResourceValue:#YES forKey:NSURLIsUserImmutableKey error:NULL].
Keep in mind that you won't be able to change the file in any other way, either, until that flag is removed. If the other process is determined to rename the file, it has permission to remove the flag just like your process does.
Also keep in mind that a system such as this is inherently race-prone.
You really ought to use separate names for the files, or separate directories, or ditch that library that doesn't give you the control you need.
Set the user immutable flag chflags(...,uchg). This will keep the other process from changing your file unless it takes action to clear the bit. Of course I don't know how the other process will react to you putting things in it's way, but that wasn't the question.
You can use chflags() on an HFS+ (Mac OS X) file system to set the UF_APPEND attribute. (Do a man 2 chflags.) That will permit appending to the file, but not deleting or renaming, even by the same user.
You can, but it unlikely will solve your problem. I strongly suspect this is an X-Y problem, and almost certainly the correct solution is to redesign some part of this system entirely, probably by changing your file names, using unique temporary files, moving to another directory, or reworking the usage of the library (libraries only do what callers tell them to do; and libraries are just code anyway). You shouldn't try to defeat another process; you're all working for the same user.
All that said, sure, you can prevent your own userid from renaming over file. Just deny yourself permission. You can modify the file:
chmod 400 a.txt
That says that you can read the file but may not write it. However, if you already have an open file handle, you may continue to use it (so you can keep writing to the file, even though another process running as the same user may not).
Similarly, you may change permissions on the directory:
chmod 500 .
This would prevent the rename because file names are kept in the directory.
Related
This is going to be a question with a lot of hypotheticals, but it's been on my mind for a while now and I finally want to get some perspectives on how to tackle this "issue". For the sake of the question, I'll make up an example requirement of how the program I want to make would work on a conceptual level without too many specifics.
The Problem
I want to create a program to keep track of miscellaneous info for files and folders. This miscellaneous info can be anything from comments, authors, to more specific info like the original source of the file (a URL for example), categories, tags, and more. All this info is kept track of in an SQLite database.
Now... how would you create a connection to the file (or folder) to the database? Whatever file is added to the program, the file should continue to operate on an independent level from the program, meaning you should be able to edit, copy, move, rename or do anything else with the file you would usually do with your OS of choice - even deleting it.
You should even be able to archive it, zip it, upload it somewhere or do other things that temporarily or permanently removes the file from your system, without losing the connection to the database. The program itself doesn't actually ever touch the files themselves, unless to generate a new entry in the database, but obviously, there should be some kind of reference in the file to a database entry in the program.
Yes, I know that if you delete the file, you would have a dead entry in the database. For now, just treat this as an unfortunate reality that can't be solved unless you incorporate the file more closely into the program.
Possible solutions and why I decided against them
Reference inside Filename
Probably the most obvious choice, you could just have a reference inside the filename to point to a database entry, for example by including the id at the start of the filename:
#1 my-example-file.txt
#12814 this-is-one-of-many-files.txt
Obviously, that goes against what I established earlier, as you would be restricted from freely renaming the file. You would always have to keep in mind to not mess with the id inside the filename, or else the connection to your program is broken. Unfortunately, that is the best bet I currently have, but I would like to avoid using that approach if possible.
Alternate Data Streams (ADS)
A pretty cool feature I recently discovered that's available on NTFS file systems, ADS allows you to store different streams of data for your files, to grossly simplify it. You could attach a data stream to your file that saves the id for the database entry in the program, and a regular user would never be able to mess directly with that.
However, since this is a feature reserved for specific file systems, there's some ugly side effects to ADS, as you can easily lose that part of the file by:
moving/copying it to a file system that doesn't support ADS, such as the file systems most often used in removable drives
uploading it to a cloud then later downloading it
moving it to another OS that might not support ADS or treats it in an unexpected way
zipping it
Thus I can't really rely on ADS either.
Now I'm trying to use .dir-locals.el for my own projects.
However it is saved at the end of init.el whenever I choose to save it permanently.
I'd like to change it to an another seperate file - eg ~/mydirlocals.el.
Please let me know what could be the solution for this.
PS : I've already tried to change custom file to a seperate one. But unfortunately it saved my dir-local variables with other custom variables.
I want to avoid this and save my dir-locals variables in a completely seperate file.
You can't use the customize interface to update .dir-locals.el files. Customize is for your own config. If you want to edit/update a directory-local variables file, you need to either edit that file directly, or use M-x add-dir-local-variable.
The latter command will prompt you for the details. Note that no default value is offered at the prompt for the value of the variable, but that you can type M-n or <down> to obtain the variable's value in the current buffer.
Note also that the command does not ask which directory the variables are local to -- it will create/update a .dir-locals.el file in the default directory for the current buffer. Issuing the command from a dired buffer for the intended directory is a safe approach, naturally, but you may wish to do so from a buffer in the mode for which you wish to add variables -- that way the default suggestion for the mode, and the current values of the variables in question, will be more useful to you.
(If there is no file of the appropriate type in the directory, you can always C-xC-f a new/unsaved buffer of an appropriate filename, use add-dir-local-variable as many times as necessary, and then when you're done just kill any new buffers you created without saving them.)
That all said, I'm still not 100% sure what your requirement is, as your question is a little confused; but you may also like to know that you can use directory-local variables without a .dir-locals.el file at all, as you can alternatively configure them entirely in your init file.
See https://www.emacswiki.org/emacs/DirectoryVariables for details and examples of that.
Disclaimer: I haven't used dir-local vars, but I have used file-local vars. I'm guessing dir-locals work the same.
The local variables aren't getting saved at the end of init.el or custom.el, the safe values are. As in, Emacs doesn't trust them by default, unless they match some sort of predicate indicating they're OK. This is a good policy, because file and dir locals can cause Emacs to run arbitrary code just by opening a file. When you apply permanently, you're telling Emacs that that value is safe; it basically just makes a predicate that matches the exact value and stores that with customize.
If you want to prevent the prompt (and thus the saving), you need mark the variables as safe with your own predicate.
For example, I set
(put 'adaptive-wrap-extra-indent 'safe-local-variable 'integerp)
which means that adaptive-wrap-extra-indent is OK as long as it's an integer. I know this is OK because I added that variable to the adaptive-wrap package (though I didn't know about safe locals at the time; I submitted a bug to fix it, which appears to be ignored). Clearly you can use any predicate, including (lambda (x) t), though I'd recommend against that.
I recently recovered a 1.5TB external HDD that crashed. The program I used to recover the files was Active Undelete Enterprise, it's excellent. When the files were successfully recovered they were all saved with a .efs extension so files looked like mydocument.docx.efs. At first I thought they were encrypted and needed to be decrypted, I spent 10 mins on it and realized I just need to remove the .efs from the entire filename and the mydocument.docx works perfectly. Problem is now I have over 55,000 files within hundreds of folders where I need to simply remove the .efs after each file. Does anyone know how to do this?
From a command prompt window, navigate to the top level directory where these files reside.
Type the command
DIR /S/B >>filelist.txt
This command will give you a bare format file listing of the current directory plus all nested subdirectories without any extraneous information. The list will be contained in the text file named "filelist.txt" or whatever else you choose to call it. I would then use this text file in a text editor to convert every line of text from, for example,
C:\Users\dlucas\.gimp-2.8\mathmap\file1.png.efs
to
rename c:\Users\dlucas\.gimp-2.8\mathmap\file1.png.efs file1.png
to give a simple example of a file that I just found on my system using this method.
You will need to use a text editor with a columnar editing capability since you have to modify som many files. Old programmer's editors such as CodeWright made this really simple while modern editors such as Eclipse or Notepad++ make this a little more difficult and may require a columnar editing plugin, depending on version. You basically have to make a columnar copy of all of the text in the file, and then paste the copy off to the far right - far enough that a second column of filenames and paths won't overwrite any of the existing file names and paths. You can then use columnar editing features to select and delete the path names of the text in the 2nd column since the rename command requires that the 2nd argument be simply the base filename and extension without the path information. You can use the columnar editing features to prepend every line with "RENAME ". If you attempt to do this without columnar editing features, you will find it slow going!
An alternate way to do this is to use a command formed from a "regular expression" to create the rename command. If you are not familiar with "regular expressions", ask a programmer friend as this is not an easy topic to learn from scratch. If you are familiar with regular expressions, this is probably the simplest way to perform this task. I haven't used them in many years and no longer recall the exact syntax to use or I would tell you myself.
Regardless of what kind of editor you use, the goal is to turn this ASCII file list of paths and filenames into a batch file (simply rename file1.txt to file1.bat when you are finished editing). You can then run the batch file by typing file1.bat at a command prompt.
I have just run into this same problem myself using the same really wonderful tool that you used. I am writing this while waiting for the undelete program to finish. That it restores files with this extra extension seems very anti-intuitive so I will look for an option to make it not do this when it finishes. If I find one, I will post a new answer here that is more specific to this tool. Otherwise, I am going to have rename all kazillion files just as you had to.
You experienced this problem because the disk that you recovered your files to "does not support encryption", according to the Active# UNDELETE documentation. The documentation offers no further explanation of what kind of disks support encryption, etc.
They offer a Decrypt command that restores the file's proper names as a post processing step. Unfortunately, this requires that you "include" each and every file to be decrypted, with no support for wildcards and parsing subdirectories so that is a non-starter, in my opinion given that both of us have hundreds of thousands of files to be renamed.
I did find that by selecting a normal fixed (non-removable) hard drive as the destination of the recovery effort, that the resulting files do not end up encrypted (i.e., they are recovered with the proper file name and extension). I originally chose a large USB based flash drive and the files were stored in their "encrypted" state (not really encrypted, but possibly potentially so and thus they give the .efs extension). Of course, this meant that I had to run the command all over again after switching to a regular hard drive (takes about 16 hours to recover 80GB worth of files due to presence of many sector CRC errors).
I guess everybody read about the recent issues with the new iOS 5.
e.g. at http://iPhoneIncubator.com/blog/data-management/local-file-storage-in-ios-5
My Magazine-App holds about 1k html/css/image/video files for every issue and I want to know if anything is missing.
So my question is how to check all required files are present a very smart way?
I would expect iOS to clean caches atomically; i.e. iOS will either clean you app's cache or it will not. It shouldn't partially remove files from your cache. However, I don't believe this is documented behaviour, so it's a bad idea to assume this.
When you download your content, you have to know what to download for each issue, right? So you already have some kind of manifest file that references every file that is needed by each magazine. You should save this manifest locally so that you can refer to it when checking for the existence of files. You can still save this in the caches directory, if it's missing, you know your cache has been at least partially cleaned.
Also, I don't believe there's any guarantee that iOS won't clean caches while your application is running. As well as checking at the point where the user opens the magazine, you should check when your application becomes active after it has been put in the background, and your code should be robust in the face of missing files (i.e. pop up a warning message instead of crashing).
You could zip all the files relating to each issue in an archive (objective-c wrapper for zlib here, to make life easier for you). That way you can make sure that the system either deletes your issue or not, without having to worry about individual files.
In a POSIX environment, I want to remove a file from disk, but calculate its checksum before removing it, to make sure it was not changed. Is locking enough? Should I open it, unlink, calculate checksum, and then close it (so the OS can remove its inode)? Is there any way to ensure no other process has an open file descriptor on the file?
To give a bit of context, the code performs synchronization of files across hosts, and there's an opportunity for data loss if a remote host removes a file but the file is being changed locally.
Your proposal of open,unlink,checksum,close won't work as is, because you'll be stuck if the checksum doesn't match (there is no POSIX-portable way of creating a link to a file given by a file descriptor). A better variant is rename,checksum,unlink,close, which lets you undo the rename or redo the copy if the checksum doesn't match. You'll still need to think of what you want to do if a third program has recreated the file in the meantime.
POSIX offers only cooperative locks. If you have control over the programs that may modify the file, make sure they use locks; if that's not an option, you're stuck without locks.
There is no portable way to see what (or even whether) processes have opened a file. On most Unix systems, lsof will show you, but this is not universal, not robust (a program could open the files just after lsof has finished looking), and incomplete (if the files are exported over NFS, there may be no way to know about active clients).
You may benefit from looking at what other synchronization programs are doing, such as rsync and unison.