Trying to set up a simple backup solution for my wife's computer. Have a volume on my server upstairs mounted locally using OSX automount, so it should just be a simple
rsync -a sourceDir targetDir
When I look at the files it syncs over though, all metadata is lost on jpg files. The created date is preserved on the file and the modified date ends up being the timestamp when the rsync runs, but I can't imagine why EXIF data (Device, exposure etc) would disappear when it should just be a straight file copy. Hoping someone has run into this before and can shed some light on it.
This can't be a rsync problem, there should be something else going on. rsync just does a binary copy from source to destination, the most probable explanation is a simple user error (e.g. you copied from the wrong source directory, source files where already without EXIF data, and so on).
For normal copies on reliable hardware, rsync is without doubt the best tool for the job, especially considering the huge amount of filesystems it has to cope with.
There are some corner cases where rsync may not behave as it should, at least with default parameters. For example, right now I'm investigating on an issue where, copying to a "not-so-reliable" USB drive, rsync continued to copy happily even when the drive disconnected from USB and the device disappeared.
Related
I like rsync. I can see what files will be deleted first. But what happens if during the backup, a sector of the source disk fails? Files could be deleted from the destination that should not be. However, if I check the log file for all deletion files first, then use the log file as instructions to rsync, then a source disk failure during backup should result in a lower probability of data loss.
I've read the man page and have to conclude that the answer is no. If not rsync, then what?
You can mitigate source disk failure risk using
--delete-after receiver deletes after transfer, not during
That will not delete files if a IO error is produced during copy.
But for ensuring integrity of your backup, I think the right way is using:
--only-write-batch=FILE like --write-batch but w/o updating destination
That will write diffs into a file. Once batch is created, you move it to destination machine, and apply diffs with:
--read-batch=FILE read a batched update from FILE
Recently, I encountered an unknown problem causing particular folder in NTFS folder to be corrupted in multiple computers. I need to detect if the folder is corrupted and perform actions like relocate the folder or send notifications. However I do not know how to do it yet. The normal APIs, like OpenFile/CreateFile seems to be malfunctioning with the corrupted folder and I can not use them to determine if a Folder is corrupted. So I plan to parse MTF structure and check for problem directly.
Therefore, I began to study the NTFS MFT structure. I found that $Volume has a dirty flag to determine if a drive needs chkdisk. But it is not directly related to file corruption and will be set if Windows is shutdown unexpectedly. DI failed to find a particular flag or anything to determine if an INDEX or FILE is corrupted in MFT structure.
Could I know if there is a way to determine a corrupted NTFS Folder?
Any help is appreciated!
I found 3 things that are related with NTFS disk corruption issues. It is incomplete; however, without updated NTFS source code, it is very hard to find out what Microsoft was really doing in chkdisk. I will just post what I found it in case if anyone needs to know it.
1 Dirty Flag in $BadClus of "File Records" section
If the flag in $BadClus is set to ON, then the operating system will perform a disk scan at boot-up. I believe NTFS module would set the flag to ON if encounter disk operation.
2 "BAAD" in identification field of a file record
If there is something wrong with file record, for example USA/USN unmatched, then MFT may replace "FILE" with "BAAD" in identification field of a file record structure. It can be used to identify corrupted file/directory quickly.
3 Compare USA/USN in every FILE/INDX record
Both FILE/INDX structure contains USA/USN for corruption check. Scan through the system and compare USA and USN could help you discover corruption issue.
I use tar --listed-incremental=... to perform incremental backups on my computer, and for the most part it works like a charm. However, it doesn't seem to play well with my old FAT partition. It thinks that all the files have changed and backs up everything. I read somewhere that tar incremental backups rely on inodes, and that's why it doesn't work well with FAT. Is there a way to fake it, or force it to just use name and modification date instead of inode?
I have been working on an algorithm in Python, and I was using Vim to edit this file. I opened it up, did a save, and it came up with an Error something like it occasionally does:
"WARNING: YOUR FILE CANNOT BE SAVED! ALL CHANGES WILL BE LOST! CANNOT WRITE THE FILE!"
As this happens occasionally, I did what I normally do, and I hit :q! to quit without writing any changes. No harm, no foul. When I looked at my file, everything had been erased! Everything!
I talked around the office, and it seems that the nfs mount was full, and so that was why I couldn't save anything. There was a huge script generating a ton of data, which caused the mount to be full temporarily. I believe the NFS mount is from NetApp. I found 2 files in my current directory.
One was last saved two days ago, and one was today. They are in the format of:
.nfs.xxxxxxxxxxx
When I try to attempt to open up this file, I see some of my code, here and there, splattered among unknown characters. Apparently, this must be a binary representation of the state of the file.
Is there any way to recover this file from this NFS mount? If there is a shortcut to recover this file in Emacs, I will switch to Emacs from vim!
So, I did find a way to recover the file. I found two ways, in fact. Since it was on a NetApp NFS mount, I was able to use the snapshots feature. When you are in a directory just do
ls .snapshot
And this will pull up any snapshots that your system administrators have set. For us, we have an hourly.0, hourly.1, and nightly.0, and nightly.1 backups. So, we can go back two days, and in the same day, we can go back one hour (the current hour, and the previous).
The other way was to rename the file to a vim swap file like this.
mv .nfs.xxx my_vim_file.cpp.swp
vim my_vim_file.cpp.swp
Then attempt to open it up in Vim, and it should ask you if you want to Recover the swap file, say yes, and it should be back!
Apparently your Netapp uses NFS to mount its volumes (as opposed to iSCSI, for example). Generally, each VM is stored on a unique volume (aka datastore) on the Netapp filer. To find out the volumes and snapshots, and then restore a snapshot, here are the commands to execute at the command line:
# list all volumes, snapshots are taken of volumes
vol status
# list the snapshots available for a particular volume
snap list <vol_name>
# restore a snapshot, nightly.1 for example
snap restore <vol_name> nightly.1
That's it. All that's left is to turn the VM back on and see if you've restored far back enough. If not, then do another "snap restore" but with an older snapshot.
Note that this procedure assumes your administrator didn't disable snapshots (Netapp has a snapshot schedule by default) and that the Netapp is licensed for snaprestore (use the "license" command to verify). This procedure can further be simplified if you have the Netapp OnCommand System Manager, which is a GUI for managing the Netapp. Reverting a snapshot in the GUI is simple:
Go to Storage > Volumes > click on a volume > click on Snapshot Copies (at the bottom)
Choose a snapshot and restore
I'm a huge fan of robocopy and use it extensively to copy between various servers I need to update.
Lately I've been archiving to an Amazon S3 account that I access via a mapped drive using JungleDisk. I then robocopy my files from local PC to S3.
Sometimes I get a very strange 'Incorrect function' error message in robocopy and the file fails to copy. I've tried xcopy and straightforward copy and paste between file explorer windows. In each case I get some variation of the 'Incorrect function' or 'Illegal MS-DOS function' and the file will never copy.
I delete the target but to no avail.
Any ideas?
Don't know if you're allowed to answer your own questions but I think I've fixed it...
I found this in the jungledisk support forums
The quick solution is to zip the
files, delete the original, then unzip
the files because zip can't handle
extended attributes. Another solution
is to move them to a FAT filesystem,
then move again to NTFS filesystem,
because FAT don't manage extended
attributes.
In both cases the result is the
deletion of extended attributes, and
the files can be moved to the
jungledisk.
The files can have extended attributes
for different reasons, expecially
migrations from other filesystems: in
my case was the migration of a CVS
repository from a ext2 filesystem to
NTFS.
Seems to have worked for me...
I've had similar issues from both OSX and linux. At first I was not concerned by it but then it occurred to me these issues could result in potential data contamination or backup failure. So I have abandon JungleDisk for everything except my lightweight work.
Zipping/taring files was not an option for me because of the size of my data set. With this approach you have to upload your entire data set each and every time.
I'm not sure which attributes you refer to but could you robocopy with the /COPY:DT switch to strip off the attributes?