I use tar --listed-incremental=... to perform incremental backups on my computer, and for the most part it works like a charm. However, it doesn't seem to play well with my old FAT partition. It thinks that all the files have changed and backs up everything. I read somewhere that tar incremental backups rely on inodes, and that's why it doesn't work well with FAT. Is there a way to fake it, or force it to just use name and modification date instead of inode?
Related
NOTE: This is not about how to use Git.. This is about under the hood working.
Can you explain how Git can checkout files so quickly from one branch to another?
For example, giving I have two branches, each branch has 10Gb of files...
I am able to checkout from one branch to the other in under a second.
When I checkout, the new files are on disk and ready to go.
However, if I was to delete 10gb and then restore 10gb of files it would take a few minutes.
I understand the basics of Git Objects and how Commit'ish, Trees and blobs work however, when checking out a diferent branch, the file needs to restored from the blobs and any existing untracked files removed. How is this happening so quickly???
I am assuming this is done using some form of HDD pointers..
I read over the Git code but my C skills are rusty.
Trying to set up a simple backup solution for my wife's computer. Have a volume on my server upstairs mounted locally using OSX automount, so it should just be a simple
rsync -a sourceDir targetDir
When I look at the files it syncs over though, all metadata is lost on jpg files. The created date is preserved on the file and the modified date ends up being the timestamp when the rsync runs, but I can't imagine why EXIF data (Device, exposure etc) would disappear when it should just be a straight file copy. Hoping someone has run into this before and can shed some light on it.
This can't be a rsync problem, there should be something else going on. rsync just does a binary copy from source to destination, the most probable explanation is a simple user error (e.g. you copied from the wrong source directory, source files where already without EXIF data, and so on).
For normal copies on reliable hardware, rsync is without doubt the best tool for the job, especially considering the huge amount of filesystems it has to cope with.
There are some corner cases where rsync may not behave as it should, at least with default parameters. For example, right now I'm investigating on an issue where, copying to a "not-so-reliable" USB drive, rsync continued to copy happily even when the drive disconnected from USB and the device disappeared.
What I want to do is to backup my personal data (on a Linux boy) possibly using rsync. I want to include only certain files (jpg,nb,pdf,..) and exclude everything else. This is easily possible with rsync. What I also want to do is to apply certain operation on those files I backup since I need to save some disk space. So my idea is re resize my jpg's, zip my pdf's,...
What do you think is the best approach to solve this (as easy as possible)?
Kind Regards, André
First question how much space do you have at the destination?
mkdir -p /storage/backups/`date +\%Y-\%m-\%d`-`date +\%A`/$host/$username
rsync -avz /storage/backups/`date --date=yesterday +\%Y-\%m-\%d`-`date--date=yesterday +\%A`/$host/$username/ /storage/backups/`date +\%Y-\%m-\%d`-`date +\%A`/$host/$username/
rsync -avz -e ssh --delete --exclude='....' /home/username/ /storage/backups/`date +\%Y-\%m-\%d`-`date +\%A`/$host/$username/
This will keep the backups nice and neat just use the include/exclude option of rsync to only sycn file types you want to back up or use the way i described in this post Local rsync inclusion/exclusion
I have a drupal site, and I am storing the codebase in a git repository. This seems to be working out well, but I'm also making changes to the database. I'm considering doing periodic dumps of the database and committing to git. I had a few questions about this.
If I overwrite the file, will git think it is a brand new file or will it recognize that it is an altered version of the same file.
Will this potentialy make my repo huge (the database is 16mb)
Can I zip this file? or will this mess Git up ... the zipped version is only 3mb
Any other suggestions?
If you have enough space, a non-compressed dump in source control is pretty handy because you can compare using a diff program what rows were added/modified/deleted.
Another solution is to use the features module which is supposed to capture drupal config in code. It stores this captured data as a feature module which you can put into version control.
For my database applications, I store scripts of DDL statements (like CREATE TABLE) in some sort of version control system. These scripts sometimes include static "seed" data as well. All the version control systems I use are good at recognizing differences in these files, and they are much smaller than the full database with data.
For the dynamically-generated data, I store backups (e.g. from mysqldump) in an appropriate location (depending on the importance of the data, that may include offsite backups).
1) It's all text, so GIT will just see it as it would any other file.
2) No, due to the above it should add 16mb to the repo (or less, due to GITs own compression), it won't add a new file every time, just the changes, so the repo will change by the size of the additions to the repository
3) No, or GIT won't be able to see the differences - GIT does it's own compression anyway
I have been working on an algorithm in Python, and I was using Vim to edit this file. I opened it up, did a save, and it came up with an Error something like it occasionally does:
"WARNING: YOUR FILE CANNOT BE SAVED! ALL CHANGES WILL BE LOST! CANNOT WRITE THE FILE!"
As this happens occasionally, I did what I normally do, and I hit :q! to quit without writing any changes. No harm, no foul. When I looked at my file, everything had been erased! Everything!
I talked around the office, and it seems that the nfs mount was full, and so that was why I couldn't save anything. There was a huge script generating a ton of data, which caused the mount to be full temporarily. I believe the NFS mount is from NetApp. I found 2 files in my current directory.
One was last saved two days ago, and one was today. They are in the format of:
.nfs.xxxxxxxxxxx
When I try to attempt to open up this file, I see some of my code, here and there, splattered among unknown characters. Apparently, this must be a binary representation of the state of the file.
Is there any way to recover this file from this NFS mount? If there is a shortcut to recover this file in Emacs, I will switch to Emacs from vim!
So, I did find a way to recover the file. I found two ways, in fact. Since it was on a NetApp NFS mount, I was able to use the snapshots feature. When you are in a directory just do
ls .snapshot
And this will pull up any snapshots that your system administrators have set. For us, we have an hourly.0, hourly.1, and nightly.0, and nightly.1 backups. So, we can go back two days, and in the same day, we can go back one hour (the current hour, and the previous).
The other way was to rename the file to a vim swap file like this.
mv .nfs.xxx my_vim_file.cpp.swp
vim my_vim_file.cpp.swp
Then attempt to open it up in Vim, and it should ask you if you want to Recover the swap file, say yes, and it should be back!
Apparently your Netapp uses NFS to mount its volumes (as opposed to iSCSI, for example). Generally, each VM is stored on a unique volume (aka datastore) on the Netapp filer. To find out the volumes and snapshots, and then restore a snapshot, here are the commands to execute at the command line:
# list all volumes, snapshots are taken of volumes
vol status
# list the snapshots available for a particular volume
snap list <vol_name>
# restore a snapshot, nightly.1 for example
snap restore <vol_name> nightly.1
That's it. All that's left is to turn the VM back on and see if you've restored far back enough. If not, then do another "snap restore" but with an older snapshot.
Note that this procedure assumes your administrator didn't disable snapshots (Netapp has a snapshot schedule by default) and that the Netapp is licensed for snaprestore (use the "license" command to verify). This procedure can further be simplified if you have the Netapp OnCommand System Manager, which is a GUI for managing the Netapp. Reverting a snapshot in the GUI is simple:
Go to Storage > Volumes > click on a volume > click on Snapshot Copies (at the bottom)
Choose a snapshot and restore