I use rsync to back up the files on my server, and mysqldump to back up my database. Here's my concern:
A mysqldump on my database takes about 30 seconds. I have a table called photos that stores info about the images a user has uploaded, including the path to the file. I am worried about what will happen when photos are uploaded or deleted during the 30 seconds it takes to complete the mysqldump. If that happened and I were then to restore the rsync'd files and the mysqldump data, I could then be looking at a database that contains rows pointing to deleted photos, or missing rows for photos that were successfully uploaded.
How can I make sure the mysqldump exactly matches the rsync?
Thanks in advance,
Brian
Use LOCK TABLES to block any write activity from the tables you're backing up. Then unlock them once your mysqldump is finished.
I think the answer is simple, just run rsync AFTER you complete mysqldump :) This way at worst you will have a couple NEW files that are not in the db dump, but you will not have inconsistent db entries.
You could MD5 the resulting mysqldump (on the server) and the transfered (locally) by rsync, then compare the two hashes to ensure they match.
Another alternative, is to set mysqldump on a version controlled file (with git or svn or your favorite vcs). The advantage with git, for example, is that you could easily setup some post-commit hooks to push the changes to a remote server, and the upload would be just the differences between versions, not the entire dump. This way you could think in decreasing the backup period.
Related
I run a Redis databse inhouse here and want to make a "snapshot of the snapshot".
What the hell? Yes. I want to move the .rdb file once a day into a S3's bucket. Also, it should be a scheduled operation (probably using a cronTab function).
So here comes my question in fact: will I face trouble if the cronTab job starts running in the middle of a SAVE operation (from redis to .rdb)? There is no problem of losing some data, I just want it to work without any obstruction.
Thanks!
When Redis writes out the RDB to disk, it writes to a temporary file. When the save process is done writing it, it then renames/moves it to the "dump.rdb" file (or whatever you've changed it to if you have done so). This is an atomic action. As such you should be fine with the method you propose.
If you want more control over it you could use a tool such as https://github.com/therealbill/redis-buagent which connects as a slave and generates it's own RDB, storing it in memory then into S3 (or wherever else you want to store such as Cloud Files or a local file) or by using redis-cli --rdb to generate a "local" RDB file for you to copy to S3.
I wanted to restore all the data I save using redis BGSAVE command.It saves the data to its default location /var/lib/redis/6379/dump.rdb .The data contains hashmaps,key-value pairs .How to get back the data to redis from the dump.rdb file?
I am using RESTORE command but it is not solving the purpose!
Just restart the server. On startup it will read the dump. It never has to read the dump during its operation, so there's no command for it.
RESTORE can be useful, but it's per key command. Meaning you have to parse the dump yourself, extract key names and their serialized values and only then call RESTORE for each key. Also, it was implemented to support migrating keys between two running servers. Not exactly your use-case.
Restarting the server is easier, isn't it? :)
I have a with mysqldumb created backup file. It's about 15GB and contains a lot of blobs. Max size per blob is 30MB.
mysqldump -uuser -ppass --compress --quick --skip-opt supertext > supertext.sql
Now when I try to restore the backup, the process just gets stuck.
mysql -uuser -ppass dev_supertext < supertext.sql
It get stuck while writing back the biggest table with the blobs. There is no error message and mysql is still running fine.
This is on a 64bit 5.1.48 community edition for Windows server.
max_allowed_packet is set to 40MB and is not the problem. I had that before.
Any other settings I could check or something I can monitor during the restore?
Didn't see anything special in the query or error log. Maybe there is a timeout?
Just FYI:
I've already posted this question in the MySQL Forum, but got no response.
http://forums.mysql.com/read.php?28,377143
Thanks for any tips.
Are you positive it is only the big table with blobs? Try running the dump sans that table. Do that table individually and if it still gets stuck, break it up.
Create the inserts into 3-4 groups and see if any go through. Process of elimination will help narrow down if theres a row specific issue (I.e. corrupted data?) or if mysql is simply taking a while to write.
I'd advise opening up a second mysql shell or using phpmyadmin to refresh the table view and see if new records are being written. MySQL isn't verbose on its dumps. It may simply be taking a while to load in all the inserts.
I'd like to dump a MySQL database in such a way that a file is created for the definition of each table, and another file is created for the data in each table. I'd like this to be done in a way that guarantees database integrity by locking the entire database for the duration of the dump. What is the best way to do this? Similarly, what's the best way to lock the database while restoring a set of these dump files?
edit
I can't assume that mysql will have permission to write to files.
If you are using InnoDB tables and therefore have transaction support, you don't need to lock the database at all you can just add the --single-transaction command line option. This gives you a consistent snapshot without locking anything by using the transaction mechanism.
If you don't have transaction support you can get what you describe with the --lock-tables command line option, this will lock the databases.
It's also worth noting that READ LOCKs aren't quite as good as they sound since any write operation will lock which will lock subsequent read operations, see this article for more detials
Edit:
I don't think it's possible to dump the format and data separately without the file writing permission and the --tab option. I think your only option is to roll your own mysqldump script that uses one of the transaction or READ LOCK mechanisms that mysqldump uses. Failing that it may be easier to just postprocess a single dumpfile into the format you want.
We have a database currently sitting on 15000 RPM drives that is simply a logging database and we want to move it to 10000 RPM drives. While we can easily detach the database, move the files and reattach, that would cause a minor outage that we're trying to avoid.
So we're considering using DBCC ShrinkFile with EMPTYFILE. We'll create a data and a transaction file on the 10000 RPM drive slightly larger than the existing files on the 15000 RPM drive and then execute the DBCC ShrinkFile with EMPTYFILE to migrate the data.
What kind of impact will that have?
I've tried this and had mixed luck. I've had instances where the file couldn't be emptied because it was the primary file in the primary filegroup, but I've also had instances where it's worked completely fine.
It does hold huge locks in the database while it's working, though. If you're trying to do it on a live production system that's got end user queries running, forget it. They're going to have problems because it'll take a while.
Why not using log shipping. Create new database on 10.000 rpm disks. Setup log shipping from db on 15K RPM to DB on 10k RPM. When both DB's are insync stop log shipping and switch to the database on 15K RPM.
Is this a system connected to a SAN, or is it direct attached storage? If its a SAN do a SAN side migration to the new raid group and the server won't ever know there was a change.