How can I dump a redis that's running on database 0 and restore it in my local machine on a different database (8) ?
I already secure copied the dump file:
scp hostname#/var/lib/redis/dump.rdb .
But if I change my local redis dump.rdb with this one, I'll get the data on database 0. How can I restore it to a specific database?
Firstly note that the use of numbered/shared Redis databases is inadvisable. You really should consider using dedicated Redis servers with a single DB (0) on them (more info at: https://redislabs.com/blog/benchmark-shared-vs-dedicated-redis-instances)
Redis does not offer a straightforward way to do this, but there are two basic ways one could go about it:
Pre-processing: modify the dump.rdb file to load into your database of choosing. You could build a tool for that or perhaps use one of the existing ones. Jan-Erik has done an outstanding job of documenting the RDB v7 format at http://rdb.fnordig.de/file_format.html so all you need to do is basically change the Database Selector byte.
Post-restore: use the MOVE command on the output of SCANing your restored database - should be easily scriptable.
I ended up creating a script in Ruby to dump and restore the keys I wanted. (Please note that this approach is slow, takes around 1 min for 200 keys) .
Get the keys to dump / restore
ssh hostname redis-cli --scan --pattern 'awesome_filter_pattern*'
Open an ssh connection to the production server
Dump the remote key
dump = ssh.exec!("redis-cli dump #{key}").chomp
Restore it on localhost
$redis.connection.restore(key, 0, dump)
Related
I am following the documents about how to restore Redis and I am at complete loss at this point.
The document says
127.0.0.1:6379> SAVE
OK
This command will create the dump.rdb file in your redis directory.
Which it does, it creates the exact same file for me in /usr/lib/redis which is alright I guess.
To restore redis data just move redis backup file (dump.rdb) into your redis directory and start the server. To get your redis directory use CONFIG command can be used. The CONFIG GET command is used to read the configuration parameters of a running Redis server.
127.0.0.1:6379> CONFIG get dir
1) "dir"
2) "/var/lib/redis/6379"
Here is where it makes no sense to me. The .rdb file for me is already saved in /var/lib/redis/ and I have no sub folder to that. I don't understand what "dir" is doing there and how I can restore my database.
Please enlighten me. I don't seem to be able to save it or I cannot find it perhaps.
Okay so basically, the rdb file saved in /var/lib/redis/ is saved every time the server stops and this can be moved to another folder and be used as a point for restoring every time redis service starts.
Is there any way to schedule redis back-ups at a specific time of day (e.g. 3:00 AM GMT) - preferably via a setting in the accompanying conf file?
I already understand that one can set a backup rule in redis configuration (e.g. save every X hours if Y keys have changed).
But how does one schedule the said backup at a particular time of day? Would love to know something basic, but effective. In case it matters, my redis version is 5.0.3
So far I know it is currently not possible from inside redis. But its achievable using crontab. Here is a short example:
create a backup script file:
/tmp/backup.sh
echo save | redis-cli >> /tmp/redis-backup.log
If using sockets, the above would be:
echo save | redis-cli -s /var/run/redis.sock >> /tmp/redis-backup.log
The socket location in your system may vary.
Next, give execute permission to the script:
chmod +x /tmp/backup.sh
Finally, make an entry in crontab: crontab -e
0 3 * * * /tmp/backup.sh
This will run backup.sh in exactly 3AM.
In case you want to disable redis saving setup in the conf (without restarting the redis instance), the best way is to log into redis-cli and issue CONFIG SET save "". Double check that it worked via CONFIG GET save. Finally, don't forget to change the save settings in the relevant conf file as well. Lastly, it's wiser to use bgsave instead of save if tackling a redis instance in production.
For more, checkout these links:
How To Back Up and Restore Your Redis Data
Cron Scheduler
How To Start/Stop/Restart Cron Service In Linux
I'm very new to RIAK. I have a cluster with 5 nodes and I want to backup the data on the cluster. I ran the following command to backup data.
[root#PCPRIAK33 local]# riak-admin backup localhost riak /var/local/temp all
However I am getting the following error.
Attempting to restart script through sudo -H -u riak
{"init terminating in do_boot",{{nocatch,{could_not_reach_node,localhost}},[{riak_kv_backup,ensure_connected,1,[{file,"src/riak_kv_backup.erl"},{line,171}]},{riak_kv_backup,backup,3,[{file,"src/riak_kv_backup.erl"},{line,40}]},{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,572}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
init terminating in do_boot ()
Can you please help me out? :)
Thanks.
I'll answer your immediate question (how to use riak-admin backup) first, but see the comments on preferred methods of backing up, at the end.
The command is:
riak-admin backup <node name> <erlang cookie> <file name with path> all
The node name you can find in your riak vm.args file (look for the line that looks like -name riak#127.0.0.1). It'll be of the form riak#xx.xx.xx.xx with the IP address. So, on my local machine, a single node is named riak#127.0.0.1.
The erlang cookie is also found in the vm.args file, it will most likely be erlang.
The file name parameter should be a fully-qualified path to the actual file name (meaning, you can't give it just a directory name). The filename and extension are arbitrary. So, I would use something like cluster_backup.riak.
So, to put it all together, your backup command should look like:
riak-admin backup riak#<your node ip> riak /var/local/temp/cluster_backup.riak all
Now, having said all that, I don't recommend using the riak-admin backup and restore commands to back up your whole cluster. For several reasons. One, it stores every replica of every object. Meaning, if you're running with the default replica value of n=3, you will be storing 3 copies of each object in your backup file.
Two, the code invoked by that command is single-threaded, and not connection pooled. So all in all, it's going to be SLOW to restore and backup.
Instead, I recommend one of the following approaches:
Take filesystem level snapshots of the data directories of each node. This is the approach currently recommended by Basho, and detailed here: http://docs.basho.com/riak/latest/ops/running/backups/
If you definitely want a "logical" backup (meaning, an export of the objects contained in the cluster), you can use an experimental standalone tool such as the Riak Data Migrator (but see the limitations in the Readme).
I recommend testing out / timing each of these approaches, to see which one is faster for your situation.
I want to switch from MySQL to MongoDB but great data losses (more than 1 hour) are not acceptable for me.
I need to have 3 backup plans:
Hourly backup plan. Data is flushed to disk every X minutes and if something wrong with the server I shall be sure that after reboot it will have all data at least for an hour ago. Can I configure it?
Daily backup plan. Data is synced to backup disk every day so even if server explodes I can recover data for yesterday in some hours. Should I use fsync, master-slave or something else? I would like to have minimal traffic so ideally only changes will be sent.
Weekly backup plan. Data is synced to second backup disk so if both server and first backup disk explode I have at least data for last week. Here this is the question of reliability so it's ok to send all data via network.
How can I do it?
The fsync command flushes the data to disk. It is executed each 60 seconds by default, but can be configured using the --syncdelay command line parameter.
The documentation on backups has some good pointers for daily and weekly backups. For the daily backup, a master-slave configuration seems like the best option, as it will only sync changes.
For the weekly backup you can also use a master-slave configuration, or replication. Another option is the mongodump utility, which will back-up the entire database. It is capable of creating backups while the database is running, so you can run it on the main database or one of the slaves. You can also lock the slave before backing it up.
Try this backup script if you want to create a backup from slave mongodb database to S3.
DB host (secondary preferred as to avoid impacting primary performance)
HOST='SomeHost/mongodbtest-slave'
DB name
DBNAME=***
S3 bucket name
BUCKET=*-backup
Linux user account
USER=ubuntu
Current time
TIME=/bin/date +%d-%m-%Y-%T
Password
PASSWORD=somePassword#!2*1
Username
USERNAME=someUsername
Backup directory
DEST=/home/ubuntu/tmp
Tar file of backup directory
TAR=$DEST/../$TIME.tar
Create backup dir (-p to avoid warning if already exists)
/bin/mkdir -p $DEST
Log
echo "Backing up $HOST/$DBNAME to s3://$BUCKET/ on $TIME";
Dump from mongodb host into backup directory
mongodump --port 27017 -d DBNAME -u USERNAME -p PASSWORD -o $DEST
Create tar of backup directory
/bin/tar cvf $TAR -C $DEST .
Upload tar to s3
/usr/bin/aws s3 cp $TAR s3://$BUCKET/
Remove tar file locally
/bin/rm -f $TAR
Remove backup directory
/bin/rm -rf $DEST
All done
echo "Backup available at https://s3.amazonaws.com/$BUCKET/$TIME.tar
You can use the steps above put them in a shell executable file and execute this at any interval using crontab commands.
If you want to outsource the backup solution entirely, MongoDB Management Service takes snapshots every six hours. The default retention policy on the snapshots will allow you to get point-in-time restore for 24 hours, daily snapshots for a week, weekly snapshots for a month, and monthly snapshots for a year.
This FAQ has the full retention policy.
The backup service continually backs up your replica set by reading the oplog so the overhead is lower than full local periodic snapshots.
May be you can use automongobackup .
On the first point.
MongoDB has a term like 'durable write operation'. If journaling is enabled, you may only lose data that has not been writed in the log. This is a very small amount of time (100 milliseconds by default)
On the second and third points.
You can set master slave replication, but this will not protect you from data errors (for example, if important data is accidentally deleted). Therefore, you need to provide regular backups one way or another.
There are several approaches here:
LVM snapshot is a good solution if your filesystem supports it and the journaling is enabled in your mongoDB. For more details see here
mongodump -You can create a shell script that will run scheduled backups via cron and send them to storage. Here is an example of a good script
Backup as a Service. There are many solutions to make the backup for you.
What is the most efficient, secure way to pipe the contents of a postgresSQL database into a compressed tarfile, then copy to another machine?
This would be used for localhosting development, or backing up to a remote server, using *nix based machines at both ends.
This page has a complete backup script for a webserver, including the pg_dump output.
Here is the syntax it uses:
BACKUP="/backup/$NOW"
PFILE="$(hostname).$(date +'%T').pg.sql.gz"
PGSQLUSER="vivek"
PGDUMP="/usr/bin/pg_dump"
$PGDUMP -x -D -U${PGSQLUSER} | $GZIP -c > ${BACKUP}/${PFILE}
After you have gzipped it, you can transfer it to the other server with scp, rsync or nfs depending on your network and services.
pg_dump is indeed the proper solution. Be sure to read the man page. In Espo's example, some options are questionable (-x and -D) and may not suit you.
As with every other database manipulation, test a lot!