I have a redis as big as 4G on one 4G memory machine,I want to split it into two 2G redis instance so that I can run the two on two different machines.
how to do that?
thx
AFAIK there is no easy way to do it.
One way to do it is to use the redis-rdb-tools package from Sripathi Krishnan. The procedure is:
choose a strategy to shard your data (i.e a function which distributes the keys over the instances)
write a Python script to parse a Redis dump file, connect to several instances, and apply the commands to insert the data on the correct instances
dump the Redis instance
flush the instance
create and start the second instance
run the script on the dump of the first instance
See more information at https://github.com/sripathikrishnan/redis-rdb-tools
you can use redis cluster
Redis Cluster provides a way to run a Redis installation where data is
automatically sharded across multiple Redis nodes.
Every Redis Cluster node requires two TCP connections open.
http://redis.io/topics/cluster-tutorial
Related
I have 3 redis instance with redis. One is the master and the other two, are the slaves. I have connected to master node and get info by redis-cli with INFO command. I can see the parameter cluster_enabled:0 and
#Replication
role:master
connected_slaves:2
slave0:ip=xxxxx,port=6379,state=online,offset=15924636776,lag=1
slave1:ip=xxxxx,port=6379,state=online,offset=15924636776,lag=0
And the keyspace, each node has different dbs. I need to migrate all data to a single memorystore in GCP but I don't know how. Anyone can help me?
Since the two nodes are slaves and clustering is not enabled, you only need to replicate the master node. RIOT is a great tool for migrating data in and out of Redis.
However, if you say DB by node do you mean redis DB that you access by select? In that case you'll need to prefix keys as there may be overlap between the keysets of the DBs.
I think setting up another Redis cluster in a single node configuration is the least of your worries.
The real challenge for you would be migrating all your records over to the new setup. This is not a simple question to answer and would depend heavily on multiple factors:
The total size of your data being migrated
Is this is a live Database in production
Do you want to keep the two DB schemas in your new configuration separate?
Ok, I believe currently your Redis Instances are hosted on Google Compute Engine.
And you are looking to migrate to Memorystore for Redis.
As mentioned here, you can leverage Redis snapshots for this. It provides you step-wise instructions on how to achieve this, leveraging GCS buckets as transient storage.
import data into Cloud Memorystore instances using RDB (Redis Database Backup) snapshots, as well as back up data from existing Redis instances.
I have multiple redis instances. I made a cluster using different port. Now I want to transfer the data from pre-existing redis instances to the cluster. I know how to transfer data from one instance to the cluster but when the instances are greater than one, I am not able to do it.
You need to define some sort of sharding strategy for your redis cluster. Database Sharding So basically you need to have a certain consistent hashing strategy which will decide given a key, the shard or the redis instance in your cluster the key will go to. You need to have a certain script for this data migration that will have an array of all the redis instances in your cluster.
Then for a given key which you read from the standalone redis, you will use the hashing mechanism to find out the sharding index or the redis instance from the list you maintained earlier to use and accordingly you will write the data in that cluster node. My assumption in all this is that you have an in house redis cluster setup as opposed to the one which Redis Labs provide.
We are using redis. We have two set of data. One set of data(Assume it is using the prefix redis:local: eg: redis:local:key1) is used by the main application and no need of replication.
Another set of data (Prefix redis:replicate: eg: redis:replicate:key2) is used by the main application and should be replicated to slave redis instances.
I have two questions.
Is it possible to configure redis to replicate only keys with prefix redis:replicate:?
If that is not possible, Is it possible to configure redis to replicate only one database? We will store the first set of data in database-0 and the second set of data in database-1. So we have to replicate only database-1.
Currenly, we are running two instances of redis to solve the issue.
Redis only supports replication of whole instances. Limiting replication to a key prefix or database is not possible.
Running two instances of Redis is simplest and reliable option.
Another way would be to write a custom replication program which is difficult and failure prone in comparison.
There is also another question concerning replication of only one database: Replicate a single Redis database from an instance that has multiple databases
We have a single Redis instance with a good amount of data (over 100GB). We also have an empty Redis Cluster with 6 nodes. What would be the best way to move all that data from the stand-alone instance to the Redis Cluster and make it distribute it evenly?
After some searching around, I came across a post detailing how to move data over to a cluster. It may take some time to move lots of data over but this is the best way I've seen so far.
You can read about it here: https://fnordig.de/2014/03/11/redis-cluster-with-pre-existing-data/
You could make it easier by using redis-rdb-tools and a cluster proxy programm like redis-cerberus after you dump data into an RDB file
rdb --command protocol RDB_FILE_PATH | nc PROXY_HOST PROXY_PORT
Piping an AOF file into a proxy maybe doesn't work somehow if the AOF file contains cross-slots commands like RPOPLPUSH (depending on the proxy's implementation). However if you are actually using this kind of commands, you are not supposed to use a cluster.
Since the redis cluster is still a work in progress, I want to build a simplied one by myselfin the current stage. The system should support data sharding,load balance and master-slave backup. A preliminary plan is as follows:
Master-slave: use multiple master-slave pairs in different locations to enhance the data security. Matsters are responsible for the write operation, while both masters and slaves can provide the read service. Datas are sent to all the masters during one write operation. Use Keepalived between the master and the slave to detect failures and switch master-slave automatically.
Data sharding: write a consistant hash on the client side to support data sharding during write/read in case the memory is not enougth in single machine.
Load balance: use LVS to redirect the read request to the corresponding server for the load balance.
My question is how to combine the LVS and the data sharding together?
For example, because of data sharding, all keys are splited and stored in server A,B and C without overlap. Considering the slave backup and other master-slave pairs, the system will contain 1(A,B,C), 2(A,B,C) , 3(A,B,C) and so on, where each one has three servers. How to configure the LVS to support the redirection in such a situation when a read request comes? Or is there other approachs in redis to achieve the same goal?
Thanks:)
You can get really close to what you need by using:
twemproxy shard data across multiple redis nodes (it also supports node ejection and connection pooling)
redis slave master/slave replication
redis sentinel to handle master failover
depending on your needs you probably need some script listening to fail overs (see sentinel docs) and clean things up when a master goes down