We are using redis. We have two set of data. One set of data(Assume it is using the prefix redis:local: eg: redis:local:key1) is used by the main application and no need of replication.
Another set of data (Prefix redis:replicate: eg: redis:replicate:key2) is used by the main application and should be replicated to slave redis instances.
I have two questions.
Is it possible to configure redis to replicate only keys with prefix redis:replicate:?
If that is not possible, Is it possible to configure redis to replicate only one database? We will store the first set of data in database-0 and the second set of data in database-1. So we have to replicate only database-1.
Currenly, we are running two instances of redis to solve the issue.
Redis only supports replication of whole instances. Limiting replication to a key prefix or database is not possible.
Running two instances of Redis is simplest and reliable option.
Another way would be to write a custom replication program which is difficult and failure prone in comparison.
There is also another question concerning replication of only one database: Replicate a single Redis database from an instance that has multiple databases
Related
I have 3 redis instance with redis. One is the master and the other two, are the slaves. I have connected to master node and get info by redis-cli with INFO command. I can see the parameter cluster_enabled:0 and
#Replication
role:master
connected_slaves:2
slave0:ip=xxxxx,port=6379,state=online,offset=15924636776,lag=1
slave1:ip=xxxxx,port=6379,state=online,offset=15924636776,lag=0
And the keyspace, each node has different dbs. I need to migrate all data to a single memorystore in GCP but I don't know how. Anyone can help me?
Since the two nodes are slaves and clustering is not enabled, you only need to replicate the master node. RIOT is a great tool for migrating data in and out of Redis.
However, if you say DB by node do you mean redis DB that you access by select? In that case you'll need to prefix keys as there may be overlap between the keysets of the DBs.
I think setting up another Redis cluster in a single node configuration is the least of your worries.
The real challenge for you would be migrating all your records over to the new setup. This is not a simple question to answer and would depend heavily on multiple factors:
The total size of your data being migrated
Is this is a live Database in production
Do you want to keep the two DB schemas in your new configuration separate?
Ok, I believe currently your Redis Instances are hosted on Google Compute Engine.
And you are looking to migrate to Memorystore for Redis.
As mentioned here, you can leverage Redis snapshots for this. It provides you step-wise instructions on how to achieve this, leveraging GCS buckets as transient storage.
import data into Cloud Memorystore instances using RDB (Redis Database Backup) snapshots, as well as back up data from existing Redis instances.
I have multiple redis instances. I made a cluster using different port. Now I want to transfer the data from pre-existing redis instances to the cluster. I know how to transfer data from one instance to the cluster but when the instances are greater than one, I am not able to do it.
You need to define some sort of sharding strategy for your redis cluster. Database Sharding So basically you need to have a certain consistent hashing strategy which will decide given a key, the shard or the redis instance in your cluster the key will go to. You need to have a certain script for this data migration that will have an array of all the redis instances in your cluster.
Then for a given key which you read from the standalone redis, you will use the hashing mechanism to find out the sharding index or the redis instance from the list you maintained earlier to use and accordingly you will write the data in that cluster node. My assumption in all this is that you have an in house redis cluster setup as opposed to the one which Redis Labs provide.
I'm a newbie to Redis and I was wondering if someone could help me to understand if it can be the right tool.
This is my scenario:
I have many different nodes, everyone behaving like a master and accepting clients connections to read and write a few geographical data data and the timestamp of the incoming record.
Each master node could be hosted onto a drone that only randomly get in touch and can comunicate with others, accordind to network conditions; when this happens they should synchronize their data according to their age (only the ones more recent than a specified time).
Is there any way to achieve this by Redis or do I have to implement this feature at application level?
I tried master/slaves configuration without success and I was wondering if Redis Cluster can somewhat meet my neeeds.
I googled around, but what I found had not an answer good for me
https://serverfault.com/questions/717406/redis-multi-master-replication
Using Redis Replication on different machines (multi master)
Teo, as a matter of fact, redis don't have a multi master replication.
And the cluster shard it's data through different instances. Say you have only two redis instances. Instance1 will accept store and retrieve instance1 and instance2 data. But he will ask for, and store in, instance2 every key that does not belong to his shard.
This is not, I think, really what you want. You could give a try to PostgreSQL+BDR as PostgreSQL supports nosql store and BDR provides a real master master replication (https://wiki.postgresql.org/wiki/BDR_Project) if that's really what you need.
I work with both today (and also MongoDB). Each one with a different goal. Redis would provide a smaller overhead and memory use, fast connection and fast replication. But it won't provide multi master (if you really need it).
I am using 5 databases in my redis server. I want to evict keys belonging to a particular DB using LRU mechanism. Is it possible ?
I read this: how-to-make-redis-choose-lru-eviction-policy-for-only-some-of-the-keys.
But all my databases are using time to live for their entries. So cant use volatile-lru policy.
I tried volatile-ttl policy but other databases are having less ttl for their keys. So they will get evicted which I dont want.
That's one of the effects of using numbered/shared database - they all share the same configuration and resources. You should consider using separate Redis servers, one for each of your databases, to have better control over what gets evicted and when. Even more importantly, using dedicated instances allows you to better utilize the cores that you server has.
Since the redis cluster is still a work in progress, I want to build a simplied one by myselfin the current stage. The system should support data sharding,load balance and master-slave backup. A preliminary plan is as follows:
Master-slave: use multiple master-slave pairs in different locations to enhance the data security. Matsters are responsible for the write operation, while both masters and slaves can provide the read service. Datas are sent to all the masters during one write operation. Use Keepalived between the master and the slave to detect failures and switch master-slave automatically.
Data sharding: write a consistant hash on the client side to support data sharding during write/read in case the memory is not enougth in single machine.
Load balance: use LVS to redirect the read request to the corresponding server for the load balance.
My question is how to combine the LVS and the data sharding together?
For example, because of data sharding, all keys are splited and stored in server A,B and C without overlap. Considering the slave backup and other master-slave pairs, the system will contain 1(A,B,C), 2(A,B,C) , 3(A,B,C) and so on, where each one has three servers. How to configure the LVS to support the redirection in such a situation when a read request comes? Or is there other approachs in redis to achieve the same goal?
Thanks:)
You can get really close to what you need by using:
twemproxy shard data across multiple redis nodes (it also supports node ejection and connection pooling)
redis slave master/slave replication
redis sentinel to handle master failover
depending on your needs you probably need some script listening to fail overs (see sentinel docs) and clean things up when a master goes down