I have 3 Redis clusters each with 3 master nodes and 3 slave nodes. I want to make one of clusters as the primary cluster and the remaining two as the secondary cluster meaning data from the slave/master nodes of the primary cluster will be replicated to the master nodes of the secondary cluster.
In other words, data mapped to each slot of primary cluster would be replicated to corresponding slot of master node in secondary clusters.
Is there any simple way to achieve this? Please help.
Thank you in advance.
Master nodes are separated by the concept of hashslot as mentioned here https://redis.io/topics/cluster-tutorial, so replication of one master node to other makes no sense. For you use case you can simply have single master node with 2 slave nodes so that data replicated from master node(primary cluster) will be replicated to other slave nodes(secondary cluster).
Related
In cluster mode of Redis, is a piece of data with a specific key has to be stored in a specific node, no matter what data structure the it has (e.g. List/Hash)?
For example, I have a hash map:
HMSET website google www.google.com yahoo www.yahoo.com
The key of the hash map is "website", and the hash map has data {google:www.google.com, yahoo:www.yahoo.com}. In my understanding, the hash map is stored in only one node of the cluster. It will be not efficient when the hash map is large (e.g. 400M key-value pairs in one hash map).
My question is: is there a way to automatically distribute the contents of the hash map of the same key among the cluster? For example, store pair {google:www.google.com} in node 0 and store pair {yahoo www.yahoo.com} in node 1, when the key of the hash map is still "website"?
In cluster mode of Redis, is a piece of data with a specific key has to be stored in a specific node, no matter what data structure the it has (e.g. List/Hash)?
Yes - every key is mapped to a hash slot, that a single cluster instance manages.
My question is: is there a way to automatically distribute the contents of the hash map of the same key among the cluster?
No - data is distributed between nodes at key level. A given key's data structure cannot be distributed between multiple shards. To distribute the data, you'll have to model it using more keys.
Correctly modeling your needs requires knowing what type of operations you'll be performing against your distributed "hash map" and their respective frequencies. Feel free to add this information to the question, or open a new one that is more focused on your requirements.
As I know from Redis cluster tutorial, cluster has only 16384 slots (0 - 16383). The hashslots are calculated by following command: CRC16 (KEY) mod 16384. So for example CRC16 of some key equals 16385 and hash slot will be 1. For another key CRC16 equals 32769 and hash slot will be again 1. Is it cause some conflict? Or first one will be rewritten by second one?
If i understand your question, no this is not a conflict. Each key belogns to one hashsolt but each hashslot can have many keys.
CLUSTER GETKEYSINSLOT slot count: https://redis.io/commands/cluster-keyslot
I have a Redis implementation with 6 nodes (3 masters 3 slaves - cluster enabled). I have load in every master an amount of keys.
So, my question is:
Is it possible to actual copy one key from 127.0.0.1:30001 to 127.0.0.1:30002?
For example lets say that my key has the name "testkey". If i copy this key from 30001 to 30002, when i want to get the key from 30001 or from 30002 the response must fetch the value of "testkey" in both calls.
No, that not how it works.
Keys in the cluster are assigned to hash slots and slots are assigned to master nodes. The keys' assignment is done by hashing their names (or the hash tag in them) so it is consistent, meaning that a given key name always hashes to the same slot.
A key can exist only once in the keyspace, but the slot it belongs to can be moved between masters. To scale reads from that key you can use the slave of the applicable master.
A good point to start understanding how the cluster works is by referring to the [tutorial](https://redis.io/topics/cluster-tutorial].
I am seeing multiple out_ prefix keys in my Redis ( version 3+ ) . Can any one help me understand these keys ? Are these internal , temporary keys ?
The application does not create these keys .
[02.66%] Biggest zset found so far 'out_GVwthhmz' with 765025 members
[09.91%] Biggest zset found so far 'out_JIwnd8Cu' with 798580 members
[69.70%] Biggest zset found so far 'out_UcP6p4YL' with 801308 members
Thanks
There are no internal/temporary keys that Redis creates - these are definitely something that was created by an application. Good luck hunting for it - possible routes to investigate:
Run grep out_* on your source code
Use MONITOR and CLIENT LIST to hone onto the offender
I'm trying to design a database for a server and database inventory and I'm looking for a good database design. We have tables for server clusters, stand alone servers and databases. I would like to represent the following relationships in the database:
A one to many relationship from cluster to servers.
A one to many relationship from database to cluster/server.
The difficulty is in the second relationship becuse the clusters and servers are in separate tables and a cluster is made up of servers. What is the best way to represent this relationship?
It sounds like you have this in your relational view of the situation.
Cluster : name, other attributes of cluster
Server : name, optional FK to cluster, other attributes of a server
Database : name, (FK to cluster OR FK to server)
The issue is that you have a somewhat more complex real-world situation, one that relational technology doesn't reflect cleanly.
Host -- an abstract superclass for places a database can run.
Cluster (extends Host) : name, etc.
Server (extends Host) : name, optional FK to cluster.
Database : FK to Host
You have several choices for handling this kind of "subentity" problem.
Collapse Host, Cluster and Server into a single table. This leads to a recursive relationship among Host (as Cluster) and Host (as Server). This is kind of annoying, but it does create a single table for Host, Cluster and Server. The resulting table has a lot of nulls (Cluster rows use one bunch of columns, Server rows use a different set of columns.) You have to add a column to discriminate among the subentities of Host.
Push Host information down into Cluster and Server. This is useful when you have a lot of common information in the Host table, and very little subclass-specific information in the Cluster or Server tables. The Cluster and Server tables look very similar (essentially clones of Host) with a few columns that are different.
Use a Join between (Host and Cluster) or (Host and Server) based on a discriminator in Host. While fairly complex, this scales well because all Databases are joined to a Host, and the complete list of Hosts is a union of Hosts which join to Server plus Hosts which join to Cluster.
Use optional FK fields in Database. This requires a union between Database joined to Cluster plus Database joined to Server to get a full list of databases. Each Database might have to have a discriminator so that you could distinguish among the various combinations of NULL values in the two FK fields. There are four possible combinations, of which two are sensible, and two might be prohibited. Trying to simply use two nullable FK's doesn't usually work out well, so you often need a status flag to separate Database on Cluster from Database on Server from Database not assigned to anything, from Database with unknown hosting from any other status that might be relevant.
Option 1: Have two fields in the database table. One refers to server, the other to cluster. Keep one of them null.
Option 2: Another approach is to add an entry in cluster for each stand-alone server also and link only to that table.
Option 1 is really not the cleanest solution (i do agree with the comments), so go for option 2 :)