There is a Paragraph in https://redis.io/topics/cluster-tutorial which says:
Redis Cluster has support for synchronous writes when absolutely needed, implemented via the WAIT command, this makes losing writes a lot less likely, however note that Redis Cluster does not implement strong consistency even when synchronous replication is used: it is always possible under more complex failure scenarios that a slave that was not able to receive the write is elected as master.
What I understand, Synchronous write is supposed to be working as follows:
1. Client Z sends data D to B
2. B sends data D to B1, B2 and B3
3. B waits for ack from B1, B2 and B3
4. B returns response success to Z
Where Z is a client, B is Master node of shard, B1, B2 and B3 are replicas of B.
My question is even in complex scenarios, how can it fails in Synchronous write as Synchronous write?
Related
Application 1 set a value in Redis.
And we have two instance of application 2 which are running and we would like only one instance should read this value from Redis (please note application 2 takes around 30 sec to 1 min process data )
Can Instance-1 application 2 acquire lock redis key which is created by application 1 , so that instance-2 of application 2 will not read and do the same operation ?
No, there's no concept of record lock in Redis. If you need to achieve some sort of locking you have to use another set of data structures to mimic that behavior. For example
List: You can use a list and then POP the item from the list or...
Redis Stream: Using Redis Stream with ConsumerGroup so that each consumer in your Group only sees a portion of the whole data the needs to be processed and it guarantees you that, when an item is delivered to a consumer, it is not going to be delivered to another one.
I'm reading Redis in action e-book chapter about semaphores. Here is the python code to implement semaphore using redis
def acquire_semaphore(conn, semname, limit, timeout=10):
identifier = str(uuid.uuid4())
now = time.time()
pipeline = conn.pipeline(True)
pipeline.zremrangebyscore(semname, '-inf', now - timeout)
pipeline.zadd(semname, identifier, now)
pipeline.zrank(semname, identifier)
if pipeline.execute()[-1] < limit:
return identifier
conn.zrem(semname, identifier)
return None
This basic semaphore works well—it’s simple, and it’s very fast. But
relying on every process having access to the same system time in
order to get the semaphore can cause problems if we have multiple
hosts. This isn’t a huge problem for our specific use case, but if we
had two systems A and B, where A ran even 10 milliseconds faster than
B, then if A got the last semaphore, and B tried to get a semaphore
within 10 milliseconds, B would actually “steal” A’s semaphore without
A knowing it.
I didn't catch what does it mean: if A ran even 10 milliseconds faster than B then B would actually “steal” A’s semaphore without A knowing it.
My thoughts: A's time is 10:10:10:200 and B's time 10:10:10:190 and A've got semaphore. Then B tries to get semaphore within 10 ms (now B's local time is 10:10:10:200). B will delete expired items and add itslef. How B can steal A's semaphore? At the same time if A's time is 10:59 and B's time is 11:02 it's possible when B can remove A's semaphore because of time difference. But it's not the case described in book.
If B runs 10 ms slower than A, then B's score is smaller than A, since we use the local time as the score of sorted set.
So B's rank, i.e. pipeline.zrank(semname, identifier), is smaller than A's rank, which is smaller than limit, i.e. if pipeline.execute()[-1] < limit:. And B thought it got the semaphore, i.e. return identifier. In fact, it steals the semaphore from A.
I make a program that can convert numbers to some Serial numbers by some rules and check if Serial numbers is used. And I use redis to do the check work.
First, get num1 in slave. when result is not nil , it means serial number is used , so return 'used'.
Second, if result is nil, set num1 in master and return 'new' (once return , the nums mean 'used')
The problem is that master will crash before it finish the process of sync with slave , so the number maybe not in slave. At this time get num1 in slave , it return 'new',but the num1 is used.
how to ensure data consistency between master and slave in redis?
Read about the WAIT command - it allows you to specify the number of slaves that were updated with the most recent change before taking further action.
Redis uses asynchronous replication and it is not possible to ensure the slave actually received a given write. There will always be a window for data loss.
Replication Docs
I have problems on designing message queue oriented design on a clustered fault tolerant system.
There are three sites A B C.
B is clustered as B1 B2 B3.
C is clustered as C1 C2 C3.
A message X (1000-5000 Xs per second actually) is sent to B cluster from Site A.
B converts it to Y and sends it to C.
C converts it to Z and sends it to B.
If the connection between A and B fails A sends X to C. Now C converts it to Z and sends Z to B.
If all connections of A fail then A stores Xs locally.
I have looked at the Apache Activemq Artemis formerly Jboss HornetQ.
But i cannot exactly figure out whether it fulfils my above scenario.
Could you help me? Thanks for your answers.
My query is regarding engineering the priority value of a process. In my system, process A is running in RR at priority 83. Now I have another process B in RR, I want B's priority to be higher than A (i.e. I want B to be scheduled always compared to A).
To do this, what value should I choose for B. I have read in code that there is a penalty/bonus of 5 depending upon process's history.
Also, If I choose value 84 Or 85, is there any chance in some situations that my process is ignored.
Please help in engineering this value.
Now I got it. Real time tasks(FF/RR) are not governed by penalty/bonus rules. With O(1) scheduler, task with higher priority will be chosen. In my case process B will be scheduled if its priority is greater than process A.
Penalty/bonus is for SCHED_OTHER/SCHED_NORMAL.