I am reading https://redis.io/docs/manual/transactions/, and I am not sure I fully understand the importance of WATCH in a Redis transaction. From https://redis.io/docs/manual/transactions/#usage, I am led to believe EXEC is atomic:
> MULTI
OK
> INCR foo
QUEUED
> INCR bar
QUEUED
> EXEC
1) (integer) 1
2) (integer) 1
But between MULTI and EXEC, the values at keys foo and bar could have been changed (e.g. in response to a request from another client). Is that why WATCH is useful? To error out the transaction if e.g. foo or bar changed after MULTI and before EXEC?
Do I have it correct? Anything I am missing?
I am led to believe EXEC is atomic
YES
Is that why WATCH is useful? To error out the transaction if e.g. foo or bar changed after MULTI and before EXEC?
NOT exactly. Redis fails the transaction if some key the client WATCHed before the transaction has been changed (after WATCH and before EXEC). The watched key might even not used in transaction.
MULTI-EXEC ensures commands between them runs atomically, while WATCH provides check-and-set(CAS) behavior.
With MULTI-EXEC, Redis ensures running INCR foo and INCR bar atomically, otherwise, Redis might run some other commands between INCR foo and INCR bar (these commands might and might not change foo or bar). If you watch some key (either foo, or bar, or even any other keys) before the transaction, and the watched key has been modified since the WATCH command, the transaction fails.
UPDATE
The doc you referred has a good example of using WATCH: implement INCR with GET and SET. In order to implement INCR, you need to 1) GET the old counter from Redis, 2) incr it on client side, and 3) SET the new value to Redis. In order to make it work, you need to ensure no other clients update/change the counter after you run step 1, otherwise, you might write incorrect value and should fail the transaction. 2 clients both get the old counter, i.e. 1, and incr it locally, and client 1 update it to 2, then client 2 update it to 2 again (the counter should be 3). In this case, WATCH is helpful.
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
If some other client update the counter after we watch it, the transaction fails.
When do watched keys no longer watched?
EXEC is called.
UNWATCH is called.
DISCARD is called.
connection is closed.
Related
I'm using Redis on a clustered db (locally). I'm trying the MULTI command, but it seems that it is not working. Individual commands work and I can see how the shard moves.
Is there anything else I should be doing to make MULTI work? The documentation is unclear about whether or not it should work. https://redis.io/topics/cluster-spec
In the example below I just set individual keys (note how the port=cluster changes), then trying a multi command. The command executes before EXEC is called
127.0.0.1:30001> set a 1
-> Redirected to slot [15495] located at 127.0.0.1:30003
OK
127.0.0.1:30003> set b 2
-> Redirected to slot [3300] located at 127.0.0.1:30001
OK
127.0.0.1:30001> MULTI
OK
127.0.0.1:30001> HSET c f val
-> Redirected to slot [7365] located at 127.0.0.1:30002
(integer) 1
127.0.0.1:30002> HSET c f2 val2
(integer) 1
127.0.0.1:30002> EXEC
(error) ERR EXEC without MULTI
127.0.0.1:30002> HGET c f
"val"
127.0.0.1:30002>
MULTI transactions, as well as any multi-key operations, are supported only within a single hashslot in a clustered Redis deployment.
I am using Redis for cache in my application which is configured in spring beans, spring-data-redis 1.7.1, jedis 2.9.0.
I would like to know how to set the race condition ttl in the configuration.
Please comment if you have any suggestions.
If I understand you right, you want the same as that Ruby repo, but in Java.
For that case you may want to put a technical lock key along the one you need.
get yourkey
(nil)
get <yourkey>::lock
// if (nil) then calculate, if t then wait. assuming (nil) here
setex <yourkey>::lock 30 t
OK
// calcultions
set <yourkey> <result>
OK
del <yourkey>::lock
(integer) 1
Here with setex you set a lock key with TTL of 30 sec. You can put another TTL if you want.
There is one problem with the code above - some time will pass before checking a lock and aquiring it. To properly aquire a lock EVAL can be used: eval "local lk=KEYS[1]..'::lock' local lock=redis.call('get',lk) if (lock==false) then redis.call('setex',lk,KEYS[2],'t') return 1 else return 0 end" 2 <yourkey> 30 This either returns 0 if there is no lock or puts a lock and returns 1.
As i want to use transaction in redis. I have read documentation of redis transaction and found below.
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
but my question is if redis transaction is executed sequentially and atomic then can't we achieve above same behavior with below statements?
MULTI
val = GET mykey
val = val + 1
SET mykey $val
EXEC
Thanks,
NO, you CANNOT achieve that.
With MULTI and EXEC, you can run multiple commands atomically on the server side, and get all replies of these commands after EXEC returns.
In your case, val = val + 1 has to be run on the client side, and it CANNOT be a part of the MULTI commands. Also, before EXEC returns, you CANNOT get the reply of val = GET mykey. So you CANNOT increase val before SET mykey val. Instead you have to run GET command outside MULTI, and use WATCH command to ensure the key haven't changed before when you update it.
Another solution to achieve your goal, i.e. make val = val + 1 run on the server side: you can use Lua scripting. Lua scripting is a replacement of MULTI and EXEC commands, and it runs atomically on the server side. In fact, it's a better solution to achieve transaction.
In the redis protocol specification, under the "Multi-bulk replies section":
A Multi bulk reply is used to return an array of other replies. Every element of a Multi Bulk Reply can be of any kind, including a nested Multi Bulk Reply.
However, I can't figure out a way to get Redis to return such output. Can anyone provide an example?
Only certain commands (especially those returning list of values) return multi-bulk replies, you can try by using LRANGE for example but you can check the command reference for more details.
Usually multi-bulk replies are only 1-level deep but some Redis commands can return nested multi-bulk replies (max 2 levels), notably EXEC (depending on the commands executed while inside the transaction context) and both EVAL / EVALSHA (depending on the value returned by the Lua script).
Here is an example using EXEC:
redis 127.0.0.1:6379> MULTI
OK
redis 127.0.0.1:6379> LPUSH metavars foo foobar hoge
QUEUED
redis 127.0.0.1:6379> LRANGE metavars 0 -1
QUEUED
redis 127.0.0.1:6379> EXEC
1) (integer) 4
2) 1) "hoge"
2) "foobar"
3) "foo"
4) "metavars"
The second element of the multi-bulk reply to EXEC is a multi-bulk itsef.
PS: I added a clarification in the comments regarding the actual maximum level of nesting of multi-bulk replies when using Lua scripts. tl;dr: there's basically no limit.
Could you please explain me following example from "The Little Redis Book":
With the code above, we wouldn't be able to implement our own incr
command since they are all executed together once exec is called. From
code, we can't do:
redis.multi()
current = redis.get('powerlevel')
redis.set('powerlevel', current + 1)
redis.exec()
That isn't how Redis transactions work. But, if we add a watch to
powerlevel, we can do:
redis.watch('powerlevel')
current = redis.get('powerlevel')
redis.multi()
redis.set('powerlevel', current + 1)
redis.exec()
If another client changes the value of powerlevel after we've called
watch on it, our transaction will fail. If no client changes the
value, the set will work. We can execute this code in a loop until it
works.
Why we can't execute increment in transaction that can't be interrupted by other command? Why we need to iterate instead and wait until nobody changes value before transaction starts?
There are several questions here.
1) Why we can't execute increment in transaction that can't be interrupted by other command?
Please note first that Redis "transactions" are completely different than what most people think transactions are in classical DBMS.
# Does not work
redis.multi()
current = redis.get('powerlevel')
redis.set('powerlevel', current + 1)
redis.exec()
You need to understand what is executed on server-side (in Redis), and what is executed on client-side (in your script). In the above code, the GET and SET commands will be executed on Redis side, but assignment to current and calculation of current + 1 are supposed to be executed on client side.
To guarantee atomicity, a MULTI/EXEC block delays the execution of Redis commands until the exec. So the client will only pile up the GET and SET commands in memory, and execute them in one shot and atomically in the end. Of course, the attempt to assign current to the result of GET and incrementation will occur well before. Actually the redis.get method will only return the string "QUEUED" to signal the command is delayed, and the incrementation will not work.
In MULTI/EXEC blocks you can only use commands whose parameters can be fully known before the begining of the block. You may want to read the documentation for more information.
2) Why we need to iterate instead and wait until nobody changes value before transaction starts?
This is an example of concurrent optimistic pattern.
If we used no WATCH/MULTI/EXEC, we would have a potential race condition:
# Initial arbitrary value
powerlevel = 10
session A: GET powerlevel -> 10
session B: GET powerlevel -> 10
session A: current = 10 + 1
session B: current = 10 + 1
session A: SET powerlevel 11
session B: SET powerlevel 11
# In the end we have 11 instead of 12 -> wrong
Now let's add a WATCH/MULTI/EXEC block. With a WATCH clause, the commands between MULTI and EXEC are executed only if the value has not changed.
# Initial arbitrary value
powerlevel = 10
session A: WATCH powerlevel
session B: WATCH powerlevel
session A: GET powerlevel -> 10
session B: GET powerlevel -> 10
session A: current = 10 + 1
session B: current = 10 + 1
session A: MULTI
session B: MULTI
session A: SET powerlevel 11 -> QUEUED
session B: SET powerlevel 11 -> QUEUED
session A: EXEC -> success! powerlevel is now 11
session B: EXEC -> failure, because powerlevel has changed and was watched
# In the end, we have 11, and session B knows it has to attempt the transaction again
# Hopefully, it will work fine this time.
So you do not have to iterate to wait until nobody changes the value, but rather to attempt the operation again and again until Redis is sure the values are consistent and signals it is successful.
In most cases, if the "transactions" are fast enough and the probability to have contention is low, the updates are very efficient. Now, if there is contention, some extra operations will have to be done for some "transactions" (due to the iteration and retries). But the data will always be consistent and no locking is required.