I used to use PostgreSQL sequence SELECT nextval('number'); to make sure I get a new number even if there are several clients since nextval is guaranteed to return distinct and increasing values.
I wanted to use the same mechanism with Redis.
I thought using the INCR command but I want to make sure I well understand that it will give the same guarantee than the nextval command?
There is no risk of getting the same number when several processes run this command or do I have to use a Redis lock?
Thanks
Redis is a single-threaded engine, so the execution of all commands is serialized. It always provides atomicity and isolation (in the sense of ACID) for each individual command.
The consequence is applying a single INCR command and exploiting its result is safe (even if multiple connections do it concurrently).
Related
I will be storing key-value pairs in Redis but the number of keys will be just 4. Since there will be multiple processes updating the values parallelly, I plan to use Redis transactions using WATCH, MULTI and EXEC commands.
My algorithm is something like this:
GET key
WATCH key
MULTI
SET key new_val
EXEC
My main concern is that, since WATCH uses optimistic locking, when I will have multiple processes (much more than the number of keys, which are only 4) trying to update values, the transaction failure rate will be very high.
Is this correct? Is there a way to prevent this?
It is unclear why you'd need a transaction here unless you're doing something in your application with the reply for GET key. I'll therefore assume that you are using the value for something meaningful, otherwise, you can drop the transaction semantics and just call SET key new_val.
Optimistic locking is mainly intended to be used in cases where there's low contention for the resources. Since the use case that you're describing is clearly the opposite, it would probably result in high failure rates. This isn't saying that Redis and your application will not work, but it does mean there a potential for a lot of wasted effort.
I would advise that you consider switching if possible to using Redis' server-side Lua scripts. These are blocking, atomic, and let you read and meaningfully manipulate the data in Redis programmatically. See the EVAL command for details.
Can XREAD (or perhaps another command) be used to atomically detect whether data was written to a Redis stream?
More specifically:
Suppose you added some data to a Redis stream in one process and saw that the data was added successfully with some auto generated key.
XADD somestream foo bar
After this XADD completes, you immediately run the following read in another process.
XREAD COUNT 1000 STREAMS somestream 0-0
Is this XREAD guaranteed to return data? The documentation is not clear about whether a successful XADD guarantees that readers will immediately see the added data, or whether there might be some small delay.
Redis's famous single threaded architecture answers that question. When you execute XADD on one process(client side) and after another process(client side) executes XREAD then the server execute them consecutively which guarantees that the data will be there before XREAD is executed.
The next quotes are from The Little Redis Book
Every Redis command is atomic, including the ones that do multiple things. Additionally, Redis has support for transactions when using multiple commands.
You might not know it, but Redis is actually single-threaded, which is how every command is guaranteed to be atomic.
While one command is executing, no other command will run. (We’ll briefly talk about scaling in a later chapter.) This
is particularly useful when you consider that some commands do multiple things.
One way to execute commands in REDIS, is via the EVAL script.
Redis uses the same Lua interpreter to run all the commands. Also
Redis guarantees that a script is executed in an atomic way: no other
script or Redis command will be executed while a script is being
executed.
Since redis is single threaded, why do we need EVAL to offer atomicity? I would expect that this is implied by the one running thread.
Am I missing something? Apologies if my question is pretty simple, I am quite new to redis
Every (data path) command in Redis is indeed atomic. EVAL allows you to compose an "atomic" command with a script that can include many Redis commands, not to mention control structures and some other utilities that are helpful to implement server-side logic. To achieve the similar "atomicity" of multiple commands you can also use MULTI/EXEC blocks (i.e. transactions) by the way.
Without an EVAL or a MULTI/EXEC block, your commands will run one after another, but other clients' commands may interleave between them. Using a script or transaction eliminates that.
Redis uses a single thread to execute commands from many different clients. So if you want a group of commands from one client to be executed in sequence, you need a way to direct Redis to do that. That's what EVAL is for. Without it, Redis could interleave the execution of commands from other clients in with yours.
I have some scripts that touch a handful of keys. What happens if Elasticache decides to reshard while my script is running? Will it wait for my script to complete before it moves the underlying keys? Or should I assume that it is not the case and design my application with this edge case in mind?
One example would be a script that increment 2 keys at once. I could receive a "cluster error" which means something went wrong and I have to execute my script again (and potentially end up with one key being incremented twice and the other once)
Assuming you are talking about a Lua script, for as long as you're passing the keys in the arguments (and not hardcoded in the script) you should be good. It will be all or nothing. If you are not using a Lua script - consider doing so
From EVAL command:
All Redis commands must be analyzed before execution to determine
which keys the command will operate on. In order for this to be true
for EVAL, keys must be passed explicitly. This is useful in many ways,
but especially to make sure Redis Cluster can forward your request to
the appropriate cluster node.
From AWS ElastiCache - Best Practices: Online Cluster Resizing:
During resharding, we recommend the following:
Avoid expensive commands – Avoid running any computationally and I/O
intensive operations, such as the KEYS and SMEMBERS commands. We
suggest this approach because these operations increase the load on
the cluster and have an impact on the performance of the cluster.
Instead, use the SCAN and SSCAN commands.
Follow Lua best practices – Avoid long running Lua scripts, and always
declare keys used in Lua scripts up front. We recommend this approach
to determine that the Lua script is not using cross slot commands.
Ensure that the keys used in Lua scripts belong to the same slot.
I understand that functionally Multi/Exec and Pipelining are designed to serve different purpose and features.
However, considering only performance for block writes, which would perform better. My understanding is that the Multi/Exec would create a single request, and Pipe lining will create individual requests, but will avoid RTT.
Multi/Exec is slower.
Multi/Exec also create individual requests which stored at server side that would be executed one by one when server receiving 'EXEC', and two more requests Multi & Exec.
Every request executed in transaction would check if the watched keys changed, which pipelining would never do.