Can XREAD (or perhaps another command) be used to atomically detect whether data was written to a Redis stream?
More specifically:
Suppose you added some data to a Redis stream in one process and saw that the data was added successfully with some auto generated key.
XADD somestream foo bar
After this XADD completes, you immediately run the following read in another process.
XREAD COUNT 1000 STREAMS somestream 0-0
Is this XREAD guaranteed to return data? The documentation is not clear about whether a successful XADD guarantees that readers will immediately see the added data, or whether there might be some small delay.
Redis's famous single threaded architecture answers that question. When you execute XADD on one process(client side) and after another process(client side) executes XREAD then the server execute them consecutively which guarantees that the data will be there before XREAD is executed.
The next quotes are from The Little Redis Book
Every Redis command is atomic, including the ones that do multiple things. Additionally, Redis has support for transactions when using multiple commands.
You might not know it, but Redis is actually single-threaded, which is how every command is guaranteed to be atomic.
While one command is executing, no other command will run. (We’ll briefly talk about scaling in a later chapter.) This
is particularly useful when you consider that some commands do multiple things.
Related
First of all, my understanding is: redis is a single-process program, all commands are executed in first-in-first-out order. If this is the case, we don't need the watch command, but this is not the case.
I want to find out more about the order of execution of the redis command. Thanks in advance
You are correct, the Redis server will execute, the command in the order they are received independently of the client.
That said, it is interesting to know that you have some features like transaction and pipelining that do not have a direct impact on the execution order (not totally for a transaction, as you will see below)
Transactions
In a transaction, "all the commands in a transaction are serialized and executed sequentially". All the commands are executed as a single isolated operation.
So when you are running commands in the transaction, it is not possible to have commands from another client to be executed before the end of the transaction.
Pipelining
As described above the operation will be executed in order (FIFO), using pipelining that does not change, but what is different is that the client is able to send multiple commands without waiting for the response.
I let you look into the details of all this and test it in your application if needed.
As you now in Redis database when you run KEYS * command the Redis will lock database until keys return all keys.
I want to create 2 separate db in Redis and create some key in each of them ,then select one of them and run keys command on that db.
Will Redis lock all available db till answer ready or only lock selected db?
TL;DR: yes.
Redis doesn't lock - it blocks on (almost1) all commands because it is single threaded. When the server executes a command, be it a simple GET or the evil KEYS, it is busy serving it and does nothing else. The longer it takes a command to complete, the longer the server is blocked.
KEYS is a long-running command because it always traverses the entire keyspace (regardless the pattern), not to mention the potentially-huge reply it makes.
That same single thread of execution also handles numbered, a.k.a. shared, databases. Any operation you perform on one of the databases blocks the entire server, all databases included. More information can be found at: https://redislabs.com/blog/benchmark-shared-vs-dedicated-redis-instances/
1 BGSAVE, for example, is one of the few commands that do not block. As of v4, there's also UNLINK and more are planned to be added.
all. I write a data item to Redis. Then later, I read the data item out of Redis.
Since there may be multiple servers taking these Redis requests and satisfying them, if I make the write request 1 ms before I make the read request (suppose they're both being done by the same process) am I assured that the read request won't be processed first, and I get a response back like "that data item doesn't exist"?
Assuming that the commands are being issued in sequence, you can assume that they will be atomic and single-threaded operations. Read more about this in this stackoverflow answer.
The above is True for a single Redis server, and not guaranteed for a Cluster behavior (thanks #mwp). In that situation, I'd recommend adding a check at the client level. If key doesn't exist when Redis makes its GET call, by default a nil value is returned.
Last note: depending on your implementation you may want to look into storing your redis items in a list, LPUSH-ing the write requests and BRPOP-ing the out values so you would always be guaranteed a value exists...
I used to use PostgreSQL sequence SELECT nextval('number'); to make sure I get a new number even if there are several clients since nextval is guaranteed to return distinct and increasing values.
I wanted to use the same mechanism with Redis.
I thought using the INCR command but I want to make sure I well understand that it will give the same guarantee than the nextval command?
There is no risk of getting the same number when several processes run this command or do I have to use a Redis lock?
Thanks
Redis is a single-threaded engine, so the execution of all commands is serialized. It always provides atomicity and isolation (in the sense of ACID) for each individual command.
The consequence is applying a single INCR command and exploiting its result is safe (even if multiple connections do it concurrently).
If redis gets overloaded, can I configure it to drop set requests? I have an application where data is updated in real time (10-15 times a second per item) for a large number of items. The values are outdated quickly and I don't need any kind of consistency.
I would also like to compute parallel sum of the values that are written in real time. What's the best option here? LUA executed in redis? Small app located on the same box as redis using UNIX sockets?
When Redis gets overloaded it will just slow down its clients. For most commands, the protocol itself is synchronous.
Redis supports pipelining though, but there is no way a client can cancel the traffic still in the pipeline, but not yet acknowledged by the server. Redis itself does not really queue the incoming traffic, the TCP stack does it.
So it is not possible to configure the Redis server to drop set requests. However, it is possible to implement a last value queue on client side:
the queue is actually a represented by 2 maps indexed by your items (only one value stored per item). The primary map will be used by the application. The secondary map will be used by a specific thread. The 2 maps content can be swapped in an atomic way.
a specific thread is blocking when the primary map is empty. When it is not, it swaps the content of the two maps, sends the content of the secondary map asynchronously to Redis using aggressive pipelining and variadic parameters commands. It also receives ack from Redis.
while the thread is working with the secondary map, the application can still fill the primary map. If Redis is too slow, the application will only accumulate last values in the primary map.
This strategy could be implemented in C with hiredis and the event loop of your choice.
However, it is not trivial to implement, so I would first check if the performance of Redis against all the traffic is not enough for my purpose. It is not uncommon to benchmark Redis at more than 500K op/s these days (using a single core). Nothing prevents you to shard your data on multiple Redis instances if needed.
You will likely saturate the network links before the CPU of the Redis server. That's why it is better to implement the last value queue (if needed) on client side rather than server side.
Regarding the sum computation, I would try to calculate and maintain it in real time. For instance, the GETSET command can be used to set a new value while returning the previous one.
Instead of just setting your values, you could do:
[old value] = GETSET item <new value>
INCRBY mysum [new value] - [old value]
The mysum key will contain the sum of your values for all the items at any time. With Redis 2.6, you can use Lua to encaspulate this calculation to save roundtrips.
Running a big batch to calculate statistics on existing data (this is how I understand your "parallel" sum) is not really suitable for Redis. It is not designed for map/reduce like computation.