As you now in Redis database when you run KEYS * command the Redis will lock database until keys return all keys.
I want to create 2 separate db in Redis and create some key in each of them ,then select one of them and run keys command on that db.
Will Redis lock all available db till answer ready or only lock selected db?
TL;DR: yes.
Redis doesn't lock - it blocks on (almost1) all commands because it is single threaded. When the server executes a command, be it a simple GET or the evil KEYS, it is busy serving it and does nothing else. The longer it takes a command to complete, the longer the server is blocked.
KEYS is a long-running command because it always traverses the entire keyspace (regardless the pattern), not to mention the potentially-huge reply it makes.
That same single thread of execution also handles numbered, a.k.a. shared, databases. Any operation you perform on one of the databases blocks the entire server, all databases included. More information can be found at: https://redislabs.com/blog/benchmark-shared-vs-dedicated-redis-instances/
1 BGSAVE, for example, is one of the few commands that do not block. As of v4, there's also UNLINK and more are planned to be added.
Related
Two issues
Do lua scripts really solve all cases for redis transactions?
What are best practices for asynchronous transactions from one client?
Let me explain, first issue
Redis transactions are limited, with an inability to unwatch specific keys, and all keys being unwatched upon exec; we are limited to a single ongoing transaction on a given client.
I've seen threads where many redis users claim that lua scripts are all they need. Even the redis official docs state they may remove transactions in favour of lua scripts. However, there are cases where this is insufficient, such as the most standard case: using redis as a cache.
Let's say we want to cache some data from a persistent data store, in redis. Here's a quick process:
Check cache -> miss
Load data from database
Store in redis
However, what if, between step 2 (loading data), and step 3 (storing in redis) the data is updated by another client?
The data stored in redis would be stale. So... we use a redis transaction right? We watch the key before loading from db, and if the key is updated somewhere else before storage, storage would fail. Great! However, within an atomic lua script, we cannot load data from an external database, so lua cannot be used here. Hopefully I'm simply missing something, or there is something wrong with our process.
Moving on to the 2nd issue (asynchronous transactions)
Let's say we have a socket.io cluster which processes various messages, and requests for a game, for high speed communication between server and client. This cluster is written in node.js with appropriate use of promises and asynchronous concepts.
Say two requests hit a server in our cluster, which require data to be loaded and cached in redis. Using our transaction from above, multiple keys could be watched, and multiple multi->exec transactions would run in overlapping order on one redis connection. Once the first exec is run, all watched keys will be unwatched, even if the other transaction is still running. This may allow the second transaction to succeed when it should have failed.
These overlaps could happen in totally separate requests happening on the same server, or even sometimes in the same request if multiple data types need to load at the same time.
What is best practice here? Do we need to create a separate redis connection for every individual transaction? Seems like we would lose a lot of speed, and we would see many connections created just from one server if this is case.
As an alternative we could use redlock / mutex locking instead of redis transactions, but this is slow by comparison.
Any help appreciated!
I have received the following, after my query was escalated to redis engineers:
Hi Jeremy,
Your method using multiple backend connections would be the expected way to handle the problem. We do not see anything wrong with multiple backend connections, each using an optimistic Redis transaction (WATCH/MULTI/EXEC) - there is no chance that the “second transaction will succeed where it should have failed”.
Using LUA is not a good fit for this problem.
Best Regards,
The Redis Labs Team
We are trying to implement caching for our multi-tenant application. We are planning to create new Redis DB for each tenant.
We have one scenario where we need to use Redis Transactions. While going through this post https://redis.io/topics/transactions, we found that
All the commands in a transaction are serialized and executed
sequentially. It can never happen that a request issued by another
client is served in the middle of the execution of a Redis
transaction. This guarantees that the commands are executed as a
single isolated operation.
Is this read blocking will only apply to database level or at full instance level?
The guarantee you quoted applies to the instance, not the database. A command for DB 2 will not run in the middle of a transaction for DB 1.
You can find more information about multiple databases (including an argument by the creator of Redis against using them at all) in this question.
When I run a rename command, I think it does something like this,
Use new name for new data
remove reference for old name
remove old data (this can take some time if it’s large)
For clients accessing this data, is there ever a time where any of these happen?
The key does not exist
The data is not in a good state
Redis hangs during access
What steps are performed during a Redis rename command?
Since Redis has single threaded execution of commands, the rename will be atomic, so the answer to 1 and 2 are no. The thing about it "removing old data" is only if the destination key already points to a large structure that it needs to delete (Redis will clobber it.) The original data object will not be copied. Only hash table entries pointing to it might be moved around. Since rehashing in Redis is incremental, this should essentially be constant time.
Redis will always "hang" on slow commands due to the single threaded command execution. So for 3, it can always be yes depending on what you're doing, but in this case, only if you're doing significantly large implicit delete.
Edit: as of Redis 4.0 you can actually specify the config option lazyfree-lazy-server-del yes (default is no) and the server will actually delete asynchronously for side-effect deletes such as this. In other words, instead of delete blocking, the object will be queued for background deletion. This would effectively make RENAME constant time. See sample cfg: https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf
I want to be able to access a very recent copy of my master Redis server keys. It doesn't have to be completely up to date as I will be polling the read only copy but I don't want the transactions and Lua scripts I run on the master instance to block on the read only instance as I SCAN through the keys on my read only instance.
Can anyone confirm/deny this behaviour?
It won't block the slaves from anything, but while the master is busy processing your logic replication will be stopped. Once the logic ends (possibly generating writes), replication will resume with the previous buffered contents and the new ones (if any).
We have big shopping and product dealing system. We have faced lots problem with MySQL so after few r&D we planned to use Redis and we start integrating Redis in our system.
Following this previously directly hitting the database now we have moved the Redis system
User shopping cart details
Affiliates clicks tracking records
We have product dealing user data.
other site stats.
I am not only storing the data in Redis system i have written crons which moves Redis data in MySQL data at time intervals. This is the main point i am facing the issues.
Bellow points i am looking for solution
Is their any other ways to dump big data from Redis to MySQL?
Redis fail our store data in file so is it possible to store that data directly to MySQL database?
Is Redis have any trigger system using that i can avoid the crons like queue system?
Is their any other way to dump big data from Redis to MySQL?
Redis has the possibility (using bgsave) to generate a dump of the data in a non blocking and consistent way.
https://github.com/sripathikrishnan/redis-rdb-tools
You could use Sripathi Krishnan's well-known package to parse a redis dump file (RDB) in Python, and populate the MySQL instance offline. Or you can convert the Redis dump to JSON format, and write scripts in any language you want to populate MySQL.
This solution is only interesting if you want to copy the complete data of the Redis instance into MySQL.
Does Redis have any trigger system that i can use to avoid the crons like queue system?
Redis has no trigger concept, but nothing prevents you to post events in Redis queues each time something must be copied to MySQL. For instance, instead of:
# Add an item to a user shopping cart
RPUSH user:<id>:cart <item>
you could execute:
# Add an item to a user shopping cart
MULTI
RPUSH user:<id>:cart <item>
RPUSH cart_to_mysql <id>:<item>
EXEC
The MULTI/EXEC block makes it atomic and consistent. Then you just have to write a little daemon waiting on items of the cart_to_mysql queue (using BLPOP commands). For each dequeued item, the daemon has to fetch the relevant data from Redis, and populate the MySQL instance.
Redis fail our store data in file so is it possible to store that data directly to MySQL database?
I'm not sure I understand the question here. But if you use the above solution, the latency between Redis updates and MySQL updates will be quite limited. So if Redis fails, you will only loose the very last operations (contrary to a solution based on cron jobs). It is of course not possible to have 100% consistency in the propagation of data though.