We're having a severe config/product bug on our installation. We've been experiencing concurrency related errors that we've been blaming to Jedis usage, but it seems that it might be a product / config issue.
This is a single redis installation with over 4M keys. Whenever we run a long running command from redis-cli, like a keys *, our client code (Jedis based) starts to throw errors, like trying to cast a string into a binary (typical concurrency errors in Jedis conf). The worst scenario is that sometimes it seems that it returns wrong keys. We were using a Jedis instance in each actor instance, so it shouldn't be an issue but we changed to JedisPool nevertheless. But the problem remained (we are using Jedis 2.6.2).
But the main thing was when trying from different redis-cli. We run KEYS * that stays a long time running and then a GET command which returned. It was our understanding that the KEYS * should block everyone, but the GET command keeps working. This also happens with a SLEEP command.
Is this related to a config setting or this is something that shouldn't happen, or the KEYS command isn't blocking and my problem lies elsewhere?
Redis.io documentation for KEYS clearly states that KEYS is a debug command and should not be used in production:
Warning: consider KEYS as a command that should only be used in
production environments with extreme care. It may ruin performance
when it is executed against large databases. This command is intended
for debugging and special operations, such as changing your keyspace
layout. Don't use KEYS in your regular application code. If you're
looking for a way to find keys in a subset of your keyspace, consider
using SCAN or sets.
Related
I have a Redis cluster that is still active and not sure if it is still been used. Logged in and tried command KEYS * and found around 30k+ keys.
How to identify if it is still been used if not can process to terminate?
First of all, you shouldn't use KEYS command in production, it can jam your Redis. Use SCAN instead.
If you mean you would like to know if some queries are still being made against Redis, you can use the Monitor command. Be careful though, it reduces your throughput by about 50%, so only use when necessary.
The web application I am working with is written in Django and is using Redis to cache some data from Elasticsearch. Yesterday everything was working fine, but today it started to give me an error. I looked over the structure of the data redis is storing for the key and some of the deep inner values for keys were changed to lists instead of dicts (that they are supposed to be). So, overnight redis data was modified by someone or something. Now I need a way to figure out which code changed it. If I launch the app after doing redic-cli flushdb and start using it, navigating here and there everything is working fine, and I can't find any apparent wrong code this way. The data for redis is set only in one place in the app code and it is set correctly. I looked into redis.log but it does not say which key it modified and when but this data could be crucial here.
So, I need to find out which code mistakenly modified the key. It could be separately run code by someone, could be some hidden specific side of the app (I doubt it is the case), or some bug within the redis itself. Maybe I would need to introduce some kind of additional observer that would run each time keys are changed writing when and which key was modified in redis. I am stuck and not sure what I could do here. Any suggestions would be greatly appreciated.
A few things you may try with Redis:
MONITOR is a debugging command that streams back every command processed by the Redis server. You may then see what command is modifying your key, from what client connection.
Redis Keyspace Notifications allow you to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way. You can subscribe to the key of interest.
CLIENT LIST command returns information and statistics about the client connections server in a mostly human readable format.
As you are suspicious from another code or someone modifying your data, you may want to use Redis 6 with ACLs, to control what clients can do what operations on what keys.
I have some scripts that touch a handful of keys. What happens if Elasticache decides to reshard while my script is running? Will it wait for my script to complete before it moves the underlying keys? Or should I assume that it is not the case and design my application with this edge case in mind?
One example would be a script that increment 2 keys at once. I could receive a "cluster error" which means something went wrong and I have to execute my script again (and potentially end up with one key being incremented twice and the other once)
Assuming you are talking about a Lua script, for as long as you're passing the keys in the arguments (and not hardcoded in the script) you should be good. It will be all or nothing. If you are not using a Lua script - consider doing so
From EVAL command:
All Redis commands must be analyzed before execution to determine
which keys the command will operate on. In order for this to be true
for EVAL, keys must be passed explicitly. This is useful in many ways,
but especially to make sure Redis Cluster can forward your request to
the appropriate cluster node.
From AWS ElastiCache - Best Practices: Online Cluster Resizing:
During resharding, we recommend the following:
Avoid expensive commands – Avoid running any computationally and I/O
intensive operations, such as the KEYS and SMEMBERS commands. We
suggest this approach because these operations increase the load on
the cluster and have an impact on the performance of the cluster.
Instead, use the SCAN and SSCAN commands.
Follow Lua best practices – Avoid long running Lua scripts, and always
declare keys used in Lua scripts up front. We recommend this approach
to determine that the Lua script is not using cross slot commands.
Ensure that the keys used in Lua scripts belong to the same slot.
I have an issue with using lua scripts running on redis
When running INFO command
used_memory_lua_human:1.08G
The usage of Lua is not extensive (single set and get commands)
How can I reduce this value?
Apparently Redis caches every Lua script that is running on it, in order to spare loading it.
This is a good feature given the set of the scripts is limited.
The problem caused because we are formatting the script with different variable every execution, and it gets a different identifier every execution.
The solution I found is running SCRIPT FLUSH command after every execution, in order to remove it from cache
The explanation of why this is happening is explained by gCoh's answer very well.
The only problem is that running SCRIPT FLUSH can lead to a lot of errors and even loss of processing if you use EVALSHA to run LUA scripts. (For example, a library like scripto runs LUA scripts using EVALSHA). Read more about EVALSHA here
EVALSHA keeps the script cached in redis server and gives out a hash when you load the script for the first time. When you run the script second time, rather than loading the script on the server it just sends the script hash and redis executes it.
Now if you run SCRIPT FLUSH it will remove all the cached scripts and your SHA hashes will become obsolete, which will lead to runtime errors when the script is being executed and it isn't found.
For me it was this: NOSCRIPT No matching script. Please use EVAL.
This should be handled in the scripto library, where if the script isn't found it should load it again but until them, please take care that this doesn't happen with you.
Problem:
I am getting junk values like "OK" for redis get call.
This issue reproduces often over a particular period of time irrespective of the keys trying to obtain through get command.
I'm Using :
Redis version 2.8
Jedis client 2.5.1 to connect Redis
Please suggest some solution to resolve this issue.
The problem is outlined in this page. From the article:
I learned a hard lesson when enabling Redis transactions in the Spring RedisTemplate class redisTemplate.setEnableTransactionSupport(true);: Redis started returning junk data after running for a few days, causing serious data corruption. A similar case was reported on StackOverflow.
By running a monitor command, my team discovered that after a Redis operation or RedisCallback, Spring doesn't close the Redis connection automatically, as it should do. Reusing an unclosed connection may return junk data from an unexpected key in Redis. Interestingly, this issue doesn't show up when transaction support is set to false in RedisTemplate.
We discovered that we could make Spring close Redis connections automatically by configuring a PlatformTransactionManager (such as DataSourceTransactionManager) in the Spring context, then using the #Transactional annotation to declare the scope of Redis transactions.
Based on this experience, we believe it's good practice to configure two separate RedisTemplates in the Spring context: One with transaction set to false is used on most Redis operations; the other with transaction enabled is only applied to Redis transactions. Of course PlatformTransactionManager and #Transactional must be declared to prevent junk values from being returned.
Moreover, we learned the downside of mixing a Redis transaction with a relational database transaction, in this case JDBC. Mixed transactions do not behave as you would expect.