I'm doing some diagnostics on a system (I didn't develop), and it's running Redis and Bull to queue API calls. I need to see if/why some API calls are failing. I have access to a web interface of Redis Commander.
There's 90,000+ keys, and when you expand the "tree", it takes several minutes to load.
I would just like a command to run that shows me the last 10 keys added.
(or a command that tells me the ID # of the last key, and another command that let's me display the data of the key, without needing to expand the tree)
I'm guessing this is really simple, I'm just inexperienced with Redis.
MONITOR is a debugging command that streams back every command processed by the Redis server. It can help in understanding what is happening to the database. This command can both be used via redis-cli and via telnet.
Related
I am connecting to an AWS redis cluster using following command
redis-cli -c -h host.amazonaws.com -p 6379
I pushed two key "X1" and "X2" into redis cache from a springboot application (API methods are not annotated with #Cacheable) and now when I run KEYS * from cli terminal it would list either "X1" or "X2" but not both. GET for both keys works fine though.
info keyspace return following;
Keyspace
db0:keys=11,expires=1,avg_ttl=1975400
What am I missing here?
You probably have cluster mode enabled. In cluster mode, the data you store is partitioned by key. One of the advantages of this is that you can now have a larger dataset than would reasonably fit on one machine (hundreds of terabytes, if you want) since every shard has some fraction of the entire data set.
A downside is that multi-key commands no long work like you would expect if the keys end up in different hash slots. The KEYS command is such a multi-key command.
To make a long story short:
KEYS is apparently giving you only the keys on the cluster node you're hitting. It would perhaps have been nicer to give you an error, instead, but it doesn't.
GET is unaffected: redis-cli, with the -c flag, knows how to find the right cluster node (perhaps after hitting the wrong one and being told the key has MOVED).
If you ask every individual primary node in your cluster for KEYS *, and add up all the results, you should get all the keys. This question has some examples of using the redis-cli to do this.
The web application I am working with is written in Django and is using Redis to cache some data from Elasticsearch. Yesterday everything was working fine, but today it started to give me an error. I looked over the structure of the data redis is storing for the key and some of the deep inner values for keys were changed to lists instead of dicts (that they are supposed to be). So, overnight redis data was modified by someone or something. Now I need a way to figure out which code changed it. If I launch the app after doing redic-cli flushdb and start using it, navigating here and there everything is working fine, and I can't find any apparent wrong code this way. The data for redis is set only in one place in the app code and it is set correctly. I looked into redis.log but it does not say which key it modified and when but this data could be crucial here.
So, I need to find out which code mistakenly modified the key. It could be separately run code by someone, could be some hidden specific side of the app (I doubt it is the case), or some bug within the redis itself. Maybe I would need to introduce some kind of additional observer that would run each time keys are changed writing when and which key was modified in redis. I am stuck and not sure what I could do here. Any suggestions would be greatly appreciated.
A few things you may try with Redis:
MONITOR is a debugging command that streams back every command processed by the Redis server. You may then see what command is modifying your key, from what client connection.
Redis Keyspace Notifications allow you to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way. You can subscribe to the key of interest.
CLIENT LIST command returns information and statistics about the client connections server in a mostly human readable format.
As you are suspicious from another code or someone modifying your data, you may want to use Redis 6 with ACLs, to control what clients can do what operations on what keys.
I'm getting over 10000 updates in 60 seconds in my Redis server and this triggers the background save which consumes resources.
I want to track the changed keys so that I can debug my app (which method causing this much change).
Is there a way to get updated keys?
While MONITOR is perfectly valid, it does include everything that gets sent to Redis. That means filtering read requests, pings, ...
Instead I recommend that you check the keyspace notifications documentation and configure your database the AK flags. By subscribing to the __keyspace:* pattern you'll be notified about every change to keys.
As I learned, it's only possible by using MONITOR command and figure out from output.
We want to use pan.sh to execute multiple kettle transformations. After exploring the script I found that it internally calls spoon.sh script which runs in PDI. Now the problem is every time a new transformation starts it create a separate JVM for its executions(invoked via a .bat file), however I want to group them to use single JVM to overcome memory constraints that the multiple JVM are putting on the batch server.
Could somebody guide me on how can I achieve this or share the documentation/resources with me.
Thanks for the good work.
Use Carte. This is exactly what this is for. You can startup a server (on the local box if you like) and then submit your jobs to it. One JVM, one heap, shared resource.
Benefit of that is then scalability, so when your box becomes too busy just add another one, also using carte and start sending some of the jobs to that other server.
There's an old but still current blog here:
http://diethardsteiner.blogspot.co.uk/2011/01/pentaho-data-integration-remote.html
As well as doco on the pentaho website.
Starting the server is as simple as:
carte.sh <hostname> <port>
There is also a status page, which you can use to query your carte servers, so if you have a cluster of servers, you can pick a quiet one to send your job to.
We're having a severe config/product bug on our installation. We've been experiencing concurrency related errors that we've been blaming to Jedis usage, but it seems that it might be a product / config issue.
This is a single redis installation with over 4M keys. Whenever we run a long running command from redis-cli, like a keys *, our client code (Jedis based) starts to throw errors, like trying to cast a string into a binary (typical concurrency errors in Jedis conf). The worst scenario is that sometimes it seems that it returns wrong keys. We were using a Jedis instance in each actor instance, so it shouldn't be an issue but we changed to JedisPool nevertheless. But the problem remained (we are using Jedis 2.6.2).
But the main thing was when trying from different redis-cli. We run KEYS * that stays a long time running and then a GET command which returned. It was our understanding that the KEYS * should block everyone, but the GET command keeps working. This also happens with a SLEEP command.
Is this related to a config setting or this is something that shouldn't happen, or the KEYS command isn't blocking and my problem lies elsewhere?
Redis.io documentation for KEYS clearly states that KEYS is a debug command and should not be used in production:
Warning: consider KEYS as a command that should only be used in
production environments with extreme care. It may ruin performance
when it is executed against large databases. This command is intended
for debugging and special operations, such as changing your keyspace
layout. Don't use KEYS in your regular application code. If you're
looking for a way to find keys in a subset of your keyspace, consider
using SCAN or sets.