Because of the security concerns I had to rename DEL command on all of the 3 Redis Sentinel nodes. Master is able to expire its keys and remove them off the memory. However Slaves are not able to remove the keys. Eventually memory gets maxed up and node becomes unstable.
I checked the redis.log file:
== CRITICAL == This replica is sending an error to its master: 'unknown command DEL, with args beginning with: test-11146, ' after processing the command ''
When I comment out renaming the DEL command on redis.conf file, things get back to working again.
I think this issue has been around for a while as I came across a few Google Searches with similar context.
rename-command PEXPIRE WF4RSp2s3a
rename-command DEL m6T1V5Fn1a
rename-command CONFIG 3Zg871qH1a
I would appreciate if someone can point me in the right direction.
Related
I want to reset the statistics redis shows me in INFO command.
I read about https://redis.io/commands/config-resetstat
But for some reason it returns error:
redis> CONFIG RESETSTAT
ERR unknown command `CONFIG`, with args beginning with: `RESETSTAT`,
What am I doing wrong? I don't want to restart server, since I use digitalocean and I don't have direct SSH access, I have only access to redis-cli.
#for_stack is right, i want to expand the answer. It is not possible to use config command unless you/administrator modify the configuration file which requires a server access and restart.
How they disabled?
The CONFIG command was renamed into an unguessable name.
rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
It is also possible to completely disable it (or any other command) by renaming it to the empty string.
rename-command CONFIG ""
You need to request your administrator(if it is possible) to remove that lines from the configuration file.
After running fine for a while, I am getting write error on my redis instance:
(error) MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.
In the log I see:
9948:C 22 Mar 20:49:32.241 # Failed opening the RDB file root (in server root dir /var/spool/cron) for saving: Read-only file system
However, my redis config file is /etc/redis/redis.conf as confirmed by:
redis-cli -p 6379 info | grep 'config_file'
config_file:/etc/redis/redis.conf
And there I have:
dir /mnt/data/redis
And indeed, there is a snapshot there.
But despite the above, redis now thinks my data directory is
redis-cli -p 6379 CONFIG GET dir
1) "dir"
2) "/var/spool/cron"
Corresponding to the error I was getting as quoted above.
Can anyone tell me why/how my data directory is changing after redis starts, such that it is no longer what is specified in the config file?
So the answer is that the redis server was hacked and the configuration changed, which is very easy to do as it turns out. (I should point out that I had no reason to think it wasn't easy to do. I just assumed security by obscurity was sufficient in this case--wrong. No matter, this was just a playground not any sort of production server).
So don't open your redis port to the world. Use security groups if on AWS to limit access to machines that need it, or use AUTH (which is still not awesome because then all clients need to know the single password which also apparently gets sent in the clear), or have some middleware controlling access.
Hacking redis is easy to do, can compromise your data, and even enable unauthorized SSH access to your server. And that's why you shouldn't highline.
I have a Redis cluster of 6 nodes, running in my Kubernetes cluster as a stateful set. Since it was for testing and not yet on production, all of the Redis nodes were on the same machine. Of course, the machine failed, and all of Redis' nodes crashed immediately.
When the machine was back alive the pods were recreated and was given different cluster ips, therefore they couldn't re-connect with each other.
I need to find a solution for a disaster case such as this. Let's assume all the nodes were reassigned with different ips, how can I configure the nodes to get to other ips?
The slaves are easy to reset with the CLUSTER RESET command, but the masters contain slots and data that shouldn't be deleted.
Should I manually re-write nodes.conf? I'm afraid this will make it even worse? I there a known method to deal with it?
Thanks!
Found a solution:
The first step is to change the current pod ip in nodes.conf when the pod is starting. You can achieve that with this script
#!/bin/sh
CLUSTER_CONFIG="/data/nodes.conf"
if [ -f ${CLUSTER_CONFIG} ]; then
if [ -z "${POD_IP}" ]; then
echo "Unable to determine Pod IP address!"
exit 1
fi
echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}
fi
exec "$#"
You should start any pod by calling this script and passing to it the original redis-server start command.
Now every pod in the cluster has the correct IP of itself set.
Make sure the cluster's pods are stable and don't crash.
Manually edit nodes.conf in one of the pods. Set the correct IPs instead of the deprecated ones.
Restart the pod you've edit with redis-cli shutdown. Kubernetes will set up a new pod for it. The new pod's IP will be set by the script I added above.
In my opinion you shouldn't rely on Pods' internal IP address at all, when referencing your Redis cluster in any place within your application. Pods are mortal, this means they are designed to crash. So when node dies, they are destroyed too. When node resurrects, PODs are recreated with new IP addresses.
The proper way to target your PODs would be via their DNS names (as explained here), if you created your Redis cluster as a Stateful application.
I recently configured Redis to use AOF as well as RDB snapshotting.
However, it does not look like the AOF is replayed correctly on server startup.
I stopped the service. Then I made sure /var/redis/appendonly.aof is valid using redis-check-aof.
Then I started the server again. In this moment, the RDB file was empty. That's another issue I need to look into - Redis started losing all the data from time to time.
In the log file I can see the AOF is supposed to be loaded correctly:
DB loaded from append only file: 1.474 seconds
However, when I try to read a value which I know should be there, I get nothing:
127.0.0.1:6379> get iQube:Live:wordCount:2015:11:13:10:6
(nil)
In the AOF though, there are commands like this:
INCRBY
$36
iQube:Live:wordCount:2015:11:13:10:6
$1
2
*2
$4
Is there something else I need to do to make this work?
My fault. I did not secure the server properly and became target of probably the most typical attack to Redis. In effect, the AOF file contained flushall commands which wiped the DB clean upon loading.
At the very least, I recommend putting these three lines to redis.conf:
rename-command CONFIG someverylongandveryunguessablestring
rename-command FLUSHDB ""
rename-command FLUSHALL ""
How does one upgrade to a newer version of Redis with zero downtime? Redis slaves are read-only, so it seems like you'd have to take down the master and your site would be read-only for 45 seconds or more while you waited for it to reload the DB.
Is there a way around this?
Redis Team has very good documentation on this
Core Steps:
Setup your new Redis instance as a slave for your current Redis instance. In order to do so you need a different server, or a server that has enough RAM to keep two instances of Redis running at the same time.
If you use a single server, make sure that the slave is started in a different port than the master instance, otherwise the slave will not be able to start at all.
Wait for the replication initial synchronization to complete (check the slave log file).
Make sure using INFO that there are the same number of keys in the master and in the slave. Check with redis-cli that the slave is working as you wish and is replying to your commands.
Configure all your clients in order to use the new instance (that is, the slave).
Once you are sure that the master is no longer receiving any query (you can check this with the MONITOR command), elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
Full Documentation:
Upgrading or restarting a Redis instance without downtime
When taking the node offline, promote the slave to master using the SLAVEOF command, then when you bring it back online you set it up as a slave and it will copy all data from the online node.
You may also need to make sure your client can handle changed/missing master nodes appropriately.
If you want to get really fancy, you can set up your client to promote a slave if it detects an error writing to the master.
You can use Redis Sentinel for doing this, the sentinel will automatically promote a slave as new master.
you can find more info here http://redis.io/topics/sentinel.
Sentinel is a system used to manage redis servers , it monitors the redis master and slaves continuously, and whenever a master goes down it will automatically promote a slave in to master. and when the old master is UP it will be made as slave of the new master.
Here there will be no downtime or manual configuration of config file is needed.
You can visit above link to find out how to configure sentinel for your redis servers.
Note, you may have to check and set the following config to write to your slave.
("Since Redis 2.6 by default slaves are read-only")
redis-cli config set slave-read-only no
-- Example
-bash-4.1$ redis-cli info
Server
redis_version:2.6.9
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
(error) READONLY You can't write against a read only slave.
-bash-4.1$ redis-cli slaveof no one
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"
-bash-4.1$ redis-cli config set slave-read-only no
OK
-bash-4.1$ redis-cli slaveof admin2.mypersonalsite.com 6379
OK
-bash-4.1$ redis-cli set temp 42
OK
-bash-4.1$ redis-cli get temp
"42"