i use reddison client but when the client has error “MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk”
Unable to send PING command over channel: [id: 0x04130153, L:/171.20.0.8:38080 - R:10.3.236.102/10.3.236.102:6379]
org.redisson.client.RedisException: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.. channel: [id: 0x04130153, L:/171.20.0.8:38080 - R:10.3.236.102/10.3.236.102:6379] command: (PING), params: []
the redis server has no error
{"log":"3443340:C 09 Apr 00:12:41.648 * DB saved on disk\n","stream":"stdout","time":"2022-04-09T00:12:41.649083457Z"}
{"log":"3443340:C 09 Apr 00:12:41.772 * RDB: 38 MB of memory used by copy-on-write\n","stream":"stdout","time":"2022-04-09T00:12:41.77335587Z"}
{"log":"7:M 09 Apr 00:12:42.024 * Background saving terminated with success\n","stream":"stdout","time":"2022-04-09T00:12:42.025019006Z"}
{"log":"7:M 09 Apr 00:12:45.027 *
beacuse the server time is not correct
Related
Redis change its own config dir to /etc/cron.d and dbfile to ntp instead of default configuration. Once we restart the redis it will reset to /var/lib/redis and dump.rdb but after awhile, it gives "Failed opening the RDB file" error
Default dire and rdb file has correct permission and redis only allow for internal IPs.
cli output
127.0.0.1:6381> CONFIG GET dir
1) "dir"
2) "/etc/cron.d"
127.0.0.1:6381> CONFIG GET "dbfilename"
1) "dbfilename"
2) "ntp"
/var/log/redis/redis-server.log
3204:M 21 May 16:07:19.124 * Background saving terminated with success
3204:M 21 May 16:12:18.962 * 10000 changes in 60 seconds. Saving...
3204:M 21 May 16:12:18.967 * Background saving started by pid 25469
25469:C 21 May 16:12:20.931 * DB saved on disk
25469:C 21 May 16:12:20.934 * RDB: 3 MB of memory used by copy-on-write
3204:M 21 May 16:12:20.968 * Background saving terminated with success
3204:M 21 May 16:17:21.082 * 10 changes in 300 seconds. Saving...
3204:M 21 May 16:17:21.088 * Background saving started by pid 25865
25865:C 21 May 16:17:22.800 * DB saved on disk
25865:C 21 May 16:17:22.803 * RDB: 3 MB of memory used by copy-on-write
3204:M 21 May 16:17:22.891 * Background saving terminated with success
3204:M 21 May 16:17:43.669 # Failed opening the RDB file root (in server root dir /var/spool/cron) for saving: Read-only file system
3204:M 21 May 16:17:45.320 # Failed opening the RDB file ntp (in server root dir /etc/cron.d) for saving: Read-only file system
3204:M 21 May 16:22:23.086 * 10 changes in 300 seconds. Saving...
3204:M 21 May 16:22:23.092 * Background saving started by pid 26264
26264:C 21 May 16:22:23.093 # Failed opening the RDB file ntp (in server root dir /etc/cron.d) for saving: Read-only file system
3204:M 21 May 16:22:23.194 # Background saving error
3204:M 21 May 16:22:29.104 * 10 changes in 300 seconds. Saving...
3204:M 21 May 16:22:29.109 * Background saving started by pid 26265
26265:C 21 May 16:22:29.109 # Failed opening the RDB file ntp (in server root dir /etc/cron.d) for saving: Read-only file system
3204:M 21 May 16:22:29.209 # Background saving error
3204:M 21 May 16:22:35.016 * 10 changes in 300 seconds. Saving...
Is your server publicly accessibly over the internet?
The most likely explanation is that somebody is connecting to the redis and sending commands to reconfigure it remotely, trying to take control over the server.
There are bots scanning the internet 24/7 looking for exposed software and known vulnerabilities. Quick rule of thumb is that a new service coming up online will be discovered and attacked in less than 5 minutes. (Try running an unpatched Windows XP server and be amazed how short it lasts).
Consider that redis and potentially the whole server was compromised. I hope for you that there was no sensitive information in this redis or that's a data breach.
Block public access, decommission the virtual machine, setup a new one from scratch.
Related redis ticket: https://github.com/antirez/redis/issues/3594
I have set up a Redis cluster on Kubernetes, the cluster state is OK and the replica is connected to the master. Also as per the logs, the full synchronization is also completed. The logs are as follows:-
9:M 22 Oct 12:24:18.209 * Slave 192.168.1.41:6379 asks for synchronization
9:M 22 Oct 12:24:18.209 * Partial resynchronization not accepted: Replication ID mismatch (Slave asked for '794b9c74abe40ac90c752f32a102078e063ff636', my replication IDs are '0f499740a46665d12fab921838297273279ad136' and '0000000000000000000000000000000000000000')
9:M 22 Oct 12:24:18.209 * Starting BGSAVE for SYNC with target: disk
9:M 22 Oct 12:24:18.211 * Background saving started by pid 231
231:C 22 Oct 12:24:18.215 * DB saved on disk
231:C 22 Oct 12:24:18.216 * RDB: 4 MB of memory used by copy-on-write
9:M 22 Oct 12:24:18.224 * Background saving terminated with success
9:M 22 Oct 12:24:18.224 * Synchronization with slave 192.168.1.41:6379 succeeded
Still, when I check the List of the PubSub Channels on the replica, it does not show the channels and thus it breaks the PubSub flow.
Any help/advise is appreciated.
I have some trouble in starting Redis on CentOS7 with systemctl. What should I do to troubleshoot?
I can use the normal command to start the Redis. Like:
# /etc/init.d/redis start
or
/usr/local/bin/redis-server /etc/redis/config.conf
And here is my redis.service file which I put into /lib/systemd/system:
[Unit]
Description=Redis persistent key-value database
After=network.target
[Service]
Type=forking
PIDFILE=/var/run/redis_6379.pid
ExecStart=/etc/init.d/redis start
ExecStop=/etc/init.d/redis stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
But when I use command systemctl start redis to start redis server. I got nothing.
I try to use systemctl status redis to read the systemctl log, it shows me these messages:
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)
Active: active (exited) since Fri 2018-08-31 15:45:37 CST; 2 days ago
Aug 31 15:45:37 redisserver001 systemd[1]: Starting LSB: start and stop redis_6379...
Aug 31 15:45:37 redisserver001 systemd[1]: Started LSB: start and stop redis_6379.
Aug 31 15:45:37 redisserver001 redis[24755]: /var/run/redis_6379.pid exists, process is already running or crashed
Sep 03 10:31:21 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:6] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 10:33:13 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:6] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 10:45:32 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:7] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 11:08:28 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:7] Unknown lvalue 'PIDFILE' in section 'Service'
The following items is the key configration that I think could impact the redis running. But I donn't know where I've make mistakes. Please help. Thanks a lot.
pidfile /var/run/redis_6379.pid
daemonize yes
supervised systemd
If an application specifies the "pidfile" property in the service file, then its the responsibility of the application to write the pid of the main process into that file, before the service initialization is complete. You need to make sure that your application is doing that. Systemd will read this value, and will prevent another forked process from being created if the user executes the "systemctl start ", and the pid file already exists. From the output you posted, it seems like systemd believes that the redis process is already running (because of the presence of the pid file, and doesnt create a new one). You can set the pid in the "ExecStartPost" clause of the service file. Something like:
ExecStartPost=/bin/sh -c 'umask 022; pgrep YOURSERVICE > /var/run/YOURSERVICE.pid'
The option must be PIDFile (case sensetive). From the manpage man systemd.service
PIDFile=
Takes a path referring to the PID file of the service. Usage of this option is recommended for
services where Type= is set to forking. The path specified typically points to a file below /run/. If
a relative path is specified it is hence prefixed with /run/. The service manager will read the PID
of the main process of the service from this file after start-up of the service. The service manager
will not write to the file configured here, although it will remove the file after the service has
shut down if it still exists. The PID file does not need to be owned by a privileged user, but if it
is owned by an unprivileged user additional safety restrictions are enforced: the file may not be a
symlink to a file owned by a different user (neither directly nor indirectly), and the PID file must
refer to a process already belonging to the service.
I use a celery worker server with redis as the broker url (for receiving tasks) as well as the result backend.
BROKER_URL = 'redis://localhost:6379/2'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/2'
app = Celery('myceleryapp', broker=BROKER_URL,backend=CELERY_RESULT_BACKEND)
I launch the celery worker server using celery -A myceleryapp worker -l info -c 8
The worker processes start processing my tasks from the redis queue until at some point, I receive the infamous MISCONF redis error and the celery worker process terminates.
Unrecoverable error: ResponseError('MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.',)
I checked the redis log files in /var/log/redis and the tail end of the file has the following
24745:C 19 Aug 09:20:26.169 * RDB: 0 MB of memory used by copy-on-write
1590:M 19 Aug 09:20:26.247 * Background saving terminated with success
1590:M 19 Aug 09:25:27.080 * 10 changes in 300 seconds. Saving...
1590:M 19 Aug 09:25:27.081 * Background saving started by pid 25397
25397:C 19 Aug 09:25:27.082 # Write error saving DB on disk: No space left on device
1590:M 19 Aug 09:25:27.181 # Backgroun1590:M 19 Aug 09:51:03.042 * 1 changes in 900 seconds. Saving...
1590:M 19 Aug 09:51:03.042 * Background saving started by pid 26341
26341:C 19 Aug 09:51:03.405 * DB saved on disk
26341:C 19 Aug 09:51:03.405 * RDB: 22 MB of memory used by copy-on-write
1590:M 19 Aug 09:51:03.487 * Background saving terminated with success
The dump.rdb file is being written to /var/lib/redis/dump.rdb.
Since the logs reported a No space left on device, I checked the disk space where /var is mounted and there seems to be sufficient space left (1.2GB).
How do I get to the root cause of this error if there is enough disk space? Of course, to prevent this error from happening, I could set config set stop-writes-on-bgsave-error no in redis-cli. But I want to get to the root cause of this error. Any help or pointers?
Maybe this is caused by the swap file. Because the swap file took the 1.2Gb space of your disk. So redis complains No space to write.
Try this "swapon -s" command to check this.
I think 1.2Gb is not enough if this disk accept the RAM page swap. you should change the dir of RDB in a more big dir.
I have had a node web service running successfully on an aws ubuntu server for over a month, with the requests cached using redis.
Yesterday I started getting the following error from some of my routes:
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
I was able to stop the error occurring by using:
config set stop-writes-on-bgsave-error no
as suggested in the answers to this question, but it doesn't actually solve the underlying problem.
To find the underlying problem I checked the logs and found the following had started happening:
[1105] 09 Aug 13:17:14.800 - 0 clients connected (0 slaves), 797680 bytes in use
[1105] 09 Aug 13:17:15.101 * 1 changes in 900 seconds. Saving...
[1105] 09 Aug 13:17:15.101 * Background saving started by pid 28090
[28090] 09 Aug 13:17:15.101 # Failed opening .rdb for saving: Permission denied
[1105] 09 Aug 13:17:15.201 # Background saving error
Over the weekend no one had been using the server, but before the weekend the logs were fine, and we were getting no errors:
[12521] 06 Aug 04:49:27.308 - 0 clients connected (0 slaves), 803352 bytes in use
[12521] 06 Aug 04:49:29.012 * 1 changes in 900 seconds. Saving...
[12521] 06 Aug 04:49:29.012 * Background saving started by pid 26663
[26663] 06 Aug 04:49:29.014 * DB saved on disk
[26663] 06 Aug 04:49:29.014 * RDB: 2 MB of memory used by copy-on-write
[12521] 06 Aug 04:49:29.112 * Background saving terminated with success
As I said, no one has touched this server in the intervening time.
Looking around for people having the same problem I found this question. I checked the ownership and permissions on the directory and db file as suggested in the answers there:
drwxr-xr-x 2 redis redis 26 Aug 6 06:55 redis
-rw-r--r-- 1 redis redis 18 Aug 6 06:55 dump-6379.rdb
The permissions and ownership both look ok to me, but I have noticed that the date on the file and folder is between the last time I saw the service working and the first time it failed. Unfortunately that hasn't really helped me with what to do next and I am at a bit of a loss.
I am looking for suggestions for next steps to find the cause of the problem, or at least a way of making redis able to write again.