Following error in mod_security error log:
/var/lib/mod_security/default_SESSION": No space left on device
We are using DETECTION_ONLY.
ls -lh /var/lib/mod_security
total 11M
-rw-r----- 1 daemon daemon 8.6M Mar 12 02:13 default_SESSION.dir
-rw-r----- 1 daemon daemon 118G Mar 12 03:24 default_SESSION.pag
The folder size is 11M showing but the file inside this is showing 118G.
What may be the reason? Since it already writing to log, what is the purpose of default_SESSION file? Can we prevent write to this file?
Related
That's weird because it happens only once every 5 nights.
I have a cron job which restarts httpd service every night, but I'm receiving this error once every 5 nights.
Apr 24 04:33:03 system systemd: httpd.service: main process exited, code=exited, status=254/n/a
Apr 24 04:33:03 system systemd: Unit httpd.service entered failed state.
Apr 24 04:33:03 system systemd: httpd.service failed.
And this is my script to restart httpd
...
/usr/sbin/httpd -k restart
...
I've checked permissions of /tmp folder and it all looks good.
File: ‘/tmp’
Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: 803h/2051d Inode: 2359297 Links: 24
Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-04-25 10:05:41.726022978 +0300
Modify: 2022-04-25 10:07:02.568018159 +0300
Change: 2022-04-25 10:07:02.568018159 +0300
Birth: -
And this is what i found in error_log of httpd.
Fatal Error Unable to create lock file: Bad file descriptor (9)
I have some trouble in starting Redis on CentOS7 with systemctl. What should I do to troubleshoot?
I can use the normal command to start the Redis. Like:
# /etc/init.d/redis start
or
/usr/local/bin/redis-server /etc/redis/config.conf
And here is my redis.service file which I put into /lib/systemd/system:
[Unit]
Description=Redis persistent key-value database
After=network.target
[Service]
Type=forking
PIDFILE=/var/run/redis_6379.pid
ExecStart=/etc/init.d/redis start
ExecStop=/etc/init.d/redis stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
But when I use command systemctl start redis to start redis server. I got nothing.
I try to use systemctl status redis to read the systemctl log, it shows me these messages:
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)
Active: active (exited) since Fri 2018-08-31 15:45:37 CST; 2 days ago
Aug 31 15:45:37 redisserver001 systemd[1]: Starting LSB: start and stop redis_6379...
Aug 31 15:45:37 redisserver001 systemd[1]: Started LSB: start and stop redis_6379.
Aug 31 15:45:37 redisserver001 redis[24755]: /var/run/redis_6379.pid exists, process is already running or crashed
Sep 03 10:31:21 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:6] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 10:33:13 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:6] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 10:45:32 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:7] Unknown lvalue 'PIDFILE' in section 'Service'
Sep 03 11:08:28 redisserver001 systemd[1]: [/usr/lib/systemd/system/redis.service:7] Unknown lvalue 'PIDFILE' in section 'Service'
The following items is the key configration that I think could impact the redis running. But I donn't know where I've make mistakes. Please help. Thanks a lot.
pidfile /var/run/redis_6379.pid
daemonize yes
supervised systemd
If an application specifies the "pidfile" property in the service file, then its the responsibility of the application to write the pid of the main process into that file, before the service initialization is complete. You need to make sure that your application is doing that. Systemd will read this value, and will prevent another forked process from being created if the user executes the "systemctl start ", and the pid file already exists. From the output you posted, it seems like systemd believes that the redis process is already running (because of the presence of the pid file, and doesnt create a new one). You can set the pid in the "ExecStartPost" clause of the service file. Something like:
ExecStartPost=/bin/sh -c 'umask 022; pgrep YOURSERVICE > /var/run/YOURSERVICE.pid'
The option must be PIDFile (case sensetive). From the manpage man systemd.service
PIDFile=
Takes a path referring to the PID file of the service. Usage of this option is recommended for
services where Type= is set to forking. The path specified typically points to a file below /run/. If
a relative path is specified it is hence prefixed with /run/. The service manager will read the PID
of the main process of the service from this file after start-up of the service. The service manager
will not write to the file configured here, although it will remove the file after the service has
shut down if it still exists. The PID file does not need to be owned by a privileged user, but if it
is owned by an unprivileged user additional safety restrictions are enforced: the file may not be a
symlink to a file owned by a different user (neither directly nor indirectly), and the PID file must
refer to a process already belonging to the service.
I use a celery worker server with redis as the broker url (for receiving tasks) as well as the result backend.
BROKER_URL = 'redis://localhost:6379/2'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/2'
app = Celery('myceleryapp', broker=BROKER_URL,backend=CELERY_RESULT_BACKEND)
I launch the celery worker server using celery -A myceleryapp worker -l info -c 8
The worker processes start processing my tasks from the redis queue until at some point, I receive the infamous MISCONF redis error and the celery worker process terminates.
Unrecoverable error: ResponseError('MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.',)
I checked the redis log files in /var/log/redis and the tail end of the file has the following
24745:C 19 Aug 09:20:26.169 * RDB: 0 MB of memory used by copy-on-write
1590:M 19 Aug 09:20:26.247 * Background saving terminated with success
1590:M 19 Aug 09:25:27.080 * 10 changes in 300 seconds. Saving...
1590:M 19 Aug 09:25:27.081 * Background saving started by pid 25397
25397:C 19 Aug 09:25:27.082 # Write error saving DB on disk: No space left on device
1590:M 19 Aug 09:25:27.181 # Backgroun1590:M 19 Aug 09:51:03.042 * 1 changes in 900 seconds. Saving...
1590:M 19 Aug 09:51:03.042 * Background saving started by pid 26341
26341:C 19 Aug 09:51:03.405 * DB saved on disk
26341:C 19 Aug 09:51:03.405 * RDB: 22 MB of memory used by copy-on-write
1590:M 19 Aug 09:51:03.487 * Background saving terminated with success
The dump.rdb file is being written to /var/lib/redis/dump.rdb.
Since the logs reported a No space left on device, I checked the disk space where /var is mounted and there seems to be sufficient space left (1.2GB).
How do I get to the root cause of this error if there is enough disk space? Of course, to prevent this error from happening, I could set config set stop-writes-on-bgsave-error no in redis-cli. But I want to get to the root cause of this error. Any help or pointers?
Maybe this is caused by the swap file. Because the swap file took the 1.2Gb space of your disk. So redis complains No space to write.
Try this "swapon -s" command to check this.
I think 1.2Gb is not enough if this disk accept the RAM page swap. you should change the dir of RDB in a more big dir.
I have had a node web service running successfully on an aws ubuntu server for over a month, with the requests cached using redis.
Yesterday I started getting the following error from some of my routes:
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
I was able to stop the error occurring by using:
config set stop-writes-on-bgsave-error no
as suggested in the answers to this question, but it doesn't actually solve the underlying problem.
To find the underlying problem I checked the logs and found the following had started happening:
[1105] 09 Aug 13:17:14.800 - 0 clients connected (0 slaves), 797680 bytes in use
[1105] 09 Aug 13:17:15.101 * 1 changes in 900 seconds. Saving...
[1105] 09 Aug 13:17:15.101 * Background saving started by pid 28090
[28090] 09 Aug 13:17:15.101 # Failed opening .rdb for saving: Permission denied
[1105] 09 Aug 13:17:15.201 # Background saving error
Over the weekend no one had been using the server, but before the weekend the logs were fine, and we were getting no errors:
[12521] 06 Aug 04:49:27.308 - 0 clients connected (0 slaves), 803352 bytes in use
[12521] 06 Aug 04:49:29.012 * 1 changes in 900 seconds. Saving...
[12521] 06 Aug 04:49:29.012 * Background saving started by pid 26663
[26663] 06 Aug 04:49:29.014 * DB saved on disk
[26663] 06 Aug 04:49:29.014 * RDB: 2 MB of memory used by copy-on-write
[12521] 06 Aug 04:49:29.112 * Background saving terminated with success
As I said, no one has touched this server in the intervening time.
Looking around for people having the same problem I found this question. I checked the ownership and permissions on the directory and db file as suggested in the answers there:
drwxr-xr-x 2 redis redis 26 Aug 6 06:55 redis
-rw-r--r-- 1 redis redis 18 Aug 6 06:55 dump-6379.rdb
The permissions and ownership both look ok to me, but I have noticed that the date on the file and folder is between the last time I saw the service working and the first time it failed. Unfortunately that hasn't really helped me with what to do next and I am at a bit of a loss.
I am looking for suggestions for next steps to find the cause of the problem, or at least a way of making redis able to write again.
I am trying to install Redmine on CentOS 6.3 but I continue to get this error in the log file
Passenger could not be initialized because of this error: Unable to start
the Phusion Passenger watchdog (/usr/lib/ruby/gems/1.8/gems/passenger-4.0.20/buildout
/agents/PassengerWatchdog): Permission denied (errno=13)
I have been looking online and cannot find this error anywhere or any way to fix it. I have tried changing permissions to the folder to 777 and apache:apache but neither work.
The only solution that I have come up with to get redmine to work is to set SELinux to disabled or permissive (which I do not want to do).
Does anyone have another way to fix this problem that leaves SELinux enabled?
Found the SELinux log file under /var/log/messages
here is the end of the file
Oct 16 14:07:30 localhost pulseaudio[2329]: alsa-util.c: Disabling timer-based scheduling because running inside a VM.
Oct 16 14:07:30 localhost rtkit-daemon[2183]: Sucessfully made thread 2331 of process 2329 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Oct 16 14:07:30 localhost pulseaudio[2329]: alsa-util.c: Disabling timer-based scheduling because running inside a VM.
Oct 16 14:07:30 localhost rtkit-daemon[2183]: Sucessfully made thread 2332 of process 2329 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Oct 16 14:07:31 localhost rtkit-daemon[2183]: Sucessfully made thread 2427 of process 2427 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Oct 16 14:07:31 localhost pulseaudio[2427]: pid.c: Daemon already running.
Oct 16 14:08:04 localhost kernel: type=1400 audit(1381957684.726:5): avc: denied { execute_no_trans } for pid=2663 comm="httpd" path="/usr/lib/ruby/gems/1.8/gems/passenger-4.0.20/buildout/agents/PassengerWatchdog" dev=dm-0 ino=1048752 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:lib_t:s0 tclass=file
Oct 16 14:08:04 localhost kernel: type=1400 audit(1381957684.760:6): avc: denied { execute_no_trans } for pid=2668 comm="httpd" path="/usr/lib/ruby/gems/1.8/gems/passenger-4.0.20/buildout/agents/PassengerWatchdog" dev=dm-0 ino=1048752 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:lib_t:s0 tclass=file
Oct 16 14:09:11 localhost pulseaudio[2329]: alsa-sink.c: ALSA woke us up to write new data to the device, but there was actually nothing to write!
Oct 16 14:09:11 localhost pulseaudio[2329]: alsa-sink.c: Most likely this is a bug in the ALSA driver 'snd_intel8x0'. Please report this issue to the ALSA developers.
Oct 16 14:09:11 localhost pulseaudio[2329]: alsa-sink.c: We were woken up with POLLOUT set -- however a subsequent snd_pcm_avail() returned 0 or another value < min_avail.
any suggestions?
So, you can fix this by using audit2allow (yum install audit-libs-python audit-libs).
SELinux logs to /var/log/audit/audit.log. If you tail and capture the output from restarting the web service (service httpd restart) you can then run the new output through audit2allow and make a module to install under selinux...
So, assuming you have captured it into a file called "audit_tmp":
cat audit_tmp | audit2allow -D -M passenger
This will create a file called passenger.pp which you can apply using:
semodule -i passenger.pp
Doing this will unblock the first thing that was stopping passenger from loading - but be aware that there will probably be more so you will need to repeats the process again until it works. I hope that makes sense!
Take a look at /var/log/syslog. That file contains SELinux error messages, which tell you how to fix up any permission problems.