getting this error during startup time of scylla node as I am loading data
Nov 12 21:55:13 usw1-im-stage-scylladb1 scylla[53703]: [shard 0] database - Keyspace product_prod: Reading CF cleanup_transaction id=bb0a0640-058f-11ea-b8e4-00000000000c version=dde3ee6f-185b-37ba-80fb-6425cce4532f
Nov 12 22:10:02 usw1-im-stage-scylladb1 systemd[1]: scylla-server.service start operation timed out. Terminating.
running this on scylla enterprise 2019.1.2
Scylla's documentation includes KBs and FAQ. I think this KB is spot-on, what you are looking for:
https://docs.scylladb.com/troubleshooting/scylla_wont_start/#solution
Here is the solution suggested there:
Locate the directory with the systemd files where the scylla-server.service resides.
For Centos operating systems it is expected to be under /usr/lib/systemd/system/scylla-server.service
For Ubuntu operating systems it is expected to be under /etc/systemd/system/scylla-server.service.d
Create the following directory (if not exist)
Centos
sudo mkdir /usr/lib/systemd/system/scylla-server.service
Ubuntu
sudo mkdir /etc/systemd/system/scylla-server.service.d
Create a file inside that directory named 10-timeout.conf, with the following contents:
[Service]
TimeoutStartSec=9000
Reload the systemd Daemon for the new configurations to take in effect.
systemctl daemon-reload
Redis went quite on me.
user#mycomputer:~$ redis-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
I try to restart the service by doing this
sudo /etc/init.d/redis_6379 stop
/var/run/redis/redis.pid exists, process is already running or crashed
But no luck. Logs didn't show an error as well.
Got it fixed by backing up the redis.rdp file mine is located at
/var/lib/redis
check your config file "/etc/redis/redis.conf" for the rdp file's location and do this
sudo mv /var/lib/redis/redis.rdp /var/lib/redis/redis_backup.rdp
Then recreate the the redis.rdp file
sudo touch redis.rdp
Run the redis-server with the conf and it should work
sudo redis-server /etc/redis/redis.conf
Get it fixed in a tidy way: Recreate the the redis.rdp file as suggested here in one of answer, will purge all the cache recorded so far and redis will start up fresh with no cache data.
This is a warning message to notify system crash / improper shutdown: "/var/run/redis/redis.pid exists, process is already running or crashed"
Just delete /var/run/redis/redis.pid file and restart the server again.
Note: You might have lost latest cache changes due to untidy shutdown, which weren't flushed into the disk. This data loss can be minimized using frequent disk flush configuration in redis conf file(in my case it is #/etc/redis/6379.conf)
save 900 1
save 300 10
save 60 10000
Or try AOF persistence, more details [here][1]
Depends on how you installed redis, the pid can be found on /var/run/redis_6379.pid.
What happened is that redis crashed, but the pid is still there. So you just have to delete it.
sudo rm -f /var/run/redis_6379.pid
Then start redis again:
sudo /etc/init.d/redis_6379 start
If you can't find it, I suggest installing redis "more properly". Follow redis quickstart guide in the Installing Redis more properly section.
You can find it here:
https://redis.io/topics/quickstart
Run the redis-server with config.
sudo redis-server redis.conf
So after restarting my httpd redis crashed (due to the number of sudden requests sent via httpd and written on redis) and now when I try to restart redis on my centos 6.5 server I get the following error:
[root#host /]# /usr/sbin/redis-server restart
[1705] 17 Apr 00:30:49 # Fatal error, can't open config file 'restart'
I have also tried to login to redis using redis-cli and I get an error stating the connection to the server failed.
What options do I have to safely restart the server?
From the /src directory where you downloaded and unzipped your redis source, run the following. This is for RHEL based systems.
make install
# (OR)
sudo cp src/redis-server /usr/local/bin/
sudo cp src/redis-cli /usr/local/bin/
I have a redis server 2.8 installed using ubuntu apt-get on ubuntu 12.04.
I have copied a dump.rdb from an other database. Now when I try to start the new server, I constantly get:
[35763] 04 Mar 01:51:47.088 * 1 changes in 900 seconds. Saving...
[35763] 04 Mar 01:51:47.088 * Background saving started by pid 43313
[43313] 04 Mar 01:51:47.088 # Failed opening .rdb for saving: Permission denied
How can I solve this?
You should check your redis.conf file to see the permissions in dir and dbfilename. If the file named in the dbfilename which is located in the path specified in the dir path exists and the permission is also right. then the problem should be fixed.
Hope this will help someone.
P.S.
To find the redis.conf file location, you can use the #ps ax | grep redis to check. Usually it will be passed to the redis-server as input file.
For the dir permissions:it should be 755, for the dbfilename, it should be 644
Sometimes you also need to use top command to check whether the user:group of the redis-server and the owner of dir are consistent. i.e. The redis-server is running by redis:redis, but the dir is under root:root. In this case, you need to chown redis:redis -R dir.
Non of the above worked for me.. I've seen everyone around being so concerned on BGSAVE.. but while you're not on production, SAVE gives you a way more straight forward answer: ERR. BGSAVE does not, unless you inspect logs.
After digging dozens of posts I did not find any clue. The only thing that fixed was stopping the redis service and running it manually.
At first I thought it could be related to the user on behalf of redis was running. Not at all: the actual difference was the damn systemd subsystem which at some point in the redis config service file (/etc/systemd/system/redis.service) had the following:
ReadWriteDirectories: -/etc/redis
WoW super cool! ended up this was preventing redis from accessing anywhere in the system at all even though the permissions would perfectly allow it.
How naive of me to think that permission were just enough to ensure something had the proper rights.. (yes, I'm being ironic).
My /lib/system/systemd/redis-server.service file contained the following:
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
My /etc/redis/redis.conf file stated that the database should be located in /data/redis
dir /data/redis
The systemd config file above effectively makes /data/redis read-only.
Once I changed the redis.conf file to read:
dir /var/lib/redis
I stopped getting the error.
My permission issue seemed to be the result of the Redis user being unable to modify the parent folder (/var/lib/redis/6379) for the purposes of creating a temporary file.
This was seen in an strace of the redis-server process:
open("temp-1833.rdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)
The issue was resolved after running the following command:
setfacl -m d:u:redis:rwX,u:redis:rwX /var/lib/redis/6379
For windows only :
This means the user does not have permission for this.
BY default owner of this file is NETWORK SERVICE, which has very limited access and need to changed(as per documentation)
solution :
go to ur redis folder.
right click --> go to properties--> security tab.
click on advanced.
click on Add to add ur user.
click on select a principal.
enter ur user (for eg GLOBAL\xxx).
click on check names and click on ok
give permissions to this user.
finally change the owner to this user.
Check the configuration 'dbfilename' in your redis.conf. Your redis running process have no write permission in the path.
In my case all rights were correct(I mean that the most stared answer doesn't help me). BUT! Redis used an incorrect path to file. In config it was correct, but from rails-cli it returned '/proc'.
This answer helped me - https://serverfault.com/questions/800295/redis-spontaneously-failed-failed-opening-rdb-for-saving-permission-denied
Warning
For exact question it doesn't matter, but my situation looked like someone hacked server. Link to explanation. So check your setup properly.
supervised systemd is intended solely for Type=notify and daemonize yes corresponds to Type=forking.
sudo vim /etc/systemd/system/redis.service
When you see the service file edit the Type=forking
[Unit]
Description=Redis In-Memory Data Store
After=network.target
[Service]
User=redis
Type=forking
Group=redis
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
ExecStop=/usr/bin/redis-cli shutdown
Restart=always
[Install]
WantedBy=multi-user.target
Open up this file
sudo vim /etc/redis/redis.conf
Add these changes to it
daemonize yes
supervised no
In my case when I typed the following command sudo tail -F /var/log/redis/redis-server.log I get this log:
987:C 08 Dec 22:28:30.344 # Can't chdir to '/var/lib/redis': Permission denied
1047:C 08 Dec 22:28:30.565 # Can't chdir to '/var/lib/redis': Permission denied
1095:C 08 Dec 22:28:30.876 # Can't chdir to '/var/lib/redis': Permission denied
1119:C 08 Dec 22:28:31.165 # Can't chdir to '/var/lib/redis': Permission denied
1151:C 08 Dec 22:28:31.413 # Can't chdir to '/var/lib/redis': Permission denied
1500:C 08 Dec 22:30:44.706 # Can't chdir to '/var/lib/redis': Permission denied
1523:C 08 Dec 22:30:45.194 # Can't chdir to '/var/lib/redis': Permission denied
1545:C 08 Dec 22:30:45.442 # Can't chdir to '/var/lib/redis': Permission denied
1568:C 08 Dec 22:30:45.696 # Can't chdir to '/var/lib/redis': Permission denied
1590:C 08 Dec 22:30:45.940 # Can't chdir to '/var/lib/redis': Permission denied
That means the user redis doesn't have permission on /var/lib/redis.
That's why I typed this command sudo ls -l /var/lib/redis to see the permissions in this directory. I get the following log:
-rw-r--r-- 1 root root 885 Dec 8 22:12 dump.rdb
That means it was associated with root instead of redis.
Then I typed the following command to change the owner of that directory: sudo chown -R redis:redis /var/lib/redis/.
Then I restarted redis-server by the following command: sudo systemctl restart redis-server.
Boom!! It worked.
Hope this will work for someone, who have similar issue as mine.
If anyone encounters this again and doesn't have a problem upgrading, just upgrade your Redis installation to the latest version. I encountered this problem with Redis 2.8.15, and upgraded to Redis 2.8.22 that was available at the time of this writing. A sysadmin in my company assured me that this was a bug with Redis 2.8.15, and the problem just went away after upgrading.
I had the same issue with redis used by Sidekiq in Rails application, rm -rf ./tmp/ worked like charm.
I spent some time on this until i realised that my cmd line session was running in the wrong drive :/. Just in case this might help anyone else!
The lock file in the log directory is what was causing this error for me. I was able to clear the error by deleting the lock file:
rm /var/log/redis/lock.
This happened when another system was restored to this one while redis was still running.
No one hasn't mentioned about SELinux.
On Centos most probably you will have such error when selinux mode = enforcing.
Just check getenforce and if it set to 'enforcing' , hit setenforce 0 and try one more time to run service.
If you are on Windows and the Redis folder was installed in C: \ Program Files \ Redis for example, you will have a problem with access permission. Modifying files within the Program Files folder, usually requires administrator permission and dump.rdb is within this context. In your redis.conf, modify the default directory to anywhere outside the Program Files folder:
from: dir ./
to: dir ../../Exceptions/Redis/
Note that I went up the directory twice to leave the Program Files folder and outside of it I created any directory (C:\Exceptions\Redis). In this directory Redis can save the .rdb file without permission problems.
Does anyone know what can cause this error in Apache's error log (particularly in Arch Linux):
PID file /run/httpd/httpd.pid not readable (yet?) after start.
The error is not stopping Apache from running normally (once it is started), but is potentially slowing down my service restart time significantly.
Any guidance would be appreciated.
The solution is to adjust the file /usr/lib/tmpfiles.d/apache.conf. The default values are wrong. You have to change this
d /run/httpd 0755 root root -
to
d /run/httpd 0755 http http -
.
After restarting apache the PID file will be created with chown http:http and not chown root:root.
Ressources:
https://mailman.archlinux.org/pipermail/arch-general/2013-July/033891.html
https://mailman.archlinux.org/pipermail/arch-general/2013-July/033900.html
My take is that this bug is a feature: Nothing I've done has been able to get rid of it (tried everything suggested here and then some) yet the software runs just fine. So, I take it as a slightly mis-written success message - I've started!
I have met this issue:
PID file *.pid not readable (yet?) after start.
Recent days, my apache server was compiling from the package-issued version is httpd-2.4.12.tar.gz. Here is my httpd.service contents, following this my problem got fixed. Hope this can help anyone who have met this.
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
ExecStart=/usr/local/apache/bin/apachectl start -DFOREGROUND
PIDFile=/run/apache/apache.pid
ExecReload=/usr/local/apache/bin/apachectl restart
ExecStop=/user/local/apache/bin/apachectl stop
PrivateTmp=True
[Install]
WantedBy=multi-user.target