getting this error during startup time of scylla node as I am loading data
Nov 12 21:55:13 usw1-im-stage-scylladb1 scylla[53703]: [shard 0] database - Keyspace product_prod: Reading CF cleanup_transaction id=bb0a0640-058f-11ea-b8e4-00000000000c version=dde3ee6f-185b-37ba-80fb-6425cce4532f
Nov 12 22:10:02 usw1-im-stage-scylladb1 systemd[1]: scylla-server.service start operation timed out. Terminating.
running this on scylla enterprise 2019.1.2
Scylla's documentation includes KBs and FAQ. I think this KB is spot-on, what you are looking for:
https://docs.scylladb.com/troubleshooting/scylla_wont_start/#solution
Here is the solution suggested there:
Locate the directory with the systemd files where the scylla-server.service resides.
For Centos operating systems it is expected to be under /usr/lib/systemd/system/scylla-server.service
For Ubuntu operating systems it is expected to be under /etc/systemd/system/scylla-server.service.d
Create the following directory (if not exist)
Centos
sudo mkdir /usr/lib/systemd/system/scylla-server.service
Ubuntu
sudo mkdir /etc/systemd/system/scylla-server.service.d
Create a file inside that directory named 10-timeout.conf, with the following contents:
[Service]
TimeoutStartSec=9000
Reload the systemd Daemon for the new configurations to take in effect.
systemctl daemon-reload
I want to start the httpd service on a CentOS 7 container. But the systemctl command doesn't work in containers. In CentOS 6 I can start httpd by simply using the /etc/init.d/apachectl -d command. But in CentOS 7 I can't find any apachectl file in /*/systemd/.
So how can I start httpd service in CentOS 7 container?
The best way is to make your own centos7 image where you install httpd
FROM centos:7
RUN yum -y install httpd; yum clean all; systemctl enable httpd.service
EXPOSE 80
Build your image with docker build -t my-centos:7 .
Systemd cannot run without SYS_ADMIN. That's why I set the following vars.
$ docker run -it -p 80:80 -e "container=docker" --privileged=true -d --security-opt seccomp:unconfined --cap-add=SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:ro my-centos:7 bash -c "/usr/sbin/init"
Verify container is running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
967581bdf31a my-centos:7 "bash -c /usr/sbin/in" 1 seconds ago Up 1 seconds 0.0.0.0:80->80/tcp gigantic_stallman
Verifiy httpd is started
$ docker exec -it gigantic_stallman /bin/bash -c "systemctl status httpd"
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-12-28 11:44:04 UTC; 2min 20s ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 61 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /docker/967581bdf31a3b741a5e857720e199614d816b05a2132271f3adf910f0ed3207/system.slice/httpd.service
├─61 /usr/sbin/httpd -DFOREGROUND
├─66 /usr/sbin/httpd -DFOREGROUND
├─67 /usr/sbin/httpd -DFOREGROUND
├─68 /usr/sbin/httpd -DFOREGROUND
├─69 /usr/sbin/httpd -DFOREGROUND
└─70 /usr/sbin/httpd -DFOREGROUND
Dec 28 11:44:04 967581bdf31a systemd[1]: Starting The Apache HTTP Server...
Dec 28 11:44:04 967581bdf31a httpd[61]: AH00558: httpd: Could not reliably d...e
Dec 28 11:44:04 967581bdf31a systemd[1]: Started The Apache HTTP Server.
TL;DR: For short answer please see the other author's Answer.
My question was wrong here because it doesn't align with the containerization philosophy IMO. As these kinds of questions would be asked by new user I am going to explain a few things that's indirectly related to this question.
What is a container?
From OCI's runtime Specification, approximately,
A container contains the unit of a deliverable software.
A container will encapsulate a software component and it's dependencies.
A container should be portable and platform agnostic.
And one of the major component to achieve containerization is container runtime or in general linux container. Container runtime is a piece of software that is responsible for running containers.
Examples of a few container runtimes are, containerd, docker-engine, crio, mcr etc.
Why the question is wrong?
In general and by design, a linux container is an isolated process (these days virtual machines are also considered as containers). So in an ideal situation we should create a container just with one process which is our deliverable software.
In the question, I was thinking of using systemd to manage the process inside the container because I was neither aware about difference between a virtual machine and a container nor the principles of OCI's specification.
Also, systemd or systemV is system management daemon that is required to manage systems with hundreds or thousands of process. As the desired number of process in a container is only one so we do not need a process management daemon or any other unnecessary tools like ssh, htop, net-tools, firewalld etc.
How do we run the deliverable software?
The ideal way to run an application inside the container is to use it as the container's Entrypoint or CMD. That means, when we run the container, it will try to initiate the Entrypoint and it will start it with default command defined in the CMD. Either way, the first process (PID 1) should be our desired application/software.
So when we build the container image, we should define the entrypoint of that container. For example, I have an httpd and a redis container.
╰──➤ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
23789e2d0416 redis "docker-entrypoint.s…" 36 seconds ago Up 35 seconds 6379/tcp elegant_ganguly
9be725968ff3 httpd "httpd-foreground" 14 minutes ago Up 14 minutes 80/tcp app1
So let's check the first process of the both containers (cat /proc/1/cmdline),
╰──➤ for i in $(docker ps -q); do docker inspect $i --format 'ImageName: {{.Config.Image}}'; printf "First PID: "; docker exec -i $i sh -c "cat /proc/1/cmdline";echo; done
ImageName: redis
First PID: redis-server *:6379
ImageName: httpd
First PID: httpd-DFOREGROUND
Let's try to see the same thing with ps
╰──➤ for i in $(docker ps -q); do docker inspect $i --format 'ImageName: {{.Config.Image}}'; docker run -i --rm --pid container:$i ubuntu sh -c "ps aux | head -n2"; done
ImageName: redis
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
999 1 0.4 0.0 56024 7760 ? Ssl 14:58 0:08 redis-server *:6379
ImageName: httpd
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 7300 4380 ? Ss 14:44 0:00 httpd -DFOREGROUND
So if we are using these images; majority of the time we do not need to start it separately because it's probably already invoked by the entrypoint.
But if we want to create our own container image for our own software we can do that just by mentioning the entrypoint like the both of the httpd and redis image did here and here. You can also use CMD and Entrypoint from the command line when you run the container with the help of --entrypoint or provide the command after container name like the following (here I am using while true; do date; sleep 1; done as the default CMD),
╰──➤ docker run -d --rm ubuntu sh -c "while true; do date; sleep 1; done"
35c6352a55f25335e1bd0874493f2a31155ef752d008eb6718923d1f04ab2c14
Now let's check the first PID,
╰──➤ docker run -i --rm --pid container:35c6352a55f25 ubuntu sh -c "ps aux | head -n2"
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2308 832 ? Ss 15:42 0:00 sh -c while true; do date; sleep 1; done
Run the docker pull command to download the Docker image, including Apache named httpd.
#docker pull httpd
Check the docker images using
#docker images
Now run the docker command to invoke the image you downloaded.
#docker run -d --name docker-apache -p 80:80 -d httpd
Mapping the local computer's port 80 to the container's port 80 (-p 80:80).
Try to verify whether the apache web server is working by accessing the server IP or hostname in the browser.
I am getting below error when i try to run httpd.
[root#localhost bin]# ./apachectl -k start
./apachectl: line 78: 22059 Segmentation fault (core dumped) $HTTPD $ARGV
This is the apachectl startup script lines are as:
74 if [ "x$ARGV" = "x" ] ; then
75 ARGV="-h"
76 fi
77
78 case $ARGV in
79 start|stop|restart|graceful|graceful-stop)
80 $HTTPD -k $ARGV
81 ERROR=$?
82 ;;
83 startssl|sslstart|start-SSL)
84 echo The startssl option is no longer supported.
85 echo Please edit httpd.conf to include the SSL configuration settings
86 echo and then use "apachectl start".
87 ERROR=2
88 ;;
Checked the messages on /var/log
Mar 21 09:45:11 localhost kernel: httpd[22059]: segfault at 8 ip 00000039d360b453 sp 00007fff7e5e80a0 error 4 in ld-2.12.so[39d3600000+20000]
Mar 21 09:45:11 localhost abrtd: Directory 'ccpp-2016-03-21-09:45:11-22059' creation detected
Mar 21 09:45:11 localhost abrt[22060]: Saved core dump of pid 22059 (/apps/httpd-2.2.22/bin/httpd) to /var/spool/abrt/ccpp-2016-03-21-09:45:11-22059 (1904640 bytes)
Mar 21 09:45:11 localhost abrtd: Executable '/apps/httpd-2.2.22/bin/httpd' doesn't belong to any package
Mar 21 09:45:11 localhost abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-03-21-09:45:11-22059' exited with 1
Mar 21 09:45:11 localhost abrtd: Corrupted or bad directory '/var/spool/abrt/ccpp-2016-03-21-09:45:11-22059', deleting
Do I need to install something for "Executable '/apps/httpd-2.2.22/bin/httpd' doesn't belong to any package" error ? Sorry, am completely new to httpd and linux installations.
Was able to fix this. I commented this line in httpd.conf
Include conf/mod-jk.conf
and saw no error. Checked mod-jk.conf is having entry for mod_jk.so. Tried changing mod_jk.so and things worked fine.
Seems like the version of mod_jk.so was a wrong one.
I also got the same error but my Operating system was ubuntu. I checked error log in /var/log/syslog and there I found libphp5.6.so has some error like below.
Feb 7 11:57:45 Varun kernel: [ 698.096661] apache2[3897]: segfault at 841f4f ip 00007f979593f3ae sp 00007ffca776ac40 error 4 in libphp5.6.so[7f97956d5000+3b3000]
Apache normal log files were not showing above error.
So I disabled php5.6 via sudo a2dismod php5.6 && sudo service apache2 reload
After that apache worked fine.
I am running apache
$ httpd -v
Server version: Apache/2.2.15 (Unix)
Server built: Oct 16 2014 14:48:21
and it runs fine but when I do yum install mod_ssl, apache seg faults. It seg faults right away when I try to list modules using httpd -M.
$ httpd -M
Segmentation fault
If I try to restart apache, it seg faults again
$ sudo service httpd restart
Stopping httpd: [ OK ]
Starting httpd: /bin/bash: line 1: 1627 Segmentation fault /usr/sbin/httpd
[FAILED]
the problem is with CentOS it seems, because I found this resource:
https://chrisjean.com/adding-ssl-support-to-apache-on-centos/
so all I had to do was make sure Apache was loading the SSL module and everything is ok. Hopefully this helps someone else in the future.
The Redis startup script is supposed to create a pid file at startup, but I've confirmed all the settings I can find, and no pid file is ever created.
I installed redis by:
$ yum install redis
$ chkconfig redis on
$ service redis start
In my config file (/etc/redis.conf) I checked to make sure these were enabled:
daemonize yes
pidfile /var/run/redis/redis.pid
And in the startup script (/etc/init.d/redis) there is:
exec="/usr/sbin/$name"
pidfile="/var/run/redis/redis.pid"
REDIS_CONFIG="/etc/redis.conf"
[ -e /etc/sysconfig/redis ] && . /etc/sysconfig/redis
lockfile=/var/lock/subsys/redis
start() {
[ -f $REDIS_CONFIG ] || exit 6
[ -x $exec ] || exit 5
echo -n $"Starting $name: "
daemon --user ${REDIS_USER-redis} "$exec $REDIS_CONFIG"
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $name: "
killproc -p $pidfile $name
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
These are the settings that came by default with the install. Any idea why no pid file is created? I need to use it for Monit.
(The system is RHEL 6.4 btw)
For those experiencing on Debian buster:
Editing
nano /etc/systemd/system/redis.service
and adding this line below redis [Service]
ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/redis/redis.pid"
It suppose to look like this:
[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
ExecStop=/bin/kill -s TERM $MAINPID
ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/redis/redis.pid"
PIDFile=/run/redis/redis-server.pid
then:
sudo systemctl daemon-reload
sudo systemctl restart redis.service
Check redis.service status:
sudo systemctl status redis.service
The pid file now should appear.
On my Ubuntu 18.04, I was getting the same error.
Error reported by redis (on /var/log/redis/redis-server.log):
# Creating Server TCP listening socket ::1:6379: bind: Cannot assign requested address
This is because I've disabled IPv6 on this host and redis-server package (version 5:4.0.9-1) for Ubuntu comes with:
bind 127.0.0.1 ::1
Editing /etc/redis/redis.conf and removing the ::1 address solves the problem. Example:
bind 127.0.0.1
Edit: As pointed out in the comments (thanks to #nicholas-vasilaki and #tommyalvarez), by default redis only allows connections from localhost. Commenting all the line, using:
# bind 127.0.0.1 ::1
works, but makes redis listen from the network (not only from localhost).
More details can be found in redis configuration file.
Problem was that the user redis did not have permission to create the pid file (or directory it was in). Fix:
sudo mkdir /var/run/redis
sudo chown redis /var/run/redis
Then I killed and restarted redis and sure enough, there was redis.pid
In CentOs 7 i need to add to the file:
$ vi /usr/lib/systemd/system/redis.service
The next line:
ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/redis/redis.pid"
And then restart the service:
$ sudo systemctl daemon-reload
$ sudo systemctl restart redis.service
Reference:
CentOs 7: Systemd & PID File
i had a similar problem on Debian Buster, systemd complains about the missing PID file, even though the file exists and redis is running.
on my system the solution using "echo $MAINPID > /run/redis/redis.pid" works by accident, although/because the real PID file is set to /run/redis/redis-server.pid (spot the different filenames!) and on my system the content of /run/redis/redis.pid (the one of the echo) was empty.
in a discussion on systemd-devel#lists.freedesktop.org someone writes:
... systemd will add the MAINPID environment variable any time it
knows what the main PID is. It learns this by reading the PID file ...
So by the time ExecStartPost runs, the main PID may or may not be
known.
having an empty MAINPID environment variable can be even harmful: if you notice the different PID filenames in the suggested solution, and correct it, you may end up in a situation where the PID file written by redis gets overwritten by an empty file. this happened to me, the result was that systemctl start redis.service never finished.
i also noticed that another server with 100% same OS and configuration, but different hardware did not have this problem.
my conclusion is that it just hits some sort of race condition, systemd seems to look for a PID file just a little too early. on my system, whatever command i used as ExecStartPost, it will add enough delay to make the error disappear.
therefore a solution is to use "sleep 1" (sleep 0.1 works too, but 1 second may be on the safe side):
ExecStartPost=/bin/sleep 1
/etc/systemd/system/redis.service now looks like:
[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
ExecStartPost=/bin/sleep 1
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/run/redis/redis-server.pid
...
an alternative solution is to use "supervised systemd":
/etc/redis/redis.conf:
# If you run Redis from upstart or systemd, Redis can interact with your
# supervision tree. Options:
# supervised no - no supervision interaction
# supervised upstart - signal upstart by putting Redis into SIGSTOP mode
# supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET
# supervised auto - detect upstart or systemd method based on
# UPSTART_JOB or NOTIFY_SOCKET environment variables
# Note: these supervision methods only signal "process is ready."
# They do not enable continuous liveness pings back to your supervisor.
supervised systemd
override the redis-server.service file using:
systemctl edit redis-server.service
and enter the following:
[Service]
Type=notify
reload the service and the error should be gone:
sudo systemctl restart redis.service
sudo systemctl status redis.service
Here from 2018
Before start, I am on Ubuntu 18.04.I wrote this if anyone comes here
by searching same error.
In my case error is the same but problem is so different. No solutions that proposed here worked.
So I checked logs if they are exist and looked for is there anything useful. Found them on;
cat /var/log/redis/redis-server.log
Searched logs and found that problem is that another service is listening same port.
2963:C 21 Sep 11:07:33.007 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2963:C 21 Sep 11:07:33.008 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=2963, just started
2963:C 21 Sep 11:07:33.008 # Configuration loaded
2974:M 21 Sep 11:07:33.009 # Creating Server TCP listening socket 127.0.0.1:6379: bind: Address already in use
I checked who is listening.
netstat anp | grep 6379
Found it.
tcp6 0 0 :::6379 :::* LISTEN 3036/docker-proxy
It was docker image of redis that installed by another tool
root#yavuz:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6a94d401700 redis:3.2 "docker-entrypoint.s…" 20 hours ago Up 3 hours 0.0.0.0:6379->6379/tcp incubatorsuperset_redis_1
So I stopped docker image
root#yavuz:~# docker stop incubatorsuperset_redis_1
And redis-server started without problem.
root#yavuz:~# systemctl start redis-server
root#yavuz:~# systemctl status redis-server
● redis-server.service - Advanced key-value store
Active: active (running) since Fri 2018-09-21 11:10:34 +03; 1min 49s ago
Process: 3671 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
For CentOS:
In my case name of Redis server is redis.service, start it edit
systemctl edit redis.service
Add this:
[Service]
ExecStartPost=/bin/sh -c "echo $MAINPID > /var/run/redis/redis.pid"
PIDFile=/var/run/redis/redis.pid
Im my case it create file: /etc/systemd/system/redis.service.d/override.conf
After restart service:
systemctl daemon-reload
systemctl restart redis
And the pid file is:
cat /var/run/redis/redis.pid
=> 19755
sudo nano /etc/redis/redis.conf
Inside the file, find the supervised directive. This directive allows you to declare an init system to manage Redis as a service, providing you with more control over its operation. The supervised directive is set to no by default. Since you are running Ubuntu, which uses the systemd init system, change this to systemd.
My default, Redis does not run as a daemon, and that is why it does not create a pid file. If you look at /etc/redis/redis.conf, it says so explicitly under General.
#By default Redis does not run as a daemon. Use 'yes' if you need it...
daemonize no
So all you need to do is to change it to daemonize yes
For people struggling with getting it to work on Ubuntu 18.04 you need to edit /etc/redis/redis.conf and update the pidfile declaration to following:
pidfile "/var/run/redis/redis-server.pid"
Ubuntu 18. /var/run/redis had the wrong permissions:
drwxr-sr-x 2 redis redis 60 Apr 27 12:22 redis
Changed to 755 (drwxrwxr-x) and the pid file now appears.