I'm an unable to start apache2 service because of the following error when it tries to start the Passenger UstRouter:
[ 2023-02-14 04:32:47.6529 353516/7f992aa77980 age/Wat/WatchdogMain.cpp:1291 ]: Starting Passenger watchdog...
[ 2023-02-14 04:32:47.6911 353519/7fa5a50be980 age/Cor/CoreMain.cpp:982 ]: Starting Passenger core...
[ 2023-02-14 04:32:47.6913 353519/7fa5a50be980 age/Cor/CoreMain.cpp:235 ]: Passenger core running in multi-application mode.
[ 2023-02-14 04:32:47.6940 353519/7fa5a50be980 age/Cor/CoreMain.cpp:732 ]: Passenger core online, PID 353519
[ 2023-02-14 04:32:47.7323 353524/7f4e122d5980 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2023-02-14 04:32:47.7440 353524/7f4e122d5980 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 353524
AH00016: Configuration Failed
[ 2023-02-14 04:32:47.7792 353519/7fa59ffff700 age/Cor/CoreMain.cpp:531 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2023-02-14 04:32:47.7794 353519/7fa5a50be980 age/Cor/CoreMain.cpp:900 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2023-02-14 04:32:47.7795 353524/7f4e121e3700 age/Ust/UstRouterMain.cpp:421 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2023-02-14 04:32:47.7797 353519/7fa59f7fe700 Ser/Server.h:816 ]: [ServerThr.2] Freed 128 spare client objects
[ 2023-02-14 04:32:47.7798 353519/7fa59f7fe700 Ser/Server.h:464 ]: [ServerThr.2] Shutdown finished
[ 2023-02-14 04:32:47.7813 353519/7fa59effd700 Ser/Server.h:816 ]: [ApiServer] Freed 0 spare client objects
[ 2023-02-14 04:32:47.7813 353519/7fa59effd700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
[ 2023-02-14 04:32:47.7814 353519/7fa59ffff700 Ser/Server.h:816 ]: [ServerThr.1] Freed 128 spare client objects
[ 2023-02-14 04:32:47.7814 353519/7fa59ffff700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
I'm not familiar with Passenger UstRouter at all, so I've never tried to modify anything to with Passenger and thus unsure of what caused the problem in it's configuration. What steps can I follow to resolve this issue?
Related
My site/server is under DDOS ATTACK and apache keep shutting down, here is the error-log.
Would someone explain that and if there is a solution to this please advise!
Anyone has any suggestions? Appreciate it!
Server informations:
16/32 CPU cores, 128 GB RAM
Running nginx as reserve proxy in front of apache mpm_worker with php-fpm.
[ N 2022-06-02 20:32:49.6396 13663/T8 age/Cor/CoreMain.cpp:671 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2022-06-02 20:32:49.6397 13663/T1 age/Cor/CoreMain.cpp:1246 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2022-06-02 20:32:49.6398 13663/T8 Ser/Server.h:902 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/Ta Ser/Server.h:902 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/T10 Ser/Server.h:902 ]: [ServerThr.15] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/Te Ser/Server.h:902 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/Ta Ser/Server.h:558 ]: [ServerThr.2] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/T10 Ser/Server.h:558 ]: [ServerThr.15] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/Tk Ser/Server.h:902 ]: [ServerThr.7] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/To Ser/Server.h:902 ]: [ServerThr.9] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/Ti Ser/Server.h:902 ]: [ServerThr.6] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Te Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/Tw Ser/Server.h:902 ]: [ServerThr.13] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T16 Ser/Server.h:902 ]: [ServerThr.18] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6398 13663/Tm Ser/Server.h:902 ]: [ServerThr.8] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/To Ser/Server.h:558 ]: [ServerThr.9] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/Ti Ser/Server.h:558 ]: [ServerThr.6] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/T1g Ser/Server.h:902 ]: [ServerThr.23] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1k Ser/Server.h:902 ]: [ServerThr.25] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Tq Ser/Server.h:902 ]: [ServerThr.10] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T18 Ser/Server.h:902 ]: [ServerThr.19] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1a Ser/Server.h:902 ]: [ServerThr.20] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Tw Ser/Server.h:558 ]: [ServerThr.13] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/Tu Ser/Server.h:902 ]: [ServerThr.12] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Ty Ser/Server.h:902 ]: [ServerThr.14] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T12 Ser/Server.h:902 ]: [ServerThr.16] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Tk Ser/Server.h:558 ]: [ServerThr.7] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/Tm Ser/Server.h:558 ]: [ServerThr.8] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/Ts Ser/Server.h:902 ]: [ServerThr.11] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1o Ser/Server.h:902 ]: [ServerThr.27] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1c Ser/Server.h:902 ]: [ServerThr.21] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1g Ser/Server.h:558 ]: [ServerThr.23] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/T1e Ser/Server.h:902 ]: [ServerThr.22] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1k Ser/Server.h:558 ]: [ServerThr.25] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/T1w Ser/Server.h:902 ]: [ServerThr.31] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1i Ser/Server.h:902 ]: [ServerThr.24] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1y Ser/Server.h:902 ]: [ServerThr.32] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1s Ser/Server.h:902 ]: [ServerThr.29] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T1q Ser/Server.h:902 ]: [ServerThr.28] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/Tq Ser/Server.h:558 ]: [ServerThr.10] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/T18 Ser/Server.h:558 ]: [ServerThr.19] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T1a Ser/Server.h:558 ]: [ServerThr.20] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/Tu Ser/Server.h:558 ]: [ServerThr.12] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T14 Ser/Server.h:902 ]: [ServerThr.17] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6400 13663/Ty Ser/Server.h:558 ]: [ServerThr.14] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T12 Ser/Server.h:558 ]: [ServerThr.16] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/Ts Ser/Server.h:558 ]: [ServerThr.11] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T1o Ser/Server.h:558 ]: [ServerThr.27] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T1c Ser/Server.h:558 ]: [ServerThr.21] Shutdown finished
[ N 2022-06-02 20:32:49.6400 13663/T1u Ser/Server.h:902 ]: [ServerThr.30] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6401 13663/T1e Ser/Server.h:558 ]: [ServerThr.22] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T1w Ser/Server.h:558 ]: [ServerThr.31] Shutdown finished
[ N 2022-06-02 20:32:49.6402 13663/T1u Ser/Server.h:558 ]: [ServerThr.30] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T1y Ser/Server.h:558 ]: [ServerThr.32] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T1s Ser/Server.h:558 ]: [ServerThr.29] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T1q Ser/Server.h:558 ]: [ServerThr.28] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T14 Ser/Server.h:558 ]: [ServerThr.17] Shutdown finished
[ N 2022-06-02 20:32:49.6398 13663/Tc Ser/Server.h:902 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6399 13663/T16 Ser/Server.h:558 ]: [ServerThr.18] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/T1m Ser/Server.h:902 ]: [ServerThr.26] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6402 13663/Tc Ser/Server.h:558 ]: [ServerThr.3] Shutdown finished
[ N 2022-06-02 20:32:49.6399 13663/Tg Ser/Server.h:902 ]: [ServerThr.5] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6402 13663/Tg Ser/Server.h:558 ]: [ServerThr.5] Shutdown finished
[ N 2022-06-02 20:32:49.6402 13663/T1m Ser/Server.h:558 ]: [ServerThr.26] Shutdown finished
[ N 2022-06-02 20:32:49.6401 13663/T1i Ser/Server.h:558 ]: [ServerThr.24] Shutdown finished
[ N 2022-06-02 20:32:49.6403 13663/T20 Ser/Server.h:902 ]: [ApiServer] Freed 0 spare client objects
[ N 2022-06-02 20:32:49.6403 13663/T20 Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2022-06-02 20:32:49.6814 15272/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2022-06-02 20:32:49.7127 15289/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2022-06-02 20:32:49.7129 15289/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2022-06-02 20:32:49.7892 15289/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 15289
[ N 2022-06-02 20:32:50.0844 13663/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
Some steps:
Create a firewall and open it only to your public IP, you can use ufw
After you confirm you are in a DDoS attack, install this module for apache mod_evasive
Some other recommendations are to install a cache in front, like Cloudfare or Cloudfront.
Shutdown is most likely due to logs filling up - this is an ancient article but I have a hunch based on your description of the problem: https://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time
Handling DDoS is difficult and resource-intensive.
Maybe not the answer you want to hear, but I strongly recommend using a service like CloudFlare to prevent DDoS - much easier than trying to deal with it on your own.
RabbitMQ 3.10.1
rabbitmq-diagnostics status
...
Config files
* /etc/rabbitmq/rabbitmq.config
...
rabbitmq.config:
[
{rabbit,
[
{heartbeat, 90}
]
}
].
RabbitMQ Management show 5s heartbeat
And log:
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> closing AMQP connection <0.5979.0> (xxx.xxx.xxx.xxx:3555 -> xxx.xxx.xxx.xxx:5672):
2022-05-13 19:56:43.235925+03:00 [error] <0.5979.0> missed heartbeats from client, timeout: 5s
How to fix this?
Set the heartbeat to 90s in the client. Most clients are able to set the heartbeat (from the client). RabbitMQ will respect the heartbeat suggested by the client. More about that here: https://www.rabbitmq.com/heartbeats.html#heartbeats-timeout
I currently try to setup redmine in Archlinux (VM) using the tutorial:
https://wiki.archlinux.org/index.php/Redmine
Redmine can be started using "ruby bin/rails server webrick -e production" (install test).
i have already run "passenger-install-apache2-module" and added the lines to "/etc/httpd/conf/httpd.conf".
This is where the Arch tutorial ends.
I can also start it calling "passenger start" in the redmine directory, but i cant get appache loading it automatically.
For starting i currently have added:
<VirtualHost *:3000>
DocumentRoot /ust/share/webapps/redmine/public
<Directory /ust/share/webapps/redmine/public>
AllowOverride all
Options -MultiViews
</Directory>
</VirtualHost>
(This last part was a try from http://linoxide.com/linux-how-to/setup-redmine-fedora-22/)
PS: i also tried to call the "ruby bin/rails server webrick -e production" line using systemctl (as describet at the end of arch) but this one only gets errors i cant understand.
PPS: Ther also is a Nginx (port 80) on the system, Apache is on 8080 (just in case that might interfere.)
EDIT: Trying to get Logs (i hope to grb the correct files):
/var/log/httpd/error_log:
[ 2017-02-06 23:16:42.5833 854/7fbcbe96dd00 age/Wat/WatchdogMain.cpp:1281 ]: Starting Passenger watchdog...
[ 2017-02-06 23:16:42.6193 1081/7fd495dacd00 age/Cor/CoreMain.cpp:1070 ]: Starting Passenger core...
[ 2017-02-06 23:16:42.6197 1081/7fd495dacd00 age/Cor/CoreMain.cpp:245 ]: Passenger core running in multi-application mode.
[ 2017-02-06 23:16:42.6245 1081/7fd495dacd00 age/Cor/CoreMain.cpp:820 ]: Passenger core online, PID 1081
[ 2017-02-06 23:16:42.6279 1088/7f56ce6b3d00 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2017-02-06 23:16:42.7834 1088/7f56ce6b3d00 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 1088
[ 2017-02-06 23:16:42.9048 1081/7fd483fff700 age/Cor/CoreMain.cpp:585 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdo$
[ 2017-02-06 23:16:42.9049 1081/7fd495dacd00 age/Cor/CoreMain.cpp:989 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2017-02-06 23:16:42.9049 1088/7f56c8a30700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force s$
[ 2017-02-06 23:16:42.9049 1088/7f56ce6b3d00 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnecte$
[ 2017-02-06 23:16:42.9049 1088/7f56bbfff700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2017-02-06 23:16:42.9049 1088/7f56bbfff700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2017-02-06 23:16:42.9050 1081/7fd482f7c700 Ser/Server.h:817 ]: [ServerThr.3] Freed 128 spare client objects
[ 2017-02-06 23:16:42.9050 1081/7fd482f7c700 Ser/Server.h:464 ]: [ServerThr.3] Shutdown finished
[ 2017-02-06 23:16:42.9051 1081/7fd483fff700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects
[ 2017-02-06 23:16:42.9051 1081/7fd483fff700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
[ 2017-02-06 23:16:42.9052 1081/7fd4837fe700 Ser/Server.h:817 ]: [ServerThr.2] Freed 128 spare client objects
[ 2017-02-06 23:16:42.9052 1081/7fd4837fe700 Ser/Server.h:464 ]: [ServerThr.2] Shutdown finished
[ 2017-02-06 23:16:42.9054 1088/7f56c8a30700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2017-02-06 23:16:42.9055 1088/7f56ce6b3d00 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
[ 2017-02-06 23:16:42.9055 1081/7fd4826fa700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2017-02-06 23:16:42.9055 1081/7fd4826fa700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
[ 2017-02-06 23:16:43.0683 1127/7f64d2621d00 age/Wat/WatchdogMain.cpp:1281 ]: Starting Passenger watchdog...
[ 2017-02-06 23:16:43.0715 1130/7fa99d107d00 age/Cor/CoreMain.cpp:1070 ]: Starting Passenger core...
[ 2017-02-06 23:16:43.0716 1130/7fa99d107d00 age/Cor/CoreMain.cpp:245 ]: Passenger core running in multi-application mode.
[ 2017-02-06 23:16:43.0737 1130/7fa99d107d00 age/Cor/CoreMain.cpp:820 ]: Passenger core online, PID 1130
[ 2017-02-06 23:16:43.0772 1136/7f1a31534d00 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2017-02-06 23:16:43.0776 1136/7f1a31534d00 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 1136
[Mon Feb 06 23:16:43.078579 2017] [mpm_event:notice] [pid 838:tid 139947678662592] AH00489: Apache/2.4.25 (Unix) Phusion_Passenger/5.1.2 configured -- resuming n$
[Mon Feb 06 23:16:43.078699 2017] [core:notice] [pid 838:tid 139947678662592] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'
[ 2017-02-06 23:16:43.1472 1081/7fd495dacd00 age/Cor/CoreMain.cpp:1055 ]: Passenger core shutdown finished
[ 2017-02-06 23:16:45.6587 1130/7fa997cae700 age/Cor/SecurityUpdateChecker.h:464 ]: Security update check: no update found (next check in 24 hours)
/var/log/httpd/access_log
127.0.0.1 - - [06/Feb/2017:23:16:58 +0100] "GET / HTTP/1.1" 200 481
127.0.0.1 - - [06/Feb/2017:23:17:01 +0100] "GET /favicon.ico HTTP/1.1" 404 1126
127.0.0.1 - - [06/Feb/2017:23:18:35 +0100] "GET / HTTP/1.1" 200 481
127.0.0.1 - - [06/Feb/2017:23:18:35 +0100] "GET / HTTP/1.1" 200 481
127.0.0.1 - - [06/Feb/2017:23:18:35 +0100] "GET /icons/blank.gif HTTP/1.1" 304 -
127.0.0.1 - - [06/Feb/2017:23:18:35 +0100] "GET /favicon.ico HTTP/1.1" 404 1126
/usr/share/webapps/redmine/log dosent contain a file for 2017-02-06
(neither rubys log nor passenger?)
for "journalctl -S today -u httpd" i only get:
-- Logs begin at Sat 2016-09-17 11:12:13 CEST, end at Mon 2017-02-06 23:36:36 CET. --
Feb 06 23:16:42 arch-devel systemd[1]: Started Apache Web Server.
Edit 2:
Moved block to /etc/httpd/conf/extra/httpd-vhosts.conf, and enable it in /etc/httpd/conf/httpd.conf
(Effectively the block was moved to before the "passenger" block, no other configuration was given in httpd.conf)
After the next restart ther were acces problems (Apache cant acces /usr), which were solved by changing httpd-vhosts.conf:
<VirtualHost *:3000>
DocumentRoot /usr/share/webapps/redmine/public
<Directory /usr/share/webapps/redmine/public>
Options -MultiViews
AllowOverride all
Require all granted
Allow from all
</Directory>
ErrorLog "/var/log/httpd/redmine_log"
</VirtualHost>
When I run Apache twill HelloWorld example as stated in http://twill.incubator.apache.org/GettingStarted.html I get this log:
10:44:47.888 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable
application_1443786884805_0185: ACCEPTED
10:44:48.383 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn -
Got ping response for sessionid: 0x15028da0ff0009d after 0ms
10:44:48.889 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #45
10:44:48.894 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root]
DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #45
10:44:48.894 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms
10:44:48.895 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable application_1443786884805_0185:
ACCEPTED
10:44:49.711 [Kafka-Consumer-log-0] DEBUG o.a.t.i.k.client.SimpleKafkaConsumer -
No leader for topic partition TopicPartition{topic=log, partition=0}.
10:44:49.895 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #46
10:44:49.902 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #46
10:44:49.902 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 7ms
10:44:49.902 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application status for HelloWorldRunnable application_1443786884805_0185:
FAILED
10:44:50.902 [ STARTING] INFO o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 is in state
FAILED
10:44:50.903 [ STARTING] INFO o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 is not in running state. Shutting down controller.
10:44:50.907 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #47
10:44:50.908 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn -
Reading reply sessionid:0x15028da0ff0009d, packet::
clientPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0
serverPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0
finished:false header:: 10,3 replyHeader:: 10,1797,-101 request:: '/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0,T response::
10:44:50.913 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client -
IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #47
10:44:50.913 [ STOPPING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms
10:44:50.916 [ STOPPING] DEBUG o.a.twill.yarn.YarnTwillController -
Yarn application HelloWorldRunnable application_1443786884805_0185 completed with status
FAILED
The Application gets ACCEPTed but then transitions to the "FAILED" state.
The YARN Web UI shows this as the error (very unspecific):
Application application_1443786884805_0185 failed 2 times due to AM Container for appattempt_1443786884805_0185_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://gin1.dev:8088/proxy/application_1443786884805_0185/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1443786884805_0185_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
And the node log shows:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2693)
at java.lang.Class.privateGetMethodRecursive(Class.java:3040)
at java.lang.Class.getMethod0(Class.java:3010)
at java.lang.Class.getMethod(Class.java:1776)
at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:85)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
What can be wrong? I supposed that hadoop classes should be on classpath in yarn application. How to fix that?
I assume you are launching the application with the result of hadoop classpath in the classpath (as shown in the example). You need to make sure the result of running hadoop classpath on launcher box points to local paths that contain hadoop jars. The other thing you might want to check is the stdout file in the container log directory. It prints out classpath that it uses to launch the application and see if you find hadoop jars there.
I have an Eucalyptus-based cloud using Xen with 3 nodes. When I run a VM more than twice the number of physical machines in the xen cluster, the extra VMs never boot up, they always change state from pending to terminate, and the log file shows the node controller shutoff the extra VM after it check the instances number. When I manually shutdown some VM, then I can easily start up some VM exactly as the number I've shutdown.
The log shows this error:
ERROR: [Tue Jan 17 12:51:05 2012][004231][EUCAERROR ] libvirt: POST
operation failed: xend_post: error from xen daemon: (xend.err 'Device
2049 (vbd) could not be connected. /etc/xen/scripts/block failed;
error detected.') (code=11) [Tue Jan 17 12:51:05
2012][004231][EUCADEBUG ] doDescribeResource() invoked [Tue Jan 17
12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances() invoked [Tue
Jan 17 12:51:06 2012][004231][EUCADEBUG ] doDescribeInstances():
instanceId=i-3BB00757 publicIp=192.168.4.165 privateIp=192.168.4.165
mac=d0:0d:3B:B0:07:57 vlan=-1 networkIndex=-1 [Tue Jan 17 12:51:06
2012][004231][EUCADEBUG ] doDescribeInstances(): instanceId=i-473807EE
publicIp=192.168.4.112 privateIp=192.168.4.112 mac=d0:0d:47:38:07:EE
vlan=-1 networkIndex=-1 [Tue Jan 17 12:51:06 2012][004231][EUCADEBUG ]
doDescribeInstances(): instanceId=i-348A06AE publicIp=0.0.0.0
privateIp=0.0.0.0 mac=d0:0d:34:8A:06:AE vlan=-1 networkIndex=-1 [Tue
Jan 17 12:51:06 2012][004231][EUCAERROR ] libvirt: POST operation
failed: xend_post: error from xen daemon: (xend.err 'Device 2049 (vbd)
could not be connected. /etc/xen/scripts/block failed; error
detected.') (code=11) [Tue Jan 17 12:51:08 2012][004231][EUCAERROR ]
libvirt: POST operation failed: xend_post: error from xen daemon:
(xend.err 'Device 2049 (vbd) could not be connected.
/etc/xen/scripts/block failed; error detected.') (code=11) [Tue Jan 17
12:51:09 2012][004231][EUCAERROR ] libvirt: POST operation failed:
xend_post: error from xen daemon: (xend.err 'Device 2051 (vbd) could
not be connected. /etc/xen/scripts/block failed; error detected.')
(code=11) [Tue Jan 17 12:51:10 2012][004231][EUCAERROR ] libvirt: POST
operation failed: xend_post: error from xen daemon: (xend.err 'Device
2051 (vbd) could not be connected. /etc/xen/scripts/block failed;
error detected.') (code=11) [Tue Jan 17 12:51:10
2012][004231][EUCAFATAL ] hypervisor failed to start domain [Tue Jan
17 12:51:10 2012][004231][EUCADEBUG ] state change for instance
i-348A06AE: Staging -> Shutoff (Extant) [Tue Jan 17 12:51:10
2012][004231][EUCAERROR ] libvirt: Domain not found:
xenUnifiedDomainLookupByName (code=42) [Tue Jan 17 12:51:10
2012][004231][EUCAINFO ] cleaning up state for instance i-348A06AE
Is this a limitation for CentOS 5.* and RedHat 5.? Is it fixed on CentOS 6.?
http://docs.vmd.citrix.com/XenServer/5.0.0/1.0/en_gb/guest.html#rhel5_limitations
"Only 3 virtual network interfaces are supported in versions below 5.2. For 5.2 and above, 7 virtual network interfaces are supported."
I opened this topic in both CentOS and Eucalyptus forums without luck:
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=35491&forum=38
http://open.eucalyptus.com/forum/unable-run-more-4-instances-even-extra-resources
Thanks, Leandro
The error you are seeing seems related to disk not to network (Device 2049 (vbd) could not be connected. /etc/xen/scripts/block failed; error detected). I don't see that script on a CentOS 5.7 install I have here, so you may want to dig into that script to see if there is some obvious error.
The solution was to increase number of loop devices in CentOS 5, as detailed in the following link:
http://open.eucalyptus.com/participate/wiki/increase-number-loop-devices-centos-5
As the link is no longer valid, you have to go to archive.org or see this similar answer:
how to increase the loop devices number