nova-compute service state is down - rabbitmq

Question

I have an all-in-one-setup with my controller and compute services running on the same node.all my nova and other dependent services are up and running. However, when i try to launch an instance the state of the nova-compute process becomes down. Because of this the instance is stuck in spawning state.
> [root#localhost nova(keystone_admin)]# nova service-list
> +----+------------------+-----------------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status |
> State | Updated_at | Disabled Reason |
> +----+------------------+-----------------------+----------+---------+-------+----------------------------+-----------------+ | 6 | nova-cert | localhost.localdomain | internal | enabled |
> up | 2016-11-04T07:24:32.000000 | - | | 7 |
> nova-consoleauth | localhost.localdomain | internal | enabled | up
> | 2016-11-04T07:24:32.000000 | - | | 8 | nova-scheduler
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:33.000000 | - | | 9 | nova-conductor
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:33.000000 | - | | 11 | nova-compute
> | localhost.localdomain | nova | enabled | **down** |
> 2016-11-04T06:43:03.000000 | - | | 12 | nova-console
> | localhost.localdomain | internal | enabled | up |
> 2016-11-04T07:24:32.000000 | - |
====
[root#localhost nova(keystone_admin)]# systemctl status
openstack-nova-compute.service -l ● openstack-nova-compute.service -
OpenStack Nova Compute Server Loaded: loaded
(/usr/lib/systemd/system/openstack-nova-compute.service; enabled;
vendor preset: disabled) Active: active (running) since Fri
2016-11-04 12:08:54 IST; 49min ago Main PID: 37586 (nova-compute)
CGroup: /system.slice/openstack-nova-compute.service
└─37586 /usr/bin/python2 /usr/bin/nova-compute
Nov 04 12:08:46 localhost.localdomain systemd[1]: Starting OpenStack
Nova Compute Server... Nov 04 12:08:53 localhost.localdomain
nova-compute[37586]: Option "verbose" from group "DEFAULT" is
deprecated for removal. Its value may be silently ignored in the
future. Nov 04 12:08:53 localhost.localdomain nova-compute[37586]:
Option "notification_driver" from group "DEFAULT" is deprecated. Use
option "driver" from group "oslo_messaging_notifications". Nov 04
12:08:54 localhost.localdomain systemd[1]: Started OpenStack Nova
Compute Server.
========
The status for the nova compute process is perfectly fine.
My rabbitmq service is also running
FYI,
[root#localhost nova(keystone_admin)]# systemctl status
rabbitmq-server ● rabbitmq-server.service - RabbitMQ broker Loaded:
loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled;
vendor preset: disabled) Drop-In:
/etc/systemd/system/rabbitmq-server.service.d
└─limits.conf
Active: active (running) since Thu 2016-11-03 12:32:08 IST; 24h ago
Main PID: 1835 (beam.smp) CGroup:
/system.slice/rabbitmq-server.service
├─1835 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home
/var/lib/rabbitmq --...
├─1964 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
├─5873 inet_gethost 4
└─5875 inet_gethost 4
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
{user,<<"guest">>, Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: [administrator], Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]:
rabbit_auth_backend_internal,...}, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: <<"/">>, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: [{<<...>>,...},{...}],
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
<0.14812.0>,<0.14816.0>]}}, Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: {restart_type,intrinsic}, Nov 04 12:13:12
localhost.localdomain rabbitmq-server[1835]: {shutdown,4294967295},
Nov 04 12:13:12 localhost.localdomain rabbitmq-server[1835]:
{child_type,worker}]}]}} Nov 04 12:13:12 localhost.localdomain
rabbitmq-server[1835]: function_clause
=======
[root#localhost nova(keystone_admin)]# netstat -anp | grep 5672 | grep
37586 tcp 0 0 10.1.10.22:55628 10.1.10.22:5672
ESTABLISHED 37586/python2 tcp 0 0 10.1.10.22:56204
10.1.10.22:5672 ESTABLISHED 37586/python2 tcp 0 0 10.1.10.22:56959 10.1.10.22:5672 ESTABLISHED 37586/python2
===== 37586 is the nova-compute process id.
I have checked the logs for nova-compute, nova-api and nova-conductor and there are no errors.
I have checked the nova scheduler logs and there are some errors stating refused to connect to rabbitmq and the database service.
**
2016-11-03 12:24:50.930 2092 ERROR nova.servicegroup.drivers.db
DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't
connect to MySQL server on '10.1.10 .22' ([Errno 111] ECONNREFUSED)")
2016-11-03 12:24:53.811 2092 ERROR oslo.messaging._drivers.impl_rabbit
[-] AMQP server on 10.1.10.22:5672 is unreachable: [Errno 111]
ECONNREFUSED. Trying again in
**
16 seconds.
=======
Can someone suggest what should i do to handle it.
As i am on the same node, why are these services not reachable?

Answer 1

If nova-compute is down, there are two possible reasons:
a. nova-compute is actually down
b. it cannot communicate with rabbit, or nova-conductor cannot communicate with rabbit.
As far as I can see in your logs, you have issue with rabbit: "10.1.10.22:5672 is unreachable". Check if you have rabbit listening on this IP/port. Check if you can connect to rabbit from compute host. I usually use nc 10.1.10.22 5672 to see if there are connection or not.
Check if nova settings for rabbit are correct. Example of correct settings:
[DEFAULT]
rpc_backend=rabbit
rabbit_host=rabbitmq-ip-here
rabbit_port=5672
rabbit_hosts=$rabbit_host:$rabbit_port
rabbit_use_ssl=false
rabbit_userid=guest
rabbit_password=guest
rabbit_login_method=AMQPLAIN
rabbit_virtual_host=/compute
Check logs in the /var/log/nova/*.log
Enable debug=true in the [DEFAULT] section of nova.conf