celerybeat shutdown - initscript order? - rabbitmq

Question

I'm trying to setup rabbitmq/celery/django-celery/django so that it is "rebootproof", i.e. it all just comes back up by itself. Everything seems to work fine except this:
When I reboot, all services get started, but it seems celeryd is started before rabbitmq, and celerybeat gets subsequently terminated because it can't connect (?):
[2011-06-14 00:48:35,128: WARNING/MainProcess] celery#inquire has started.
[2011-06-14 00:48:35,130: INFO/Beat] child process calling self.run()
[2011-06-14 00:48:35,131: INFO/Beat] Celerybeat: Starting...
[2011-06-14 00:48:35,134: ERROR/MainProcess] Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 2 seconds...
[2011-06-14 00:48:35,688: INFO/Beat] process shutting down
[2011-06-14 00:48:35,689: WARNING/Beat] Process Beat:
[2011-06-14 00:48:35,689: WARNING/Beat] Traceback (most recent call last):
...
[2011-06-14 00:48:35,756: WARNING/Beat] File "/home/inquire/inquire.env/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 220, in create_transport
[2011-06-14 00:48:35,760: WARNING/Beat] return TCPTransport(host, connect_timeout)
[2011-06-14 00:48:35,761: WARNING/Beat] File "/home/inquire/inquire.env/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 58, in __init__
[2011-06-14 00:48:35,761: WARNING/Beat] self.sock.connect((host, port))
[2011-06-14 00:48:35,761: WARNING/Beat] File "<string>", line 1, in connect
[2011-06-14 00:48:35,761: WARNING/Beat] error: [Errno 111] Connection refused
[2011-06-14 00:48:35,761: INFO/Beat] process exiting with exitcode 1
[2011-06-14 00:48:37,137: ERROR/MainProcess] Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 4 seconds...
On Ubuntu, I installed rabbitmq-server with apt, django-celery with pip into my virtualenv, then I symlinked the "celeryd" initscript I got from https://github.com/ask/celery/tree/master/contrib/debian/init.d in /etc/init.d, configured it in /etc/default/celeryd to use the django celeryd from my virtualenv, and made it "rebootproof" via (maybe "defaults" is the problem?)
update-rc.d celeryd defaults
Rather than running celeryd and celerybeat with separate initscripts, I just configured celeryd to include Beat (maybe that's the problem?):
CELERYD_OPTS="-v 2 -B -s celery -E"
Any pointers how to solve this issue?
If I
sudo /etc/init.d/celeryd restart
there are no complaints:
[2011-06-14 00:54:29,157: WARNING/MainProcess] celery#inquire has started.
[2011-06-14 00:54:29,161: INFO/Beat] child process calling self.run()
[2011-06-14 00:54:29,162: INFO/Beat] Celerybeat: Starting...
but I need to eliminate the need for any manual steps.

Answer 1

celerybeat's dependency on the broker service was indeed the issue.
Rather than installing the initscript with
update-rc.d celeryd defaults
with the rabbitmq-server script being installed as sequence number 20 for start and kill, celerybeat's dependency must be resolved by explicitly starting it after (and killing it before) rabbitmq-server by using
update-rc.d celeryd defaults 21 19
NB: I've actually opted for the separate celerybeat service instead of the -B invocation, and only did 21 19 for that script, i.e. the one with the problem.

Answer 2

I think that the problem is not in celery it self but in your script, probably when celeryd starts the broker is not listening yet.
I'm using almost your same command and I don't have any issue, launch the celeryd script with -B option is not wrong.
I think that on your reboot script you have to wait for rabbitmq complete restart before launch celeryd, maybe with test of connection too.