celery with redis - unix timeout - redis

Question

I have an app using celery for async-task and i use redis for its broker and result backend and i set redis to use unix socket.
here is my url for celery and broker
brok = 'redis+socket://:ABc#/tmp/redis.sock'
app = Celery('NTWBT', backend=brok, broker=brok)
app.conf.update(
BROKER_URL=brok,
BROKER_TRANSPORT_OPTIONS={
"visibility_timeout": 3600
},
CELERY_RESULT_BACKEND=brok,
CELERY_ACCEPT_CONTENT=['pickle', 'json', 'msgpack', 'yaml'],
)
but every time i add a job celery gives me this error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 283, in trace_task
uuid, retval, SUCCESS, request=task_request,
File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 257, in store_result
request=request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 491, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/redis.py", line 160, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/usr/local/lib/python2.7/dist-packages/celery/backends/redis.py", line 149, in ensure
**retry_policy
File "/usr/local/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 246, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/backends/redis.py", line 169, in _set
pipe.execute()
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2620, in execute
self.shard_hint)
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 897, in get_connection
connection = self.make_connection()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 906, in make_connection
return self.connection_class(**self.connection_kwargs)
TypeError: __init__() got an unexpected keyword argument 'socket_connect_timeout'
which option should i use for celery to donot set timeout for its redis connection?

Answer 1

it seems that this problem is related to a version of a redis-server installed on your system, the socket_connect_timeout was first introduced in redis 2.10.0.
so you need to update your version of redis.
if you are running on ubuntu server you can install the official apt repositories:
$ sudo apt-get install -y python-software-properties
$ sudo add-apt-repository -y ppa:rwky/redis
$ sudo apt-get update
$ sudo apt-get install -y redis-server
and update to the last version of celery.
This is the github issue in celery because not only you run into this problem : https://github.com/celery/celery/issues/2903
and if everything not work in any way for you, i suggest to use rabbitmq instead of Redis :
$ sudo apt-get install rabbitmq-server
$ sudo pip install librabbitmq
and in your app configure celery with this CELERY_BROKER_URL:
'amqp://guest:guest#localhost:5672//'
i hope this answer will fit all your needs.
Cheers

Answer 2

There are bugs in several libraries that causes this exception in Celery:
https://github.com/celery/celery/issues/2903
https://github.com/celery/kombu/pull/590
If you use Redis with UNIX socket as the broker, there's no easy fix yet. Unless you monkey-patch celery, kombu and/or redis-py libraries...
For now, I recommend that you use Redis with TCP connection, or switch to another broker, e.g. RabbitMQ.