I'm using Celery along with Redis and I was wondering how big all the parameters could be until Celery or Redis refuses them.
For instance :
mode = 'delete'
file_content = None
with open('/tmp/output.txt', 'r') as f:
file_content = f.read()
celery_task.s(file_content, mode).apply_async()
It's not possible to pass the path (/tmp/output.txt) since the task will be executed on another server.
How big can "output.txt" be before having Celery or Redis refuse it?
(I tried to search on both Celery and Redis documentation, but Redis's doc depends on how Celery uses Redis, and I couldn't find this).
Related
I am trying to run Airflow Celery worker with redis backend but cant get the connection between the worker and redis to work.
Here is how the configuration looks like:
broker_url = redis://:<my-password>#localhost:6379/0
I am getting error:
ValueError: invalid literal for int() with base 10: 'my-password-string'
Anyone know any fix for this?
That's because there're special characters in your password, you can simply do this:
e.g. your password is my#pw#
do url-encoding, then you get my%23pw%23
append another % before % to allow airflow read the config successfully
then the final password is my%%23pw%%23, so e.g. the redis broker url in airflow.cfg should be like this:
broker_url = redis://:my%%23pw%%23#redis-host:6379/0
So i solved it now.
My password was generated from random characters piped through base64 and it was causing problems.
I changed my password to bit shorter and simplier one and airflow worker now runs normally.
Using pyiron, I build up my script and I would submit it in cluster for running , I was wondering How can I do that ?
Note: Vasp is already installed in my Cluster.
pyiron uses pysqa to submit jobs to a queuing system:
https://github.com/pyiron/pysqa
With sample queuing configurations available at:
https://github.com/pyiron/pysqa/tree/master/tests/config
So in your pyiron resources directory you create a folder named queues which contains the pysqa queuing system configuration.
Once this is done you can use:
job.server.list_queues()
to view the available queues and:
job.server.view_queues()
to get more information about the individual queue and finally submit the job using:
job.server.queue = 'queue_name'
where queue_name is the name of the queue you want to select and then specify the cores and run_time using:
job.server.cores = 8
job.server.run_time = 30000
Finally when you call job.run() it is automatically submitted to the queue.
every time I try to use delay() nothing happens.
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379')
#app.task
def run(msg):
print(msg)
word="xxx"
run.delay(word)
I checked redis with ping and it seems to be working.
Or maybe I should use something different to run background threads in Flask app?
You need to start worker in order to pick up the task to process.
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#id9
Currently i have plan a process of using a script to add key values into redis using rpush.
I am currently using bin/sh to launch redis.
But what is the syntax to rpush my keyvalue into redis server.
#!/bin/sh
redis-cli -h 172.16.114.54
rpush stlfile "fad.stl" // how to rpush in with the correct syntax?
What kind of language is more suitable for redis so maybe i could change now to ease my future processes
Seems like using python would be easier to code Redis
from this post i found here i can easily install python.
ImportError: No module named redis
i managed to create a python script with this
import redis
r = redis.StrictRedis(host='172.16.114.54', port=6379, db=0)
r.lpush ('stlfile', 'variable.stl')
r.lrange ('stlfile', 0, -1)
when checked with another client i managed to see variable.stl went in :D
My question builds off this one: Temporary queue made in Celery
My application needs to retrieve results, as it uploads them to an S3 file. however, the number of temporary queues being made is causing my broker to crash (machine doesn't have enough memory). I want to delete the temporary queue once the corresponding result as been retrieved. In my celery client script, I am iterating through a list of of results (where each result is from function.delay() ):
for result in result_list:
while True:
if result.ready():
#do something with result
#I WANT TO DELETE TEMPORARY QUEUE HERE
Is there any way I can achieve the above -- deleting the temporary queue once the result has been retrieved?
I would have used CELERY_TASK_RESULT_EXPIRES option in my celeryconfig , but I don't know when I can safely clean up the temporary queue, as the result may not have been retrieved. Is there anyway I can delete specific queues in this script (note that I have the queue Id from the result).
ADDITIONAL NOTE:
I am running all rabbitmq servers in a cluster with HA enabled.
The way I did this was to use the rabbitmqadmin from rabbitmq. I downloaded it via
wget localhost:15672/cli/rabbitmqadmin
after installing the management plugin
rabbitmq-plugins enable rabbitmq_management
Make sure your user has the administrator tag for rabbitmq, or you will not be able to perform commands. I then deleted the queue in my script using python subprocess import and rabbitmqadmin delete queue name='' . Keep in mind that the queue name is the same as the corresponding result id, except without the hyphens.
Also make sure you add the params -v myvhost -u myusername -p mypassword in rabbitmqadmin commands, default vhost is /.
I believe this will delete queues across all nodes in a cluster, though I am not completely sure of this.