All the examples I have seen of executing/scheduling Celery tasks are like this:
add.delay()
I was wondering if I could do something like this with Celery:
celery_app.publish(topic='my-topic')
And in other codebase/service:
#task(topic='my-topic')
def mytask():
do_stuf()
This way I don't need to know which tasks have to do something when an event happens.
I probably have some missconceptions causing this question, but I couldn't find the answer myself.
No topics, just queues. And yes, you can send task to any queue. Subscribing to a queue is a worker-level, remote command, so that is possible too.
Also, you can't send arbitrary messages to queues, just Celery tasks. If you want to produce/consume arbitrary messages, use kombu.
Related
I'm looking for a solution to have scheduled messages with RabbitMQ, so not only delaying the messages as described in several sources but schedule it to have a message e.g. every day.
If not RabbitMQ, any other solutions out there you can think of and you'd suggest for a microservices environment using a message-bus?
So it's really about combining the concept of a task-scheduler and a message bus ...
Or is it better to use a job scheduler just to push messages to the message queue, e.g. using rundeck in combination with RabbitMQ?
Or is it better to use a job scheduler just to push messages to the message queue, e.g. using rundeck in combination with RabbitMQ?
yes.
RabbitMQ is not designed to handle scheduling, and attempting to use it for that will just be painful (at best).
It is best to use another scheduling system, like cron jobs or rundeck or any of the other numerous scheduling tools available. From that tool, you can execute code that will push messages across RabbitMQ, triggering work in other parts of your system.
You may try ActiveMQ, It supports crontab schedule, and it provides a web console to setup the schedule too. If you want to schedule from code, it may looks like:
MessageProducer producer = session.createProducer(destination);
TextMessage message = session.createTextMessage("test msg");
message.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_CRON, "0 * * * *");
producer.send(message);
I'm trying to achieve load balancing between different types of messages. I would not know in advance what the messages coming in might be until they hit the queue. I know I can try resequencing the messages, but I was thinking that maybe if there was a way to have the various consumers round robin between either queues or between topics, this would solve my problem.
The main problem i'm trying to solve is that I have many services sending messages to one queue with many consumers feeding off one queue. I do not want one type of service monopolizing the entire worker cluster. Again I don't know in advance what the messages that are going to hit the queue are going to be.
To try to clearly repeat my question:
Is there a way to tell the consumers to round robin between either existing queues or topics?
Thank you in advance.
I found the answer to my question on another post just had to know to look there. I resolved my problem by not creating AMQ consumer but a JMS listener with a composite destination as specified in this post: jms-listener-dynamically-choose-destinations. It turns out the JMS listener automatically round robins though all the queues you assign to it.
Consumers on a Queue will already do round robin processing of the messages on the Queue. The one thing to keep in mind is consumer prefetch which can allow one consumer to grab many messages before others arrive on the Queue so you may need to adjust prefetch depending on your scenario.
Read up on the differences between Queue and Topic here.
I am starting with ActiveMQ and I have a usecase. i have n producers sending messages into a Queue Q1. I want to stop the delivery of messages (i.e. i do not want consumers to consume those messages). I want to store the messages for sometime without those being consumed.
I was looking at ways this can be achieved. These two things came into my mind based on what i browsed through.
Using Mirrored queues, so that I can wiretap the messages and save into a virtual queue.
Possibly stop consumers from doing a PULL on the queue.
Another dirty way of doing this is by making consumers not send ack messages once its consumed a message from the queue.
We are currently not happy with either of these.
Any other way you can suggest.
Thanks in advance.
If you always want message delivery to be delayed you can use the scheduler feature of ActiveMQ to delay delivery until a set time or a fixed delay etc.
Other strategies might also work but it really up to you to design something that fits your use case. You can try to use Apache Camel to define a route that implements the logic of your use case to either dispatch a message to a Queue or send it to the scheduler for delayed processing. It all really depends on your use case and requirements.
Is there a simple way to log (perhaps partially log) all messages that pass through all queues on ActiveMQ? The logging-interceptor does NOT do it.
Yes, I understand its somewhat ludicrous, but its still a requirement.
How about from a Camel context?
I was looking around for an answer like this, and there's actually a relatively simple way to do this now - Mirrored Queues.
http://activemq.apache.org/mirrored-queues.html
Achieved by embedding a camel context with a message handler. Then we wiretap any queue that we want to log and let the handler consume from the queue. The handler sends messages to any database you like.
Logged.
Pretty new to RabbitMQ and we're still in the investigation stage to see if it's a good fit for our use cases--
We've readily come to the conclusion that our desired topology would have us deploying a few topic based exchanges, and then filtering from there to specific queues. For example, let's say we have a user and an upload exchange, where the user queue might receive messages where the topic is "new-registration" or "friend-request" and the upload exchange might receive messages like "video-upload" or "picture-upload".
Creating the queues, getting them routed to the appropriate queue, and then building listeners to handle the messages for the various queues has been quite straight forward.
What's unclear to me however is if it's possible to do a fanout on a topic exchange?
I.e. I have named queues that are bound to my topic exchange, but I'd like to be able to just throw tons of instances of my listeners at those queues to prevent single points of failure. But to the best of my knowledge, RabbitMQ treats these listeners in a straight forward round robin fashion--e.g. every Nth message always go to the same Nth listener rather than dispatching messages to the first available consumer. This is generally acceptable to us but given the load we anticipate, we'd like to avoid the possibility of hot spots developing amongst our consumer farm.
So, is there some way, either in the queue or exchange configuration or in the consumer code, where we can point our listeners to a topic queue but have the listeners treated in a fanout fashion?
Yes, by having the listeners bind using different queue names, they will be treated in a fanout fashion.
Fanout is 1:N though, i.e. each task can be delivered to multiple listeners like pub-sub. Note that this isn't restricted to a fanout exchange, but also applies if you bind multiple queues to a direct or topic exchange with the same binding key. (Installing the management plugin and looking at the exchanges there may be useful to visualize the bindings in effect.)
Your current setup is a task queue. Each task/message is delivered to exactly one worker/listener. Throw more listeners at the same queue name, and they will process the tasks round-robin as you say. With "fanout" (separate queues for a topic) you will process a task multiple times.
Depending on your platform there may be existing work queue solutions that meet your requirements, such as Resque or DelayedJob for Ruby, Celery for Python or perhaps Octobot or Akka for the JVM.
I don't know for a fact, but I strongly suspect that RabbitMQ will skip consumers with unacknowledged messages, so it should never bottleneck on a single stuck consumer. The comments on their FAQ seem to suggest that RabbitMQ will make an effort to keep things chugging along even in the presence of troublesome consumers.
This is a late answer, but in case others come across this question...
It sounds like what you want is fair dispatch rather than a fan out model (which would publish a given message to every queue).
Fair dispatch will give a message to the next available worker rather than using a simple round-robin approach. This should avoid the "hotspots" you are concerned about, without delivering the same message to multiple consumers.
If this is what you are looking for, then see the "Fair Dispatch" section on this page in the Rabbit docs. A prefetch count of 1 is the key here.