I have a route in my API which kicks off a job to a message queue (I'm using the AMQP package). I would like to leave the connection open and respond to the web request when the job finishes.
With AMQP, I have no way of knowing when a job I started will finish. Instead, I can subscribe to a channel that will report when a job finishes.
Here's some pseudo-code:
main = do
doneChannel = openAMQPChannel "done"
-- assume some other program subscribes to jobs, performs them
-- and puts them onto "done" when finished.
jobsChannel = openAMQPChannel "jobs"
forkIO $ subscribeAMQP doneChannel onDone
startServer
onDone jobId info = do
-- How can I return this info in my request?
-- assume it has unique identifying information
???
startServer jobs = do
Warp.run 8000 $ do
post "/jobs" $ \payload -> do
jobId <- generateUniqueId
publishAMQP jobs jobId payload
-- TODO wait until onDone happens for my particular request
info <- waitForJobToComplete jobId
return info
(See here for the real AMQP interface. I'm using Servant for routing)
Is there some way to do this with Control.Concurrent? MVar and Chan don't seem to be addressable by an ID like this. How could I implement it in such a way that I could handle thousands of requests at once?
Any other ideas?
I want to consume multiple messages from specific queue or a specific exchange with a given key.
so the scenario is as follow:
Publisher publish message 1 over queue 1
Publisher publish message 2 over queue 1
Publisher publish message 3 over queue 1
Publisher publish message 4 over queue 2
Publisher publish message 5 over queue 2
..
Consumer consume messages from queue 1
get [message 1, message 2, message 3] all at once and handle them in one call back
listen_to(queue_name , num_of_msg_to_fetch or all, function(messages){
//do some stuff with the returned list
});
the messages are not coming at the same time, it is like events and i want to collect them in a queue, package them and send them to a third party.
I also read this post:
http://rabbitmq.1065348.n5.nabble.com/Consuming-multiple-messages-at-a-time-td27195.html
Thanks
Don't consume directly from the queue as queues follow round robin algorithm(an AMQP mandate)
Use shovel to transfer the queue contents to a fanout exchange and consume messages right from this exchange. You get all messages across all connected consumers. :)
If you want to consume multiple messages from specific queue, you can try as below.
channel.queueDeclare(QUEUE_NAME, false, false,false, null);
Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
logger.info("Recieved Message --> " + message);
}
};
You might need to conceptually separate domain-message from RMQ-message. As a producer you'd then bundle multiple domain messages into a single RMQ-message and .produce() it to RMQ. Remember this kind of design introduces timeouts and latencies due to the existence of a window (you might take some impression from Kafka that does bundling to optimize I/O at the cost of latency).
As a consumer then, you'd have a consumer, with typical .handleDelivery implementation that would transform the received body for the processing: byte[] -> Set[DomainMessage] -> your listener.
I have got 10 queues on activemq server.
I have producer which want to push messages on one of the queue (the producer will select the queue randomly run time to put message on queue), how can I pass destination name in createProducer method.
I understand that I need to pass an object of type Destination. the producer would know the queues name on the server. Is it possible to pass (or convert) a string to Destination object type and pass that to createproducer method.
Thanks
If I understand your problem correctly;
If you're running Java and have a valid session, you could use Session.createQueue();
// Create a Destination using the queue name
Destination destination = session.createQueue("queue name");
// Create a MessageProducer from the Session to the Queue
MessageProducer producer = session.createProducer(destination);
Here is a complete example of doing this at the Apache site.
In a Mule flow, I would like to add an Exception Handler that forwards messages to a "retry queue" when there is an exception. However, I don't want this retry logic to run automatically. Instead, I'd rather receive a notification so I can review the errors and then decide whether to retry all messages in the queue or not.
I don't want to receive a notification for every exception. I'd rather have a scheduled job that runs every 15 minutes and checks to see if there are messages in this retry queue and then only send the notification if there are.
Is there any way to determine how many messages are currently in a persistent VM queue?
Assuming you use the default VM queue persistence mechanism and that the VM connector is named vmConnector, you can do this:
final String queueName = "retryQueue";
int messageCount = 0;
final VMConnector vmConnector = (VMConnector) muleContext.getRegistry()
.lookupConnector("vmConnector");
for (final Serializable key : vmConnector.getQueueProfile().getObjectStore().allKeys())
{
final QueueKey queueKey = (QueueKey) key;
if (queueName.equals(queueKey.queueName))
{
messageCount++;
}
}
System.out.printf("Queue %s has %d pending messages%n", queueName, messageCount);
I'm trying to build RPC service at PHP using RabbitMQ similar to this example: http://www.rabbitmq.com/tutorials/tutorial-six-java.html
I'm using this PECL extension: http://pecl.php.net/package/amqp (version 1.0.3)
The problem is that my Callback Queue (declared at Client script) is locked for a Server when I add flag AMQP_EXCLUSIVE to it.
Here is my Server
// connect to server
$cnn = new AMQPConnection('...');
$cnn->connect();
$channel = new AMQPChannel($cnn);
// create exchange
$exchangeName = 'k-exchange';
$exchange = new AMQPExchange($channel);
$exchange->setName($exchangeName);
$exchange->setType(AMQP_EX_TYPE_DIRECT);
$exchange->declare();
// declare queue to consume messages from
$queue = new \AMQPQueue($channel);
$queue->setName('tempQueue');
$queue->declare();
// start consuming messages
$queue->consume(function($envelope, $queue)
use ($channel, $exchange) {
// create callback queue
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setName($envelope->getReplyTo());
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
/* WARNING: Following code line causes error. See rabbit logs below:
* connection <0.1224.10>, channel 1 - error:
* {amqp_error,resource_locked,
* "cannot obtain exclusive access to locked queue 'amq.gen-Q6J...' in vhost '/'",
* 'queue.bind'}
*/
$callbackQueue->bind($exchange->getName(), 'rpc_reply');
// trying to publish response back to client's callback queue
$exchange->publish(
json_encode(array('processed by remote service!')),
'rpc_reply',
AMQP_MANDATORY & AMQP_IMMEDIATE
);
$queue->ack($envelope->getDeliveryTag());
});
And here is my Client.php
// connect to server
$cnn = new AMQPConnection('...');
$cnn->connect();
$channel = new AMQPChannel($cnn);
// create exchange
$exchangeName = 'k-exchange';
$exchange = new AMQPExchange($channel);
$exchange->setName($exchangeName);
$exchange->setType(AMQP_EX_TYPE_DIRECT);
$exchange->declare();
// create a queue which we send messages to server via
$queue = new \AMQPQueue($channel);
$queue->setName('tempQueue');
$queue->declare();
// binding exchange to queue
$queue->bind($exchangeName, 'temp_action');
// create correlation_id
$correlationId = sha1(time() . rand(0, 1000000));
// create anonymous callback queue to get server response response via
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
$callbackQueue->declare();
// publishing message to exchange (passing it to server)
$exchange->publish(
json_encode(array('process me!')),
'temp_action',
AMQP_MANDATORY,
array(
'reply_to' => $callbackQueue->getName(), // pass callback queue name
'correlation_id' => $correlationId
)
);
// going to wait for remote service complete tasks. tick once a second
$attempts = 0;
while ($attempts < 5)
{
echo 'Attempt ' . $attempts . PHP_EOL;
$envelope = $callbackQueue->get();
if ($envelope) {
echo 'Got response! ';
print_r($envelope->getBody());
echo PHP_EOL;
exit;
}
sleep(1);
$attempts++;
}
So in the end I just see error in RabbitMQ's logs:
connection <0.1224.10>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 'amq.gen-Q6J...' in vhost '/'",
'queue.bind'}
Question:
What is the proper way to create a callbackQueue object in a Server.php?
It appears that my Server.php has a different from Client.php connection to a RabbitMQ server. What should I do here?
How should I "share" the same (to Client.php's) connection at Server.php side.
UPDATE
Here are some more RabbitMQ Logs
My Server.php connection (Id is: <0.22322.27>)
=INFO REPORT==== 20-Jun-2012::13:30:22 ===
accepting AMQP connection <0.22322.27> (127.0.0.1:58457 -> 127.0.0.1:5672)
My Client.php connection (Id is: <0.22465.27>)
=INFO REPORT==== 20-Jun-2012::13:30:38 ===
accepting AMQP connection <0.22465.27> (127.0.0.1:58458 -> 127.0.0.1:5672)
Now I see Server.php causes error:
=ERROR REPORT==== 20-Jun-2012::13:30:38 ===
connection <0.22322.27>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 'amq.gen-g6Q...' in vhost '/'",
'queue.bind'}
My Assumption
I suspect since Client.php and Server.php do not share connection with the same Id it's impossible for them both to use exclusive queue declared in Client.php
There are a few issues with your implementation:
Exchange Declaration
Manually setting the reply queue opposed to
using a temporary queue
Use of AMQP_EXCLUSIVE in both directions
Exchange Declaration
You don't need to declare an exchange (AMQPExchange) to publish messages. In this RPC example, you need to use it as a way of broadcasting a message (e.g. temporary queue or temporary exchange). All communication will occur directly on the QUEUE and theoretically bypasses the exchange.
$exchange = new AMQPExchange($channel);
$exchange->publish(...);
QUEUEs & Reply To:
When you use AMQPQueue::setName() along with AMQPQueue::declare(), you are binding to a queue with a user defined name. If you declare the queue without a name, this is known as a temporary queue. This is useful when you need to receive a broadcasted message from a specific routing key. For this reason, RabbitMQ / AMQP generates a random temporary name. Since the queue name is made for a given instance to consume information exclusively, for its own sake, it is disposed of when the connection is closed.
When an RPC client wants to publish a message (AMQPExchange::publish()), it must specify a reply-to as one of the publish parameters. In this way, the RPC server can fetch the randomly generated name when it receives a request. It uses the reply-to name as the name of the QUEUE on which server will reply to the given client. Along with the temporary queue name, the instance must send a correlationId to ensure that the reply message it receives is unique to the request instance.
Client
$exchange = new AMQPExchange($channel);
$rpcServerQueueName = 'rpc_queue';
$client_queue = new AMQPQueue($this->channel);
$client_queue->setFlags(AMQP_EXCLUSIVE);
$client_queue->declareQueue();
$callbackQueueName = $client_queue->getName(); //e.g. amq.gen-JzTY20BRgKO-HjmUJj0wLg
//Set Publish Attributes
$corrId = uniqid();
$attributes = array(
'correlation_id' => $corrId,
'reply_to' => $this->callbackQueueName
);
$exchange->publish(
json_encode(['request message']),
$rpcServerQueueName,
AMQP_NOPARAM,
$attributes
);
//listen for response
$callback = function(AMQPEnvelope $message, AMQPQueue $q) {
if($message->getCorrelationId() == $this->corrId) {
$this->response = $message->getBody();
$q->nack($message->getDeliveryTag());
return false; //return false to signal to consume that you're done. other wise it continues to block
}
};
$client_queue->consume($callback);
Server
$exchange = new AMQPExchange($channel);
$rpcServerQueueName = 'rpc_queue';
$srvr_queue = new AMQPQueue($channel);
$srvr_queue->setName($rpcServerQueueName); //intentionally declares the rpc_server queue name
$srvr_queue->declareQueue();
...
$srvr_queue->consume(function(AMQPEnvelope $message, AMQPQueue $q) use (&$exchange) {
//publish with the exchange instance to the reply to queue
$exchange->publish(
json_encode(['response message']), //reponse message
$message->getReplyTo(), //get the reply to queue from the message
AMQP_NOPARAM, //disable all other params
$message->getCorrelationId() //obtain and respond with correlation id
);
//acknowledge receipt of the message
$q->ack($message->getDeliveryTag());
});
AMQP_EXCLUSIVE
In this case, EXCLUSIVE is only used on the Rpc client's temporary queue for each instance so that it can publish a message. In other words, the client creates a disposable temporary queue for it self to receive an answer from the RPC server exclusively. This insures no other channel thread can post on that queue. It is locked for the client and its responder only. It's important to note that AQMP_EXCLUSIVE does not prevent the RPC server from responding on the client's reply-to queue. AMQP_EXCLUSIVE pertains to two separate threads (channels instances) trying to publish to the same queue resource. When this occurs, the queue is essentially locked for subsequent connections. The same behavior occurs with an exchange declaration.
#Denis: Your implementation in this case is correct up to a point
Bad - don't re-declare the Queue in the server. That's the client's job
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setName($envelope->getReplyTo());
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
...
$callbackQueue->bind($exchange->getName(), 'rpc_reply');
You're trying to bind to a QUEUE called tempQueue. But you've already created a queue called tempQueue in the client.php. Depending on which service is started first, the other will throw an error. So you can cut out all of that and just keep the last part:
// trying to publish response back to client's callback queue
$exchange->publish(
json_encode(array('processed by remote service!')),
'rpc_reply', //<--BAD Should be: $envelope->getReplyTo()
AMQP_MANDATORY & AMQP_IMMEDIATE
);
Then modify the above by replacing:
'rpc_reply'
with
$envelope->getReplyTo()
Don't Declare a Queue Name on the client side
// create a queue which we send messages to server via
$queue = new \AMQPQueue($channel);
//$queue->setName('tempQueue'); //remove this line
//add exclusivity
$queue->setFlags(AMQP_EXCLUSIVE);
$queue->declare();
//no need for binding... we're communicating on the queue directly
//there is no one listening to 'temp_action' so this implementation will send your message into limbo
//$queue->bind($exchangeName, 'temp_action'); //remove this line
My answer from this question replied on the RabbitMQ Official mailing list
While not using the same library here you have the official tutorials ported to PHP
https://github.com/rabbitmq/rabbitmq-tutorials/tree/master/php
The problem in your code is that you declare queues with different options.
So as one reply say, if you declare queue A as durable, then every other declaration of that queue must be durable. The same for the exclusive flag.
Also you don't need to redeclare a queue to publish messages to it. As an RPC server you assume that the address sent in the 'reply_to' property is already present. I think is the responsibility of the RpcClient to make sure the queue where it is waiting for replies exists already.
Addendum:
Exclusivity in queues means that the only the channel that declared the queue can access it.
On your server you should also declare your queue as exclusive. Remember, RabbitMQ queues should have the same flag. For example if you declare queue that is set to "durable" the other end should also declare the queue a "durable" So on your server put a flag $callbackQueue->setFlags(AMQP_EXCLUSIVE); somewhat like that from your client.