I'm new to RabbitMQ and I really want to figure out the detailed implementation of per connection flow control of RabbitMQ.
I read the source code of RabbitMQ server but can't find how flow-control blocking rabbitmq-java-client from publishing messages.
In rabbit_reader.erl I find control_throttle method which most likely do the flow control work.
% rabbitmq-server v3.7.x rabbit_reader.erl
control_throttle(State = #v1{connection_state = CS,
throttle = #throttle{blocked_by = Reasons} = Throttle}) ->
Throttle1 = case credit_flow:blocked() of
true ->
Throttle#throttle{blocked_by = sets:add_element(flow, Reasons)};
false ->
Throttle#throttle{blocked_by = sets:del_element(flow, Reasons)}
end,
State1 = State#v1{throttle = Throttle1},
case CS of
running -> maybe_block(State1);
%% unblock or re-enable blocking
blocked -> maybe_block(maybe_unblock(State1));
_ -> State1
end.
But in maybe_block method, it will send a "connection.blocked" message to client, which I find in rabbit-java-client is high-water-mark blocking message, not flow-control blocking.
So how does flow control block the client from publishing messages?
Where are the codes for blocking?
Does it block only on server-side or both on server and client side?
Thanks
Related
I have implemented Celery with RabbitMQ as Broker. I rely on Celery v4.4.7 since I have read that v5.0+ doesn't support RabbitMQ anymore. RabbitMQ is a MUST in my case.
Everything has been containerized then deployed as pods within Kubernetes 1.19. I am able to execute long running tasks and everything apparently looks fine at first glance. However, I have few concerns which require your expertise.
I have declared inbound and outbound queues but Celery created his owns and I do not see any message within those queues (inbound or outbound) :
inbound_queue = "_IN"
outbound_queue = "_OUT"
app = Celery()
app.conf.update(
broker_url = 'pyamqp://%s//' % path,
broker_heartbeat = None,
broker_connection_timeout = int(timeout)
result_backend = 'rpc://',
result_persistent = True,
task_queues = (
Queue(algorithm_queue, Exchange(inbound_queue), routing_key='default', auto_delete=False),
Queue(result_queue, Exchange(outbound_queue), routing_key='default', auto_delete=False),
),
task_default_queue = inbound_queue,
task_default_exchange = inbound_exchange,
task_default_exchange_type = 'direct',
task_default_routing_key = 'default',
)
#app.task(bind=True,
name='osmq.tasks.add',
queue=inbound_queue,
reply_to = outbound_queue,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 5, 'countdown': 2})
def execute(self, data):
<method_implementation>
I have implemented callbacks to get results back via REST APIs. However, randomly, it can return or not some results when the status is successfull. This is probably related to message persistency. In details, when I implement flower API to get info, status is successfull and the result is partially displayed (shortened json messages) - when I call AsyncResult, for the same status, result is either None or the right one. I do not understand the mechanism between rabbitmq queues and kombu which seems to cache the resulting message. I must guarantee to retrieve results everytime the task has been successfully executed.
def callback(uuid):
task = app.AsyncResult(uuid)
Specifically, it was that Celery 5.0+ did not support amqp:// as a result back end anymore. However, as your example, rpc:// is supported.
The relevant snippet is here: https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html#rabbitmq
We tend to always ignore_results=True in our implementation, so I can't give any practical tips of how to use rpc://, other than to infer that any response is put on an application-specific queue, instead of being able to put on a specified queue (or even different broker / rabbitmq instance) via amqp://.
i'm trying to do a simple message queue with RabitMQ i push a message with create_message
and then trying to get the message by the routing key.
it works great when the routing key is the same. the problem is when the routing key is different i keep on getting the message with the wrong routing key:
for example
def callback(ch, method, properties, body):
print("%r:%r" % (method.routing_key, body))
def create_message(self):
connection = pika.BlockingConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='www')
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='www',
routing_key="11",
body='Hello World1111!')
connection.close()
self.get_analysis_task_celery()
def get_message(self):
connection = pika.BlockingConnection(pika.ConnectionParameters(
'localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='www')
timeout = 1
connection.add_timeout(timeout, on_timeout)
channel.queue_bind(exchange="www", queue="hello", routing_key="10")
channel.basic_consume(callback,
queue='hello',
no_ack=True,
consumer_tag= "11")
channel.start_consuming()
example for my output: '11':'Hello World1111!'
what am i doing wrong?
tnx for the help
this is a total guess, since i can't see your rabbitmq server..
if you open the RabbitMQ management website and look at your exchange, you will probably see that the exchange is bound to the queue for routing key 10 and 11, both of which are bound to the same queue.
since both go to the same queue, your message will always be delivered to that queue, the consumer will always pick up the message
again, i'm guessing since i can't see your server. but check the server to make sure you don't have leftover / extra bindings
I am using Celery with Rabbitmq broker on Server A. Some tasks require interaction with another server say, Server B and I am using Rabbitmq queues for this interaction.
Queue 1 - Server A (Producer), Server B (Consumer)
Queue 2 - Server B (Producer), Server A (Consumer)
My celery is unexpectedly hanging and I have found the reason to be incorrect implementation of Server A consumer code.
channel.start_consuming() keeps polling Rabbitmq as expected however putting this in a celery task creates multiple pollers which don't expire. I can add expiry but the time completion for the data being sent to Server B cannot be guaranteed. The code pasted below is one method I used to tackle the issue but I am not convinced this is best solution.
I wish to know what I am doing wrong and what is the right way to implement this because I have failed searching for articles on the web. Any tips, insights and even links to articles would be extremely helpful.
Finally, my code -
#celery.task
def task_a(data):
do_some_processing
# Create only 1 Rabbitmq consumer instance to avoid celery hangups
task_d.delay()
#celery.task
def task_b(data):
do_some_processing
if data is not None:
task_c.delay()
#celery.task
def task_c():
data = some_data
data = json.dumps(data)
conn_params = pika.ConnectionParameters(host=RABBITMQ_HOST)
connection = pika.BlockingConnection(conn_params)
channel = connection.channel()
channel.queue_declare(queue=QUEUE_1)
channel.basic_publish(exchange='',
routing_key=QUEUE_1,
body=data)
channel.close()
#celery.task
def task_d():
def queue_helper(ch, method, properties, body):
'''
Callback from queue.
'''
data = json.loads(body)
task_b.delay(data)
conn_params = pika.ConnectionParameters(host=RABBITMQ_HOST)
connection = pika.BlockingConnection(conn_params)
channel = connection.channel()
channel.queue_declare(queue=QUEUE_2)
channel.basic_consume(queue_helper,
queue=QUEUE_2,
no_ack=True)
channel.start_consuming()
channel.close()
Up until now, my RabbitMQ consumer clients have used a prefetch value of 1. I'm looking to increase the value in order to gain performance. If I set the value to 2, will the RabbitMQ server send each consumer 2 messages at once such that I will need to parse the two messages and store the second one in a List until the first is processed and acknowledged? Or will the API handle this behind the scenes?
I'm using the Java AMQP client library:
ConnectionFactory factory = new ConnectionFactory();
...
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.basicQos(2);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(CONSUME_QUEUE_NAME, false, consumer);
while (!Thread.currentThread().isInterrupted()) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String m = new String(delivery.getBody(), "UTF-8");
// Will m contain two messages? Will I have to each message and keep track of them within a List?
...
}
The api handles this behind the scenes, so there are no worries there for you.
Regarding which message gets where, RMQ will just deliver by using round robin, that is if you have the queue: 1 2 3 4 5 6 and consumer1 and consumer2.
consumer1 will have 1 3 5
consumer2 will have 2 4 6
Should the connection die to any of your consumers the prefetched messages will be redelivered to the active consumers using the same delivery method.
This should be interesting reading and a good starting point to figure more exactly what happens:
Tutorial no.2 which I'm sure you've read
Reliability
The api internally queue messages in a blocking queue.
Setting the prefetch count more than 1 is actually a good idea since your worker need not wait for each and every message to arrive. It can read up to N messages (where N is the prefetch count). It can start working on a message as soon as it has finished the previous one.
Also, you have the option to acknowledge multiple messages at once instead of acknowledging individually.
channel.basicAck(lastDeliveryTag, true);
boolean true indicates to acknowledge all the messages upto and including the supplied lastDeliveryTag
I'm trying to build RPC service at PHP using RabbitMQ similar to this example: http://www.rabbitmq.com/tutorials/tutorial-six-java.html
I'm using this PECL extension: http://pecl.php.net/package/amqp (version 1.0.3)
The problem is that my Callback Queue (declared at Client script) is locked for a Server when I add flag AMQP_EXCLUSIVE to it.
Here is my Server
// connect to server
$cnn = new AMQPConnection('...');
$cnn->connect();
$channel = new AMQPChannel($cnn);
// create exchange
$exchangeName = 'k-exchange';
$exchange = new AMQPExchange($channel);
$exchange->setName($exchangeName);
$exchange->setType(AMQP_EX_TYPE_DIRECT);
$exchange->declare();
// declare queue to consume messages from
$queue = new \AMQPQueue($channel);
$queue->setName('tempQueue');
$queue->declare();
// start consuming messages
$queue->consume(function($envelope, $queue)
use ($channel, $exchange) {
// create callback queue
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setName($envelope->getReplyTo());
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
/* WARNING: Following code line causes error. See rabbit logs below:
* connection <0.1224.10>, channel 1 - error:
* {amqp_error,resource_locked,
* "cannot obtain exclusive access to locked queue 'amq.gen-Q6J...' in vhost '/'",
* 'queue.bind'}
*/
$callbackQueue->bind($exchange->getName(), 'rpc_reply');
// trying to publish response back to client's callback queue
$exchange->publish(
json_encode(array('processed by remote service!')),
'rpc_reply',
AMQP_MANDATORY & AMQP_IMMEDIATE
);
$queue->ack($envelope->getDeliveryTag());
});
And here is my Client.php
// connect to server
$cnn = new AMQPConnection('...');
$cnn->connect();
$channel = new AMQPChannel($cnn);
// create exchange
$exchangeName = 'k-exchange';
$exchange = new AMQPExchange($channel);
$exchange->setName($exchangeName);
$exchange->setType(AMQP_EX_TYPE_DIRECT);
$exchange->declare();
// create a queue which we send messages to server via
$queue = new \AMQPQueue($channel);
$queue->setName('tempQueue');
$queue->declare();
// binding exchange to queue
$queue->bind($exchangeName, 'temp_action');
// create correlation_id
$correlationId = sha1(time() . rand(0, 1000000));
// create anonymous callback queue to get server response response via
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
$callbackQueue->declare();
// publishing message to exchange (passing it to server)
$exchange->publish(
json_encode(array('process me!')),
'temp_action',
AMQP_MANDATORY,
array(
'reply_to' => $callbackQueue->getName(), // pass callback queue name
'correlation_id' => $correlationId
)
);
// going to wait for remote service complete tasks. tick once a second
$attempts = 0;
while ($attempts < 5)
{
echo 'Attempt ' . $attempts . PHP_EOL;
$envelope = $callbackQueue->get();
if ($envelope) {
echo 'Got response! ';
print_r($envelope->getBody());
echo PHP_EOL;
exit;
}
sleep(1);
$attempts++;
}
So in the end I just see error in RabbitMQ's logs:
connection <0.1224.10>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 'amq.gen-Q6J...' in vhost '/'",
'queue.bind'}
Question:
What is the proper way to create a callbackQueue object in a Server.php?
It appears that my Server.php has a different from Client.php connection to a RabbitMQ server. What should I do here?
How should I "share" the same (to Client.php's) connection at Server.php side.
UPDATE
Here are some more RabbitMQ Logs
My Server.php connection (Id is: <0.22322.27>)
=INFO REPORT==== 20-Jun-2012::13:30:22 ===
accepting AMQP connection <0.22322.27> (127.0.0.1:58457 -> 127.0.0.1:5672)
My Client.php connection (Id is: <0.22465.27>)
=INFO REPORT==== 20-Jun-2012::13:30:38 ===
accepting AMQP connection <0.22465.27> (127.0.0.1:58458 -> 127.0.0.1:5672)
Now I see Server.php causes error:
=ERROR REPORT==== 20-Jun-2012::13:30:38 ===
connection <0.22322.27>, channel 1 - error:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue 'amq.gen-g6Q...' in vhost '/'",
'queue.bind'}
My Assumption
I suspect since Client.php and Server.php do not share connection with the same Id it's impossible for them both to use exclusive queue declared in Client.php
There are a few issues with your implementation:
Exchange Declaration
Manually setting the reply queue opposed to
using a temporary queue
Use of AMQP_EXCLUSIVE in both directions
Exchange Declaration
You don't need to declare an exchange (AMQPExchange) to publish messages. In this RPC example, you need to use it as a way of broadcasting a message (e.g. temporary queue or temporary exchange). All communication will occur directly on the QUEUE and theoretically bypasses the exchange.
$exchange = new AMQPExchange($channel);
$exchange->publish(...);
QUEUEs & Reply To:
When you use AMQPQueue::setName() along with AMQPQueue::declare(), you are binding to a queue with a user defined name. If you declare the queue without a name, this is known as a temporary queue. This is useful when you need to receive a broadcasted message from a specific routing key. For this reason, RabbitMQ / AMQP generates a random temporary name. Since the queue name is made for a given instance to consume information exclusively, for its own sake, it is disposed of when the connection is closed.
When an RPC client wants to publish a message (AMQPExchange::publish()), it must specify a reply-to as one of the publish parameters. In this way, the RPC server can fetch the randomly generated name when it receives a request. It uses the reply-to name as the name of the QUEUE on which server will reply to the given client. Along with the temporary queue name, the instance must send a correlationId to ensure that the reply message it receives is unique to the request instance.
Client
$exchange = new AMQPExchange($channel);
$rpcServerQueueName = 'rpc_queue';
$client_queue = new AMQPQueue($this->channel);
$client_queue->setFlags(AMQP_EXCLUSIVE);
$client_queue->declareQueue();
$callbackQueueName = $client_queue->getName(); //e.g. amq.gen-JzTY20BRgKO-HjmUJj0wLg
//Set Publish Attributes
$corrId = uniqid();
$attributes = array(
'correlation_id' => $corrId,
'reply_to' => $this->callbackQueueName
);
$exchange->publish(
json_encode(['request message']),
$rpcServerQueueName,
AMQP_NOPARAM,
$attributes
);
//listen for response
$callback = function(AMQPEnvelope $message, AMQPQueue $q) {
if($message->getCorrelationId() == $this->corrId) {
$this->response = $message->getBody();
$q->nack($message->getDeliveryTag());
return false; //return false to signal to consume that you're done. other wise it continues to block
}
};
$client_queue->consume($callback);
Server
$exchange = new AMQPExchange($channel);
$rpcServerQueueName = 'rpc_queue';
$srvr_queue = new AMQPQueue($channel);
$srvr_queue->setName($rpcServerQueueName); //intentionally declares the rpc_server queue name
$srvr_queue->declareQueue();
...
$srvr_queue->consume(function(AMQPEnvelope $message, AMQPQueue $q) use (&$exchange) {
//publish with the exchange instance to the reply to queue
$exchange->publish(
json_encode(['response message']), //reponse message
$message->getReplyTo(), //get the reply to queue from the message
AMQP_NOPARAM, //disable all other params
$message->getCorrelationId() //obtain and respond with correlation id
);
//acknowledge receipt of the message
$q->ack($message->getDeliveryTag());
});
AMQP_EXCLUSIVE
In this case, EXCLUSIVE is only used on the Rpc client's temporary queue for each instance so that it can publish a message. In other words, the client creates a disposable temporary queue for it self to receive an answer from the RPC server exclusively. This insures no other channel thread can post on that queue. It is locked for the client and its responder only. It's important to note that AQMP_EXCLUSIVE does not prevent the RPC server from responding on the client's reply-to queue. AMQP_EXCLUSIVE pertains to two separate threads (channels instances) trying to publish to the same queue resource. When this occurs, the queue is essentially locked for subsequent connections. The same behavior occurs with an exchange declaration.
#Denis: Your implementation in this case is correct up to a point
Bad - don't re-declare the Queue in the server. That's the client's job
$callbackQueue = new \AMQPQueue($channel);
$callbackQueue->setName($envelope->getReplyTo());
$callbackQueue->setFlags(AMQP_EXCLUSIVE); // set EXCLUSIVE flag
...
$callbackQueue->bind($exchange->getName(), 'rpc_reply');
You're trying to bind to a QUEUE called tempQueue. But you've already created a queue called tempQueue in the client.php. Depending on which service is started first, the other will throw an error. So you can cut out all of that and just keep the last part:
// trying to publish response back to client's callback queue
$exchange->publish(
json_encode(array('processed by remote service!')),
'rpc_reply', //<--BAD Should be: $envelope->getReplyTo()
AMQP_MANDATORY & AMQP_IMMEDIATE
);
Then modify the above by replacing:
'rpc_reply'
with
$envelope->getReplyTo()
Don't Declare a Queue Name on the client side
// create a queue which we send messages to server via
$queue = new \AMQPQueue($channel);
//$queue->setName('tempQueue'); //remove this line
//add exclusivity
$queue->setFlags(AMQP_EXCLUSIVE);
$queue->declare();
//no need for binding... we're communicating on the queue directly
//there is no one listening to 'temp_action' so this implementation will send your message into limbo
//$queue->bind($exchangeName, 'temp_action'); //remove this line
My answer from this question replied on the RabbitMQ Official mailing list
While not using the same library here you have the official tutorials ported to PHP
https://github.com/rabbitmq/rabbitmq-tutorials/tree/master/php
The problem in your code is that you declare queues with different options.
So as one reply say, if you declare queue A as durable, then every other declaration of that queue must be durable. The same for the exclusive flag.
Also you don't need to redeclare a queue to publish messages to it. As an RPC server you assume that the address sent in the 'reply_to' property is already present. I think is the responsibility of the RpcClient to make sure the queue where it is waiting for replies exists already.
Addendum:
Exclusivity in queues means that the only the channel that declared the queue can access it.
On your server you should also declare your queue as exclusive. Remember, RabbitMQ queues should have the same flag. For example if you declare queue that is set to "durable" the other end should also declare the queue a "durable" So on your server put a flag $callbackQueue->setFlags(AMQP_EXCLUSIVE); somewhat like that from your client.