I'm running ActiveMQ 5.14.5. I have a Queue with some Pending messages. Screenshot from the console:
There are no active consumers.
The console reports that there are 21651 messages. However if I try and view them, it appears to be empty:
Furthermore when I try to call receive() on my org.apache.activemq.jms.pool.PooledConnection it blocks and receives no messages.
I'm fairly sure that there are messages there, and they should be retrieved. This used to work, and has stopped working.
Is there an explanation for this? There aren't any errors in the log.
Edit:
I'm using the Java client in Clojure. I didn't want to share it because it might confuse matters, but here it is. I'm using a Pooled factory in a couple of different threads. But I think the above example using the console is self-contained.
(let [factory (org.apache.activemq.ActiveMQConnectionFactory.
"Username"
"Password"
"URI")
pooled-connection-factory (org.apache.activemq.jms.pool.PooledConnectionFactory.)]
(.setConnectionFactory pooled-connection-factory factory)
(.start pooled-connection-factory)
(with-open [connection (.createConnection factory)]
(let [session (.createSession connection false, javax.jms.Session/AUTO_ACKNOWLEDGE)
destination (.createQueue session (:queue-name config))
consumer (.createConsumer session destination)]
(.start connection)
(loop [message (.receive consumer)]
(println (.getText ^org.apache.activemq.command.ActiveMQTextMessage message))
(recur (.receive consumer))))))
Related
SOLVED PROBLEM.
Just create 2 different queues, like rpc.queue and pubsub.queue. Then you can use multiple messaging pattern in one service without any problem.
I create one rails service using Bunny and ConnectionPool Gem. This service "in my mind (cause not yet implemented)" handle multiple RMQ pattern such as Direct Messaging and RPC. These patterns initialized with different object of Connection Class and defined inside initalizer folder.
Initializer looks like this:
# RMQ Initializer for RabbitMQ Connection
class RMQ
include Lontara::RMQ
# NOTE: Call 2 Server caused errors
def self.start(url:, queue:, rpc_exchange:, pubsub_exchange:)
# Then start the consumer and subscriber
Server::RPCConsumer.new(Connection.new(url:), queue:, exchange: rpc_exchange).consume
Server::Subscriber.new(Connection.new(url:), queue:, exchange: pubsub_exchange).subscribe
end
end
RMQ.start(
url: ENV.fetch('RABBITMQ_URL', 'amqp://guest:guest#rmqserver:5672'),
queue: ENV.fetch('RABBITMQ_QUEUE_VOUCHER', 'lontara-dev.voucher'),
rpc_exchange: ENV.fetch('RABBITMQ_EXCHANGE_RPC', 'lontara-dev.rpc'),
pubsub_exchange: ENV.fetch('RABBITMQ_EXCHANGE_PUBSUB', 'lontara-dev.pubsub')
)
and Connection class:
module Lontara
module RMQ
# Class Connection initializing the connection to RabbitMQ.
class Connection
def initialize(url: ENV['RABBITMQ_URL'])
#connection = Bunny.new(url)
connection.start
#channel = channel_pool.with(&:create_channel)
yield self if block_given?
end
def close
channel.close
connection.close
end
attr_reader :connection, :channel
private
def channel_pool
#channel_pool ||= ConnectionPool.new { #connection }
end
end
end
end
The problem goes whenever these 2 Server:: (RPC and Subscriber) activated. Impacted only when use RPC as messaging, the problem is RPC Publisher does not get response from Consumer.
These steps (when RPC produce error) are:
Run Rails server
Open new terminal, and open rails console in same project
Create Request to Consumer using RPCPublisher
Publisher get response. Then send request again... On this step not get response.
Job is pending, i push ctrl+c to terminate job. Send request again, and get response...
Try again like step 4, and error...
But, if Server::Publisher not initialized on initializer, nothing error happened.
I assumed this error happened cause of thread... But i don't really get helped from other articles on internet.
My expectation is so simple:
RPC Connection requested for Get related (because RPC can reply this request) or other action requires response. And Pub/Sub (Direct) request for Create, Update, Delete since this type didn't need it.
Your answer really help me... Thankyou !
I am continuously listening on redis streams using the spring reactive api(using lettuce driver). I am using a standalone connection. It seems like the reactor's event loop opens a new connection every time it reads the messages instead of keeping the connection open. I see a lot of TIME_WAIT ports in my machine when i run my program. Is this normal? Is there a way to let lettuce know to re-use the connection instead of reconnecting every time?
This is my code:
StreamReceiver<String, MapRecord<String, String, String>> receiver = StreamReceiver.create(factory);
return receiver
.receive(Consumer.from(keyCacheStreamsConfig.getConsumerGroup(), keyCacheStreamsConfig.getConsumer()),
StreamOffset.create(keyCacheStreamsConfig.getStreamName(), ReadOffset.lastConsumed()))//
// flatMap reads 256 messages by default and processes them in the given scheduler
.flatMap(record -> Mono.fromCallable(() -> consumer.consume(record)).subscribeOn(Schedulers.boundedElastic()))//
.doOnError(t -> {
log.error("Error processing.", t);
streamConnections.get(nodeName).setDirty(true);
})//
.onErrorContinue((err, elem) -> log.error("Error processing message. Continue listening."))//
.subscribe();
Looks like the spring-data-redis library re-uses the connection only if the poll timeout is set to '0' in the stream receiver options and pass it as the second argument in StreamReceiver.create(factory, options). Figured by looking into spring-data-redis' source code.
I have two servers, call them A and B. B runs RabbitMQ, while A connects to RabbitMQ via Kombu. If I restart RabbitMQ on B, the kombu connection breaks, and the messages are no longer delivered. I then have to reset the process on A to re-establish the connection. Is there a better approach, i.e. is there a way for Kombu to re-connect automatically, even if the RabbitMQ process is restarted?
My basic code implementation is below, thanks in advance! :)
def start_consumer(routing_key, incoming_exchange_name, outgoing_exchange_name):
global rabbitmq_producer
incoming_exchange = kombu.Exchange(name=incoming_exchange_name, type='direct')
incoming_queue = kombu.Queue(name=routing_key+'_'+incoming_exchange_name, exchange=incoming_exchange, routing_key=routing_key)#, auto_delete=True)
outgoing_exchange = kombu.Exchange(name=outgoing_exchange_name, type='direct')
rabbitmq_producer = kombu.Producer(settings.rabbitmq_connection0, exchange=outgoing_exchange, serializer='json', compression=None, auto_declare=True)
settings.rabbitmq_connection0.connect()
if settings.rabbitmq_connection0.connected:
callbacks=[]
queues=[]
callbacks.append(callback)
# if push_queue:
# callbacks.append(push_message_callback)
queues.append(incoming_queue)
print 'opening a new *incoming* rabbitmq connection to the %s exchange for the %s queue' % (incoming_exchange.name, incoming_queue.name)
incoming_exchange(settings.rabbitmq_connection0).declare()
incoming_queue(settings.rabbitmq_connection0).declare()
print 'opening a new *outgoing* rabbitmq connection to the %s exchange' % outgoing_exchange.name
outgoing_exchange(settings.rabbitmq_connection0).declare()
with settings.rabbitmq_connection0.Consumer(queues=queues, callbacks=callbacks) as consumer:
while True:
settings.rabbitmq_connection0.drain_events()
On the consumer side, kombu.mixins.ConsumerMixin handles reconnecting when the connection goes away (and also does heartbeats, etc., and lets you write less code). There doesn't seem to be a ProducerMixin, unfortunately but you could potentially dig into the code and adapt it...?
I'm evaluating RabbitMQ and while the general impression (of AMQP as such, and also RabbitMQ) is positive, I'm not very impressed by the result.
I'm attempting to publish and consume messages simultaneously and have achieved very poor message rates. I have a durable direct exchange, which is bound to a durable queue and I publish persistent messages to that exchange. The average size of the message body is about 1000 bytes.
My publishing happens roughly as follows:
AMQP.BasicProperties.Builder bldr = new AMQP.BasicProperties.Builder();
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("guest");
factory.setPassword("guest");
factory.setVirtualHost("/");
factory.setHost("my-host");
factory.setPort(5672);
Connection conn = null;
Channel channel = null;
ObjectMapper mapper = new ObjectMapper(); //com.fasterxml.jackson.databind.ObjectMapper
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.confirmSelect();
} catch (IOException e) {}
for(Message m : messageList) { //the size of messageList happens to be 9945
try {
channel.basicPublish("exchange", "", bldr.deliveryMode(2).contentType("application/json").build(), mapper.writeValueAsBytes(cm));
} catch (Exception e) {}
}
try {
channel.waitForConfirms();
channel.close();
conn.close();
} catch (Exception e1) {}
And consuming messages from the bound queue happens as so:
AMQP.BasicProperties.Builder bldr = new AMQP.BasicProperties.Builder();
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("guest");
factory.setPassword("guest");
factory.setVirtualHost("/");
factory.setHost("my-host");
factory.setPort(5672);
Connection conn = null;
Channel channel = null;
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.basicQos(100);
while (true) {
GetResponse r = channel.basicGet("rawDataQueue", false);
if(r!=null)
channel.basicAck(r.getEnvelope().getDeliveryTag(), false);
}
} catch (IOException e) {}
The problem is that when the message publisher (or several of them) and consumer (or several of them) run simultaneously then the publisher(s) appear to run at full throttle and the RabbitMQ management web interface shows a publishing rate of, say, ~2...3K messages per second, but a consumption rate of 0.5...3 per consumer. When the publisher(s) finish then I get a consumption rate of, say, 300...600 messages per consumer. When not setting the QOS prefetch value for the Java client, then a little less, when setting it to 100 or 250, then a bit more.
When experimenting with throttling the consumers somewhat, I have managed to achieve simultaneous numbers like ~400 published and ~50 consumed messages per second which is marginally better but only marginally.
Here's, a quote from the RabbitMQ blog entry which claims that queues are fastest when they're empty which very well may be, but slowing the consumption rate to a crawl when there are a few thousand persistent messages sitting in the queue is still rather unacceptable.
Higher QOS prefetching values may help a bit but are IMHO not a solution as such.
What, if anything, can be done to achieve reasonable throughput rates (2 consumed messages per consumer per second is not reasonable in any circumstance)? This is only a simple one direct exchange - one binding - one queue situation, should I expect more performance degradation with more complicated configurations? When searching around the internet there have also been suggestions to drop durability, but I'm afraid in my case that is not an option. I'd be very happy if somebody would point out that I'm stupid and that there is an evident and straightforward solution of some kind :)
Have you tried with the autoAck option? That should improve your performance. It is much faster than getting the messages one by one and ack'ing them. Increasing the prefetch count should make it even better too.
Also, what is the size of the messages you are sending and consuming including headers? Are you experiencing any flow-control in the broker?
Another question, are you creating a connection and channel every time you send/get a message? If so, that's wrong. You should be creating a connection once, and use a channel per thread (probably in a thread-local fashion) to send and receive messages. You can have multiple channels per connection. There is no official documentation about this, but if you read articles and forums this seems to be the best performance practice.
Last thing, have you considered using the basicConsume instead of basicGet? It should also make it faster.
Based on my experience, I have been able to run a cluster sending and consuming at rates around 20000 messages per second with non-persistent messages. I guess that if you are using durable and persistent messages the performance would decrease a little, but not 10x.
Operating system could schedule your process to the next time slot, if sleep is used. This could create significant performance decrease.
I'd like to send a message to a RabbitMQ server and then wait for a reply message (on a "reply-to" queue). Of course, I don't want to wait forever in case the application processing these messages is down - there needs to be a timeout. It sounds like a very basic task, yet I can't find a way to do this. I've now run into this problem with Java API.
The RabbitMQ Java client library now supports a timeout argument to its QueueConsumer.nextDelivery() method.
For instance, the RPC tutorial uses the following code:
channel.basicPublish("", requestQueueName, props, message.getBytes());
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
if (delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
Now, you can use consumer.nextDelivery(1000) to wait for maximum one second. If the timeout is reached, the method returns null.
channel.basicPublish("", requestQueueName, props, message.getBytes());
while (true) {
// Use a timeout of 1000 milliseconds
QueueingConsumer.Delivery delivery = consumer.nextDelivery(1000);
// Test if delivery is null, meaning the timeout was reached.
if (delivery != null &&
delivery.getProperties().getCorrelationId().equals(corrId)) {
response = new String(delivery.getBody());
break;
}
}
com.rabbitmq.client.QueueingConsumer has a nextDelivery(long timeout) method, which will do what you want. However, this has been deprecated.
Writing your own timeout isn't so hard, although it may be better to have an ongoing thread and a list of in-time identifiers, rather than adding and removing consumers and associated timeout threads all the time.
Edit to add: Noticed the date on this after replying!
There is similar question. Although it's answers doesn't use java, maybe you can get some hints.
Wait for a single RabbitMQ message with a timeout
I approached this problem using C# by creating an object to keep track of the response to a particular message. It sets up a unique reply queue for a message, and subscribes to it. If the response is not received in a specified timeframe, a countdown timer cancels the subscription, which deletes the queue. Separately, I have methods that can be synchronous from my main thread (uses a semaphore) or asynchronous (uses a callback) to utilize this functionality.
Basically, the implementation looks like this:
//Synchronous case:
//Throws TimeoutException if timeout happens
var msg = messageClient.SendAndWait(theMessage);
//Asynchronous case
//myCallback receives an exception message if there is a timeout
messageClient.SendAndCallback(theMessage, myCallback);