We use the rabbitmq message delay plugin (rabbitmq_delayed_message_exchange) for delaying messages. Is it possible for debugging and monitoring purposes, to show holded / delayed messages in rabbitmq admin web interface?
Link: https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/
Bye,
Ben
No; delayed messages are not visible in the admin UI.
As an alternative you can route the messages to a real queue, with a TTL defined as well as dead lettering which will cause expired message to be routed to the final queue.
You can set a fixed TTL on the temporary queue or use the expiration property on individual messages.
EDIT
#SpringBootApplication
public class So50760600Application {
public static void main(String[] args) {
SpringApplication.run(So50760600Application.class, args);
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
return args -> template.convertAndSend("", "temp", "foo", m -> {
m.getMessageProperties().setExpiration("5000");
return m;
});
}
#RabbitListener(queues = "final")
public void in(String in, #Header("x-death") List<?> death) {
System.out.println(in + ", x-death:" + death);
}
#Bean
public Queue temp() {
Map<String, Object> args = new HashMap<>();
args.put("x-message-ttl", 10000); // default (max)
args.put("x-dead-letter-exchange", "");
args.put("x-dead-letter-routing-key", "final");
return new Queue("temp", true, false, false, args);
}
#Bean
public Queue finalQ() {
return new Queue("final");
}
}
and
foo:[{reason=expired, original-expiration=5000, count=1, exchange=, time=Fri Jun 08 10:43:42 EDT 2018, routing-keys=[temp], queue=temp}]
Related
I have a scenario where I want to "pull" messages of a RabbitMQ queue/topic and process them one at a time.
Specifically if there are already messages sitting on the queue when the consumer starts up.
I have tried the following with no success (meaning, each of these options reads the queue until it is either empty or until another thread closes the context).
1.Stopping route immediately it is first processed
final CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
RouteDefinition route = from("rabbitmq:harley?queue=IN&declare=false&autoDelete=false&hostname=localhost&portNumber=5672");
route.process(new Processor() {
Thread stopThread;
#Override
public void process(final Exchange exchange) throws Exception {
String name = exchange.getIn().getHeader(Exchange.FILE_NAME_ONLY, String.class);
String body = exchange.getIn().getBody(String.class);
// Doo some stuff
routeComplete[0] = true;
if (stopThread == null) {
stopThread = new Thread() {
#Override
public void run() {
try {
((DefaultCamelContext)exchange.getContext()).stopRoute("RabbitRoute");
} catch (Exception e) {}
}
};
}
stopThread.start();
}
});
}
});
context.start();
while(!routeComplete[0].booleanValue())
Thread.sleep(100);
context.stop();
}
Similar to 1 but using a latch rather than a while loop and sleep.
Using a PollingConsumer
final CamelContext context = new DefaultCamelContext();
context.start();
Endpoint re = context.getEndpoint(srcRoute);
re.start();
try {
PollingConsumer consumer = re.createPollingConsumer();
consumer.start();
Exchange exchange = consumer.receive();
String bb = exchange.getIn().getBody(String.class);
consumer.stop();
} catch(Exception e){
String mm = e.getMessage();
}
Using a ConsumerTemplate() - code similar to above.
I have also tried enabling preFetch and setting the max number of exchanges to 1.
None of these appear to work, if there are 3 messages on the queue, all are read before I am able to stop the route.
If I were to use the standard RabbitMQ Java API I would use a basicGet() call which lets me read a single message, but for other reasons I would prefer to use a Camel consumer.
Has anyone successfully been able to process a single message on a queue that holds multiple messages using a Camel RabbitMQ Consumer?
Thanks.
This is not the primary intention of the component as its for continued received. But I have created a ticket to look into supporting a basicGet (single receive). There is a new spring based rabbitmq component coming in 3.8 onwards so its going to be implemeneted there (first): https://issues.apache.org/jira/browse/CAMEL-16048
Given I have application with AMQP anonymous queue and fanout exchange:
#Bean
public Queue cacheUpdateAnonymousQueue() {
return new AnonymousQueue();
}
public static final String CACHE_UPDATE_FANOUT_EXCHANGE = "cache.update.fanout";
#Bean
FanoutExchange cacheUpdateExchange() {
return new FanoutExchange(CACHE_UPDATE_FANOUT_EXCHANGE);
}
#Bean
Binding cacheUpdateQueueToCacheUpdateExchange() {
return bind(cacheUpdateAnonymousQueue())
.to(cacheUpdateExchange());
}
and Spring Integration flow:
#Bean
public IntegrationFlow cacheOutputFlow() {
return from(channelConfig.cacheUpdateOutputChannel())
.transform(objectToJsonTransformer())
.handle(outboundAdapter())
.get();
}
And I use outbound adapter:
public MessageHandler outboundAdapter() {
rabbitTemplate.setChannelTransacted(true);
return outboundAdapter(rabbitTemplate)
.exchangeName(CACHE_UPDATE_FANOUT_EXCHANGE)
.get();
}
I can see in logs:
o.s.amqp.rabbit.core.RabbitTemplate: Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,4), conn: Proxy#40976c4b Shared Rabbit Connection: SimpleConnection#1cfaa28d [delegate=amqp://guest#127.0.0.1:5672/, localPort= 56042]
o.s.amqp.rabbit.core.RabbitTemplate: Publishing message on exchange [cache.update.fanout], routingKey = []
but message is not delivered to queue bound to cache.update.fanout exchange.
When I set rabbitTemplate.setChannelTransacted(false); in outbound adapter, then I can see in logs:
o.s.amqp.rabbit.core.RabbitTemplate : Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,1), conn: Proxy#11a1389d Shared Rabbit Connection: SimpleConnection#444c6abf [delegate=amqp://guest#127.0.0.1:5672/, localPort= 56552]
o.s.amqp.rabbit.core.RabbitTemplate : Publishing message on exchange [cache.update.fanout], routingKey = []
and message is delivered to queue.
Why is message not delivered in first case?
Why doesn't RabbitTemplate indicate something?
Your logs have different exchange names; I just tested it like this...
#SpringBootApplication
public class So60993877Application {
public static void main(String[] args) {
SpringApplication.run(So60993877Application.class, args);
}
#Bean
public Queue cacheUpdateAnonymousQueue() {
return new AnonymousQueue();
}
public static final String CACHE_UPDATE_FANOUT_EXCHANGE = "cache.update.fanout";
#Bean
FanoutExchange cacheUpdateExchange() {
return new FanoutExchange(CACHE_UPDATE_FANOUT_EXCHANGE);
}
#Bean
Binding cacheUpdateQueueToCacheUpdateExchange() {
return BindingBuilder.bind(cacheUpdateAnonymousQueue())
.to(cacheUpdateExchange());
}
#RabbitListener(queues = "#{cacheUpdateAnonymousQueue.name}")
public void listen(String in) {
System.out.println(in);
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
return args -> {
template.convertAndSend(CACHE_UPDATE_FANOUT_EXCHANGE,
cacheUpdateAnonymousQueue().getName(), "foo");
template.setChannelTransacted(true);
template.convertAndSend(CACHE_UPDATE_FANOUT_EXCHANGE,
cacheUpdateAnonymousQueue().getName(), "bar");
};
}
}
With no problems.
foo
bar
With confirms and returns enabled:
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
template.setReturnCallback((message, replyCode, replyText, exchange, routingKey) ->
LOG.info("Return: " + message));
template.setConfirmCallback((correlationData, ack, cause) ->
LOG.info("Confirm: " + correlationData + ": " + ack));
return args -> {
template.convertAndSend(CACHE_UPDATE_FANOUT_EXCHANGE, cacheUpdateAnonymousQueue().getName(),
"foo", new CorrelationData("foo"));
// template.setChannelTransacted(true);
template.convertAndSend(CACHE_UPDATE_FANOUT_EXCHANGE, cacheUpdateAnonymousQueue().getName(),
"bar", new CorrelationData("bar"));
template.convertAndSend("missingExchange", cacheUpdateAnonymousQueue().getName(), "baz",
new CorrelationData("baz"));
Thread.sleep(5000);
};
}
I connect RabbitMQ with sprin cloud config:
#Bean
public ConnectionFactory rabbitConnectionFactory() {
Map<String, Object> properties = new HashMap<String, Object>();
properties.put("publisherConfirms", true);
RabbitConnectionFactoryConfig rabbitConfig = new RabbitConnectionFactoryConfig(properties);
return connectionFactory().rabbitConnectionFactory(rabbitConfig);
}
2.Set rabbitTemplate.setMandatory(true) and setConfirmCallback():
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory);
template.setMandatory(true);
template.setMessageConverter(new Jackson2JsonMessageConverter());
template.setConfirmCallback((correlationData, ack, cause) -> {
if (!ack) {
System.out.println("send message failed: " + cause + correlationData.toString());
} else {
System.out.println("Publisher Confirm" + correlationData.toString());
}
});
return template;
}
3.Send message to queue to invoke the publisherConfirm and print log.
#Component
public class TestSender {
#Autowired
private RabbitTemplate rabbitTemplate;
#Scheduled(cron = "0/5 * * * * ? ")
public void send() {
this.rabbitTemplate.convertAndSend(EXCHANGE, "routingkey", "hello world",
(Message m) -> {
m.getMessageProperties().setHeader("tenant", "aaaaa");
return m;
}, new CorrelationData(UUID.randomUUID().toString()));
Date date = new Date();
System.out.println("Sender Msg Successfully - " + date);
}
}
But publisherConfirm have not worked.The log have not been printed. Howerver true or false, log shouldn't been absent.
Mandatory is not needed for confirms, only returns.
Some things to try:
Turn on DEBUG logging to see it it helps; there are some logs generated regarding confirms.
Add some code
.
template.execute(channel -> {
system.out.println(channel.getClass());
return null;
}
If you don't see PublisherCallbackChannelImpl then it means the configuration didn't work for some reason. Again DEBUG logging should help with the configuration debugging.
If you still can't figure it out, strip your application to the bare minimum that exhibits the behavior and post the complete application.
We are trying to implement Retry mechanism on client exceptions. We want to be able to set different routing key, ttl and retry count based on the content in each message. We want to keep the handler simple, i.e; for handleMessage to throw exception. How do we handle this exception and send the message to DLX with appropriate parameters. On retry if the failure happens again - message would be discarded (acknowledged) , or will be put back on DLX with incrementing the retry count. where would we implement this logic and how would be wired?
========================
With Gary's direction, I was able to implement. Here are excerpts ..
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
jsonMessageHandler.setQueueName(queueName);
container.setQueueNames(queueName);
container.setMessageListener(jsonMessageListenerAdapter());
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
return container;
}
#Bean
public MessageListenerAdapter messageListenerAdapter() {
return new MessageListenerAdapter(messageHandler,messageConverter);
}
#Bean
public MessageListenerAdapter jsonMessageListenerAdapter() {
return new MessageListenerAdapter(jsonMessageHandler);
}
#Bean
RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless().recoverer(republishMessageRecoverer).maxAttempts(1).build();
}
#Bean
RepublishMessageRecoverer republishMessageRecoverer() {
return new MyRepublishMessageRecoverer(rabbitTemplate());
}
==========
public class MyRepublishMessageRecoverer extends RepublishMessageRecoverer {
// - constructor
#Override
public void recover(Message message, Throwable cause) {
//Deal with headers
long currentCount = 0;
List xDeathList = (List)message.getMessageProperties().getHeaders().get("x-death");
if(xDeathList != null && xDeathList.size() > 0) {
currentCount = (Long)((Map)(xDeathList.get(0))).get("count");
}
if(currentCount < context.getRules().getNumberOfRetries()) {
//message sent to DLX
this.retryTemplate.send(handlerProperties.getSystem(), message);
} else {
//message ignored
}
throw new AmqpRejectAndDontRequeueException(cause);
}
}
You can't modify a rejected message, it is routed to the DLX/DLQ unchanged (except x-death headers are added by the broker).
You have to republish to the DLX/DLQ yourself if you want to change message properties.
You can use Spring Retry with a customized RepublishMessageRecoverer to do this.
My case is rabbitmq server got out of space, just as below
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ramonubuntu--vg-root 6299376 5956336 0 100% /
The producer publishes message to server(the message needs to be persisted), and then will be blocked forever, it will keeping waiting the response of publishing. Sure we should avoid the situation of server out of space, but is there any timeout mechanism to let producer quit the waiting?
I have tried heartbeat and SO_TIMEOUT, they both don't work, as the network works fine. Below is my producer.
protected void publish(byte[] message) throws Exception {
// ConnectionFactory can be reused between threads.
ConnectionFactory factory = new SoTimeoutConnectionFactory();
factory.setHost(this.getHost());
factory.setVirtualHost("te");
factory.setPort(5672);
factory.setUsername("amqp");
factory.setPassword("amqp");
factory.setConnectionTimeout(10 * 1000);
// doesn't help if server got out of space
factory.setRequestedHeartbeat(1);
final Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
// declare a 'topic' type of exchange
channel.exchangeDeclare(this.exchangeName, "topic", true);
channel.addReturnListener(new ReturnListener() {
#Override
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey,
AMQP.BasicProperties properties, byte[] body) throws IOException {
logger.warn("[X]Returned message(replyCode:" + replyCode + ",replyText:" + replyText
+ ",exchange:" + exchange + ",routingKey:" + routingKey + ",body:" + new String(body));
}
});
channel.confirmSelect();
channel.addConfirmListener(new ConfirmListener() {
#Override
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
logger.info("Ack: " + deliveryTag);
// RabbitMessagePublishMain.this.release(connection);
}
#Override
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
logger.info("Nack: " + deliveryTag);
// RabbitMessagePublishMain.this.release(connection);
}
});
channel.basicPublish(this.exchangeName, RabbitMessageConsumerMain.EXCHANGE_NAME + ".-1", true,
MessageProperties.PERSISTENT_BASIC, message);
channel.waitForConfirmsOrDie(10*1000);
// now we can close connection
connection.close();
}
It will block at 'channel.waitForConfirmsOrDie(10*1000);', and the SotimeoutConnectionFactory,
public class SoTimeoutConnectionFactory extends ConnectionFactory {
#Override
protected void configureSocket(Socket socket) throws IOException {
super.configureSocket(socket);
socket.setSoTimeout(10 * 1000);
}
}
Also I captured the network between producer and rabbimq,
Please help.
You need to implement Connection Block/Unblocked.
This is basically a way of notifying the publisher that the server is running out of resources. The advantage with this is that the publisher will also be notified once it is safe to publish again.
I would recommend that you take a look at this article. A simple way of implementing this is to have a flag that indicates if it is safe to publish, if it is not wait until it is.
As an example you can take a look on how I implemented this in one of my Python examples.