Spring AMQP message resiliency in case broker connectivity down - rabbitmq

I have a Use Case of managing Spring AMQP client message resiliency in case of RabbitMQ server connectivity down,
For the same I have used Spring Retry
RabbitTemplate template = // get template from some bean
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
// 10 sec
backOffPolicy.setInitialInterval(10000);
// 1 hr
backOffPolicy.setMultiplier(360.0);
// 1 hr max interval
backOffPolicy.setMaxInterval(3600001);
retryTemplate.setBackOffPolicy(backOffPolicy);
template.setRetryTemplate(retryTemplate);
template.convertAndSend("Direct-Exchange",
"Test.Routing.Key", AMQPMessage);
But when a try to test it and bring broker down it hangs at template.convertAndSend() and never recovers even when RabbitMQ broker connectivity is restored

Your retry configuration is rather extreme. Did you wait for an hour?
It works fine for me...
#SpringBootApplication
public class So44300651Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So44300651Application.class, args).close();
}
#Autowired
private RabbitTemplate template;
#Override
public void run(String... args) throws Exception {
template.convertAndSend("foo");
}
#Bean
public RabbitTemplate template(ConnectionFactory cf) {
RabbitTemplate template = new RabbitTemplate(cf);
RetryTemplate retry = new RetryTemplate();
ExponentialBackOffPolicy backOff = new ExponentialBackOffPolicy();
backOff.setInitialInterval(10_000);
backOff.setMultiplier(1.5);
backOff.setMaxInterval(15_000);
retry.setBackOffPolicy(backOff);
template.setRetryTemplate(retry);
return template;
}
}
Result:
09:19:45.592 [main] DEBUG o.s.retry.support.RetryTemplate - Retry: count=0
09:19:45.603 [main] DEBUG o.s.r.b.ExponentialBackOffPolicy - Sleeping for 10000
09:19:55.607 [main] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=1
09:19:55.607 [main] DEBUG o.s.retry.support.RetryTemplate - Retry: count=1
09:19:55.608 [main] DEBUG o.s.r.b.ExponentialBackOffPolicy - Sleeping for 15000
09:20:10.610 [main] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=2
09:20:10.610 [main] DEBUG o.s.retry.support.RetryTemplate - Retry: count=2
09:20:10.654 [main] INFO o.s.a.r.c.CachingConnectionFactory - Created new connection: SimpleConnection#13cf7d52 [delegate=amqp://guest#127.0.0.1:5672/, localPort= 56958]

Related

Multiple consumer issue in ActiveMQ with Wildfly

I am trying to build a basic registration form that sends the form data to a stateless bean through a Servlet. But inside ActiveMQ console I see two consumers for same queue also I need to register twice to add a new user in the list.
Any help is appreciated.
Logs:
WARNING: All illegal access operations will be denied in a future release
15:35:34,892 WARN [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0101: Duplicate servlet mapping /Register found
15:35:35,008 INFO [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 84) Initializing Mojarra 2.3.14.SP01 for context '/demowarproject'
15:35:35,436 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 84) WFLYUT0021: Registered web context: '/demowarproject' for server 'default-server'
15:35:35,509 INFO [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "activemq-rar-5.17.2.rar" (runtime-name : "activemq-rar-5.17.2.rar")
15:35:35,510 INFO [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "demowarproject.war" (runtime-name : "demowarproject.war")
15:35:35,510 INFO [org.jboss.as.server] (ServerService Thread Pool -- 46) WFLYSRV0010: Deployed "demoejbprojectEAR.ear" (runtime-name : "demoejbprojectEAR.ear")
15:35:35,551 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
15:35:35,555 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (Controller Boot Thread) Starting
15:35:35,561 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (Controller Boot Thread) Starting
15:35:35,559 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (default-threads - 1) Establishing connection to broker [tcp://localhost:61616]
15:35:35,565 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (default-threads - 2) Establishing connection to broker [tcp://localhost:61616]
15:35:35,567 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 21.0.2.Final (WildFly Core 13.0.3.Final) started in 8578ms - Started 648 of 874 services (390 services are lazy, passive or on-demand)
15:35:35,574 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
15:35:35,574 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
15:35:35,801 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (default-threads - 2) Successfully established connection to broker [tcp://localhost:61616]
15:35:35,801 INFO [org.apache.activemq.ra.ActiveMQEndpointWorker] (default-threads - 1) Successfully established connection to broker [tcp://localhost:61616]
15:35:50,821 INFO [stdout] (default task-1) message sent!
15:35:51,154 INFO [stdout] (default-threads - 3) Registered!!, User count is: 1
15:35:51,158 INFO [stdout] (default-threads - 3) All users are:
15:35:51,158 INFO [stdout] (default-threads - 3) 87KumarPrashant#gmail.com
15:35:51,158 INFO [stdout] (default-threads - 3)
15:36:10,979 INFO [stdout] (default task-1) message sent!
15:36:10,995 INFO [stdout] (default-threads - 4) Registered!!, User count is: 1
15:36:10,996 INFO [stdout] (default-threads - 4) All users are:
15:36:10,996 INFO [stdout] (default-threads - 4) 87KumarPrashant#gmail.com
15:36:10,996 INFO [stdout] (default-threads - 4)
15:36:12,270 INFO [stdout] (default task-1) message sent!
15:36:12,286 INFO [stdout] (default-threads - 5) Email already registered 1
15:36:12,287 INFO [stdout] (default-threads - 5) All users are:
15:36:12,287 INFO [stdout] (default-threads - 5) 87KumarPrashant#gmail.com
15:36:12,287 INFO [stdout] (default-threads - 5)
15:36:13,438 INFO [stdout] (default task-1) message sent!
15:36:13,448 INFO [stdout] (default-threads - 6) Email already registered 1
15:36:13,449 INFO [stdout] (default-threads - 6) All users are:
15:36:13,449 INFO [stdout] (default-threads - 6) 87KumarPrashant#gmail.com
15:36:13,449 INFO [stdout] (default-threads - 6)
As you can see it needs twice to register before it says already registered, but user count is not increased second time (weird). And WildFly console shows two consumers for the same queue (also weird) as I am not using the same queue anywhere else.
ActiveMQ Console showing 2 consumers instead of 1:
Here are related files:
MDB:
import java.util.ArrayList;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.ObjectMessage;
#MessageDriven(activationConfig = {
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "destination", propertyValue = "jms/queue/demo")})
public class DemoEjb implements MessageListener
{
static ArrayList<User> usersList = new ArrayList<User>();
static int userCount = 0;
User u = null;
public void onMessage(Message message)
{
try
{
if (message instanceof ObjectMessage)
{
User user = (User) ((ObjectMessage) message).getObject();
if(user.getOperation().equals("register"))
registerUser(user);
// else
// loginUser(user);
}
} catch (Exception e)
{
e.printStackTrace();
}
}
public void registerUser(User user)
{
if (isUserPresent(user.getEmail())) {
System.out.println("Email already registered " + userCount);
giveAllUsers();
return;
}
addUser(user);
System.out.println("Registered!!, User count is: " + userCount);
giveAllUsers();
}
public boolean isUserPresent(String email)
{
return usersList.stream().anyMatch(d -> d.getEmail().equals(email));
}
public void addUser(User newUser)
{
usersList.add(newUser);
userCount++;
}
public void giveAllUsers()
{
System.out.println("All users are: ");
for(User u: usersList)
{
System.out.println(u.getEmail());
}
System.out.println();
}
}
User (POJO):
import java.io.Serializable;
public class User implements Serializable
{
private static final long serialVersionUID = 1L;
private String name;
private String password;
private String email;
private String operation;
public User(String name, String email, String password, String operation)
{
this.name = name;
this.email = email;
this.password = password;
this.operation = operation;
}
public String getName()
{
return this.name;
}
public void setOperation(String operation) {
this.operation = operation;
}
public String getPassword()
{
return this.password;
}
public String getEmail()
{
return this.email;
}
public void setName(String name)
{
this.name = name;
}
public void setPassword(String pass)
{
this.password = pass;
}
public String getOperation() {
return this.operation;
}
}
Servlet:
import java.io.IOException;
import java.io.PrintWriter;
import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.MessageProducer;
import javax.jms.ObjectMessage;
import javax.jms.Session;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.activemq.ActiveMQConnectionFactory;
import com.enovate.demoejb.User;
#WebServlet("/Register")
public class Register extends HttpServlet
{
private static final long serialVersionUID = 1L;
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException
{
String name = request.getParameter("userName");
String pass = request.getParameter("userPassword");
String email = request.getParameter("userEmail");
User user = new User(name, email, pass, "register");
try
{
System.out.println("message sent!");
sendMessage(user);
PrintWriter out = response.getWriter();
out.println("Message sent!");
} catch (Exception e)
{
e.printStackTrace();
}
}
public void sendMessage(User user) throws Exception
{
Connection connection = null;
try
{
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
connection = connectionFactory.createConnection();
connection.start();
String queue = "jms/queue/demo";
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destinationQueue = session.createQueue(queue);
MessageProducer publisher = session.createProducer(destinationQueue);
ObjectMessage objectMessage = session.createObjectMessage();
objectMessage.setObject(user);
publisher.send(objectMessage);
session.close();
}
finally
{
closeConnection(connection);
}
}
private void closeConnection(Connection con)
{
try
{
if (con != null)
con.close();
}
catch(JMSException jmse)
{
System.out.println("Could not close connection " + con +" exception was " + jmse);
}
}
}
HTML file:
<html>
<head>
<meta charset="ISO-8859-1">
<title>Register</title>
</head>
<body>
<form method="post" action="Register">
Name: <input name="userName" type="text"><br>
Email: <input name="userEmail" type="email"><br>
Password: <input name="userPassword" type="password"><br>
<input type="submit" value="Register">
</form>
</body>
</html>
Web.xml:
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_4_0.xsd" id="WebApp_ID" version="4.0">
<display-name>demowarproject</display-name>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
<welcome-file>index.jsp</welcome-file>
<welcome-file>index.htm</welcome-file>
<welcome-file>default.html</welcome-file>
<welcome-file>default.jsp</welcome-file>
<welcome-file>default.htm</welcome-file>
</welcome-file-list>
<servlet>
<description></description>
<display-name>Register</display-name>
<servlet-name>Register</servlet-name>
<servlet-class>com.enovate.demoservlet.Register</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>Register</servlet-name>
<url-pattern>/Register</url-pattern>
</servlet-mapping>
</web-app>
I was expecting it should work fine and have only one consumer.
It feels like there are two consumer created (don't know how) and then each try accepts the data one after the other.

#method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out

I have searched a few on the internet, but have not found anything for me hence posting here.
The error message "#method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out" is well describing the consumer_timeout (ack timeout) happening. I'm using CachingConnectionFactory.
As per the monitor internal, it keeps looping and trying to create a new channel but the channel number is the same as the previous one and again shutting down the same channel. It's not able to restart the consumer. I want to drop that old channel and create a new channel like before. Why its opening the same channel (i.e. channel 8) again after closing? Any suggestion would be a great help.
Below is the sample of the log for ref :
2022-08-09 09:25:27.688 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t] started
2022-08-09 21:27:01.124 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-09 21:27:01.124 DEBUG 1 --- [.232.164.84:443] m.m.r.amqp.client.AmqpChannelListener : AMQP channel closed [false] channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-09 21:27:02.641 ERROR 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Consumer canceled - channel closed SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t]
2022-08-09 21:27:02.641 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Closing cached Channel: PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxxxxxx/,8)
2022-08-09 21:27:07.641 DEBUG 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Attempting to restart consumer SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t]
2022-08-09 21:27:07.658 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Creating cached Rabbit Channel from PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxx:443/,8)
2022-08-09 21:27:07.658 DEBUG 1 --- [enerContainer-1] m.m.r.amqp.client.AmqpChannelListener : AMQP channel opened [false] 8
2022-08-09 21:27:07.685 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt] started
2022-08-10 09:27:55.005 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-10 09:27:55.005 DEBUG 1 --- [.232.164.84:443] m.m.r.amqp.client.AmqpChannelListener : AMQP channel closed [false] channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-10 09:27:57.641 ERROR 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Consumer canceled - channel closed SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt]
2022-08-10 09:27:57.641 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Closing cached Channel: PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxxxxxxx:443/,8)
2022-08-10 09:27:57.641 DEBUG 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Attempting to restart consumer SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt]
2022-08-10 09:27:57.659 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Creating cached Rabbit Channel from PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxx:443/,8)
2022-08-10 09:27:57.659 DEBUG 1 --- [enerContainer-1] m.m.r.amqp.client.AmqpChannelListener : AMQP channel opened [false] 8
2022-08-10 09:27:57.687 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-RKgc4e0o1wii-OBE9ai69A identity=68a0033a] started
2022-08-10 21:28:01.145 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
AmqpConnectionFactory.java
#Slf4j
#Configuration
public class AmqpConnectionFactory implements InitializingBean, ApplicationListener<ContextClosedEvent> {
private static final Map<Class<? extends Throwable>,Boolean> EXCEPTION_MAP = new HashMap<>();
private static final long INITIAL_RETRY_INTERVAL = 1000L;
private static final int MAX_RETRY_ATTEMPTS = 20;
#Autowired
ApplicationContext applicationContext;
#Autowired
QueueConfig queueConfig;
private RabbitTemplate rabbitTemplate;
private RabbitAdmin rabbitAdmin;
private volatile CachingConnectionFactory cachingConnectionFactory;
private boolean initialized = false;
#Override
public void afterPropertiesSet() {
try {
initialize();
} catch (Exception ex) {
//For now, we're going to say this is ok
}
}
public boolean isInitialized() {
return initialized;
}
public void reinitialize(String host, String port, boolean useSsl, String username, String password) throws Exception {
queueConfig.setHost(host);
queueConfig.setPort(Integer.valueOf(port));
queueConfig.setSslEnabled(useSsl);
queueConfig.setUsername(username);
queueConfig.setPassword(password);
initialize();
}
public void initialize() throws Exception {
if (!StringUtils.hasLength(queueConfig.getHost()) || !StringUtils.hasLength(queueConfig.getUsername())) {
log.info("AMQP configuration is disabled or not complete");
return;
}
log.info("Attempting to connect: {}:{} as {} [ssl={}]", queueConfig.getHost(), queueConfig.getPort(), queueConfig.getUsername(), queueConfig.isSslEnabled());
if (queueConfig.isSslEnabled()) {
cachingConnectionFactory = new CachingConnectionFactory(
new SSLConnectionFactory(queueConfig)) {
#Override
public void onApplicationEvent(ContextClosedEvent event) {
waitForEngineTaskExecutor();
super.onApplicationEvent(event);
}
};
} else {
cachingConnectionFactory = new CachingConnectionFactory(
new BaseConnectionFactory(queueConfig)) {
#Override
public void onApplicationEvent(ContextClosedEvent event) {
waitForEngineTaskExecutor();
super.onApplicationEvent(event);
}
};
}
cachingConnectionFactory.addChannelListener(new AmqpChannelListener());
cachingConnectionFactory.addConnectionListener(new AmqpConnectionListener());
cachingConnectionFactory.setChannelCacheSize(queueConfig.getChannelCacheSize());
cachingConnectionFactory.setConnectionCacheSize(queueConfig.getConnectionCacheSize());
cachingConnectionFactory.setConnectionLimit(queueConfig.getConnectionLimit());
cachingConnectionFactory.setConnectionNameStrategy(f -> HostInfo.getHostname());
cachingConnectionFactory.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
log.info("Connection Last cachingConnectionFactory {} ", cachingConnectionFactory );
try {
rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setRetryTemplate(amqpRetryTemplate());
} catch (Exception ex) {
log.warn("AMQP credentials may be insufficient for pub/sub capabilities");
}
try {
rabbitAdmin = new RabbitAdmin(cachingConnectionFactory);
rabbitAdmin.setRetryTemplate(amqpRetryTemplate());
rabbitAdmin.setIgnoreDeclarationExceptions(true);
} catch (Exception ex) {
log.warn("AMQP credentials may be insufficient for Admin capabilities");
}
initialized = true;
}
private void waitForEngineTaskExecutor() {
try {
ThreadPoolTaskExecutor engineTaskExecutor = (ThreadPoolTaskExecutor)applicationContext.getBean("engineTaskExecutor");
if (engineTaskExecutor != null) {
try {
while(engineTaskExecutor.getActiveCount() > 0) {
log.warn("There are {} jobs currently executing", engineTaskExecutor.getActiveCount());
Thread.currentThread().sleep(10000);
}
log.info("All jobs have been completed");
} catch (Exception ex) {
log.warn("Termination signal received: {}", ex.getMessage());
}
} else {
log.info("EngineTaskExecutor was not initialized at shutdown");
}
} catch (NoSuchBeanDefinitionException ex) {
//We don't have to wait b/c there is no engineTaskExecutor bean
}
}
#Retryable(include = {AmqpAuthenticationException.class, AmqpConnectException.class},
maxAttempts = MAX_RETRY_ATTEMPTS,
backoff = #Backoff(delay = INITIAL_RETRY_INTERVAL,
multiplier = ExponentialBackOffPolicy.DEFAULT_MULTIPLIER,
maxDelay = ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL))
public void waitForConnection() throws Exception {
connect();
}
#Retryable(include = {AmqpAuthenticationException.class, AmqpConnectException.class},
maxAttempts = Integer.MAX_VALUE,
backoff = #Backoff(delay = 5000L,
multiplier = ExponentialBackOffPolicy.DEFAULT_MULTIPLIER,
maxDelay = ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL))
public void waitForConnectionIndefinitely() throws Exception {
connect();
}
public boolean isConnected() {
try {
connect();
return true;
} catch (Exception ex) {
}
return false;
}
private void connect() throws Exception {
if (!initialized) initialize();
if (cachingConnectionFactory == null) throw new AmqpConnectException(new RuntimeException("Connection not yet intialized"));
try {
cachingConnectionFactory.createConnection();
} catch (AmqpAuthenticationException | AmqpConnectException ex) {
log.debug("AMQP connection failure: " + ex.getMessage());
throw ex;
}
}
#Bean
public RetryTemplate amqpRetryTemplate() {
//Note this retryTemplate may not handle initial connection failures
RetryTemplate retryTemplate = new RetryTemplate();
EXCEPTION_MAP.put(AmqpConnectException.class, true);
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(MAX_RETRY_ATTEMPTS, EXCEPTION_MAP, true));
ExponentialBackOffPolicy backOffPolicy = new ExponentialRandomBackOffPolicy();
backOffPolicy.setInitialInterval(INITIAL_RETRY_INTERVAL);
backOffPolicy.setMultiplier(ExponentialBackOffPolicy.DEFAULT_MULTIPLIER);
backOffPolicy.setMaxInterval(ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL);
retryTemplate.setBackOffPolicy(backOffPolicy);
return retryTemplate;
}
public RabbitTemplate getRabbitTemplate() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize RabbitTemplate", ex);
}
return rabbitTemplate;
}
// RabbitAdmin Bean is defined in AmqpRabbitAdmin
public RabbitAdmin getRabbitAdmin() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize RabbitAdmin", ex);
}
return rabbitAdmin;
}
public CachingConnectionFactory getConnectionFactory() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize CachingConnectionFactory", ex);
}
return cachingConnectionFactory;
}
#Override
public void onApplicationEvent(ContextClosedEvent e) {
waitForEngineTaskExecutor();
}
}
CartMessageListenerContainer.java
#Slf4j
#Service
public class CartMessageListenerContainer extends DirectMessageListenerContainer implements SmartLifecycle, ApplicationListener<ContextClosedEvent> {
AmqpConnectionFactory amqpConnectionFactory;
CartMessageListener CartMessageListener;
LocalWorkerAmqpConfig localWorkerAmqpConfig;
LocalWorkerConfig localWorkerConfig;
#Autowired
CartMessageListenerContainer(AmqpConnectionFactory amqpConnectionFactory, CartMessageListener CartMessageListener, LocalWorkerAmqpConfig localWorkerAmqpConfig, LocalWorkerConfig localWorkerConfig) throws Exception {
log.info("Initializing CartMessageListenerContainer...");
this.amqpConnectionFactory = amqpConnectionFactory;
this.CartMessageListener = CartMessageListener;
this.localWorkerAmqpConfig = localWorkerAmqpConfig;
this.localWorkerConfig = localWorkerConfig;
amqpConnectionFactory.waitForConnection();
this.setAcknowledgeMode(AcknowledgeMode.MANUAL);
this.setAutoDeclare(Boolean.FALSE);
this.setAutoStartup(Boolean.TRUE);
this.setConnectionFactory(amqpConnectionFactory.getConnectionFactory());
this.setIdleEventInterval(30000);
this.setMessageListener(CartMessageListener);
this.setMismatchedQueuesFatal(Boolean.FALSE);
this.setMissingQueuesFatal(Boolean.FALSE);
this.setPrefetchCount(localWorkerConfig.getConcurrency());
}
#Override
public void start() {
log.debug("Starting CartMessageListenerContainer...");
try {
List<Queue> cartQueues = new ArrayList<>();
cartQueues.add(localWorkerAmqpConfig.localDefaultQueue());
if (localWorkerConfig.getDestinationId() != null && !localWorkerConfig.getDestinationId().isEmpty()) {
try {
Long.valueOf(localWorkerConfig.getDestinationId());
Queue destinationQueue = new Queue(WorkerQueueNames.getJobsSubscribeToName(localWorkerConfig.getDestinationId()), true, false, false);
cartQueues.add(destinationQueue);
} catch (Exception ex) {
log.warn("DestinationId [{}] is invalid or you do not have access to it", localWorkerConfig.getDestinationId());
}
}
this.setQueues(cartQueues.toArray(new Queue[cartQueues.size()]));
this.lazyLoad();
super.start();
} catch (Exception ex) {
log.error("Failed to start CartMessageListenerContainer", ex);
}
}
#Override
public void stop() {
log.info("Stopping CartMessageListenerContainer...");
super.stop();
}
#Override
protected void handleListenerException(Throwable ex) {
//Overriding this b/c default impl logs a stacktrace after the container shuts down
if (isActive()) {
// Regular case: failed while active.
// Invoke ErrorHandler if available.
invokeErrorHandler(ex);
}
}
#Override
public void onApplicationEvent(ContextClosedEvent e) {
log.info("Destroying CartMessageListenerContainer...");
destroy();
}
}
BaseConnectionFactory.java
#Slf4j
public class BaseConnectionFactory extends ConnectionFactory {
protected BaseConnectionFactory(QueueConfig queueConfig) {
this.setHost(queueConfig.getHost());
this.setPort(queueConfig.getPort());
this.setUsername(queueConfig.getUsername());
this.setPassword(queueConfig.getPassword());
this.setRequestedHeartbeat(queueConfig.getConnectionTimeout());
//This ConnectionFactory will be wrapped in Spring CachingConnectionFactory,
//which prefers to use its own recovery mechanisms
this.setAutomaticRecoveryEnabled(false);
this.setExceptionHandler(new BaseConnectionExceptionHandler());
//shutdown consumers once the corresponding queue is deleted rather than notifying them
Map<String, Object> clientProperties = new HashMap<>();
clientProperties.put("product", ProductInfo.SHORT_NAME);
clientProperties.put("information", ProductInfo.LONG_NAME);
clientProperties.put("copyright", ProductInfo.COPYRIGHT);
clientProperties.put("version", ProductInfo.VERSION);
clientProperties.put("hostname", HostInfo.getHostname());
Map<String, Object> clientCapabilities = new HashMap<>();
clientCapabilities.put("exchange_exchange_bindings", Boolean.TRUE);
clientCapabilities.put("consumer_cancel_notify", Boolean.TRUE);
clientCapabilities.put("basic.nack", Boolean.TRUE);
clientCapabilities.put("publisher_confirms", Boolean.TRUE);
clientCapabilities.put("connection.blocked", Boolean.TRUE);
clientCapabilities.put("authentication_failure_close", Boolean.TRUE);
clientProperties.put("capabilities", clientCapabilities);
this.setClientProperties(clientProperties);
}
protected static class BaseConnectionExceptionHandler extends ForgivingExceptionHandler {
#Override
public void handleUnexpectedConnectionDriverException(Connection conn, Throwable exception) {
log.trace("AMQP connection driver failure: {}", exception.getMessage());
}
#Override
public void handleConnectionRecoveryException(Connection conn, Throwable exception) {
// ignore java.net.ConnectException as those are
// expected during recovery and will only produce noisy
// traces
if (exception instanceof ConnectException) {
// no-op
} else {
log.warn("AMQP connection recovery warning: {}", exception.getMessage());
}
}
#Override
public void handleChannelRecoveryException(Channel ch, Throwable exception) {
log.warn("AMQP channel recovery warning: {}", exception.getMessage());
}
}
}

RabbitMQ CachingConnectionFactory and publisherReturns configuration

A continuation of Make Spring RabbitMQ fail on missing exchange
I register MessageListenerContainer for multiple queues.
Where and how should I configure channel returnListener? - I thing the way I have done it is wrong. I inserted CachingConnectionFactory configuration into createQueueBMessageListener(...) - method responsible for creating one of multiple MessageListeners.
1. How should be CachingConnectionFactory additional configuration done Spring and Rabbit way? By now I didn't configure it in Java (only by application.properties and admins in K8S environment). I only injected ConnectionFactory and set it as connectionFactory in SimpleMessageListenerContainer (as in createQueueAMessageListener(...)), I even didn't know it's CachingConnectionFactory.
Is there something like CachingConnectionFactoryConfigurer?
2. Why is ReturnListener.handleReturn(..) not executed? ChannelListener.onCreate(...) is executed.
3. Checking missing exchange exception in cachingConnectionFactory.setCloseExceptionLogger and doing System.exit(1) there seems wrong to me, isn't it? But this is all I managed to do by now. I want to application not start when there is no exchange during binding creation. When I throw exception there application still starts. ReturnListener.handleReturn seems a better place for it, but it isn't executed when configured as below.
4. How can I stop Spring Application Context gracefully instead of System.exit(1)? - throwing exception doesn't stop Application Context. How to make RabbitMq fail to start in such situation? - when a creation of #Bean Binding at Spring Application Context start fails.
#Bean
MessageListenerContainer createQueueAMessageListener(SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory,
ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = rabbitListenerContainerFactory.createListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("queueA");
MessageConverter jsonMessageConverter = null;
container.setMessageListener(new MessageListenerAdapter(new Object(), jsonMessageConverter));
return container;
}
#Bean
MessageListenerContainer createQueueBMessageListener(SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory,
ConnectionFactory connectionFactory,
CachingConnectionFactory cachingConnectionFactory) {
// I think configuring CachingConnectionFactory here is a lame, isn't it? Of course connectionFactory is redundant now, I left it to show how was it done earlier.
// Where and how should I add listeners to CachingConnectionFactory?
cachingConnectionFactory.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
cachingConnectionFactory.setPublisherReturns(true);
cachingConnectionFactory.addChannelListener(new ChannelListener() {
#Override
public void onCreate(final Channel channel, final boolean transactional) {
log.info("channelListener onCreate - this is executed");
channel.addReturnListener(new ReturnListener() {
#Override
public void handleReturn(final int replyCode, final String replyText, final String exchange, final String routingKey,
final AMQP.BasicProperties properties,
final byte[] body) throws IOException
{
log.info("!!! Why is this not executed ?!!! handleReturn replyCode: " + replyCode + " replyText: " + replyText);
}
});
}
});
cachingConnectionFactory.addConnectionListener(new ConnectionListener() {
#Override
public void onCreate(final Connection connection) {
log.info("connectionListener onCreate - this is executed" + connection);
}
});
cachingConnectionFactory.setCloseExceptionLogger(new ConditionalExceptionLogger() {
#Override
public void log(final Log logger, final String message, final Throwable t) {
try {
logger.error(message + ": " + t.getMessage());
if (t.getMessage().contains("reply-code=404, reply-text=NOT_FOUND")) {
// throw new RuntimeException(); it doesn't stop Spring ApplicationContext from starting
log.error("Executing System.exit(1) command.");
// System.exit(1);
}
} catch (Exception e) {
log.error("err in listener ", e);
}
}
});
SimpleMessageListenerContainer container = rabbitListenerContainerFactory.createListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("queueB");
MessageConverter jsonMessageConverter = null;
container.setMessageListener(new MessageListenerAdapter(new Object(), jsonMessageConverter));
return container;
}
#Bean
MessageListenerContainer createQueueCMessageListener(SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory,
ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = rabbitListenerContainerFactory.createListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("queueC");
MessageConverter jsonMessageConverter = null;
container.setMessageListener(new MessageListenerAdapter(new Object(), jsonMessageConverter));
return container;
}
// I think configuring CachingConnectionFactory here is a lame, isn't it?
It is not "lame"; that is the normal way of configuring beans with additional properties not exposed directly by Boot.
It should be called; have you tried debugging?
Why don't you do what I advised here Make Spring RabbitMQ fail on missing exchange - it's much simpler.
close() it - but, since you are using Spring Boot, it will do that for you - it registers a JVM shutdown hook that closes the context.
EDIT
Binding to a non-existent exchange will fail; you just need to force it to happen before the application is fully initialized, e.g. in an ApplicationRunner.
#SpringBootApplication
public class So70212347Application {
public static void main(String[] args) {
SpringApplication.run(So70212347Application.class, args);
}
#Bean
Binding binding() {
return new Binding("foo", DestinationType.QUEUE, "doesn't exist", "foo", null);
}
#Bean
Queue queue() {
return new Queue("foo");
}
#Bean
ApplicationRunner runner(ConnectionFactory cf) {
return args -> {
cf.createConnection().close();
};
}
}
Created new connection: rabbitConnectionFactory#6a0cbc6f:0/SimpleConnection#6cd164a6 [delegate=amqp://guest#127.0.0.1:5672/, localPort= 62884]
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
Application run failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:748)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:309)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1290)
at com.example.demo.So70212347Application.main(So70212347Application.java:16)
Caused by: org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2192)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2138)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2118)
at org.springframework.amqp.rabbit.core.RabbitAdmin.initialize(RabbitAdmin.java:691)
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$null$10(RabbitAdmin.java:619)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:209)
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$afterPropertiesSet$11(RabbitAdmin.java:618)
at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.lambda$onCreate$0(CompositeConnectionListener.java:38)
at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807)
at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.onCreate(CompositeConnectionListener.java:38)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:730)
at com.example.demo.So70212347Application.lambda$0(So70212347Application.java:33)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:758)
... 5 common frames omitted
Caused by: java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147)
at com.rabbitmq.client.impl.ChannelN.queueBind(ChannelN.java:1077)
at com.rabbitmq.client.impl.ChannelN.queueBind(ChannelN.java:46)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1157)
at com.sun.proxy.$Proxy47.queueBind(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareBindings(RabbitAdmin.java:870)
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$initialize$12(RabbitAdmin.java:694)
at org.springframework.amqp.rabbit.core.RabbitTemplate.invokeAction(RabbitTemplate.java:2227)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2186)
... 18 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'doesn't exist' in vhost '/', class-id=50, method-id=20)
...

RabbitMQ delayed message plugin - How to show delayed message in admin UI?

We use the rabbitmq message delay plugin (rabbitmq_delayed_message_exchange) for delaying messages. Is it possible for debugging and monitoring purposes, to show holded / delayed messages in rabbitmq admin web interface?
Link: https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/
Bye,
Ben
No; delayed messages are not visible in the admin UI.
As an alternative you can route the messages to a real queue, with a TTL defined as well as dead lettering which will cause expired message to be routed to the final queue.
You can set a fixed TTL on the temporary queue or use the expiration property on individual messages.
EDIT
#SpringBootApplication
public class So50760600Application {
public static void main(String[] args) {
SpringApplication.run(So50760600Application.class, args);
}
#Bean
public ApplicationRunner runner(RabbitTemplate template) {
return args -> template.convertAndSend("", "temp", "foo", m -> {
m.getMessageProperties().setExpiration("5000");
return m;
});
}
#RabbitListener(queues = "final")
public void in(String in, #Header("x-death") List<?> death) {
System.out.println(in + ", x-death:" + death);
}
#Bean
public Queue temp() {
Map<String, Object> args = new HashMap<>();
args.put("x-message-ttl", 10000); // default (max)
args.put("x-dead-letter-exchange", "");
args.put("x-dead-letter-routing-key", "final");
return new Queue("temp", true, false, false, args);
}
#Bean
public Queue finalQ() {
return new Queue("final");
}
}
and
foo:[{reason=expired, original-expiration=5000, count=1, exchange=, time=Fri Jun 08 10:43:42 EDT 2018, routing-keys=[temp], queue=temp}]

Timeout of basicPublish when server is outofspace

My case is rabbitmq server got out of space, just as below
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ramonubuntu--vg-root 6299376 5956336 0 100% /
The producer publishes message to server(the message needs to be persisted), and then will be blocked forever, it will keeping waiting the response of publishing. Sure we should avoid the situation of server out of space, but is there any timeout mechanism to let producer quit the waiting?
I have tried heartbeat and SO_TIMEOUT, they both don't work, as the network works fine. Below is my producer.
protected void publish(byte[] message) throws Exception {
// ConnectionFactory can be reused between threads.
ConnectionFactory factory = new SoTimeoutConnectionFactory();
factory.setHost(this.getHost());
factory.setVirtualHost("te");
factory.setPort(5672);
factory.setUsername("amqp");
factory.setPassword("amqp");
factory.setConnectionTimeout(10 * 1000);
// doesn't help if server got out of space
factory.setRequestedHeartbeat(1);
final Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
// declare a 'topic' type of exchange
channel.exchangeDeclare(this.exchangeName, "topic", true);
channel.addReturnListener(new ReturnListener() {
#Override
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey,
AMQP.BasicProperties properties, byte[] body) throws IOException {
logger.warn("[X]Returned message(replyCode:" + replyCode + ",replyText:" + replyText
+ ",exchange:" + exchange + ",routingKey:" + routingKey + ",body:" + new String(body));
}
});
channel.confirmSelect();
channel.addConfirmListener(new ConfirmListener() {
#Override
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
logger.info("Ack: " + deliveryTag);
// RabbitMessagePublishMain.this.release(connection);
}
#Override
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
logger.info("Nack: " + deliveryTag);
// RabbitMessagePublishMain.this.release(connection);
}
});
channel.basicPublish(this.exchangeName, RabbitMessageConsumerMain.EXCHANGE_NAME + ".-1", true,
MessageProperties.PERSISTENT_BASIC, message);
channel.waitForConfirmsOrDie(10*1000);
// now we can close connection
connection.close();
}
It will block at 'channel.waitForConfirmsOrDie(10*1000);', and the SotimeoutConnectionFactory,
public class SoTimeoutConnectionFactory extends ConnectionFactory {
#Override
protected void configureSocket(Socket socket) throws IOException {
super.configureSocket(socket);
socket.setSoTimeout(10 * 1000);
}
}
Also I captured the network between producer and rabbimq,
Please help.
You need to implement Connection Block/Unblocked.
This is basically a way of notifying the publisher that the server is running out of resources. The advantage with this is that the publisher will also be notified once it is safe to publish again.
I would recommend that you take a look at this article. A simple way of implementing this is to have a flag that indicates if it is safe to publish, if it is not wait until it is.
As an example you can take a look on how I implemented this in one of my Python examples.