I have implemented RabbitMQ and stuck in one place where it seems as if my sender is not able to send any data to the queue.
My Producer class:
#Service
public class MessageSender {
#Autowired
private AmqpTemplate template;
public void send(String text) {
template.convertAndSend(text);
}
}
my spring config file looks like this:
<rabbit:connection-factory id="connectionFactory"
addresses="${connectionFactory.addresses}" channel-cache-size="${connectionFactory.channel-cache-size}"
virtual-host="${connectionFactory.vhost}" username="${connectionFactory.user}"
password="${connectionFactory.password}" />
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" queue="myQueue" exchange="myExchange" routing-key="dummyKey"/>
<rabbit:queue name="myQueue" />
<rabbit:direct-exchange name="myExchange">
<rabbit:bindings>
<rabbit:binding queue="myQueue" />
</rabbit:bindings>
</rabbit:direct-exchange>
<rabbit:listener-container connection-factory="connectionFactory">
<rabbit:listener ref="messageHandler" method="onMessage" queue-names="myQueue" />
</rabbit:listener-container>
<rabbit:admin connection-factory="connectionFactory" />
<bean id="messageHandler" class="com.tm.email.sender.spring.MessageHandler" />
I cannot find out what the issue is.
The below mentioned thing works perfectly. I can easily push the messages on to the queue and then my sender class works perfectly fine.
public class Send {
private final static String QUEUE_NAME = "myQueue";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
int i = 0;
while(i < 10000){
HashMap message = new HashMap();
message.put("message number", i);
channel.basicPublish("", QUEUE_NAME, null, SerializationUtils.serialize(message));
i++;
}
channel.close();
connection.close();
}
}
You have default routing-key="dummyKey" on the <rabbit:template> definition. That means that your template.convertAndSend(text); will send a Message to the myExchange through the dummyKey routing-key.
Since you don't provide the binding for that myQueue via dummyKey you messages just lost.
From the raw AMQP code you just send message to the default Exchange ("") using a default routing-key for each queue - its name.
For the listener it doesn't matter from where and how the message has appeared in the queue.
That's why the second 'send' variant works well.
So, to fix your use-case with AmqpTemplate, you have to add this:
<rabbit:binding queue="myQueue" key="dummyKey"/>
Related
Does ActiveMQ cluster (master-slave) offer high availability on failover mode?
I am publishing messages on a topic but when I kill active node and my consumer connect to the other node It loses several messages.
I need only topic mode, because I need to have different pods consuming messages at the same time. We have made test with durable messages and subscriptions wit we got the same problem.
If ActiveMQ doesn't support this functionality, is there other broker that could do this?
We have used Spring boot to develop sender and consumer apps.
SENDER:
To send messages we used and API REST that sends 5000 messages to a topic. If some convertAndSend fails (ActiveMQ node 1 killed) we retry it until the failover.
This is my sender class:
#RestController
public class RestApiController {
#Autowired
private JmsTemplate jmsTemplate;
#RequestMapping(value = "/produce")
public String produce() {
String result = "Done";
for (int i = 0; i < 5000; i++) {
boolean repetir = true;
while (repetir) {
try {
jmsTemplate.convertAndSend("TestTopic", "Message " + i);
repetir = false;
result = "Done";
} catch (JmsException e) {
e.printStackTrace();
result = "ERROR";
}
}
}
return result;
}
}
This is de sender's application.properties:
spring.activemq.broker-url=failover:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)?initialReconnectDelay=1&backup=true spring.activemq.user=admin
spring.activemq.password=admin
spring.jms.pub-sub-domain=true
server.port=8081
CONSUMER:
This is my listener class:
#Service
public class ContactTransactionReceiver {
#JmsListener(destination = "TestTopic")
public void receiveMessageSendMessage(Message message) throws Exception {
System.out.println(((TextMessage) message).getText());
}
}
This is the consumer's application.properties:
spring.activemq.broker-url=failover:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)?initialReconnectDelay=1&backup=true
spring.activemq.user=admin
spring.activemq.password=admin
spring.jms.pub-sub-domain=true
server.port=8082
ACTIVEMQ NODE 1
We have included this configuration in activemq.xml for HA, that refers to node2:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="static:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)" />
</networkConnectors>
We have proved too master-slave:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="masterslave:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)" />
</networkConnectors>
ACTIVEMQ NODE 2
We have included this configuration in activemq.xml for HA, that refers to node2:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="static:(tcp://172.18.13.45:61626,tcp://172.18.13.45:61616)" />
</networkConnectors>
We have proved too master-slave:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="masterslave:(tcp://172.18.13.45:61626,tcp://172.18.13.45:61616)" />
</networkConnectors>
You can find the full code and ActiveMQ configuration files in:
https://github.com/PedroRamirezTOR/ActiveMQ-HA-Sender-Consumer.git
Thanks in advance!
ActiveMQ 5.x does support high availability via a master-slave configuration. See the documentation for more details.
I noticed in your xmls that both master & slave are referring to a local KashaDB. I believe to have true HA Failover, you need to reference a shared network drive.
http://activemq.apache.org/shared-file-system-master-slave
<persistenceAdapter>
<kahaDB directory="/sharedFileSystem/sharedBrokerData"/>
</persistenceAdapter>
We are trying to run JMS application using ActiveMQ, we are facing a problem with the message receiver. Please find below process used by the application and test.
A scheduler based JMS application starts
A persistent activemq test message is placed into a queue
The scheduled event tries to receive the message from the queue(this is not a consumer but it is just a receiver jmsTemplate.receive(queue))
Even though the message is there in the queue which can be browsed using activemq console, the application doesn't see it and says no message received in the queue. This issue happening even for a single input message, I have tried all the config options suggested over the support sites but nothing works. This is how my activemq.xml broker tag is now:
<?xml version="1.0" encoding="UTF-8"?>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" persistent="true" dataDirectory="${activemq.data}" schedulePeriodForDestinationPurge="10000" deleteAllMessagesOnStartup="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" maxPageSize="5000" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000" />
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false" />
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb" />
</persistenceAdapter>
</broker>
Can someone please suggest anything that I am missing?
Edit:
Here is the receiver code
private javax.jms.Message getMessageFromQueue(String jmsCorrelationID, String queueName)
throws JMSException, EdiRoutingException, DcQueueNameMapperException {
String messageSelector = "JMSCorrelationID = '" + jmsCorrelationID + "'";
javax.jms.Message message = jmsTemplate.receiveSelected(queueName, messageSelector);
if(null != message) {
TextMessage textMessage = (TextMessage) message;
auditMessage("DATA", textMessage.getText());
}
return message;
}
Here is the producer code:
public void sendMessage(String queueName, final String jmsCorrelationId, final String message) {
jmsTemplate.send(queueName, session -> {
TextMessage textMessage = session.createTextMessage(message);
if (StringUtils.isNotBlank(jmsCorrelationId)) {
textMessage.setJMSCorrelationID(jmsCorrelationId);
}
return textMessage;
});
}
I am trying to implement a module to send messages from a CXF client to a server (SOAP endpoint) using HTTPS. I am able to achieve this by following the guide here: https://camel.apache.org/how-to-switch-the-cxf-consumer-between-http-and-https-without-touching-the-spring-configuration.html
The following configuration is key:
<ctx:property-placeholder location="classpath:orderEntry.cfg" />
<!-- other properties -->
<http:conduit name="{http://www.company.com/product/orderEntry/service/1}OrderEntry.http-conduit">
<http:tlsClientParameters disableCNCheck="true">
<sec:trustManagers>
<sec:keyStore type="JKS" password="${trustStore.password}" file="${trustStore.file}"/>
</sec:trustManagers>
<!-- other config -->
</http:tlsClientParameters>
</http:conduit>
The above configuration refers to a config file that has these properties stored:
orderEntry.cfg
--------------
endpointUri=https://localhost:8181/OrderEntry
trustStore.password=password
trustStore.file=etc/myApp.ts
As noted earlier, I am able to send messages via https when I follow the guide.
But I am concerned about the password being stored in plain text here. Is there a way that I can have the password wired from Java code (which can probably read the password from an encrypted source) and provide it to the http conduit when it needs it?
Have you tried location attribute value with file prefix?
E.g. location="file:/my/custom/location/orderEntry.cfg"
See: https://stackoverflow.com/a/17303537
Update:
If it works with your custom bean, you can try create trust managers as a bean and inject it into the conduit configuration like bellow:
blueprint.xml
<bean id="serviceTrustManager"
class="my.app.security.KeyStores" factory-method="loadTrustManagers">
<argument index="0" value="${my.app.service.trustStorePath}"/>
<argument index="1" value="${my.app.service.trustStoreEncryptedPassword}"/>
</bean>
<http:conduit name="{http://www.company.com/product/orderEntry/service/1}OrderEntry.http-conduit">
<http:tlsClientParameters disableCNCheck="true">
<sec:trustManagers ref="serviceTrustManager"/>
</http:tlsClientParameters>
</http:conduit>
Java code:
public class KeyStores {
public static TrustManager[] loadTrustManagers(String trustStorePath, String trustStoreEncryptedPassword) {
String trustStoreDecryptedPassword = PasswordDescriptor.decryptPassword(trustStoreEncryptedPassword); //Password decryption logic here
KeyStore trustStore = KeyStores.loadKeyStore("JKS", trustStorePath, trustStoreDecryptedPassword); //IO logic here
TrustManagerFactory trustFactory;
try {
trustFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustFactory.init(trustStore);
} catch (NoSuchAlgorithmException | KeyStoreException ex) {
throw new IllegalStateException(ex);
}
return trustFactory.getTrustManagers();
}
}
I have super simple scenario: one broker and one consumer with durable subscription.
This is the code of my consumer app:
package test;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.jms.Topic;
import org.apache.activemq.ActiveMQConnectionFactory;
import pojo.Event;
import pojo.StockUpdate;
public class Consumer
{
private static transient ConnectionFactory factory;
private transient Connection connection;
private transient Session session;
public static int counter = 0;
public Consumer(String brokerURL) throws JMSException
{
factory = new ActiveMQConnectionFactory(brokerURL);
connection = factory.createConnection();
connection.setClientID("CLUSTER_CLIENT_1");
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
}
public void close() throws JMSException
{
if (connection != null)
{
connection.close();
}
}
public static void main(String[] args) throws JMSException
{
try
{
// extract topics from the rest of arguments
String[] topics = new String[2];
topics[0] = "CSCO";
topics[1] = "ORCL";
// define connection URI
Consumer consumer = new Consumer("failover:(tcp://localhost:61616)?maxReconnectAttempts=-1&useExponentialBackOff=true");
for (String stock : topics)
{
try
{
Destination destination = consumer.getSession().createTopic("STOCKS." + stock);
// consumer.getSession().
MessageConsumer messageConsumer = consumer.getSession().createDurableSubscriber((Topic) destination, "STOCKS_DURABLE_CONSUMER_" + stock);
messageConsumer.setMessageListener(new Listener());
}
catch (JMSException e)
{
e.printStackTrace();
}
}
}
catch (Throwable t)
{
t.printStackTrace();
}
}
public Session getSession()
{
return session;
}
}
class Listener implements MessageListener
{
public void onMessage(Message message)
{
try
{
TextMessage textMessage = (TextMessage) message;
String json = textMessage.getText();
Event event = StockUpdate.fromJSON(json, StockUpdate.class);
System.out.println("Consumed message #:" + ++Consumer.counter + "\n" + event);
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Here is my activemq.xml
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="R6_cluster_broker1" persistent="true">
<networkConnectors>
<networkConnector uri="static:(failover:(tcp://remote_master:61616,tcp://remote_slave:61617))"/>
</networkConnectors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="/work/temp/kahadb"/>
</persistenceAdapter>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<!-- <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/> -->
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file
Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
When I have both broker and consumer running and then stop the broker my consumer exits few moments after. As far I understood it must attempt to reconnect, but it is not the case. What am I doing wrong, please advise.
!NOTE! I launch my consumer in Eclipse, i do not build a standalone jar for this task.
I have updated my broker to the latest 5.9.1 and did the same to my consumer. Result is the same - after I stop the broker my consumer dies few seconds after. It works fine if broker is up and running.
For anyone who can't setup the failover via the URI/URL parameter in the ConnectionFactory ,because of schema not found in ARTEMIS :
As far as I know in the ActiveMQ the URL is the following:
failover:(tcp://localhost:61616,tcp://localhost:51516)?randomize=false
But in Artemis the above will fail ,because of schema not found so remove just the failover: prefix:
(tcp://localhost:61616,tcp://localhost:51516)?randomize=false
Alright, the problem was actually in my code: there was nothing that would prevent main thread from exiting. Since thread that implements failover is a daemon thread, consumer app terminated right after there was nothing to hold on to (no non-daemon threads).
Most likely you are using a version of ActiveMQ that has a bug which causes there to be all daemon threads which means there's nothing to keep the client running. Upgrade to a later version such as v5.9.1 and see if that helps. If not post more information as you haven't really provided much.
I use mule to process the files from sftp, but I need to process the file in order(the sequence is order by name of file).
For example:
there are 3 files in sftp:
1.txt,2.txt,3.txt
I need to process these three files one by one and have to in the order of file's name.(first to process 1.txt then 2.txt at last 3.txt)
I use service-overrides in connector and extend the SftpMessageReceiver and override the poll() method ,in the method I make these files's name by order. But it doesn't work.Who can tell me where is wrong.
the class I write
public class NewSftpMessageReceiver extends SftpMessageReceiver {
private SftpReceiverRequesterUtil sftpRRUtil = null;
public NewSftpMessageReceiver(SftpConnector connector,
FlowConstruct flow,
InboundEndpoint endpoint,
long frequency) throws
CreateException {
super(connector, flow, endpoint);
this.setFrequency(frequency);
sftpRRUtil = new SftpReceiverRequesterUtil(endpoint);
}
public NewSftpMessageReceiver(SftpConnector connector, FlowConstruct flow, InboundEndpoint endpoint)
throws
CreateException {
super(connector, flow, endpoint);
sftpRRUtil = new SftpReceiverRequesterUtil(endpoint);
}
#Override
public void poll() throws
Exception {
try {
String[] files = sftpRRUtil.getAvailableFiles(false);
//do something here.....
for (String file : files) {
if (getLifecycleState().isStopping()) {
break;
}
routeFile(file);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
the config file like below
<sftp:connector name="incoming-mandate-connector"
archiveDir="${esb.archive.path}/mandate/incoming"
useTempFileTimestampSuffix="true"
autoDelete="true"
identityFile="${esb.incoming.sftp.identityFile}"
passphrase="${esb.incoming.sftp.passphrase}"
sizeCheckWaitTime="${esb.incoming.sftp.sizeCheckWaitTime}">
<service-overrides messageReceiver="com.xxy.NewSftpMessageReceiver"/>
</sftp:connector>
<sftp:endpoint name="incoming-mandate-ep"
address="${esb.incoming.connector}"
connector-ref="incoming-mandate-connector"
keepFileOnError="true" encoding="${esb.encoding}">
<file:filename-wildcard-filter pattern="${esb.incoming.filePattern}"/>
</sftp:endpoint>
the flow like blew
<flow name="process">
<quartz:inbound-endpoint name="quartz-endpoint" cronExpression="${esb.cronExpression}" jobName="incoming-mandate-job">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="incoming-mandate-ep"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<processor ref="mandate-processor"/>
<transformer ref="mandate-transformer"/>
<mulexml:jaxb-object-to-xml-transformer jaxbContext-ref="jaxb" name="mandate-jaxb-transformer"/>
<byte-array-to-string-transformer mimeType="application/xml"/>
<message-properties-transformer scope="outbound">
<add-message-property key="${esb.username}" value="${esb.password}"/>
</message-properties-transformer>
<outbound-endpoint ref= "mandate-service-ep"/>
</flow>
The reason why your message receiver is not being called is because you have a Quartz endpoint that calls the SFTP transport, so in fact you are using a message requester, not a receiver.
You would need to create a custom message requester instead of a receiver and also a custom requester factory (that creates instances of the requester)
You can configure it as follows:
<sftp:connector name="sftpConnector">
<spring:property name="serviceOverrides">
<spring:map>
<spring:entry key="requester.factory" value="your.package.YourCustomSftpRequesterFactory"/>
</spring:map>
</spring:property>
</ftp:connector>