In ActiveMQ, is there a way to send a copy of message from one queue to another queue in a remote broker? - activemq

Problem statement: There are two Queues in two different brokers. Each Queue has one Consumer to it. The producer is dropping messages on the first Queue. We would want to send a copy of message to the second Queue. For visualization
                   Producer
                         |
Broker1 --> Queue1 --> Consumer1
                       | (copy)
Broker2 --> Queue2 --> Consumer2 (consumes same message as Consumer1 but is independent of Consumer1)
The ask is
Only 1 queue is created in each broker. I have achieved the above with 4 Queues but looking for more optimized solution.
Prefer no topics to be used.
To be done only through activemq provided configuration.
What have I done till now:
I managed to do the above with 4 queues.
In Broker1, Queue1 forwarding a copy to a Virtual Destination Queue. Also, sending the messages in Virtual Destination to broker 2 through network connector.
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="Queue1" forwardOnly="false">
<forwardTo>
<queue physicalName="IntermediateQueue"/>
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
<networkConnectors>
<networkConnector
name="Q:broker1->broker2"
uri="static:(tcp://localhost:31616)"
duplex="false"
staticBridge="true">
<staticallyIncludedDestinations>
<queue physicalName="IntermediateQueue"/>
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
In Broker2, forwarding all messages received in the intermediate Queue to the actual destination queue.
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="IntermediateQueue">
<forwardTo>
<queue physicalName="FinalDestinationQueue" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Appreciate any help, as going through activemq documentation and forums didn't yield an optimized answer to this problem.

You are essentially re-creating pub+sub and then adding in a transmission-queue pattern for multi-broker integration. There are valid use cases to do this and your approach is valid and within the intended design of Composite Destinations and Network Connectors. The trade-off in this approach is the heavy administration and configuration management that is required.
I understand you prefer to not use topics. However, you may consider looking at Virtual Topics1 which solve this problem in an elegant way and allows you to add new consumers dynamically and without having to modify the broker configuration.
Producer send to Topic:
topic://VT.ORDER.EVENT
Consumer(s) read from special named Queues
clientA: queue://VQ.CLIENTA.VT.ORDER.EVENT
clientB: queue://VQ.CLIENTB.VT.ORDER.EVENT
ref: Virtual Topics

Related

explicit configuration of durable topic

It seems quite hard to find a way to configure explicitly activeMQ topics
Is there a cofn file and a reference page to make configuration explicitly
The idea is to declare new durable topics and consumers (with static Id)
any advice ?
As the ActiveMQ documentation states, you can configure destinations in the XML configuration (activemq.xml), .e.g:
<broker xmlns="http://activemq.apache.org/schema/core">
<destinations>
<queue physicalName="FOO.BAR" />
<topic physicalName="SOME.TOPIC" />
</destinations>
</broker>

ActiveMQ doesn't propagate topics on failover reconnection

I'm attempting to use a network of brokers that bridges two LANs over a duplex WAN connector:
There are actually many subscribers in our setup, each connecting to a different "Broker A", if that makes sense. All of the Broker A instances have their own connections to a single "Broker B".
Software and configurations:
ActiveMQ 5.14.0, Java 8
All brokers have non-persistent topics only; advisory messages are on.
OS: Linux (RHEL 6)
When I initially bring everything online, regardless of the order in which I bring things online, communication between the publisher and subscriber works flawlessly. I've had the system up-and-running for weeks at a time without issue.
What I've observed is that if broker C is restarted, no new topics that show up in broker B ever appear in broker C. New topics are still appearing in broker B as they are created by the subscriber(s). Neither existing nor new topics ever propagate across the WAN to broker C. I've verified this using jconsole.
If I restart broker B, the problem goes away immediately. The topics contained in broker B (according to jconsole) are the same as they were prior to restart, but now they've magically appeared in C.
Brokers B and C have the same configuration (shown below). The only difference is that B creates a duplex network connector to C created using the following code:
final NetworkConnector wanNC = new DiscoveryNetworkConnector(
new URI(String.format("static:(failover:(tcp://%s:%d))", parentNode, port)));
wanNC.setCheckDuplicateMessagesOnDuplex(true);
wanNC.setDecreaseNetworkConsumerPriority(true);
wanNC.setDuplex(true);
wanNC.setName(NetworkUtils.getHostName());
wanNC.setNetworkTTL(10);
wanNC.setSuppressDuplicateTopicSubscriptions(false);
broker.addNetworkConnector(wanNC);
broker.xml:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" />
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="${broker.id}" start="false"
offlineDurableSubscriberTimeout="5000" offlineDurableSubscriberTaskSchedule="5000"
persistent="false" useJmx="true" schedulePeriodForDestinationPurge="86400000">
[...]
<networkConnectors>
<networkConnector name="z-broker-${broker.id}-x-nc"
decreaseNetworkConsumerPriority="true"
networkTTL="10"
uri="multicast://225.5.5.5:6555?group=TO_X">
<excludedDestinations>
<topic physicalName="X.A" />
</excludedDestinations>
</networkConnector>
<networkConnector name="z-broker-${broker.id}-y-nc"
decreaseNetworkConsumerPriority="true"
networkTTL="10"
uri="multicast://225.5.5.5:6555?group=TO_Y">
<excludedDestinations>
<topic physicalName="X.B.>" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://${broker.ip}:${broker.port}?maximumConnections=1000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://225.5.5.5:6555?group=TO_Z" />
</transportConnectors>
</broker>
</beans>
Why don't topics from broker B (existing or new) ever show up in broker C?
Why does restarting broker B solve the issue immediately?
Apparently the trick was changing the network connector URI from
static:(failover:(tcp://<ip>:<port>))
to
static:(tcp://<ip>:<port>)
I didn't need failover transport for any reason since the connection is intended as a network bridge and there's a single remote address.
For whatever reason, using failover prevented topics from propagating on reconnect.

ActiveMQ messages getting stuck in networked broker setup

I have two ActiveMQ brokers in a network setup. The clients are configured with randomize=true and are able to connect fine. However, the messages do not get forwarded from one broker to the other and remain in the queue. For example, I have a particular queue which has multiple producers and one consumer. If I look at the queue on the broker to which the one consumer is connected to, all messages are dequeued immediately. However, on the other broker messages get queued and do not get drained.
Listed below are my networkConnectors and transportConnectors setup for the two brokers. I have tried adding duplex="true" as well as changing the networkTTL to 1 and those didn't seem to make any difference.
BrokerA:
<networkConnectors>
<networkConnector name="LocalBrokerToB"
networkTTL="2"
uri="static:(tcp://hostnameB:61617)"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="nioConnectorFront" uri="nio://hostnameA:61616?maximumConnections=1024 "/>
<transportConnector name="nioConnectorBack" uri="tcp://hostnameA:61617?maximumConnections=1024 "/>
</transportConnectors>
BrokerB:
<networkConnectors>
<networkConnector name="LocalBrokerToA"
networkTTL="2"
uri="static:(tcp://hostnameA:61617)"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="nioConnectorFront" uri="nio://hostnameB:61616?maximumConnections=1024 "/>
<transportConnector name="nioConnectorBack" uri="tcp://hostnameB:61617?maximumConnections=1024 "/>
</transportConnectors>
Any ideas on what could be the problem? An example configuration that someone has working would be a great help.
You should connect the networkConnector to the transport connector of the other broker. That is port 61616 in your example, not 61617.
You should verify in the broker logs or via Web Console / JMX that the network connection actually gets established.
Adding duplex="true" let's one of the brokers initiate the connection which is great in case of firewalls etc. In your case, that should not matter.

Maintain order of messages while forwarding messages between two ActiveMQ brokers

I have an ActiveMQ setup where a source broker living in one data center forwards all messages arriving on certain topics to a destination broker in another data center. The consumer application consumes messages only from the destination broker. (This setup is mainly to ensure fast and efficient forwarding of messages between the two data centers.)
The configuration for forwarding looks something like this:
<networkConnectors>
<networkConnector name="Q:DontForwardQueueMessages"
uri="static:(tcp://destination-broker.example.com:61616)"
duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector name="T:ForwardSampleMessages"
uri="static:(tcp://destination-broker.example.com:61616)"
duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
<staticallyIncludedDestinations>
<topic physicalName="SampleTopic1" />
<topic physicalName="SampleTopic2" />
<topic physicalName="SampleTopic3" />
<topic physicalName="SampleTopic4" />
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
Our application needs message order to be maintained. However, we are losing messages when the destination broker goes down. Messages arriving at the source broker pile up in the topic, but do not get forwarded when the connection with the destination broker is re-established. However, messages arriving after re-connection are forwarded as usual.
I'm looking for a way I can configure the setup so that:
All messages waiting at the source are sent as soon as the destination is re-connected, maintaining the correct order,
Messages arriving after re-connection wait for older messages to be forwarded before they are forwarded.
It looks like it was a poor design choice to have messages forwarded from a Topic. As per the ActiveMQ documentation:
Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message.
The destination broker acts like a subscriber to the source topic from which messages are being forwarded. So, when messages arrive in the source topic in the absence of a subscriber (destination disconnected), they are not available to anyone.
As a solution, I changed the design:
Remove the Virtual Destination configuration in the destination broker
Add the same Virtual Destination configuration in the source broker (so now messages are distributed into their respective queues right here)
Add networkConnector rules to the source broker to forward messages in these queues to corresponding queues on the destination broker.
Now since messages at the source are in a queue, they will be consumed in the order in which they were received, and no messages are lost, even if the brokers are disconnected from each other.

ActiveMQ Topic policyEntry

I'm only trying to use ActiveMQ queue mechanism and not topic. Does the below activemq.xml configuration force any clients to use a topic instead of a queue of can I ignore the policyEntry topic=">"? I'm seeing issues where multiple consumers sometimes (very small percentage) pick up the same message and process it. In the admin console it shows I have a queue. I do see that the default activemq.xml contains a policyEntry for both topic and queue. Its a bit odd that not all messages are consumed by multiple consumer threads if this config is indeed invalid for a queue based approach.
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" persistent="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
The line you pointed out only means that policy is defined for all topics,
In activemq '>' does the same thing as '*' does in other languages i.e it is default handler for all types.
You can remove that configuartion if you have doubts.
As your pending message stratergy is defined in the policyEntry it will be applicable to all topics,not queues.