Proxying internal traffic to Azure Service Bus - rabbitmq

What I'm trying to achieve is interoperability between RabbitMQ clients in an internal network, and Azure Service Bus consumers running in Azure.
The RabbitMQ clients need to publish and subscribe, as do the Azure Service Bus consumers - so I need some kind of 'bi-directional proxy'. A diagram of what I'm trying to achieve:
+
Internal network | Azure
|
|
+--------+ | +----------+
| Client +---+ | +---+ Consumer |
+--------+ | | | +----------+
| | |
| +-----------------+ | +-------------------+ |
+-+ RabbitMQ Broker +---------+ Azure Service Bus +--+
| +-----------------+ | +-------------------+ |
| | |
+--------+ | | | +----------+
| Client +---+ | +---+ Consumer |
+--------+ | +----------+
|
|
|
+
AFAIK, both the RabbitMQ broker and Azure Service Bus can do AMQP 1.0. I've looked at the rabbit shovel plugin, but I think this would only handle publishing of messages from the internal clients to Azure, and wouldn't allow the clients to subscribe to messages published by the Azure consumers? Or have I got this wrong, and shovel will work for this?
If shovel won't work, is there any other way to achieve this

I got a response from a couple of Pivotal engineers, over on the rabbitmq-users group.
I was incorrect - shovel is actually bi-directional, acting as both publisher and consumer, so you can both send and receive messages from a remote broker.
I should then be able to fulfill my requirements by installing and configuring the shovel plugin in the internal RabbitMQ broker.
There are some gotchas when connecting to an Azure Service Bus broker though:
If you need to use a queue with sessions, you should use the GroupId property for the session ID (amqpMessage.Properties.GroupId = mySessionId)
If you need to use a queue with partitions, you should use the x-opt-partition-key annotation for the partition ID (amqpMessage.MessageAnnotations[new Symbol("x-opt-partition-key")] = myPartitionKey)

Related

ActiveMQ Network of Brokers pull message from broker 1

I am very new to ActiveMQ and I try to understand if I can achieve the following:
I have a producer which publishes messages to broker 1. In a secure network there is a broker 2 and the receiver. The connection between broker 1 and broker 2 can only be in the direction from broker 2 towards broker 1, because I can only communicate out of the secure network, but not in it.
All the documentation about network of brokers just show the way depicted in the following picture (found on https://dzone.com/articles/active-mq-network-brokers)
I need something similar, but the connection should be from broker-2 to broker-1.
Is something like this possible? Maybe with a polling from broker-2 to broker-1
Yes. Have Broker2 connect to Broker1 and establish a duplex connection
<networkConnector uri="static:(tcp://broker1:61616)" duplex="true" ... >
see: Network of Brokers

Spring JMS consumer does not have parent span Id rightly set

My sample application makes use of
spring-boot-starter-activemq (2.3.1)
spring-boot-starter-web (2.3.1)
pring-cloud-starter-sleuth (2.2.3)
On the published message I see the X-Parent-Span-Id, X-Trace-Id and b3 properties rightly set. However the logs in the JMS Listener (AMQ consumer) have a different parentId (X-B3-ParentSpanId).
Why do the logs on the AMQ consumer side not have the same parentId as present in the message?
Note - The traceId shows up fine.
Flow
Server 1 --> Server 2 (AMQ producer) ---> AMQ consumer
Server1 logs
{"#timestamp":"2020-07-26T18:42:19.258+01:00","#version":"1","message":"Received greet request","logger_name":"com.spike.server.HelloController","thread_name":"http-nio-8080-exec-9","level":"INFO","level_value":20000,"traceId":"e4ac9c00ba990cf2","spanId":"e4ac9c00ba990cf2","spanExportable":"true","X-Span-Export":"true","X-B3-SpanId":"e4ac9c00ba990cf2","X-B3-TraceId":"e4ac9c00ba990cf2"}
Server 2 (AMQ producer) logs
{"#timestamp":"2020-07-26T18:42:19.262+01:00","#version":"1","message":"Received audit request","logger_name":"com.spike.upstream.AuditController","thread_name":"http-nio-8081-exec-9","level":"INFO","level_value":20000,"traceId":"e4ac9c00ba990cf2","spanId":"aed4a3863c141dde","spanExportable":"true","X-Span-Export":"true","X-B3-SpanId":"aed4a3863c141dde","X-B3-ParentSpanId":"e4ac9c00ba990cf2","X-B3-TraceId":"e4ac9c00ba990cf2","parentId":"e4ac9c00ba990cf2"}
AMQ consumer logs
{"#timestamp":"2020-07-26T18:42:19.270+01:00","#version":"1","message":"Received message: hello world","logger_name":"com.spike.consumer.Consumer","thread_name":"DefaultMessageListenerContainer-1","level":"INFO","level_value":20000,"traceId":"e4ac9c00ba990cf2","spanId":"9f7928f65ee2479d","spanExportable":"true","X-Span-Export":"true","X-B3-SpanId":"9f7928f65ee2479d","X-B3-ParentSpanId":"70f25884c40b2dc7","X-B3-TraceId":"e4ac9c00ba990cf2","parentId":"70f25884c40b2dc7"}
I think I got what you're asking about. So the mismatch in parent Ids is related to the fact that between the consumer and the producer we introduce a "jms" service and if the broker was correlated you would have seen how long it takes exactly in the broker. Check out the following images in Imgur https://imgur.com/a/rpcFz4l

console producer is trying to connect non-leader and non-replica/isr node in Apache kafka

My kafka cluster is having 9 nodes .
Bcz I am trying to implement SSL, I am currently trying to set up SSL on only 1 node.
Now as I have done SSL setup for node with id 1001 and to test this, I created a topic 'SSLtest' which has leader as well as replica on 1001 only.
When I try to push data from console producer on same machine to the topic 'SSLtest' (i.e 1001), it tries to connect machine 1006 as log says :
Error when handling request {topics=[SSLtest],allow_auto_topic_creation=true} (kafka.server.KafkaApis)
kafka.common.BrokerEndPointNotAvailableException: End point with listener name SSL not found for broker 1006.
Why can't it push data to 1001 directly. What is it trying to do on 1006.

why each openstack nova services has 3 rabbitmq consumers associated with it

In openstack nova, different types of rabbitmq consumers ((ex. topic consumer, node_topic consumer, fanout consumer) are associated with each nova services (ex. nova-schedule, nova-compute etc) . Following lines is an Excerpt from https://github.com/openstack/nova/blob/master/nova/service.py which uses 3 consumers for each nova service.FYI: service.py is a wrapper for generating nova-services.
self.conn.create_consumer(self.topic, rpc_dispatcher, fanout=False)
node_topic = '%s.%s' % (self.topic, self.host)
self.conn.create_consumer(node_topic, rpc_dispatcher, fanout=False)
self.conn.create_consumer(self.topic, rpc_dispatcher, fanout=True)
I guess, each consumer is associated with a different Rabbit Queue. I think, we may need a node_topic consumer for sending messages directly to a node. what may be the purposes of two other consumers ( topic, & fanout consumer ) in each service?
Also, when I list exchanges of rabbitmq in my devstack node with the following command, it shows a whole lot of (11685) messages
stack#9734efd5-6fcd-4127-92b4-6715a66fda9d:/opt/stack$ sudo rabbitmqctl list_exchanges | wc -l
11685
can someone explain why are there so many exchanges in the rabbitmq of openstack devstack implemenation?

Pre-built AMQP and STOMP client (as in GUI client)

I'm looking for a client (as in GUI client, not client library) to play with our MQ server and familiarize myself with its semantics. Something that will send and receive messages at the press of a button (or a text command) and maybe even update me about the status of the server queues and messages. Administration would be a bonus. The UI doesn't have to be graphical (i.e. command line clients are fine).
The server will probably run RabbitMQ so anything RabbitMQ-specific is fine, as is ActiveMQ. But I'd rather have a generic AMQP or STOMP tool.
So, does anything of the sort exist?
I know some management and monitoring tools come with both server distributions, but no clients, right?
For Apache ActiveMQ, there is
the web admin console at http://localhost:8161/admin/
the ApacheActiveMQBrowser project on Sourceforge:
An open source project of developing
Message admin gui based tools for
Apache ActiveMQ.
HermesJMS, it does not mention ActiveMQ 5 (only 3 and 4) on the plugin page, but there is an active user forum
The rabbitmq-management plugin that comes with RabbitMQ (and enabled by rabbitmq-plugins enable rabbitmq_management) has a web-based interface that listens on the port 15672 and can do everything you are (I was) asking for.
Check out the BQL RabbitMQ plugin.
It gives you an SQL-style language for AMQP. For instance,
BQL> create exchange myexchange;
ok
BQL> create durable queue 'myqueue'
ok
BQL> select name,messages from queues where 'durable'=true order by name
----------------------
| name | messages |
----------------------
| myqueue | 0 |
Obviously, it's RabbitMQ specific.
If you're willing to do a bit of coding, you could take a look at the examples in the RabbitMQ Java and .NET clients:
Java examples
.NET examples
They're not quite graphical, but trying to understand them forces you to ask the right questions.
It's been a while, but I remember thinking that the best way to familiarize yourself with AMQP is to read the 0-9-1 spec and write some simple programs; in particular, the protocol documentation on that site gives a lot of examples.
Command line tools (written in C) to send and receive AMQP messages: http://github.com/rmt/amqptools