I want to use Mass Transit with RabbitMQ and Kafka for message broker.
and in deployment phase can decide which one be used for transporting messages.
I know what's ConsumeContext,ISendEndpointProvider, or IPublishEndpoint. And according of the link this Interfaces shared between RabbitMQ and Kafka for publishing messages and consuming that.
I configured both RabbitMQ and Kafka in startup of my .Net Core App and by appSetting in deployment phase can decide which one must be register.
this approach is correct for having and configure both Kafka and RabbitMq in my app?
Using MassTransit, Kafka is a Rider which is configured along with a bus – and the bus still requires a supported transport. If you have RabbitMQ, that would be your bus transport, and the Kafka Rider would exist alongside the bus using the bus for publishing and sending messages.
With Kafka, messages can only be consumed or produced – they cannot be published, nor can they be sent. The ITopicProducer<T> interface is used to produce messages to Kafka topics.
Calling Publish or Send on the ConsumeContext will publish or send those messages to the rider's bus (which may be RabbitMQ, or any supported transport included the InMemory transport). Producing messages to Kafka topics must be done using the ITopicProducer<T> (which may be injected as a dependency to the consumer).
You can't "switch out" RabbitMQ with Kafka as the two services are very different.
Related
EDIT2: My issue here was caused by an insufficient understanding of how transport connectors work in ActiveMQ. TL;DR is that ActiveMQ will implicitly "transform" or "relay" messages between your transport connector configurations defined in activemq.xml.
EDIT: Additional info, the STOMP messages received by the Angular application are used for debugging and demo purposes. Hence, simply converting the OpenWire message to a blob of readable text is sufficient.
I'm creating an Angular application (preferably website, avoiding native applications), which objective is to "tap in" by web sockets on an ActiveMQ server and subscribe to OpenWire messages. How do I let ActiveMQ transform OpenWire messages to STOMP messages and send these to any clients (i.e. my Angular application) connected to the ActiveMQ WebSocket connector?
In addtiion, it would be nice-to-have if I could transform STOMP to OpenWire as well.
It must be Angular
Avoiding the use of native applications on the client-side is preferable although not a deal-breaker.
Adding extra processing stress on the ActiveMQ server must be done with caution.
To the best of my knowledge, it is only possible to let Angular "talk directly" with the ActiveMQ server by STOMP messages send by web socket, if I am to avoid using native applications.
I already have an Angular application capable of STOMP communication by web sockets (e.g. something like https://github.com/stomp-js/ng2-stompjs-angular7).
I am missing information on how to configure the ActiveMQ server to transform OpenWire-->STOMP through its transport connectors.
In my understanding, what I am trying to do should be possible. It is noted by other users but not how. E.g. users hint that what I want is possible in ActiveMQ but not Apollo: ActiveMQ to Apollo transition, Openwire to Stomp protocol configuration.
I expect (preferably) the need to use something like an ActiveMQ transformer (e.g. adding transformer to the connector configuration: AMQP & Openwire - Activemq broker and 2 different consumers) or maybe writing an ActiveMQ plugin (http://activemq.apache.org/developing-plugins.html). On ActiveMQ's website, an existing transformer is mentioned (http://activemq.apache.org/stomp.html Message Transformations section):
Currently, ActiveMQ comes with a transformer that can transform XML/JSON text to Java objects
... but no mention of how to use this and I am unsure if I can benefit from this and if this means that there are no transformers for OpenWire-->STOMP or vice versa.
I expect I might have misunderstood some of the concepts, and a "you're going in a wrong direction, do this instead" can work out as a good answer for me. At the time of writing, I expect I will have to create an ActiveMQ plugin using their Message Transformer interface (http://activemq.apache.org/message-transformation.html) although their sub links are 404. I hope to achieve a more simple solution, e.g. an existing OpenWire-->STOMP transformer:
<transportConnector name="openwire" uri="{some-openwire-uri}?transport.transformer=stomp"/>
ActiveMQ will "transform" any Openwire message into a STOMP message and vice versa as needed based on client connections. I an Openwire based JMS client connects and places a message onto a queue and a STOMP based client comes along and subscribes to that queue the message will be converted into a STOMP message to send to that client.
Without knowing more about what issue you are having it is hard to provide more insight than that though. There are some cases where the transformation from Openwire to STOMP might not yield exactly the right thing for you such as a MapMessage or StreamMessage and definitely an ObjectMessage so some care needs to be taken about cross protocol messaging.
You do of course need to add a transport connector for each of the protocols you want to support, Openwire, STOMP, AMQP etc. The clients need something to connect to, then once they connect the broker manages the message transformations amongst subscriptions on Topics and Queues.
We are working towards an architecture like one below but we will have micro services on cloud and some on premises which will talk to each other using queue(s) and bus(es),
Now I am confused with where we should host MassTransit and RabbitMq, also should it be a ASP.NET Core project on its own ? if yes what I will be doing in it ? starting a bus ? creating queues ? I am not able to move forward with this
The simple MassTransit examples are just that, the absolute simplest examples of interacting with queues.
RabbitMQ is your message broker. It is hosted separately.
MassTransit is a development framework that makes it much easier to interact with RabbitMQ (or Azure Service Bus) by abstracting away the implementation-specific "plumbing."
You write any number of .NET services that either publish messages to a queue, or subscribe to queues.
We recently worked on something similar, the way we did it is:
RabbitMQ was hosted separately, and buses/queues creation and management were done from the services that use messaging.
For each service that receives messages you use Maastransit to create a queue because service will be receiving messages using this queue.
You will be using publish/subscribe way of messaging so as mentioned above, inside each service, create a queue with logical name and connect to RabbitMQ server address.
Services that represent senders will publish messages of a custom type you create, and services that represent receivers will subscribe to this type of messages by having a consumer for this type registered inside the bus created.
Hope it helps.
Tech Stack:
.Net 4.6.1
Erlang 18.3
Language: C# 6
NServiceBus 6
RabbitMQ 3.6.3
Windows 7
Context:
We are trying to implement a dumb event publisher with smart subscription in NServiceBus on the RabbitMQ transport.
All interested receivers subscribe to an event.
A Publisher publishes the event to an event channel.
All subscribers receive a copy of the event.
Note that I did not say each TYPE of subscriber receives a copy of the message. If there are multiple instances of a service running, and they all have active subscriptions to an event, each INSTANCE of the subscribing service should get a copy of the message.
However, NServiceBus' notion of Publish-Subscribe delivers a published event to one and only one receiver on a given channel. In our case, one instance of a given subscribed service.
I hesitate to list messaging "patterns" because they don't seem to be named or described particularly consistently. However, I believe we are esentially trying to implement the "Multicast" version of the Publish-Subscribe Channel messaging pattern from the Enterprise Integration Patterns (Hohpe and Woolf) Book.
Problem Statement:
Our business case is this:
We have a configuration service that provides application configuration to all other services in our broader application.
Each service requests its configuration from the configuration service on startup by sending a ConfigurationRequest message.
The configuration services replies to the specific instance that made the ConfigurationRequest with a ConfigurationResponse. This is done using NServiceBus' Full-Duplex (commonly called Request-Response) feature.
There is a website that can modify configuration globally. When it does so, it notifies the configuration service with an UpdateConfiguration command.
The configuration service publishes a ConfigurationUpdated event that all other services are subscribed to.
Each service can have multiple instances running on more than one server. ALL instances of a service need to update their configuration, not just one instance.
Each service instance is calling NServiceBus' EndpointConfiguration.MakeInstanceUniquelyIdentifiable with a distinct discriminator.
Currently when we publish the ConfigurationUpdated event, only one instance of each service type gets the message. The events are distributed round-robin, instead of each instance getting a copy of the message.
We have worked around this problem by keeping a record of running services instances and sending the ConfigurationUpdated event (as a Command in NServiceBus) to each of them individually, but Pub-Sub implies that we should have dumb publishers and smart subscribers, and our workaround is the opposite... Our publisher looks up a listof each subscriber and sends to it implicitly. Is there some configuration I am missing on the subscriber side that would allow each instance of each service to get a copy of the published ConfigurationUpdated event? If not, where should I look in NServiceBus to start implementing such a feature? Routing topology, perhaps?
i'm not familiar enough with nservicebus to know how to do it with that toolset, but the RabbitMQ implementation would be a "fanout" exchange with a queue per consumer.
Every queue bound to a fanout exchange will get a copy of the message. If there is a single consumer for each of those queues, then you will be sending a copy of the message to each of the consumers.
It sounds like you have multiple consumers connected to the same queue. Maybe there's a way to tell nservicebus to create a queue for each consumer instance?
p.s. you're right about pub-sub pattern being multicast. i talk about this in my RMQ Patterns ebook (https://leanpub.com/rmq-patterns) and that's what the EIP book would say, as well.
I am using RabbitMQ as a Stomp broker for Spring Websocket application. The client uses SockJS library to connect to the websocket interface.
Every queue created on the RabbitMQ by the Spring is durable while topics are non durable. Is there any way to make the queues non durable as well?
I do not think I can configure on the application side. I played a bit with RabbitMQ configuration but could not set it up either.
Example destination on RabbitMQ used for SUBSCRIBE and SEND:
services-user-_385b304f-7a8f-4cf4-a0f1-d6ceed6b8c92
It will be possible to specify properties for endpoints as of RabbitMQ 3.6.0 according to comment in RabbitMQ issues - https://github.com/rabbitmq/rabbitmq-stomp/issues/24#issuecomment-137896165:
as of 3.6.0, it will be possible to explicitly define properties for endpoints such as /topic/ and /queue using subscription headers: durable, auto-delete, and exclusive, respectively.
As a workaround you can try to create queues by your own using AMQP protocol and then refer to that queues from STOMP protocol.
we are using ActiveMQ for message Queuing with openwire transport.In this context there will be one producer and one consumer with a message listener registered. We heard about MQTT protocol and its support in activeMQ. But i saw examples only for Publisher/subscriber semantics , where subscriber need to call receive method explicitly to get the published message. Can I use mqtt with Producer/Consumer envirnment. Please give a sample..
The MQTT protocol is based a publish / subscribe based model, it has no queuing semantics built into the protocol. If you need Queue's then you need to stick to openwire clients or use a STOMP based client which supports both Topics and Queues.