Does NServiceBus, at any point, for any reason, have to post empty messages to MSMQ, and if so, why and when does it happen? Longer explanation below.
A project I work on makes use of NServiceBus version 4. That version does not allow "multi-hosting" of event handlers for different queues in a single process, which may be inconvenient if your project contains 40 or so different queues.
To overcome this problem in development, I made a small "router" app, which listens to all the necessary MSMQ queues and simply forwards all messages from them into a single "unified" queue. That "unified" queue is specified as the queue name for the "unified endpoint" process, which references all the handlers for all the messages that would normally be handled from those various queues.
The setup kind-of works, it seems (with most handlers, at least), but there is one mysterious behaviour (which, I presume, may have something to do with the set-up not working with some other handlers). Namely, as soon as the project starts up, my "router" immediately discovers a number of empty MSMQ messages posted to the queues it has to listen to. Apparently, NSB is publishing those messages during start-up for some reason (and most probably the router is snitching them up before NSB has the chance to look at them again).
I am sure this is not an artefact of my implementation as this does not happen unless NSB is also started. I am curious about the reasons.
NServiceBus, by default, autosubscribes to all handled events if it knows the endpoint which publishes them. These empty messages you see might be the subscribe messages because they are being sent during the endpoint start-up phase.
The mechanics behind the subscribe messages are documented here. TL;DR for transports that do not provide publishing natively (e.g. MSMQ, Azure Storage Queues) NServiceBus emulates it using subscribe messages and internal subscription lists (storages).
You can verify this by checking the message intent header. If they are not subscribe messages, please share the complete list of headers of such a message for further investigation.
Related
I use Masstransit in my .net web application to connect to RabbitMq.
Sometime after site recycle, I see lots of consumers on a single queue. I didn't set competing consumers and in normal situation I should have only one consumer per queue.
When this problem happens, my messages get processed very slowly (I assume the time depends on my retry policy) and I have to showdown the site and start it again.
I use Masstransit 5.2.3/RabbitMq 3.7.6.
Could anyone give any clue what the problem could be?
Tech Stack:
.Net 4.6.1
Erlang 18.3
Language: C# 6
NServiceBus 6
RabbitMQ 3.6.3
Windows 7
Context:
We are trying to implement a dumb event publisher with smart subscription in NServiceBus on the RabbitMQ transport.
All interested receivers subscribe to an event.
A Publisher publishes the event to an event channel.
All subscribers receive a copy of the event.
Note that I did not say each TYPE of subscriber receives a copy of the message. If there are multiple instances of a service running, and they all have active subscriptions to an event, each INSTANCE of the subscribing service should get a copy of the message.
However, NServiceBus' notion of Publish-Subscribe delivers a published event to one and only one receiver on a given channel. In our case, one instance of a given subscribed service.
I hesitate to list messaging "patterns" because they don't seem to be named or described particularly consistently. However, I believe we are esentially trying to implement the "Multicast" version of the Publish-Subscribe Channel messaging pattern from the Enterprise Integration Patterns (Hohpe and Woolf) Book.
Problem Statement:
Our business case is this:
We have a configuration service that provides application configuration to all other services in our broader application.
Each service requests its configuration from the configuration service on startup by sending a ConfigurationRequest message.
The configuration services replies to the specific instance that made the ConfigurationRequest with a ConfigurationResponse. This is done using NServiceBus' Full-Duplex (commonly called Request-Response) feature.
There is a website that can modify configuration globally. When it does so, it notifies the configuration service with an UpdateConfiguration command.
The configuration service publishes a ConfigurationUpdated event that all other services are subscribed to.
Each service can have multiple instances running on more than one server. ALL instances of a service need to update their configuration, not just one instance.
Each service instance is calling NServiceBus' EndpointConfiguration.MakeInstanceUniquelyIdentifiable with a distinct discriminator.
Currently when we publish the ConfigurationUpdated event, only one instance of each service type gets the message. The events are distributed round-robin, instead of each instance getting a copy of the message.
We have worked around this problem by keeping a record of running services instances and sending the ConfigurationUpdated event (as a Command in NServiceBus) to each of them individually, but Pub-Sub implies that we should have dumb publishers and smart subscribers, and our workaround is the opposite... Our publisher looks up a listof each subscriber and sends to it implicitly. Is there some configuration I am missing on the subscriber side that would allow each instance of each service to get a copy of the published ConfigurationUpdated event? If not, where should I look in NServiceBus to start implementing such a feature? Routing topology, perhaps?
i'm not familiar enough with nservicebus to know how to do it with that toolset, but the RabbitMQ implementation would be a "fanout" exchange with a queue per consumer.
Every queue bound to a fanout exchange will get a copy of the message. If there is a single consumer for each of those queues, then you will be sending a copy of the message to each of the consumers.
It sounds like you have multiple consumers connected to the same queue. Maybe there's a way to tell nservicebus to create a queue for each consumer instance?
p.s. you're right about pub-sub pattern being multicast. i talk about this in my RMQ Patterns ebook (https://leanpub.com/rmq-patterns) and that's what the EIP book would say, as well.
Right now we have a queuing system(activemq) which is storing the messages. And we have written a separate java application that will read the queue and then trigger a exe to do some processing. But we want to do away with this extra application that is linking our activemq and exe. So i want to know whether any queuing system houses a code which will help me run the exe without any extra code written by me.
Any inputs regarding which queuing system can get me this done will be greatly appreciated.
This isn't really how Message Brokers work. You could however embed a broker inside your own application or create a broker plugin to do something. In the end though the best way to do this is to create your own client that can implement your business logic and let the Message Broker do what it was designed to do, route message traffic.
If you want more of a 'push' solution rather than a producer-consumer solution (which sounds to me like you are) you could look into the use of WebSockets. That would be another way of dealing with messages.
As others have said it doesn't look like using a message broker is the solution you want if you don't want to have some additional middleware to provide asynchronous communication.
So you just need something to launch an EXE on message arrival?
Message Queuing Triggers
Just some additional information for you... Triggered applications are natively supported in IBM Websphere MQ via a Trigger Monitor application that runs as a service (in windows implementations) or a daemon (in UNixish implementations).
When a message arrives in a queue, the MQ software will generate another message ("Trigger" message and send it to the "Trigger" queue, which is being monitored by the Trigger Manager app. The app then starts the required application.
So your implementation of an "app to start an app" is not odd ball or strange at all.. IBM do it in their implementation. I see nothing wrong with your implementation and if you can integrate it tighter to activeMQ then you are on a winner.
What about IBM MQ's triggering feature ?
WebSphere MQ provides a feature that enables an application or channel to be started automatically when there are messages available to retrieve from a queue. A message is put to a queue defined as triggered. If a series of conditions are met, the queue manager sends a trigger message to an initiation queue.
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q026940_.htm
All,
I'm looking for advice over the following scenario:
I have a component running in one part of the corporate network that sends messages to an application logic component for processing. These components might reside on the same server, different servers in the same network (LAN ot WAN) or live outside in the cloud. The application server should be scalable and resilient.
The messages are related in that the sequence they arrive is important. They are time-stamped with the client timestamp.
My thinking is that I'll get the clients to use WCF basicHttpBinding (some are based on .NET CF which only has basic) to send messages to the Application Server (this is because we can guarantee port 80/443 will be open for outgoing connections). Server accepts these, and writes these into a queue. This queue can be scaled out if needed over multiple machines.
I'm hesitant to use MSMQ for the queue though as to properly scale out we are going to have to install seperate private queues on each application server and round-robin monitor the queues. I'm concerned though that we could lose a message on a server that's gone down until the server is restored, and we could end up processing a later message from a different server and disrupt the sequence.
What I'd prefer is a central queue (e.g. a database table) that all application servers monitor.
With this in mind, what I'd like to do is to create a custom WCF binding, similar to netMsmqBinding, but that uses the DB table instead but I'm confused as to whether I can simply create a custom transport or a I need a full binding, and whether the binding will allow the client to send over HTTP. I've looked around the internet but I'm a little confused as to where to start.
I could not bother with the custom WCF binding but it seems a good way to introduce scalability if I do need to seperate the servers.
Any suggestions please would be helpful, including alternatives.
Many thanks
I would start with MSMQ because it is exactly for this purpouse. Use single transactional queue on clustered machine and let application servers to take messages for processing from this queue. Each message processing has to be part of distributed transaction (MSDTC).
This scenario will ensure:
clustered queue host will ensure that if one cluster node fails the other will still be able to handle requests
sending each message as recoverable - it means that message will be persisted on hard drive (not only in memory) so in critical failure of the whole cluster you will still have all messages.
transactional queue will ensure that all message transport operations will be atomic - moving message from outgoing queue to destination queue will be processed as transaction. It means that original message from outgoing queue will be kept in queue until ack from destination queue arrives. Transactional processing can ensure in order delivery.
Distributed transaction will allow application servers consuming messages in transaction. Message will not be deleted from queue until application server commits transaction or transaction time outs.
MSMQ is also available on .NET CF so you can send messages directly to queue without intermediate non-reliable web service layer.
It should be possible to configure MSMQ over HTTP (but I have never used it so I'm not sure how it cooperates with previous mentioned features).
Your proposed solution will be pretty hard. You will end up in building BizTalk's MessageBox. But if you really want to do it, check Omar's post about building database queue table.
We are using MSMQ right now with WCF activation feature, it enables us not to pull queue to read messages. It like push message to application.
As we are looking at porting from MSMQ to RabbitMQ going through what we need from message queue.
I can't anything regarding RabbitMQ .net client support for receiving message notification from subscribed queue?
Is there anything in RabbitMQ with .net which can do push notification to subscriber like MSMQ?
Or we need service running which constantly checks for message?
In AMQP (and RabbitMQ), there are two ways to retrieve messages: basic.get and basic.consume.
Basic.get is used to poll the server for a message. If one exists, it is returned to the client. If not, a get-empty is returned (the .NET method returns null).
Basic.consume sets the consumer for the queue. The broker pushes messages to the consumer as they arrive. You can either derive DefaultBasicConsumer, which gives you your own custom consumer, or you can use the Subscription Message Pattern, which gives you a blocking nextDelivery().
For more information, check out the API guide linked above and the .NET Client Userguide. Also, a great place to ask RabbitMQ-related questions is the rabbitmq-discuss mailing list.
I think you are after something like the EventingBasicConsumer. See also this question/answer
That is a feature provided by WAS (Windows Activation Service). Right now WAS has listener adapters for net.pipe, net.msmq and net.tcp (and its port sharing service). I guess you would need a specific AMQP listener adapter.
This may help http://msdn.microsoft.com/en-us/library/ms789006.aspx