NServiceBus Listener Missing Messages - nservicebus

I am just starting to play with nservice bus and am trying to get publishing working. I have a listener which seems to be missing some messages. It is configured with
<MsmqTransportConfig
InputQueue="InformMessages"
ErrorQueue="error"
NumberOfWorkerThreads="5"
MaxRetries="5"
/>
Interestingly if I set NumberOfWorkerThreads to 1 it consistently misses every other message. For larger values it seems less determinate. In my message handler I have
class MessageHandler : IMessageHandler<ICourseRegister>
{
public void Handle(ICourseRegister message)
{
Console.WriteLine("Message dun got gotted");
Console.WriteLine("Course name is: " + message.CourseName);
}
private IBus bus;
public IBus Bus
{
set { this.bus = value; }
}
}
and the bus is configured with
var bus = NServiceBus.Configure.With()
.SpringBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(false)
.UnicastBus()
.ImpersonateSender(false)
.LoadMessageHandlers()
.CreateBus()
.Start();
Is there something I need to do and the end of Handler such that it is freed ready to receive the next message or some configuration I need to do so that there is a client side queue to retain messages if the handler is busy. The time between sending messages doesn't seem to matter, it could be 20 seconds and the listener still doesn't get all the messages.

After taking a look at the solution you sent over, I see what the problem is.
You've specified the same input queue for both processes - the publisher and the subscriber. As a result, you have both processes fighting over the messages which (I think) you intend to be going only to the subscriber.
Give each process its own input queue and everything should be right with the world :-)

If you're working similar to the pub/sub sample, what you could be seeing is "polymorphic subscriptions".
In the pub/sub sample, the publisher alternately publishes an interface IEvent and a concrete class EventMessage (which inherits IEvent).
If a subscriber is subscribed to the concrete class, then it won't receive the interface messages. This is true for all kinds of hierarchies, if you subscribe to the specific class, and the publisher publishes a message of the type of its base class, the message won't be dispatched.
The polymorphic subscriptions works the other way around. If you subscribe to the base, any subclass that the publisher publishes will arrive.
This has nothing to do with threading.
Hope that helps.

Related

MassTransit / RabbitMQ - why so many messages get skipped?

I'm working with 2 .NET Core console applications in a producer/consumer scenario with MassTransit/RabbitMQ. I need to ensure that even if NO consumers are up-and-running, the messages from the producer are still queued up successfully. That didn't seem to work with Publish() - the messages just disappeared, so I'm using Send() instead. The messages at least get queued up, but without any consumers running the messages all end up in the "_skipped" queue.
So that's my first question: is this the right approach based on the requirement (even if NO consumers are up-and-running, the messages from the producer are still queued up successfully)?
With Send(), my consumer does indeed work, but still many messages are falling through the cracks and getting dumped into to the "_skipped" queue. The consumer's logic is minimal (just logging the message at the moment) so it's not a long-running process.
So that's my second question: why are so many messages still getting dumped into the "_skipped" queue?
And that leads into my third question: does this mean my consumer needs to listen to the "_skipped" queue as well?
I am unsure what code you need to see for this question, but here's a screenshot from the RabbitMQ management UI:
Producer configuration:
static IHostBuilder CreateHostBuilder(string[] args)
{
return Host.CreateDefaultBuilder()
.ConfigureServices((hostContext, services) =>
{
services.Configure<ApplicationConfiguration>(hostContext.Configuration.GetSection(nameof(ApplicationConfiguration)));
services.AddMassTransit(cfg =>
{
cfg.AddBus(ConfigureBus);
});
services.AddHostedService<CardMessageProducer>();
})
.UseConsoleLifetime()
.UseSerilog();
}
static IBusControl ConfigureBus(IServiceProvider provider)
{
var options = provider.GetRequiredService<IOptions<ApplicationConfiguration>>().Value;
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(options.RabbitMQ_ConnectionString), h =>
{
h.Username(options.RabbitMQ_Username);
h.Password(options.RabbitMQ_Password);
});
cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName, e =>
{
EndpointConvention.Map<CardMessage>(e.InputAddress);
});
});
}
Producer code:
Bus.Send(message);
Consumer configuration:
static IHostBuilder CreateHostBuilder(string[] args)
{
return Host.CreateDefaultBuilder()
.ConfigureServices((hostContext, services) =>
{
services.AddSingleton<CardMessageConsumer>();
services.Configure<ApplicationConfiguration>(hostContext.Configuration.GetSection(nameof(ApplicationConfiguration)));
services.AddMassTransit(cfg =>
{
cfg.AddBus(ConfigureBus);
});
services.AddHostedService<MassTransitHostedService>();
})
.UseConsoleLifetime()
.UseSerilog();
}
static IBusControl ConfigureBus(IServiceProvider provider)
{
var options = provider.GetRequiredService<IOptions<ApplicationConfiguration>>().Value;
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri(options.RabbitMQ_ConnectionString), h =>
{
h.Username(options.RabbitMQ_Username);
h.Password(options.RabbitMQ_Password);
});
cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName, e =>
{
e.Consumer<CardMessageConsumer>(provider);
});
//cfg.ReceiveEndpoint(host, typeof(CardMessage).FullName + "_skipped", e =>
//{
// e.Consumer<CardMessageConsumer>(provider);
//});
});
}
Consumer code:
class CardMessageConsumer : IConsumer<CardMessage>
{
private readonly ILogger<CardMessageConsumer> logger;
private readonly ApplicationConfiguration configuration;
private long counter;
public CardMessageConsumer(ILogger<CardMessageConsumer> logger, IOptions<ApplicationConfiguration> options)
{
this.logger = logger;
this.configuration = options.Value;
}
public async Task Consume(ConsumeContext<CardMessage> context)
{
this.counter++;
this.logger.LogTrace($"Message #{this.counter} consumed: {context.Message}");
}
}
In MassTransit, the _skipped queue is the implementation of the dead letter queue concept. Messages get there because they don't get consumed.
MassTransit with RMQ always delivers a message to an exchange, not to a queue. By default, each MassTransit endpoint creates (if there's no existing queue) a queue with the endpoint name, an exchange with the same name and binds them together. When the application has a configured consumer (or handler), an exchange for that message type (using the message type as the exchange name) also gets created and the endpoint exchange gets bound to the message type exchange. So, when you use Publish, the message is published to the message type exchange and gets delivered accordingly, using the endpoint binding (or multiple bindings). When you use Send, the message type exchange is not being used, so the message gets directly to the destination exchange. And, as #maldworth correctly stated, every MassTransit endpoint only expects to get messages that it can consume. If it doesn't know how to consume the message - the message is moved to the dead letter queue. This, as well as the poison message queue, are fundamental patterns of messaging.
If you need messages to queue up to be consumed later, the best way is to have the wiring set up, but the endpoint itself (I mean the application) should not be running. As soon as the application starts, it will consume all queued messages.
When the consumer starts the bus bus.Start(), one of the things it does is create all exchanges and queues for the transport. If you have a requirement that publish/send happens before the consumer, your only option is to run DeployTopologyOnly. Unfortunately this feature is not documented in official docs, but the unit tests are here: https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit.RabbitMqTransport.Tests/BuildTopology_Specs.cs
The skipped queue happens when messages are sent to a consumer that doesn't know how to process.
For example if you have a consumer that can process IConsumer<MyMessageA> which is on receive endpoint name "my-queue-a". But then your message producer does Send<MyMessageB>(Uri("my-queue-a")...), Well this is a problem. The consumer only understands the A, it doesn't know how to process B. And so it just moves it to a skipped queue and continues on.
In my case, the same queue listens to multiple consumers at the same time

sending acknowledgement from consumer to producer and handle it in activemq and rabbitmq

As I know ActiveMQ has a feature called AUTO Acknowledge that actually inform the broker that message has been received (not acknowledging the producer).
I want to know if it is possible to send acknowledgement from consumer to producer in ActiveMQ or RabbitMQ. then I want to handle the acknowledgment message in producer and if it wouldn't receive acknowledge then sending the message again to the consumer.
You want to perform a synchronous usecase over an asynchronous medium.
In RabbitMQ's case you can use RPC, as described here - https://www.rabbitmq.com/tutorials/tutorial-six-python.html
and
https://www.rabbitmq.com/direct-reply-to.html
Please notice that even authors advise to avoid it:
When in doubt avoid RPC. If you can, you should use an asynchronous pipeline - instead of RPC-like blocking, results are asynchronously pushed to a next computation stage.
RabbitMQ Java client provides auto-acking through com.rabbitmq.client.Channel.basicConsume.
At least for ActiveMQ - this is built in. You have to turn it on in activemq.xml
<policyEntry queue=">" advisoryForConsumed="true"/>
Simply listen the advisory topic for the queue you want to monitor consumed messages for. Then you can extract message id:s and what not to "tick off" outstanding requests.
For a complete end-to-end acknowledgement, I recommend something more custom. I.e. your producer-app should listen to some "response" queue that receives responses about the status of the produced message. I.e. if processing failed - you may want to know why etc..
Anyway, here is some code with a producer that also listens to acknowledgements from ActiveMQ.
public void run() throws Exception {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
conn = cf.createConnection();
sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination dest = sess.createQueue("duck");
MessageConsumer mc = sess.createConsumer(AdvisorySupport.getMessageConsumedAdvisoryTopic(dest));
mc.setMessageListener(this);
conn.start();
MessageProducer mp = sess.createProducer(sess.createQueue("duck"));
mp.send(sess.createTextMessage("quack"));
}
public void onMessage(Message msg) {
try {
String msgId = msg.getStringProperty("orignalMessageId");
System.out.println("Msg: " + msgId + " consumed");
} catch ( Exception e) {
e.printStackTrace();
}
}

Keep messages in queue while consumer is offline

I use Masstransit in C# project.
I have a publisher and consumer services, and when both of them are up, then there are no problems. But if the consumer goes offline, published messages don't go to the queue. They just disappear.
The expected behavior is to keep messages in the queue until the consumer is started, and then send them to it. I've found several topics in google groups with same questions, but it wasn't clear for me how to solve that problem.
It seems strange to me that this functionality isn't provided out of the box because, in my understanding, it is the main purpose of RabbitMQ and MT.
The way I create publisher bus:
public static IBusControl CreateBus()
{
return Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://RMQ-TEST"), h =>
{
h.Username("test");
h.Password("test");
});
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<IProductDescriptionChangedEvent>(
content => content.CompleteTask);
});
});
}
And the consumer:
public static void StartRmqBus()
{
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://RMQ-TEST"), h =>
{
h.Username("test");
h.Password("test");
});
cfg.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Consumer<ProductChangedConsumer>();
});
});
bus.Start();
}
EDIT:
Here is one more interesting feature: if I stop both services and manually put a message to the queue via admin interface of MT, the message is waiting in test_queue. But when I start publisher or consumer service, it falls to test_queue_error queue.
You use the same queue for published and consumer, plus publisher has a consumer for this message type, as you pointed out in your own answer.
If your publisher does not consume messages, it is better to remove the receiving endpoint from it at all and then your service will be send-only.
If you have several services, where each of them need to have their own consumers for the same message type - this is how pub-sub works and you must have different queues per service. This is described in the Common Gotchas section of the documentation. In such scenario, each service will get it's own copy of the published message.
If you have one queue - you get competing consumers and this scenario is only valid for horizontal scalability, where you run several instance of the same services to increase the number of processed messages if the processing is too slow. In such case all these instances will consume messages from the same queue. In this scenario only one instance will get a message.
It seems like my publisher was set up incorrectly. After removing this part:
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<IProductDescriptionChangedEvent>(
content => content.CompleteTask);
});
it started to work as expected. Looks like it consumed its own messages, that's why I didn't see messages in the queue when the consumer was down.

NServiceBus - put message back on bus

I have a bus endpoint that processes a message, however I now want to put another message back on to the bus inside this handler.
To do this I need to get a reference to the bus.
However if I try the following:
public class ServerEndpoint: IWantToRunAtStartup
{
public static IBus Bus { get; set; }
public void Run()
{
int x = 5;
Bus = NServiceBus.Configure.With()
.DisableTimeoutManager()
.Log4Net()
.DefaultBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(false)
.PurgeOnStartup(false)
.UnicastBus()
.ImpersonateSender(false)
.CreateBus()
.Start();
}
public void Stop()
{
}
}
Then I get all sort of config errors, even if I put the Unicast and Msmq Configs in the app.config.
Could anyone provide me with pointers, I'm hoping this is a fairly straightforward mistake!!
Thanks
Duncan
EDIT: This is on the server. The thing that is confusing me is that the bus gets configured on the web app using the above code, in the global.asax App_Start() method. But when trying to do this in the ServerEndpoint (the actual 'bus' dll, run inside the host) this doesn't work.
You use Dependency Injection for that, see http://support.nservicebus.com/customer/portal/articles/862398-how-do-i-get-a-reference-to-ibus-in-my-message-handler-
Thanks for all the feedback - I was trying to run before I could walk.
By the looks of things, to send a message to another endpoint, I need to set up a different assembly (ServiceBus.Host).
The Bus is created automatically using DI, so this wasn't the problem after all.
I watched the excellent Pluralsight videos by Andreas Ohlund (http://pluralsight.com/training/Courses/TableOfContents/nservicebus) and followed the examples he built here (See BusStop - Billing Take 2)

Multiple subscribers in NServiceBus

I'm getting started with NServiceBus and have a question about the Pubsub sample.
My intention was to have multiple instances of Publisher1 running and receiving the message sent from the publisher. I also hacked the Publisher to only send messages of the eventMessage type.
But if I start the publisher and three instances of Subscriber1, only one of them gets the message at a time.
why is that? Is it a config setting or something else?
This is what I've tried which returns an exception "Exception when starting endpoint, error has been logged. Reason: Cannot configure property before the component has been configured. Please call 'Configure' first.":
using NServiceBus;
namespace Subscriber1
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server
{
}
public class OverrideInputQueue : IWantCustomInitialization
{
public void Init()
{
Configure
.Instance
.Configurer
.ConfigureProperty<NServiceBus.Config.MsmqTransportConfig>(t => t.InputQueue, "testQueue");
}
}
}
/J
NServiceBus assumes that you have one input queue per process. Make sure that each of your subscribers are configured with a unique input queue. If not all three will be polling the same queue producing the behavior you're describing.
To do this you would probably have to copy paste sub1 to 3 different folders, modfying the app.config and start them up.
Hope this helps!
You should use this-
Configure.Instance.Configurer.ConfigureProperty<NServiceBus.Unicast.Transport.Msmq.MsmqTransport>(msmq => msmq.InputQueue, "SomeQueueHere");
Make sure you use MsmqTransport and not MsmqTransportConfig as you mentioned.