I have a particular scenario with RabbitMQ that needs to have dynamically created queues and binds to exchanges, that are also dynamically created (not by me). This creation and binding is triggered by a new SignalR subscription.
This issue: https://github.com/MassTransit/MassTransit/issues/398 is about it, but I still don't know the answer.
Seems that mass transit is not very flexible on creating things on the move.
How can I achieve this? What if I stop the bus and recreate all the queues and bindings plus the new one, and start the bus again?
Thanks in advance.
Receive endpoints can be connected via the bus, as shown in the documentation.
For example:
var handle = bus.ConnectReceiveEndpoint("queue-name", x =>
{
x.Consumer<SomeConsumer>();
})
// the code below waits for the receive endpoint to be ready
// and throws an exception if a fault occurs
var ready = await handle.Ready;
Related
I have a situation where I have an asp.net core application which registers a subscription client to a topic on startup (IHostedService), this subscription client essentially has a dictionary of callbacks that need to be fired whenever it detects a new message in a topic with an id (this id is stored on the message properties). This dictionary lives throughout the lifetime of the application, and is in memory.
Everything works fine on a single instance of the asp.net core app service on azure, as soon as I scale up to 2, I notice that sometimes the callbacks in the subscription are not firing. This makes sense, as we have two instances now, each with its own dictionary store of callbacks.
So I updated the code to check if the id of the subscription exists, if not, abandon message, if yes, get the callback and invoke it.
public async Task HandleMessage(Microsoft.Azure.ServiceBus.Message message, CancellationToken cancellationToken)
{
var queueItem = this.converter.DeserializeItem(message);
var sessionId = // get the session id from the message
if (string.IsNullOrEmpty(sessionId))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
if (!this.subscriptions.TryGetValue(sessionId, out var subscription))
{
await this.subscriptionClient.AbandonAsync(message.SystemProperties.LockToken);
return;
}
await subscription.Call(queueItem);
// subscription was found and executed. Complete message
await this.subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, the problem still occurs. My only guess is that when calling AbandonAsync, the same instance is picking up the message again?
I guess what I am really trying to ask is, if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
if I have multiple instances of a topic subscription client all pointing to the same subscriber for the topic, is it possible for all the instances to get a copy of the message? Or is that not guaranteed.
No. If it's the same subscription all clients are pointing to, only one will be receiving that message.
You're running into an issue of scaling out with competing consumers. If you're scaling out, you never know what instance will pick the message. And since your state is local (in memory of each instance), this will fail from time to time. Additional downside is the cost. By fetching messages on the "wrong" instance and abandoning, you're going to pay higher cost on the messaging side.
To address this issue you either need to have a shared/centralized or change your architecture around this.
I managed to solve the issue by making use of service bus sessions. What I was trying to do with the dictionary of callbacks is basically a session manager anyway!
Service bus sessions allow me to have multiple instances of a session client all pointing to the same subscription. However, each instance will only know or care about the sessions it is currently dealing with.
i have a azure worker role with an nservicebus host 4.7.5 . This host sends events on azure servicebus transport and to a topic. Is there a way to either delay the sending of the event or setting some properties to make sure that the message appears after a delay on the topic subscription? The host sends out events after it notices a change in the primary database. There are several secondary databases into which the primary data write is replicated. The receivers are also azure worker roles that use nservicebus host and have subscription to the topics.
By the time the receivers receive the message, due to replication lag the secondaries may have out of sync data.
one option is to use primary database to read but that is a route which i dont want to take.
Would it be possible to fail-early in your subscription endpoints and let the retries take care of it? You can fine-tune the retry times/delays to make sure your secondary databases are updated before the message is retried.
You still need to find the best way to lookup your data from the database and a way to differentiate between the version in the event. You could use version numbers or last update dates in case of updates, or just lookup by an identifier in case of creation.
The endpoint reading data off the secondary database might have a event handler like this:
public class CustomerCreationHandler : IHandlesMessage<CustomerCreated>
{
public void Handle(CustomerCreated #event)
{
var customer = Database.Load(#event.CustomerId);
if(customer == null)
{
throw new CustomerNotFoundException("Customer was not found.");
}
//Your business logic goes here
}
}
You can control how many times the event handler will retry and how much delay there'll be between each attempt. In this case, the message will be retried by First-Level retries and then handed over to Second-Level retries which is configured below.
class ProvideConfiguration :
IProvideConfiguration<SecondLevelRetriesConfig>
{
public SecondLevelRetriesConfig GetConfiguration()
{
return new SecondLevelRetriesConfig
{
Enabled = true,
NumberOfRetries = 2,
TimeIncrease = TimeSpan.FromSeconds(10)
};
}
}
Alternatively, instead of just publishing the event, you can send a deferred message to the same endpoint to then publish the actual event after certain amount of time is passed.
I'm attempting to create a high level test in my solution, and I want to 'catch' messages sent to the bus.
Here's what I do:
nUnit [SetUp] spins up the WebAPI project in IISExpress
SetUp also creates the bus
Send a HTTP request to the API
Verify whatever I want to verify
The WebAPI part of the whole test works fine. The creation of the bus and kicking it off seems great too. It even finds my fake message handler. The problem is the handler never receives the command from the queue, they just stay in the RabbitMQ queue forever.
Here's how the bus is being configured:
var bus = Configure.With()
.DefineEndpointName("Local")
.Log4Net()
.UseTransport<global::NServiceBus.RabbitMQ>()
.UseInMemoryTimeoutPersister()
.RijndaelEncryptionService()
.UnicastBus();
.CreateBus();
In the log from NServiceBus starting up, I see that my fake handler is being associated with the command:
2014-09-24 15:29:59,007 [Runner thread] DEBUG NServiceBus.Unicast.MessageHandlerRegistry
[(null)] <(null)> - Associated 'Bloo.MyCommand' message with 'Blah.FakeMyCommandHandler' handler
So seeing as the message lands in the correct RabbitMQ queue, I'm assuming everything up until the handler point is working fine.
I've tried putting waits in my [TearDown] so that the bus lives a little longer - hoping to give the handler time to receive the message. I've also tried spinning off the in-memory bus for the consumer part of the interactoin into a new thread with no luck.
Has anyone else tried this?
This is only the first step, what I would love to do is create a fake bus that records messages being sent to it. The need for RabbitMQ is just to get myself going (the bounds of my solution are WebAPI on the front and the bus at the back).
Cheers
You forgot to call .Start() on the bus, that's why it didn't listen for messages.
See here for more info: http://docs.particular.net/nservicebus/hosting-nservicebus-in-your-own-process-v4.x
Also, consider using NServiceBus.Testing for unit testing your handlers and sagas:
https://www.nuget.org/packages/NServiceBus.Testing
I'm guessing your messages are just sitting in your queue forever because your end point is listening on "Local.MachineName" queue instead of "Local"
If you set the ScaleOut to be SingleBrokerQueue this should sort the issue.
Configure.ScaleOut(s => s.UseSingleBrokerQueue());
var bus = Configure.With()
.DefineEndpointName("Local")
...
If you are attempting to do full integration tests, using actual queues, then this answer won't help you.
If you are doing more focused tests, i.e. testing individual components that rely on the bus, I would recommend that you use a mocking framework (I like Moq) and mock out IBus. You can then verify that messages you expected to be sent to the bus were indeed sent.
my existing code which uses BlockingQueue creates a list of BlockingQueue (like private List> queues;) in which I can put messages to process.
However due to persistence issue we plan to shift to activemq.
Can anyone please help me if we can get a list of activemq queue (in java program not from configuration file). I know that I can use createQueue on session to create a single instance of the queue but I want list of queue like done for BlockingQueue.
Any help would be much appreciated.
You can get a list of the available queues using DestinationSource from your connection.
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
ActiveMQConnection connection = (ActiveMQConnection)connectionFactory.createConnection();
DestinationSource ds = connection.getDestinationSource();
Set<ActiveMQQueue> queues = ds.getQueues();
edit:
to create a queue take a look at ActiveMQ Hello world sample link What the code does there is creating a connection to an activeMQ-broker embedded in the jvm
// Create a ConnectionFactory
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost");
// Create a Connection
Connection connection = connectionFactory.createConnection();
connection.start();
// Create a Session
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createQueue("TEST.FOO");
The thing that might not be obvious with above code is that the line:
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost");
will not only setup a connection to a broker, but embedded a broker inside the connection if there isn't one already. Explained at the bottom of this page
That feature can be turned of using (you need a broker, but if you want to set it up in other way):
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("vm://localhost?create=false");
I really like ActiveMQ, but it offers a lot more than persistence, so things might seem a little overly complex when doing the simple things. But hope that will not scare you.
To create a list of queues, you have to create that list, then create each queue individually from the session object.
Queue q = session.createQueue("someQueueName")
This, however, does not really "create" a queue in that sense, since a queue is a persistent "thing" in the ActiveMQ process/server. This will only create a reference to an ActiveMQ queue given an identifier/name.
I'm not sure why you need ten queues right up. Typically, you have one queue per event type or use case (or similar), then use concurrent consumers to process in parallel.
But of course, you can always do somethings similar by a simple for loop, creating one queue at a time and attaching them to an arraylist. Note that you cannot get type safe queues with only Event objects in them.
You can send ObjectMessages with events though. Just create one:
Event event = createEvent(..); // Given "Event" is serializable (need to be able to persist it).
Message m = session.createObjectMessage(event);
// Send message as usual in ActiveMQ.
It might be that you need to rethink one or a few things in your code when converting from BlockingQueues to persistent ActiveMQ queues.
For some reason I can not post to the masstransit google group, even though I joined, I am told that I do not have permission to post to this group. So I am going to post here...
Now for my problem:
I am using MassTransit v2.7.2, with AutoFac v2.6.3. I am trying to configure Autofac to scan an assembly and register my consumers; all types that implement the IConsumer interface. This seems to work.
I am using the MassTransit.AutofacIntegration assembly and the LoadFrom(...) extension method to register the consumers from the container with MassTransit when I configure the bus. Here is the code:
var builder = new ContainerBuilder();
builder
.RegisterAssemblyTypes(typeof (CreateElectionCommandHandler).Assembly)
.Where(type => type.Implements<IConsumer>())
.AsSelf();
var container = builder.Build();
var localBus = ServiceBusFactory.New(configurator =>
{
//configurator.ReceiveFrom("loopback://localhost/testqueue");
configurator.ReceiveFrom("rabbitmq://localhost/commandqueue");
configurator.UseRabbitMq();
configurator.Subscribe(sbc => sbc.LoadFrom(container));
});
Assert.IsTrue(container.IsRegistered<CreateElectionCommandHandler>());
Assert.IsTrue(container.IsRegistered<TerminateElectionCommandHandler>());
Assert.AreEqual(1, localBus.HasSubscription<CreateElection>().Count());
Assert.AreEqual(1, localBus.HasSubscription<TerminateElection>().Count());
If I run the above code using the loopback
configurator.ReceiveFrom("loopback://localhost/testqueue");
configuration (comment out the rabbitmq conifig), the test will pass.
If I comment out the "loopback" config and comment in the
configurator.ReceiveFrom("rabbitmq://localhost/commandqueue");
configurator.UseRabbitMq();
config, the test will fail. (Note: The rabbitmq queue is already up and running - I have been using it as part of my POC). Specifically, it will fail on the assertion:
Assert.AreEqual(1, localBus.HasSubscription<CreateElection>().Count());
Assert.AreEqual(1, localBus.HasSubscription<TerminateElection>().Count());
Can anybody help me understand what is going on here? I am new to MT so fully anticipating that I am missing something, or not configuring something correctly.
Am I correct to assume that if there are no message subscriptions registered, then the bus will not be able to deliver to any of my consumers (even though the consumers are registered)?
Any help much appreciated!
With RabbitMQ, subscriptions are not added to the outbound bus until a message is published. This is due to how the classes are inspected and outbound endpoints to the appropriate exchanges are created and added to the pipeline.
So yes, this test will fail with RabbitMQ, but it will in fact work properly when the message is published.
The HasSubscription() calls are really meant for verifying that consumers and such are properly configured using the loopback transport, and really just for vetting out things that are not really integration issues but just making sure the internal MassTransit code is working.
So, if you were to add a Publish() call of one of those types, and then call the HasSubscription() extension method, it would pass.
I would check to see if MassTransit creates an exchange the message types in question. Messages are sent to the exchange and all consumer queues are bound to the exchange. You can look at the Rabbit config to see if that's happened or not as well. And with no consumers registered, no messages will be delivered. Chris has been working on adding options to error is there's consumers so you can handle it in your code.
I would join the mailing list https://groups.google.com/forum/?fromgroups=#!forum/masstransit-discuss to get help. There's a lot more people that can ask the right questions to get you where you need to be.