I have a clear understanding of consuming messages: http://docs.masstransit-project.com/en/latest/usage/consumer.html
these implementations only handle ONE message at a time.
I need to handle multiple messages at a time, in bulk, in batches.
Mass Transit now has an experimental feature to process individual message's in a batch.
Configure your bus:
_massTransitBus = Bus.Factory.CreateUsingRabbitMq(
cfg =>
{
var host = cfg.Host(new Uri("amqp://#localhost"),
cfg =>
{
cfg.Username("");
cfg.Password("");
});
cfg.ReceiveEndpoint(
host,
"queuename",
e =>
{
e.PrefetchCount = 30;
e.Batch<MySingularEvent>(
ss =>
{
ss.MessageLimit = 30;
ss.TimeLimit = TimeSpan.FromMilliseconds(1000);
ss.Consumer(() => new BatchSingularEventConsumer());
});
});
});
And Create your Consumer:
public class BatchSingularEventConsumer: IConsumer<Batch<MySingularEvent>>
{
public Task Consume(ConsumeContext<Batch<MySingularEvent>> context)
{
Console.WriteLine($"Number of messages consumed {context.Message.Length}");
return Task.CompletedTask;
}
}
You can configure your Batch with a Message Limit and a Time Limit.
I suggest reading Chris Patterson's issue on the matter Batch Message Consumption especially the part regarding prefetch
The batch size must be less than or equal to any prefetch counts or concurrent message delivery limits in order reach the size limit. If other limits prevent the batch size from being reached, the consumer will never be called.
Batch consumption is also documented on the MassTransit website.
As it turns out, today you can do this:
public class MyConsumer : IConsumer<Batch<MyMessage>>
{
public async Task Consume(ConsumeContext<Batch<MyMessage>> context)
{
...
}
}
Related
I'm following this tutorial to create a hosted service. The program runs as expected. However, I want to process the queued items concurrently.
In my app, there are 4 clients, each of these clients can process 4 items at a time. So at any given time, 16 items should be processed in parallel.
So based on these requirements, I've modified the code a bit:
In the MonitorLoop class:
private int count = 0;
private async ValueTask MonitorAsync()
{
while (!_cancellationToken.IsCancellationRequested)
{
await _taskQueue.QueueAsync(BuildWorkItem);
Interlocked.Increment(ref count);
Console.WriteLine($"Count: {count}");
}
}
and in the same class:
if (delayLoop == 3)
{
_logger.LogInformation("Queued Background Task {Guid} is complete.", guid);
Interlocked.Decrement(ref count);
}
This shows that, if I set the "Capacity" as 4, the value will never increase after 5.
Basically, if the queue is full, it will wait until there's room for one more.
The problem is that the items are processed one at a time.
Here's the code for the BackgroundProcessing method on the QueuedHostedService class:
private async Task BackgroundProcessing(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(stoppingToken);
try
{
//instead of getting a single item from the queue, somehow, here
//we should be able to process them in parallel for 4 clients
//with a limit for maximum items each client can process
await workItem(stoppingToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error occurred executing {WorkItem}.", nameof(workItem));
}
}
}
I want to process them in parallel. I'm not sure if using Channel as the queue in the system is the best solution. Maybe I should have a ConcurrentQueue instead. But again, I'm not sure how to achieve a robust implementation that can have 4 clients with 4 threads each.
If you want four processors, then you can refactor the code to use four instances of your main loop, and use Task.WhenAll to (asynchronously) wait for all of them to complete:
private async Task BackgroundProcessing(CancellationToken stoppingToken)
{
var task1 = ProcessAsync(stoppingToken);
var task2 = ProcessAsync(stoppingToken);
var task3 = ProcessAsync(stoppingToken);
var task4 = ProcessAsync(stoppingToken);
await Task.WhenAll(task1, task2, task3, task4);
async Task ProcessAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(stoppingToken);
try
{
await workItem(stoppingToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error occurred executing {WorkItem}.", nameof(workItem));
}
}
}
}
I'm not sure how to achieve a robust implementation
If you want a robust implementation, then you can't use that tutorial, sorry. The primary problem with that kind of background work is that it will be lost on any app restart. And app restarts are normal: the server can lose power or crash, OS or runtime patches can be installed, IIS will recycle your app periodically, and whenever you deploy your code, the app will restart. And whenever any of these things happen, all in-memory queues like channels will lose all their work.
A production-quality implementation requires a durable queue at the very least. I also recommend a separate background processor. I have a blog series on the subject that may help you get started.
I am using Akka.NET to develop a logistics simulation.
Having tried various patterns, it seems to me that FSM-type behaviour using become will substantially simplify development.
The system has a repeating clock tick message that all relevant actors receive in order to simulate accelerated passage of time for the entire simulation system. This clock tick message should be handled by all actors that are subscribed to it regardless of which message loop is currently active for any specific actor.
Am I correct in thinking that the only way to handle the clock message in all message loops is by explicitly checking for it in all message loops, or is there a way of defining messages that are handled regardless of which message loop is active?
If the former is the case my idea is to check for a clock tick message in a ReceiveAny, which all the message loops need to have anyway, and to then pass it on to an appropriate handler.
You could use Stashing to Stash the messages while Simulating. I came up with the following code sample to better explain how that works:
// See https://aka.ms/new-console-template for more information
using Akka.Actor;
using Akka.NET_StackOverflow_Questions_tryout.Questions;
var actorSystem = ActorSystem.Create("stackOverFlow");
var sim = actorSystem.ActorOf(Props.Create(()=> new StackOverflow71079733()));
sim.Tell(5000L);
sim.Tell("string");
sim.Tell(1000L);
sim.Tell("strin2");
sim.Tell("strin3");
Console.ReadLine();
public class StackOverflow71079733 : ReceiveActor, IWithUnboundedStash
{
public IStash Stash { get ; set ; }
private readonly IActorRef _simActor;
public StackOverflow71079733()
{
_simActor = Context.ActorOf<SimulationActor>();
ClockTickMessage();
}
private void Simulate(long ticks)
{
Console.WriteLine($"Ticks: {ticks}");
Receive<Done>(d =>
{
Console.WriteLine("Simulation done");
Become(ClockTickMessage);
Stash?.Unstash();
});
// you can add additional messages that may to be handled while the simulation is happening
// e.g:
Receive<string>(s => Console.WriteLine($"received in '{s}' in simulation"));
//While the simulation is on-going, add the incoming message into a queue/stash it
// so that it is not lost and can be picked and handled after stimulation is done
ReceiveAny(any =>
{
Stash.Stash();
Console.WriteLine($"Stashed Ticks: {any}");
});
_simActor.Tell(ticks);
}
private void ClockTickMessage()
{
// you can create an object to represent the ClockTickMessage
Receive<long>(ticks =>
{
Become(() => Simulate(ticks));
});
}
}
/// <summary>
/// We need to run simulation in a another actor so that the parent actor can keep receiving ClockTicksMessages
/// In case the sim takes a long time to become
/// </summary>
public sealed class SimulationActor : ReceiveActor
{
private IActorRef _sender;
public SimulationActor()
{
Receive<long>(l =>
{
_sender = Sender;
Thread.Sleep(TimeSpan.FromMilliseconds(l));
_sender.Tell(Done.Instance);
});
}
}
public sealed class Done
{
public static Done Instance = new Done();
}
My, already "legacy" by now, implementation of a pub/sub solution using ServiceStack quickly ran out of clients, when it reached the 20 client limit.
We do something like:
_redisConsumer = MqClientFactory.Instance.GetRedisClient(); // Returns a IRedisClient
_subscription = _redisConsumer.CreateSubscription();
_subscription.OnSubscribe = channel => CoreLog.Instance.Info($"Subscription started on {eventChannelName}");
_subscription.OnUnSubscribe = channel => CoreLog.Instance.Warning($"Unsubscribed from {eventChannelName}");
_subscription.OnMessage = (channel, msg) =>
{
try
{
onMessageReceived(CoreRequestJsonEnvelope.CreateCoreRequestFromJson(msg));
}
catch (Exception ex)
{
CoreLog.Instance.Exception(ex);
}
};
// Since it blocks execution, we put this in a Task:
Task.Run(() =>
{
try
{
_subscription.SubscribeToChannels(eventChannelName); // blocking
}
catch(Exception e)
{
}
});
and when we have enough different channels to listen too, it runs out.
I then thought, that maybe instead of taking a new IRedisClient for each subscription, I could use the same IRedisClient for all of them, so:
_redisConsumer = mySavedRedisClient;
...
but that returns Unknown reply on multi-request after a few seconds/executions.
Lastly, I looked at the RedisPubSubServer, but it seems that I need to specify the channels in the constructor, and I cannot change after that. I do need to add and remove channels in runtime, and channels are not known from start.
What is the recommended approach?
Is it to increaase the Max-limit and continue as before?
Is it to use RedisPubSub, but how to handle dynamic channels?
What does "unknown reply on multi-request" actually mean?
Thanks!
It's not clear what 20 client limit you're referring to & how the client limit is dependent on channels or subscribers, but if this is your App own limit than sounds like increasing it would be the easiest solution.
ServiceStack.Redis doesn't support changing the subscribed channels after a subscription has started. Instead of managing the IRedisSubscription yourself you may want to consider ServiceStack.Redis Managed Pub/Sub Server which manages the background subscription thread with added resiliency and support for auto retries.
Whilst you can't change the subscribed channels at runtime, you can modify the modify the Channels collection and restart the subscription to create a new subscription to the updated channels list, e.g:
var pubSub = new RedisPubSubServer(clientsManager, chan1);
pubSub.Start();
//...
pubSub.Channels = new[] { chan1, chan2 };
pubSub.Restart();
Otherwise depending on your use-case you may be able to subscribe to a channel pattern which allows you to subscribe to a multiple dynamic channels matching wildcard channel pattern:
var pubSub = new RedisPubSubServer(clientsManager) {
ChannelsMatching = new[] { "chan:*" }
}
.Start();
Where it will handle any messages clients send that matches the channel pattern.
In version 5 of nServiceBus I have a Behavior that keeps track of messages in flight.
In the Behavior I was able to access DeliveryOptions(SendOptions) and see the Destination Queue, in NSB 6 with the change to the Behavior I can't seem to access the destination of the message any more.
Does anyone know of to access the destination of an outgoing message from a Behavior?
Previous code in v5:
public class PendingCommandBehavior : IBehavior<OutgoingContext>
{
public void Invoke(OutgoingContext context, Action next)
{
var sendOptions = context.DeliveryOptions as Nsb.Unicast.SendOptions;
if (sendOptions != null && context.OutgoingMessage.MessageIntent == Nsb.MessageIntentEnum.Send)
{
var destinationEndpoint = sendOptions.Destination.Queue;
Code in v6:
public class PendingCommandBehavior : Behavior<IOutgoingSendContext>
{
public override async Task Invoke(IOutgoingSendContext context, Func<Task> next)
{
// context doesn't have any destination queue information???
The IOutgoingSendContext is too early in the pipeline to capture the physical destination. Each outgoing send operation will go through the following contexts (in order) in NServiceBus version 6:
IOutgoingSendContext
IOutgoingLogicalMessageContext
IOutgoingPhysicalMessageContext
IRoutingContext
IBatchDispatchContext (if you are sending from inside a message handler)
IDispatchContext
After IOutgoingSendContext a routing strategy is selected but it is not converted into a physical address until after IRoutingContext.
For that reason, if you want to track physical addresses, the best bet is to sit in the IDispatchContext. This context will contain a collection of TransportOperations, each of which has an AddressTag. This will either be an instance of UnicastAddressTag with a Destination or an instance of MulticastAddressTag with a MessageType.
Here is some code to get you started:
public override Task Invoke(IDispatchContext context, Func<Task> next)
{
foreach (var operation in context.Operations)
{
if (operation.AddressTag is UnicastAddressTag unicastAddressTag)
{
var destinationEndpoint = unicastAddressTag.Destination;
}
}
return next();
}
For more info about the NServiceBus version 6 pipeline, see Steps, Stages and Connectors in the NServiceBus documentation.
I have started producer and consumer concurrently. After 6 hours producer produced around 6 crores messages into queue and stopped producer after 6 hours but consumer is running continuously, even after running 18 hours still 4 crores messages are in queue. Could any one please let me know why consumer performance is very slow?
Thanks in advance!
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory());
container.setQueueNames(this.queueName);
container.setMessageListener(new MessageListenerAdapter(new TestMessageHandler(), new JsonMessageConverter()));
return container;
}
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(
"localhost");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setMessageConverter(new JsonMessageConverter());
template.setRoutingKey(this.queueName);
template.setQueue(this.queueName);
return template;
}
public class TestMessageHandler {
// receive messages
public void handleMessage(MessageBeanTest msgBean) {
// Storing bean data into CSV file
}
}
As per Gary's suggestion you can set them as follows. Check out #RabbitListener
#Bean
public SimpleRabbitListenerContainerFactory listenerContainer( {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(baseConfig.connectionFactory());
factory.setConcurrentConsumers(7); // choose a value
factory.setPrefetchCount(1); // how many messages per consumer at a time
factory.setMaxConcurrentConsumers(10); // choose a value
factory.setDefaultRequeueRejected(false); // if you want to deadletter
return factory;
}
According to WikiPedia, crore == 10,000,000 so you mean 60 million.
The container can only process messages as fast as your listener does - you need to analyze what you are doing with each message.
You also need to experiment with the container concurrency settings (concurrentConsumers), prefetch, etc, to obtain the optimum performance, but it still ends up being your listener that takes the majority of the processing time; the container has very little overhead. Increasing the concurrency won't help if your listener is not well constructed.
If you are using transactions, that will significantly slow down consumption.
Try using a listener that does nothing with the message.
Finally, you should always show configuration when asking questions like this.