Can NServiceBus process events to different queues at different rates? - nservicebus

Question

I have two subscribers to the same event. One subscriber writes to the database and the other caches data in memory. The former takes much longer than the latter and that's OK since the caching is more time critical than writing to the database. Some times the DB writer gets behind and its queue starts to grow, and that's OK (as long as eventually it catches up). But, it's unacceptable for the caching subscriber to get behind. It's OK for it to can get ahead of the DB writer when the DB writer cannot keep up. I want the two queues to be processed as fast as possible without the processing of one affecting the processing of the other.
But, what I see is that when the DB writer queue grows, the caching subscriber's queue grows. The two queues have the same number of pending items. They seem to be in lock step.
Profiling shows that the DB write takes about 500 times longer than the memory cache (which is not surprising). So, the caching subscriber could easily keep up, but it seems to be held back by the DB writer subscriber.
I'll apologize for code up front. It would be easier to understand if not for the endpoint wrapper code.
This method creates the endpoint as en event publisher. (My wrapper code provides different endpoint types for different purposes. I find the NSB endpoint abstraction to be lacking):
public async Task Start()
{
if (_endpoint == null)
_endpoint = await EventPublisherEndpoint.Start(EndpointName);
}
Here's the EventPublisherEndpoint:
public sealed class EventPublisherEndpoint : Endpoint, IEventPublisherEndpoint
{
private EventPublisherEndpoint(IEndpointInstance nsbEndpoint, string name) : base(nsbEndpoint, name) { }
/// <summary>
/// Create and start endpoint.
/// </summary>
public static async Task<IEventPublisherEndpoint> Start(string endpointName)
{
var ep = await ConfigureEndpoint(new EndpointConfig(), endpointName);
return new EventPublisherEndpoint(ep, endpointName);
}
public async Task Publish(object message)
{
await NsbEndpoint.Publish(message);
}
}
Here's the general endpoint factory:
protected static async Task<IEndpointInstance> ConfigureEndpoint(
IEndpointConfig config,
string endpointName,
Action<EndpointConfiguration> configureEndpoint = null,
Action<TransportExtensions<MsmqTransport>> configureTransport = null,
Action<PersistenceExtensions<SqlPersistence>> configurePersistence = null)
{
if (config == null)
throw new ArgumentNullException(nameof(config));
if (endpointName == null)
throw new ArgumentNullException(nameof(endpointName));
var fullName = GetFullName(endpointName);
try
{
IEndpointInstance ep = null;
await Logger.OperationAsync("Start endpoint: " + endpointName, async () =>
{
// eliminate existing error queue to clear it; is re-created below
string errorEndpointName = $"{fullName}.error";
EliminateQueue(errorEndpointName);
var endpointConfiguration = new EndpointConfiguration(fullName);
endpointConfiguration.UseSerialization<JsonSerializer>();
endpointConfiguration.SendFailedMessagesTo(errorEndpointName);
endpointConfiguration.EnableInstallers();
endpointConfiguration.DoNotCreateQueues(); // created explicitly (below)
endpointConfiguration.DefineCriticalErrorAction(OnCriticalError);
endpointConfiguration.LimitMessageProcessingConcurrencyTo(config.MaxConcurrency);
configureEndpoint?.Invoke(endpointConfiguration);
// NOTE:
// Using TransportTransactionMode.None disables retries.
// Using default (whatever that value is) results in runtime error on endpoint
// config that distributed transactions are not enabled (whatever that means).
// Stumbled into SendsAtomicWithReceive which seems to work; no error and
// retries happen.
const TransportTransactionMode transactionMode = TransportTransactionMode.SendsAtomicWithReceive;
var transport = endpointConfiguration
.UseTransport<MsmqTransport>()
.Transactions(transactionMode);
{
var routing = transport.Routing();
var instanceMappingFile = routing.InstanceMappingFile();
instanceMappingFile.FilePath(NsbInstanceMappingFile.DefaultFilePath);
}
configureTransport?.Invoke(transport);
var persistence = endpointConfiguration.UsePersistence<SqlPersistence>();
persistence.SqlVariant(SqlVariant.MsSqlServer);
persistence.ConnectionBuilder(() => new SqlConnection(config.PersistConnectionString));
persistence.TablePrefix(fullName + ".");
configurePersistence?.Invoke(persistence);
var subscriptions = persistence.SubscriptionSettings();
subscriptions.CacheFor(SubscriptionCachePeriod);
EnsureQueuesForEndpoint(fullName);
ep = await NServiceBus.Endpoint.Start(endpointConfiguration);
});
return ep;
}
catch (Exception exception)
{
throw new ApplicationException($"Unable to start endpoint '{endpointName}' for connection [{config.PersistConnectionString}].", exception);
}
}
Here's the publish code (inside a logging call):
public async Task Publish(DeviceOutput message)
{
await Logger.OperationWithoutBeginLogOrThrowAsync(
() => $"Publishing {message.GetType().Name} message: {message.ToString().Truncate()}.",
async () => { await _endpoint.Publish(message); });
}
Here's the event class:
public sealed class Datagram : DeviceOutput, IEvent
{
public Datagram(long deviceID, DatagramType payloadType, string payload) : base(deviceID)
{
Payload = payload;
PayloadType = payloadType;
}
public string Payload { get; set; }
public DatagramType PayloadType { get; set; }
public override string ToString()
{
return base.ToString() + Environment.NewLine + $"PayloadType:{PayloadType} Payload:{Payload.Truncate()}";
}
}
Here's one of the subscriber endpoint config:
public async Task Start()
{
if (_endpoint == null)
{
_endpoint = await EventSubscriberEndpoint<Datagram>.Start(
endpointName: "datagram-store-subscriber",
publisherEndpointName: DatagramPublisher.EndpointName);
}
}
And the associated handler:
public async Task Handle(Datagram datagram, IMessageHandlerContext context)
{
await _storer.Store(datagram);
}
Here's the other subscriber endpoint config:
public async Task Start()
{
if (_endpoint == null)
{
_endpoint = await EventSubscriberEndpoint<Datagram>.Start(
endpointName: "datagram-live-subscriber",
publisherEndpointName: DatagramPublisher.EndpointName);
}
}
And associated handler:
public async Task Handle(Datagram datagram, IMessageHandlerContext context)
{
//var start = DateTime.Now;
if (IsLiveDataEnabled)
{
await Logger.OperationWithoutBeginLogAsync(
() => $"Saving datagram: {datagram}",
async () => { await WriteData(datagram); });
}
//Logger.Info($"{DateTime.Now - start}");
}
Here's EventSubscriberEndpoint:
public sealed class EventSubscriberEndpoint<T> : Endpoint, IEventSubscriberEndpoint where T : IEvent
{
private EventSubscriberEndpoint(IEndpointInstance nsbEndpoint, string name) : base(nsbEndpoint, name) {}
public async Task Unsubscribe()
{
await NsbEndpoint.Unsubscribe(typeof(T), new UnsubscribeOptions());
}
/// <summary>
/// Create and start endpoint.
/// </summary>
public static async Task<IEventSubscriberEndpoint> Start(
string endpointName,
string publisherEndpointName,
Type[] excludeSubscriberTypes = null)
{
return await Start(endpointName, publisherEndpointName, typeof(T), excludeSubscriberTypes);
}
/// <summary>
/// Non-generic version of Start.
/// </summary>
private static async Task<IEventSubscriberEndpoint> Start(
string endpointName,
string publisherEndpointName,
Type messageType,
Type[] excludeSubscriberTypes = null)
{
return await Start(new EndpointConfig(), endpointName, publisherEndpointName, messageType, excludeSubscriberTypes);
}
private static async Task<IEventSubscriberEndpoint> Start(
IEndpointConfig config,
string endpointName,
string publisherEndpointName,
Type messageType,
Type[] excludeSubscriberTypes = null)
{
if (publisherEndpointName == null)
throw new ArgumentNullException(nameof(publisherEndpointName));
if (messageType == null)
throw new ArgumentNullException(nameof(messageType));
var ep = await ConfigureEndpoint(config,
endpointName,
endpointConfiguration =>
{
if (excludeSubscriberTypes != null)
endpointConfiguration.AssemblyScanner().ExcludeTypes(excludeSubscriberTypes);
}, transport =>
{
transport.Routing().RegisterPublisher(messageType, GetFullName(publisherEndpointName));
});
return new EventSubscriberEndpoint<T>(ep, endpointName);
}
}
Is there special config that allows queues to be processed independently of each other?
NServiceBus 6.2.1