We are using NServiceBus 6.4, and we've already disabled retries by setting the delayed and immediate to 0. However, when an exception is caught, are retry is still triggered. How can we completely disable retry even when there's an exception?
config.DisableFeature<TimeoutManager>();
recoverability.Delayed(delayed => { delayed.NumberOfRetries(0); });
recoverability.Immediate(immediate => { immediate.NumberOfRetries(0); });
Documentation on disabling retries can be found here:
https://docs.particular.net/nservicebus/recoverability/configure-immediate-retries#disabling
https://docs.particular.net/nservicebus/recoverability/configure-delayed-retries#disabling-through-code
It's exactly what you did. So it should not work anymore.
Related
I'm working on a Kafka consumer in kotlin/javalin, using the standard kafka library org.apache.kafka.clients.consumer, and struggling a bit with the poll function, as it seems to never throw any errors that can be caught, it just writes warn/errors to the console. For example, when it's not able to reach the broker, it logges a warning that "Connection to node -1 could not be established. Broker may not be available.":
{
"timestamp": "2022-12-14T13:30:58.673+01:00",
"level": "WARN",
"thread": "main",
"logger": "org.apache.kafka.clients.NetworkClient",
"message": "[Consumer clientId=xxx, groupId=xxx] Connection to node -1 (localhost/127.0.0.1:1000) could not be established. Broker may not be available."
}
But it doesn't actually throw any errors, so it's pretty much impossible to handle the error, if you would like to do anything other than just continue to poll forever. Does anyone know if there is some way to configure this behavior? Or am I missing something?
The relevant code
consumer = createConsumer() // This returns a Consumer<String?, String?>
consumer.subscribe(listOf(TOPIC))
while (true) {
val records = consumer.poll(Duration.ofSeconds(1))
records.iterator().forEach {
println(it.key())
}
consumer.commitSync() // Commit offset after finished processing entries
}
I can trigger a timeout-error if I call the partitionsFor-function from the consumer, so this can work as a liveness-probe, but this feels more like a hack than the intended way to do it.
try {
var committed = consumer.partitionsFor(TOPIC)
} catch (e: Exception) {
println(e)
}
Thanks!
The client is dumb, and expects you to provide the correct values.
You can use AdminClient.describeCluster() with the same address to verify connection, then catch/throw RuntimeException from that.
Otherwise, the consumer will retry and update the metadata for your bootstrap.servers until it can connect.
I'm using in react-native the package #aspnet/signalr to connect with my server.
All work correctly until the app is in foreground, I'm able to reconnect if I lose the connection without receiving errors.
When I open the app after a long time in in background I can reconnect immediately to my server but I receive the error Error: Connection disconnected with error 'Error: Server timeout elapsed without receiving a message from the server.'
How can I intercept this error?
This is a piece of my code:
connection = new signalR.HubConnectionBuilder()
.withUrl("http://192.168.xxx.xxx/notificationHub?userId=" + authInfo.userId)
.build();
connection.on("receiveMessage", data => {
console.log('*** MESSAGGIO RICEVUTO ***');
Alert.alert(data);
});
connection.start()
.then(() => console.log("Connessione avvenuta"))
.catch((err) => console.log(err);
connection.onclose()
.then(() => connection.start(););
Thanks
Error: Connection disconnected with error 'Error: Server timeout elapsed without receiving a message from the server.'
The default timeout value of serverTimeoutInMilliseconds is 30,000 milliseconds (30 seconds), if this timeout elapses without receiving any messages from the server, the connection might be terminated with above error.
To troubleshoot the issue, please check if you just update KeepAliveInterval setting of your SignalR hub but not change the serverTimeoutInMilliseconds value on your client side.
And the recommended serverTimeoutInMilliseconds value is double the KeepAliveInterval value.
Update:
Is there a way to intercept this error and manage the error without warning?
If you do not want the signalR client log this error in browser console tab, you can try to modify the LogLevel to None.
.configureLogging(signalR.LogLevel.None)
Then manage error in onclose callbacks, like below.
connection.onclose(error => {
//...
console.log("Connection Disconnected");
});
I am using serilog with asp net core 3
It is setup and logging exceptions automatically - so i have no error handling to log the errors.
I was attempting to add extra context properties to logged items, and have added middleware to log these.
LogContext.PushProperty("Email", email);
LogContext.PushProperty("Url", url);
These are added if i manually log myself but any logs added automatically when an error occurs does not have these items added.
Any ideas?
NOTE: i have read this...
https://blog.datalust.co/smart-logging-middleware-for-asp-net-core/
This is the closest i have found to working around the issue, but this catches the exception and manually logs it, which is a shame if this is the only way it can be done.
At first, LogContext is a stack; properties that are pushed onto the stack must be popped back off by disposing the object returned from PushProperty():
using (LogContext.PushProperty("Email", email))
using (LogContext.PushProperty("Url", url)){
// middleware code
...
}
Otherwise, the behavior may be non-deterministic.
but any logs added automatically when an error occurs does not have these items added
I assume that error logging occurs outside of the scope, where these properties don't exist. In this case, try ThrowContextEnricher to enrich the exception log with properties from the original context where the exception was thrown.
// call it once on app startup
ThrowContextEnricher.EnsureInitialized();
...
// then each throwing will capture context,
// so you can enrich log in exception handler:
catch (Exception ex)
{
using (LogContext.Push(new ThrowContextEnricher()))
{
Log.Error(ex, "Exception!");
}
}
Or simply register ThrowContextEnricher globally at LoggerConfiguration (in this case ThrowContextEnricher.EnsureInitialized() is not required). So every exception log will be enriched:
Log.Logger = new LoggerConfiguration()
.Enrich.With<ThrowContextEnricher>()
.Enrich.FromLogContext()
...
.CreateLogger();
Disclaimer: I am the author of that library, and I also left an example in this answer.
As per Customizing Error Handling "Throwing the exception in the catch block will forward the message to the error queue. If that's not desired, remove the throw from the catch block to indicate that the message has been successfully processed." That's not true for me even if I simply swallow any kind of exception in a behavior:
public override async Task Invoke(IInvokeHandlerContext context, Func<Task> next)
{
try
{
await next().ConfigureAwait(false);
}
catch (Exception ex)
{
}
}
I put a breakpoint there and made sure execution hit the catch block. Nevertheless after intimidate and delayed retries messages inevitably ends up in error queue. And I have no more Behaviours in the pipeline besides this one.
Only if I run context.DoNotContinueDispatchingCurrentMessageToHandlers(); inside the catch block it prevents sending error to the error queue, but it also prevents any further immediate and delayed retries.
Any idea on why it works in contravention of Particular NserviceBus documentation is very appreciated
NserviceBus ver. used: 6.4.3
UPDATE:
I want only certain type of exceptions not being sent to an error queue in NServiceBus 6, however to make test case more clear and narrow down the root cause of an issue I use just type Exception. After throwing exception, execution certainly hits the empty catch block. Here is more code to that:
public class EndpointConfig : IConfigureThisEndpoint
{
public void Customize(EndpointConfiguration endpointConfiguration)
{
endpointConfiguration.DefineEndpointName("testEndpoint");
endpointConfiguration.UseSerialization<XmlSerializer>();
endpointConfiguration.DisableFeature<AutoSubscribe>();
configure
.Conventions()
.DefiningCommandsAs(t => t.IsMatched("Command"))
.DefiningEventsAs(t => t.IsMatched("Event"))
.DefiningMessagesAs(t => t.IsMatched("Message"));
var transport = endpointConfiguration.UseTransport<MsmqTransport>();
var routing = transport.Routing();
var rountingConfigurator = container.GetInstance<IRountingConfiguration>();
rountingConfigurator.ApplyRountingConfig(routing);
var instanceMappingFile = routing.InstanceMappingFile();
instanceMappingFile.FilePath("routing.xml");
transport.Transactions(TransportTransactionMode.TransactionScope);
endpointConfiguration.Pipeline.Register(
new CustomFaultMechanismBehavior(),
"Behavior to add custom handling logic for certain type of exceptions");
endpointConfiguration.UseContainer<StructureMapBuilder>(c => c.ExistingContainer(container));
var recoverability = endpointConfiguration.Recoverability();
recoverability.Immediate(immediate =>
{
immediate.NumberOfRetries(2);
});
endpointConfiguration.LimitMessageProcessingConcurrencyTo(16);
recoverability.Delayed(delayed =>
{
delayed.NumberOfRetries(2);
});
endpointConfiguration.SendFailedMessagesTo("errorQueue");
...
}
}
public class CustomFaultMechanismBehavior : Behavior<IInvokeHandlerContext>
{
public override async Task Invoke(IInvokeHandlerContext context, Func<Task> next)
{
try
{
await next().ConfigureAwait(false);
}
catch (Exception ex)
{
}
}
}
UPDATE 2
I think I know what's going on: message is handled by first handler that throws an exception which is caught by the Behavior catch block, but then NServiceBus runtime tries to instantiate second handler class which is also supposed to handle the message (it handles class the message is derived from). That's where another exception is thrown in a constructor of one of dependent class. StructureMap tries to instantiate the handler and all its dependent services declared in the constructor and in the process runs into the exception. And this exception is not caught by CustomFaultMechanismBehavior.
So my I rephrase my question now: Is there any way to suppress errors (ignore error queue) occurring inside constructor or simply during StructureMap classes initialization? Seems like the described way does not cover this kind of situations
Your behavior is activated on Handler invocation. This means you are catching exceptions happening inside the Handle method so any other exception, e.g. in the Constructor of the handler would not be caught.
To change the way you 'capture' the exceptions, you can change the way the behavior is activated, e.g. change it from Behavior<IInvokeHandlerContext> to Behavior<ITransportReceiveContext> which is activated when the transport receives a message. You can investigate on different stages and behaviors to see which one suits your purpose best.
Greetings to MT experts.
In my app I have default retry policy that sends message each 3 minutes for 30 minutes in total. And if there are many failed messages that are affected with this policy (more than 16) other messages are not handled (even successfull ones). This is a huge problem, because if there are 16 broken messages, then whole queue is blocked for 30 minutes.
I'm sure there is a solution for this, but I haven't found any.
The solution is redelivery aka second-level retries.
Here is the documentation.
There are two ways to use it:
Explicit redelivery from a consumer, called on exception:
public async Task Consume(ConsumeContext<ScheduleNotification> context)
{
try
{
// try to update the database
}
catch (CustomerNotFoundException exception)
{
// schedule redelivery in one minute
context.Redeliver(TimeSpan.FromMinutes(1));
}
}
Or using configuration and policy (part of the endpoint configuration delegate):
ep.Consumer<CSomeConsumer>(c => c.Message<SomeMessage>(
x => x.UseDelayedRedelivery(
p =>
{
p.Handle<SqlException>(e => e.Message.Contains("Timeout"));
p.Exponential(40, TimeSpan.FromSeconds(10), TimeSpan.FromHours(1),
TimeSpan.FromSeconds(4));
})));
Remember that you must have scheduling configured to use this feature. It can be done using Quartz or RabbitMQ/AzureSB integrated scheduling features.
I think you're looking for the circuit breaker pattern, this can be applied to the masstransit with the following:
cfg.ReceiveEndpoint(host, "customer_update_queue", e =>
{
e.UseCircuitBreaker(cb =>
{
cb.TrackingPeriod = TimeSpan.FromMinutes(1);
cb.TripThreshold = 15;
cb.ActiveThreshold = 10;
cb.ResetInterval = TimeSpan.FromMinutes(5);
});
// other configuration
});
More information can be found in the docs:
http://masstransit-project.com/MassTransit/advanced/middleware/circuit-breaker.html