As I understand it, Hangfire does not support async methods yet. As a workaround, I wrapped my async method calls with AsyncContext.Run() from AsyncEx to make it appear to be synchronous from Hangfire point of view. Exception seems to be bubbled up correctly as expected (unwrapped from AggregateException).
public void Task()
{
AsyncContext.Run(() => TaskAsync());
}
private async Task TaskAsync()
{
//...
}
However, when TaskAsync throws TaskCanceledException, Hangfire does not correctly mark it as "Failed". Instead it will try to process the job again. If TaskAsync keeps on throwing TaskCanceledException, it will be stuck in that state indefinitely instead of stop retrying after 10 times as usual.
It seems to be because Hangfire treats OperationCanceledException as its own control flow, instead of treating it as an exception originating from the job. e.g. here, and here.
Is there any way to get around it, other than wrapping all my Hangfire jobs with catch TaskCanceledException ?
For those who face the same problem as myself, this bug has been fixed in Hangfire 1.4.7.
As per the changeset, Hangfire now checks that the InnerException is not a TaskCanceledException.
Related
I'm designing a simple toy interpreter, and I have a custom exception like so:
class ValError(varName: String) : Exception("$varName is val, cannot reassign.")
The problem is, when it's thrown, it prints out the whole call stacktrace like so:
ValError: foo is val, cannot reassign.
at com.github.me.dynamik.interpreter.Environment.get(Environment.kt:62)
at com.github.me.dynamik.interpreter.TreeWalker.visitVariableExpr(TreeWalker.kt:154)
at com.github.me.dynamik.expressions.VariableExpr.evaluateBy(Expression.kt:57)
but I only want it to print out the first line
ValError: foo is val, cannot reassign.
How do I accomplish this?
Short answer: catch the exception, and print its message field. You can do this in a try…catch block in your top-level function, or (better) in an UncaughtExceptionHandler.
Long boring answer:
An exception will get passed up from each function to its caller, until it finds one with an enclosing try…catch block.
So one approach is to catch the exception yourself, e.g.:
fun main() {
try {
// All your top-level code here…
} catch (x: Exception) {
System.err.println(x.message)
}
}
If you're using a logging framework, you can use that instead, of course.
If there's no enclosing try…catch, it will call the thread's UncaughtExceptionHandler — or, if the thread doesn't have one, its ThreadGroup's one, or failing that, the default one.
If you don't provide a default one, the system's one calls its printStackTrace() method, which prints the exception message and all its backtrace to the standard error stream — which is why you see the trace shown in the question.
So another approach is to set an UncaughtExceptionHandler at one of those levels, e.g.:
Thread.setDefaultUncaughtExceptionHandler { t, x ->
System.err.println(x.message)
}
That's a lot safer, because it works for all threads.
If you're writing a simple app that only uses a single thread, then it can be straightforward to catch the exception in your top-level method. But if you're writing something which uses a GUI toolkit or web framework, or uses coroutines or another execution framework, or start any threads manually, then there are likely to be other threads running. This has two effects:
First, you won't have access to the top-level methods of those threads, and so you won't be able to catch exceptions there. So an UncaughtExceptionHandler is the only way. (The default one is safest, as you probably won't have access to the Threads nor even ThreadGroups involved.)
Second, although the thread with the exception will stop running, any other threads won't. So the JVM won't shut down, and other threads will carry on running, unaware. This leaves your app in an inconsistent state, which could be disastrous: you could end up with jobs queueing up but never run, or other threads stalling waiting for data, or requests going unanswered, or just about anything at all depending how your app is structured and which thread died. And because the JVM continues, system-level monitors or service managers won't know there's anything wrong.
So after logging the exception, in some cases it can be a very good idea for your handler to shut down the whole app.
(BTW, a third approach would be to use a custom Exception and override its printStackTrace() method to print only the message. But that's not a good idea; it defeats the intent of that method, which could cause all sorts of problems for anything that uses it.)
Use try..catch
Kotlin
try {
// perform tasks
throw ValError("x")
} catch (e: ValError) {
println(e.message)
}
Java
try{
// perform task
throw new ValError("x")
}
catch(ValError e){
System.out.println(e.getMessage());
}
I'd like to add a service that executes some initialization operations for the system when it's first created.
I'd imagine it would be a stateless service (with cluster admin rights) that should self-destruct when it's done it's thing. I am under the impression that exiting the RunAsync function allows me to indicate that I'm finished (or in an error state). However, then it still hangs around on the application's context and annoyingly looking like it's "active" when it's not really doing anything at all.
Is it possible for a service to remove itself?
I think maybe we could try using the FabricClient.ServiceManager's DeleteServiceAsync (using parameters based on the service context) inside an OnCloseAsync override but I've not been able to prove that might work and it feels a little funky:
var client = new FabricClient();
await client.ServiceManager.DeleteServiceAsync(new DeleteServiceDescription(Context.ServiceName));
Is there a better way?
Returning from RunAsync will end the code in RunAsync (indicate completion), so SF won't start RunAsync again (It would if it returned an exception, for example). RunAsync completion doesn't cause the service to be deleted. As mentioned, for example, the service might be done with background work but still listening for incoming messages.
The best way to shut down a service is to call DeleteServiceAsync. This can be done by the service itself or another service, or from outside the cluster. Services can self-delete, so for services whose work is done we typically see await DeleteServiceAsync as the last line of RunAsync, after which the method just exits. Something like:
RunAsync(CancellationToken ct)
{
while(!workCompleted && !ct.IsCancellationRequested)
{
if(!DoneWithWork())
{
DoWork()
}
if(DoneWithWork())
{
workCompleted == true;
await DeleteServiceAsync(...)
}
}
}
The goal is to ensure that if your service is actually done doing the work it cleans itself up, but doesn't trigger its own deletion for the other reasons that a CancellationToken can get signaled, such as shutting down due to some upgrade or cluster resource balancing.
As mentioned already, returning from RunAsync will end this method only, but the service will continue to run and hence not be deleted.
DeleteServiceAsync certainly is the way to go - however it's not quite as simple as just calling it because if you're not careful it will deadlock on the current thread (especially in local developer cluster). You would also likely get a few short-lived health warnings about RunAsync taking a long time to terminate and/or target replica size not being met.
In any case - solution is quite simple - just do this:
private async Task DeleteSelf(CancellationToken cancellationToken)
{
using (var client = new FabricClient())
{
await client.ServiceManager.DeleteServiceAsync(new DeleteServiceDescription(this.Context.ServiceName), TimeSpan.FromMinutes(1), cancellationToken);
}
}
Then, in last line of my RunAsync method I call:
await DeleteSelf(cancellationToken).ConfigureAwait(false);
The ConfigureAwait(false) will help with deadlock issue as it will essentially return to a new thread synchronization context - i.e. not try to return to "caller context".
I have an Endpoint with a Handle method. I would like to do something immediately before and immediately following Handle. I was able to get the step working before by imolementing LogCommandEntryBehavior : Behavior<IIncomingLogicalMessageContext>. What needs to be implemented to happen immediately following Handle?
In NServiceBus Version 6, the pipeline consists of a series of Stages, each one nested inside the previous like a set of Russian Dolls. For an incoming message, the stages are (in order):
ITransportReceiveContext,
IIncomingPhysicalMessageContext,
IIncomingLogicalMessageContext, and
IInvokeHandlerContext
When you create a behavior within a stage, you get passed a delegate called next(). When you call next() you execute the next behavior in the pipeline (which may move the pipeline to the next stage). The call to next() returns a Task which indicates when this inner part of the pipeline is done.
That gives you the opportunity to invoke your code before moving on to the next stage, and invoke more code after the next stage has been completed like this:
public class LogCommandEntryBehavior : Behavior<IIncomingLogicalMessageContext>
{
public override async Task Invoke(IIncomingLogicalMessageContext context, Func<Task> next)
{
// custom logic before calling the next step in the pipeline.
await next().ConfigureAwait(false);
// custom logic after all inner steps in the pipeline completed.
}
}
If you want to log information about the handling of a message, I recommend looking at the IInvokeHandlerContext stage. It contains information about how the message was handled and will be called once for every handler that is invoked (in cases where you have multiple). If you don't need info at that level then IIncomingLogicalMessageContext is probably all you need.
You can read more about the Version 6 pipeline in the Particular Docs site:
Steps, Stages and Connectors
Manipulate Pipeline with Behaviors
I have an interesting use case where certain exception types mean "This message is no longer valid and should be ignored" but this code doesn't have any awareness of the Bus in order to call Bus.DoNotContinueDispatchingCurrentMessageToHandlers().
I loathe boilerplate code like try/catch blocks that need to be present in every single message handler. So I started implementing a UnitOfWork to handle and swallow the exception, but I can't find a way to tell the framework that "Yes, this code generated an exception, but forget about that and just complete the transaction."
Bus.DoNotContinueDispatchingCurrentMessageToHandlers() does not work. I tried having an ITransport injected and calling AbortHandlingCurrentMessage() but that caused the entire universe to blow up. Even stepping through the source code I seem to be at a loss.
Note that it very well may be that this is a horrible idea, because faking that there is no exception when there is in fact an exceptional case would cause the transaction to commit, causing who knows how many bad unknown side effects. So it would be preferable to have a method that still rolls back the transaction but discards the message. But I would be interested in a potential "Yes I know what I'm doing, commit the transaction regardless of the exception" option as well.
As of NServiceBus version 4.4 you can control this by injecting a behavior into our handler pipeline.
This let's you control which exceptions to mute.
class MyExceptionFilteringBehavior : IBehavior<HandlerInvocationContext>
{
public void Invoke(HandlerInvocationContext context, Action next)
{
try
{
//invoke the handler/rest of the pipeline
next();
}
//catch specific exceptions or
catch (Exception ex)
{
//modify this to your liking
if (ex.Message == "Lets filter on this text")
return;
throw;
}
}
There are several samples of how this works:
http://docs.particular.net/samples/pipeline/
That said I totally agree with Ramon that this trick should only be used if you can't change to design to avoid this.
A dirty solution would be having a unit of work test the exception, put the message id in a shared 'ignore' bag (concurrent dictionary in memory, db, what works for you) , let it fail so that everything is rolled back, in the retry have a generic message handler compare the message ID and let that call Bus.DoNotContinueDispatchingCurrentMessageToHandlers()
If you do not want to work with a unit of work then you could try to use the AppDomain.FirstChanceException.
I wouldn't advice any of these as good solution :-)
Why would you like to 'swallow' unhandled exceptions?
If you want to ignore an exception then you should catch these in the handler and then just return and log this.
What I'm more interested in is what about state? You maybe have already writen to a store. Shouldn't these writes be rolled back? If you swallow an exception the transaction commits.
It seems to me you are running in a kind of 'at least once delivery' environment. THen you need to store some kind of message id somewhere.
Or is it an action initiated by several actors based on a stale state? In that case you need to have first/last write wins construction that just ignores a command based on a stale item.
If you handl an event then swallowing a exception seems not correct. They usually say that you can ignore commands but that you always have to handle events.
Shouldn't this be part of validation? If you validate a command then you can decide to ignore it.
I am currently using this code tu call a webservice to get data for an application page.
Everything is fine until I try to call this method 10 times in a row without waiting for the first call to finish.
Doing so is causing me to have a problem with closures and I get the same result object for all my results.
has anyone faced this with Rx.Net before? if so does anyone have an idea or a lead so that I may resolve this issue.
public void GetPage(long pageId, Action<PageDTO> observer)
{
Observable.FromEvent<GetPageCompletedEventArgs>(
handler => Service.GetPageCompleted += handler,
handler => Service.GetPageCompleted -= handler)
.Select(eventHandler => eventHandler.EventArgs.Result)
.Take(1) // necessary to ensure the observable unsubscribes
.ObserveOnDispatcher() // controls which thread the observer runs on;
.Subscribe(observer, HandleError);
Service.GetPageAsync(pageId);
}
Is Service always the same instance? If so, you're going to run into all kinds of craziness whereby GetPageCompleted events will be handled by the FromEvent created by a different call (with different arguments), which would explain why your results are the same for all methods that were called at the same time.
You can get around this specific issue by using the Begin/End methods, though you will still likely run into problems with contention on the underlying connection.
public void GetPage(long pageId, Action<PageDTO> observer)
{
Observable.FromAsyncPattern<long, PageDTO>(
service.BeginGetAwards, service.EndGetAwards)(pageId)
.ObserveOnDispatcher() // controls which thread the observer runs on;
.Subscribe(observer, HandleError);
}