I am working on a Hangfire queue implementation to handle requests to an external API.
I can only handle 1 request at a time, so I set the WorkerCount to 1.
The API runs long running tasks and can only work with one task at the time.
I am using a fire-and-forget job queue, and the idea is to call the API, which will create an async call to the long running task, and return to the caller the confirmation that the task has been initialised. Once finished, the caller should be notified about the result of the execution.
If I leave it like this, then the Hangfire queue will finish the job successfully and immediately take the next job from the queue and call the API again, which I don't want to do as the API can only run one task at a time.
So I decided to add a custom state to the hangfire state machine so that after processing, the job would be in that state instead of being finished. Once the API is done with the long running task, the idea is to change the job status to "successful" and finish the job.
This is the code I use for the new state:
public class WaitingModelCompletionState : IState
{
public static readonly string StateName = "WaitingModelCompletion";
public string Name => StateName;
public string Reason => "Waiting for the model completion from an external service.";
public bool IsFinal => false;
public bool IgnoreJobLoadException => false;
public Dictionary<string, string> SerializeData() => new Dictionary<string, string>();
public class Handler : IStateHandler
{
public const string StateStatTKey = "stats:waitinmodelcompletion";
public string StateName => WaitingModelCompletionState.StateName;
public void Apply(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
transaction.IncrementCounter(StateStatTKey);
}
public void Unapply(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
transaction.DecrementCounter(StateStatTKey);
}
}
}
public class WaitingModelCompletionFilter : IElectStateFilter
{
public void OnStateElection(ElectStateContext context)
{
if (context.CurrentState == ProcessingState.StateName
&& context.CandidateState is SucceededState
)
{
context.CandidateState = new WaitingModelCompletionState();
}
}
}
However, for some reason the IsFinal attribute is ignored and the job is terminated even though it is in the new state.
This is my output in the dashboard.
And this is my output if I move from enqueed to the new state, instead of moving from processing to the new state
Related
I am new to SignalR and I would like to build such app -- every second a hub sends current time to all connected clients.
I found tutorial, but it is for .Net Framework (not Core): https://learn.microsoft.com/en-us/aspnet/signalr/overview/getting-started/tutorial-high-frequency-realtime-with-signalr So on one hand I don't know how to translate it to .Net Core SignalR, on the other hand I don't know how to write it from scratch (the limiting condition is the fact a hub is a volatile entity, so I cannot have state in it).
I need something static (I guess) with state -- let's say Broadcaster, when I create some cyclic action which in turn will send updates to clients. If such approach is OK, how to initialize this Broadcaster?
Currently I added such static class:
public static class CrazyBroadcaster
{
public static void Initialize(IServiceProvider serviceProvider)
{
var scope = serviceProvider.CreateScope();
var hub = scope.ServiceProvider.GetRequiredService<IHubContext<ChatHub>>();
var sub = Observable.Interval(TimeSpan.FromSeconds(1)).Subscribe(_ => hub.Clients.All.SendAsync("Bar", DateTimeOffset.UtcNow));
}
}
Yes, I know it is leaky. I call this method at the end of Startup.Configure, probably tons of violations here, but so far it is my best shot.
The missing piece was hosted service, i.e. the code that runs in the background -- https://learn.microsoft.com/en-US/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-2.2.
So my crazy class is now transformed into:
public sealed class HostedBroadcaster : IHostedService, IDisposable
{
private readonly IHubContext<ChatHub> hubContext;
private IDisposable subscription;
public HostedBroadcaster(IHubContext<ChatHub> hubContext)
{
this.hubContext = hubContext;
}
public void Dispose()
{
this.subscription?.Dispose();
}
public Task StartAsync(CancellationToken cancellationToken)
{
this.subscription = Observable.Interval(TimeSpan.FromSeconds(1)).Subscribe(_ => hubContext.Clients.All.SendAsync("Bar", DateTimeOffset.UtcNow));
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
this.subscription?.Dispose();
return Task.CompletedTask;
}
}
Currently my web API is able to run on a schedule and trigger another end point in order to sync data. The services that needs to be called are stored in a yml file. I have managed to get it working for one service to run a schedule. What I want is to be able to save multiple endpoints with schedules of their own and for them to be scheduled and executed at the right time.
Here is the code that I have now
I have done this using iHostedService interface.
This is the HostService class that implements iHostedService
public abstract class HostedService : IHostedService
{
private Task _executingTask;
private CancellationTokenSource _cts;
public Task StartAsync(CancellationToken cancellationToken)
{
_cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
_executingTask = ExecuteAsync(_cts.Token);
// If the task is completed then return it, otherwise it's running
return _executingTask.IsCompleted ? _executingTask : Task.CompletedTask;
}
public async Task StopAsync(CancellationToken cancellationToken)
{
// Stop called without start
if (_executingTask == null)
{
return;
}
// Signal cancel
_cts.Cancel();
// Wait until the task completes or the stop token triggers
await Task.WhenAny(_executingTask, Task.Delay(-1, cancellationToken));
cancellationToken.ThrowIfCancellationRequested();
}
// cancel
protected abstract Task ExecuteAsync(CancellationToken cancellationToken);
}
I am then extending this class and implementing what needs to be done in the ExecuteAsync as follows
public class DataRefreshService : HostedService
{
private readonly DataFetchService _dataFetchService;
public DataRefreshService(DataFetchService randomStringProvider)
{
_dataFetchService = randomStringProvider;
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
try
{
while (!cancellationToken.IsCancellationRequested)
{
await _dataFetchService.UpdateData(cancellationToken);
TimeSpan span = _dataFetchService.GetNextTrigger();
await Task.Delay(span, cancellationToken);
}
} catch (Exception)
{
await StopAsync(cancellationToken);
throw new Exception("Error trigger Sync service");
}
}
}
This is what I have added to the Startup.cs file
services.AddSingleton<DataFetchService>();
services.AddSingleton<IHostedService, DataRefreshService>();
You could try
services.AddHostedService<DataRefreshService>;
You could also try in making the DataRefreshService inherit from
Microsoft.Extensions.Hosting.BackgroundService
You can read more about that here
Using NServiceBus (v6), is there a way to ensure that a property is set in the SagaData object before the Saga Handler for a message is fired?
Our environment is multi-tenant so I want to ensure that the correct CustomerId is used for db access etc.. and that developers don't forget to pull this value from the incoming message/message header.
For example given this saga data ...
public interface ICustomerSagaData : IContainSagaData
{
Guid CustomerId { get; set; }
}
public class SomeProcessSagaData : ICustomerSagaData
{
// IContainSagaData and other properties removed for brevity ...
#region ICustomerSagaData properties
public virtual Guid CustomerId { get; set; }
#endregion
}
... and the following Saga ...
public class SomeProcessSagaSaga :
Saga<SomeProcessSagaData>,
IAmStartedByMessages<StartProcess>
{
public async Task Handle(StartProcess message, IMessageHandlerContext context)
{
// How do I ensure that Data.CustomerId is already set at this point?
}
// ConfigureHowToFindSaga etc ...
}
I initially tried inserting a behaviour into the pipeline e.g.
public class MyInvokeHandlerBehavior : Behavior<IInvokeHandlerContext>
{
public override async Task Invoke(IInvokeHandlerContext context, Func<Task> next)
{
// Ideally I'd like to set the CustomerId here before the
// Saga Handler is invoked but calls to ...
// context.Extensions.TryGet(out activeSagaInstance);
// return a null activeSagaInstance
await next().ConfigureAwait(false);
// This is the only point I can get the saga data object but
// as mentioned above the hander has already been invoked
ActiveSagaInstance activeSagaInstance;
if (context.Extensions.TryGet(out activeSagaInstance))
{
var instance = activeSagaInstance.Instance.Entity as ICustomerSagaData;
if (instance != null)
{
Guid customerId;
if (Guid.TryParse(context.Headers["CustomerId"), out customerId))
{
instance.CustomerId = customerId;
}
}
}
}
}
... but this only allows access to the SagaData instance after the handler has been fired.
Late answer, but you need to make sure your behaviour executes after the SagaPersistenceBehavior.
In your IConfigureThisEndpoint implementation:
public virtual void Customize(EndpointConfiguration configuration)
{
configuration.Pipeline.Register<Registration>();
}
public class Registration : RegisterStep
{
public Registration()
: base(
stepId: "AuditMutator",
behavior: typeof(AuditMutator),
description: "Sets up for auditing")
{
this.InsertAfterIfExists("InvokeSaga");
}
}
So to answer your question directly Data.CustomerId is not going to be set when you handle StartProcess messages. You will need to set that with the id coming off of the message.
public async Task Handle(StartProcess message, IMessageHandlerContext context)
{
Data.CustomerId = message.CustomerId;
}
That being said your sample above is missing a crucial piece which is the code for determining how a saga can be looked up for continuation of processing:
protected override void ConfigureHowToFindSaga(SagaPropertyMapper<SomeProcessSagaData> mapper)
{
mapper.ConfigureMapping<StartProcess>(message => message.CustomerId)
.ToSaga(sagaData => sagaData.CustomerId);
}
Each time you send a message type that is handled by a saga you need to have the ConfigureHowToFindSaga() method configured so it can look up the previously started saga to continue processing with. So in essence you are going to start a new saga for every customerid you send with a StartProcess message. You can read more about it here: https://docs.particular.net/nservicebus/sagas/
So the real question now is do you really need to be using a saga at this point? The sample only seems to be handling one type of message so do you really need to be saving the state of CustomerId? The overhead of the saga isn't necessary in your sample and I believe a regular handler would be just fine based on the example above.
I'm wrestling with a situation where we currently use the IBus interface (NServiceBus v5) in domain event handlers to send commands to a backend service for processing. With the IBus, these commands could be sent regardless of what triggered the event, whether while receiving a Web API request or as part of an NServiceBus handler (common domain model). But, in NServiceBus v6, with the shift to context specific interfaces, IEndpointInstance or IMessageHandlerContext, it seems that my domain event handlers now need to become context aware. And further, it looks like the IMessageHandlerContext is only available via method injection, so I may have to sprinkle this parameter all throughout the call stack?
Is there some approach that I'm not seeing whereby I can keep my domain event handlers context unaware? Or have I followed some bad practice that's revealing itself through this code smell?
EDIT
Here's an attempt at boiling down the scenario to the most relevant pieces. There's an order in the domain model whose status may change. When the status of the order changes, we've been firing off a StatusChanged domain event through a publisher. A subscriber to this particular domain event writes out a record of the status change and also sends out an NServiceBus command to communicate this status out - the handler for this particular command will follow some further logic on whether to send out emails, SMS messages, etc., the details of which I don't think are relevant.
Order Domain Object
public class Order
{
private OrderStatusCode _statusCode;
public OrderStatusCode StatusCode
{
get { return _statusCode; }
private set { _statusCode = value; }
}
public void ChangeStatus(OrderStatusCode status)
{
Status = status;
Publish(new StatusChanged(CreateOrderSnapshot(), status));
}
protected void Publish<T>(T #event) where T : IDomainEvent
{
DomainEventPublisher.Instance.Publish(#event);
}
}
Domain Event Publisher
public class DomainEventPublisher : IDomainEventPublisher
{
private static IDomainEventPublisher _instance;
public static IDomainEventPublisher Instance
{
get { return _instance ?? (_instance = new DomainEventPublisher()); }
}
public ISubscriptionService SubscriptionService { get; set; }
public void Publish<T>(T #event) where T : IDomainEvent
{
if (SubscriptionService == null) return;
var subscriptions = SubscriptionService.GetSubscriptions<T>();
subscriptions.ToList().ForEach(x => PublishToConsumer(x, #event).GetAwaiter().GetResult());
}
private static async Task PublishToConsumer<T>(IEventSubscriber<T> x, T eventMessage) where T : IDomainEvent
{
await x.HandleEvent(eventMessage);
}
}
Status Changed Domain Event Handler
public class StatusChangedHandler : IEventSubscriber<StatusChanged>
{
private readonly IBus _bus;
private readonly IOrdersRepository _ordersRepository;
public StatusChangedHandler(IBus bus, IOrdersRepository ordersRepository)
{
_bus = bus;
_ordersRepository = ordersRepository;
}
public async Task HandleEvent(StatusChanged #event)
{
var statusTrailEntry = new OrderStatusTrailEntry(#event.OrderSnapshot, #event.Status);
var txOptions = new TransactionOptions { IsolationLevel = IsolationLevel.ReadCommitted };
using (
var scope = new TransactionScope(TransactionScopeOption.Required, txOptions))
{
await _ordersRepository.SaveStatusTrail(statusTrailEntry);
if (communicateStatus)
{
_bus.Send(new SendCommunicationCommand(#event.OrderSnapshot, #event.Status));
}
scope.Complete();
}
}
}
The things is, up until now none of the sample code above has needed to know whether the status changed as a result of a request coming in through a Web API request or as a result of a status being changed within the context of an NServiceBus message handler (within a windows service) - the IBus interface is not context specific. But with the differentiation between IEndpointInstance and IMessageHandlerContext in NServiceBus v6, I don't feel that I have the same flexibility.
If I understand correctly, I'm able to register the IEndpointInstance with my container and inject into the EventSubscriber, so I'd be covered in the case of a Web API call, but I'd also need to add an IMessageHandlerContext as a parameter to optionally be passed down through the call stack from ChangeStatus to the Publisher and finally to the Domain Event Subscriber if the status happens to be changed within the context of a message handler. Really doesn't feel right to be adding this parameter all throughout the call stack.
I use a call to a webservice in order to wake up a process which processes some images.
The process takes 10minutes,so I don't want the asp.net client waiting for that.
What I did was a simple call async to the service to start the processing.
But it looks like ConvertToCubeService service doesn't bother to start.
ConvertToCubeService is a workflow service. I call it from within the activity of another workflow service.
I am sure that the code reaces the async call to ConvertToCubeService .
public sealed class CallProcessingAsync : CodeActivity
{
// Define an activity input argument of type string
public InArgument<string> EquirectangularImagePath { get; set; }
public InArgument<string> SaveImageDirectoryPath { get; set; }
public InArgument<string> ImageName { get; set; }
// If your activity returns a value, derive from CodeActivity<TResult>
// and return the value from the Execute method.
protected override void Execute(CodeActivityContext context)
{
// Obtain the runtime value of the Text input argument
ConvertToCubeService.ServiceClient client = new ConvertToCubeService.ServiceClient();
ConvertToCubeService.ConvertToCubeFaces param = new ConvertToCubeService.ConvertToCubeFaces();
var apPath = System.Web.Hosting.HostingEnvironment.ApplicationPhysicalPath;
param.EquirectangularFilePath=apPath+context.GetValue(this.EquirectangularImagePath);
param.imageName = context.GetValue(this.ImageName);
param.SaveCubicFacesDirectory = apPath + context.GetValue(this.SaveImageDirectoryPath);
client.ConvertToCubeFacesCompleted += new EventHandler<ConvertToCubeService.ConvertToCubeFacesCompletedEventArgs>(client_ConvertToCubeFacesCompleted);
client.ConvertToCubeFacesAsync(param);//call async the server to do the work
using (var db = new panonestEntities())
{
var p = db.Panoramas.Where(x => x.ImageName.Equals(param.imageName)).First();
p.Status = "Called processing. wait";
db.SaveChanges();
}
}
void client_ConvertToCubeFacesCompleted(object sender, ConvertToCubeService.ConvertToCubeFacesCompletedEventArgs e)
{
using (var db = new panonestEntities())
{
}
}
}
You shouldn't run something that takes 10 minutes on your web server. You should write a windows service for that. You could have a simple webservice that puts a record into a table to trigger the windows service to do its work
.
I think better way would be not to do the async stuff client side but server side because when you ignore the response, you won't know if the call even has reached the server.
I would wirte the server operation in that way, that it starts the long running task asynchronous and then returns quickly.
The client can wait on the call and can be sure, that the call arrived at the server successfully and the operation was started.