While following Crm system Saga Example in Rebus documentation I am getting a ConcurrencyException with the following Details when handling the event LegalInfoAcquiredInFirstSystem and LegalInfoAcquiredInSecondSystem.
Error Details are as follows
message_id: 150d3d50-1de4-4a2f-bd60-660fc441412e
delivery_mode: 2
headers:
rbs2-content-type: application/json;charset=utf-8
rbs2-corr-id: 0848197f-3ebb-442a-a80b-49c4c30dc0ca
rbs2-corr-seq: 2
rbs2-error-details: System.AggregateException: 1 unhandled exceptions (Update of saga with ID 95395c60-cf2e-48da-89ff-fdf192ce53b9 did not succeed because someone else beat us to it)
---> Rebus.Exceptions.ConcurrencyException: Update of saga with ID 95395c60-cf2e-48da-89ff-fdf192ce53b9 did not succeed because someone else beat us to it
at Rebus.SqlServer.Sagas.SqlServerSagaStorage.Update(ISagaData sagaData, IEnumerable1 correlationProperties) at Rebus.Sagas.LoadSagaDataStep.SaveSagaData(RelevantSagaInfo sagaDataToUpdate, Boolean insert) at Rebus.Sagas.LoadSagaDataStep.SaveSagaData(RelevantSagaInfo sagaDataToUpdate, Boolean insert) at Rebus.Sagas.LoadSagaDataStep.Process(IncomingStepContext context, Func1 next)
at Rebus.Pipeline.Receive.ActivateHandlersStep.Process(IncomingStepContext context, Func1 next) at Rebus.Pipeline.Receive.HandleRoutingSlipsStep.Process(IncomingStepContext context, Func1 next)
at Rebus.Pipeline.Receive.DeserializeIncomingMessageStep.Process(IncomingStepContext context, Func1 next) at Rebus.DataBus.ClaimCheck.HydrateIncomingMessageStep.Process(IncomingStepContext context, Func1 next)
at Rebus.Pipeline.Receive.HandleDeferredMessagesStep.Process(IncomingStepContext context, Func1 next) at Rebus.Retry.FailFast.FailFastStep.Process(IncomingStepContext context, Func1 next)
at Rebus.Retry.Simple.SimpleRetryStrategyStep.DispatchWithTrackerIdentifier(Func`1 next, String identifierToTrackMessageBy, ITransactionContext transactionContext, String messageId, String secondLevelMessageId)
--- End of inner exception stack trace ---
rbs2-intent: pub
rbs2-msg-id: 150d3d50-1de4-4a2f-bd60-660fc441412e
rbs2-msg-type: Crm.Messages.Events.LegalInfoAcquiredInFirstSystem, Crm.Messages.Events
rbs2-return-address: RebusQueue
rbs2-sender-address: RebusQueue
rbs2-senttime: 2021-08-07T23:27:49.8601606+03:00
rbs2-source-queue: RebusQueue
content_encoding: utf-8
content_type: application/json
Payload
100 bytes
Encoding: string
{"$type":"Crm.Messages.Events.LegalInfoAcquiredInFirstSystem, Crm.Messages.Events","CorrId":"70001"}
My Rebus Configuration are as follows:
``
public void ConfigureServices(IServiceCollection services)
{
AppSettings settings = new AppSettings();
Configuration.Bind(settings);
services.AutoRegisterHandlersFromAssemblyOf<AcquireLegalInformationFromFirstSystemHandler>();
services.AddControllers();
services.AddLogging(logging => logging.AddConsole());
services.AddRebus((configure, serviceProvider) => configure
.Transport(x =>
{
x.UseRabbitMq($"amqp://{settings.Settings.UserName}:{settings.Settings.Password}#{settings.Settings.HostName}", settings.Settings.EndpointQueueName);
})
.Options(o => o.SetBusName("RebusSaga"))
.Options(o => o.SimpleRetryStrategy(errorQueueAddress: settings.Settings.ErrorQueueName, maxDeliveryAttempts: 1, secondLevelRetriesEnabled: false))
.Sagas(s =>
{
s.StoreInSqlServer(settings.ConnectionStrings.RebusContext, "Sagas", "SagaIndex", true);
})
.Timeouts(s => s.StoreInSqlServer(settings.ConnectionStrings.RebusContext, "Timeouts",true))
.Routing(r => r.TypeBased()
.MapAssemblyOf<Crm.Messages.Events.CustomerCreated>(settings.Settings.EndpointQueueName)
.MapFallback("RebusErrors"))
);
}
``
The Error Occurs in the Saga handlers below
``
public async Task Handle(LegalInfoAcquiredInFirstSystem first)
{
Data.GotLegalInfoFromFirstSystem = true;
await PossiblyPerformCompleteAction();
}
public async Task Handle(LegalInfoAcquiredInSecondSystem first)
{
Data.GotLegalInfoFromSecondSystem = true;
await PossiblyPerformCompleteAction();
}
async Task PossiblyPerformCompleteAction()
{
if (Data.GotLegalInfoFromFirstSystem && Data.GotLegalInfoFromSecondSystem)
{
await bus.Publish(new CustomerIsLegallyOk { CrmCustomerId = Data.CrmCustomerId });
MarkAsComplete();
}
}
``
What could be the possible source of this error.
Thanks
I finally solved the problem by adding s.EnforceExclusiveAccess(); to the Saga options.
Related
I've decided to write a custom AuthorizationHandler for a custom Policy I'm using :
// I pass this to AddPolicy in startup.cs
public class MyRequirement : IAuthorizationRequirement {
public MyRequirement () { ... }
}
public class MyAuthorizationHandler : AuthorizationHandler<MyRequirement> {
public MyAuthorizationHandler() { }
protected override async Task HandleRequirementAsync(AuthorizationHandlerContext context, MyRequirement requirement) {
if (context.Resource is HttpContext httpContext) {
var endpoint = httpContext.GetEndpoint();
if ( /* conditions for hard failure */ ) { context.Fail(); return; }
if ( /* conditions for success */) { context.Succeed(requirement); return; }
// Neither a success nor a failure, simply a different response.
httpContext.Response.StatusCode = 404;
httpContext.Response.ContentType = "application/json";
await httpContext.Response.WriteAsync("Blah blah NotFound").ConfigureAwait(false);
return;
}
context.Fail();
}
}
I've seen similar code snippets in other StackOverlflow answers. (e.g. here : How to change status code & add message from failed AuthorizationHandler policy )
Problem : this doesn't seem to generate a "valid" 404 response.
I think so for two reasons:
When I look at Chrome's network tab, the response is NOT "404", instead it's net::ERR_HTTP2_PROTOCOL_ERROR 404
When I look at the response data, there's only headers. My custom error text ("Blah blah NotFound") does not appear anywhere.
What am I doing wrong?
Note : I've tried returning immediately after setting the 404, without doing context.Fail() but I get the same result.
The root cause:
My Web Api had several middlewares working with the response value. Those middleware were chained up in Startup.cs, using the traditional app.UseXXX().
Chrome was receiving 404 (along with my custom response body) from my Requirement middleware (hurray!), but Chrome is "smart" and by design continues to receive the response even after that initial 404 -- for as long as the server continues generating some response data.
Because of that, Chrome eventually came across a different response added by another of the chained up middlewares. The 404 was still there, but the response body was slightly changed.
And since chrome is paranoid, it would display this net::ERR_HTTP2_PROTOCOL_ERROR to indicate that someone had messed up the consistency of the response somewhere along the chain of responders.
==========
The solution :
Finalize your response with Response.CompleteAsync() to prevent any other subsequent middleware from changing it further :
protected override async Task HandleRequirementAsync(AuthorizationHandlerContext context, MyRequirement requirement) {
if (context.Resource is HttpContext httpContext) {
var endpoint = httpContext.GetEndpoint();
if ( /* conditions for hard failure */ ) { context.Fail(); return; }
if ( /* conditions for success */) {
context.Succeed(requirement);
return;
}
// Neither a requirement success nor a requirement failure, just a different response :
httpContext.Response.StatusCode = 404;
httpContext.Response.ContentType = "application/json";
await httpContext.Response.WriteAsync("Blah blah NotFound");
await httpContext.Response.CompleteAsync(); // <-- THIS!!!
return;
}
context.Fail();
}
Please note : if your 'HandleRequirementAsync' function does not have the 'async' keyword, then do not use 'await' inside of it, and do return Task.CompletedTask; instead of just return;
Below is a work demo based on your code, you can refer to it.
public class MyAuthorizationHandler : AuthorizationHandler<MyRequirement>
{
private readonly IHttpContextAccessor httpContextAccessor;
public MyAuthorizationHandler(IHttpContextAccessor httpContextAccessor)
{
this.httpContextAccessor = httpContextAccessor;
}
protected override Task HandleRequirementAsync(AuthorizationHandlerContext context, MyRequirement requirement)
{
if ( /* conditions for success */) {
context.Succeed(requirement);
return;
}
// If it fails at this point, I want to return 404 because of reasons.
else
{
var httpContext = httpContextAccessor.HttpContext;
httpContext.Response.StatusCode = 404;
httpContext.Response.ContentType = "application/json";
httpContext.Response.WriteAsync("Blah blah NotFound").ConfigureAwait(false);
}
return Task.CompletedTask;
}
}
result:
I am trying out the new MassTransit IJobConsumer implementation, and although I've tried to follow the documentation, the JobConsumer I have written is never being run/hit.
I have:
created the JobConsumer which has a run method that runs the code I need it to
public class CalculationStartRunJobConsumer : IJobConsumer<ICalculationStartRun>
{
private readonly ICalculationRunQueue runQueue;
public CalculationStartRunJobConsumer(ICalculationRunQueue runQueue)
{
this.runQueue = runQueue;
}
public Task Run(JobContext<ICalculationStartRun> context)
{
return Task.Run(
() =>
{
var longRunningJob = new LongRunningJob<ICalculationStartRun>
{
Job = context.Job,
CancellationToken = context.CancellationToken,
JobId = context.JobId,
};
runQueue.StartSpecial(longRunningJob);
},
context.CancellationToken);
}
}
I have registered that consumer trying both ConnectReceiveEndpoint and AddConsumer
Configured the ServiceInstance as shown in the documentation
services.AddMassTransit(busRegistrationConfigurator =>
{
// TODO: Get rid of this ugly if statement.
if (consumerTypes != null)
{
foreach (var consumerType in consumerTypes)
{
busRegistrationConfigurator.AddConsumer(consumerType);
}
}
if(requestClientType != null)
{
busRegistrationConfigurator.AddRequestClient(requestClientType);
}
busRegistrationConfigurator.UsingRabbitMq((context, cfg) =>
{
cfg.UseNewtonsoftJsonSerializer();
cfg.UseNewtonsoftJsonDeserializer();
cfg.ConfigureNewtonsoftJsonSerializer(settings =>
{
// The serializer by default omits fields that are set to their default value, but this causes unintended effects
settings.NullValueHandling = NullValueHandling.Include;
settings.DefaultValueHandling = DefaultValueHandling.Include;
return settings;
});
cfg.Host(
messagingHostInfo.HostAddress,
hostConfigurator =>
{
hostConfigurator.Username(messagingHostInfo.UserName);
hostConfigurator.Password(messagingHostInfo.Password);
});
cfg.ServiceInstance(instance =>
{
instance.ConfigureJobServiceEndpoints(serviceCfg =>
{
serviceCfg.FinalizeCompleted = true;
});
instance.ConfigureEndpoints(context);
});
});
});
Seen that the queue for the job does appear in the queue for RabbitMQ
When I call .Send to send a message to that queue, it does not activate the Run method on the JobConsumer.
public async Task Send<T>(string queueName, T message) where T : class
{
var endpointUri = GetEndpointUri(messagingHostInfo.HostAddress, queueName);
var sendEndpoint = await bus.GetSendEndpoint(endpointUri);
await sendEndpoint.Send(message);
}
Can anyone help?
Software
MassTransit 8.0.2
MassTransit.RabbitMq 8.0.2
MassTransit.NewtonsoftJson 8.0.2
.NET6
Using in-memory for JobConsumer
The setup of any type of repository for long running jobs is missing. We needed to either:
explicitly specify that it was using InMemory (missing from the docs)
Setup saga repositories using e.g. EF Core.
As recommended by MassTransit, we went with the option of setting up saga repositories by implementing databases and interacting with them using EF Core.
I made two methods for testing. GetAllUsers working but GetAllRoles throws an error like below. I don't get why this is happening? and how do I fix this?
Update
I solved the problem by adding a nuget package NewJson, but I am getting the new error below.
Error
A possible object cycle was detected which is not supported. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32.
new error
A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
SQLConnection with DB
public void Configure(IWebHostBuilder builder)
{
builder.ConfigureServices((context, services) =>
{
services.AddDbContext<CDSPORTALContext>(options =>
{
options.UseSqlServer(
context.Configuration.GetConnectionString("CDSPORTALContextConnection"));
}, ServiceLifetime.Transient );
});
}
AcountController
[HttpGet("getAllUsers")]
public async Task<IActionResult> GetAllUsers()
{
var users = await _userManager.Users.ToListAsync();
return Ok(users);
}
[HttpGet("/getAllRoles")]
public async Task<IActionResult> GetAllRoles()
{
var roles = await _roleManager.Roles.ToListAsync();
//var roles = await _appDbContext.Roles.ToListAsync();
return Ok(roles);
}
I have a scenario where I call an api in one of my handlers and that Api can go down for like 6 hours per month. Therefore, I designed a retry logic with 1sec retry, 1 minute retry and a 6 hour retry. This all works fine but then I found that long delay retries are not a good option.Could you please give me your experience about this?
Thank you!
If I were you, I would use Rebus' ability to defer messages to the future to implement this functionality.
You will need to track the number of failed delivery attempts manually though, by attaching and updating headers on the deferred message.
Something like this should do the trick:
public class YourHandler : IHandleMessages<MakeExternalApiCall>
{
const string DeliveryAttemptHeaderKey = "delivery-attempt";
public YourHandler(IMessageContext context, IBus bus)
{
_context = context;
_bus = bus;
}
public async Task Handle(MakeExternalApiCall message)
{
try
{
await MakeCallToExternalWebApi();
}
catch(Exception exception)
{
var deliveryAttempt = GetDeliveryAttempt();
if (deliveryAttempt > 5)
{
await _bus.Advanced.TransportMessage.Forward("error");
}
else
{
var delay = GetNextDelay(deliveryAttempt);
var headers = new Dictionary<string, string> {
{DeliveryAttemptHeaderKey, (deliveryAttempt+1).ToString()}
};
await bus.Defer(delay.Value, message, headers);
}
}
}
int GetDeliveryAttempt() => _context.Headers.TryGetValue(DeliveryAttemptHeaderKey, out var deliveryAttempt)
? deliveryAttempt
: 0;
TimeSpan GetNextDelay() => ...
}
When running in production, please remember to configure some kind of persistent subscription storage – e.g. SQL Server – otherwise, your deferred messages will be lost in the event of a restart.
You can configure it like this (after having installed the Rebus.SqlServer package):
Configure.With(...)
.(...)
.Timeouts(t => t.StoreInSqlServer(...))
.Start();
I am new to MassTransit and messaging. I am trying to write a unit test for a IBus.Publish and am not able to succeed with the result.
I am observing the fault from RabbitMQ and my observer looks like this:
public class FaultObserver : IReceiveObserver
{
public FaultObserver(IRequestUpdater statusUpdater,Lazy<IBus> bus)
{
this.statusUpdater = statusUpdater;
this.bus = bus;
}
public async Task ConsumeFault<T>(ConsumeContext<T> context, TimeSpan duration, string consumerType, Exception exception) where T : class
{
}
}
and my tests looks like the below code
var bus = fixture.Freeze<Mock<IBus>>();
bus.Setup(bu => bu.Publish<ReportFailedEvent>(It.IsAny<ReportFailedEvent>(),It.IsAny<CancellationToken>())).Verifiable();
var sut = fixture.Create<ReportRequestedFaultObserver>();
// Act
await sut.ConsumeFault(context.Object,new TimeSpan(0,0,1),string.Empty,exception);
// Assert
//bus.Verify(b => b.Publish<ReportFailedEvent>(It.IsAny<ReportFailedEvent>(), It.IsAny<CancellationToken>()), Times.Exactly(1));
bus.Verify(b =>b.Publish<ReportFailedEvent>(new ReportFailedEvent(request,exception.Message),It.IsAny<CancellationToken>()),Times.Once());
my setup looks like
[SetUp]
public void SetUp()
{
fixture.Customize(new AutoMoqCustomization());
var inMemoryTransportCache = new InMemoryTransportCache(Environment.ProcessorCount);
bus = Bus.Factory.CreateUsingInMemory(configure =>
{
configure.SetTransportProvider(inMemoryTransportCache);
configure.ReceiveEndpoint("input_queue", configurator =>
{
configurator.Handler<(cc =>
{
});
});
});
bus.Start().Ready.Wait();
}
I am not able to mock my Publish method. Does anyone knows what I am doing wrong?
Your mock for the bus is wrong. It should be:
bus.Setup(bu => bu.Publish<ReportFailedEvent>(
It.IsAny<object>(), It.IsAny<CancellationToken>()));
Publish takes an object and a CancellationToken, as per the interface definition in MassTransit:
Task Publish(object message, CancellationToken cancellationToken = default(CancellationToken));
Also, if you want to check the contents of the message, you can use the Moq Callback extension:
ReportFailedEvent message = null;
bus
.Setup(bu => bu.Publish<ReportFailedEvent>(It.IsAny<object>(), It.IsAny<CancellationToken>()))
.Callback<object, CancellationToken>((a, b) => { message = a as ReportFailedEvent; });
//.. your system under test code....
Assert.That(message.Property, Is.EqualTo(expectation));
If you are setting the priority, the MassTransit interface looks like this:
public static Task Publish<T>(
this IPublishEndpoint endpoint,
T message,
Action<PublishContext<T>> callback,
CancellationToken cancellationToken = default(CancellationToken))
where T : class;
And an example would be:
bus.Publish<ReportFailedEvent>(message, context =>
{
context.SetPriority(messagePriority.Priority);
});
And the accompanying Moq is:
ReportFailedEvent message = null;
bus
.Setup(bu => bu.Publish<ReportFailedEvent>(It.IsAny<object>(), It.IaAny<Action<PublishContext<ReportFailedEvent>>>(), It.IsAny<CancellationToken>()))
.Callback<object, Action<PublishContext<ReportFailedEvent>>, CancellationToken>((a, b, c) => { message = a as ReportFailedEvent; });
Just as a side note, ideally you should be publishing the interfaces rather than classes, so IReportFailedEvent rather than ReportFailedEvent.