I'm using Hangfire for background jobs, and Serilog for logging. I'm trying to enrich my serilogs with a TrackingId so that all logs from a specific Hangfire job will have the same TrackingId that I can filter on.
I configure Serilog like this in Startup.cs:
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(Configuration)
.WriteTo.Seq(serverUrl: serverUrl, apiKey: apiKey)
// Enrich the logs with a tracking id. Will be a new value per request
.Enrich.WithProperty("TrackingId", Guid.NewGuid())
.CreateLogger();
And I enqueue jobs like this:
BackgroundJob.Enqueue<MyService>(myService => myService.DoIt(someParameter));
But doing like this will not set a separate TrackingId per Hangfire job. Is there any way I can achieve that?
For what it's worth, I ended up pulling this off using the server/client filter and GlobalJobFilters registration shown below. One annoying issue I ran into is that the AutomaticRetryAttribute is added by default to the GlobalJobFilters collection, and that class will log errors for failed jobs without knowledge of the Serilog LogContext created in our custom JobLoggerAttribute. Personally, I know I will only allow manual retry, so I just removed that attribute and handled the error within the IServerFilter.OnPerformed method. Check the end of my post to see how to remove it if that works for you.
If you are going to allow automatic retry, then you will need to: 1) create a custom attribute that decorates the AutomaticRetryAttribute and makes it aware of a custom LogContext, 2) again remove the default AutomaticRetryAttribute from the GlobalJobFilters collection, and 3) add your decorator attribute to the collection.
public class JobLoggerAttribute : JobFilterAttribute, IClientFilter, IServerFilter
{
private ILogger _log;
public void OnCreating(CreatingContext filterContext)
{
_log = GetLogger();
_log.Information("Job is being created for {JobType} with arguments {JobArguments}", filterContext.Job.Type.Name, filterContext.Job.Args);
}
public void OnCreated(CreatedContext filterContext)
{
_log.Information("Job {JobId} has been created.", filterContext.BackgroundJob.Id);
}
public void OnPerforming(PerformingContext filterContext)
{
if (_log == null)
_log = GetLogger();
_log.Information("Job {JobId} is performing.", filterContext.BackgroundJob.Id);
}
public void OnPerformed(PerformedContext filterContext)
{
_log.Information("Job {JobId} has performed.", filterContext.BackgroundJob.Id);
if (filterContext.Exception != null)
{
_log.Error(
filterContext.Exception,
"Job {JobId} failed due to an exception.",
filterContext.BackgroundJob.Id);
}
_log = null;
}
private ILogger GetLogger()
{
return Log.ForContext(GetType()).ForContext("HangfireRequestId", Guid.NewGuid());
}
}
And the registration...
GlobalJobFilters.Filters.Add(new JobLoggerAttribute());
Removing the AutomaticRetryAttribute...
var automaticRetryFilter = GlobalJobFilters.Filters.Where(x => x.Instance is AutomaticRetryAttribute).Single();
GlobalJobFilters.Filters.Remove(automaticRetryFilter.Instance);
Related
Hangfire DisableConcurrentExecution attribute not working as expected.
I have one method and that can be called with different Id. I want to prevent concurrent execution of method if same Id is passed.
string jobName= $"{Id} - Entry Job";
_recurringJobManager.AddOrUpdate<EntryJob>(jobName, j => j.RunAsync(Id, Null), "0 2 * * *");
My EntryJob interface having RunAsync method.
public class EntryJob: IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
public async Task RunAsync(int Id, SomeObj obj)
{
//Some coe
}
}
And interface look like this
[DisableConcurrentExecution(3600)] <-- Tried here
public interface IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
Task RunAsync(int Id, SomeObj obj);
}
Now I want to prevent RunAsync method to call multiple times if Id is same. I have tried to put DisableConcurrentExecution on top of the RunAsync method at both location inside interface declaration and also from where Interface is implemented.
But it seems like not working for me. Is there any way to prevent concurrency based on Id?
The existing implementation of DisableConcurrentExecution does not support this. It will prevent concurrent executions of the method with any args. It would be fairly simple to add support in. Note below is untested pseudo-code:
public class DisableConcurrentExecutionWithArgAttribute : JobFilterAttribute, IServerFilter
{
private readonly int _timeoutInSeconds;
private readonly int _argPos;
// add additional param to pass in which method arg you want to use for
// deduping jobs
public DisableConcurrentExecutionAttribute(int timeoutInSeconds, int argPos)
{
if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");
_timeoutInSeconds = timeoutInSeconds;
_argPos = argPos;
}
public void OnPerforming(PerformingContext filterContext)
{
var resource = GetResource(filterContext.BackgroundJob.Job);
var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);
var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
filterContext.Items["DistributedLock"] = distributedLock;
}
public void OnPerformed(PerformedContext filterContext)
{
if (!filterContext.Items.ContainsKey("DistributedLock"))
{
throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
}
var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
distributedLock.Dispose();
}
private static string GetResource(Job job)
{
// adjust locked resource to include the argument to make it unique
// for a given ID
return $"{job.Type.ToGenericTypeString()}.{job.Method.Name}.{job.Args[_argPos].ToString()}";
}
}
I want to schedule a task on Hazelcast that runs at a fixed interval and updates the IMap with some data that I get after hitting a rest endpoint. Below is a sample code:
// Main class
IScheduledExecutorService service = hazelcast.getScheduledExecutorService("default");
service.scheduleAtFixedRate(TaskUtils.named("my-task", myTask), 30, 1);
// Task
#Singleton
public class MyTask implements Runnable, Serializable {
RestClient restClient;
IMap<String, JsonObject> map;
#Inject
MyTask() { // Inject hazelcast and restclient
map = hazelcastInstace.getMap("my-map");
this.restClient = restClient;
}
#Override
public void run() {
Collection<JSONObject> values = map.values(new MyCustomFilter());
for(JSONObject obj : values) {
// query endpoint based on id
map.submitToKey(key, response);
}
}
private static class MyCustomFilter implements Predicate<String, JSONObject> {
public boolean apply(Map.Entry<String, JSONObject> map) {
// logic to filter relevant keys
}
}
}
When I try to execute this on the cluster, I get:
java.io.NotSerializableException: com.hazelcast.map.impl.proxy.MapProxyImpl
Now I need the IMap to selectively query only some keys based on PredicateFilter and this needs to be a background scheduled job so stuck here on how to take this forward. Any help appreciated. TIA
Try making your task also implement HazelcastInstanceAware
When you submit your task, it is serialized, sent to the grid to run, deserialized when it is received, and the run() method is called.
If your task implements HazelcastInstanceAware, then between deserialization and run(), Hazelcast will call the method setHazelcastInstance(HazelcastInstance instance) to pass your code a reference to the particular Hazelcast instance it is running in. From there you can just do instance.getMap("my-map") and store the map reference in a transient field that the run() method can use.
i'm trying to add additional Baggage to the existing span on a HTTP server, i want to add a path variable to the span to be accessed from log MDC and to be propagated on the wire to the next server i call via http or kafka.
my setup : spring cloud sleuth Hoxton.SR5 and spring boot 2.2.5
i tried adding the following setup and configuration:
spring:
sleuth:
propagation-keys: context-id, context-type
log:
slf4j:
whitelisted-mdc-keys: context-id, context-type
and added http interceptor :
public class HttpContextInterceptor implements HandlerInterceptor {
private final Tracer tracer;
private final HttpContextSupplier httpContextSupplier;
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
if (httpContextSupplier != null) {
addContext(request, handler);
}
return true;
}
private void addContext(HttpServletRequest request, Object handler) {
final Context context = httpContextSupplier.getContext(request);
if (!StringUtils.isEmpty(context.getContextId())) {
ExtraFieldPropagation.set(tracer.currentSpan().context(), TracingHeadersConsts.HEADER_CONTEXT_ID, context.getContextId());
}
if (!StringUtils.isEmpty(context.getContextType())) {
ExtraFieldPropagation.set(tracer.currentSpan().context(), TracingHeadersConsts.HEADER_CONTEXT_TYPE, context.getContextType());
}
}
}
and http filter to affect the current span(according to the spring docs)
public class TracingFilter extends OncePerRequestFilter {
private final Tracer tracer;
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
try (Tracer.SpanInScope ws = tracer.withSpanInScope(tracer.currentSpan())){
filterChain.doFilter(request, response);
}
}
}
the problem is the logs doesn't contain my custom context-id, context-type, although is see it in the span context.
what i'm missing ?
Similar question Spring cloud sleuth adding tag
and answer to it https://stackoverflow.com/a/66554834
For some context: This is from the Spring Docs.
In order to automatically set the baggage values to Slf4j’s MDC, you have to set the spring.sleuth.baggage.correlation-fields property with a list of allowed local or remote keys. E.g. spring.sleuth.baggage.correlation-fields=country-code will set the value of the country-code baggage into MDC.
Note that the extra field is propagated and added to MDC starting with the next downstream trace context. To immediately add the extra field to MDC in the current trace context, configure the field to flush on update.
// configuration
#Bean
BaggageField countryCodeField() {
return BaggageField.create("country-code");
}
#Bean
ScopeDecorator mdcScopeDecorator() {
return MDCScopeDecorator.newBuilder()
.clear()
.add(SingleCorrelationField.newBuilder(countryCodeField())
.flushOnUpdate()
.build())
.build();
}
// service
#Autowired
BaggageField countryCodeField;
countryCodeField.updateValue("new-value");
A way to flush MDC in current span is also described in official Sleuth 2.0 -> 3.0 migration guide
#Configuration
class BusinessProcessBaggageConfiguration {
BaggageField BUSINESS_PROCESS = BaggageField.create("bp");
/** {#link BaggageField#updateValue(TraceContext, String)} now flushes to MDC */
#Bean
CorrelationScopeCustomizer flushBusinessProcessToMDCOnUpdate() {
return b -> b.add(
SingleCorrelationField.newBuilder(BUSINESS_PROCESS).flushOnUpdate().build()
);
}
}
I have a WCF service which has its Thread.CurrentPrincipal set in the ServiceConfiguration.ClaimsAuthorizationManager.
When I implement the service asynchronously like this:
public IAsyncResult BeginMethod1(AsyncCallback callback, object state)
{
// Audit log call (uses Thread.CurrentPrincipal)
var task = Task<int>.Factory.StartNew(this.WorkerFunction, state);
return task.ContinueWith(res => callback(task));
}
public string EndMethod1(IAsyncResult ar)
{
// Audit log result (uses Thread.CurrentPrincipal)
return ar.AsyncState as string;
}
private int WorkerFunction(object state)
{
// perform work
}
I find that the Thread.CurrentPrincipal is set to the correct ClaimsPrincipal in the Begin-method and also in the WorkerFunction, but in the End-method it's set to a GenericPrincipal.
I know I can enable ASP.NET compatibility for the service and use HttpContext.Current.User which has the correct principal in all methods, but I'd rather not do this.
Is there a way to force the Thread.CurrentPrincipal to the correct ClaimsPrincipal without turning on ASP.NET compatibility?
Starting with a summary of WCF extension points, you'll see the one that is expressly designed to solve your problem. It is called a CallContextInitializer. Take a look at this article which gives CallContextInitializer sample code.
If you make an ICallContextInitializer extension, you will be given control over both the BeginXXX thread context AND the EndXXX thread context. You are saying that the ClaimsAuthorizationManager has correctly established the user principal in your BeginXXX(...) method. In that case, you then make for yourself a custom ICallContextInitializer which either assigns or records the CurrentPrincipal, depending on whether it is handling your BeginXXX() or your EndXXX(). Something like:
public object BeforeInvoke(System.ServiceModel.InstanceContext instanceContext, System.ServiceModel.IClientChannel channel, System.ServiceModel.Channels.Message request){
object principal = null;
if (request.Properties.TryGetValue("userPrincipal", out principal))
{
//If we got here, it means we're about to call the EndXXX(...) method.
Thread.CurrentPrincipal = (IPrincipal)principal;
}
else
{
//If we got here, it means we're about to call the BeginXXX(...) method.
request.Properties["userPrincipal"] = Thread.CurrentPrincipal;
}
...
}
To clarify further, consider two cases. Suppose you implemented both an ICallContextInitializer and an IParameterInspector. Suppose that these hooks are expected to execute with a synchronous WCF service and with an async WCF service (which is your special case).
Below are the sequence of events and the explanation of what is happening:
Synchronous Case
ICallContextInitializer.BeforeInvoke();
IParemeterInspector.BeforeCall();
//...service executes...
IParameterInspector.AfterCall();
ICallContextInitializer.AfterInvoke();
Nothing surprising in the above code. But now look below at what happens with asynchronous service operations...
Asynchronous Case
ICallContextInitializer.BeforeInvoke(); //TryGetValue() fails, so this records the UserPrincipal.
IParameterInspector.BeforeCall();
//...Your BeginXXX() routine now executes...
ICallContextInitializer.AfterInvoke();
//...Now your Task async code executes (or finishes executing)...
ICallContextInitializercut.BeforeInvoke(); //TryGetValue succeeds, so this assigns the UserPrincipal.
//...Your EndXXX() routine now executes...
IParameterInspector.AfterCall();
ICallContextInitializer.AfterInvoke();
As you can see, the CallContextInitializer ensures you have opportunity to initialize values such as your CurrentPrincipal just before the EndXXX() routine runs. It therefore doesn't matter that the EndXXX() routine assuredly is executing on a different thread than did the BeginXXX() routine. And yes, the System.ServiceModel.Channels.Message object which is storing your user principal between Begin/End methods, is preserved and properly transmitted by WCF even though the thread changed.
Overall, this approach allows your EndXXX(IAsyncresult) to execute with the correct IPrincipal, without having to explicitly re-establish the CurrentPrincipal in the EndXXX() routine. And as with any WCF behavior, you can decide if this applies to individual operations, all operations on a contract, or all operations on an endpoint.
Not really the answer to my question, but an alternate approach of implementing the WCF service (in .NET 4.5) that does not exhibit the same issues with Thread.CurrentPrincipal.
public async Task<string> Method1()
{
// Audit log call (uses Thread.CurrentPrincipal)
try
{
return await Task.Factory.StartNew(() => this.WorkerFunction());
}
finally
{
// Audit log result (uses Thread.CurrentPrincipal)
}
}
private string WorkerFunction()
{
// perform work
return string.Empty;
}
The valid approach to this is to create an extension:
public class SLOperationContext : IExtension<OperationContext>
{
private readonly IDictionary<string, object> items;
private static ReaderWriterLockSlim _instanceLock = new ReaderWriterLockSlim();
private SLOperationContext()
{
items = new Dictionary<string, object>();
}
public IDictionary<string, object> Items
{
get { return items; }
}
public static SLOperationContext Current
{
get
{
SLOperationContext context = OperationContext.Current.Extensions.Find<SLOperationContext>();
if (context == null)
{
_instanceLock.EnterWriteLock();
context = new SLOperationContext();
OperationContext.Current.Extensions.Add(context);
_instanceLock.ExitWriteLock();
}
return context;
}
}
public void Attach(OperationContext owner) { }
public void Detach(OperationContext owner) { }
}
Now this extension is used as a container for objects that you want to persist between thread switching as OperationContext.Current will remain the same.
Now you can use this in BeginMethod1 to save current user:
SLOperationContext.Current.Items["Principal"] = OperationContext.Current.ClaimsPrincipal;
And then in EndMethod1 you can get the user by typing:
ClaimsPrincipal principal = SLOperationContext.Current.Items["Principal"];
EDIT (Another approach):
public IAsyncResult BeginMethod1(AsyncCallback callback, object state)
{
var task = Task.Factory.StartNew(this.WorkerFunction, state);
var ec = ExecutionContext.Capture();
return task.ContinueWith(res =>
ExecutionContext.Run(ec, (_) => callback(task), null));
}
I'm having an issue getting OutputCaching to work with HttpContext.RewritePath for a WCF 4.0 WebHttp service.
My service is localized. The idea is that you call a URL like so:
/languageCode/ServiceName/Method
e.g.
/en/MyService/GetItems
And it'll return the results localized to the correct language.
My scheme is based on this article. The idea is to create a derivative of RouteBase that creates a unique, "private" route to the real service. When the user makes a request, the language code is unpacked from the URL and set as the culture for the current thread, and then HttpContext.RewritePath is used to load the actual service.
For the life of me I can't figure out how to work OutputCaching into the mix. I've decorated my service method with AspNetCacheProfile and am seeing my own VaryByCustom override called. However despite receiving a duplicate result from VaryByCustom, .NET continues into my service method anyway.
Lots of code below, sorry for the dump but I suspect it's all relevant.
How I add a route in Global.asax.cs
RouteTable.Routes.Add(new CulturedServiceRoute(
"devices",
new StructureMapServiceHostFactory(),
typeof(DeviceService)));
VaryByCustom override in Global.asax.cs:
public override string GetVaryByCustomString(
HttpContext context, string custom)
{
// This method gets called twice: Once for the initial request, then a
// second time for the rewritten URL. I only want it to be called once!
if (custom == "XmlDataFreshness")
{
var outputString = String.Format("{0}|{1}|{2}",
XmlDataLoader.LastUpdatedTicks,
context.Request.RawUrl,
context.Request.HttpMethod);
return outputString;
}
return base.GetVaryByCustomString(context, custom);
}
This is the dynamic service route class.
public class CulturedServiceRoute : RouteBase, IRouteHandler
{
private readonly string _virtualPath = null;
private readonly ServiceRoute _innerServiceRoute = null;
private readonly Route _innerRoute = null;
public CulturedServiceRoute(
string pathPrefix,
ServiceHostFactoryBase serviceHostFactory,
Type serviceType)
{
if (pathPrefix.IndexOf("{") >= 0)
{
throw new ArgumentException(
"Path prefix cannot include route parameters.",
"pathPrefix");
}
if (!pathPrefix.StartsWith("/")) pathPrefix = "/" + pathPrefix;
pathPrefix = "{culture}" + pathPrefix;
_virtualPath = String.Format("Cultured/{0}/", serviceType.FullName);
_innerServiceRoute = new ServiceRoute(
_virtualPath, serviceHostFactory, serviceType);
_innerRoute = new Route(pathPrefix, this);
}
public override RouteData GetRouteData(
HttpContextBase httpContext)
{
return _innerRoute.GetRouteData(httpContext);
}
public override VirtualPathData GetVirtualPath(
RequestContext requestContext, RouteValueDictionary values)
{
return null;
}
public IHttpHandler GetHttpHandler(RequestContext requestContext)
{
// This method is called even if VaryByCustom
// returns a duplicate response!
var culture = requestContext.RouteData.Values["culture"].ToString();
var ci = new CultureInfo(culture);
Thread.CurrentThread.CurrentUICulture = ci;
Thread.CurrentThread.CurrentCulture =
CultureInfo.CreateSpecificCulture(ci.Name);
requestContext.HttpContext.RewritePath("~/" + _virtualPath, true);
return _innerServiceRoute.RouteHandler.GetHttpHandler(requestContext);
}
}
Finally, the relevant portions of the service itself:
[ServiceContract]
[AspNetCompatibilityRequirements(
RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
public class DeviceService
{
[AspNetCacheProfile("MyCacheProfile")]
[WebGet(UriTemplate = "")]
public IEnumerable<DeviceListItemModel> GetDevices()
{
// This is called AFTER the first VaryByCustom override is called.
// I'd expect it not to be called unless VaryByCustom changes!
var devices =
from d in _deviceRepository.GetAll()
where d.ReleaseDate < DateTime.Now
orderby d.Id descending
select new DeviceListItemModel(d);
return devices;
}
UPDATE: My cache profile:
<caching>
<outputCacheSettings>
<outputCacheProfiles>
<add name="MyCacheProfile" varyByCustom="XmlDataFreshness"
varyByHeader="accept" varyByParam="*" location="Server"
duration="3600" />
</outputCacheProfiles>
</outputCacheSettings>
</caching>
Hmmm seems like a valid approach to me. Is the cache profile configured correctly? Is varyByCustom called multiple times and certain to return the same result when the cache does not need to be updated?