I need to add immovable feature to my project job scheduling solution, I have add locked property at Allocation class like this :
private boolean locked = false;
update the NotSourceOrSinkAllocationFilter class to include locked checking like this :
JobType jobType = allocation.getJob().getJobType();
return !allocation.isLocked() && jobType != JobType.SOURCE && jobType != JobType.SINK;
I tried to run it first, when the result appears, I immediately terminate the solving earlier. Then move one single allocation and set it to locked. When I run it again, it will gave me a infeasible solution.
Here is my screenshot:
initializes schedule:
adhoc decision:
try to moved it, lock it, and solve it again
unexpected result come out (it just come back to the previous allocation)
Please help me to add this immovable feature. Thanks.
Related
I´ve got a Job that runs a TaskletStep, then a chunk-based step and then another TaskletStep.
In each of these steps, errors (in the form of Exceptions) can occur.
The chunk-based step looks like this:
stepBuilderFactory
.get("step2")
.chunk<SomeItem, SomeItem>(1)
.reader(flatFileItemReader)
.processor(itemProcessor)
.writer {}
.faultTolerant()
.skipPolicy { _ , _ -> true } // skip all Exceptions and continue
.taskExecutor(taskExecutor)
.throttleLimit(taskExecutor.corePoolSize)
.build()
The whole job definition:
jobBuilderFactory.get("job1")
.validator(validator())
.preventRestart()
.start(taskletStep1)
.next(step2)
.next(taskletStep2)
.build()
I expected that Spring Batch somehow picks up the Exceptions that occur along the way, so I can then create a Report including them after the Job has finished processing. Looking at the different contexts, there´s also fields that should contain failureExceptions. However, it seems there´s no such information (especially for the chunked step).
What would be a good approach if I need information about:
what Exceptions did occur in which Job execution
which Item was the one that triggered it
The JobExecution provides a method to get all failure exceptions that happened during the job. You can use that in a JobExecutionListener#afterJob(JobExecution jobExecution) to generate your report.
In regards to which items caused the issue, this will depend on where the exception happens (during the read, process or write operation). For this requirement, you can use one of the ItemReadListener, ItemProcessListener or ItemWriteListener to keep record of the those items (For example, by adding them to the job execution context to be able to get access to them in the JobExecutionListener#afterJob method for your report).
It may happen that data that enters Flink job triggers exception either due to bug in code or lack of validation.
My goal is to provide consistent way of exception handling that our team could use within Flink jobs that won't cause any downtime in production.
Restart strategies do not seem to be applicable here as:
simple restart won't fix issue and we fall into restart loop
we cannot simply skip event
they can be good for OOME or some transient issues
we cannot add custom one
try/catch block in "keyBy" function does not fully help as:
there's no way to skip event in "keyBy" after exception is handled
Sample code:
env.addSource(kafkaConsumer)
.keyBy(keySelector) // must return one result for one entry
.flatMap(mapFunction) // we can skip some entries here in case of errors
.addSink(new PrintSinkFunction<>());
env.execute("Flink Application");
I'd like to have ability to skip processing of event that caused issue in "keyBy" and similar methods that are supposed to return exactly one result.
Beside the suggestion of #phanhuy152 (which seems totally legit to me) why not filter before keyBy?
env.addSource(kafkaConsumer)
.filter(invalidKeys)
.keyBy(keySelector) // must return one result for one entry
.flatMap(mapFunction) // we can skip some entries here in case of errors
.addSink(new PrintSinkFunction<>());
env.execute("Flink Application");
Can you reserve a special value like "NULL" for the keyBy to return in such case? Then your flatMap function can skip when encounter such value?
I have an eclipse plugin which has some performance issues. Looking into the progress view sometimes there are multiple jobs waiting and from the code most of it's arhitecture is based on classes which extend WorkspaceJobs mixed with Guava EventBus events. The current solution involves also nested jobs...
I read the documentation, I understand their purpose, but I don't get it why would I use a workspace job when I could run syncexec/asyncexec from methods which get triggered when an event is sent on the bus?
For example instead of creating 3 jobs which wait one for another, I could create an event which triggers what would have executed Job 1, then when the method is finished, it would have sent a different event type which will trigger a method that does what Job 2 would have done and so on...
So instead of:
WorkspaceJob Job1 = new WorkspaceJob("Job1");
Job1.schedule();
WorkspaceJob Job2 = new WorkspaceJob("Job2");
Job2.schedule();
WorkspaceJob Job1 = new WorkspaceJob("Job3");
Job3.schedule();
I could use:
#Subsribe
public replaceJob1(StartJob1Event event) {
//do what runInWorkspace() of Job1 would have done
com.something.getStaticEventBus().post(new Job1FinishedEvent());
}
#Subsribe
public replaceJob2(Job1FinishedEvent event) {
//do what `runInWorkspace()` of Job2 would have done
com.something.getStaticEventBus().post(new Job2FinishedEvent());
}
#Subsribe
public replaceJob3(Job2FinishedEvent event) {
//do what `runInWorkspace()` of Job3 would have done
com.something.getStaticEventBus().post(new Job3FinishedEvent());
}
I didn't tried it yet because I simplified the ideas as much as I could and the problem is more complex than that, but I think that the EventBus would win in terms of performance over the WorkspaceJobs.
Can anyone confirm my idea or tell my why this I shouldn't try this( except for the fact that I must have a good arhitecture of my events)?
WorkspaceJob delays resource change events until the job finishes. This prevents components listening for resource changes receiving half completed changes. This may or may not be important to your application.
I can't comment on the Guava code as I don't know anything about it - but note that if your code is long running you must make sure it runs in a background thread (which WorkbenchJob does).
I am using Hangfire to trigger a database retrieval operation as a background job.
This operation is only supposed to happen once, and can be triggered in multiple ways. (for example, in the UI whenever a user drags and drops a tool, I need to fire that job in the background. But if another tool is dragged and dropped, I don't want to fire the background job as it's already prefetched from the database).
This is what my code looks like now:
var jobId = BackgroundJob.Enqueue<BackgroundModelHelper>( (x) => x.PreFetchBillingByTimePeriods(organizationId) );
What I want is some kind of check before I execute above statement, to find if a background job has already been fired; if yes, then do not fire another and if not, then enqueue this .
for example:
bool prefetchIsFired = false;
// find out if a background job has already been fired. If yes, set prefetchIsFired to true.
if (!prefetchIsFired)
var jobId = BackgroundJob.Enqueue<BackgroundModelHelper>( (x) => x.PreFetchBillingByTimePeriods(organizationId, null) );
You can use a filter (DisableMultipleQueuedItemsFilter) on your job method like here : https://discuss.hangfire.io/t/how-do-i-prevent-creation-of-duplicate-jobs/1222/4
I modified the OMGStop() method to something more like this:
public void cancelUpdates() {
//TODO potential bug here
if(pi == null)
setPendingIntent();
mgr.cancel(pi);
//Should one of these work?
stopService(new Intent(applicationContext, LocationPoller.class));
stopService(new Intent(applicationContext, LocationPollerService.class));
I'm storing pi (PendingIntent) as a member in my activity class. And this works fine to remove the PendingIntent from the AlarmManager.
However...
I would like to be able to stop the current location poll if there is one going on. Is it possible with your current design? I thought I could just stop the service, but the GPS continues to run.
Basically what i'm trying to do is stop everything when the user (me on my trip) changes a preference (such as the timeout, or USE GPS or update period. And then recreate everything with the new values.
Thanks,
Great code BTW - Exactly what I want for tracking my cross country journey :)
I faced the same issue. - On occasion I let my mousepointer hover over "mgr.setRepeating(..)" and read some of Eclipse's (Indigo) hints:
"If there is already an alarm scheduled for the same IntentSender, it will first be canceled."
But the IntentSender could be gone by then.
This led me to the following "solution" (in original CommonsWare code):
if(pi == null) {
Intent i = new Intent(this, LocationPoller.class);
pi = PendingIntent.getBroadcast(this, 0, i, 0);
mgr.setRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP,
SystemClock.elapsedRealtime() + 1000, PERIOD, pi);
}
mgr.cancel();
By adding 1000 msec I tried to make sure the AlarmManager has no chance to start before being hit by a cancel().
HTH, regards.
PS: I'd like to quote mparkes: "Great code"!
Is it possible with your current design?
No, sorry. That's theoretically possible to add, but probably a bit tricky, and definitely not there at the moment.
Basically what i'm trying to do is stop everything when the user (me on my trip) changes a preference (such as the timeout, or USE GPS or update period. And then recreate everything with the new values.
That is a perfectly reasonable concept, just not what LocationPoller supports. LocationPoller was designed more for the "check every hour" sorts of scenarios, where it is statistically unlikely that a check is going on while the user happens to be manipulating your app's UI.