I have a BPMN process that should handle 2 alternative scenarios:
TaskA -> TaskB -> Last Task
OR
TaskA -> TaskX -> (TaskY and TaskB in parallel) -> Last Task
I can't find what's is the proper way to join the parallel tasks.
I have designed this solution, but it doesn't look fine to me:
for the first scenario, the parallel gateway looks like a fork rather than a join.
How should I design this case (without having to duplicate tasks) ?
I think the following diagram do just what you want:
I use an inclusive gateway that will always take the transition that goes to "Task B" and based on condition also execute "Task Y" in parallel.
Same condition is also use to include or skip "Task X".
I create a runnable version of this process for Bonita BPM and it seems to behave like what you expect.
Related
I´ve got a Job that runs a TaskletStep, then a chunk-based step and then another TaskletStep.
In each of these steps, errors (in the form of Exceptions) can occur.
The chunk-based step looks like this:
stepBuilderFactory
.get("step2")
.chunk<SomeItem, SomeItem>(1)
.reader(flatFileItemReader)
.processor(itemProcessor)
.writer {}
.faultTolerant()
.skipPolicy { _ , _ -> true } // skip all Exceptions and continue
.taskExecutor(taskExecutor)
.throttleLimit(taskExecutor.corePoolSize)
.build()
The whole job definition:
jobBuilderFactory.get("job1")
.validator(validator())
.preventRestart()
.start(taskletStep1)
.next(step2)
.next(taskletStep2)
.build()
I expected that Spring Batch somehow picks up the Exceptions that occur along the way, so I can then create a Report including them after the Job has finished processing. Looking at the different contexts, there´s also fields that should contain failureExceptions. However, it seems there´s no such information (especially for the chunked step).
What would be a good approach if I need information about:
what Exceptions did occur in which Job execution
which Item was the one that triggered it
The JobExecution provides a method to get all failure exceptions that happened during the job. You can use that in a JobExecutionListener#afterJob(JobExecution jobExecution) to generate your report.
In regards to which items caused the issue, this will depend on where the exception happens (during the read, process or write operation). For this requirement, you can use one of the ItemReadListener, ItemProcessListener or ItemWriteListener to keep record of the those items (For example, by adding them to the job execution context to be able to get access to them in the JobExecutionListener#afterJob method for your report).
i have a daily( 09:00am) box containing 10 jobs inside it. All child jobs are sequentially scheduled to run.
On Monday, jobs 1,2 &3 completed and job4 failed. And coz of this, the downstream is stalled and the box is running infinetly( until some actions taken manually)
But the requirement is to run this box again on Tue 09:00am. I heard of Kickstart attribute to kick off the box on next scheduled time irrespective of last run status.
Can someone tell about this kick_start attribute? Also suggest me any other way to schedule this box daily.
TIA
Never heard of the kick_start attribute and could not find it in the R11.3.5 reference guide.
I would look at the box_terminator: y that will fail the box if a job in it fails and the job_terminator: y that will terminate and fail a job if the box it is in fails.
box_criteria is another attribute that may help as you can define what success or failure looks like. For example if you don't care if job4 fails, define box_criteria: s(job3).
Course that only sets your box to FA where it will run the next time it's starting conditions are met. It does nothing to run the downstream for the current run.
Have fun and test, test, test.
I have a common test suite that attempts to create an ets table for use in all suites and all test cases. It looks like so:
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
init_per_suite(Config) ->
TabId = ets:new(conns, [set]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId} | Config].
end_per_suite(Config) ->
ets:delete(?config(table, Config)).
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
The ets_tests function failed with a badarg. Creating/destroying the ets table per testcase, which looks like so:
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
init_per_testcase(Config) ->
TabId = ets:new(conns, [set]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId} | Config].
end_per_testcase(Config) ->
ets:delete(?config(table, Config)).
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
Running this, I find that it functions beautifully.
I'm confused by this behavior and unable to determine why this would happen, form the docs. Questions:
Why does this happen?
How can I have an ets table to share between a per suite and per testcase?
As was already mentioned in the answer by Pascal and as discussed in the User Guide only init_per_testcase and end_per_testcase run in the same process as the testcase. Since ETS tables are bound to a owner process your only way to have a ETS table persist during a whole suite or group is to give it away or define a heir process.
You can easily spawn a process in your init_per_suite or init_per_group functions, set it as heir for the ETS table and pass its pid along in the config.
To clean up all you need is to kill this process in your end_per_suite or end_per_group functions.
-module(an_example_SUITE).
-include_lib("common_test/include/ct.hrl").
-compile(export_all).
all() -> [ets_tests].
ets_owner() ->
receive
stop -> exit(normal);
Any -> ets_owner()
end.
init_per_suite(Config) ->
Pid = spawn(fun ets_owner/0),
TabId = ets:new(conns, [set, protected, {heir, Pid, []}]),
ets:insert(TabId, {foo, 2131}),
[{table,TabId},{table_owner, Pid} | Config].
end_per_suite(Config) ->
?config(table_owner, Config) ! stop.
ets_tests(Config) ->
TabId = ?config(table, Config),
[{foo, 2131}] = ets:lookup(TabId, foo).
You also need to make sure you can still access your table from the testcase process, by making it either protectedor public
An ets table is attached to a process and destroyed as soon as the process ends, unless you use the the give_away function (which is not feasible I fear in this case)
As state in the common tets doc, each test case and the init_per_suite and end_per_suite are run in separate processes, so the ets table is destroyed as soon as you leave the init_per_suite function.
fron common_test doc
init_per_suite and end_per_suite will execute on dedicated Erlang
processes, just like the test cases do. The result of these functions
is however not included in the test run statistics of successful,
failed and skipped cases.
from ets doc
The default owner is the process that created the table. Table
ownership can be transferred at process termination by using the heir
option or explicitly by calling give_away/3.
I need to design a Redis-driven scalable task scheduling system.
Requirements:
Multiple worker processes.
Many tasks, but long periods of idleness are possible.
Reasonable timing precision.
Minimal resource waste when idle.
Should use synchronous Redis API.
Should work for Redis 2.4 (i.e. no features from upcoming 2.6).
Should not use other means of RPC than Redis.
Pseudo-API: schedule_task(timestamp, task_data). Timestamp is in integer seconds.
Basic idea:
Listen for upcoming tasks on list.
Put tasks to buckets per timestamp.
Sleep until the closest timestamp.
If a new task appears with timestamp less than closest one, wake up.
Process all upcoming tasks with timestamp ≤ now, in batches (assuming
that task execution is fast).
Make sure that concurrent worker wouldn't process same tasks. At the same time, make sure that no tasks are lost if we crash while processing them.
So far I can't figure out how to fit this in Redis primitives...
Any clues?
Note that there is a similar old question: Delayed execution / scheduling with Redis? In this new question I introduce more details (most importantly, many workers). So far I was not able to figure out how to apply old answers here — thus, a new question.
Here's another solution that builds on a couple of others [1]. It uses the redis WATCH command to remove the race condition without using lua in redis 2.6.
The basic scheme is:
Use a redis zset for scheduled tasks and redis queues for ready to run tasks.
Have a dispatcher poll the zset and move tasks that are ready to run into the redis queues. You may want more than 1 dispatcher for redundancy but you probably don't need or want many.
Have as many workers as you want which do blocking pops on the redis queues.
I haven't tested it :-)
The foo job creator would do:
def schedule_task(queue, data, delay_secs):
# This calculation for run_at isn't great- it won't deal well with daylight
# savings changes, leap seconds, and other time anomalies. Improvements
# welcome :-)
run_at = time.time() + delay_secs
# If you're using redis-py's Redis class and not StrictRedis, swap run_at &
# the dict.
redis.zadd(SCHEDULED_ZSET_KEY, run_at, {'queue': queue, 'data': data})
schedule_task('foo_queue', foo_data, 60)
The dispatcher(s) would look like:
while working:
redis.watch(SCHEDULED_ZSET_KEY)
min_score = 0
max_score = time.time()
results = redis.zrangebyscore(
SCHEDULED_ZSET_KEY, min_score, max_score, start=0, num=1, withscores=False)
if results is None or len(results) == 0:
redis.unwatch()
sleep(1)
else: # len(results) == 1
redis.multi()
redis.rpush(results[0]['queue'], results[0]['data'])
redis.zrem(SCHEDULED_ZSET_KEY, results[0])
redis.exec()
The foo worker would look like:
while working:
task_data = redis.blpop('foo_queue', POP_TIMEOUT)
if task_data:
foo(task_data)
[1] This solution is based on not_a_golfer's, one at http://www.saltycrane.com/blog/2011/11/unique-python-redis-based-queue-delay/, and the redis docs for transactions.
You didn't specify the language you're using. You have at least 3 alternatives of doing this without writing a single line of code in Python at least.
Celery has an optional redis broker.
http://celeryproject.org/
resque is an extremely popular redis task queue using redis.
https://github.com/defunkt/resque
RQ is a simple and small redis based queue that aims to "take the good stuff from celery and resque" and be much simpler to work with.
http://python-rq.org/
You can at least look at their design if you can't use them.
But to answer your question - what you want can be done with redis. I've actually written more or less that in the past.
EDIT:
As for modeling what you want on redis, this is what I would do:
queuing a task with a timestamp will be done directly by the client - you put the task in a sorted set with the timestamp as the score and the task as the value (see ZADD).
A central dispatcher wakes every N seconds, checks out the first timestamps on this set, and if there are tasks ready for execution, it pushes the task to a "to be executed NOW" list. This can be done with ZREVRANGEBYSCORE on the "waiting" sorted set, getting all items with timestamp<=now, so you get all the ready items at once. pushing is done by RPUSH.
workers use BLPOP on the "to be executed NOW" list, wake when there is something to work on, and do their thing. This is safe since redis is single threaded, and no 2 workers will ever take the same task.
once finished, the workers put the result back in a response queue, which is checked by the dispatcher or another thread. You can add a "pending" bucket to avoid failures or something like that.
so the code will look something like this (this is just pseudo code):
client:
ZADD "new_tasks" <TIMESTAMP> <TASK_INFO>
dispatcher:
while working:
tasks = ZREVRANGEBYSCORE "new_tasks" <NOW> 0 #this will only take tasks with timestamp lower/equal than now
for task in tasks:
#do the delete and queue as a transaction
MULTI
RPUSH "to_be_executed" task
ZREM "new_tasks" task
EXEC
sleep(1)
I didn't add the response queue handling, but it's more or less like the worker:
worker:
while working:
task = BLPOP "to_be_executed" <TIMEOUT>
if task:
response = work_on_task(task)
RPUSH "results" response
EDit: stateless atomic dispatcher :
while working:
MULTI
ZREVRANGE "new_tasks" 0 1
ZREMRANGEBYRANK "new_tasks" 0 1
task = EXEC
#this is the only risky place - you can solve it by using Lua internall in 2.6
SADD "tmp" task
if task.timestamp <= now:
MULTI
RPUSH "to_be_executed" task
SREM "tmp" task
EXEC
else:
MULTI
ZADD "new_tasks" task.timestamp task
SREM "tmp" task
EXEC
sleep(RESOLUTION)
If you're looking for ready solution on Java. Redisson is right for you. It allows to schedule and execute tasks (with cron-expression support) in distributed way on Redisson nodes using familiar ScheduledExecutorService api and based on Redis queue.
Here is an example. First define a task using java.lang.Runnable interface. Each task can access to Redis instance via injected RedissonClient object.
public class RunnableTask implements Runnable {
#RInject
private RedissonClient redissonClient;
#Override
public void run() throws Exception {
RMap<String, Integer> map = redissonClient.getMap("myMap");
Long result = 0;
for (Integer value : map.values()) {
result += value;
}
redissonClient.getTopic("myMapTopic").publish(result);
}
}
Now it's ready to sumbit it into ScheduledExecutorService:
RScheduledExecutorService executorService = redisson.getExecutorService("myExecutor");
ScheduledFuture<?> future = executorService.schedule(new CallableTask(), 10, 20, TimeUnit.MINUTES);
future.get();
// or cancel it
future.cancel(true);
Examples with cron expressions:
executorService.schedule(new RunnableTask(), CronSchedule.of("10 0/5 * * * ?"));
executorService.schedule(new RunnableTask(), CronSchedule.dailyAtHourAndMinute(10, 5));
executorService.schedule(new RunnableTask(), CronSchedule.weeklyOnDayAndHourAndMinute(12, 4, Calendar.MONDAY, Calendar.FRIDAY));
All tasks are executed on Redisson node.
A combined approach seems plausible:
No new task timestamp may be less than current time (clamp if less). Assuming reliable NTP synch.
All tasks go to bucket-lists at keys, suffixed with task timestamp.
Additionally, all task timestamps go to a dedicated zset (key and score — timestamp itself).
New tasks are accepted from clients via separate Redis list.
Loop: Fetch oldest N expired timestamps via zrangebyscore ... limit.
BLPOP with timeout on new tasks list and lists for fetched timestamps.
If got an old task, process it. If new — add to bucket and zset.
Check if processed buckets are empty. If so — delete list and entrt from zset. Probably do not check very recently expired buckets, to safeguard against time synchronization issues. End loop.
Critique? Comments? Alternatives?
Lua
I made something similar to what's been suggested here, but optimized the sleep duration to be more precise. This solution is good if you have few inserts into the delayed task queue. Here's how I did it with a Lua script:
local laterChannel = KEYS[1]
local nowChannel = KEYS[2]
local currentTime = tonumber(KEYS[3])
local first = redis.call("zrange", laterChannel, 0, 0, "WITHSCORES")
if (#first ~= 2)
then
return "2147483647"
end
local execTime = tonumber(first[2])
local event = first[1]
if (currentTime >= execTime)
then
redis.call("zrem", laterChannel, event)
redis.call("rpush", nowChannel, event)
return "0"
else
return tostring(execTime - currentTime)
end
It uses two "channels". laterChannel is a ZSET and nowChannel is a LIST. Whenever it's time to execute a task, the event is moved from the the ZSET to the LIST. The Lua script with respond with how many MS the dispatcher should sleep until the next poll. If the ZSET is empty, sleep forever. If it's time to execute something, do not sleep(i e poll again immediately). Otherwise, sleep until it's time to execute the next task.
So what if something is added while the dispatcher is sleeping?
This solution works in conjunction with key space events. You basically need to subscribe to the key of laterChannel and whenever there is an add event, you wake up all the dispatcher so they can poll again.
Then you have another dispatcher that uses the blocking left pop on nowChannel. This means:
You can have the dispatcher across multiple instances(i e it's scaling)
The polling is atomic so you won't have any race conditions or double events
The task is executed by any of the instances that are free
There are ways to optimize this even more. For example, instead of returning "0", you fetch the next item from the zset and return the correct amount of time to sleep directly.
Expiration
If you can not use Lua scripts, you can use key space events on expired documents.
Subscribe to the channel and receive the event when Redis evicts it. Then, grab a lock. The first instance to do so will move it to a list(the "execute now" channel). Then you don't have to worry about sleeps and polling. Redis will tell you when it's time to execute something.
execute_later(timestamp, eventId, event) {
SET eventId event EXP timestamp
SET "lock:" + eventId, ""
}
subscribeToEvictions(eventId) {
var deletedCount = DEL eventId
if (deletedCount == 1) {
// move to list
}
}
This however has it own downsides. For example, if you have many nodes, all of them will receive the event and try to get the lock. But I still think it's overall less requests any anything suggested here.
Let's say I have a test case with some steps in it. Now, let's say that step 3 needs to be repeated after you complete steps 4 and 5 ... so that when you do step 6 you are in the right place.
Is it good practice to tell the tester to repeat a step? Or would it be better copy-and-paste the repeated step into the step where you would need to repeat it?
I am hearing arguments that it is not industry standard to tell the tester to repeat steps and that one might not pass certain certifications if test cases are written in this manner.
Example:
*Step 1: Click the View Event Log button; Expected Results: Event Log window appears
Step 2: Close the event log window (X) or OK; Expected Results: The Event Log window disappears
Step 3: Repeat Step 1; Expected Results: Expected Results from Step 1
Step 4: Click the Cancel button; Expected Results: The Event Log window closes and any changes (such as clearing the log) are not applied
Step 5: Repeat Step 1; Expected Results: Expected Results from Step 1
Step 6: Click the Clear button and hit apply; Expected Results: The log is cleared
...*
Some people think that I should be copying-and-pasting what is in Step 1 each time I need to repeat that step rather than just simply saying that the tester should repeat the step. Any input as to industry standards, potential downfalls, etc ... would be greatly appreciated.
Test case design does not really follow an industry standard, if you were trying to get certified, listing to repeat a step is a no-no. I personally think that's crap. I see no problem asking a tester to repeat a step. As a believer in agile methodology, I prefer much simpler test cases so a tester has more time to test scenarios rather than design test cases (or a developer more time to develop if you are in a cross functional team). If your looking for more input from a larger testing community try http://www.qaforums.com/
Test cases should be as independent as possible and not verifying two outcomes in a single test case. The test cases should not be designed in a way where the tester has to repeat any previous step. In this case, a new test case should be written because it is a new path. The prons for this approach is that at the end of execution you'll have a clear picture of the test coverage and the pass/fail % of the requirements because all the test cases are independent.