I'm wondering if there is a way to get the parent execution of an execution in camunda. What I'm trying to achieve is basically the following:
This is a simple process involving a parallel gateway. Each of the flows is composed of a service task (external) and a user task.
In each "Pre: Task X" service task, I want to set some variables that I will use afterward in their respective user tasks. I want each execution flow of the parallel gateway to have their own variables and non-accessible from the other flows. How can I achieve this?
I was doing some tests and I found the following:
When the process is instantiated, I get instantly 5 execution instances.
What I understand is that one belongs to the process, the next two belong to each flow of the parallel gateway, and the last two belong to each of the service tasks.
If I call "complete" for one of the service tasks on the REST API with localVariables, they will instantly disappear and no further be available because they will be tied to the execution associated to the external task, which is terminated after the task completion.
Is there a way in which I can get the parent execution of the task, which in this case would be the parallel execution flow. So I can set localVariables at this level?
Thanks in advance for the valuable help
Regards
First of all 5 executions doesn't mean they are active. In your case there should only be 2 executions active when you start a new instance for the process. You can set your variables in respective executions as return value of the respective service tasks.
You can set variables for process instance but do respect you have 2 executions and 1 process instance. You can not set same variable for multiple executions.
Related
I have a number of pipelines that need to cycle depending on availability of data. If the data is not there wait and try again. The pipe behaviours are largely controlled by a database which captures logs which are used to make decisions about processing.
I read the Microsoft documentation about the Execute Pipeline activity which states that
The Execute Pipeline activity allows a Data Factory or Synapse
pipeline to invoke another pipeline.
It does not explicitly state that it is impossible though. I tried to reference Pipe_A from Pipe_A but the pipe is not visible in the drop down. I need a work-around for this restriction.
Constraints:
The pipe must not call all pipes again, just the pipe in question. The preceding pipe is running all pipes in parallel.
I don't know how many iterations are needed and cannot specify this quantity.
As far as possible best effort has been implemented and this pattern should continue.
Ideas:
Create a intermediary pipe that can be referenced. This is no good I would need to do this for every pipe that requires this behaviour because dynamic content is not allowed for pipe selection. This approach would also pollute the Data Factory workspace.
Direct control flow backwards after waiting inside the same pipeline if condition is met. This won't work either, the If activity does not allow expression of flow within the same context as the If activity itself.
I thought about externalising this behaviour to a Python application which could be attached to an Azure Function if needed. The application would handle the scheduling and waiting. The application could call any pipe it needed and could itself be invoked by the pipe in question. This seems drastic!
Finally, I discovered an activity Until which has do while behaviour. I could wrap these pipes in Until, the pipe executes and finishes and sets database state to 'finished' or cannot finish and sets the state to incomplete and waits. The expression then either kicks off another execution or it does not. Additional conditional logic can be included as required in the procedure that will be used to set a value to variable used by the expression in the Until. I would need a variable per pipe.
I think idea 4 makes sense, I thought I would post this anyway in case people can spot limitations in this approach and/or recommend an approach.
Yes, absolutely agree with All About BI, its seems in your scenario the best suited ADF Activity is Until :
The Until activity in ADF functions as a wrapper and parent component
for iterations, with inner child activities comprising the block of
items to iterate over. The result (s) from those inner child
activities must then be used in the parent Until expression to
determine if another iteration is necessary. Alternatively, if the
pipeline can be maintained
The assessment condition for the Until activity might comprise outputs from other activities, pipeline parameters, or variables.
When used in conjunction with the Wait activity, the Until activity allows you to create loop conditions to periodically check the status of specific operations. Here are some examples:
Check to see if the database table has been updated with new rows.
Check to see if the SQL job is complete.
Check to see whether any new files have been added to a specific
folder.
The task is to run defined number of transformations (.ktr) in parallel.
Each transformation opens it's own database connection to read data.
But we have a limitation on given user, who has only 5 allowed parallel connection to DB and let's consider that this could not be changed.
So when I start job depicted below, only 5 transformations finish their work successfully, and other 5 fails with db connection error.
I know that there is an option to redraw job scheme to have only 5 parallel sequences, but I don't like this approach, as it requires reimplementation when count of threads changes.
Is it possible to configure some kind of pool of executors, so Pentaho job will understand that even if there were 10 transformations provided, only random 5 could be processed in parallel?
I am assuming that you know the number of parallel database connections available. If you know this, use switch/case component and then number of transformations in parallel. Second option is to use job-executor.In Job Executor, if you can set variable which in turn call the job accordingly. For example, you are calling a job using job-executor with value
c:/data-integrator/parallel_job_${5}.kjb where 5 is number of connections available
or
c:/data-integrator/parallel_job_${7}.kjb where 7 is number of connections available
Is this making sense to you.
The concept is the following:
Catch database connection error during transformation run attempt
Wait a couple of seconds
Retry run of a transformation
Look at attached transformation picture. It works for me.
Disadvantages:
A lot of connection errors in the logs, which could confuse.
Given solution could turn in infinite loop (but could be amended to avoid it)
I'm new to Apache Ignite (using 2.7) and I'm looking to create a set of compute tasks that also query data from a cache. I see in the docs the concept of collocated processing but I don't see any examples in the repo. Couple of things I'm unclear on:
1) I want to query the cache from within the task, do I need to create another instance of Cache using Ignite.start or Client mode from within this task, or is there some implicit variable I can use from the context to query the cache.
2) Specifically I'd like to to execute this task as the result of a Continuous Query callback, are there any example detailing that?
thanks
You should inject an instance of Ignite into your task - this is preferred approach.
This may be tricky - make sure to not run this task synchronously since you should not acquire any locks from Continuous Query callback. Maybe Async() methods are OK. The preferred approach is to schedule a taks into your own thread pool to handle procesing latter, and return from callback. Make sure that you don't wait on thread pool as it exhausts (since the common strategy is to run task synchronously if pool is full).
I have the following scenario
I have to do create multiple instances(count determined by call to DB) of 5 different business processes which are modelled in separate workflows
For illustration naming them
Workflow 1
Workflow 2
Workflow 3
Workflow 4
Workflow 5
I will have to start multiple instances of the above workflows based on some data in the database
I will also need a parent workflow(to manage creating the above workflows) which will essentially do the below steps
Note: This workflow should never die, unless stopped externally. I want this workflow as a scheduler to create instances of other workflows. This workflow will be started when the container starts
Step1. Read data from the database using a REST API in a service task.
. The data from step1 will tell the following information
Workflow 1 -> create 5 instances
Workflow 2 -> create 2 instances
Workflow 3 -> create 1 instances
Workflow 4 -> nothing yet to create
Workflow 5 -> nothing yet to create
Note: We have some thresholds set, which ensures not many PI are being created by this process
Step 2: I am trying to start these process instances using the java API of the RuntimeService in the next service task
runtimeService.startProcessInstanceByKey("workflow1");. * 5 times
runtimeService.startProcessInstanceByKey("workflow2");. * 2 times
runtimeService.startProcessInstanceByKey("workflow3");. * 1 time
Not starting workflow 4 and workflow 5 as there is no need in this iteration
I am calling this number of times based on the data in step2
I am expecting that all these process instances will be started async
The purpose of this workflow is only to start the other workflows
Step 3 . After i have finished starting all the process instances for workflow1 to workflow5
I am doing some process cleanup and sending the flow back to Step 1
This keeps going in a loop and does the same steps again
I observed that the execution of the workflows(workflow1 to workflow5) are not triggering at all . until the main workflow is stopped
I have tried different mechanism but unsuccessful in achieving the use case
What is the best approach to model this? I am not sure what has to be done to achieve this.Can someone please help with the same?
I am using the spring boot camunda starter to do the same
I have attached the master workflow which has 3 service tasks
Get Data (explained earlier)
Schedule Workflow(Start child workflows)
Cleanup
From the answer to my question on the Camunda forum:
In BPMN, you can mark an activity as multi-instance, meaning that it is executed multiple times either based on static configuration or based on a dynamic condition or collection variable. See https://camunda.org/bpmn/reference/#activities-task (scroll to Multiple Instance) for an introduction on this and https://docs.camunda.org/manual/7.7/reference/bpmn20/tasks/task-markers/#multiple-instance for the Camunda implementation reference. In Camunda modeler, the multi-instance marker can be toggled via the context menu of an activity.
I want to manage a huge workflow in Camunda.
I have decided to split this into different processes like Create, Configuration, Review & Confirm. Each of these processes have 10 to 15 tasks. These processes should be executed in sequence.
If I want to design my workflow like this, how will I link each process. What is the proper way for Camunda modular design.
You would probably go with some kind of SubProcess. If you plan to model different processes you most likely will use Call Activities and execute them one ofter another in some kind of root process.
Beware of the fact that each sub process starts its own process instance and thus you have to handle different execution scopes. That will be relevant if you request information from the system like e.g. the List of UserTasks. You can not use the processInstanceId of the root process in this case and will have to use a businessKey.
You also have to handle the process variables and decide which variables you want to propagate to the sub process.