I'm working on kubeflow(kf) pipelines that calls other kf pipelines.
We use a top level parent pipeline, to call out multiple child pipelines in parallel. During development and testing it is often the case where we would like to kill a complete run (all parent and child pipelines). Is there a destructor like mechanism in kf pipelines to terminate such "child" pipelines when a parent is terminated.
You can change your pipeline launcher components so that they catch the Kubernetes' shutdown signal and terminate the pipeline they've started.
You can also try using Exit handler (although it cannot access any in-pipeline data).
Related
I have a kogito setup(data-index, jobs-service, infinispan, kafka etc.) and works with my Quarkus application.
My application has a parent .bpmn2 process, and in that process, I create a subprocess when a signal is received. See the process image:
And I have the same logic in child process, it goes like 3 levels down. Thus, I have a hierarchy in data index resources. When I fetch the parent process via data-index in GraphQL, I see the childProcessInstances etc. I also persist the data in infinispan, so when restarting the application, my data persists.
But now I only have a small application and I do not know how it works in a big project, working many parent and child processes as nested.
And I wonder how my Quarkus application handles a big workload of these processes? Is it possible to run infinite multiple procesess? Thanks in advance.
I have a pipeline that executes another pipeline in azure data factory v2. In the executed (child) pipeline I assign a value to a variable I want returned in the master pipeline, is this possible?
Thanks
Pipelines are independent entities - while you can execute "child" pipelines, there is no functional connection between the two. One way around this is to have the child pipeline write the value to some form of intermediate storage (blob storage, a SQL table, etc), and then have the "parent" pipeline read the value after the child pipeline completes. You should also make sure the Execute Pipeline activity has the "Wait on completion" property checked. If you don't want the value retained in the storage medium, you could have the parent pipeline delete the data once it has processed it.
I'm new to Apache Ignite (using 2.7) and I'm looking to create a set of compute tasks that also query data from a cache. I see in the docs the concept of collocated processing but I don't see any examples in the repo. Couple of things I'm unclear on:
1) I want to query the cache from within the task, do I need to create another instance of Cache using Ignite.start or Client mode from within this task, or is there some implicit variable I can use from the context to query the cache.
2) Specifically I'd like to to execute this task as the result of a Continuous Query callback, are there any example detailing that?
thanks
You should inject an instance of Ignite into your task - this is preferred approach.
This may be tricky - make sure to not run this task synchronously since you should not acquire any locks from Continuous Query callback. Maybe Async() methods are OK. The preferred approach is to schedule a taks into your own thread pool to handle procesing latter, and return from callback. Make sure that you don't wait on thread pool as it exhausts (since the common strategy is to run task synchronously if pool is full).
I was going through the Process States. The first state was "Creating Process". What do we mean by creating the process? Is it the completion the program and saving into the hard disk?
The state 'Creating Process' is what you refer to before it goes into a ready state, that is ready to be scheduled by the OS. The process creation state refers to the initial setup of the essentials of a process. In UNIX, at system boot, the first user level process is created called 'init' which is the parent of all the other processes.
UNIX fork() is used to create a new process. During this process creation, the fork() will create a new address space for the child process, a process id (pid) will be assigned, all the mapping of the parent process will be copied into the child's address space and the new program will be loaded in to the child process's address space. This is what happens in 'Process Creation' and once fork() it is followed by a exec() call which will allow the child to run its own program.
Is do_fork() safe from preemption? In other words, can the parent process allocate a new task struct and then get preempted, before getting a chance to insert the new task struct into the ready queue?
It's not safe from preemption.
The do_fork calls copy_process which in turn does the sched_fork that initializes the task. Afterwards the do_fork calls wake_up_new_task in order to put it on the run queue.
This is separated in order to be able to kill or terminate a process before being scheduled.
The sched_fork disables preemption, but enables it once its done with its work, making it possible for the kernel to preempt before calling the wake_up_new_task and putting it on the run queue.
This is based on my knowledge of the 2.6 kernel.