Adding a dependency to a batch task at runtime - batch-processing

I have a batch job in AX 2012 R2 that runs, essentially iterating over a table and creating an instance of a class (that extends RunBaseBatch) that gets added as a task.
I also have some post processing items I need to do, after all the tasks have completed.
So far, the following is working:
while select stagingTable where stagingTable.OperationNo == params.paramOperationNo()
{
batchHeader = this.getCurrentBatchHeader();
batchTask = OperationTask::construct();
batchHeader.addRuntimeTask(batchTask,this.getCurrentBatchTask().RecId);
}
batchHeader.save();
postTask = PostProcessingTask::construct();
batchHeader.addRuntimeTask(postTask,this.getCurrentBatchTask().RecId);
batchHeader.addDependency(postTask,batchTask,BatchDependencyStatus::FinishedOrError);
batchHeader.save();
My thought is that this will add a dependency on the post process task to not start until we get Finished or Error on the last task added in the loop. What I get instead is an exception "The dependency could not be created because task '' does not exist."
I'm uncertain what I'm missing, as the tasks all get added executed successfully, it seems that just the dependency doesn't want to work.

Several things, where this code is being called matters. Is the code already in batch? Is the code calling in doBatch() before/after the super? etc.
You have a while-select, does this create multiple batch tasks? If it does, then you need to create a dependency on each batch task object. This is one problem I see. If your while-select statement only selects 1 record and adds one task, then the problem is something else, but you shouldn't do a while-select to select one record.
Also, you call batchHeader.save(); two times. I'd probably remove the first call. I'd need to see what is instantiating your code.
Where you have this.getCurrentBatchTask().RecId, depending on if your code is in batch or not, try replacing that with BatchHeader::getCurrentBatchTask().RecId
And where you have batchHeader = this.getCurrentBatchHeader(); replace that with batchHeader = BatchHeader::getCurrentBatchHeader();
EDIT Try this code (fix whatever to make it compile)
BatchHeader batchHeader = BatchHeader::getCurrentBatchHeader();
Set set = new Set(Types::Class);
SetEnumerator se;
BatchTask batchTask;
PostTask postTask;
while select stagingTable where stagingTable.OperationNo == params.paramOperationNo()
{
batchTask = OperationTask::construct();
set.add(batchTask);
batchHeader.addRuntimeTask(batchTask,BatchHeader::getCurrentBatchTask().RecId);
}
// Create post task
postTask = PostProcessingTask::construct();
batchHeader.addRuntimeTask(postTask,BatchHeader::getCurrentBatchTask().RecId);
// Create dependencies
se = set.getEnumerator();
while (se.moveNext())
{
batchTask = se.current(); // Task to make dependent on
batchHeader.addDependency(postTask,batchTask,BatchDependencyStatus::FinishedOrError);
}
batchHeader.save();

Related

Polarion API: How to update multiple work items in the same action

I created a script which updates work items in a polarion document. However, right now each workitem update is a single save. But my script updates all work items in the document, which results in a large number of save actions in the api and thus a large set of clutter in the history.
If you edit a polarion document yourself, it will update all workitems.
Is it possible to do this same thing with the polarion API?
I tried
Using the tracker service to update work items. This only allows a single work item to be updated.
Using the web development tools to try and get information from the API. This seems to use a UniversalService for which no documentation is available at the API site https://almdemo.polarion.com/polarion/sdk/index.html
Update 1:
I am using the python polarion package to update the workitems.Python polarion Based on the answer by #boaz I tried the following code:
project = client.getProject(project_name)
doc = project.getDocument(document_location)
workitems = doc.getWorkitems()
session_service = client.getService("Session")
tracker_service = client.getService("Tracker")
session_service.beginTransaction()
for workitem in workitems:
workitem.description = workitem._polarion.TextType(
content=str(datetime.datetime.now()), type='text/html', contentLossy=False)
update_list = {
"uri": workitem.uri,
"description": workitem.description
}
tracker_service.updateWorkItem(update_list)
session_service.endTransaction(False)
The login step that #boaz indicated is done in the backend (See: https://github.com/jesper-raemaekers/python-polarion/blob/3e61527cf0f1f3c8614a30289a0a3409d2d8712d/polarion/polarion.py#L103)
However, this gives the following Java exception:
java.lang.RuntimeException: java.lang.NullPointerException
Update 2
There seems to be an issue with the session. If I call the following code:
session_service.logIn(user, password)
print(session_service.hasSubject())
it prints False.
The same thing happens when using the transaction:
session_service.beginTransaction()
print(session_service.transactionExists())
also prints False
Try wrapping your changes in a SessionWebService transaction, see JavaDoc:
sessionService = factory.getSessionService();
sessionService.logIn(prop.getProperty("user"),
prop.getProperty("passwd"));
sessionService.beginTransaction();
// ...
// your changes
// ...
sessionService.endTransaction(false);
sessionService.endSession();
As shown in the example in Polarion/polarion/SDK/examples/com.polarion.example.importer.
This will commit all your changes in one single SVN commit.

NextFlow: How to fail if channel is empty ( .ifEmpty() )

I'd like for my NextFlow pipeline to fail if a specific channel is empty because, as is, the pipeline will continue as though nothing is wrong, but the process depending on the channel never starts. The answer to a related post states that we generally shouldn't check if a channel is empty, but I'm not sure how else to handle this.
The issue I'm having in the below example is that it always fails, but the process is called if I comment out the .ifEmpty() statement.
Here's a basic example:
/*
* There are .cram files in this folder
*/
params.input_sample_folder = 'path/to/folder/*'
samples = Channel.fromPath(params.input_sample_folder, checkIfExists: true)
.filter( ~/.*(\.sam|\.bam|\.cram)/ )
.ifEmpty( exit 1,
"ERROR: Did not find any samples in ${params.input_sample_folder})
workflow{
PROCESS_SAMPLES( samples )
}
Ultimate questions:
My guess is that the channel does not fill immediately. Is that true? If so, when does it fill?
How should I handle this situation? I want to fail if the channel doesn't get populated. e.g., I was surprised to learn that the channel remains empty if I only provide a folder path without a glob/wildcard character (/path/to/folder/; no * or *.cram, etc.). I don't think I can handle it in the process itself, because the process never gets called if the channel is legitimately empty.
Really appreciate your help.
Setting checkIfExists: true will actually throw an exception for you if the specified files do not exist on your file system. The trick is to specify the files you need when you create the channel, rather than filtering for them downstream. For example, all you need is:
params.input_sample_folder = 'path/to/folder'
samples = Channel.fromPath(
"${params.input_sample_folder}/*.{sam,bam,cram}",
checkIfExists: true,
)
Or arguably better; since this gives the user full control over the input files:
params.input_sample_files = 'path/to/folder/*.{sam,bam,cram}'
samples = Channel.fromPath( params.input_sample_files, checkIfExists: true )
Either way, both will have your pipeline fail with exit status 1 and the following message in red when no matching files exist:
No files match pattern `*.{sam,bam,cram}` at path: path/to/folder/
As per the docs, the ifEmpty operator is really just intended to emit a default value when a channel becomes empty. To avoid having to check if a channel is empty, the general solution is to just avoid creating an empty channel in the first place. There's lots of ways to do this, but one way might look like:
import org.apache.log4j.Logger
nextflow.enable.dsl=2
def find_sample_files( input_dir ) {
def pattern = ~/.*(\.sam|\.bam|\.cram)/
def results = []
input_dir.eachFileMatch(pattern) { item ->
results.add( item )
}
return results
}
params.input_sample_folder = 'path/to/folder'
workflow {
input_sample_folder = file( params.input_sample_folder )
input_sample_files = find_sample_files( input_sample_folder )
if ( !input_sample_files ) {
log.error("ERROR: Did not find any samples in ${params.input_sample_folder}")
System.exit(1)
}
sample_files = Channel.of( input_sample_files )
sample_files.view()
}

The Nhibernate transaction has been successfully committed, but the result of the query is still the unmodified value

Execute the following server code, and then check the promotion table and task table in the database. The related fields have been updated correctly, which indicates that the transaction has been successfully committed.
using (ITransaction tx = session.BeginTransaction())
{
try
{
Promotion p = session.Get<Promotion>(request.PromotionId);
p.Status = PromotionStatus.Canceled;
foreach (Task task in p.Tasks)
{
if (task.AnnounceStatus == TaskAnnounceStatus.New)
{
task.AnnounceStatus = TaskAnnounceStatus.PromotionCanceled;
task.CancelTime = DateTime.Now;
//session.Update(task);
}
}
tx.Commit();
}
catch
{
tx.Rollback();
throw;
}
}
Then execute the following query(Query A), the data obtained is also the updated value. It looks like everything is very good.
tasks = session.Query<Task>().Where(p => p.AnnounceStatus == Model.TaskAnnounceStatus.New && p.ProcessStatus == Model.TaskProcessStatus.New).ToList();
However, if I execute a query on the task using the following code before committing the transaction, the result of the above query(Query A) will get the old unmodified value. At the same time, what you see in the database is still the correctly updated value.
Task task = session.Get<Task>(taskId);
So I modified the first piece of code and explicitly called the update method (see the code at the comment), and everything worked fine this time.
My guess is that Nhibernate's cache is causing the above problem. I use syscache2 to manage the second-level cache, the cache was set to ReadWrite, and use sessionFacotry.getCurrentSession to manage Nhibernate's session.
Hope someone can help me explain how this works.
You execute query session.Get<Task>(taskId); first. This loads the entity in first level cache.
Then in your transaction, you Get the Promotion entity. The Task is the IEnumerable property of it. As lazy load may be, your foreach loop iterate through Task entity with ID taskID - Modifies it - Updates it - Transaction is successful. As all this is happening inside the transaction, your initial entity returned by session.Get<Task>(taskId); is not updated. It still hold the old values.
Then, you again session.Query<Task>() outside the transaction. This time, NHibernate see that the entity with same identifier is already loaded in session cache (with session.Get<Task>(taskId); query), it does not load that entity again, it simply returns the entity already in session cache. As that entity hold the old values, you see the problem.
To confirm this, put all these queries inside the transaction block and check the result.
Alternatively, manage so scope of session properly.
Understand that your ISession is your Unit Of Work; scope it carefully.

How would I implement multithreading in this situation? Is it possible?

I have a form with a listview in it that the user can add tasks too. Then the user can click a button then the application goes through each task in the listview 1 by 1 an executes it.
These tasks are more like instructions that actually complete tasks.I do this by having a class with a loop in it that goes through each item and it then completes a task I set for each item(instruction). In order to start the parsing I have a button on a form that calls that function. IE: RunTask(listview1, 1) - basically all this does it starts the loop I have in my class , with a specified listview and which item to start on.
Everything works perfect except the screen locks up, so I cannot implement a stop feature to stop the application from parsing these listview items. I just don't understand how I can implement this without crossthreading, since the thread that I would like to run seperate will always access this listview. It is not feasable to redesign the program to get rid of the listview. I tried application.doevents although it caused way too man bugs. I have been researching for days on how to fix this but I have NO idea. Hopefully someone cans hed some light.
Also I had already added a background worker to solve the issue, although I had to obviously set checkforillegalcrossthreadcalls = false and I know this isn't smart.
Try doing something like this. Take you list view and turn it into a set of values that aren't UI related. Like this:
string[] values =
this
.listView1
.Items
.Cast<ListViewItem>()
.Select(x => x.Text)
.ToArray();
Then you can use the parallel task library to run your task in the background:
var tokenSource = new System.Threading.CancellationTokenSource();
var token = tokenSource.Token;
var task = System.Threading.Tasks.Task.Factory
.StartNew(() => RunTasks(values, 1), token);
If you need to cancel the task you can do this:
tokenSource.Cancel();
But to handle UI updates when the task is finished do this:
task.ContinueWith(r =>
{
/* When compete code */
});
Make sure that you invoke the UI updates so that they go on the UI thread.
My apologies that I didn't write this in VB.NET. My VB is getting rusty.

How do I dynamically trigger downstream builds in jenkins?

We want to dynamically trigger integration tests in different downstream builds in jenkins. We have a parametrized integration test project that takes a test name as a parameter. We dynamically determine our test names from the git repo.
We have a parent project that uses jenkins-cli to start a build of the integration project for each test found in the source code. The parent project and integration project are related via matching fingerprints.
The problem with this approach is that the aggregate test results doesn't work. I think the problem is that the "downstream" integration tests are started via jenkins-cli, so jenkins doesn't realize they are downstream.
I've looked at many jenkins plugins to try to get this working. The Join and Parameterized Trigger plugins don't help because they expect a static list of projects to build. The parameter factories available for Parameterized Trigger won't work either because there's no factory to create an arbitrary list of parameters. The Log Trigger plugin won't work.
The Groovy Postbuild Plugin looks like it should work, but I couldn't figure out how to trigger a build from it.
def job = hudson.model.Hudson.instance.getJob("job")
def params = new StringParameterValue('PARAMTEST', "somestring")
def paramsAction = new ParametersAction(params)
def cause = new hudson.model.Cause.UpstreamCause(currentBuild)
def causeAction = new hudson.model.CauseAction(cause)
hudson.model.Hudson.instance.queue.schedule(job, 0, causeAction, paramsAction)
This is what finally worked for me.
NOTE: The Pipeline Plugin should render this question moot, but I haven't had a chance to update our infrastructure.
To start a downstream job without parameters:
job = manager.hudson.getItem(name)
cause = new hudson.model.Cause.UpstreamCause(manager.build)
causeAction = new hudson.model.CauseAction(cause)
manager.hudson.queue.schedule(job, 0, causeAction)
To start a downstream job with parameters, you have to add a ParametersAction. Suppose Job1 has parameters A and C which default to "B" and "D" respectively. I.e.:
A == "B"
C == "D"
Suppose Job2 has the same A and B parameters, but also takes parameter E which defaults to "F". The following post build script in Job1 will copy its A and C parameters and set parameter E to the concatenation of A's and C's values:
params = []
val = ''
manager.build.properties.actions.each {
if (it instanceof hudson.model.ParametersAction) {
it.parameters.each {
value = it.createVariableResolver(manager.build).resolve(it.name)
params += it
val += value
}
}
}
params += new hudson.model.StringParameterValue('E', val)
paramsAction = new hudson.model.ParametersAction(params)
jobName = 'Job2'
job = manager.hudson.getItem(jobName)
cause = new hudson.model.Cause.UpstreamCause(manager.build)
causeAction = new hudson.model.CauseAction(cause)
def waitingItem = manager.hudson.queue.schedule(job, 0, causeAction, paramsAction)
def childFuture = waitingItem.getFuture()
def childBuild = childFuture.get()
hudson.plugins.parameterizedtrigger.BuildInfoExporterAction.addBuildInfoExporterAction(
manager.build, childProjectName, childBuild.number, childBuild.result
)
You have to add $JENKINS_HOME/plugins/parameterized-trigger/WEB-INF/classes to the Groovy Postbuild plugin's Additional groovy classpath.
Execute this Groovy script
import hudson.model.*
import jenkins.model.*
def build = Thread.currentThread().executable
def jobPattern = "PUTHEREYOURJOBNAME"
def matchedJobs = Jenkins.instance.items.findAll { job ->
job.name =~ /$jobPattern/
}
matchedJobs.each { job ->
println "Scheduling job name is: ${job.name}"
job.scheduleBuild(1, new Cause.UpstreamCause(build), new ParametersAction([ new StringParameterValue("PROPERTY1", "PROPERTY1VALUE"),new StringParameterValue("PROPERTY2", "PROPERTY2VALUE")]))
}
If you don't need to pass in properties from one build to the other just take the ParametersAction out.
The build you scheduled will have the same "cause" as your initial build. That's a nice way to pass in the "Changes". If you don't need this just do not use new Cause.UpstreamCause(build) in the function call
Since you are already starting the downstream jobs dynamically, how about you wait until they done and copy the test result files (I would archive them on the downstream jobs and then just download the 'build' artifacts) to the parent workspace. You might need to aggregate the files manually, depending if the Test plugin can work with several test result pages. In the post build step of the parent jobs configure the appropriate test plugin.
Using the Groovy Postbuild Plugin, maybe something like this will work (haven't tried it)
def job = hudson.getItem(jobname)
hudson.queue.schedule(job)
I am actually surprised that if you fingerprint both jobs (e.g. with the BUILD_TAG variable of the parent job) the aggregated results are not picked up. In my understanding Jenkins simply looks at md5sums to relate jobs (Aggregate downstream test results and triggering via the cli should not affect aggregating results. Somehow, there is something additional going on to maintain the upstream/downstream relation that I am not aware of...
This worked for me using "Execute system groovy
script"
import hudson.model.*
def currentBuild = Thread.currentThread().executable
def job = hudson.model.Hudson.instance.getJob("jobname")
def params = new StringParameterValue('paramname', "somestring")
def paramsAction = new ParametersAction(params)
def cause = new hudson.model.Cause.UpstreamCause(currentBuild)
def causeAction = new hudson.model.CauseAction(cause)
hudson.model.Hudson.instance.queue.schedule(job, 0, causeAction, paramsAction)