Unable to open transformation: null - Running a Job from Java - pentaho

I have simple Job (named A) which starts a simple transformation (named A). The transformation
contains only a dummy component.
They are both stored into a db repository.
If I start the job from kitchen, everything runs fine:
./kitchen.sh -rep=spoon -user=<user> -pass=<pwd> -job A
Then I have written a simple java code:
JobMeta jobMeta = repository.loadJob(jobName, directory, null, null);
org.pentaho.di.job.Job job = new org.pentaho.di.job.Job(null, jobMeta);
job.getJobMeta().setInternalKettleVariables(job);
job.setLogLevel(LogLevel.ERROR);
job.setName(Thread.currentThread().getName());
job.start();
job.waitUntilFinished();
if (job.getResult() != null && job.getResult().getNrErrors() != 0) {
...
}
else {
...
}
Problem is that running the java program I always get the following error:
A - Unable to open transformation: null
A - java.lang.NullPointerException
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:698)
at org.pentaho.di.job.Job.execute(Job.java:589)
at org.pentaho.di.job.Job.execute(Job.java:728)
at org.pentaho.di.job.Job.execute(Job.java:443)
at org.pentaho.di.job.Job.run(Job.java:363)
I have googled for this error without success and I am stucking there.
Any suggestion ?

The solution seems to be replacing the line inside kitchen
org.pentaho.di.job.Job job = new org.pentaho.di.job.Job(null, jobMeta);
with
org.pentaho.di.job.Job job = new org.pentaho.di.job.Job(repository, jobMeta);
Hoping that this helps someone else.

Related

Gatling feeder/parameter issue - Exception in thread "main" java.lang.UnsupportedOperationException

I just involved the new project for API test for our service by using Gatling. At this point, I want to search query, below is the code:
def chnSendToRender(testData: FeederBuilderBase[String]): ChainBuilder = {
feed(testData)
exec(api.AdvanceSearch.searchAsset(s"{\"all\":[{\"all:aggregate:text\":{\"contains\":\"#{edlAssetId}_Rendered\"}}]}", "#{authToken}")
.check(status.is(200).saveAs("searchStatus"))
.check(jsonPath("$..asset:id").findAll.optional.saveAs("renderedAssetList"))
)
.doIf(session => session("searchStatus").as[Int] == 200) {
exec { session =>
printConsoleLog("Rendered Asset ID List: " + session("renderedAssetList").as[String], "INFO")
session
}
}
}
I declared the feeder already in the simulation scala file:
class GVRERenderEditor_new extends Simulation {
private val edlToRender = csv("data/render/edl_asset_ids.csv").queue
private val chnPostRender = components.notifications.notice.JobsPolling_new.chnSendToRender(edlToRender)
private val scnSendEDLForRender = scenario("Search Post Render")
.exitBlockOnFail(exec(preSimAuth))
.exec(chnPostRender)
setUp(
scnSendEDLForRender.inject(atOnceUsers(1)).protocols(httpProtocol)
)
.maxDuration(sessionDuration.seconds)
.assertions(global.successfulRequests.percent.is(100))
}
But Gatling test failed to run, showing this error: Exception in thread "main" java.lang.UnsupportedOperationException: There were no requests sent during the simulation, reports won't be generated
If I hardcode the #{edlAssetId} (put the real edlAssetId in that query), I will get result. I think I passed the parameter wrongly in this case. I've tried to print the output in console log but no luck. What's wrong with this code? I would appreciate your help. Thanks!
feed(testData)
exec(api.AdvanceSearch.searchAsset(s"{\"all\":[{\"all:aggregate:text\":{\"contains\":\"#{edlAssetId}_Rendered\"}}]}", "#{authToken}")
.check(status.is(200).saveAs("searchStatus"))
.check(jsonPath("$..asset:id").findAll.optional.saveAs("renderedAssetList"))
)
You're missing a . (dot) before the exec to attach it to the feed.
As a result, your method is returning the last instruction, ie the exec only.

IntelliJ IDEA LiveTemplate auto increment between usages

I am trying to make my life easier with Live Templates in intelliJ
I need to increment some param by 1 every-time I use the snippet.
So I tried to develop some groovyScript, and I am close, but my groovy capabilities keeps me back. the number is not incremented by 1, but incremented by 57 for some reason... (UTF-8?)
here is the script:
File file = new File("out.txt");
int code = Integer.parseInt(file.getText('UTF-8'));
code=code+1;
try{
if(_1){
code = Integer.parseInt(_1);
}
} catch(Exception e){}
file.text = code.toString();
return code
So whenever there's param passed to this script (with _1) the initial value is set, and otherwise simply incremented.
this script needs to be passed to the live template param with:
groovyScript("File file = new File(\"out.txt\");int code = Integer.parseInt(file.getText(\'UTF-8\'));code=code+1;String propName = \'_1\';if(this.hasProperty(propName) && this.\"$propName\"){code = Integer.parseInt(_1);};file.text =code.toString();return code", "<optional initial value>")

Why is a DOORS Module sometimes null when trying to edit the Module via DXL?

I'm new to DXL programming language in IBM DOORS. However, I think I have managed to do many interesting things: create Modules, create Objects, create Links, delete Objects etc.
However, I have a very specific problem regarding "null" Modules. I've just written null between "" because the modules exist and they are referenced with a correct name.
When doing this:
Module m1 = edit("1. MY_MODULE", false)
save(m1)
close(m1)
An error like this appears:
enter image description here
You could not understand what does that mean as it is spanish. Basically states this: "Module null parameter in the first position of the argument." That means that the "m1" is null, as the parameter for save() method is null.
The point is that it is an error which appears only sometimes. It seems that the Module is null as it has been previously opened and DOORS does not close properly.
Is there any way, any method...whatever to avoid this error?
I assume that the script cannot find the module when another folder is active.
Try
Module m1 = edit ("/myproject/myfolder/mysubfolder/1. MY_MODULE", false)
There might be many reasons that the module can't be opened in edit mode. For example: User do not have write access OR Module is being used by other user, etc.
However, you can get around the error with the below code snippet:
Module m = edit('My_module', false)
if(!null m) {
//execute program
...
}
else {
//do something
}
I hope this helps.
How does your script work? do you open the same module again and again and sometimes get the error or do you open lots of modules and for some of them it works and for others it doesn't? In the latter case, perhaps you misspelled the path. You could add some sanity checks like
string fullPathToMod = "/myproject/myfolder.."
Item i = item fullPathToMod;
if null i then error "there is no item called " fullPathToMod
if "Module" != type i then error "there is an item, but it's not a module, it's a " type i
This is how the Code is structured:
void checkModule(string folderPath, string mName, Skip list, int listSize, int listLastIndex, string headers[], string heading[], string headerKey, bool uniqueKey, string combinedKey[]){
if (module mName){
Folder f = folder(folderPath)
current = f
Module m = edit(folderPath""mName, false)
current = m
Object o = first(m) // error sometimes: Se ha pasado un parametro Module null en una posiciĆ³n de argumento 1
if (o == null){
loadModule(m, list, listSize, listLastIndex, headers, heading)
} else {
updateModule(m, mName, list, listSize, listLastIndex, heading, headerKey, headers, uniqueKey, combinedKey)
save(m)
close(m)
}
if (lastError() != ""){
print "Error: " lastError() "\n"
}
} else {
print "No module " mName ".\n"
}
}
Exactly it breaks in line:
current = m
But as said, only sometimes, not always.
BTW, I'm executing this script via Batch, via Java code. One curious thing is that if I close DOORS, and execute the script it does execute correctly. It is as if it needs to be closed in order to edit modules correctly.
I pressume current can be used more than once with different types of Items. I guess it should not be wrong, but it breaks saying (more or less):
Null value passed to DXL commmand (current Module).
Obviously, it means that m is null, but I cannot see any reason for that.

Why would I get a lot of None.get exceptions while draining a streaming pipeline?

I am running into issues where I have a streaming scio pipeline running on Dataflow that is deduplicating messages and performing some counting by key. When I try to drain the pipeline I get a large amount of None.get exceptions supposedly thrown in my deduplicate step (I am basing this assumption off the label I am observing in the stackdriver log).
We are currently running on scio version 0.7.0-beta1 and beam version 2.8.0. I have tried protecting as much as I can in my code from any potential Nones but this appears like it is occurring further down inside of the deduplicate step.
The error I am getting is the following:
"java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at com.spotify.scio.util.Functions$$anon$2.mergeAccumulators(Functions.scala:227)
at com.spotify.scio.util.Functions$$anon$2.mergeAccumulators(Functions.scala:220)
at org.apache.beam.runners.dataflow.worker.WindmillStateInternals$WindmillCombiningState.getAccum(WindmillStateInternals.java:958)
at org.apache.beam.runners.dataflow.worker.WindmillStateInternals$WindmillCombiningState.read(WindmillStateInternals.java:920)
at org.apache.beam.runners.core.SystemReduceFn.onTrigger(SystemReduceFn.java:125)
at org.apache.beam.runners.core.ReduceFnRunner.onTrigger(ReduceFnRunner.java:1060)
at org.apache.beam.runners.core.ReduceFnRunner.onTimers(ReduceFnRunner.java:768)
at org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:95)
at org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
at org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:135)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:45)
at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:50)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:202)
at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:160)
at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1226)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:141)
at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:965)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
As you can see, this never really enters my code and I am unsure how I should go about finding this issue. Perhaps it has something to do with the "LateDataDroppingDoFnRunner"? Our allowed lateness is relatively large (3 days with windows being an hour long).
val input = PubsubIO.readStrings()
.fromSubscription(subscription)
.withTimestampAttribute("ts")
.withName("Window messages")
.withFixedWindows(
duration = windowSize,
options = WindowOptions(
trigger = AfterWatermark.pastEndOfWindow()
.withEarlyFirings(AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(earlyFiring))
.withLateFirings(AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(lateFiring)),
accumulationMode = ACCUMULATING_FIRED_PANES,
allowedLateness = allowedLateness
)
)
.withName(s"Deduplicate messages")
.distinctBy[String](f = getId)
...
// I am being overly cautious here because I have been having
// so much trouble debugging this
def getId(message: Map[String, Any]): String = {
message match {
case null => {
logger.warn("message is null when getting id")
""
}
case message => {
message.get("id") match {
case None => {
logger.warn("id is null in message")
""
}
case id => id.get.toString
}
}
}
}
I am confused how I could possibly be getting a None.get here and why that would only occur when I am draining.
Can I have some advice on how I should go about debugging this error or where I should be looking?

How to get the result of the KieSession build (i.e. the rules compiler errors)?

I am testing DROOLS 7.0 with a simple test Rule set using the following code:
KieContainer kc = KieServices.Factory.get().getKieClasspathContainer();
KieSession ksession = kc.newKieSession("DroolsTestKS");
...
The KieSession instance is returned even if there are errors in the rule .drl file, and no exception is thrown. I would like to check the result of the rules compilation.
The Drools reference (see 4.2.2.4) says that the build result can be obtained with:
KieServices kieServices = KieServices.Factory.get();
KieBuilder kieBuilder = kieServices.newKieBuilder( kfs ).buildAll();
assertEquals( 0, kieBuilder.getResults().getMessages( Message.Level.ERROR ).size() );
where kfs is a KieFileSystem instance, but the examples on how to build such a KieFileSystem in the previous pages of the manual are much more complex and a little bit confused IMHO.
Is there a way to have the Session buid result (i.e. to access a KieBuilder ) when creating a the KieSession with the simple two lines of code I show at the beginning of this post ?
I am answering to my question, because I've just found a solution:
KieContainer kc = KieServices.Factory.get().getKieClasspathContainer();
Results rs = kc.verify("KBase");
if (rs.hasMessages(Level.ERROR)) {
System.out.println("ERRORI DROOLS: " + rs.getMessages());
... // handle this
}
I'm wondering if with this validation the actual rules compilation is executed twice or not ... but anyway this method seems to work.