Pentaho PDI: Final value of previous row's calculated field - pentaho

Question

I tried to use the Analytik Query step to access some calculated field of the previous row. Turns out that the rows are all calculated in parallel and that accessing the previous row's fields gives you the current value they have during their processing, which is kind of random. It does not seem to be possible to obtain the final value of a field of a previous row. Or is there any other way than the Analytik Query step? I imagine all I need is a checkbox "Wait for previous rows to complete"...
What I need this for: I am processing time dependent data and doing a state recognition. When I am currently in state A, I do other stuff with my data then when I am in state B. So I need to know the state of the previous data row (which is determined not before the end of my transformation).
It can be done is Excel really easy, so I guess there must be some way in PDI. :-)
Thanks for any help!

Answer 1

If i have understood your question correctly, you may try using the Block this step until steps finish. This step waits until all the step copies that are specified in the dialog have finished. Read the link for more.
Hope this helps:)

Answer 2

I believe that it can be resolved by using the User Defined Java Class (UDJC) step.
If you sort the rows before processing them, the Sort By step would wait for the last row set by default.
Here's the most basic example of writing an output row for each input row. One important thing to keep in mind with the User Defined Java Class step, is the fact that they rewrite your whole data set, therefore need to be well thought of, especially if you do look-backs at previous rows. I hope this helps a bit.
// A class member that stores the previous row:
public Object[] previousRow;
public boolean processRow(StepMetaInterface smi, StepDataInterface sdi)
throws KettleException {
// Fetching row:
Object[] r = getRow();
// Check if r is null:
if (r == null) {
setOutputDone();
return false;
}
// Get some field's value:
String someFieldValue = get(Fields.In, "someFieldName").getString(r);
// Log value if you want:
logBasic("current field value is " + someFieldValue);
// Generate an output row object:
Object[] outputRow = RowDataUtil.createResizedCopy(r, data.outputRowMeta.size());
// Modify row's field values if needed:
get(Fields.Out, "someFieldName").setValue(outputRow, "a modified value here");
// Write row:
putRow(data.outputRowMeta, outputRow);
// Assign current row to previous row:
previousRow = r;
return true;
}
EDIT:
One more important thing to note about PDI - the blocking method, either by blocking steps or by the Sort by step, is done by checking row sets rather than single rows.
How can this be verified?
Right click --> Transformation Settings --> Miscellaneous --> Nr of rows in rowset.
The default value is 10000 rows. PDI developers often create a deadlock by using one of the blocking steps with a row set size that doesn't fit their data volume - do keep that in mind.

Answer 3

Use "Identify last row in a stream" & "Filter rows" transformations. The 1st transformation checks if its the last row and returns a Boolean value and the later can be used to filter the records based on the Boolean value returned.