Kettle - Text File Unstructured - pentaho

Question

I have a text file that its structure is not in a single line, it is certain that the lines start with zero (0). Below is the sample:
header : TEXT<br>
header : TEXT<br>
header : TEXT<br>
line 1 : 0TEXT Name Other Field<br>
line 2 : TEXT Other Field Phone<br>
line 3 : 0TEXT Name Other Field<br>
line 4 : TEXT Other Field Phone<br>
line 5 : 0TEXT textexttexttext <br>
line 6 : 0TEXT Name Other Field<br>
line 7 : TEXT Other Field Phone<br>
line 8 : 0TEXT Name Other Field<br>
line 9 : TEXT Other Field Phone<br>
What I want to do is get through a regex evaluation the NAME and the PHONE fields and store this values.
Name, Phone
Name, Phone
The regex part is ok, I already did it.
What I need to know is how to get the values from two different lines and put it in the same register?
I found this forum http://forums.pentaho.com/showthread.php?53288-Reading-multi-line-records-from-text-file-newbie and tried to apply a javascript suggested, but it didn't work for me, maybe I did something wrong..
I really did some simple wrong and fixed it.
js..
var x;
var charInitial = line.toString().charAt(0);
if(charInitial == '0') {
x = line.toString();
}
else{
x += line.toString();
}
With this script I get the rows separated, I want to concatenate them and after apply the regex. I can concatenate all the rows that belong to the group, and with a regex I can drop that ones that are unnecessary.
Thanks

Answer 1

Given that you have those records in multiple rows, you have the following options:
1) Group by: as long as you can identify your rows that belong together via some set of keys, you can use a group by and create two new fields, Name and Phone, obtained by "Concatenate fields separated by" (not the "concatenate fields separated by ,", mind that). If the values are either what you want to keep or null, the concatenation works;
2) De-normalize. Same principle applies, you need a set of keys to identify records that belong together, but you will need both your Name and Phone to be in the same field (e.g, Value) and you need another field with the key (either Name or Phone).
3) Perhaps the best one: Analytic Query: Use "Lag N rows forward and get field" with N=1 and you get the phone number of the next row. After this step you have rows with a Not null name and the next row's phone number; rows with a null name and a null phone number. Filter the rows you want after and you're done.
This is just a generic idea. You have to sort out the details.