In pseudo code I want to achieve this within groovy.
file
a a1 a2 a3
b b1 b2 b3
c c1 c2 c3
file = /path/to/file
for (line in file) {
var_1=line1.a
var_2=line.a2
var_3=line.a3
}
echo $var_1
echo $var_2
echo $var_3
new File('/path/to/file').withReader("UTF-8"){r->
def headers=null
r.eachLine{line, index->
line = line.split(' ') //one space is a column delimiter
if(index==1){
//convert first line to a map - [ column-index : column-name ]
headers=line.toList().indexed()
}else{
//convert line to headers-to-values map to be able to access value by column name
line = headers.collectEntries{k,v-> [v, line[k]] }
println line.a
println line.b
println line.c
}
}
}
Related
I am trying to remove a String at a certain index in my list of Strings however I can't seem to invoke the list.remove() method in groovy.
public List getCassetteTypes(socket, numOfSlots){
//get the cassettes layout
sendCommand(socket, 'syst:layout? ')
String systLayoutStr = readCommand(socket)
//this String looks like: '1 ABC, 2 DEF, 3 SPN, ....'
List listOfCassetteTypes = new ArrayList<String>()
//I split the String at ',' because for each cassetteName, I want to remove the number before it
listOfCassetteTypes = systLayoutStr.split(',')
for(int i = 0; i < numOfSlots; i++){
//remove any white spaces
listOfCassetteTypes[i] = listOfCassetteTypes[i].trim()
//remove the numerical value
listOfCassetteTypes[i] = listOfCassetteTypes[i].replace((i + 1) + ' ', '')
/* if the cassette name is 'SPN',
I want to remove it and append '-EXT' to the cassetteName before it,
because 'SPN' means the previous slot is extended,
'SPN' itself isn't a cassette */
if(listOfCassetteTypes[i].equalsIgnoreCase('spn')){
listOfCassetteTypes[i - 1] = listOfCassetteTypes[i - 1].concat('-EXT')
//this is what is not working for me, everything else is fine.
listOfCassetteTypes = listOfCassetteTypes.remove(i)
}
}
return listOfCassetteTypes
}
I've tried several different ways but none of them seem to work.
Instead of manipulating the list, you could process each entry in a pair with it's successor... I believe this does what you're after?
def layoutStr = '1 ABC, 2 DEF, 3 SPN, 4 GHI'
def splitted = layoutStr.split(',')
*.trim() // remove white space from all the entries (note *)
*.dropWhile { it ==~ /[0-9 ]/ } // drop until you hit a char that isn't a number or space
.collate(2, 1, true) // group them with the next element in the list
.findAll { it[0] != 'SPN' } // if a group starts with SPN, drop it
.collect {
// If the successor is SPN add -EXT, otherwise, just return the element
it[1] == 'SPN' ? "${it[0]}-EXT" : it[0]
}
assert splitted == ['ABC', 'DEF-EXT', 'GHI']
Followup question
To just get the numbers of the ones not SPN:
def layoutStr = '1 ABC, 2 DEF, 3 SPN, 4 GHI'
def splitted = layoutStr.split(',')
*.trim() // remove white space from all the entries (note *)
*.split(/\s+/) // Split all the entries on whitespace
.findResults { it[1] == 'SPN' ? null : it[0] } // Only keep those that don't have SPN
Note that this is a list of Strings, not Integers... If you need integers then:
.findResults { it[1] == 'SPN' ? null : it[0] as Integer }
I am trying to solve a problem where my csv data looks like below:
A|B|C
"Jon"|"PR | RP"|"MN"
"Pam | Map"|"Ecom"|"unity"
"What"|"is"this" happening"|"?"
That is, it is pipe delimited and has quotes as text qualifier but it also has pipe and quotes with in the data values. I have already tried
Update based on the comments
I tried to select | as delimiter and " as Text Qualifier but when trying to import data to OLEDB Destination i receive the following error:
couldn't find column delimiter for column B
You have to change the Column Delimiter property to | (vertical bar) and the Text Qualifier property to " within the Flat File Connection Manager
If these is still not working then you have some bad rows in the Flat File Source which you must handle using the Error Output:
Flat File source Error Output connection in SSIS
SQL SERVER – SSIS Component Error Outputs
I actually ended up writing a c sharp script to remove the initial and last quote and set the column delimiter to quote pipe quote ("|") in SSIS. Code was as below:
public void Main()
{
String folderSource = "path";
String folderTarget = "path";
foreach (string file in System.IO.Directory.GetFiles(folderSource))
{
String targetfilepath = folderTarget + System.IO.Path.GetFileName(file);
System.IO.File.Delete(targetfilepath);
int icount = 1;
foreach (String row in System.IO.File.ReadAllLines(file))
{
if (icount == 1)
{
System.IO.File.AppendAllText(targetfilepath, row.Replace("|", "\"|\""));
}
else
{
System.IO.File.AppendAllText(targetfilepath, row.Substring(1, row.Length - 2));
}
icount = icount + 1;
System.IO.File.AppendAllText(targetfilepath, Environment.NewLine);
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Using tcl, i want to pass variable parameters to function.
i tried this code
proc launch_proc { msg proc_name {params {}} } {
puts "params launch_proc is $params \n"
}
proc test { param } {
puts "param test is $param \n"
launch_proc "1.5.2 test param" test_standard {{*}$param param1 param2 param3"
}
}
test value
--> params launch_proc is {*}$param param1 param2 param3"
$param is not evaluated (i use tcl 8.5)
Tcl has support for this using the keyword args as the last parameter in the argument list.
Here is an example directly from the wiki:
proc demo {first {second "none"} args} {
puts "first = $first"
puts "second = $second"
puts "args = $args"
}
demo one
demo one two
demo one two three four five
results in
first = one
second = none
args =
first = one
second = two
args =
first = one
second = two
args = three four five
You can use the expand syntax for this as well which makes the following two calls to demo equivalent.
demo one two three four five
set params {three four five}
demo one two {*}$params
You're passing a list and need to instead send each item as a parameter to the proc.
proc test {p1 p2 p3} {
puts "$p1 - $p2 - $p3"
}
set value {one two three}
# This should work in tcl 8.5+
test {*}$value
set value {four five six}
# For tcl < 8.5
foreach {p1 p2 p3} $value {
test $p1 $p2 $p3
break
}
# or
set value {seven eight nine}
test [lindex $value 0] [lindex $value 1] [lindex $value 2]
Output:
$ tclsh test.tcl
one - two - three
four - five - six
seven - eight - nine
You need upvar, and use list when you construct the params list for launch_proc
proc test {varname} {
upvar 1 $varname value
puts "param test is $varname => $value"
launch_proc "1.5.2 test param" test_standard [list {*}$value par1 par2 par3]
}
proc launch_proc {msg proc_name {params {}}} {
puts "params launch_proc: [join $params ,]"
}
set value {"a b c" "d e f" "g h i"}
test value
param test is value => "a b c" "d e f" "g h i"
params launch_proc: a b c,d e f,g h i,par1,par2,par3
I was wondering if it is possible to combine column values based upon a condition. Let me explain...
Let say my data looks like this
Id name offset
1 Jan 100
2 Janssen 104
3 Klaas 150
4 Jan 160
5 Janssen 164
An my output should be this
Id fullname offsets
1 Jan Janssen [ 100, 160 ]
I would like to combine the name values from two rows where the offset of the two rows are no more apart then 1 character.
My question is if this type of data manipulation is possible with and if it is could someone share some code and explaination?
Please be gentle but this little piece of code return some what what I want...
ArrayList<String> persons = new ArrayList<String>();
// write your code here
String _previous = "";
//Sample output form entities.txt
//USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Berkowitz,PERSON,9,10660
//USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,10685
File file = new File("entities.txt");
try {
//
// Create a new Scanner object which will read the data
// from the file passed in. To check if there are more
// line to read from it we check by calling the
// scanner.hasNextLine() method. We then read line one
// by one till all line is read.
//
Scanner scanner = new Scanner(file);
while (scanner.hasNextLine()) {
if(_previous == "" || _previous == null)
_previous = scanner.nextLine();
String _current = scanner.nextLine();
//Compare the lines, if there offset is = 1
int x = Integer.parseInt(_previous.split(",")[3]) + Integer.parseInt(_previous.split(",")[4]);
int y = Integer.parseInt(_current.split(",")[4]);
if(y-x == 1){
persons.add(_previous.split(",")[1] + " " + _current.split(",")[1]);
if(scanner.hasNextLine()){
_current = scanner.nextLine();
}
}else{
persons.add(_previous.split(",")[1]);
}
_previous = _current;
}
} catch (Exception e) {
e.printStackTrace();
}
for(String person : persons){
System.out.println(person);
}
Working of this piece sample data
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Richard,PERSON,7,2732
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,2740
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,2756
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,3093
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,3195
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Berkowitz,PERSON,9,3220
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Berkowitz,PERSON,9,10660
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,10685
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Lea,PERSON,3,10858
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Lea,PERSON,3,11063
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Ken,PERSON,3,11186
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Marottoli,PERSON,9,11234
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Berkowitz,PERSON,9,17073
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Lea,PERSON,3,17095
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Stephanie,PERSON,9,17330
USER.A-GovDocs-f83c6ca3-9585-4c66-b9b0-f4c3bd57ccf4,Putt,PERSON,4,17340
Which produces this output
Richard Marottoli
Marottoli
Marottoli
Marottoli
Berkowitz
Berkowitz
Marottoli
Lea
Lea
Ken
Marottoli
Berkowitz
Lea
Stephanie Putt
Kind regards
Load the table using below create table
drop table if exists default.stack;
create external table default.stack
(junk string,
name string,
cat string,
len int,
off int
)
ROW FORMAT DELIMITED
FIELDS terminated by ','
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location 'hdfs://nameservice1/....';
Use below query to get your desired output.
select max(name), off from (
select CASE when b.name is not null then
concat(b.name," ",a.name)
else
a.name
end as name
,Case WHEN b.off1 is not null
then b.off1
else a.off
end as off
from default.stack a
left outer join (select name
,len+off+ 1 as off
,off as off1
from default.stack) b
on a.off = b.off ) a
group by off
order by off;
I have tested this it generates your desired result.
I have following tuple H1 and I want to strsplit its $0 into tuple.However I always get an error message:
DUMP H1:
(item32;item31;,1)
m = FOREACH H1 GENERATE STRSPLIT($0, ";", 50);
ERROR 1000: Error during parsing. Lexical error at line 1, column 40.
Encountered: after : "\";"
Anyone knows what's wrong with the script?
There is an escaping problem in the pig parsing routines when it encounters this semicolon.
You can use a unicode escape sequence for a semicolon: \u003B. However this must also be slash escaped and put in a single quoted string. Alternatively, you can rewrite the command over multiple lines, as per Neil's answer. In all cases, this must be a single quoted string.
H1 = LOAD 'h1.txt' as (splitme:chararray, name);
A1 = FOREACH H1 GENERATE STRSPLIT(splitme,'\\u003B'); -- OK
B1 = FOREACH H1 GENERATE STRSPLIT(splitme,';'); -- ERROR
C1 = FOREACH H1 GENERATE STRSPLIT(splitme,':'); -- OK
D1 = FOREACH H1 { -- OK
splitup = STRSPLIT( splitme, ';' );
GENERATE splitup;
}
A2 = FOREACH H1 GENERATE STRSPLIT(splitme,"\\u003B"); -- ERROR
B2 = FOREACH H1 GENERATE STRSPLIT(splitme,";"); -- ERROR
C2 = FOREACH H1 GENERATE STRSPLIT(splitme,":"); -- ERROR
D2 = FOREACH H1 { -- ERROR
splitup = STRSPLIT( splitme, ";" );
GENERATE splitup;
}
Dump H1;
(item32;item31;,1)
Dump A1;
((item32,item31))
Dump C1;
((item32;item31;))
Dump D1;
((item32,item31))
STRSPLIT on a semi-colon is tricky. I got it to work by putting it inside of a block.
raw = LOAD 'cname.txt' as (name,cname_string:chararray);
xx = FOREACH raw {
cname_split = STRSPLIT(cname_string,';');
GENERATE cname_split;
}
Funny enough, this is how I originally implemented my STRSPLIT() command. Only after trying to get it to split on a semicolon did I run into the same issue.