Combine multiple lines between flags in one line in AWK - awk

Example file:
Pattern 1
AAAAAAAAAA
BBBBBBBBBB
Pattern 2
I want to print the lines between two patterns in a file in one line.
From a previous question How to print lines between two patterns, inclusive or exclusive (in sed, AWK or Perl)? I found the very nice
awk '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag' file
With output:
AAAAAAAAAA
BBBBBBBBBB
My desired output:
AAAAAAAAAABBBBBBBBBB

Have your awk program in this way, written and tested in GNU awk.
awk '
/Pattern 2/{
if(found){
print val
}
found=""
next
}
/Pattern 1/{
found=1
next
}
found{
val=val $0
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
/Pattern 2/{ ##Checking if Pattern 2 is found here then do following.
if(found){ ##Checking if found is set then do following.
print val ##Printing val here.
}
found="" ##Nullifying found here.
next ##next will skip all statements from here.
}
/Pattern 1/{ ##Checking if Pattern 1 is found in current line.
found=1 ##Setting found to 1 here.
next ##next will skip all statements from here.
}
found{ ##Checking condition if found is SET then do following.
val=val $0 ##Creating val variable here which is keep adding current line values in it.
}
' Input_file ##Mentioning Input_file name here.

You may use this awk:
awk '/Pattern 2/ {if (s!="") print s; s=f=""} f {s = s $0} /Pattern 1/ {f=1}' file
AAAAAAAAAABBBBBBBBBB

And also with awk:
awk -v RS= '!/Pattern/ {sub(/\n/,"");print}' file
AAAAAAAAAABBBBBBBBBB

With GNU awk for multi-char RS and assuming your "Pattern"s really to take up whole lines and can't occur elsewhere in your input (easy fix if that's wrong):
$ awk -v RS='Pattern 2' 'sub(/.*Pattern 1/,""){gsub(/\n/,""); print}' file
AAAAAAAAAABBBBBBBBBB
or with any awk:
awk 'f{ if (/Pattern 2/){print buf; f=0} else buf=buf $0 } /Pattern 1/{f=1; buf=""}' file
AAAAAAAAAABBBBBBBBBB

You can set the output record separator to an empty string by using -v ORS=:
awk -v ORS= '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag' file
See an online demo.
To print a newline at the end, add END{print "\n"}:
awk -v ORS= '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag; END{print "\n"}' file > newfile
See the Ubuntu 18 screenshot:

Related

Optimise Bash scripting for large data

I have written a bash script where trying to obtain a new file from two files.
File1:
1000846364118,9,369,9901,0,2020.05.20 13:20:52,2020.07.14 16:38:11,2021.03.14 00:00:00,U,2020.07.14 16:38:11
1000683648398,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,U,2020.01.01 01:25:05
1000534726081,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,X,2020.01.01 01:25:05
File2:
1000846364118;0;;2021.04.04;9914;100084636;ISATD;U;TEST;1234567890;2;;0;0;0;0;2020.10.12.00:00:00;0;0
1000830686890;0;;2021.03.02;9807;100083068;ISATD;U;TEST;1234567891;2;;0;0;0;0;2020.10.12.00:00:01;0;0
1000835819335;0;;2021.03.21;9990;100083581;ISATD;U;TEST;1234567892;2;;0;0;0;0;2020.10.12.00:00:03;0;0
1000683648398;0;;2020.10.31;9829;100068364;ISATD;U;TEST;1234567893;2;;0;0;0;0;2020.10.12.00:00:06;0;0
New file will have rows from file1 only which is having pattern 'U' in it with extra column where 10th field(123456789X) of file2 will be there. So my final output will be like this:
1000846364118,9,369,9901,0,2020.05.20 13:20:52,2020.07.14 16:38:11,2021.03.14 00:00:00,U,2020.07.14 16:38:11,1234567890
1000683648398,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,U,2020.01.01 01:25:05,1234567893
My script is below and working fine but the only issue is the data with which I am plying is huge and to generate the file output it is taking too much time. I put a timespan after every step and found that for loop portion is taking hours to generate few KB data wherein I am playing with few hundred MBs of data. Need help to optimise it.
cat /dev/null > new_file
used_Serial_Number=`grep U file1 | awk -F "," '{print $1}'`
echo "Serial no extracted at `date`" # Till this portion is getting completed in 2-3mins
for i in $used_Serial_Number; do
msisdn=`grep $i file2 | awk -F ";" '{print $10}'`
grep $i file1 | awk -v msisdn=$msisdn -F "," 'BEGIN { OFS = "," } { print $0 , msisdn }' >> new_file
done
Could you please try following, written and tested with shown samples in GNU awk. In case your 9th field of Input_file1 could be u OR U then change from $9=="U" TO tolower($9)=="u" for matching both cases.
awk '
BEGIN{
FS=";"
OFS=","
}
FNR==NR{
a[$1]=$10
next
}
($1 in a) && $9=="U"{
print $0,a[$1]
}
' Input_file2 FS="," Input_file1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
FS=";" ##Setting FS as ; here.
OFS="," ##Setting OFS as , here.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when Input_file2 is being read.
a[$1]=$10 ##Creating array a with index $1 and value is $10 here.
next ##next will skip all further statements from here.
}
($1 in a) && $9=="U"{ ##Checking if $1 is in a and 9th field is U then do following.
print $0,a[$1] ##Printing current line along with value of a with index of $1 here.
}
' file2 FS="," file1 ##Mentioning Input_file2 then setting FS as , and mentioning Input_file1 here.

How to skip first line between two patterns in awk?

I have the next script
cat foo.txt | awk '/ERROR/,/INFO/'
With the input of:
FooFoo
ERROR
Foo1
INFO
FooFoo
Now the result is:
ERROR
Foo1
INFO
I am looking for the next result:
Foo1
INFO
How I can make it work?
Thanks for your help
Give this a try:
awk '/ERROR/,/INFO/' foo.txt | tail -n +2
If your input is from a file, you don't need the cat. just awk '...' file
Could you please try following, written and tested with shown samples in GNU awk.
awk '
/ERROR/{
found=1
next
}
found{
val=(val?val ORS:"")$0
}
/INFO/{
print val
val=count=found=""
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
/ERROR/{ ##Checking if line contains ERROR then do following.
found=1 ##Setting found variable here.
next ##next will skip all further statements from here.
}
found{ ##Checking here if found is SET then do following.
val=(val?val ORS:"")$0 ##Creating variable val and keep adding value to it in form of current line.
}
/INFO/{ ##Checking condition if INFO is found in current line then do following.
print val ##Printing val here.
val=count="" ##Nullifying val and count here.
}
' Input_file ##Mentioning Input_file name here.
Like this:
awk '
seen # a true (1) condition makes awk to print current line
/ERROR/{seen=1} # if we grep ERROR, assign 1 to seen flag
/INFO/{seen=0} # if we grep INFO, assign 0 to seen flag
' file
Output
Foo1
INFO

How to display the original line number in awk after removing common lines

I have the following code that only displays the lines that are not in the first file:
'NR==FNR{a[$0];next} !($0 in a)' compareAgainst myFile
How can I include the original line number next to the output?
Could you please try following.
awk 'NR==FNR{a[$0];next} !($0 in a){print FNR,$0}' compareAgainst myFile
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
NR==FNR{ ##Checking condition FNR==NR which will be TRUE when first Input_file compareAgainst is being read.
a[$0] ##Creating array a with index $0 here.
next ##next will skip all further statements from here.
}
!($0 in a){ ##Checking condition if current Line is not present in array a then do following.
print FNR,$0 ##Printing line number and current line here.
}
' compareAgainst myFile ##Mentioning Input_file names here.

How can I sum values in column based on the value in another column in Awk

INPUT :
TT,SS,ECID,CDID,ODID,Symbol,Side,LastQty,LastPx,CumQty,AvgPx,
"20191008-13:32:52","RO","0284","378MT","r7ot","SPD","1","100","290.67","400","290.67",
"20191008-13:33:13","RO","02DJ","378MT","r7o","SPD","1","100","290.68","2248","290.655",
"20191008-13:33:26","RO","FATS","378MTA","r7ot","PDF","1","100","290.92","2751","290.608",
Output should be :
SPD 200
PDF 100
Tried doing it using but doesn't work
$ awk '{a[$3]+=$4}END{for(i in a) print i,a[i]}' file
EDIT2: Since OP has old awk where FPAT is not there so as per samples I added following code then.
awk -F, '{gsub(/\r/,"")} FNR>1{gsub(/"/,"",$8);gsub(/"/,"",$6);a[$6]+=$8} END{for(i in a){print i,a[i] | "sort -k1"}}' Input_file
EDIT: Since OP changed Input_file completely so adding this solution now. Written and tested with GNU awk.
awk -v FPAT='[^,]*|"[^"]+"' '
gsub(/\r/,"")
FNR>1{
gsub(/"/,"",$8)
gsub(/"/,"",$6)
a[$6]+=$8
}
END{
for(i in a){
print i,a[i]
}
}
' Input_file
OR to sort output in alphabetic order try following.
awk -v FPAT='[^,]*|"[^"]+"' '{gsub(/\r/,"")} FNR>1{gsub(/"/,"",$8);gsub(/"/,"",$6);a[$6]+=$8} END{for(i in a){print i,a[i] | "sort -k1"}}' Input_file
You were close, problem with your approach is you haven't set field separator as , in your code but your Input_file has separator as , so it is not having $3 at all and hence not working. Could you please try following.
awk -F"[[:space:]]*,[[:space:]]*" 'FNR>1{a[$3]+=$4} END{for(i in a){print i,a[i]}}' Input_file
PS: Thanks to oguz ismail for letting know about field separator set.

Grepping all strings on the same line from multiple files

Trying to find a way to grep all names on one line for 100 files. grepping all names available in each file must appear on the same line.
FILE1
"company":"COMPANY1","companyDisplayName":"CM1","company":"COMPANY2","companyDisplayName":"CM2","company":"COMPANY3","companyDisplayName":"CM3",
FILE2
"company":"COMPANY99","companyDisplayName":"CM99"
The output i actually want is, ( include file name as prefix.)
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99
i tried grep -oP '(?<="company":")[^"]*' * but i get results like this :
FILE1:COMPANY1
FILE1:COMPANY2
FILE1:COMPANY3
FILE2:COMPANY99
Could you please try following.
awk -F'[,:]' '
BEGIN{
OFS=","
}
{
for(i=1;i<=NF;i++){
if($i=="\"company\""){
val=(val?val OFS:"")$(i+1)
}
}
gsub(/\"/,"",val)
print FILENAME":"val
val=""
}
' Input_file1 Input_file2
Explanation: Adding explanation for above code.
awk -F'[,:]' ' ##Starting awk program here and setting field separator as colon OR comma here for all lines of Input_file(s).
BEGIN{ ##Starting BEGIN section of awk here.
OFS="," ##Setting OFS as comma here.
} ##Closing BEGIN BLOCK here.
{ ##Starting main BLOCK here.
for(i=1;i<=NF;i++){ ##Starting a for loop which starts from i=1 to till value of NF.
if($i=="\"company\""){ ##Checking condition if field value is equal to "company" then do following.
val=(val?val OFS:"")$(i+1) ##Creating a variable named val and concatenating its own value to it each time cursor comes here.
} ##Closing BLOCK for if condition here.
} ##Closing BLOCK for, for loop here.
gsub(/\"/,"",val) ##Using gsub to gklobally substitute all " in variable val here.
print FILENAME":"val ##Printing filename colon and variable val here.
val="" ##Nullifying variable val here.
} ##Closing main BLOCK here.
' Input_file1 Input_file2 ##Mentioning Input_file names here.
Output will be as follows.
Input_file1:COMPANY1,COMPANY2,COMPANY3
Input_file2:COMPANY99
EDIT: Adding solution in case OP needs to use grep and want to get final output from its output(though I will recommend to use awk solution itself since we are NOT using multiple commands or sub-shells).
grep -oP '(?<="company":")[^"]*' * | awk 'BEGIN{FS=":";OFS=","} prev!=$1 && val{print prev":"val;val=""} {val=(val?val OFS:"")$2;prev=$1} END{if(val){print prev":"val}}'
There are two tools that can take the output of your grep command and reformat it the way you want. First tool is GNU datamash. Second is tsv-summarize from eBay's tsv-utils package (disclaimer: I'm the author). Both tools solve this in similar ways:
$ # The grep output
$ echo $'FILE1:COMPANY1\nFILE1:COMPANY2\nFILE1:COMPANY3\nFILE2:COMPANY99' > grep-output.txt
$ cat grep-output.txt
FILE1:COMPANY1
FILE1:COMPANY2
FILE1:COMPANY3
FILE2:COMPANY99
$ # Using GNU datamash
$ cat grep-output.txt | datamash -field-separator : --group 1 unique 2
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99
$ # Using tsv-summarize
$ cat grep-output.txt | tsv-summarize --delimiter : --group-by 1 --unique-values 2 --values-delimiter ,
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99