remove all lines in a file containing a string from another file - awk

I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.

EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d '\r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk), append > temp_file && mv temp_file Input_file in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.

Related

Combine multiple lines between flags in one line in AWK

Example file:
Pattern 1
AAAAAAAAAA
BBBBBBBBBB
Pattern 2
I want to print the lines between two patterns in a file in one line.
From a previous question How to print lines between two patterns, inclusive or exclusive (in sed, AWK or Perl)? I found the very nice
awk '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag' file
With output:
AAAAAAAAAA
BBBBBBBBBB
My desired output:
AAAAAAAAAABBBBBBBBBB
Have your awk program in this way, written and tested in GNU awk.
awk '
/Pattern 2/{
if(found){
print val
}
found=""
next
}
/Pattern 1/{
found=1
next
}
found{
val=val $0
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
/Pattern 2/{ ##Checking if Pattern 2 is found here then do following.
if(found){ ##Checking if found is set then do following.
print val ##Printing val here.
}
found="" ##Nullifying found here.
next ##next will skip all statements from here.
}
/Pattern 1/{ ##Checking if Pattern 1 is found in current line.
found=1 ##Setting found to 1 here.
next ##next will skip all statements from here.
}
found{ ##Checking condition if found is SET then do following.
val=val $0 ##Creating val variable here which is keep adding current line values in it.
}
' Input_file ##Mentioning Input_file name here.
You may use this awk:
awk '/Pattern 2/ {if (s!="") print s; s=f=""} f {s = s $0} /Pattern 1/ {f=1}' file
AAAAAAAAAABBBBBBBBBB
And also with awk:
awk -v RS= '!/Pattern/ {sub(/\n/,"");print}' file
AAAAAAAAAABBBBBBBBBB
With GNU awk for multi-char RS and assuming your "Pattern"s really to take up whole lines and can't occur elsewhere in your input (easy fix if that's wrong):
$ awk -v RS='Pattern 2' 'sub(/.*Pattern 1/,""){gsub(/\n/,""); print}' file
AAAAAAAAAABBBBBBBBBB
or with any awk:
awk 'f{ if (/Pattern 2/){print buf; f=0} else buf=buf $0 } /Pattern 1/{f=1; buf=""}' file
AAAAAAAAAABBBBBBBBBB
You can set the output record separator to an empty string by using -v ORS=:
awk -v ORS= '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag' file
See an online demo.
To print a newline at the end, add END{print "\n"}:
awk -v ORS= '/Pattern 1/{flag=1; next} /Pattern 2/{flag=0} flag; END{print "\n"}' file > newfile
See the Ubuntu 18 screenshot:

Optimise Bash scripting for large data

I have written a bash script where trying to obtain a new file from two files.
File1:
1000846364118,9,369,9901,0,2020.05.20 13:20:52,2020.07.14 16:38:11,2021.03.14 00:00:00,U,2020.07.14 16:38:11
1000683648398,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,U,2020.01.01 01:25:05
1000534726081,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,X,2020.01.01 01:25:05
File2:
1000846364118;0;;2021.04.04;9914;100084636;ISATD;U;TEST;1234567890;2;;0;0;0;0;2020.10.12.00:00:00;0;0
1000830686890;0;;2021.03.02;9807;100083068;ISATD;U;TEST;1234567891;2;;0;0;0;0;2020.10.12.00:00:01;0;0
1000835819335;0;;2021.03.21;9990;100083581;ISATD;U;TEST;1234567892;2;;0;0;0;0;2020.10.12.00:00:03;0;0
1000683648398;0;;2020.10.31;9829;100068364;ISATD;U;TEST;1234567893;2;;0;0;0;0;2020.10.12.00:00:06;0;0
New file will have rows from file1 only which is having pattern 'U' in it with extra column where 10th field(123456789X) of file2 will be there. So my final output will be like this:
1000846364118,9,369,9901,0,2020.05.20 13:20:52,2020.07.14 16:38:11,2021.03.14 00:00:00,U,2020.07.14 16:38:11,1234567890
1000683648398,9,369,9901,0,2019.05.04 19:50:39,2019.06.23 14:27:17,2019.12.31 23:59:59,U,2020.01.01 01:25:05,1234567893
My script is below and working fine but the only issue is the data with which I am plying is huge and to generate the file output it is taking too much time. I put a timespan after every step and found that for loop portion is taking hours to generate few KB data wherein I am playing with few hundred MBs of data. Need help to optimise it.
cat /dev/null > new_file
used_Serial_Number=`grep U file1 | awk -F "," '{print $1}'`
echo "Serial no extracted at `date`" # Till this portion is getting completed in 2-3mins
for i in $used_Serial_Number; do
msisdn=`grep $i file2 | awk -F ";" '{print $10}'`
grep $i file1 | awk -v msisdn=$msisdn -F "," 'BEGIN { OFS = "," } { print $0 , msisdn }' >> new_file
done
Could you please try following, written and tested with shown samples in GNU awk. In case your 9th field of Input_file1 could be u OR U then change from $9=="U" TO tolower($9)=="u" for matching both cases.
awk '
BEGIN{
FS=";"
OFS=","
}
FNR==NR{
a[$1]=$10
next
}
($1 in a) && $9=="U"{
print $0,a[$1]
}
' Input_file2 FS="," Input_file1
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
FS=";" ##Setting FS as ; here.
OFS="," ##Setting OFS as , here.
}
FNR==NR{ ##Checking condition if FNR==NR which will be TRUE when Input_file2 is being read.
a[$1]=$10 ##Creating array a with index $1 and value is $10 here.
next ##next will skip all further statements from here.
}
($1 in a) && $9=="U"{ ##Checking if $1 is in a and 9th field is U then do following.
print $0,a[$1] ##Printing current line along with value of a with index of $1 here.
}
' file2 FS="," file1 ##Mentioning Input_file2 then setting FS as , and mentioning Input_file1 here.

How to display the original line number in awk after removing common lines

I have the following code that only displays the lines that are not in the first file:
'NR==FNR{a[$0];next} !($0 in a)' compareAgainst myFile
How can I include the original line number next to the output?
Could you please try following.
awk 'NR==FNR{a[$0];next} !($0 in a){print FNR,$0}' compareAgainst myFile
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
NR==FNR{ ##Checking condition FNR==NR which will be TRUE when first Input_file compareAgainst is being read.
a[$0] ##Creating array a with index $0 here.
next ##next will skip all further statements from here.
}
!($0 in a){ ##Checking condition if current Line is not present in array a then do following.
print FNR,$0 ##Printing line number and current line here.
}
' compareAgainst myFile ##Mentioning Input_file names here.

Grepping all strings on the same line from multiple files

Trying to find a way to grep all names on one line for 100 files. grepping all names available in each file must appear on the same line.
FILE1
"company":"COMPANY1","companyDisplayName":"CM1","company":"COMPANY2","companyDisplayName":"CM2","company":"COMPANY3","companyDisplayName":"CM3",
FILE2
"company":"COMPANY99","companyDisplayName":"CM99"
The output i actually want is, ( include file name as prefix.)
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99
i tried grep -oP '(?<="company":")[^"]*' * but i get results like this :
FILE1:COMPANY1
FILE1:COMPANY2
FILE1:COMPANY3
FILE2:COMPANY99
Could you please try following.
awk -F'[,:]' '
BEGIN{
OFS=","
}
{
for(i=1;i<=NF;i++){
if($i=="\"company\""){
val=(val?val OFS:"")$(i+1)
}
}
gsub(/\"/,"",val)
print FILENAME":"val
val=""
}
' Input_file1 Input_file2
Explanation: Adding explanation for above code.
awk -F'[,:]' ' ##Starting awk program here and setting field separator as colon OR comma here for all lines of Input_file(s).
BEGIN{ ##Starting BEGIN section of awk here.
OFS="," ##Setting OFS as comma here.
} ##Closing BEGIN BLOCK here.
{ ##Starting main BLOCK here.
for(i=1;i<=NF;i++){ ##Starting a for loop which starts from i=1 to till value of NF.
if($i=="\"company\""){ ##Checking condition if field value is equal to "company" then do following.
val=(val?val OFS:"")$(i+1) ##Creating a variable named val and concatenating its own value to it each time cursor comes here.
} ##Closing BLOCK for if condition here.
} ##Closing BLOCK for, for loop here.
gsub(/\"/,"",val) ##Using gsub to gklobally substitute all " in variable val here.
print FILENAME":"val ##Printing filename colon and variable val here.
val="" ##Nullifying variable val here.
} ##Closing main BLOCK here.
' Input_file1 Input_file2 ##Mentioning Input_file names here.
Output will be as follows.
Input_file1:COMPANY1,COMPANY2,COMPANY3
Input_file2:COMPANY99
EDIT: Adding solution in case OP needs to use grep and want to get final output from its output(though I will recommend to use awk solution itself since we are NOT using multiple commands or sub-shells).
grep -oP '(?<="company":")[^"]*' * | awk 'BEGIN{FS=":";OFS=","} prev!=$1 && val{print prev":"val;val=""} {val=(val?val OFS:"")$2;prev=$1} END{if(val){print prev":"val}}'
There are two tools that can take the output of your grep command and reformat it the way you want. First tool is GNU datamash. Second is tsv-summarize from eBay's tsv-utils package (disclaimer: I'm the author). Both tools solve this in similar ways:
$ # The grep output
$ echo $'FILE1:COMPANY1\nFILE1:COMPANY2\nFILE1:COMPANY3\nFILE2:COMPANY99' > grep-output.txt
$ cat grep-output.txt
FILE1:COMPANY1
FILE1:COMPANY2
FILE1:COMPANY3
FILE2:COMPANY99
$ # Using GNU datamash
$ cat grep-output.txt | datamash -field-separator : --group 1 unique 2
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99
$ # Using tsv-summarize
$ cat grep-output.txt | tsv-summarize --delimiter : --group-by 1 --unique-values 2 --values-delimiter ,
FILE1:COMPANY1,COMPANY2,COMPANY3
FILE2:COMPANY99

Use awk to limit output of transparent huge pages

How would I use awk to filter the current setting of the transparent huge pages?
Example output of transparent huge pages:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
I would like to output only the current setting:
never
Following awk may also help you on same.
awk -F"[][]" 'NF{print $2}' Input_file
Explanation: following is not exact code it is only for explanation purposes only.
-F"[][]" ##Setting field separator as ] and [ for each line in Input_file.
'NF{ ##Checking condition here if line is NOT NULL where NF is number of fields awk variable which will be set only when a line is NOT NULL.
print $2} ##If above condition is TRUE then print the 2nd field of current line of Input_file.
' Input_file ##Mentioning Input_file name here.
How about something like this?
awk -F'[\\]\\[]' '/always madvise/ && NF > 2 { print $2 }' /sys/kernel/mm/transparent_hugepage/enabled