awk 1 unexpected character '.' suddenly appeared - awk

the script was working. I added some comments and renamed it then submitted it. today my instructor told me it doesnt work and give me the error of awk 1 unexpected character '.'
the script is supposed to read a name in command line and return the student information for the name back.
right now I checked it and surprisingly it gives me the error.
I should run it by the command like this:
scriptName -v name="aname" -f filename
what is this problem and which part of my code make it?
#!/usr/bin/awk
BEGIN{
tmp=name;
nameIsValid;
if (name && tolower(name) eq ~/^[a-z]+$/ )
{
inputName=tolower(name)
nameIsValid++;
}
else
{
print "you have not entered the student name"
printf "Enter the student's name: "
getline inputName < "-"
tmp=inputName;
if (tolower(inputName) eq ~/^[a-z]+$/)
{
tmpName=inputName
nameIsValid++
}
else
{
print "Enter a valid name!"
exit
}
}
inputName=tolower(inputName)
FS=":"
}
{
if($1=="Student Number")
{
split ($0,header,FS)
}
if ($1 ~/^[0-9]+$/ && length($1)==8)
{
split($2,names," ")
if (tolower(names[1]) == inputName || tolower(names[2])==inputName )
{
counter++
for (i=1;i<=NF;i++)
{
printf"%s:%s ",header[i], $i
}
printf "\n"
}
}
}
END{
if (counter == 0 && nameIsValid)
{
printf "There is no record for the %-10s\n" , tmp
}
}

Here are the steps to fix the script:
Get rid of all those spurious NULL statements (trailing semi-colons at the end of lines).
Get rid of the unset variable eq (it is NOT an equality operator!) from all of your comparions.
Cleanup the indenting.
Get rid of that first non-functional nameIsValid; statement.
Change printf "\n" to the simpler print "".
Get rid of the useless ,FS arg to split().
Change name && tolower(name) ~ /^[a-z]+$/ to just the second part of that condition since if that matches then of course name is populated.
Get rid of all of those tolower()s and use character classes instead of explicit a-z ranges.
Get rid of the tmp variable.
Simplify your BEGIN logic.
Get rid of the unnecessary nameIsValid variable completely.
Make the awk body a bit more awk-like
And here's the result (untested since no sample input/output posted):
BEGIN {
if (name !~ /^[[:alpha:]]+$/ ) {
print "you have not entered the student name"
printf "Enter the student's name: "
getline name < "-"
}
if (name ~ /^[[:alpha:]]+$/) {
inputName=tolower(name)
FS=":"
}
else {
print "Enter a valid name!"
exit
}
}
$1=="Student Number" { split ($0,header) }
$1 ~ /^[[:digit:]]+$/ && length($1)==8 {
split(tolower($2),names," ")
if (names[1]==inputName || names[2]==inputName ) {
counter++
for (i=1;i<=NF;i++) {
printf "%s:%s ",header[i], $i
}
print ""
}
}
}
END {
if (counter == 0 && inputName) {
printf "There is no record for the %-10s\n" , name
}
}

I changed the shebang line to:
#!/usr/bin/awk -f
and then in command line didnt use -f. It is working now

Run the script in the following way:
awk -f script_name.awk input_file.txt
This seems to suppress the warnings and errors.

In my case, the problem was resetting the IFS variable to be IFS="," as suggested in this answer for splitting string into an array. So I resetted the IFS variable and got my code to work.
IFS=', '
read -r -a array <<< "$string"
IFS=' ' # reset IFS

Related

Error trying to redirect output of awk script to a new file

I am working on the following code in an awk script and I need the output to be redirected to another file within the same script.
BEGIN { FS=OFS="," }
NR==1 {print; next}
{ $9 = sprintf("%0.2f", $9) }
{ a[$0]++ }
BEGIN { FS=OFS="," }
{ gsub(/\r/,"") }
FNR==1 { $10="Survival Percentage" }
FNR > 1 && ($5+0==$5 && $6+0==$6 && $3+0==$3){
$10=sprintf("%0.2f",(($5-$6)/$3)*100)
}1
END {
if (i>0){
for (i in a){
print "i" > nj.csv
}}}
This is my code and just by executing it I get an error pointing to the point between nj and csv (nj.csv). Any idea to solve it?
gsub(/\r/,"") is almost always the wrong thing to do, you probably meant sub(/\r$/,""), and print "i" > nj.csv should be print i > "nj.csv" but idk why you have 2 identical BEGIN sections or what the overall purpose of the script is as it doesn't seem to make any sense.

GAWK does not terminate after ENDFILE block with single file

I have a gawk script below that reads a protein FASTA file and only prints out the records that don't have an X in their sequence and are within a certain range length. I wanted to modify the file in place so I had the script write to a temporary file and then rename it to the original file. The BEGINFILE and ENDFILE constructs in gawk seemed convenient for this. However, for some reason, gawk does not exit after executing the code in the ENDFILE even if it is given a single file argument. It seems to jump back to another line of code and then just hang. Does anyone know what could cause this to happen? The weird part is that this doesn't happen for every FASTA file, only a few and I can't tell what is different between the ones that trigger the bug and the ones that don't
#! /bin/gawk -f
function trim(s) {
gsub(/^[ \t]+|[ \t]+$/, "", s)
return s
}
function printFasta(header, seq, outfile, seq_line_max_chars) {
print ">" header > outfile
seq_line_max_chars = 80
start = 1
end = length(seq)
while (start <= end) {
print substr(seq, start, seq_line_max_chars) > outfile
start += seq_line_max_chars
}
}
BEGIN {
min_prot_len = 400
}
BEGINFILE {
tmp_out = FILENAME ".tmp"
}
/^>.+/ {
headerStartIdx = index($0, ">") + 1
header = trim(substr($0, headerStartIdx))
getline
sequence = ""
while ($0 !~ /(^>.+)|(^[[:space:]]*$)/) {
x_matched = match($0, "X")
if (x_matched != 0) {
next
}
gsub("*", "")
sequence = sequence $0
getline
}
if (length(sequence) >= min_prot_len) {
printFasta(header, sequence, tmp_out)
}
}
ENDFILE {
print "move called"
# system(("mv " tmp_out " " FILENAME))
}
I called the script with
$ ./filter_proteins.awk test.faa
When I run this, move called is printed and then it hangs. I tried stepping through with the debugger and I see that it reaches the ENDFILE block having processed all the lines in the input file, but when I type the next command, it jumps to the getline statement on line 44. After several iterations of next and print $0 it seems that the program is stuck reading the last line of the input file till the end of time. Perhaps this is a bug?
I am using GAWK 5.1.0
Edit
A minimal input file.
https://github.com/CuriousTim/pastebin/blob/main/mb.34.faa
When I run the script with only a few sequences, it works, but when I use the whole file, it hangs. I wasn't sure how to make a minimal example without providing the whole file.
We'll know for sure after you provide sample input to test with but my money's on the call to getline within the loop reaching the end of the file and so triggering the ENDFILE condition to be true but you're still in the loop.
Look:
$ cat file
foo
bar
$ cat tst.awk
{
while (1) {
print "about to execute getline"
getline
print "just executed getline:", $0
if (++c == 5) {
exit
}
}
}
ENDFILE {
print "*** in ENDFILE ***"
}
$ awk -f tst.awk file
about to execute getline
just executed getline: bar
about to execute getline
*** in ENDFILE ***
just executed getline: bar
about to execute getline
just executed getline: bar
about to execute getline
just executed getline: bar
about to execute getline
just executed getline: bar
Calling getline in a loop is not how you want to write an awk script and what you have in your code is the wrong syntax to use when calling getline at any time - see http://awk.freeshell.org/AllAboutGetline.
I modified my script to remove the getline in case anyone finds it useful.
#! /bin/gawk -f
function printFasta(header, seq, seq_line_max_chars) {
print ">" header
seq_line_max_chars = 80
start = 1
end = length(seq)
while (start <= end) {
print substr(seq, start, seq_line_max_chars)
start += seq_line_max_chars
}
}
BEGIN {
FS = ">"
min_prot_len = 400
max_prot_len = 700
}
NF > 1 {
if (sequence &&
length(sequence) >= min_prot_len &&
length(sequence) <= max_prot_len) {
printFasta(header, sequence)
}
}
{
if (!header) {
next
} else {
x_in_seq = match($0, "X")
if (!x_in_seq) {
gsub("*", "")
sequence = sequence $0
} else {
header = ""
}
}
}
END {
if (header) {
printFasta(header, sequence)
}
}

How to print out the file name that is being passed as an argument to awk program

I use a for loop to iterate a list of files and at each iteraction I pass one file name to an awk script.
I would like to print, not only the error code but also the name of the file that generated the error.
Something like : print error";"THENAMEOF_THE_FILE >> file.txt;
The awk script would look like this:
awk ' function errorManager(error)
{
print error >> file.txt;
}
BEGIN {error1="ERROR CODE X"}
{if (NR==1)
if(length($0) != 10)
{
errorManager(error1)
}
}
END{print "STOP"}' $1
This is what you're looking for:
awk '
function errorManager(idx, errorArr, error) {
errorArr[1] = "ERROR CODE X"
error = (idx in errorArr ? errorArr[idx] : "Unspecified Error")
printf "Error %s[%d]: %s\n", FILENAME, FNR, error | "cat>&2"
}
NR==1 {
if ( length($0) != 10 ) {
errorManager(1)
}
}
END{ print "STOP" }
' "$1" 2>"file.txt"

Sum up from line "A" to line "B" from a big file using awk

aNumber|bNumber|startDate|timeZone|duration|currencyType|cost|
22677512549|778|2014-07-02 10:16:35.000|NULL|NULL|localCurrency|0.00|
22675557361|76457227|2014-07-02 10:16:38.000|NULL|NULL|localCurrency|10.00|
22677521277|778|2014-07-02 10:16:42.000|NULL|NULL|localCurrency|0.00|
22676099496|77250331|2014-07-02 10:16:42.000|NULL|NULL|localCurrency|1.00|
22667222160|22667262389|2014-07-02 10:16:43.000|NULL|NULL|localCurrency|10.00|
22665799922|70110055|2014-07-02 10:16:45.000|NULL|NULL|localCurrency|20.00|
22676239633|433|2014-07-02 10:16:48.000|NULL|NULL|localCurrency|0.00|
22677277255|76919167|2014-07-02 10:16:51.000|NULL|NULL|localCurrency|1.00|
This is the input (sample of million of line) i have in csv file.
I want to sum up duration based on date.
My concern is i want to sum up first 1000000 lines
the awk program i'm using is:
test.awk
BEGIN { FS = "|" }
NR>1 && NR<=1000000
FNR == 1{ next }
{
sub(/ .*/,"",$3)
key=sprintf("%10s",$3)
duration[key] += $5 } END {
printf "%-10s %16s,"dAccused","Duration"
for (i in duration) {
printf "%-4s %16.2f i,duration[i]
}}
i run my script as
$awk -f test.awk 'file'
The input i have doesn't condsidered my condition NR>1 && NR<=1000000
ANY SUGGESTION? PLEASE!
You're looking for this:
BEGIN { FS = "|" }
1 < NR && NR <= 1000000 {
sub(/ .*/, "", $3)
key = sprintf("%10s",$3)
duration[key] += $5
}
END {
printf "%-10s %16s\n", "dAccused", "Duration"
for (i in duration) {
printf "%-4s %16.2f i,duration[i]
}
}
A lot of errors become obvious with proper indentation.
The reason you saw 1,000,000 lines was due to this:
NR>1 && NR<=1000000
That is a condition with no action block. The default action is to print the current record if the condition is true. That's why you see a lot of awk one-liners end with the number 1
You didn't post any expected output and your duration field is always NULL so it's still not clear what you really want output, but this is probably the right approach:
$ cat tst.awk
BEGIN { FS = "|" }
NR==1 { for (i=1;i<NF;i++) f[$i] = i; next }
{
sub(/ .*/,"",$(f["startDate"]))
sum[$(f["startDate"])] += $(f["duration"])
}
NR==1000000 { exit }
END { for (date in sum) print date, sum[date] }
$ awk -f tst.awk file
2014-07-02 0
Instead of discarding your header line, it uses it to create an array f[] that maps the field names to their order in each line so instead of having to hard-code that duration is field 4 (or whatever) you just reference it as $(f["duration"]).
Any time your input file has a header line, don't discard it - use it so your script is not coupled to the order of fields in your input file.

Parsing errors in awk blocks

awk 'BEGIN
{
INPUTFILE ='XXX'; iterator =0;
requestIterator =0;
storageFlag =T;
printFlag =F;
currentIteration =F;
recordCount =1;
while (getline < "'"$INPUTFILE"'")
{
requestArray[requestIterator]++;
requestIterator++;
}
}
if ($1 ~ /RequestId/)
{
FS = "=";
if($2 in requestArray)
{
storage[iterator] =$0;
printFlag =T;
next
}
else
{
storageFlag =F;
next
}
}
else
{
if((storageFlag =='T' && $0 != "EOE"))
{
storage[iterator]=$0; iterator++;
}
else {if(storageFlag == 'F')
{
next
}
else
{
if(printFlag == 'T')
{
for(details in storage)
{
print storage[details] >> FILE1;
delete storage[details];
}
printFlag =F;
storageFlag =T;
next
}
}'
I am facing some syntax error in the above code. Could you ppl please help me?
awk: BEGIN{INPUTFILE =XXXX;iterator =0;requestIterator =0;storageFlag =T;printFlag =F;currentIteration =F;recordCount =1;while (getline < ""){requestArray[requestIterator]++;requestIterator++;}}if ($1 ~ /RequestId/){FS = "=";if($2 in requestArray){storage[iterator] =$0;printFlag =T;next}else{storageFlag =F;next}}else{if((storageFlag ==T && $0 != EOE)){storage[iterator]=$0;iterator++;}else{if(storageFlag == F){next}else{if(printFlag == T){for(details in storage){print storage[details] >> XXXX;delete storage[details];}printFlag = F;storageFlag =T;next}}}}
awk: ^ syntax error
awk: ^ syntax error
Quotes are the problem. The first single quotes on INPUTFILE ='XXX' is going to be parsed as matching the one before BEGIN, and from then on all the parsing is broken.
Either escape the quotes or just put the awk file into a seperate file rather than "inline".
# STARTING POINT - known bad
awk 'BEGIN { INPUTFILE ='XXX'; iterator =0; ... '
Has to be rewritten to remove all of the single quotes inside the outer pair
awk 'BEGIN { INPUTFILE ="XXX"; iterator =0; ... '
Or depending on if you need doubles or singles, use doubles outside and single inside
awk "BEGIN { INPUTFILE ='XXX'; iterator =0; ... '
or escape the singles quotes so they make it through to awk and don't get consumed by the shell.
awk 'BEGIN { INPUTFILE =\'XXX\'; iterator =0; ... '
All of your problems go away if you put the awk script into a separate file rather than inlining it the shell. You can have whatever quotes you like and no one will care !!