I'm trying to execute an awk command in my go program (the awk command pulls zip codes for a specified city, San Francisco in this case, from a tab delimited file of California zip codes):
cmd := exec.Command(
"awk",
"-F",
"'\\t'",
"'{if ($4 == \"SAN FRANCISCO\") print $0; }'",
"zipcodes_ca.txt",
)
fmt.Println(cmd.Args)
var out bytes.Buffer
var stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
fmt.Println(fmt.Sprint(err) + ": " + stderr.String())
return
}
This outputs:
[awk -F '\t' '{if ($4 == "SAN FRANCISCO") print $0; }' zipcodes_ca.txt]
exit status 2: awk: syntax error at source line 1
context is
>>> ' <<<
awk: bailing out at source line 1
If I take the printed args from the command and just run that as a command awk -F '\t' '{if ($4 == "SAN FRANCISCO") print $0; }' zipcodes_ca.txt it works. But, running it through my go program seems to be having issues. I'm not sure what I'm doing wrong here. I'm guessing I'm escaping things incorrectly, but nothing I try seems to work.
I don't think that you need single quotes around arguments. They are an artifact of using shell that prevents shell from interpreting argument content. Try without them.
Related
How can I trap an error if a dynamic regular expression evaluation is bad like:
var='lazy dog'
# a fixed Regex here, but original is coming from ouside the script
Regex='*.'
#try and failed
if (var ~ Regex) foo
The goal is to manage this error as I cannot test the regex itself (it comes from external source). Using POSIX awk (AIX)
Something like this?
$ echo 'foo' |
awk -v re='*.' '
BEGIN {
cmd="awk --posix \047/" re "/\047 2>&1"
cmd | getline rslt
print "rslt="rslt
close(cmd)
}
{ print "got " $0 " but re was bad" }
'
rslt=awk: cmd. line:1: error: Invalid preceding regular expression: /*./
got foo but re was bad
I use gawk so I had to add --posix to make it not just accept that regexp as a literal * followed by any char. You'll probably have to change the awk command being called in cmd to behave sensibly for your needs with both valid and invalid regexps but you get the idea - to do something like an eval in awk you need to have awk call itself via system() or a pipe to getline. Massage to suit...
Oh, and I don't think you can get the exit status of cmd with the above syntax and you can't capture the output of a system() call within awk so you may need to test the re twice - first with system() to find out if it fails but redirecting it's output to /dev/null, and then on a failure run it again with getline to capture the error message.
Something like:
awk -v re='*.' '
BEGIN {
cmd="awk --posix \047/" re "/\047 2>&1"
if ( system(cmd " > /dev/null") ) {
close(cmd " > /dev/null")
cmd | getline rslt
print "rslt="rslt
close(cmd)
}
}
{ print "got " $0 " but re was bad" }
'
I am writing a report tool which processes the source files of some application and produce a report table with two columns, one containing the name of the file and the other containing the word TODO if the file contains a call to some deprecated function deprecated_function and DONE otherwise.
I used awk to prepare this report and my shell script looks like
report()
{
find . -type f -name '*.c' \
| xargs -n 1 awk -v deprecated="$1" '
BEGIN { status = "DONE" }
$0 ~ deprecated{ status = "TODO" }
END {
printf("%s|%s\n", FILENAME, status)
}'
}
report "deprecated_function"
The output of this script looks like
./plop-plop.c|DONE
./fizz-boum.c|TODO
This works well but I would like to rewrite the awk script so that it supports several input files instead of just one — so that I can remove the -n 1 argument to xargs. The only solutions I could figure out involve a lot of bookkeeping, because we need to track the changes of FILENAME and the END event to catch each end of file event.
awk -v deprecated="$1" '
BEGIN { status = "DONE" }
oldfilename && (oldfilename != FILENAME) {
printf("%s|%s\n", oldfilename, status);
status = DONE;
oldfilename = FILENAME;
}
$0 ~ deprecated{ status = "TODO" }
END {
printf("%s|%s\n", FILENAME, status)
}'
Maybe there is a cleaner and shorter way to handle this.
I am using FreeBSD's awk and am looking for solutions compatible with this tool.
This will work in any modern awk:
awk -v deprecated="$1" -v OFS='|' '
$0 ~ deprecated{ dep[FILENAME] }
END {
for (i=1;i<ARGC;i++)
print ARGV[i], (ARGV[i] in dep ? "TODO" : "DONE")
}
' file1 file2 ...
Any time you need to produce a report for all files and don't have GNU awk for ENDFILE, you MUST loop through ARGV[] in the END section (or loop through it in BEGIN and populate a different array for END section processing). Anything else will fail if you have empty files.
Your awk script could be something like this:
awk -v deprecated="$1" '
FNR==1 {if(file) print file "|" (f?"TODO":"DONE"); file=FILENAME; f=0}
$0 ~ deprecated {f=1}
END {print file "|" (f?"TODO":"DONE")}' file1.c file2.c # etc.
The logic is fairly similar to your program so hopefully it's all clear. FNR is the record number of the current file, which I'm using to detect the start of a new file. Admittedly there's some repetition in the END block but I don't think it's a big deal. You could always use a function if you wanted to.
Testing it out:
$ cat f1.c
int deprecated_function()
{
// some deprecated stuff
}
$ cat f2.c
int good_function()
{
// some good stuff
}
$ find -name "f?.c" -print0 | xargs -0 awk -v deprecated="deprecated" 'FNR==1 {if(file) print file "|" (f?"TODO":"DONE"); file=FILENAME; f=0} $0 ~ deprecated {f=1} END {print file "|" (f?"TODO":"DONE")}'
./f2.c|DONE
./f1.c|TODO
I have used -print0 and the -0 switch to xargs so that both programs with work file names separated by null bytes "\0" rather than spaces. This means that you won't run into problems with spaces in file names.
I am trying to append lines to some new files with awk in this way:
#!/usr/bin/awk -f
BEGIN {
FS = "[ \t|]"; }
{
print $5 "\t" $13 "\t" $14 >> "./bed/" $5 ".bed";
}
END {
}
New file is created with filename derived from a field of awk input file (5th field). I am unable to execute this script since it fails with
awk: ./blast2bed.awk:6: (FILENAME=blastout000 FNR=1) fatal: can't redirect to `./bed/AY517392.1.bed' (No such file or directory)
Any hints?
Thanks
The directory bed has to exist so create it first with mkdir bed either before you run your script or in the BEGIN block. You should also add brackets around the output file:
print $5"\t"$13"\t"$14 >> ("./bed/"$5".bed")
Notes: You don't need to end lines with ; if you have a single statement per line and the BEGIN and END blocks are optional.
I'm trying to run the command below, and its giving me the error. Thoughts on how to fix? I would rather have this be a one line command than a script.
grep "id\": \"http://room.event.assist.com/event/room/event/" failed_events.txt |
head -n1217 |
awk -F/ ' { print $7 } ' |
awk -F\" ' { print "url \= \"http\:\/\/room\.event\.assist\.com\/event\/room\/event\/'{ print $1 }'\?schema\=1\.3\.0\&form\=json\&pretty\=true\&token\=582EVTY78-03iBkTAf0JAhwOBx\&account\=room_event\"" } '
awk: non-terminated string url = "ht... at source line 1
context is
>>> <<<
awk: giving up
source line number 2
The line below exports out a single column of ID's:
grep "id\": \"http://room.event.assist.com/event/room/event/" failed_events.txt |
head -n1217 |
awk -F/ ' { print $7 } '
156512145
898545774
454658748
898432413
I'm looking to get the ID's above into a string like so:
" url = "string...'ID'string"
take a look what you have in last awk :
awk -F\"
' #single start here
{ print " #double starts for print, no ends
url \= \"http\:\/\/room\.event\.assist\.com\/event\/room\/event\/
' #single ends here???
{ print $1 }'..... #single again??? ...
(rest codes)
and you want to print exact {print } out? i don't think so. why you were nesting print ?
Most of the elements of your pipe can be expressed right inside awk.
I can't tell exactly what you want to do with the last awk script, but here are some points:
Your "grep" is really just looking for a string of text, not a
regexp.
You can save time and simplify things if you use awk's
index() function instead of a RE. Output formats are almost always
best handled using printf().
Since you haven't provided your input data, I can't test this code, so you'll need to adapt it if it doesn't work. But here goes:
awk -F/ '
BEGIN {
string="id\": \"http://room.event.assist.com/event/room/event/";
fmt="url = http://example.com/event/room/event/%s?schema=whatever\n";
}
count == 1217 { nextfile; }
index($0, string) {
split($7, a, "\"");
printf(fmt, a[0]);
count++;
}' failed_events.txt
If you like, you can use awk's -v option to pass in the string variable from a shell script calling this awk script. Or if this is a stand-alone awk script (using #! shebang), you could refer to command line options with ARGV.
I want to run the system command in an awk script and get its output stored in a variable. I've been trying to do this, but the command's output always goes to the shell and I'm not able to capture it. Any ideas on how this can be done?
Example:
$ date | awk --field-separator=! {$1 = system("strip $1"); /*more processing*/}
Should call the strip system command and instead of sending the output to the shell, should assign the output back to $1 for more processing. Rignt now, it's sending output to shell and assigning the command's retcode to $1.
Note: Coprocess is GNU awk specific.
Anyway another alternative is using getline
cmd = "strip "$1
while ( ( cmd | getline result ) > 0 ) {
print result
}
close(cmd)
Calling close(cmd) will prevent awk to throw this error after a number of calls :
fatal: cannot open pipe `…' (Too many open files)
To run a system command in awk you can either use system() or cmd | getline.
I prefer cmd | getline because it allows you to catch the value into a variable:
$ awk 'BEGIN {"date" | getline mydate; close("date"); print "returns", mydate}'
returns Thu Jul 28 10:16:55 CEST 2016
More generally, you can set the command into a variable:
awk 'BEGIN {
cmd = "date -j -f %s"
cmd | getline mydate
close(cmd)
}'
Note it is important to use close() to prevent getting a "makes too many open files" error if you have multiple results (thanks mateuscb for pointing this out in comments).
Using system(), the command output is printed automatically and the value you can catch is its return code:
$ awk 'BEGIN {d=system("date"); print "returns", d}'
Thu Jul 28 10:16:12 CEST 2016
returns 0
$ awk 'BEGIN {d=system("ls -l asdfasdfasd"); print "returns", d}'
ls: cannot access asdfasdfasd: No such file or directory
returns 2
Figured out.
We use awk's Two-way I/O
{
"strip $1" |& getline $1
}
passes $1 to strip and the getline takes output from strip back to $1
gawk '{dt=substr($4,2,11); gsub(/\//," ",dt); "date -d \""dt"\" +%s"|getline ts; print ts}'
You can use this when you need to process a grep output:
echo "some/path/exex.c:some text" | awk -F: '{ "basename "$1"" |& getline $1; print $1 " ==> " $2}'
option -F: tell awk to use : as field separator
"basename "$1"" execute shell command basename on first field
|& getline $1 reads output of previous shell command in substream
output:
exex.c ==> some text
I am using macOS's awk and I also needed exit status of the command. So I extended #ghostdog74's solution to get the exit status too:
Exit if non-zero exit status:
cmd = <your command goes here>
cmd = cmd" ; printf \"\n$?\""
last_res = ""
value = ""
while ( ( cmd | getline res ) > 0 ) {
if (value == "") {
value = last_res
} else {
value = value"\n"last_res
}
last_res = res
}
close(cmd)
# Now `res` has the exit status of the command
# and `value` has the complete output of command
if (res != 0) {
exit 1
} else {
print value
}
So basically I just changed cmd to print exit status of the command on a new line. After the execution of the above while loop, res would contain the exit status of the command and
value would contain the complete output of the command.
Honestly not a very neat way and I myself would like to know if there is some better way.