Having awk act upon results from a system command - awk

Question

I have found some excellent help here about hoe to invoke external commands from within awk and store results in a variable. What I have not been able to find is how to have awk act upon the result as it would on an ordinary input text file.
I use awk to parse a small HTML file (the status page of a running Tahoe LAFS node) in order to find some IP addresses listed. On each IP address I run an nmap scan of a specific port to see if it is open (yes, this is to become an automated Tahoe LAFS grid monitor). Using an if statement I can pick out the line of the output from nmap that contains the status (open/filtered/closed) of the port as its second field (typically "8098/TCP open unknown"). I would like to strip the line of fields 1 and 3 and only keep field 2, however, $2 of course refers to the fields in the HTML file I use as input to my awk script. I tried a user defined function which just did return $2, but that also refers to the field in the input HTML file.
Is there a way to refer to fields in an internally created variable inside an awk script? Something like a nested awk command within an awk script?

Answer 1

Use the getline "function". It sets $0 to the entire record and $1 through $NF in the usual way:
$ awk '/test/ {
> while (("ping -c 2 google.com") | getline > 0) {
> printf("$1 = %s, $2 = %s\n", $1, $2);
> }
> }'
abc
test
$1 = PING, $2 = google.com
$1 = 64, $2 = bytes
$1 = 64, $2 = bytes
$1 = , $2 =
$1 = ---, $2 = google.com
$1 = 2, $2 = packets
$1 = round-trip, $2 = min/avg/max/stddev
xyz
$
Edit: added parentheses around (cmd | getline) (it works for me without them but I guess some awk variants require it?).
Edit 2: apparently the "parentheses around getline" thing comes from a quite different issue noted in the GNU awk manuals:
According to POSIX, ‘expression | getline’ is ambiguous if expression contains unparenthesized operators other than ‘$’—for example, ‘"echo " "date" | getline’ is ambiguous because the concatenation operator is not parenthesized. You should write it as ‘("echo " "date") | getline’ if you want your program to be portable to all awk implementations.
In this case, the expression before the pipe is a single string, so there is no ambiguity. I moved the parentheses to where they would be needed for a more complex expression.
Also, it's a good idea to call close() on the command after the while loop exits. If there is another line matching test, awk will assume the existing sub-command should be read further, unless it has been close()d. As the command match is via the string, it's even better, rather than parenthesizing the left hand side of the pipe-to-getline, to store it in a variable and use that variable as an argument to close. For example:
awk '/^test / {
cmd = sprintf("ping -c %d %s", $2, $3)
while (cmd | getline > 0) print
close(cmd)
}'
(a variant without the semicolons that some dislike :-) ), which, when fed:
test 1 google.com
produces:
PING google.com (74.125.225.161): 56 data bytes
64 bytes from 74.125.225.161: icmp_seq=0 ttl=56 time=22.898 ms
--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.898/22.898/22.898/0.000 ms
Addendum (poking around on the web, I discovered that this is less obvious than I thought it was): be aware that this kind of "bare" getline, because it replaces the "current line", causes any remaining pattern-and-action rules in the script to fire on the new line contents. E.g., after the above, $0 begins with round-trip min/av , so a subsequent rule of the form /^round/ would match, even though the input line that triggered the "ping" was test 1 google.com. If this is not the last rule, it's probably appropriate to add a next directive to it. (In a complicated script I'd put that in every getline-ing action, even the last one, in case the last rule is moved, or more are added.)

Answer 2

As the relevant part of my final awk script is too large to fit as a comment, I'll post as an answer. The stripInputRecord, getIpNumber and getPortNumber functions just pick out the useful parts from the HTML code.
/address/ {
ip = stripInputRecord( $0 );
ip = getIpNumber( ip );
port[na] = stripInputRecord( $0 );
port[na] = getPortNumber( port[na] );
if (!(ip~"N/A")) {
if (ip~/loopback/) {
ip="127.0.0.1";
port[na]=stdp;
}
cmd="nmap -PN -p "stdp" "ip
cmd2="nmap -PN -p " port[na] " " ip
while ((cmd | getline)==1) {
if ($0~stdp) {
stdportstatus[na] = $2
}
}
while ((cmd2 | getline)==1) {
if ($0~port[na]) {
otherportstatus[na] = $2
}
}
}
close(cmd)
close(cmd2)
if ($0~/N\/A/) {
stdportstatus[na] = "-";
otherportstatus[na] = "-";
}
na++;
}
Thank you all (especially torek!)