Issue excluding records from output using awk [duplicate] - awk

Question

I have a file called domain which contains some domains. For example:
google.com
facebook.com
...
yahoo.com
And I have another file called site which contains some sites URLs and numbers. For example:
image.google.com 10
map.google.com 8
...
photo.facebook.com 22
game.facebook.com 15
..
Now I'm going to count the url number each domain has. For example: google.com has 10+8. So I wrote an awk script like this:
BEGIN{
while(getline dom < "./domain" > 0) {
domain[dom]=0;
}
for(dom in domain) {
while(getline < "./site" > 0) {
if($1 ~/$dom$) #if $1 end with $dom {
domain[dom]+=$2;
}
}
}
}
But the code if($1 ~/$dom$) doesn't run like I want. Because the variable $dom in the regular expression was explained literally. So, the first question is:
Is there any way to use variable $dom in a regular expression?
Then, as I'm new to writing script
Is there any better way to solve the problem I have?

Answer 1

awk can match against a variable if you don't use the // regex markers.
if ( $0 ~ regex ){ print $0; }
In this case, build up the required regex as a string
regex = dom"$"
Then match against the regex variable
if ( $1 ~ regex ) {
domain[dom]+=$2;
}

Answer 2

First of all, the variable is dom not $dom -- consider $ as an operator to extract the value of the column number stored in the variable dom
Secondly, awk will not interpolate what's between // -- that is just a string in there.
You want the match() function where the 2nd argument can be a string that is treated as the regular expression:
if (match($1, dom "$")) {...}
I would code a solution like:
awk '
FNR == NR {domain[$1] = 0; next}
{
for (dom in domain) {
if (match($1, dom "$")) {
domain[dom] += $2
break
}
}
}
END {for (dom in domain) {print dom, domain[dom]}}
' domain site

Answer 3

One way using an awk script:
BEGIN {
FS = "[. ]"
OFS = "."
}
FNR == NR {
domain[$1] = $0
next
}
FNR < NR {
if ($2 in domain) {
for ( i = 2; i < NF; i++ ) {
if ($i != "") {
line = (line ? line OFS : "") $i
}
}
total[line] += $NF
line = ""
}
}
END {
for (i in total) {
printf "%s\t%s\n", i, total[i]
}
}
Run like:
awk -f script.awk domain.txt site.txt
Results:
facebook.com 37
google.com 18

Answer 4

You clearly want to read the site file once, not once per entry in domain. Fixing that, though, is trivial.
Equally, variables in awk (other than fields $0 .. $9, etc) are not prefixed with $. In particular, $dom is the field number identified by the variable dom (typically, that's going to be 0 since domain strings don't convert to any other number).
I think you need to find a way to get the domain from the data read from the site file. I'm not sure if you need to deal with sites with country domains such as bbc.co.uk as well as sites in the GTLDs (google.com etc). Assuming you are not dealing with country domains, you can use this:
BEGIN {
while (getline dom < "./domain" > 0) domain[dom] = 0
FS = "[ .]+"
while (getline < "./site" > 0)
{
topdom = $(NF-2) "." $(NF-1)
domain[topdom] += $NF
}
for (dom in domain) print dom " " domain[dom]
}
In the second while loop, there are NF fields; $NF contains the count, and $1 .. $(NF-1) contain components of the domain. So, topdom ends up containing the top domain name, which is then used to index into the array initialized in the first loop.
Given the data in the question (minus the lines of dots), the output is:
yahoo.com 0
facebook.com 37
google.com 18

Answer 5

The problem of the answers above is that you cannot use the "metacharacters" (e.g. \< for a word boundary at the beginning of a word) if you use a string instead of a regular expression /.../.
If you had a domain xyz.com and two sites ab.xyz.com and cd.prefix_xyz.com, the numbers of the two site entries would be added to xyz.com
Here's a solution using awk's pipe and the sed command:
...
for(dom in domain) {
while(getline < "./site" > 0) {
# let sed replaces occurence of the domain at the end of the site
cmd = "echo '" $1 "' | sed 's/\\<'" dom "'$/NO_VALID_DOM/'"
cmd | getline x
close(cmd)
if (match(x, "NO_VALID_DOM")) {
domain[dom]+=$2;
}
}
close("./site") # this misses in original code
}
...