Word Occurrences Script - awk

Question

I'm putting together a script that will the count the occurrences of words in text documents.
{
$0 = tolower($0)
for ( i = 1; i <= NF; i++ )
freq[$i]++
}
BEGIN { printf "%-20s %-6s\n", "Word", "Count"}
END {
sort = "sort -k 2nr"
for (word in freq)
printf "%-20s %-6s\n", word, freq[word] | sort
close(sort)
}
It works fine so far, but I'd like to make a couple tweaks/additions:
I'm having a hard time displaying the array index number, tried freq[$i] which just spit 0's back at me
Is there any way to eliminate the whitespace (spaces) from the word count?

Answer 1

You do not need to code an own loop to scan the fields, just set RS to make each word an own record: E.g. RS=[^A-Za-z] will treat every string not completely being built from uppercase and lowercase letters as record separator.
$ echo 'Hello world! I am happy123...' | awk 'BEGIN{RS="[^A-Za-z]+"}$0'
Hello
world
I
am
happy
The single $0 matches nonempty lines.
Maybe you want to allow digits in words.. just adapt RS to your needs.
So what's left?
Transform to lowercase, count, print sorted results.
File wfreq.awk:
BEGIN { RS="[^A-Za-z]+" }
$0 { counts[tolower($0)]++ }
END{
printf "%-20s %6s\n", "Word", "Count"
sort = "sort -k 2nr"
for(word in counts)
printf "%-20s %6s\n",word,counts[word] | sort
close(sort)
}
Example run (only top 10 lines of output for not spamming the answer):
$ awk -f wfreq.awk /etc/motd | head
Word Count
the 5
debian 3
linux 3
are 2
bpo 2
gnu 2
in 2
with 2
absolutely 1
But now for something not really completely different...
To sort by a different field, just adapt the sort = "sort ..." options.
I don't use asort() because not every awk has this extension.
File wfreq2.awk:
BEGIN { RS="[^A-Za-z]+" }
$0 { counts[tolower($0)]++ }
END{
printf "%-20s %6s\n", "Word", "Count"
sort = "sort -k 1"
for(word in counts)
printf "%-20s %6s\n",word,counts[word] | sort
close(sort)
}
Example run (only top 10 lines of output for not spamming the answer):
$ awk -f wfreq2.awk /etc/motd | head
Word Count
absolutely 1
amd 1
applicable 1
are 2
bpo 2
by 1
comes 1
copyright 1
darkstar 1