Selecting a field after a string using awk - awk
I'm very new to awk having just been introduced to it over the weekend.
I have a question that I'm hoping someone may be able to help me with.
How would one select a field that follows a specific string?
How would I expand this code to select more than one field following a specific string?
As an example, for any given line in my text file I have something like
2 of 10 19/4/2014 school name random text distance 800m more random text time 2:20:22 winner someonefast.
Some attributes are very consistent so I can easily extract these fields. For example 2, 10 and the date. However, there is often a lot of variable text before the next field that I wish to extract. Hence the question. Using awk can I extract the next field following a string? For example I'm interested in the fields following the /distance/ or /time/ string in combination with $1, $3, $4, $5.
Your help will be greatly appreciated.
Andy
Using awk you can select the field following a string. Here is an example:
echo '2 of 10 19/4/2014 school name random text distance 800m more random text time 2:20:22 winner someonefast.' |
awk '{
for(i=1; i<=NF; i++) {
if ( i ~ /^[1345]$/ ) {
extract = (extract ? extract FS $i : $i)
}
if ( $i ~ /distance|time/ ) {
extract = (extract ? extract FS $(i+1): $(i+1))
}
}
print extract
}'
2 10 19/4/2014 school 800m 2:20:22
What we are doing here is basically allowing awk to split on default delimiter. We create a for loop to iterate over all fields. NF stores number of fields for a given line. So we start from 1 and go all the way to the end.
In our first conditional block, we just inspect the field number. If it is 1 or 3 or 4 or 5, we create a variable called extract which concatenates the values of these fields separated by the field separator.
In our second conditional block, we check if the value of the field is either distance or time. If it is we again append to our variable but this time instead of the current value, we do $(i+1) which is basically the value of the next field or you can say value of a field that follows a specific string.
When you have name = value situations like you do here, it's best to create an array that maps the names to the values and then just print the values for the names you're interested in, e.g.:
$ awk '{for (i=1;i<=NF;i++) v[$i]=$(i+1); print $1, $3, $4, $5, v["distance"], v["time"]}' file
2 10 19/4/2014 school 800m 2:20:22
Basic:
awk '{
for (i = 6; i <= NF; ++i) {
if ($i == "distance") distance = $(i + 1)
if ($i == "time") time = $(i + 1)
}
print $1, $3, $4, $5, distance, time
}' file
Output:
2 10 19/4/2014 school 800m 2:20:22
But it's not enough to get all other significant texts which is still part of the school name after $5. You should add another condition.
The better solution is to have another delimiter besides spaces like tabs and use \t as FS.
Related
AWK script- Not showing data
I'm trying to create a variable to sum columns 26 to 30 and 32. SO far I have this code which prints me the hearder and the output format like I want but no data is being shown. #! /usr/bin/awk -f BEGIN { FS="," } NR>1 { TotalPositiveStats= ($26+$27+$28+$29+$30+$32) } {printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%.2f %,%s,%s,%.2f %,%s,%s,%.2f %,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s, %s\n", EndYear,Rk,G,Date,Years,Days,Age,Tm,Home,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc,TotalPositiveStats } NR==1 { print "EndYear,Rk,G,Date,Years,Days,Age,Tm,HOme,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc,TotalPositiveStats" }#header Input data: EndYear,Rk,G,Date,Years,Days,Age,Tm,Home,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc 1985,1,1,10/26/1984,21,252,21.6899384,CHI,1,WSB,1,16,1,40,5,16,0.313,0,0,,6,7,0.857,1,5,6,7,2,4,5,2,16,12.5 1985,2,2,10/27/1984,21,253,21.69267625,CHI,0,MIL,0,-2,1,34,8,13,0.615,0,0,,5,5,1,3,2,5,5,2,1,3,4,21,19.4 1985,3,3,10/29/1984,21,255,21.69815195,CHI,1,MIL,1,6,1,34,13,24,0.542,0,0,,11,13,0.846,2,2,4,5,6,2,3,4,37,32.9 1985,4,4,10/30/1984,21,256,21.7008898,CHI,0,KCK,1,5,1,36,8,21,0.381,0,0,,9,9,1,2,2,4,5,3,1,6,5,25,14.7 1985,5,5,11/1/1984,21,258,21.7063655,CHI,0,DEN,0,-16,1,33,7,15,0.467,0,0,,3,4,0.75,3,2,5,5,1,1,2,4,17,13.2 1985,6,6,11/7/1984,21,264,21.72279261,CHI,0,DET,1,4,1,27,9,19,0.474,0,0,,7,9,0.778,1,3,4,3,3,1,5,5,25,14.9 1985,7,7,11/8/1984,21,265,21.72553046,CHI,0,NYK,1,15,1,33,15,22,0.682,0,0,,3,4,0.75,4,4,8,5,3,2,5,2,33,29.3 Output expected: EndYear,Rk,G,Date,Years,Days,Age,Tm,Home,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc,TotalPositiveStats 1985,1,1,10/26/1984,21,252,21.6899384,CHI,1,WSB,1,16,1,40,5,16,0.313,0,0,,6,7,0.857,1,5,6,7,2,4,5,2,16,12.5,35 1985,2,2,10/27/1984,21,253,21.69267625,CHI,0,MIL,0,-2,1,34,8,13,0.615,0,0,,5,5,1,3,2,5,5,2,1,3,4,21,19.4,34 1985,3,3,10/29/1984,21,255,21.69815195,CHI,1,MIL,1,6,1,34,13,24,0.542,0,0,,11,13,0.846,2,2,4,5,6,2,3,4,37,32.9,54 1985,4,4,10/30/1984,21,256,21.7008898,CHI,0,KCK,1,5,1,36,8,21,0.381,0,0,,9,9,1,2,2,4,5,3,1,6,5,25,14.7,38 1985,5,5,11/1/1984,21,258,21.7063655,CHI,0,DEN,0,-16,1,33,7,15,0.467,0,0,,3,4,0.75,3,2,5,5,1,1,2,4,17,13.2,29 1985,6,6,11/7/1984,21,264,21.72279261,CHI,0,DET,1,4,1,27,9,19,0.474,0,0,,7,9,0.778,1,3,4,3,3,1,5,5,25,14.9,36 1985,7,7,11/8/1984,21,265,21.72553046,CHI,0,NYK,1,15,1,33,15,22,0.682,0,0,,3,4,0.75,4,4,8,5,3,2,5,2,33,29.3,51 This script will be called like gawk -f script.awk <filename>. Currently when calling this is the output (It seems to be calculating the variable but the rest of fields are empty)
awk is well suited to summing columns: awk 'NR>1{$(NF+1)=$26+$27+$28+$29+$30+$32}1' FS=, OFS=, input-file > tmp mv tmp input-file That doesn't add a field in the header line, so you might want something like: awk '{$(NF+1) = NR>1 ? ($26+$27+$28+$29+$30+$32) : "TotalPositiveStats"}1' FS=, OFS=,
An explanation on the issues with the current printf output is covered in the 2nd half of this answer (below). It appears OP's objective is to reformat three of the current fields while also adding a new field on the end of each line. (NOTE: certain aspects of OPs code are not reflected in the expected output so I'm not 100% sure what OP is looking to generate; regardless, OP should be able to tweak the provided code to generate the desired result) Using sprintf() to reformat the three fields we can rewrite OP's current code as: awk ' BEGIN { FS=OFS="," } NR==1 { print $0, "TotalPositiveStats"; next } { TotalPositiveStats = ($26+$27+$28+$29+$30+$32) $17 = sprintf("%.3f",$17) # FG_PCT if ($20 != "") $20 = sprintf("%.3f",$20) # 3P_PCT $23 = sprintf("%.3f",$23) # FT_PCT print $0, TotalPositiveStats } ' raw.dat NOTE: while OP's printf shows a format of %.2f % for the 3 fields of interest ($17, $20, $23), the expected output shows that the fields are not actually being reformatted (eg, $17 remains %.3f, $20 is an empty string, $23 remains %.2f); I've opted to leave $20 as blank otherwise reformat all 3 fields as %.3f; OP can modify the sprintf() calls as needed This generates: EndYear,Rk,G,Date,Years,Days,Age,Tm,Home,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc,TotalPositiveStats 1985,1,1,10/26/1984,21,252,21.6899384,CHI,1,WSB,1,16,1,40,5,16,0.313,0,0,,6,7,0.857,1,5,6,7,2,4,5,2,16,12.5,40 1985,2,2,10/27/1984,21,253,21.69267625,CHI,0,MIL,0,-2,1,34,8,13,0.615,0,0,,5,5,1.000,3,2,5,5,2,1,3,4,21,19.4,37 1985,3,3,10/29/1984,21,255,21.69815195,CHI,1,MIL,1,6,1,34,13,24,0.542,0,0,,11,13,0.846,2,2,4,5,6,2,3,4,37,32.9,57 1985,4,4,10/30/1984,21,256,21.7008898,CHI,0,KCK,1,5,1,36,8,21,0.381,0,0,,9,9,1.000,2,2,4,5,3,1,6,5,25,14.7,44 1985,5,5,11/1/1984,21,258,21.7063655,CHI,0,DEN,0,-16,1,33,7,15,0.467,0,0,,3,4,0.750,3,2,5,5,1,1,2,4,17,13.2,31 1985,6,6,11/7/1984,21,264,21.72279261,CHI,0,DET,1,4,1,27,9,19,0.474,0,0,,7,9,0.778,1,3,4,3,3,1,5,5,25,14.9,41 1985,7,7,11/8/1984,21,265,21.72553046,CHI,0,NYK,1,15,1,33,15,22,0.682,0,0,,3,4,0.750,4,4,8,5,3,2,5,2,33,29.3,56 NOTE: in OP's expected output it appears the last/new field (TotalPositiveStats) does not contain the value from $30 hence the mismatch between the expected results and this answer; again, OP can modify the assignment statement for TotalPositiveStats to include/exclude fields as needed Regarding the issues with the current printf ... {printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%.2f %,%s,%s,%.2f %,%s,%s,%.2f %,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s, %s\n", EndYear,Rk,G,Date,Years,Days,Age,Tm,Home,Opp,Win,Diff,GS,MP,FG,FGA,FG_PCT,3P,3PA,3P_PCT,FT,FTA,FT_PCT,ORB,DRB,TRB,AST,STL,BLK,TOV,PF,PTS,GmSc,TotalPositiveStats} ... is referencing (awk) variables that have not been defined (eg, EndYear, Rk, G). [NOTE: one exeception is the very last variable in the list - TotalPositiveStats - which has in fact been defined earlier in the script.] The default value for undefined variables is the empty string ("") or zero (0), depending on how the awk code is referencing the variable, eg: printf "%s", EndYear => EndYear is treated as a string and the printed result is an empty string; with an output field delimiter of a comma (,) this empty strings shows up as 2 commas next to each other (,,) printf "%.2f %", FG_PCT => FG_PCT is treated as a numeric (because of the %f format) and the printed result is 0.00 % Where it gets a little interesting is when the (undefined) variable name starts with a numeric (eg, 3P) in which case the P is ignored and the entire reference is treated as a number, eg: printf "%s", 3P => 3P is processed as 3 and the printed result is 3 This should explain the 5 static values (0.00 %, 3, 3, 3.00 % and 0.00 %) printed in all output lines as well as the 'missing' values between the rest of the commas (eg, ,,,,). Obviously the last value in the line is an actual number, ie, the value of the awk variable TotalPositiveStats.
Bash one liner for calculating the average of a specific row of numbers in bash
I just began learning bash. Trying to figure out how to convert a two-liner into a one liner using bash. The First Line of code... searches the first column of the input.txt for the word - KEYWORD. captures every number in this KEYWORD row from column2 until the last column. dumps all these numbers into the values.txt file placing each number on a new line. The second line of code calculates average value for all the numbers in the first column of values txt the prints out the this value. awk '{if($1=="KEYWORD") for(i=2;i<=NF;i++) print $i}' input.txt > values.txt awk 'BEGIN{sum=0; counter=0}{sum+=$1; counter+=1}END{print sum/counter}' values.txt How do I create a one liner from this?
Something like awk ' BEGIN { count = sum = 0 } $1 == "KEYWORD" { for (n = 2; n <= NF; n++) sum += $n count += NF - 1 } END { print sum/count }' input.txt Just keep track of the sum and total count of numbers in the first (and only) pass through the file and then average them at the end instead of printing a value for each matching line.
After reviewing this problem with several people and learning some new bash/awk shortcuts, the code below appears to be the shortest answer. awk '/KEYWORD/{for(n=3;n<=NF;n++)sum+=$n;print sum/(NF-2)}' input.txt This code searches the input file for the row containing "KEYWORD". Then sums up all the field from the 3rd column to the last column. Then prints out the average value of all those numbers. (i.e. the mean).
Bash: Finding average of entries from multiple columns after reading a CSV text file
I am trying read a CSV text file and find average of weekly hours (columns 3 through 7) spent by all user-ids (column 2) ending with an even number (2,4,6,...). The input sample is as below: Computer ID,User ID,M,T,W,T,F Computer1,User3,5,7,3,5,2 Computer2,User5,8,8,8,8,8 Computer3,User4,0,8,0,8,4 Computer4,User1,5,4,5,5,8 Computer5,User2,9,8,10,0,0 Computer6,User7,4,7,8,2,5 Computer7,User6,8,8,8,0,0 Computer8,User9,5,2,0,6,8 Computer9,User8,2,5,7,3,6 Computer10,User10,8,9,9,9,10 I have written the following script: awk -F, '$2~/[24680]$/{for(i=3;i<=7;i++){a+=$i};printf "%s\t%.2g\n",$2,a/5;a=0}' user-list.txt > superuser.txt The output of this script is: User4 4 User2 5.4 User6 4.8 User8 4.6 User10 9 However, I want to change the script to only print one average for all user-Ids ending with an even number. The desired output for this would be as below (which is technically the average of all hours for the IDs ending with even numbers): 5.56 Any help would be appreciated. TIA
Trying to fix OP's attempt here and adding logic to get average of averages at last of the file's reading. Written on mobile so couldn't test it should work in case I got the thought correct by OP's description. awk -F, ' $2~/[24680]$/{ count++ for(i=3;i<=7;i++){ sum+=$i } tot+=sum/5 sum=0 } END{ print "Average of averages is: " (count?tot/count:"NaN") } ' user-list.txt > superuser.txt
You may try: awk -F, '$2 ~ /[02468]$/ { for(i=3; i<=7; i++) { s += $i ++n } } END { if (n) printf "%.2f\n", s/n }' cust.csv 5.56
awk -F, 'NR == 1 { next } { match($2,/[[:digit:]]+/);num=substr($2,RSTART,RLENGTH);if(num%2==0) { av+=($3+$4+$5+$6+$7)/5 } } END { printf "%.2f\n",av/5}' user-list.txt Ignore the first header like. Pick the number out of the userid with awk's match function. Set the num variable to this number. Check to see if the number is even with num%2. If it is average, set the variable av to av plus the average. At the end, print the average to 2 decimal places.
Print the daily average, for all even numbered user IDs: #!/bin/sh awk -F , ' (NR>1) && ($2 ~ /[02468]$/) { hours += ($3 + $4 + $5 + $6 + $7) (users++) } END { print (hours/users/5) }' \ "$1" Usage example: $ script user-list 5.56 One way to get evenness or oddness of an integer is to use modulus (%), as in N % 2. For even values of N, this sum evaluates to zero, and for odd values, it evaluates to 1. However in this case, a string operation would be required to extract the number any way, so may as well just use a single string match, to get odd or even. Also, IMO, for 5 fields, which are not going to change (days of the week), it's more succinct to just add them directly, instead of a loop. (NR>1) skips the titles line too, in case there's a conflict. Finally, you can of of course swap /[02468]$/ for /[13579]$/ to get the same data, for odd numbered users.
In a CSV file, subtotal 2 columns based on a third one, using AWK in KSH
Disclaimers: 1) English is my second language, so please forgive any grammatical horrors you may find. I am pretty confident you will be able to understand what I need despite these. 2) I have found several examples in this site that address questions/problems similar to mine, though I was unfortunately not able to figure out the modifications that would need to be introduced to fit my needs. The "Problem": I have an CSV file that looks like this: c1,c2,c3,c4,c5,134.6,,c8,c9,SERVER1,c11 c1,c2,c3,c4,c5,0,,c8,c9,SERVER1,c11 c1,c2,c3,c4,c5,0.18,,c8,c9,SERVER2,c11 c1,c2,c3,c4,c5,0,,c8,c9,SERVER2,c11 c1,c2,c3,c4,c5,416.09,,c8,c9,SERVER3,c11 c1,c2,c3,c4,c5,0,,c8,c9,SERVER3,c11 c1,c2,c3,c4,c5,12.1,,c8,c9,SERVER3,c11 c1,c2,c3,c4,c5,480.64,,c8,c9,SERVER4,c11 c1,c2,c3,c4,c5,,83.65,c8,c9,SERVER5,c11 c1,c2,c3,c4,c5,,253.15,c8,c9,SERVER6,c11 c1,c2,c3,c4,c5,,18.84,c8,c9,SERVER7,c11 c1,c2,c3,c4,c5,,8.12,c8,c9,SERVER7,c11 c1,c2,c3,c4,c5,,22.45,c8,c9,SERVER7,c11 c1,c2,c3,c4,c5,,117.81,c8,c9,SERVER8,c11 c1,c2,c3,c4,c5,,96.34,c8,c9,SERVER9,c11 Complementary facts: 1) File has 11 columns. 2) The data in columns 1, 2, 3, 4, 5, 8, 9 and 11 is irrelevant in this case. In other words, I will only work with columns 6, 7 and 10. 3) Column 10 will be typically alphanumeric strings (server names), though it may contain also "-" and/or "_". 4) Columns 6 and 7 will have exclusively numbers, with up to two decimal places (A possible value is 0). Only one of the two will have data per line, never both. What I need as an output: - A single occurrence of every string in column 10 (as column 1), then the sum (subtotal) of it's values in column 6 (as column 2) and last, the sum (subtotal) of it's values in column 7 (as column 3). - If the total for a field is "0" the field must be left empty, but still must exist (it's respective comma has to be printed). - **Note** that the strings in column 10 will be already alphabetically sorted, so there is no need to do that part of the processing with AWK. Output sample, using the sample above as an input: SERVER1,134.6,, SERVER2,0.18,, SERVER3,428.19,, SERVER4,480.64,, SERVER5,,83.65 SERVER6,,253.15 SERVER7,,26.96 I've already found within these pages not one, but two AWK oneliners that PARTIALLY accomplish what it need: awk -F "," 'NR==1{last=$10; sum=0;}{if (last != $10) {print last "," sum; last=$10; sum=0;} sum += $6;}END{print last "," sum;}' inputfile awk -F, '{a[$10]+=$6;}END{for(i in a)print i","a[i];}' inputfile My "problems" in both cases are the same: - Subtotals of 0 are printed. - I can only handle the sum of one column at a time. Whenever I try to add the second one, I get either a syntax error or it does simply not print the third column at all. Thanks in advance for your support people! Regards, MartÃn
something like this? $ awk 'BEGIN{FS=OFS=","} {s6[$10]+=$6; s7[$10]+=$7} END{for(k in s6) print k,(s6[k]?s6[k]:""),(s7[k]?s7[k]:"")}' file | sort SERVER1,134.6, SERVER2,0.18, SERVER3,428.19, SERVER4,480.64, SERVER5,,83.65 SERVER6,,253.15 SERVER7,,49.41 SERVER8,,117.81 SERVER9,,96.34 note that your treatment of commas is not consistent, you're adding an extra one when the last field is zero (count the commas)
Your posted expected output doesn't seem to match your posted sample input so we're guessing but this might be what you're looking for: $ cat tst.awk BEGIN { FS=OFS="," } $10 != prev { if (NR > 1) { print prev, sum6, sum7 } sum6 = sum7 = "" prev = $10 } $6 { sum6 += $6 } $7 { sum7 += $7 } END { print prev, sum6, sum7 } $ awk -f tst.awk file SERVER1,134.6, SERVER2,0.18, SERVER3,428.19, SERVER4,480.64, SERVER5,,83.65 SERVER6,,253.15 SERVER7,,49.41 SERVER8,,117.81 SERVER9,,96.34
Awk: printing undetermined number of columns
I have a file that contains a number of fields separated by tab. I am trying to print all columns except the first one but want to print them all in only one column with AWK. The format of the file is col 1 col 2 ... col n There are at least 2 columns in one row. Sample 2012029754 901749095 2012028240 901744459 258789 2012024782 901735922 2012026032 901738573 257784 2012027260 901742004 2003062290 901738925 257813 257822 2012026806 901741040 2012024252 901733947 257493 2012024365 901733700 2012030848 901751693 260720 260956 264843 264844 So I want to tell awk to print column 2 to column n for n greater than 2 without printing blank lines when there is no info in column n of that row, all in one column like the following. 901749095 901744459 258789 901735922 901738573 257784 901742004 901738925 257813 257822 901741040 901733947 257493 901733700 901751693 260720 260956 264843 264844 This is the first time I am using awk, so bear with me. I wrote this from command line which works: awk '{i=2; while ($i ~ /[0-9]+/) { printf "%s\n", $i i++ } }' bth.data It is more of a seeking approval than asking a question whether it is the right way of doing something like this in AWK or is there a better/shorter way of doing it. Note that the actual input file could be millions of lines. Thanks
Is this what you want as output? awk '{for(i=2; i<=NF; i++) print $i}' bth.data gives 901749095 901744459 258789 901735922 901738573 257784 901742004 901738925 257813 257822 901741040 901733947 257493 901733700 901751693 260720 260956 264843 264844 NF is one of several pre-defined awk variables. It indicates the number of fields on a given input line. For instance, it is useful if you want to always print out the last field in a line print $NF. Or of course if you want to iterate through all or part of the fields on a given line to the end of the line.
Seems like awk is the wrong tool. I would do: cut -f 2- < bth.data | tr -s '\t' '\n' Note that with -s, this avoids printing blank lines as stated in the original problem.