I'm a newbie to Bourne shell and want to do simple array simulation. This works:
COLORS='FF0000 0000FF 00FF00'
i=2
color=$(echo ${COLORS} | awk '{print $2}')
echo "color selected: $color"
What I want to do is to pass $i instead of the fixed $2 parameter in print (this will later be used in a loop). I spent hours figuring out the right combination of single and double quotes to do this, no luck.
The closest I got is
color=$("echo ${COLORS} | awk '{print "$"${i}}'")
The run result is:
+ COLORS=FF0000 0000FF 00FF00
+ i=2
+ echo FF0000 0000FF 00FF00 | awk '{print $2}'
./tempgraph.sh: ./tempgraph.sh: 37: echo FF0000 0000FF 00FF00 | awk '{print $2}': not found
+ color=
+ echo color selected:
color selected:
Any help is appreciated.
Don't waste your time trying to get the shell to expand the variable correctly in the awk command, just define a variable using -v:
echo $COLORS | awk -v col=2 '{ print $col }'
In terms of your i variable, this becomes:
i=1
echo $COLORS | awk -v col=$i '{ print $col }'
You can also get at your environment directly:
export COLORS='FF0000 0000FF 00FF00'
awk 'END {split(ENVIRON["COLORS"],colors);for(col in colors) { print "Color",col,"is",colors[col]}}' /dev/null
which gives the following output on this mac:
Color 2 is 0000FF
Color 3 is 00FF00
Color 1 is FF0000
I'd do it like this:
color=$(echo ${COLORS} | awk "{print \$$i}")
If you use '...', the content is not expanded. But you want the value of $i inserted in your script. So "..." is to be used, which does variable expanding. But you also want a $ in front of the number for AWK, so you've got to escape it (\$).
Variables assigned on invokation like -v foo=bar are available in the BEGIN where variable assigned with a simple baz=qux are not.
BEGIN { print foo, bar; }
{ print foo, bar; }
see the difference:
echo Don\'t Panic! | awk -f ./hello.awk -v foo=Hello bar=World
Hello
Hello World
Related
I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".
First, I tried with a shell script:
$ v=123test
$ echo $v
123test
$ echo "$v"
123test
Then tried awk:
$ awk 'BEGIN{print "'$v'"}'
$ 123test
$ awk 'BEGIN{print '"$v"'}'
$ 123
Why is the difference?
Lastly I tried this:
$ awk 'BEGIN{print " '$v' "}'
$ 123test
$ awk 'BEGIN{print ' "$v" '}'
awk: cmd. line:1: BEGIN{print
awk: cmd. line:1: ^ unexpected newline or end of string
I'm confused about this.
#Getting shell variables into awk
may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below. v1.5
Using -v (The best way, most portable)
Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)
variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two
This should be compatible with most awk, and the variable is available in the BEGIN block as well:
If you have multiple variables:
awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'
Warning. As Ed Morton writes, escape sequences will be interpreted so \t becomes a real tab and not \t if that is what you search for. Can be solved by using ENVIRON[] or access it via ARGV[]
PS If you have vertical bar or other regexp meta characters as separator like |?( etc, they must be double escaped. Example 3 vertical bars ||| becomes -F'\\|\\|\\|'. You can also use -F"[|][|][|]".
Example on getting data from a program/function inn to awk (here date is used)
awk -v time="$(date +"%F %H:%M" -d '-1 minute')" 'BEGIN {print time}'
Example of testing the contents of a shell variable as a regexp:
awk -v var="$variable" '$0 ~ var{print "found it"}'
Variable after code block
Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:
variable="line one\nline two"
echo "input data" | awk '{print var}' var="${variable}"
or
awk '{print var}' var="${variable}" file
Adding multiple variables:
awk '{print a,b,$0}' a="$var1" b="$var2" file
In this way we can also set different Field Separator FS for each file.
awk 'some code' FS=',' file1.txt FS=';' file2.ext
Variable after the code block will not work for the BEGIN block:
echo "input data" | awk 'BEGIN {print var}' var="${variable}"
Here-string
Variable can also be added to awk using a here-string from shells that support them (including Bash):
awk '{print $0}' <<< "$variable"
test
This is the same as:
printf '%s' "$variable" | awk '{print $0}'
P.S. this treats the variable as a file input.
ENVIRON input
As TrueY writes, you can use the ENVIRON to print Environment Variables.
Setting a variable before running AWK, you can print it out like this:
X=MyVar
awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'
MyVar /bin/bash
ARGV input
As Steven Penny writes, you can use ARGV to get the data into awk:
v="my data"
awk 'BEGIN {print ARGV[1]}' "$v"
my data
To get the data into the code itself, not just the BEGIN:
v="my data"
echo "test" | awk 'BEGIN{var=ARGV[1];ARGV[1]=""} {print var, $0}' "$v"
my data test
Variable within the code: USE WITH CAUTION
You can use a variable within the awk code, but it's messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.
This works by extracting the variable within the code, so it becomes a part of it.
If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.
variable="line one\nline two"
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
Here is an example of code injection:
variable='line one\nline two" ; for (i=1;i<=1000;++i) print i"'
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
1
2
3
.
.
1000
You can add lots of commands to awk this way. Even make it crash with non valid commands.
One valid use of this approach, though, is when you want to pass a symbol to awk to be applied to some input, e.g. a simple calculator:
$ calc() { awk -v x="$1" -v z="$3" 'BEGIN{ print x '"$2"' z }'; }
$ calc 2.7 '+' 3.4
6.1
$ calc 2.7 '*' 3.4
9.18
There is no way to do that using an awk variable populated with the value of a shell variable, you NEED the shell variable to expand to become part of the text of the awk script before awk interprets it. (see comment below by Ed M.)
Extra info:
Use of double quote
It's always good to double quote variable "$variable"
If not, multiple lines will be added as a long single line.
Example:
var="Line one
This is line two"
echo $var
Line one This is line two
echo "$var"
Line one
This is line two
Other errors you can get without double quote:
variable="line one\nline two"
awk -v var=$variable 'BEGIN {print var}'
awk: cmd. line:1: one\nline
awk: cmd. line:1: ^ backslash not last character on line
awk: cmd. line:1: one\nline
awk: cmd. line:1: ^ syntax error
And with single quote, it does not expand the value of the variable:
awk -v var='$variable' 'BEGIN {print var}'
$variable
More info about AWK and variables
Read this faq.
It seems that the good-old ENVIRON awk built-in hash is not mentioned at all. An example of its usage:
$ X=Solaris awk 'BEGIN{print ENVIRON["X"], ENVIRON["TERM"]}'
Solaris rxvt
You could pass in the command-line option -v with a variable name (v) and a value (=) of the environment variable ("${v}"):
% awk -vv="${v}" 'BEGIN { print v }'
123test
Or to make it clearer (with far fewer vs):
% environment_variable=123test
% awk -vawk_variable="${environment_variable}" 'BEGIN { print awk_variable }'
123test
You can utilize ARGV:
v=123test
awk 'BEGIN {print ARGV[1]}' "$v"
Note that if you are going to continue into the body, you will need to adjust
ARGC:
awk 'BEGIN {ARGC--} {print ARGV[2], $0}' file "$v"
I just changed #Jotne's answer for "for loop".
for i in `seq 11 20`; do host myserver-$i | awk -v i="$i" '{print "myserver-"i" " $4}'; done
I had to insert date at the beginning of the lines of a log file and it's done like below:
DATE=$(date +"%Y-%m-%d")
awk '{ print "'"$DATE"'", $0; }' /path_to_log_file/log_file.log
It can be redirect to another file to save
Pro Tip
It could come handy to create a function that handles this so you dont have to type everything every time. Using the selected solution we get...
awk_switch_columns() {
cat < /dev/stdin | awk -v a="$1" -v b="$2" " { t = \$a; \$a = \$b; \$b = t; print; } "
}
And use it as...
echo 'a b c d' | awk_switch_columns 2 4
Output:
a d c b
In the below have I 3 grep commands that I would like to replace with awk's grep. so I have tried
! /000000000000/;
! /000000000000/ $0;
! /000000000000/ $3;
where I don't get an error, but testing with both the script below and
$ echo 000000000000 | awk '{ ! /000000000000/; print }'
000000000000
it doesn't skip the lines as expected.
Question
Can anyone explain why my "not grep" doesn't work in awk?
grep -v '^#' $hosts | grep -E '[0-9A-F]{12}\b' | grep -v 000000000000 | awk '{
print "host "$5" {"
print " option host-name \""$5"\";"
gsub(/..\B/,"&:", $3)
print " hardware ethernet "$3";"
print " fixed-address "$1";"
print "}"
print ""
}' > /etc/dhcp/reservations.conf
Could you please try changing your code to:
echo 000000000000 | awk '!/000000000000/'
Problem in your attempt: $ echo 000000000000 | awk '{ ! /000000000000/; print }' Since you are checking condition ! /000000000000/ which is having ; after it so that condition works well and DO NOT print anything. But then you have print after it which is NOT COMING under that condition so it simply prints that line.
awk works on pattern{action} if you are putting semi colon in between it means that condition ends before it and statement after ; is all together a new statements for awk.
EDIT: Adding possible solution by seeing OP's attempt here, not tested at all since no samples are shown by OP. Also I am using --re-interval since my awk version is old you could remove in case you have new version of awk in your box.
awk --re-interval '!/^#/ && !/000000000000/ && /[0-9A-Fa-f]{12}/{
print "host "$5" {"
print " option host-name \""$5"\";"
gsub(/..\B/,"&:", $3)
print " hardware ethernet "$3";"
print " fixed-address "$1";"
print "}"
print ""
}' "$host" > /etc/dhcp/reservations.conf
Taking a look at your code:
$ echo 000000000000 | awk '
{
! /000000000000/ # on given input this evaluates to false
# but since its in action, affects nothing
print # this prints the record regardless of whatever happened above
}'
Adding a print may help you understand:
$ echo 000000000000 | awk '{ print ! /000000000000/; print }'
0
000000000000
Removing the !:
$ echo 000000000000 | awk '{ print /000000000000/; print }'
1
000000000000
This is all I can help you with since there is not enough information for more.
This is the simple awk command i am trying to write
grep "Inputs - " access.log | awk 'BEGIN { FS = "Inputs -" } ; { print $2 }'
i am trying to grep the file access.log for all the lines with "Input -" and trying to awk the part after the "Input -". This is giving the following error
awk: syntax error near line 1
awk: bailing out near line 1
I am confused what is the issue with this, this should work!!!!
I have also tried the following and it does not work
grep "Inputs - " L1Access.log | awk -F='Inputs' '{print $1}'
Here is a sample input text file
This is line number 1. I dont want this line to be part of grep output
This is line number 2. I want this line to be part of grep output. This has "Input -", I want to display only the part after "Input -" from this line using awk
your problem cannot be reproduced here:
kent$ cat f
foo - xxx
foo - yyy
foo - zzz
fooba
kent$ grep 'foo - ' f| awk 'BEGIN { FS = "foo -"};{print $2}'
xxx
yyy
zzz
There must be something wrong in your awk codes. Besides, if you want to do a grep and awk to extract the part after your Inputs - you can use grep to do it in single shot:
kent$ grep -Po 'foo - \K.*' f
xxx
yyy
zzz
Since you stated you want everything after the first instance "Inputs -", and since your grep is unnecessary:
nawk -F"Inputs -" 'BEGIN {OFS="Inputs -"} {line=""}; { for(i=2;i<=NF;i++) line=line OFS $i} {print line}' test
Your own answer will only print out the second element. In the event that you have more than one "Input -" you will be missing the remaining of the line. If you don't want the second (or third.. ) "Inputs -" in the output you could use:
nawk -F"Input -" '{ for(i=2;i<=NF;i++) print $i}' test
OK folks i see what my issue is. I am using solaris and in solaris the awk does not have capability for regex, meaning it does not support more than 1 charater in the field seperator. So i used nawk
Please refer to this post
Stackoverflow post
grep "Inputs - " L1Access.log | nawk 'BEGIN { FS = "Inputs -" } { print $2 }'
this worked.
You are not clear on what to get. Here is a sample file:
cat file
test Inputs - more data
Here is nothing to get
yes Inputs - This is what we need Inputs - but what about this?
You can then use awk to get data:
awk -F"Inputs - " 'NF>1 {print $2}' file
more data
This is what we need
or like this?
awk -F"Inputs - " 'NF>1 {print $NF}' file
more data
but what about this?
By setting separator to Inputs - and test for NF>1 it will only print lines with Inputs -
I'm trying to do something pretty simple but its appears more complicated than expected...
I've lines in a text file, separated by the comma and that I want to output to another file, without the first field.
Input:
echo file1,item, 12345678 | awk -F',' '{OFS = ";";$1=""; print $0}'
Output:
;item; 12345678
As you can see the spaces before 12345678 are kind of merged into one space only.
I also tried with the cut command:
echo file1,item, 12345678 | cut -d, -f2-
and I ended up with the same result.
Is there any workaround to handle this?
Actually my entire script is as follows:
cat myfile | while read l_line
do
l_line="'$l_line'"
v_OutputFile=$(echo $l_line | awk -F',' '{print $1}')
echo $(echo $l_line | cut -d, -f2-) >> ${v_OutputFile}
done
But stills in l_line all spaces but one are removed. I also created the quotes inside the file but same result.
it has nothing to do with awk. quote the string in your echo:
#with quotes
kent$ echo 'a,b, c'|awk -F, -v OFS=";" '{$1="";print $0}'
;b; c
#without quotes
kent$ echo a,b, c|awk -F, -v OFS=";" '{$1="";print $0}'
;b; c
The problem is with your invocation of the echo command you're using to feed awk the test data above. The shell is looking at this command:
echo file1,item, 12345678
and treating file1,item, and 12345678 as two separate parameters to echo. echo just prints all its parameters, separated by one space.
If you were to quote the whitespace, as follows:
echo 'file1,item, 12345678'
the shell would interpret this as a single parameter to feed to echo, so you'd get the expected result.
Update after edit to OP - having seen your full script, you could do this entirely in awk:
awk -F, '{ OFS = "," ; f = $1 ; sub("^[^,]*,","") ; print $0 >> f }' myfile
In this script , I want awk to print the variables $file, $f, $order and sum/NR (all in a single row)
#!/bin/bash
for file in pmb_mpi tau xhpl mpi_tile_io fftw ; do
for f in 2.54 1.60 800 ;do
if [ ${f} = 2.54 ]
then
for order in even odd ; do
# echo ${file}_${f}_${order}_v1.xls >> P-state-summary.xls
awk '{sum+=$2} END {print ${file}_${f}_${order}_v1.xls, sum/NR}' ${file}_${f}_${order}_v1.xls >> P-state-summary.xls
done
else
# echo ${file}_${f}_v1.xls >> P-state-summary.xls
awk '{sum+=$2} END {print ${file}_${f}_v1.xls , sum/NR}' ${file}_${f}_v1.xls >> P-state-summary.xls
fi
done
done
Colud anyone of you kindly help me with this ?
awk doesn't go out and get shell variables for you, you have to pass them in as awk variables:
pax> export x=XX
pax> export y=YY
pax> awk 'BEGIN{print x "_" y}'
_
pax> awk -vx=$x -v y=$y 'BEGIN{print x "_" y}'
XX_YY
There is another way of doing it by using double quotes instead of single quotes (so that bash substitutes the values before awk sees them), but then you have to start escaping $ symbols and all sorts of other things in your awk command:
pax> awk "BEGIN {print \"${x}_${y}\"}"
XX_YY
I prefer to use explicit variable creation.
By the way, there's another solution to your previous related question here which should work.
You can do this:
echo -n "${file}_${f}_${order}_v1.xls " >> P-state-summary.xls
# or printf "${file}_${f}_${order}_v1.xls " >> P-state-summary.xls
awk '{sum+=$2} END {print sum/NR}' "${file}_${f}_${order}_v1.xls" |
tee "${file}_${f}_avrg.xls" >> P-state-summary.xls
Using echo -n or printf without a "\n" will output the text without a newline so the output of the awk command will follow it on the same line. I added a space as a separator, but you could use anything.
Using tee will allow you to write your output to the individual files and the summary file using only one awk invocation per input (order) file.