I have a script that I need to run on every value. It basically return a number by taking an argument, like below
>>./myscript 4832
>>1100
my.csv contains the following:
123,4832
456,4833
789,4834
My command
cat my.csv | awk -F',' '{$3=system("../myscript $2");print $1,$2,$3'}
myscript is unable to understand that I'm passing the second input field $2 as argument. I need the output from the script to be added to the output as my 3rd column.
The expected output is
123,4832,1100
456,4833,17
789,4834,42
where the third field is the output from myscript with the second field as the argument.
If you are attempting to add a third field with the output from myscript $2 where $2 is the value of the second field, try
awk -F , '{ printf ("%s,%s,", $1, $2); system("../myscript " $2) }' my.csv
where we exploit the convenient fact that the output from myscript will complete the output without a newline with the calculated value and a newline.
This isn't really a good use of Awk; you might as well do
while IFS=, read -r first second; do
printf "%s,%s," "$first" "$second"
../mycript "$second"
done <my.csv
I'm assuming you require comma-separated output; changing this to space-separated is obviously a trivial modification.
The syntax you want is:
awk 'BEGIN{FS=OFS=","}
{
cmd = "./myscript \047" $2 "\047"
val = ( (cmd | getline line) > 0 ? line : "NaN" )
close(cmd)
print $0, val
}
' file
Tweak the getline part to do different error handling if you like and make sure you read and fully understand http://awk.freeshell.org/AllAboutGetline before using getline.
We can use in gnu-awk Two-Way Communications with Another Process
awk -F',' '{"../myscript "$2 |& getline v; print $1,$2,v}' my.csv
you get,
123 4832 1100
456 4833 17
789 4834 42
awk -F',' 'BEGIN { OFS=FS }{"../myscript "$2 |& getline v; print $1,$2,v}' my.csv
you get,
123,4832,1100
456,4833,17
789,4834,42
from GNU awk online documentation:
system: Execute the operating system command command and then return to the awk program. Return command’s exit status (see further on).
you need to use getline getline piped documentation
You need to specify the $2 separately in the string concatenation, that is
awk -F',' '{ system("echo \"echo " $1 "$(../myexecutable " $2 ") " $3 "\" | bash"); }' my.csv
Related
In the below awk when I echo f it is empty, but if I remove the $f I get the desired results, however the new formatting is not stored in the $d variable. Basically I am trying to convert the string in $d variable into a new formatted variable $f. Thank you :).
file
ID,1A
DATE,220102
awk
d=$(awk -F, '/Date/ {print $2}' file) | f=$(date -d "$d" +'%Y-%m-%d')
f --- desired ---
2022-01-02
You need to use it this way to return a value from awk and set a shell variable:
f=$(date -d "$(awk -F, '/DATE/ {print $2}' file)" +'%Y-%m-%d')
echo "$f"
2022-01-02
With awk:
awk 'BEGIN{FS=","; OFS="-"} $1=="DATE"{ print "20" substr($2,1,2), substr($2,3,2), substr($2,5,2) }' file
Output:
2022-01-02
See: 8 Powerful Awk Built-in Variables – FS, OFS, RS, ORS, NR, NF, FILENAME, FNR
With your shown samples please try following awk code. Written and tested in GNU awk. Using awk's match function capability to use regex ^DATE,([0-9]{2})([0-9]{2})([0-9]{2})$ for getting required output. This creates 3 capturing groups and stores matched values into array named arr once this match is done then printing 20 and all 3 values of arrays separated by - as per required output.
awk -v OFS="-" '
match($0,/^DATE,([0-9]{2})([0-9]{2})([0-9]{2})$/,arr){
print "20" arr[1],arr[2],arr[3]
}
' Input_file
While the other answers provide a more efficient method of reformatting the date (and assuming OP has no need for d in follow-on code), I'm going to focus solely on a couple issues with OP's current code:
in the awk script need to match for (all caps) DATE instead of Date
current code attempts to pipe the output from d=$(...) to the f=$(...) portion of code; while this does 'work' in that f will be assigned 2022-01-02 the problem is that the assignment to f is performed in a subprocess and upon exiting the subprocess f is effectively 'unassigned'; what OP really needs is to separate the d=$(...) and f=$(...) commands from each other so that both assignments occur in the current shell, and this can be done by replacing the pipe with a semicolon.
If we make these two simple edits:
# old code:
d=$(awk -F, '/Date/ {print $2}' file) | f=$(date -d "$d" +'%Y-%m-%d')
^^^^ ^^^
# new code:
d=$(awk -F, '/DATE/ {print $2}' file) ; f=$(date -d "$d" +'%Y-%m-%d')
^^^^ ^^^
OP's code will now generate the desired result:
$ echo "${f}"
2022-01-02
the string approaches :
{n,g}awk -F'^[^,]*,' 'gsub("^....|..", "-&", $(_=!(NF*=NF==NR)))\
($+_ = substr($+_,++_+_--))^_' OFS=20
mawk -F'^[^,]*,' '$(gsub("^....|..", "-&",
$!(NF*=NF==NR))*(_=!NF)) = substr($_,++_+_)' OFS=20
mawk2 'gsub("^....|..", "-&",
$!(NF*=NF==NR)) + sub(".",_)^_' FS='^.+,' OFS=20
the numeric approach :
mawk -F',' 'NF==NR && ($!NF = sprintf("20%.*s-%.*s-%0*.f", _+=_^=_<_,
__ = $NF, _++, substr(__,_), --_, __%(_+_*_*_)^_))'
2022-01-02
I have a file which has several sections with a header like this
$ head -n 5 test.txt
[44610] gmx#127.0.0.1
f1(cu_atomdata, NBParamGpu, Nbnxm::gpu_plist, bool), Block Size 64, Grid Size 3599, Device 0, 99 invocations
Section: Command line profiler metrics
Metric Name Metric Unit Minimum Maximum Average
-------------------------------------------------------------------------------------------- ----------- ------------ ------------ ------------
I would like to use the following awk command to get the number after Grid Size and the number before invocations. However, the following command returns nothing.
$ awk '{for (I=1;I<NF;I++) if ($I == "Grid Size") print $(I+1)}' test.txt
$
$ awk '{for (I=1;I<NF;I++) if ($I == "invocations") print $(I-1)}' test.txt
$
Any idea to fix that?
You may use this awk that loops through each field and extract your numbers based on field values:
awk '{
for (i=3;i<NF;i++)
if ($(i-2) == "Grid" && $(i-1) == "Size")
print "gridSize:", $i+0
else if ($(i+1) == "invocations")
print "invocations:", $i+0
}' file
gridSize: 3599
invocations: 99
Alternatively, you may try this gnu grep with PCRE regex:
grep -oP 'Grid Size\h+\K\d+|\d+(?=\h+invocations)' file
3599
99
\K - match reset
(?=...) - Lookahead assertion
With GNU awk latest versions try putting array within match itself:
awk '
match($0,/Grid Size [0-9]+/, arr1){
print arr1[3]
match($0,/[0-9]+ invocations/, arr2)
print arr2[1]
}
' Input_file
With your shown samples could you please try following(when I tried above it didn't work with 4.1 awk version so adding this one as an alternative here).
awk '
match($0,/Grid Size [0-9]+/){
num=split(substr($0,RSTART,RLENGTH),arr1," ")
print arr1[num]
match($0,/[0-9]+ invocations/)
split(substr($0,RSTART,RLENGTH),arr2," ")
print arr2[1]
}
' Input_file
make it even simpler :
{mawk/mawk2/gawk} 'BEGIN {
FS = "(^.+Grid Size[ ]+|" \ # before
"[,][^,]+[,][ ]+|" \ # in-between
"[ ]+invocations.*$)"; # after
} NF == 4 { print "grid size \043 : " $2 ", invocations \043 : " $3 }'
This regex gobbles everything before, in between, and after it. Because the regex touches the 2 walls at the ends, fields $1 and $4 will also be created, but as empty ones, hence the NF==4 check.
The octal code \043 is the hash symbol # - just my own personal preference of not having comment delimiter inside my strings is its original form.
a gawk approach with gensub:
$ gawk '/Grid Size/{
s=gensub(/.*Grid\sSize\s([[:digit:]]+).*,\s([[:digit:]]+) invocations/, "gridSize: \\1\ninvocations: \\2","G"); print s
}' myFile
gridSize: 3599
invocations: 99
I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".
First, I tried with a shell script:
$ v=123test
$ echo $v
123test
$ echo "$v"
123test
Then tried awk:
$ awk 'BEGIN{print "'$v'"}'
$ 123test
$ awk 'BEGIN{print '"$v"'}'
$ 123
Why is the difference?
Lastly I tried this:
$ awk 'BEGIN{print " '$v' "}'
$ 123test
$ awk 'BEGIN{print ' "$v" '}'
awk: cmd. line:1: BEGIN{print
awk: cmd. line:1: ^ unexpected newline or end of string
I'm confused about this.
#Getting shell variables into awk
may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below. v1.5
Using -v (The best way, most portable)
Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)
variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two
This should be compatible with most awk, and the variable is available in the BEGIN block as well:
If you have multiple variables:
awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'
Warning. As Ed Morton writes, escape sequences will be interpreted so \t becomes a real tab and not \t if that is what you search for. Can be solved by using ENVIRON[] or access it via ARGV[]
PS If you have vertical bar or other regexp meta characters as separator like |?( etc, they must be double escaped. Example 3 vertical bars ||| becomes -F'\\|\\|\\|'. You can also use -F"[|][|][|]".
Example on getting data from a program/function inn to awk (here date is used)
awk -v time="$(date +"%F %H:%M" -d '-1 minute')" 'BEGIN {print time}'
Example of testing the contents of a shell variable as a regexp:
awk -v var="$variable" '$0 ~ var{print "found it"}'
Variable after code block
Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:
variable="line one\nline two"
echo "input data" | awk '{print var}' var="${variable}"
or
awk '{print var}' var="${variable}" file
Adding multiple variables:
awk '{print a,b,$0}' a="$var1" b="$var2" file
In this way we can also set different Field Separator FS for each file.
awk 'some code' FS=',' file1.txt FS=';' file2.ext
Variable after the code block will not work for the BEGIN block:
echo "input data" | awk 'BEGIN {print var}' var="${variable}"
Here-string
Variable can also be added to awk using a here-string from shells that support them (including Bash):
awk '{print $0}' <<< "$variable"
test
This is the same as:
printf '%s' "$variable" | awk '{print $0}'
P.S. this treats the variable as a file input.
ENVIRON input
As TrueY writes, you can use the ENVIRON to print Environment Variables.
Setting a variable before running AWK, you can print it out like this:
X=MyVar
awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'
MyVar /bin/bash
ARGV input
As Steven Penny writes, you can use ARGV to get the data into awk:
v="my data"
awk 'BEGIN {print ARGV[1]}' "$v"
my data
To get the data into the code itself, not just the BEGIN:
v="my data"
echo "test" | awk 'BEGIN{var=ARGV[1];ARGV[1]=""} {print var, $0}' "$v"
my data test
Variable within the code: USE WITH CAUTION
You can use a variable within the awk code, but it's messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.
This works by extracting the variable within the code, so it becomes a part of it.
If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.
variable="line one\nline two"
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
Here is an example of code injection:
variable='line one\nline two" ; for (i=1;i<=1000;++i) print i"'
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
1
2
3
.
.
1000
You can add lots of commands to awk this way. Even make it crash with non valid commands.
One valid use of this approach, though, is when you want to pass a symbol to awk to be applied to some input, e.g. a simple calculator:
$ calc() { awk -v x="$1" -v z="$3" 'BEGIN{ print x '"$2"' z }'; }
$ calc 2.7 '+' 3.4
6.1
$ calc 2.7 '*' 3.4
9.18
There is no way to do that using an awk variable populated with the value of a shell variable, you NEED the shell variable to expand to become part of the text of the awk script before awk interprets it. (see comment below by Ed M.)
Extra info:
Use of double quote
It's always good to double quote variable "$variable"
If not, multiple lines will be added as a long single line.
Example:
var="Line one
This is line two"
echo $var
Line one This is line two
echo "$var"
Line one
This is line two
Other errors you can get without double quote:
variable="line one\nline two"
awk -v var=$variable 'BEGIN {print var}'
awk: cmd. line:1: one\nline
awk: cmd. line:1: ^ backslash not last character on line
awk: cmd. line:1: one\nline
awk: cmd. line:1: ^ syntax error
And with single quote, it does not expand the value of the variable:
awk -v var='$variable' 'BEGIN {print var}'
$variable
More info about AWK and variables
Read this faq.
It seems that the good-old ENVIRON awk built-in hash is not mentioned at all. An example of its usage:
$ X=Solaris awk 'BEGIN{print ENVIRON["X"], ENVIRON["TERM"]}'
Solaris rxvt
You could pass in the command-line option -v with a variable name (v) and a value (=) of the environment variable ("${v}"):
% awk -vv="${v}" 'BEGIN { print v }'
123test
Or to make it clearer (with far fewer vs):
% environment_variable=123test
% awk -vawk_variable="${environment_variable}" 'BEGIN { print awk_variable }'
123test
You can utilize ARGV:
v=123test
awk 'BEGIN {print ARGV[1]}' "$v"
Note that if you are going to continue into the body, you will need to adjust
ARGC:
awk 'BEGIN {ARGC--} {print ARGV[2], $0}' file "$v"
I just changed #Jotne's answer for "for loop".
for i in `seq 11 20`; do host myserver-$i | awk -v i="$i" '{print "myserver-"i" " $4}'; done
I had to insert date at the beginning of the lines of a log file and it's done like below:
DATE=$(date +"%Y-%m-%d")
awk '{ print "'"$DATE"'", $0; }' /path_to_log_file/log_file.log
It can be redirect to another file to save
Pro Tip
It could come handy to create a function that handles this so you dont have to type everything every time. Using the selected solution we get...
awk_switch_columns() {
cat < /dev/stdin | awk -v a="$1" -v b="$2" " { t = \$a; \$a = \$b; \$b = t; print; } "
}
And use it as...
echo 'a b c d' | awk_switch_columns 2 4
Output:
a d c b
I have tried
awk '{print FILENAME}'
And the result was full path of the file.
I want to get only the file name, example: from "test/testing.test.txt" I just want to get "testing" without ".test.txt".
Use -F to delimit by the period and print the first string before that delimiter:
awk -F'.' '{ print $1 }'
Alternatively,
ls -l | awk '{ print $9 }' | awk -F"." '{ print $1 }'
will run through the whole folder👍
(there's a fancier way to do it, but that's easy).
Use the sub and/or split functions to extract the part of FILENAME you want.
This is the simple awk command i am trying to write
grep "Inputs - " access.log | awk 'BEGIN { FS = "Inputs -" } ; { print $2 }'
i am trying to grep the file access.log for all the lines with "Input -" and trying to awk the part after the "Input -". This is giving the following error
awk: syntax error near line 1
awk: bailing out near line 1
I am confused what is the issue with this, this should work!!!!
I have also tried the following and it does not work
grep "Inputs - " L1Access.log | awk -F='Inputs' '{print $1}'
Here is a sample input text file
This is line number 1. I dont want this line to be part of grep output
This is line number 2. I want this line to be part of grep output. This has "Input -", I want to display only the part after "Input -" from this line using awk
your problem cannot be reproduced here:
kent$ cat f
foo - xxx
foo - yyy
foo - zzz
fooba
kent$ grep 'foo - ' f| awk 'BEGIN { FS = "foo -"};{print $2}'
xxx
yyy
zzz
There must be something wrong in your awk codes. Besides, if you want to do a grep and awk to extract the part after your Inputs - you can use grep to do it in single shot:
kent$ grep -Po 'foo - \K.*' f
xxx
yyy
zzz
Since you stated you want everything after the first instance "Inputs -", and since your grep is unnecessary:
nawk -F"Inputs -" 'BEGIN {OFS="Inputs -"} {line=""}; { for(i=2;i<=NF;i++) line=line OFS $i} {print line}' test
Your own answer will only print out the second element. In the event that you have more than one "Input -" you will be missing the remaining of the line. If you don't want the second (or third.. ) "Inputs -" in the output you could use:
nawk -F"Input -" '{ for(i=2;i<=NF;i++) print $i}' test
OK folks i see what my issue is. I am using solaris and in solaris the awk does not have capability for regex, meaning it does not support more than 1 charater in the field seperator. So i used nawk
Please refer to this post
Stackoverflow post
grep "Inputs - " L1Access.log | nawk 'BEGIN { FS = "Inputs -" } { print $2 }'
this worked.
You are not clear on what to get. Here is a sample file:
cat file
test Inputs - more data
Here is nothing to get
yes Inputs - This is what we need Inputs - but what about this?
You can then use awk to get data:
awk -F"Inputs - " 'NF>1 {print $2}' file
more data
This is what we need
or like this?
awk -F"Inputs - " 'NF>1 {print $NF}' file
more data
but what about this?
By setting separator to Inputs - and test for NF>1 it will only print lines with Inputs -