This started as an obscure problem with RPM scriptlets occasionally failing on awk. I narrowed it down to the following: The scriptlets use a GNU extension: length(array) construct, not supported when running in the posix mode. OK so far. What I don't understand is how running awk under sudo changes the posix compliance behavior. Here is a simple awk script that should run in the GNU mode, and should fail in posix mode.
$ cat ./try
/bin/awk 'BEGIN{x[1]=foo;x[2]=bar;print length(x);}'
$ /bin/awk --version | grep Awk
GNU Awk 4.0.2
$ id
uid=0(root) gid=0(root) groups=0(root)
$ /bin/sh ./try
awk: cmd. line:1: fatal: length: received array argument
$ sudo /bin/sh ./try
2
$
What is the underlying mechanism that changes the awk behavior?
Awk (really gawk under linux) is being controlled by the POSIXLY_CORRECT environment variable, which was occasionally being inherited from the original user's environment. The installation in question must be run by root, but at times the admin would become root with "su" which keeps the environment, thus keeping his POSIXLY_CORRECT, forcing gawk into a posix mode, and failing the GNU length(array) extension. At other times the admin would run "sudo" or "su -" to become root, start with root's clean environment and successfully run the extended gawk functionality.
Related
I'm writing a shell script which shut down some services and trying to get its pid by using the following awk script.
However, this awk script can't get pid. What's wrong with that?
ps -ef | awk -v port_no=10080 '/[m]ilk.*port=port_no/{print $2}'
The result of ps -ef is like this:
username 13155 27705 0 16:06 pts/2 00:00:00 /home/username/.rbenv/versions/2.3.6/bin/ruby /home/username/.rbenv/versions/2.3.6/bin/milk web --no-browser --host=example.com --port=10080
This process is working with a different port argument as well, so I want to kill the process only working on port=10080.
The awk script below works fine, but when I specify the port no using awk -v like the above, it doesn't work well.
ps -ef | awk '/[m]ilk.*port=10080/{print $2}'
awk version: GNU Awk 4.0.2
The syntax for pattern matching with /../ does not work with variables in the regular expression. You need to use the ~ syntax for it.
awk -v port_no=10080 '$0 ~ "[m]ilk.*port="port_no{print $2}'
If you notice the regex carefully, the regex string on the r.h.s of ~ is under the double-quotes ".." except the variable name holding the port number which shouldn't be under quotes, for the expansion to happen.
This task is easily accomplished using pgrep:
$ pgrep -f '[m]ilk.*port=10080'
Have a look at man pgrep for details.
I'm trying to run a simple awk command using Cygwin.
I want to run through a file and put each line to a file named the same as the first field, appending lines with the same value in the first field to their relevant file.
I am running this command:
gawk -F , '{print>\$1}' ./filename.csv
The only output I get from command is: "Access is denied."
C:\projects>gawk -F , '{print>\$1}' ./filename.csv
Access is denied.
This is what it looks like in total.
I've looked at the file permissions and allowed everything but I can't seem to get it to work.
Edited with extra info as requested:
C:\projects>gawk -F , '{print > \$1}' filename.csv
Access is denied.
C:\projects>$SHELL --version
'$SHELL' is not recognized as an internal or external command,
operable program or batch file.
C:\projects>gawk --version
GNU Awk 4.1.4, API: 1.1 (GNU MPFR 3.1.5, GNU MP 6.1.2)
Copyright (C) 1989, 1991-2016 Free Software Foundation.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.
I've tried running from the Cygwin Shell and I get this error now:
/cygdrive/c/projects/FileSplit
$ gawk -F , '{print > $1}' ./filename.csv
gawk: cmd. line:1: (FILENAME=./filename.csv FNR=1) fatal: expression for `>' redirection has null string value
Running it in the Cygwin Shell worked although my first field was blank on some lines so the command I ended up with was:
gawk -F , '{if(length($1)==0)print > "empty"; else print > $1}' ./filename.csv
Get rid of the backslash:
gawk -F , '{print > $1}' ./filename.csv
Given what you've now added to your question - you're running gawk from the Windows CMD prompt, not from cygwin. Don't do that, run gawk from a cygwin terminal window instead by executing something like All Programs -> Cygwin -> Cygwin64 Terminal to start a cygwin window running a bash shell.
I have created a tcsh script with a series of gawk commands in the following form:
gawk -f InputFileName > OutputFileName
After the standard call-in (#!/bin/csh -f), I utilized the following command:
set a = $<
In the InputFileName, I proceed to use ${a}.txt, but it does not even use $a when looking up the input file.
They were initially running fine on Cygwin (Windows); now they are running on a Linux and are presenting problems.
It was a text file compatibility issue.
Since I did not have dos2unix on my computer, I used the following one line of code.
awk '{ sub("\r$", ""); print }' winfile.txt > unixfile.txt
I have the following command I want to execute in a Makefile but I'm not sure how.
The command is docker rmi -f $(docker images | grep "<none>" | awk "{print \$3}")
The command executed between $(..) should produce output which is fed to docker rmi but this is not working from within the Makefile I think that's because the $ is used specially in the Makefile but I'm not sure how to modify the command to fit in there.
Any ideas?
$ in Makefiles needs to be doubled to prevent substitution by make:
docker rmi -f $$(docker images | grep "<none>" | awk "{print \$$3}")
Also, it'd be simpler to use use a singly-quoted string in the awk command to prevent expansion of $3 by the shell:
docker rmi -f $$(docker images | grep "<none>" | awk '{print $$3}')
I really recommend the latter. It's usually better to have awk code in single quotes because it tends to contain a lot of $s, and all the backslashes hurt readability.
I am new to awk script. I am trying to figure out how to run an awk file without awk -f command. I see people keep saying add "#!bin/awk -f" for the first line of an awk file. But this didn't for my awk. It still gives me "no file or directory" error.
I question is what does "#!bin/awk -f" really mean, and what does it do?
Its #!/bin/awk -f not #!bin/awk. That will probably work, but theres no guaranty. If someone who has awk installed in a different location runs your script, it won't work. What you want is this: #!/usr/bin/env awk -f.
#! is what tells bash what to use to interpret your script. It should go at the very top of your file. It's called a Shebang. Right after that, you put the path to the interpreter.
/usr/bin/env finds where awk is located, and uses that script as the interpreter. So if they installed awk into somewhere else like /usr/local/bin then it'll find it. This probably won't matter for you, but it's a good habit to get into. It's more portable, and can be shared easier.
The -f says that awk is gonna read from a file. You could do awk -f yourfilename.awk in bash, but in the shebang, -f means the rest of the code will be the file it reads from.
I hope this helped. Feel free to ask me any questions if it doesn't work, or isn't clear enough.
UPDATE
If you get the error message:
/usr/bin/env: ‘awk -f’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
then change the first line of your script to #!/usr/bin/env -S awk -f (tested with GNU bash, version 4.4.23)
You probably want
#!/bin/awk -f
(The first slash after the #! is important).
This tells unix what program it should use to 'run' the script with.
It is usually called the 'shebang' which comes from hash + bang.
If you want to run your script like this you need to make sure it is executable (chmod +x <script>).
Otherwise you can just run your script by typing the command /bin/awk -f <script>
The Shebang for Awk Explained
#! is the start of a shebang line, which tells the shell which interpreter to use for the script.
/bin/awk is the path to your awk executable. You may need to change this is your awk is installed elsewhere, or if you want to use a different version of awk.
-f is a flag to awk to tell it to interpret the flag's argument as an awk script. In a shebang, it tells some awks to interpret the remainder of the script instead of a file.
Your Shebang is (Probably) Broken
You are using #!bin/awk -f which is unlikely to work, unless you have awk installed as $PWD/bin/awk. You probably meant to use #!/bin/awk instead.
In some instances, passing a flag on the shebang line may not work with your shell or your awk. If you have the rest of the shebang line correct, you might try removing the -f flag and see if that works for you.