I need to scan a file with many different special charecters and values.
Given a set of special charecters - I need to provide the value next to it:
547 %$
236 \"
4523 &*
8876 (*
8756 "/
...
I am using an awk command with gsub in order to find the sequences as they are.
awk -v st="$match_string" 'BEGIN {gsub(/(\[|\]|\-|\$|\*|\:|\+|\"|\(|\))/,"\\\\&", st)} match($0,st) {print;exit}' file.txt
The command works great e.g.
> (*
>> 8876 (*
However I am having trouble using the command to locate the \" sequence
I am trying to add to the gsub different strings to represnt the sequence:
|\\|
|\\\\|
|\\\\"|
...
But the result is always:
> \"
>> 8756 "/
while the result I am looking for woould be:
> \"
>> 236 \"
It seems that the gsub does not work, and the \" is interpeted just as "
Any ideas?
follwoing is a short script to run -
- it should find the symbol attached to the value in first_num
- Next it should print the first value in the file attched to the symbol found
first_num=$1
echo "looking for : $first_num"
sym_to_check=$(awk -v s="$first_num" '$0~s {if ($0~s)print $2}' temp.txt)
echo "symbol - $sym_to_check"
first_val=$(awk -v s="$sym_to_check" 'BEGIN {gsub(/(\[|\]|\-|\$|\^|\*|\:|\+|\"|\(|\))/,"\\\\&",s)} $0~s {if ($0~s)print; if ($0~s)exit}' temp.txt)
echo "first val- $first_val"
suppose the txt file is:
547 %$
111 [*
222 ()
5655 (*
454 )"
35 #!
743 \"
657 #!
236 \"
4523 &*
8876 (*
456 \"
8756 "/
first run is good:
> bash temp1.sh 8876
looking for : 8876
symbol - (*
first val- 5655 (*
the script finds the first value attached to (*
but the next run is bad:
> bash temp1.sh 236
looking for : 236
symbol - \"
first val- 454 )"
the symbol is correct - looking for \" but when searching for the first value attached to it, it looks for the first symbol with "
This gives the value 454 )" instead of the desired 743 \"
The way you're initializing the awk variable st using -v st="$match_string" is by design expanding escape sequences (so \t in "$match_string" would become a literal tab char in st, for example) and you're using a regexp operator, match(), but trying to escape the regexp metachars to make it act like it's doing string instead of regexp matching and then you're doing partial matching on the whole line (e.g. $0~85 would match 1853) instead of full matching on a specific field ($1==85).
Here's how you init awk variables from the shell without interpreting escape sequences and then test for them as full-matching literal strings or numbers on a specific field rather than partial-matching regexps across the whole line:
$ match_string='\"'
$ st="$match_string" awk 'BEGIN{st=ENVIRON["st"]} $2==st{print; exit}' file
743 \"
$ awk 'BEGIN{st=ARGV[1]; ARGV[1]=""} $2==st{print; exit}' "$match_string" file
743 \"
$ awk 'BEGIN{st=ARGV[1]; ARGV[1]=""} $1==st{print; exit}' '743' file
743 \"
Not all awks support ENVIRON[] so the first approach won't work in all awks but the second will.
See How do I use shell variables in an awk script? for how to set awk variables from shell and when you want to do literal string comparisons, it's usually simpler to just use string operators like == and index() instead of using regexp operators like ~ or match() and trying to escape all the regexp metacharacters to make them act like they're strings.
If you ever DID want to escape all regexp metachars, though, then the syntax to do that would be:
gsub(/[^^]/,"[&]",st); gsub(/\^/,"\\^",st)
rather than what you have in the code in your question:
gsub(/(\[|\]|\-|\$|\*|\:|\+|\"|\(|\))/,"\\\\&", st)
See Is it possible to escape regex metacharacters reliably with sed for an explanation of why that is the correct syntax.
Related
I have a file file with content like:
stringa 8.0.1.2 stringx
stringb 12.01.0.0 stringx
I have to get a substring from field 2 (first two values with the dot).
I am currently doing cat file | awk '{print $2}' | awk -F. '{print $1"."$2}' and getting the expected output:
8.0
12.01
The query is how to do this with single awk?
I have tried with match() but not seeing an option for a back reference.
Any help would be appreciated.
You can do something like this.
$ awk '{ split($2,str,"."); print str[1]"."str[2] }' file
8.0
12.01
Also, keep in mind that your cat is not needed. Simply give the file directly to awk.
With GNU grep please try following command once.
grep -oP '^\S+\s+\K[[:digit:]]+\.[[:digit:]]+' Input_file
Explanation: Using GNU grep here. Using its -oP options to print matched part and enable PCRE with -P option here. In main program, matching from starting non-space characters followed by 1 or more spaces, then using \K option to forget that match. Then matching 1 or more digits occurrences followed by a dot; which is further followed by digits. If a match is found then it prints matched value.
I would use GNU AWK's split function as follow, let file.txt content be
stringa 8.0.1.2 stringx
stringb 12.01.0.0 stringx
then
awk '{split($2,arr,".");print arr[1]"."arr[2]}' file.txt
output
8.0
12.01
Explantion: split at . 2nd field and put elements into array arr.
(tested in gawk 4.2.1)
You could match digits . digits from the second column and print if there is a match:
awk 'match($2, /^[[:digit:]]+\.[[:digit:]]+/) {
print substr($2, RSTART, RLENGTH)
}
' file
Output
8.0
12.01
Also with GNU awk and gensub():
awk '{print gensub(/([[:digit:]]+[.][[:digit:]]+)(.*)/,"\\1","g",$2)}' file
8.0
12.01
gensub() provides the ability to specify components of a regexp in the replacement text using parentheses in the regexp to mark the components and then specifying \\n in the replacement text, where n is a digit from 1 to 9.
You should perhaps not use awk at all (or any other external program, for that matter) but rely on the field-splitting capabilities of the shell and some variable expansion. For instance:
# printf "%s\n%s\n" "stringa 8.0.1.2 stringx" \
"stringb 12.01.0.0 stringx" |\
while read first second third junk ; do
printf "=%s= =%s= =%s=\n" "$first" "$second" "$third"
done
=stringa= =8.0.1.2= =stringx=
=stringb= =12.01.0.0= =stringx=
As you can see the value is captured in the variable "$second" already and you just need to further isolate the parts you want to see - the first and second part separated by a dot. You can do that either with parameter expansion:
# variable="8.0.1.2"
# echo ${variable%.*.*}
8.0
or like this:
# variable="12.01.0.0"
# echo ${variable%${variable#*.*.}}
12.01
or you can use a further read-statement to separate the parts and then put them back together:
# variable="12.01.0.0"
# echo ${variable} | IFS=. read parta partb junk
# echo ${parta}.${partb}
12.01
So, putting all together:
# printf "%s\n%s\n" "stringa 8.0.1.2 stringx" \
"stringb 12.01.0.0 stringx" |\
while read first second third junk ; do
printf "%s\n" "$second" | IFS=. read parta partb junk
printf "%s.%s\n" "$parta" "$partb"
done
8.0
12.01
Consider the following input:
$ cat a
d:\
$ cat a.awk
{ sub("\\", "\\\\"); print $0 }
$ cat a_double.awk
{ sub("\\\\", "\\\\"); print $0 }
Now running cat a | awk -f a.awk gives
d:\
and running cat a | awk -f a_double.awk gives
d:\\
and I expect exactly the other way around. How should I interpret this?
$ awk -V
GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
Yes, its expected behavior of awk. When you run sub("\\", "\\\\") in your first script, in sub's inside "(double quotes) since we are NOT using / for matching pattern we need to escape first \(actual literal character) then for escaping we are using \ so we need to escape that also, hence it will become \\\\
\\ \\
| |
| |
first 2 chars are denoting escaping next 2 chars are denoting actual literal character \
Which is NOT happening your 1st case hence NO match so no substitution in it, in your 2nd awk script you are doing this(escaping part in regex matching section of sub) hence its matching \ perfectly.
Let's see this by example and try putting ... for checking purposes.
When Nothing happens: Since no match on
awk '{sub("\\", "....\\\\"); print $0}' Input_file
d:\
When pattern matching happens:
awk '{sub("\\\\", "...\\\\"); print $0}' Input_file
d:...\\
From man awk:
gsub(r, s [, t])
For each substring matching the regular expression r in the string t,
substitute the string s, and return the number of substitutions.
How could we could do perform actual escaping part(where we need to use only \ before character only once)? Do mention your regexp in /../ in first section of sub like and we need NOT to double escape \ here.
awk '{sub(/\\/,"&\\")} 1' Input_file
The first arg to *sub() is a regexp, not a string, so you should use regexp (/.../) rather than string ("...") delimiters. The former is a literal regexp which is used as-is while the latter defines a dynamic (or computed) regexp which forces awk to interpret the string twice, the first time to convert the string to a regexp and the second to use it as a regexp, hence double the backslashes needed for escaping. See https://www.gnu.org/software/gawk/manual/gawk.html#Computed-Regexps.
In the following we just need to escape the backslash once because we're using a literal, rather than dynamic, regexp:
$ cat a
d:\
$ awk '{sub(/\\/,"\\\\")}1' a
d:\\
Your first script would rightfully produce a syntax error in a more recent version of gawk (5.1.0) since "\\" in a dynamic regexp is equivalent to /\/ in a literal one and in that expression the backslash is escaping the final forward slash which means there is no final delimiter:
$ cat a.awk
{ sub("\\", "\\\\"); print $0 }
$ awk -f a.awk a
awk: a.awk:1: (FILENAME=a FNR=1) fatal: invalid regexp: Trailing backslash: /\/
I have the following shell command:
awk -F'\[|\]' '{print $2}'
What is this command doing? Split into fields using as delimiter [sometext]?
E.g.:
$ echo "this [line] passed to awk" | awk -F'\[|\]' '{print $2}'
line
Editor's note: Only Mawk, as used on Ubuntu by default, produces the output above.
The apparent intent is to treat literal [ and ] as field-separator characters, i.e., to split each input record into fields by each occurrence of [ and/or ], which, with the sample line, yields this as field 1 ($1), line as field 2 ($2), and passed to awk as the last field ($3).
This is achieved by a regex (regular expression) that uses alternation (|), either side of which defines a field separator (delimiter): \[ and \] in a regex are needed to represent literal [ and ], because, by default, [ and ] are so-called metacharacters (characters with special syntactical meaning).
Note that awk always interprets the value of the FS variable (-F option) as a regex.
However, the correct form is '\\[|\\]':
$ echo "this [line] passed to awk" | awk -F'\\[|\\]' '{print $2}'
line
That said, a more concise version that uses a character set ([...]) rather than alternation (|) is:
$ echo "this [line] passed to awk" | awk -F'[][]' '{print $2}'
line
Note the careful placement of ] before [ inside the enclosing [...] to make this work, and how the enclosing [...] now have special meaning: they enclose a set of characters, any of which matches.
As for why 2 \ instances are needed in '\\[|\\]':
Taken as a regex in isolation, \[|\] would work:
\[ matches literal [
\] matches literal ]
| is an alternation that matches one or the other.
However, Awk's string processing comes first:
It should, due to \ handling in a string, reduce \[|\] to [|] before interpretation as a regex.
Unfortunately, however, Mawk, the default Awk on Ubuntu, for instance, resorts to guesswork in this particular scenario.[1]
[|], interpreted as a regex, would then only match a single, literal |
Thus, the robust and portable way is to use \\ in a string literal when you mean to pass a single \ as part of a regex.
This quote from the relevant section of the GNU Awk manual sums it up well:
To get a backslash into a regular expression inside a string, you have to type two backslashes.
[1] Implementation differences:
Unfortunately, at least 1 major Awk implementation resorts to guesswork in the presence of a single \ before a regex metacharacter inside a string literal.
BSD/macOS Awk and GNU Awk act predictably and GNU Awk also issues a helpful warning when a singly \-prefixed regex metacharacter is found:
# GNU Awk: Predictable string-first processing + a helpful warning.
echo 'a[b]|c' | gawk -F'\[|\]' '{print $2}'
gawk: warning: escape sequence '\[' treated as plain '['
gawk: warning: escape sequence '\]' treated as plain ']'
c
# BSD/macOS Awk: Predictable string-first processing, no warning.
echo 'a[b]|c' | awk -F'\[|\]' '{print $2}'
c
# Mawk: *Guesses* that a *regex* was intended.
# The unambiguous form -F'\\[|\\]' works too, fortunately.
echo 'a[b]|c' | mawk -F'\[|\]' '{print $2}'
b
Optional reading: regex literals inside Awk scripts
Awk supports regex literals enclosed in /.../, the use of which bypasses the double-escaping problem.
However:
These literals (which are invariably constant) are only available inside an Awk script,
and, it seems, you can only use them as patterns or function arguments - you cannot store them in a variable.
Therefore, even though /\[|\]/ is in principle equivalent to "\\[|\\]", you can not use the following, because the regex literal cannot be assigned to (special) variable FS:
# !! DOES NOT WORK in any of the 3 major Awk implementations.
# Note that nothing is output, and no error/warning is displayed.
$ echo 'a[b]|c' | awk 'BEGIN { FS=/\[|\]/ } { print $2 }'
# Using a double-escaped *string* to house the regex again works as expected:
$ echo 'a[b]|c' | awk 'BEGIN { FS="\\[|\\]" } { print $2 }'
b
Consider a file with a list of strings,
string1
string2
...
abc\de
.
When using gawk's system command to execute a shell command
, in this case printing the strings,
cat file | gawk '{system("echo " $0)}'
the last string will be formatted to abcde. $0 denotes the whole record, here this is just the one string
Is this a limitation of gawk's system command, not being able to output the gawk variables unformatted?
Expanding on Mussé Redi's answer, observe that, in the following, the backslash does not print:
$ echo 'abc\de' | gawk '{system("echo " $0)}'
abcde
However, here, the backslash will print:
$ echo 'abc\de' | gawk '{system("echo \"" $0 "\"")}'
abc\de
The difference is that the latter command passes $0 to the shell with double-quotes around it. The double-quotes change how the shell processes the backslash.
The exact behavior will change from one shell to another.
To print while avoiding all the shell vagaries, a simple solution is:
$ echo 'abc\de' | gawk '{print $0}'
abc\de
In Bash we use double backslashes to denote an actual backslash. The function of a single backslash is escaping a character. Hence the system command is not formatting at all; Bash is.
The solution for this problem is writing a function in awk to preformat backslashes to double backslahses, afterwards passing it to the system command.
emphasized textI have some text like
CreateMainPageLink("410",$objUserData,$mnt[139]);
from which i want to extract the number 139 after the occurrence of mnt with gawk. I tried the following expression (within a pipe expression to be used on a result of a grep)
gawk '{FS="[\[\]]";print NF}'
to print the number of fields. If my field separators were [ and ] I expect to see the number 3 printed out (three fields; one before the opening rectangular bracket, one after, and the actual number I want to extract). What I get instead is one field, corresponding to the full line, and two warnings:
gawk: warning: escape sequence `\[' treated as plain `['
gawk: warning: escape sequence `\]' treated as plain `]'
I was following the example given here, but obviously there is some problem/error with my expression.
Using the following two expressions also do not work:
gawk '{FS="[]"}{print NF;}'
gawk: (FILENAME=- FNR=1) fatal: Unmatched [ or [^: /[]/
and
gawk '{FS="\[\]"}{print NF;}'
gawk: warning: escape sequence `\[' treated as plain `['
gawk: warning: escape sequence `\]' treated as plain `]'
gawk: (FILENAME=- FNR=1) fatal: Unmatched [ or [^: /[]/
gawk -F[][] '{ print $0" -> "$1"\t"$2; }'
$ gawk -F[][] '{ print $0" -> "$1"\t"$2; }'
titi[toto]tutu
titi[toto]tutu -> titi toto
1) You must set the FS before entering the main parsing loop. You could do:
awk 'BEGIN { FS="[\\[\\]]"; } { print $0" -> "$1"\t"$2; }'
Which executes the BEGIN clause before parsing the file.
I have to escape the [character twice: one because it is inside a quoted string. And another once because gawk mandate it inside a bracket expression.
I personnaly prefer to use the -F flag which is less verbose.
2) FS="[\[\]]" is wrong, because you are inside a quoted string, this escape the character inside the string. Awk will see: [[]] which is an invalid bracket expression.
3) FS="[]" is wrong because it is an empty bracket expression trying to match nothing
4) FS="\[\]" is wrong again because it is error 2) and 3) together :)
gawk manual says: The regular expressions in awk are a superset of the POSIX specification. This is why you can use either: [\\[\\]] or [][]. The later being the posix way.
To include a literal ']' in the list, make it the first character
See:
Posix Regex specification:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap09.html#tag_09_04
Posix awk specification:
http://pubs.opengroup.org/onlinepubs/009695399/utilities/awk.html
Gnu Awk manual:
http://www.gnu.org/software/gawk/manual/gawk.html#Bracket-Expressions
FS="[]" Here it looks for data inside the [] and there are none.
To use square brackets you need to write them like this [][]
This is also wrong gawk '{FS="[\[\]]";print NF}' you need FS as a variable outside expression.
Eks
echo 'CreateMainPageLink("410",$objUserData,$mnt[139]);' | awk -F[][] '{print $2}'
139
Or
awk '{print $2}' FS=[][]
Or
awk 'BEGIN {FS="[][]"} {print $2}'
All gives 139
Edit: gawk '{FS="[\[\]]";print NF}' Here you print number of fields NF and not value of it $NF. Anyway it will not help, since dividing your data with [] gives ); as last filed, use this awk '{print $(NF-1)}' FS=[][] to get second last filed.
Do you need awk? You can get the value via sed like this:
# echo 'CreateMainPageLink("410",$objUserData,$mnt[139]);' | sed -n 's:.*\[\([0-9]\+\)\].*:\1:p'
139