Extracting lines from a file if a certain substring is present - awk

Question

I have a file that looks like:
Stef NY ID=1;CITY=NY
John SE ID=0;CITY=SE
Stef SE ID=2;CITY=SE
I want to extract only those lines where ID in third column is greater than 1 so the expected output becomes:
Stef SE ID=2;CITY=SE
The bash script I have take care of removing either ID=1 or ID=0 but I don't know how to do it together. This is what I have:
awk '$3 !~ /^ID=1;/' file.txt > output.txt
But this gives me an output:
John SE ID=0;CITY=SE
Stef SE ID=2;CITY=SE
How can I add ID=0 in my bash statement above? Insights will be appreciated.

Answer 1

It's a bit fragile, but you could try:
$ cat input
Stef NY ID=1;CITY=NY
John SE ID=0;CITY=SE
Stef SE ID=2;CITY=SE
$ awk '$2>1' FS='[=;]' input
Stef SE ID=2;CITY=SE
That is, split the line on the = and ; so that the number you're looking to compare is in field 2.

Answer 2

awk '$3 !~ /^ID=1;/' file.txt > output.txt
How it works
Your AWK command (anything between the quotes) works like a filter.
$3 !~ // filters by a condition on the 3rd field ($3). The condition is a not matching (!~) regular expression (between the slashes //).
^ID=1; is a regular expression matching all lines starting with (^) ID=1.
Adjust the regex
As Charles Duffy commented you could simply change the constant literal pattern ID=1 for a more flexible one like either of those:
ID=[01]; the ID can be any char inside character-set (the set inside square brackets []), so either 0 or 1
similar set defined as range: ID=[0-1]; (from 0 to 1)
or even distinct alternatives ID=(0|1); whereas alternatives are listed in a group (wrapped inside parentheses) separated by a pipe symbol (| often meaning logical-OR)
All above will match 2 cases.

Answer 3

The expression is a regular expression so you can use:
awk '$3 !~ /^ID=[01];/' file.txt > output.txt

Answer 4

Here is a way to do numerical comparison by stripping out all unwanted characters from last field:
awk '{val=$NF; gsub(/(^|.*;)ID=|;.*/, "", val)} val+0 > 1' file
Stef SE ID=2;CITY=SE
This will also work correctly for input like this:
Stef NY ID=1;CITY=NY
Stef NY ID=01;CITY=NY
John SE ID=0;CITY=SE
Stef SE ID=2;CITY=SE
Stef SE ID=04;CITY=SE

Answer 5

Another possibility with awk could be:
awk '$NF ~ /^ID=[[:digit:]]+/ {split($NF,a,/=|;/);if (a[1]=="ID" && a[2] > 1) print $0}' file
Stef SE ID=2;CITY=SE
Initial condition: only if last field begins by the sequence
of characteres of the regexp /^ID=[[:digit:]]+/
action: split the field with the separator = or ; and then check the condition (a[1]=="ID" && a[2] > 1) If true, print the current line.