How do I awk or grep greppable Nmap output for IP address, Host, Port Number, Port Status, Protocol, Service, and Service Version (if there is one)? - awk

The greppable Nmap output looks like the following:
Host: 9.2.1.100 (hello.world.com) Status: Up
Host: 9.2.1.100 (hello.world.com) Ports: 21/closed/tcp//ftp///, 22/closed/tcp//ssh///,
23/closed/tcp//telnet///, 25/closed/tcp//smtp///, 80/open/tcp//http//Citrix Metaframe ICA Browser/,
110/filtered/tcp//pop3///, 139/open/tcp//netbios-ssn//Microsoft Windows netbios-ssn/,
443/closed/tcp//https///, 445/open/tcp//microsoft-ds//Windows Server 2003 3790 Service Pack 2 microsoft-ds/,
3389/open/tcp//ms-wbt-server//Microsoft Terminal Service/ Seq Index: 256 IP ID Seq: Incremental
My question is how do I use awk or grep to parse the output so that I get the following:
IP Address, Host, Port Status (limited to only open ports), Protocol, Service, and Service Version (if there is one)?
9.2.1.100\thello.world.com\t80\topen\ttcp\thttp\tCitrix Metaframe ICA Browser\n
9.2.1.100\thello.world.com\t139\topen\ttcp\tnetbios-ssn\tMicrosoft Windows netbios-ssn\n
...

There are a million different ways to parse this output, and this is probably not the prettiest, but I do believe this should work:
grep 'Ports:' $1 | while read -r line
do
ip=$(echo $line | grep -oP '(?<=^Host: )(\d{1,3}\.){3}\d{1,3}')
name=$(echo $line | grep -Po "(?<=^Host: $ip \()[^\)]*")
echo "$line" | grep -Po '(?<=\tPorts: )[^\t]*\t' | sed -e 's/, /\n/g' | sed -e 's%/%\t%g' | cut -d/ -f 1-3,5,7 | sed -e "s/^/$ip\t$name\t/g"
done
in this specific example you have to save this code to a file and execute it with a the name of the file that contains the nmap output as an argument.
Anyway, here you can find the full explanation about the grepable output of the nmap command. They even explain how it should be parsed, but i did not have the time to read it so i just wrote here this little script.
I hope you will find this helpful.

Related

Running "ip | grep | awk" within a sed replacement

Problem Set (Raspberry Pi OS):
I have a file example.conf that contains a line IPv4addr=XXXXX. I am attempting to change this to the IP that is generated the in the command
ipTest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
I want to automate this file change during a script install.sh, the line I am attempting is:
IPtest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
sudo sed -e "/IPv4addr/s/[^=]*$/$IPtest/" example.conf
Returns error:
sed: -e expression #1, char 32: unknown option to `s'
A simple line in that code works, such as SimpleTest='Works'
Any thoughts? I am open to other solutions as well, however I am not an experienced linux user so I am using the tools I know to work with other problem sets.
$IPtest contains the / character; try something like this:
IPtest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
sudo sed -e '/IPv4addr/s#[^=]*$#'"$IPtest"'#' example.conf
You can shorten your variable and allow awk to do the job of grep at the same time
IPtest=$(ip --brief a s | awk '/eth0/{print $3}')
Using sed grouping and back referencing
sed -i.bak "s|\([^=]*.\).*|\1$IPtest|" example.conf

Force write to Xcode 'Debugger Output' in console?

The Xcode console has a 'Debugger output' filter. I understand this is for use with lldb, and that you can get messages to print to this output by using breakpoints. My question is not how to do that.
My question is: what is the underlying mechanism Xcode itself uses to write lldb messages to Debugger Output (not Target Output)? Is there a variable similar to stdout or stderr that writes here? Is it possible, from Xcode target code (Swift/Obj-C/C), to write to this output?
Looks like Xcode uses a tty to communicate with lldb, and you can interface with the Debugger Output using that:
echo "Wheeeeeeee" > $(lsof -p $(ps -A | grep -m1 MacOS/Xcode | awk '{print $1}') | grep -m2 dev/ttys | tail -1 | awk '{print $9}')
Breaking the above down:
$ ps -A | grep -m1 MacOS/Xcode | awk '{print $1}'
21280
This gives the process ID of Xcode (21280). Using this, we can find the files it has open:
$ lsof -p 21280 | grep /dev/ttys
Xcode 21280 tres 47u CHR 16,3 0t0 3569 /dev/ttys003
Xcode 21280 tres 58u CHR 16,5 0t0 3575 /dev/ttys005
The one with the highest number (/dev/ttys005 in this case) is the one we want, so let's extract it. tail -1 will give us the last line of output, and awk '{print $9}' will give us the 9th item on the line, which is what we want!
$ lsof -p 21280 | grep /dev/ttys | tail -1 | awk '{print $9}'
/dev/ttys005
Now we can use this to write whatever we want:

Ansible grep from shell variable

I am trying to create an Ansible playbook to pull out MTU size for exact NIC (unfortunately i have 5k VMs and this exact NIC does not have the same name on all VMs). I need to parse IP from file to variable and grep by that.
My command i will use in playbook:
/sbin/ifconfig -a | grep -C 1 $IP | grep MTU | awk '{print $5}' | cut -c 5-10
And output should be looking like this:
9000
This one gnu awk command should do:
ifconfig -a | awk -v ip="$IP" -v RS= -F'MTU:' '$0~ip {split($2,a," ");print a[1]}'
9216
Another variations
ifconfig -a | awk -v ip="$IP" 'f {split($6,a,":");print a[2];exit} $0~ip{f=1}'
ifconfig -a | awk -v ip="$IP" 'f {print substr($6,5,99);exit} $0~ip{f=1}'
9216

Extract unique IPs from live tcpdump capture

I am using the following command to output IPs from live tcpdump capture
sudo tcpdump -nn -q ip -l | awk '{print $3; fflush(stdout)}' >> ips.txt
I get the following output
192.168.0.100.50771
192.168.0.100.50770
192.168.0.100.50759
Need 2 things:
Extract only the IPs, not the ports.
Generate a file with unique IPs, no duplicated, and sorted if posible.
Thank you in advance
To extract unique IPs from tcpdump you can use:
awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+).*/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' YOURFILE
So your command to see unique IPs live would be:
sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }'
This will print each IP to output as soon as they appear, so it cannot sort them. If you want to sort those, you can save the output to a file and then use sort tool:
sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' > IPFILE
sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 IPFILE
Example output:
34.216.156.21
95.46.98.113
117.18.237.29
151.101.65.69
192.168.1.101
192.168.1.102
193.239.68.8
193.239.71.100
202.96.134.133
NOTE: make sure you are using gawk. It doesn't work with mawk.
While I'm a huge Awk fan, it's worthwhile having alternatives. Consider this example using cut:
tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | sort | uniq
This is a using match (working in macOs)
sudo tcpdump -nn -q ip -l | \
awk '{match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
ip = substr($3,RSTART,RLENGTH); \
if (!seen[ip]++) print ip }'
In case want to pre-filter the input you could use something like:
sudo tcpdump -nn -q ip -l | \
awk '$3 !~ /^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ \
{match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
ip = substr($3,RSTART,RLENGTH); \
if (!seen[ip]++) print ip }'
sudo tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | awk '!x[$0]++'
Is the command that did it for me. Simple and elegant.

How to create an alias correctly

I've tested this command
$ nmap -sP 192.168.1.* | grep 192 | awk '{print $5}'
which produces this output
192.168.1.1
192.168.1.33
192.168.1.34
192.168.1.36
192.168.1.41
And then added it to my .bash_alias file and then sourced it.
# This alias shows IPs on the local network
alias list-ip="nmap -sP 192.168.1.* | grep 192 | awk '{print $5}'"
But then it produces this output
Nmap scan report for 192.168.1.1
Nmap scan report for 192.168.1.33
Nmap scan report for 192.168.1.34
Nmap scan report for 192.168.1.36
Nmap scan report for 192.168.1.41
I've got no clue on what I'm doing whrong. I just want the output to be like when I run it on command-line, and it should be.
You use double quotes, so $5 gets expanded at the time you set the alias. Try
alias list-ip="nmap -sP 192.168.1.* | grep 192 | awk '{print \$5}'"
Note that
alias list-ip='nmap -sP 192.168.1.* | grep 192 | awk "{print $5}"'
will not work because the expansion still takes place, this time when you run the alias.
You can also get rid of the awk, e.g.:
alias list-ip='nmap -sP 192.168.1.* | grep -o "192[0-9.]*"'