I am trying to pipe tshark output to awk. The tshark command works fine on its own, and when piped to other programs such as cat, it works fine (real time printing of output). However, when piped to awk, it hangs and nothing happens.
sudo tshark -i eth0 -l -f "tcp" -R 'http.request.method=="GET"' -T fields -e ip.src -e ip.dst -e
tcp.srcport -e tcp.dstport -e tcp.seq -e tcp.ack | awk '{printf("mz -A %s -B %s -tcp \"s=%s sp=%s
dp=%s\"\n", $2, $1, $5, $4, $3)}'
Here is a simplier version:
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' | awk '{print $0}'
And to compare, the following works fine (although is not very useful):
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' | cat
Thanks in advance.
I had the same problem.
I have found some partial "solutions" that are not completely portable.
Some of them point to use the fflush() or flush() awk functions or -W interactive option
http://mywiki.wooledge.org/BashFAQ/009
I tried both and none works. So awk is not the appropriate command at all.
A few of them suggest to use gawk but it neither does the trick for me.
cut command has the same problem.
My solution: In my case I just needed to put --line-buffered in GREP and not touching awk command but in your case I would try:
sed -u
with the proper regular expression. For example:
sed -u 's_\(.*\) \(.*\) \(.*\) DIFF: \(.*\)_\3 \4_'
This expression gives you the 3rd and 4th columns separate by TAB (written with ctrl+v and TAB combination). With -u option you get unbuffered output and also you have -l option that gives you line buffered output.
I hope you find this answer useful although is late
Per our previous messages in comments, maybe it will work to force closing the input and emitting a linefeed.
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' ...... \
| {
awk '{print $0}'
printf "\n"
}
Note, no pipe between awk and printf.
I hope this helps.
I found the solution here https://superuser.com/questions/742238/piping-tail-f-into-awk (by John1024).
It says:
"You don't see it in real time because, for purposes of efficiency, pipes are buffered. tail -f has to fill up the buffer, typically 4 kB, before the output is passed to awk."
The proposed solutions is to use "unbuffer" or "stdbuf -o0" commands to disable buffering. It worked for me like this:
stdbuf -o0 tshark -i ens192 -f "ip" | awk '{print $0}'
Related
Problem Set (Raspberry Pi OS):
I have a file example.conf that contains a line IPv4addr=XXXXX. I am attempting to change this to the IP that is generated the in the command
ipTest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
I want to automate this file change during a script install.sh, the line I am attempting is:
IPtest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
sudo sed -e "/IPv4addr/s/[^=]*$/$IPtest/" example.conf
Returns error:
sed: -e expression #1, char 32: unknown option to `s'
A simple line in that code works, such as SimpleTest='Works'
Any thoughts? I am open to other solutions as well, however I am not an experienced linux user so I am using the tools I know to work with other problem sets.
$IPtest contains the / character; try something like this:
IPtest=$(ip --brief a show | grep eth0 | awk '{ print $3 }')
sudo sed -e '/IPv4addr/s#[^=]*$#'"$IPtest"'#' example.conf
You can shorten your variable and allow awk to do the job of grep at the same time
IPtest=$(ip --brief a s | awk '/eth0/{print $3}')
Using sed grouping and back referencing
sed -i.bak "s|\([^=]*.\).*|\1$IPtest|" example.conf
More with my radio logs... So far the work I have completed is great, push notifications are going out perfectly, too well in fact. I am getting bombarded with every accidental connection. I want to filter the log to only send notifications if the mic is keyed up longer than 1.5 seconds (the 4th field), cutting the notifications down to real communication only. I am trying to filter the original "completelog.txt" file over to the "filteredlog.txt" file. The original looks like this:
W7MIT,TG 91,4601149,0.5,0,0.00
W7MIT,TG 91,4601149,0.8,0,0.00
VA3GJB,TG 91,4601149,1.2,0,0.00
G0IOR,TG 91,2345614,11.6,0,0.00
G0IOR,TG 91,2345614,10.6,0,0.00
G4NHP,TG 91,2345614,6.2,2,0.00
GM0TKC,TG 91,2345614,0.5,0,0.00
EA8CTB,TG 91,2345614,2.3,0,0.00
EA8CTB,TG 91,2345614,1.6,0,0.00
GM1RCP,TG 91,2345614,0.5,12,0.00
The code I have tried using is here:
#!/bin/bash
awk -v x=1.5 '$4 >= x' completelog.txt >> filteredlog.txt
Then I tried not appending (I think)
#!/bin/bash
awk -v x=1.5 '$4 >= x' completelog.txt > filteredlog.txt
Since neither worked I added sudo (don't think that is needed)
#!/bin/bash
sudo awk -v x=1.5 '$4 >= x' completelog.txt >> filteredlog.txt
And again not appending (still thinking that is what that means).
#!/bin/bash
sudo awk -v x=1.5 '$4 >= x' completelog.txt > filteredlog.txt
Then I just started guessing random things.
awk -v x=1.5 '$4 >= x' completelog.txt
And again...
awk '$4>=1.5{print}' completelog.txt
All to no avail. I did get few permission errors even though the files are chmoded and even tried with sudo. Please tell me what am I doing wrong?
The issue is that you aren't setting the delimiter (Field Separator) in awk so it's expecting whitespace delimitation by default. Instead:
awk -F"," -v x=1.5 '$4 >= x' completelog.txt >> filteredlog.txt
You can also specify this in the BEGIN block of the awk script itself by setting the FS built-in variable:
awk -v x=1.5 'BEGIN{OFS=FS=","}$4>=x' completelog.txt >> filteredlog.txt
That sets the Field Separator FS and the Output Field Separator OFS to a comma.
(not an answer, an extended comment)
The sudo [command ...] > file redirection is kind of misleading: you run the command with sudo privs, but the file redirection is done with your privs. This is because the shell processes redirections before processing the actual command.
Two workarounds:
use tee with sudo privs:
some command with args | sudo tee -a privileged_file
This runs the command as you, but appends to the file as sudo.
wrap everything in a shell with sudo privs:
sudo sh -c 'some command with args >> privileged_file'
Permission error noticed after posting, but the comma delimiter was also a great catch, thank you both.
pi-star#pi-star(rw):~$ awk -F"," '$4>=1.6{print}' completelog.txt >
filteredlog.txt
-bash: filteredlog.txt: Permission denied
pi-star#pi-star(rw):~$ touch filteredlog.txt
touch: cannot touch ‘filteredlog.txt’: Permission denied
pi-star#pi-star(rw):~$ ls -l filteredlog.txt
-rwxr-xr-x 1 root root 0 Jun 10 08:09 filteredlog.txt
pi-star#pi-star(rw):~$ sudo chmod 777 filteredlog.txt
pi-star#pi-star(rw):~$ ls -l filteredlog.txt
-rwxrwxrwx 1 root root 0 Jun 10 08:09 filteredlog.txt
pi-star#pi-star(rw):~$ awk -F"," '$4>=1.6{print}' completelog.txt >
filteredlog.txt
pi-star#pi-star(rw):~$ tail -f -n2 filteredlog.txt
PD2RAL,TG 91,2042416,6.7,1,0.00
N1AJW,TG 91,2042416,2.2,2,6.70
I'm writing a shell script which shut down some services and trying to get its pid by using the following awk script.
However, this awk script can't get pid. What's wrong with that?
ps -ef | awk -v port_no=10080 '/[m]ilk.*port=port_no/{print $2}'
The result of ps -ef is like this:
username 13155 27705 0 16:06 pts/2 00:00:00 /home/username/.rbenv/versions/2.3.6/bin/ruby /home/username/.rbenv/versions/2.3.6/bin/milk web --no-browser --host=example.com --port=10080
This process is working with a different port argument as well, so I want to kill the process only working on port=10080.
The awk script below works fine, but when I specify the port no using awk -v like the above, it doesn't work well.
ps -ef | awk '/[m]ilk.*port=10080/{print $2}'
awk version: GNU Awk 4.0.2
The syntax for pattern matching with /../ does not work with variables in the regular expression. You need to use the ~ syntax for it.
awk -v port_no=10080 '$0 ~ "[m]ilk.*port="port_no{print $2}'
If you notice the regex carefully, the regex string on the r.h.s of ~ is under the double-quotes ".." except the variable name holding the port number which shouldn't be under quotes, for the expansion to happen.
This task is easily accomplished using pgrep:
$ pgrep -f '[m]ilk.*port=10080'
Have a look at man pgrep for details.
In a script I want to search connections established between some ports gathered with another command and set on PORT variable and specific systems.
the PORT variable is pass to awk using -vp=${PORT}
but I don't know how to use "p" it inside the rest of the pattern.
his does not work:
$ lsof -i -P|awk -vp=${PORT} '$(NF-1)~/vm7.+:'$p'->(vm9|vm11).+ESTABLISHED/{print $(NF-1)}'
$ lsof -i -P|awk -vp=${PORT} '$(NF-1)~/vm7.+:'p'->(vm9|vm11).+ESTABLISHED/{print $(NF-1)}'
give this a try:
awk -v p="$PORT" '{pat="yourHost(or whatever):"p}$(NF-1)~pat{print $(NF-1)}'
build the pattern(pat) with p and check the field (NF-1)
you don't need (shouldn't have) the ESTABLISHED in pattern, since it is the last field NF instead of NF-1
Use match:
$ awk -v p=$port 'match($(NF-1),"vm7.+:" p "->(vm9|vm11)"){print $(NF-1)}'
There might be some errors as there was no test material. Removed the ESTABLISHED as it is in $NF, not $(NF-1) (in my systems, at least).
... or don't:
$ awk -v p=$port '$(NF-1) ~ "vm7.+:" p "->(vm9|vm11)" {print $(NF-1)}'
Today I learned something.
I am using the following command to output IPs from live tcpdump capture
sudo tcpdump -nn -q ip -l | awk '{print $3; fflush(stdout)}' >> ips.txt
I get the following output
192.168.0.100.50771
192.168.0.100.50770
192.168.0.100.50759
Need 2 things:
Extract only the IPs, not the ports.
Generate a file with unique IPs, no duplicated, and sorted if posible.
Thank you in advance
To extract unique IPs from tcpdump you can use:
awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+).*/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' YOURFILE
So your command to see unique IPs live would be:
sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }'
This will print each IP to output as soon as they appear, so it cannot sort them. If you want to sort those, you can save the output to a file and then use sort tool:
sudo tcpdump -nn -q ip -l | awk '{ ip = gensub(/([0-9]+.[0-9]+.[0-9]+.[0-9]+)(.*)/,"\\1","g",$3); if(!d[ip]) { print ip; d[ip]=1; fflush(stdout) } }' > IPFILE
sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 IPFILE
Example output:
34.216.156.21
95.46.98.113
117.18.237.29
151.101.65.69
192.168.1.101
192.168.1.102
193.239.68.8
193.239.71.100
202.96.134.133
NOTE: make sure you are using gawk. It doesn't work with mawk.
While I'm a huge Awk fan, it's worthwhile having alternatives. Consider this example using cut:
tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | sort | uniq
This is a using match (working in macOs)
sudo tcpdump -nn -q ip -l | \
awk '{match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
ip = substr($3,RSTART,RLENGTH); \
if (!seen[ip]++) print ip }'
In case want to pre-filter the input you could use something like:
sudo tcpdump -nn -q ip -l | \
awk '$3 !~ /^(192\.168|10\.|172\.1[6789]|172\.2[0-9]\.|172\.3[01]\.)/ \
{match($3,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/); \
ip = substr($3,RSTART,RLENGTH); \
if (!seen[ip]++) print ip }'
sudo tcpdump -n ip | cut -d ' ' -f 3 | cut -d '.' -f 1-4 | awk '!x[$0]++'
Is the command that did it for me. Simple and elegant.