I wrote a program to analyze the log files in real time. I need to feed it with the IPs. It works fine with the command:
cat /var/log/apache2/access.log | awk '{print $1}' | ./my_program
Also, I can get the IPs in real time with the command:
tail -f /var/log/apache2/access.log | awk '{print $1}'
When I pipe it to my program, my program does not receive anything:
tail -f /var/log/apache2/access.log | awk '{print $1}' | ./my_program
It seems like a matter of buffering. Is there way of piping continuous stream to my program?
I found the ultimate solution to my buffering problem here.
The problem is that stdio is being buffered, ...
Related
The Xcode console has a 'Debugger output' filter. I understand this is for use with lldb, and that you can get messages to print to this output by using breakpoints. My question is not how to do that.
My question is: what is the underlying mechanism Xcode itself uses to write lldb messages to Debugger Output (not Target Output)? Is there a variable similar to stdout or stderr that writes here? Is it possible, from Xcode target code (Swift/Obj-C/C), to write to this output?
Looks like Xcode uses a tty to communicate with lldb, and you can interface with the Debugger Output using that:
echo "Wheeeeeeee" > $(lsof -p $(ps -A | grep -m1 MacOS/Xcode | awk '{print $1}') | grep -m2 dev/ttys | tail -1 | awk '{print $9}')
Breaking the above down:
$ ps -A | grep -m1 MacOS/Xcode | awk '{print $1}'
21280
This gives the process ID of Xcode (21280). Using this, we can find the files it has open:
$ lsof -p 21280 | grep /dev/ttys
Xcode 21280 tres 47u CHR 16,3 0t0 3569 /dev/ttys003
Xcode 21280 tres 58u CHR 16,5 0t0 3575 /dev/ttys005
The one with the highest number (/dev/ttys005 in this case) is the one we want, so let's extract it. tail -1 will give us the last line of output, and awk '{print $9}' will give us the 9th item on the line, which is what we want!
$ lsof -p 21280 | grep /dev/ttys | tail -1 | awk '{print $9}'
/dev/ttys005
Now we can use this to write whatever we want:
I'm writing a shell script which shut down some services and trying to get its pid by using the following awk script.
However, this awk script can't get pid. What's wrong with that?
ps -ef | awk -v port_no=10080 '/[m]ilk.*port=port_no/{print $2}'
The result of ps -ef is like this:
username 13155 27705 0 16:06 pts/2 00:00:00 /home/username/.rbenv/versions/2.3.6/bin/ruby /home/username/.rbenv/versions/2.3.6/bin/milk web --no-browser --host=example.com --port=10080
This process is working with a different port argument as well, so I want to kill the process only working on port=10080.
The awk script below works fine, but when I specify the port no using awk -v like the above, it doesn't work well.
ps -ef | awk '/[m]ilk.*port=10080/{print $2}'
awk version: GNU Awk 4.0.2
The syntax for pattern matching with /../ does not work with variables in the regular expression. You need to use the ~ syntax for it.
awk -v port_no=10080 '$0 ~ "[m]ilk.*port="port_no{print $2}'
If you notice the regex carefully, the regex string on the r.h.s of ~ is under the double-quotes ".." except the variable name holding the port number which shouldn't be under quotes, for the expansion to happen.
This task is easily accomplished using pgrep:
$ pgrep -f '[m]ilk.*port=10080'
Have a look at man pgrep for details.
Today I've noticed that I can't use "awk" on "nginx -v".
I've tried running this command: nginx -v | awk -F/ '{print $2}'
This should of been output like this: nginx/1.4.3
But instead it gives me nginx version: nginx/1.4.3
Any idea why it would behave this way !?
Also you can't output it to file by running: nginx -v > file.txt
nginx must be writing that message to standard error, not standard output. If you want to pipe it, you have to redirect stderr to stdout:
nginx -v 2>&1 | awk -F/ '{print $2}'
I am running the following command using awk on file.txt ,currently its running the command on the ids present in file.txt from top to bottom..i want the commmand to be run in the reverse order for the ids present in file.txt..any inputs on how we can do this?
git command $(awk '{print $1}' file.txt)
file.txt contains.
97a65fd1d1b3b8055edef75e060738fed8b31d3
fb8df67ceff40b4fc078ced31110d7a42e407f16
a0631ce8a9a10391ac4dc377cd79d1adf1f3f3e2
.....
If you aren't bound to using awk then tail with the -r (for reverse) argument will do the trick...
myFile.txt
97a65fd1d1b3b8055edef75e060738fed8b31d3
fb8df67ceff40b4fc078ced31110d7a42e407f16
a0631ce8a9a10391ac4dc377cd79d1adf1f3f3e2
Now to print it in reverse...
$ tail -r myFile.txt
a0631ce8a9a10391ac4dc377cd79d1adf1f3f3e2
fb8df67ceff40b4fc078ced31110d7a42e407f16
97a65fd1d1b3b8055edef75e060738fed8b31d3
EDIT:
To output this to a file simply redirect it out...
$ tail -r myFile.txt > newFile.txt
EDIT:
Want to write to the same file? No problem!
tail -r myFile.txt > temp.txt; cat temp.txt > myFile.txt; rm temp.txt;
For some reason when I redirected tail -r to the same file it came back blank, this workaround avoids that issue by writing to a temporary "buffer" file.
To reverse the lines in a file using awk, use
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file
use $1 instead of $0 above to operate on the first field only instead of the whole line.
I am trying to pipe tshark output to awk. The tshark command works fine on its own, and when piped to other programs such as cat, it works fine (real time printing of output). However, when piped to awk, it hangs and nothing happens.
sudo tshark -i eth0 -l -f "tcp" -R 'http.request.method=="GET"' -T fields -e ip.src -e ip.dst -e
tcp.srcport -e tcp.dstport -e tcp.seq -e tcp.ack | awk '{printf("mz -A %s -B %s -tcp \"s=%s sp=%s
dp=%s\"\n", $2, $1, $5, $4, $3)}'
Here is a simplier version:
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' | awk '{print $0}'
And to compare, the following works fine (although is not very useful):
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' | cat
Thanks in advance.
I had the same problem.
I have found some partial "solutions" that are not completely portable.
Some of them point to use the fflush() or flush() awk functions or -W interactive option
http://mywiki.wooledge.org/BashFAQ/009
I tried both and none works. So awk is not the appropriate command at all.
A few of them suggest to use gawk but it neither does the trick for me.
cut command has the same problem.
My solution: In my case I just needed to put --line-buffered in GREP and not touching awk command but in your case I would try:
sed -u
with the proper regular expression. For example:
sed -u 's_\(.*\) \(.*\) \(.*\) DIFF: \(.*\)_\3 \4_'
This expression gives you the 3rd and 4th columns separate by TAB (written with ctrl+v and TAB combination). With -u option you get unbuffered output and also you have -l option that gives you line buffered output.
I hope you find this answer useful although is late
Per our previous messages in comments, maybe it will work to force closing the input and emitting a linefeed.
sudo tshark -i eth0 -f "tcp" -R 'http.request.method=="GET"' ...... \
| {
awk '{print $0}'
printf "\n"
}
Note, no pipe between awk and printf.
I hope this helps.
I found the solution here https://superuser.com/questions/742238/piping-tail-f-into-awk (by John1024).
It says:
"You don't see it in real time because, for purposes of efficiency, pipes are buffered. tail -f has to fill up the buffer, typically 4 kB, before the output is passed to awk."
The proposed solutions is to use "unbuffer" or "stdbuf -o0" commands to disable buffering. It worked for me like this:
stdbuf -o0 tshark -i ens192 -f "ip" | awk '{print $0}'