Tail and grep combo to show text in a multi line pattern - awk

Not sure if you can do this through tail and grep. Lets say I have a log file that I would like to tail. It spits out quite a bit of information when in debug mode. I want to grep for information pertaining to only my module, and the module name is in the log like so:
/*** Module Name | 2014.01.29 14:58:01
a multi line
dump of some stacks
or whatever
**/
/*** Some Other Module Name | 2014.01.29 14:58:01
this should show up in the grep
**/
So as you can imagine, the number of lines that would be spit out pertaining to "Module Name" could be 2, or 50 until the end pattern appears (**/)

From your description, it seems like this should work:
tail -f log | awk '/^\/\*\*\* Module Name/,/^\*\*\//'
but be wary of buffering issues. (Lines printed to the file will very likely see high latency before actually being printed.)

I think you will have to install pcregrep for this. Then this will work:
tail -f logfile | pcregrep -M "^\/\*\*\* Some Other Module Name \|.*(\n|.)*?^\*\*/$"
This worked in testing, but for some reason, I had to append your example text to the log file over 100 times before output started showing up. However, all of the "Some other Module Name" data that was written to the log file as soon as this line was invoked was eventually printed to stdout.

Related

Awk file shows an error

i run a simulation with udp connection and cbr application in ns2.
I run an awk file to show some results. I have 10 nodes and 2 of them (n0 and n10) send packets to each other. My problem is that i run these 2 codes to get result for both nodes but only the first one runs smoothly (n0 => n10):
1)gawk -f stats.awk src=0 dst=10 flow=0 pkt=512 file.tr
2) gawk -f stats .awk src=10 dst=0 flow=0 pkt=512 file.tr
The second code gives "division by zero attempted" error"
I don't include the stats.awk file here because it works in the first example so i guess the problem is something else? eg the flow in the second line is 0? i tried other numbers too but nothing happened.
Thanks in advance.

Read all lines at the same time individually - Solaris ksh

I need some help with a script. Solaris 10 and ksh.
I Have a file called /temp.list with this content:
192.168.0.1
192.168.0.2
192.168.0.3
So, I have a script which reads this list and executes some commands using the lines values:
FILE_TMP="/temp.list"
while IFS= read line
do
ping $line
done < "$FILE_TMP"
It works, but it executes the command on line 1. When it's over, it goes to the line 2, and it goes successively until the end. I would like to find a way to execute the command ping at the same time in each line of the list. Is there a way to do it?
Thank you in advance!
Marcus Quintella
As Ari's suggested, googling ksh multithreading will produce a lot of ideas/solutions.
A simple example:
FILE_TMP="/temp.list"
while IFS= read line
do
ping $line &
done < "$FILE_TMP"
The trailing '&' says to kick the ping command off in the background, allowing loop processing to continue while the ping command is running in the background.
'course, this is just the tip of the proverbial iceberg as you now need to consider:
multiple ping commands are going to be dumping output to stdout (ie, you're going to get a mish-mash of ping output in your console), so you'll need to give some thought as to what to do with multiple streams of output (eg, redirect to a common file? redirect to separate files?)
you need to have some idea as to how you want to go about managing and (possibly) terminating commands running in the background [ see jobs, ps, fg, bg, kill ]
if running in a shell script you'll likely find yourself wanting to suspend the main shell script processing until all background jobs have completed [ see wait ]

awk/sed - generate an error if 2nd address of range is missing

We are currently using sed to filter output of regression runs. Sometimes we have a filter that looks like this:
/copyright/,/end copyright/d
If that end copyright is ever missing, the rest of the file is deleted. I'm wondering if there's some way to generate an error for this? awk would also be okay to use. I don't really want to add code that reads the file line by line and issues an error if it hits EOF.
here's a string
copyright
2016 jan 15
end copyright
date 2016 jan 5 time 15:36
last one
I'd like to get an error if end copyright is missing. The real filter also would replace the date line with DATE, so it's more that just ripping out the copyright.
You can persuade sed to generate an error if you reach end of input (i.e. see address $) between your start and end, but it won't be a very helpful message:
/copyright/,/end copyright/{
$s//\1/ # here
d
}
This will error if end copyright is missing or on the last line, with an exit status of 1 and the helpful message:
sed: -e expression #1, char 0: invalid reference \1 on `s' command's RHS
If you're using this in a makefile, you might want to echo a helpful message first, or (better) to wrap this in something that catches the error and produces a more useful one.
I tested this with GNU sed; though if you are using GNU sed, you could more easily use its useful extension:
q [EXIT-CODE]
This command only accepts a single address.
Exit 'sed' without processing any more commands or input. Note
that the current pattern space is printed if auto-print is not
disabled with the -n options. The ability to return an exit code
from the 'sed' script is a GNU 'sed' extension.
Q [EXIT-CODE]
This command only accepts a single address.
This command is the same as 'q', but will not print the contents of
pattern space. Like 'q', it provides the ability to return an exit
code to the caller.
So you could simply write
/copyright/,/end copyright/{
$Q 42
d
}
Never use range expressions /start/,/end/ as they make trivial code very slightly briefer but require a complete rewrite or duplicate conditions when you have the tiniest requirements change. Always use a flag instead. Note that since sed doesn't support variables, it doesn't support flag variables, and so you shouldn't be using sed you should be using awk instead.
In this case your original code would be:
awk '/copyright/{f=1} !f; /end copyright/{f=0}' file
And your modified code would be:
awk '/copyright/{f=1} !f; /end copyright/{f=0} END{if (f) print "Missing end copyright"}' file
The above is obviously untested since you didn't provide any sample input/output we could test a potential solution against.
With sed you can build a loop:
sed -e '/copyright/{:a;/end copyright/d;N;ba;};' file
:a defines the label "a"
/copyright end/d deletes the pattern space, only when "end copyright" matches
N appends the next line to the pattern space
ba jumps to the label "a"
Note that d ends the loop.
In this way you can avoid to delete the text until the end.
If you don't want the text to be displayed at all and prefer an error message when a "copyright" block stays unclosed, you obviously need to wait the end of the file. You can do it with sed too storing all the lines in the buffer space until the end:
sed -n -e '/copyright/{:a;/end copyright/d;${c\ERROR MESSAGE
;};N;ba;};H;${g;p};' file
H appends the current line to the buffer space
g put the content of the buffer space to the pattern space
The file content is only displayed once the last line reached with ${g;p} otherwise when the closing "end copyright" is missing, the current line is changed in the error message with ${c\ERROR MESSAGE\n;} inside the loop.
This way you can test what returns sed before redirecting it to whatever you want.

How to store a command output in OpenVMS

Im having an issue writing a DCL in OpenVMS in that I need the DCL to call a command and capture its output (but not output the output to the screen) Later on in the DCL I then need to print that output I stored.
Heres an example:
ICE SET FASTER !This command sets my environment to the "Faster" environment.
The above command outputs this if executed directly in OpenVMS:
Initialising TEST Environment to FASTER
--------------------------------------------------------------------------------
Using Test Search rules FASTER
Using Test Search rules FASTER
--------------------------------------------------------------------------------
dcl>
So I created a DCL in an attempt to wrap this output in order to display a more simplified output. Heres my code so far:
!************************************************************************
$ !* Wrapper for setting ICE account. Outputs Environment
$ !************************************************************************
$ on error then goto ABORT_PROCESS
$ICE_DCL_MAIN:
$ ice set 'P1'
$ ICE SHOW
$ EXIT
$ABORT_PROCESS:
$ say "Error ICING to: " + P1
$ EXIT 2
[End of file]
In the lines above ICE SET 'P1' is setting the ice environment, but I dont want this output to be echoed to VMS. But what I do want is to write the output of $ICE SHOW into a variable and then echo that out later on in the DCL (most of which ive omitted for simplification purposes)
So what should be outputted should be:
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
Instead of:
Initialising TEST Environment to FASTER
--------------------------------------------------------------------------------
Using Test Search rules FASTER
Using Test Search rules FASTER
--------------------------------------------------------------------------------
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
Ive had a look through the manual and im getting a bit confused so I figured I tried here. Id appreciate any pointers. Thanks.
EDIT
Here is what ive come up with after the comments, the problem im having is when I connect to VMS using an emulator such as SecureCRT the correct output is echoed. But when I run the DCL via my SSH2 library in .NET it doesnt output anything. I guess thats because it closes the SYS$OUTPUT stream temporarily or something?
$ !************************************************************************
$ !* Wrapper for setting ICE account. Outputs Environment
$ !************************************************************************
$ on error then goto ABORT_PROCESS
$ICE_DCL_MAIN:
$ DEFINE SYS$OUTPUT NL:
$ ice set 'P1'
$ DEASSIGN SYS$OUTPUT
$ ice show
$ EXIT
$ABORT_PROCESS:
$ say "Error ICING to: " + P1
$ EXIT 2
[End of file]
EDIT 2
So I guess really I need to clarify what im trying to do here. Blocking the output doesnt so matter so much, im merely trying to capture it into a Symbol for example.
So in C# for example you can have a method that returns a string. So you'd have string myResult = vms.ICETo("FASTER"); and it would return that and store it in the variable.
I guess im looking for a similar thing in VMS so that once ive iced to the environment I can call:
$ environment == $ICE SHOW
But I of course get errors with that statement
The command $ assign/user_mode Thing Sys$Output will cause output to be redirected to Thing until you $ deassign/user_mode Sys$Output or next executable image exits. An assignment without the /USER_MODE qualifier will persist until deassigned.
Thing can be a logical name, a file specification (LOG.TXT) or the null device (NLA0:) if you simply want to flush the output.
When a command procedure is executed the output can be redirected using an /OUTPUT qualifier, e.g. $ #FOO/output=LOG.TXT.
And then there is piping ... .
You can redirect the output to a temp file and then print its content later:
$ pipe write sys$output "hi" > tmp.tmp
$ ty tmp.tmp
VMS is not Unix, DCL is not Bash: you can not easily set a DCL symbol from the output of a command.
Your ICE SHOW prints one line, correct? The first word is always "current", correct?
So you can create a hack.
First let me fake your ICE command:
$ create ice.com
$ write sys$output "current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]"
^Z
$
and I define a dcl$path pointing to the directory where this command procedure is
so that I can use/fake the command ICE
$ define dcl$path sys$disk[]
$ ice show
current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]
$
Now what you need, create a command procedure which sets a job logical
$ cre deflog.com
$ def/job/nolog mylog "current''p1'"
^Z
$
And I define a command "current" to run that command procedure:
$ current="#deflog """
Yes, you need three of the double quotes at the end of the line!
And finally:
$ pipe (ice show | #sys$pipe) && mysym="''f$log("mylog")'"
$ sh symb mysym
MYSYM = "current Test Environment is DISK$DEVELOPERS:[FASTER.DEVELOP]"
$
On the other hand, I don't know what you are referring to C# and Java. Can you elaborate on that and tell us what runs where?
You can try using: DEFINE /USER SYS$OUTPUT NL:.
It works only for the next command and you dont need to deassign.
Sharing some of my experience here. I used below methods to redirect outputs to files.
Define/Assign the user output and then execute the required command/script afterwards. Output will be written to .
$define /user sys$output <file_path>
execute your command/script
OR
assign /user <file_path> sys$output
execute your command/script
deassign sys$output
To re-direct in to null device like in Unix (mentioned in above answers), you can use 'nl:' instead of
define /user sys$output nl:
or
assign /user nl: sys$output

Nano hacks: most useful tiny programs you've coded or come across

It's the first great virtue of programmers. All of us have, at one time or another automated a task with a bit of throw-away code. Sometimes it takes a couple seconds tapping out a one-liner, sometimes we spend an exorbitant amount of time automating away a two-second task and then never use it again.
What tiny hack have you found useful enough to reuse? To make go so far as to make an alias for?
Note: before answering, please check to make sure it's not already on favourite command-line tricks using BASH or perl/ruby one-liner questions.
i found this on dotfiles.org just today. it's very simple, but clever. i felt stupid for not having thought of it myself.
###
### Handy Extract Program
###
extract () {
if [ -f $1 ] ; then
case $1 in
*.tar.bz2) tar xvjf $1 ;;
*.tar.gz) tar xvzf $1 ;;
*.bz2) bunzip2 $1 ;;
*.rar) unrar x $1 ;;
*.gz) gunzip $1 ;;
*.tar) tar xvf $1 ;;
*.tbz2) tar xvjf $1 ;;
*.tgz) tar xvzf $1 ;;
*.zip) unzip $1 ;;
*.Z) uncompress $1 ;;
*.7z) 7z x $1 ;;
*) echo "'$1' cannot be extracted via >extract<" ;;
esac
else
echo "'$1' is not a valid file"
fi
}
Here's a filter that puts commas in the middle of any large numbers in standard input.
$ cat ~/bin/comma
#!/usr/bin/perl -p
s/(\d{4,})/commify($1)/ge;
sub commify {
local $_ = shift;
1 while s/^([ -+]?\d+)(\d{3})/$1,$2/;
return $_;
}
I usually wind up using it for long output lists of big numbers, and I tire of counting decimal places. Now instead of seeing
-rw-r--r-- 1 alester alester 2244487404 Oct 6 15:38 listdetail.sql
I can run that as ls -l | comma and see
-rw-r--r-- 1 alester alester 2,244,487,404 Oct 6 15:38 listdetail.sql
This script saved my career!
Quite a few years ago, i was working remotely on a client database. I updated a shipment to change its status. But I forgot the where clause.
I'll never forget the feeling in the pit of my stomach when I saw (6834 rows affected). I basically spent the entire night going through event logs and figuring out the proper status on all those shipments. Crap!
So I wrote a script (originally in awk) that would start a transaction for any updates, and check the rows affected before committing. This prevented any surprises.
So now I never do updates from command line without going through a script like this. Here it is (now in Python):
import sys
import subprocess as sp
pgm = "isql"
if len(sys.argv) == 1:
print "Usage: \nsql sql-string [rows-affected]"
sys.exit()
sql_str = sys.argv[1].upper()
max_rows_affected = 3
if len(sys.argv) > 2:
max_rows_affected = int(sys.argv[2])
if sql_str.startswith("UPDATE"):
sql_str = "BEGIN TRANSACTION\\n" + sql_str
p1 = sp.Popen([pgm, sql_str],stdout=sp.PIPE,
shell=True)
(stdout, stderr) = p1.communicate()
print stdout
# example -> (33 rows affected)
affected = stdout.splitlines()[-1]
affected = affected.split()[0].lstrip('(')
num_affected = int(affected)
if num_affected > max_rows_affected:
print "WARNING! ", num_affected,"rows were affected, rolling back..."
sql_str = "ROLLBACK TRANSACTION"
ret_code = sp.call([pgm, sql_str], shell=True)
else:
sql_str = "COMMIT TRANSACTION"
ret_code = sp.call([pgm, sql_str], shell=True)
else:
ret_code = sp.call([pgm, sql_str], shell=True)
I use this script under assorted linuxes to check whether a directory copy between machines (or to CD/DVD) worked or whether copying (e.g. ext3 utf8 filenames -> fusebl
k) has mangled special characters in the filenames.
#!/bin/bash
## dsum Do checksums recursively over a directory.
## Typical usage: dsum <directory> > outfile
export LC_ALL=C # Optional - use sort order across different locales
if [ $# != 1 ]; then echo "Usage: ${0/*\//} <directory>" 1>&2; exit; fi
cd $1 1>&2 || exit
#findargs=-follow # Uncomment to follow symbolic links
find . $findargs -type f | sort | xargs -d'\n' cksum
Sorry, don't have the exact code handy, but I coded a regular expression for searching source code in VS.Net that allowed me to search anything not in comments. It came in very useful in a particular project I was working on, where people insisted that commenting out code was good practice, in case you wanted to go back and see what the code used to do.
I have two ruby scripts that I modify regularly to download all of various webcomics. Extremely handy! Note: They require wget, so probably linux. Note2: read these before you try them, they need a little bit of modification for each site.
Date based downloader:
#!/usr/bin/ruby -w
Day = 60 * 60 * 24
Fromat = "hjlsdahjsd/comics/st%Y%m%d.gif"
t = Time.local(2005, 2, 5)
MWF = [1,3,5]
until t == Time.local(2007, 7, 9)
if MWF.include? t.wday
`wget #{t.strftime(Fromat)}`
sleep 3
end
t += Day
end
Or you can use the number based one:
#!/usr/bin/ruby -w
Fromat = "http://fdsafdsa/comics/%08d.gif"
1.upto(986) do |i|
`wget #{sprintf(Fromat, i)}`
sleep 1
end
Instead of having to repeatedly open files in SQL Query Analyser and run them, I found the syntax needed to make a batch file, and could then run 100 at once. Oh the sweet sweet joy! I've used this ever since.
isqlw -S servername -d dbname -E -i F:\blah\whatever.sql -o F:\results.txt
This goes back to my COBOL days but I had two generic COBOL programs, one batch and one online (mainframe folks will know what these are). They were shells of a program that could take any set of parameters and/or files and be run, batch or executed in an IMS test region. I had them set up so that depending on the parameters I could access files, databases(DB2 or IMS DB) and or just manipulate working storage or whatever.
It was great because I could test that date function without guessing or test why there was truncation or why there was a database ABEND. The programs grew in size as time went on to include all sorts of tests and become a staple of the development group. Everyone knew where the code resided and included them in their unit testing as well. Those programs got so large (most of the code were commented out tests) and it was all contributed by people through the years. They saved so much time and settled so many disagreements!
I coded a Perl script to map dependencies, without going into an endless loop, For a legacy C program I inherited .... that also had a diamond dependency problem.
I wrote small program that e-mailed me when I received e-mails from friends, on an rarely used e-mail account.
I wrote another small program that sent me text messages if my home IP changes.
To name a few.
Years ago I built a suite of applications on a custom web application platform in PERL.
One cool feature was to convert SQL query strings into human readable sentences that described what the results were.
The code was relatively short but the end effect was nice.
I've got a little app that you run and it dumps a GUID into the clipboard. You can run it /noui or not. With UI, its a single button that drops a new GUID every time you click it. Without it drops a new one and then exits.
I mostly use it from within VS. I have it as an external app and mapped to a shortcut. I'm writing an app that relies heavily on xaml and guids, so I always find I need to paste a new guid into xaml...
Any time I write a clever list comprehension or use of map/reduce in python. There was one like this:
if reduce(lambda x, c: locks[x] and c, locknames, True):
print "Sub-threads terminated!"
The reason I remember that is that I came up with it myself, then saw the exact same code on somebody else's website. Now-adays it'd probably be done like:
if all(map(lambda z: locks[z], locknames)):
print "ya trik"
I've got 20 or 30 of these things lying around because once I coded up the framework for my standard console app in windows I can pretty much drop in any logic I want, so I got a lot of these little things that solve specific problems.
I guess the ones I'm using a lot right now is a console app that takes stdin and colorizes the output based on xml profiles that match regular expressions to colors. I use it for watching my log files from builds. The other one is a command line launcher so I don't pollute my PATH env var and it would exceed the limit on some systems anyway, namely win2k.
I'm constantly connecting to various linux servers from my own desktop throughout my workday, so I created a few aliases that will launch an xterm on those machines and set the title, background color, and other tweaks:
alias x="xterm" # local
alias xd="ssh -Xf me#development_host xterm -bg aliceblue -ls -sb -bc -geometry 100x30 -title Development"
alias xp="ssh -Xf me#production_host xterm -bg thistle1 ..."
I have a bunch of servers I frequently connect to, as well, but they're all on my local network. This Ruby script prints out the command to create aliases for any machine with ssh open:
#!/usr/bin/env ruby
require 'rubygems'
require 'dnssd'
handle = DNSSD.browse('_ssh._tcp') do |reply|
print "alias #{reply.name}='ssh #{reply.name}.#{reply.domain}';"
end
sleep 1
handle.stop
Use it like this in your .bash_profile:
eval `ruby ~/.alias_shares`