How to extract lines between multiline patterns? - awk

Question

I have a file which looks like:
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
<empty line here>
Total DOS and NOS and partial (IT) DOSDOWN
<empty line here>
E Total 1
<empty line here>
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
-1.4859 0.004 0.000 0.004
-1.4812 0.004 0.000 0.004
0.3563 0.708 5.510 0.708
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149
<empty line here>
Sublattice 1 Atom Fe spin DOWN
What I want is to extract all lines between (first pattern)
Total DOS and NOS and partial (IT) DOSUP
<empty line here>
E Total 1
<empty line here>
and (second pattern)
<empty line here>
Sublattice 1 Atom Fe spin DOWN
i.e. I want to get
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
-1.4859 0.004 0.000 0.004
-1.4812 0.004 0.000 0.004
0.3563 0.708 5.510 0.708
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149
So, at the end of the day I want to have lines between two multiline patterns.
As I understand awk can detect multiline patterns via state machine (see here), but I failed to do it in my case.
Any suggestion how to resolve this problem would be very much appreciated.

Answer 1

Here's a solution based on Ed Morton's trick.
awk -v RS= 'n==2; /Total DOS/ || n {n++;next} {n=0}' input.txt
Here's how this works.
RS= puts awk into multi-line mode, so that records contain blocks of lines.
n==2; prints any record processed while this condition is met.
/RE/ || n is a condition that evaluates to true if EITHER the RE (pattern) is matched within the current record or the variable n is non-zero.
{n++;next} obviously increments n and skips to the next record.
{n=0} And if we haven't already skipped to the next record, we reset n.
The effect of all this is that we print the record that is two records after the one with the matched pattern. You could of course adjust the condition that begins the counter to whatever you like. $2=="Total" for example. Salt to taste.
sh-3.2$ cat input.txt
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
Total DOS and NOS and partial (IT) DOSUP
E Total 1
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
....... ..... ..... .....
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149
blah blah blah blah
sh-3.2$ awk -v RS= 'n==2; /Total DOS and NOS/||n{n++;next} {n=0}' input.txt
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
....... ..... ..... .....
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149

Answer 2

Using sed: sed -n '5,/^$/{/^$/d}'
But that assumes that "multiline starting pattern" is always at the beginning of the file. Otherwise it gets a bit more complicated. Like this:
/Total/{N;N;N}
/Total.*Total/,/^$/{
/Total/d
/^$/d
}
Here I am assuming that 'Total' matches the beginning of multiline pattern, 'Total.*Total' matches the whole pattern. Replace N;N;N with something more complex if there are other patterns that start with first line of you multiline pattern but are shorter than 4 lines.

Answer 3

From your comments it sounds like all you need is:
awk -v RS= '/Total DOS/{tgt=NR+2} NR==tgt' file
If not then edit your question to clarify. Make it NR==tgt{print; exit} if you only want the first matching block in the file output and efficiency is a concern. Change the regexp if necessary to be as much of the Total DOS... line as you need to match to make it unique.
Here it is running against your provided sample input:
$ cat file
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
blah blah blah blah blah blah blah blah
Total DOS and NOS and partial (IT) DOSUP
E Total 1
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
....... ..... ..... .....
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149
blah blah blah blah
$ awk -v RS= '/Total DOS/{tgt=NR+2} NR==tgt' file
-1.5000 0.004 0.000 0.004
-1.4953 0.004 0.000 0.004
-1.4906 0.004 0.000 0.004
....... ..... ..... .....
0.3609 0.562 5.513 0.562
0.3656 0.381 5.515 0.381
0.3703 0.149 5.517 0.149