sed: replace many lines from a file with single notification - awk

Question

I'm looking for a solution in GNU sed, but POSIX sed is OK and awk will be OK but probably more complicated than necessary. I prefer sed for this, it should be easy but I'm stuck. Seems like a one-liner can do this, no need to create a python/bash script or anything.
my attempted solution
sed -i '218,226140d; 218i ...REMOVED...' psql.log
This deletes the desired rows, but the insert gets lost. If I move the insert to line 217 I get:
sed -i '218,226140d; 217i ...REMOVED...' psql.log
result:
┌────────────┬─────────────────────┬─────────────────┐
│ col_one │ col_two │ column_three │
├────────────┼─────────────────────┼─────────────────┤
│ CC00CBSNRY │ 553854451 │ 15003.44 │
│ CC00CBSNRY │ 1334177150 │ 5159.57 │
...REMOVED...
│ CC6XDSQGH2 │ 42385958605 │ [null] │ (line 217 in original file)
│ CC6XJ8YG5C │ 24661013005 │ [null] │ (line 226141 in original file)
│ CC6XJ9HGRG │ 44946564505 │ [null] │
│ CC6XMQW6SJ │ 34496719615 │ [null] │
└────────────┴─────────────────────┴─────────────────┘
I know - this should be good enough, but I'm annoyed that I can't get this simple one-liner to work right. What am I missing?
the problem
I keep the psql.log file as a reference for work I am doing developing SQL code. It's very useful to see iterations of the query and the results.
The problem is that sometimes I forget to limit the output and the query will generate 100k+ rows of results that aren't a helpful reference, and I'd like to delete them from my file, leaving a note that reminds me the query output has been excised.
It would be nice to match the pattern, say every output more than 50 rows I could squash down to just the first 5 rows and the last 5. However, its easy for me to mark the line numbers where I've blown up the file, so I'd be happy with just using sed to delete lines N through M, and insert the message ...REMOVED... where line N was.
Here is an example log file, added notes are in parentheses. The query text can change and the number of columns can be from 1 to 100 or more:
...
********* QUERY **********
select *
from table
where rnk <= 3
**************************
┌────────────┬─────────────────────┬─────────────────┐
│ col_one │ col_two │ column_three │
├────────────┼─────────────────────┼─────────────────┤
│ CC00CBSNRY │ 553854451 │ 15003.44 │
│ CC00CBSNRY │ 1334177150 │ 5159.57 │
│ CC6XDSQGH2 │ 42385958605 │ [null] │ (line 217)
│ CC6XF2SVWT │ 13182280615 │ [null] │
(many rows)
│ CC6XF2XWDT │ 995086081 │ [null] │
│ CC6XFX3TL1 │ 25195177405 │ [null] │
│ CC6XJ8YG5C │ 24661013005 │ [null] │ (line 226141)
│ CC6XJ9HGRG │ 44946564505 │ [null] │
│ CC6XMQW6SJ │ 34496719615 │ [null] │
└────────────┴─────────────────────┴─────────────────┘
(225926 rows)
********* QUERY **********
/* another query begins */
select * from table where X = 1 limit 20;
/* well done you remembered to limit the output */
**************************
...
acceptable output
the query text should all be untouched, and the top/bottom three rows of output kept. The annotation ...REMOVED... has been added and rows 218 through 226140 have been deleted:
********* QUERY **********
select *
from table
where rnk <= 3
**************************
┌────────────┬─────────────────────┬─────────────────┐
│ col_one │ col_two │ column_three │
├────────────┼─────────────────────┼─────────────────┤
│ CC00CBSNRY │ 553854451 │ 15003.44 │
│ CC00CBSNRY │ 1334177150 │ 5159.57 │
│ CC6XDSQGH2 │ 42385958605 │ [null] │ (line 217 in original file)
...REMOVED...
│ CC6XJ8YG5C │ 24661013005 │ [null] │ (line 226141 in original file)
│ CC6XJ9HGRG │ 44946564505 │ [null] │
│ CC6XMQW6SJ │ 34496719615 │ [null] │
└────────────┴─────────────────────┴─────────────────┘
(225926 rows)
********* QUERY **********
(etc just like example above)
update
the border comes from my .psqlrc with \pset border 2
therefore solutions depending on the ┌ character are fragile but OK
over time i've learned that manually flagging the line numbers is too time consuming, so the best solution needs a pattern match

Answer 1

There is example 'every output more than 50 rows I could squash down to just the first 5 rows and the last 5'.
With test input:
$ seq 160 | awk -vstart=10 -vmax=50 -vleft=5 '{if(NR < start) {print; next} {i++; if(i <= left || i > max - left){print}; if(i == left + 1){print "...REMOVED..."}if(i == max){i = 0}}}'
If you line put script in file, store this to squash.awk
BEGIN {
start=10;
max=50;
left=5;
}
{
if(NR < start) {
print;
next
}
i++;
if(i <= left || i > max - left) {
print
}
if(i == left + 1) {
print "...REMOVED...";
}
if(i == max) {
i = 0
}
}
For testing:
$ seq 160 | awk -f squash.awk
Variable start is line number from which squashing line will begin.
Variable max is maximum rows (in your example 50).
Variable left is how many rows will left from max first and last.
if(NR < start) {print; next} if line number less then start (in our case 10), we just print them and go to next line.
Here you can put any condition to skip squashing.
i++ it's rows counter increment.
if(i <= left || i > max - left){print} if rows counter less then 5 or more then max - 5 - print it.
if(i == left + 1){print "...REMOVED..."} when we starting skip rows - put "...REMOVED..." message
if(i == max){i = 0} if rows counter reach max, zero it

Answer 2

One in awk:
$ awk '
/^ └/ { # at the end marker
for(j=1;j<=6;j++) # output from the buffer b the wanted records
print b[j]
for(j=(i-2);j<=i;j++)
print b[j]
delete b # reset buffer
i=0 # and flag / counter
}
/^ ┌/ || i { # at the start marker or when flag up
b[++i]=$0 # gather records to buffer
next
} 1' file # print records which are not between the markers

Answer 3

This might work for you (GNU sed):
sed -r '/\o342[^\n]*$/{:a;N;//ba;s/^(([^\n]*\n){6}).*((\n[^\n]*){5})$/\1 ... REMOVED ...\3/}' file
Focus only on table data which will always contains the octal value 342. Gather up table lines in the pattern space, substitute the required value ... REMOVED ... and print. The number of lines above and below the required string can be altered here 6 (headings + 3 rows) and 5 (required string + 3 rows + table count).
To change a range use:
sed 'm,nc ... REMOVE ...' file # where m,n from and to line numbers
or:
sed -e 'ma ...REMOVE ...' -e 'm,nd' file
N.B. the d command terminates any following commands.

Answer 4

The sed man page is more helpful than you might think at first glance. The [addr]c command is exactly what is needed (note the whitespace after c is ignored):
sed -i '218,226141c ...REMOVED...' psql.log
So there is the solution for known line numbers.
Does anyone want to provide a generic solution where the line numbers aren't known? Probably awk would be the better tool but maybe sed can remove output that is too long.