How to inplace substitute the content between 2 tags with SED (bash)? - awk

I want to inplace edit a file with sed (Oracle-Linux/Bash).
The content between 2 search-tags (in form of "#"-comments) should get commented out.
Example:
Some_Values
#NORMAL_LISTENER_START
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)
(KEY = LISTENER)
)
)
)
#NORMAL_LISTENER_END
Other_Values
Should result in:
Some_Values
#NORMAL_LISTENER_START
# LISTENER =
# (DESCRIPTION =
# (ADDRESS = (PROTOCOL = IPC)
# (KEY = LISTENER)
# )
# )
# )
#NORMAL_LISTENER_END
Other_Values
The following command already achieves it, but it also puts a comment+blank in front of the search-tags:
sed -i "/#NORMAL_LISTENER_START/,/#NORMAL_LISTENER_END/ s/^/# /" ${my_file}
Now my research told me to exclude those search-tags like:
sed -i '/#NORMAL_LISTENER_START/,/#NORMAL_LISTENER_END/{//!p;} s/^/# /' ${my_file}
But it won't work - with the following message as a result:
sed: -e expression #1, char 56: extra characters after command
I need those SearchTags to be as they are, because I need them afterwards again.

If ed is available/acceptable.
printf '%s\n' 'g/#NORMAL_LISTENER_START/+1;/#NORMAL_LISTENER_END/-1s/^/#/' ,p Q | ed -s file.txt
Change Q to w if you're satisfied with the output and in-place editing will occur.
Remove the ,p If you don't want to see the output.

This might work for you (GNU sed):
sed '/#NORMAL_LISTENER_START/,/#NORMAL_LISTENER_END/{//!s/^/# /}' file
Use a range, delimited by two regexp and insert # before the lines between the regexps but not including the regexps.
Alternative:
sed '/#NORMAL_LISTENER_START/,/#NORMAL_LISTENER_END/{s/^[^#]/# &/}' file
Or if you prefer:
sed '/#NORMAL_LISTENER_START/{:a;n;/#NORMAL_LISTENER_END/!s/^/# /;ta}' file

With your shown samples only, please try following awk code. Simple explanation would be, look for specific string and set/unset vars as per that and then print updated(added # in lines) lines as per that, stop updating lines once we find line from which we need not to update lines.
awk ' /Other_Values/{found=""} found{$0=$0!~/^#/?"#"$0:$0} /Some_Values/{found=1} 1' Input_file
Above will print output on terminal, once you are happy with results you could run following code to do inplace save into Input_file.
awk ' /Other_Values/{found=""} found{$0=$0!~/^#/?"#"$0:$0} /Some_Values/{found=1} 1' Input_file > temp && mv temp Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
/Other_Values/{ found="" } ##If line contains Other_Values then nullify found here.
found { $0=$0!~/^#/?"#"$0:$0 } ##If found is SET then check if line already has # then leave it as it is OR add # in starting to it.
/Some_Values/{ found=1 } ##If line contains Some_Values then set found here.
1 ##Printing current line here.
' Input_file ##Mentioning Input_file name here.

Related

add a line between matching pattern - unix

I want to insert "123" below madguy-xyz- line in "module xyz".
There are multiple modules having similar lines. But i want to add it in only "module xyz".
module abc
njkenjkfvsfd
madguy-xyz-mafdvnskjfvn
enfvjkesn
endmodule
module xyz
njkenjkfvsfd
madguy-xyz-mafdvnskjfvn
enfvjkesn
endmodule
This is the code i tried but doesn't work,
sed -i "/module xyz/,/^endmodule/{/madguy-xyz-/a 123}" <file_name>
This is the error I got:
sed: -e expression #1, char 0: unmatched `{'
This might work for you (GNU sed):
sed '/module xyz/{:a;n;/madguy-xyz-/!ba;p;s/\S.*/123/}' file
For a line containing module xyz, continue printing lines until one containing madguy-xyz-.
Print this line too and then replace it with 123.
Another alternative solution:
sed '/module/h;G;/madguy-xyz.*\nmodule xyz/{P;s/\S.*/123/};P;d' file
Store any module line in the hold space.
Append the module line to each line.
If the first line contains madguy-xyz- and the second module xyz, print the first then substitute the second for 123.
Print the first line and delete the whole.
With your shown samples, please try following.
awk '1; /^endmodule$/{found=""};/^module xyz$/{found=1} found && /^ +madguy-xyz-/{print "123"} ' Input_file
Once you are happy with results of above command, to save output into Input_file itself try following then:
awk '1;/^endmodule$/{found=""} /^module xyz$/{found=1} found && /^ +madguy-xyz-/{print "123"} ' Input_file > temp && mv temp Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
1;
/^endmodule$/{found=""} ##Printing current line here.
/^module xyz$/{ ##Checking condition if line contains module xyz then do following.
found=1 ##Setting found to 1 here.
}
found && /^ +madguy-xyz-/{ ##Checking if found is SET and line contains madguy-xyz- then do following.
print "123" ##Printing 123 here.
}
' Input_file ##Mentioning Input_file name here.
NOTE: In case your line exactly having module xyz value then change above /module xyz/ (this condition) to $0=="module xyz" too.
With GNU sed I suggest:
sed -i -e "/module xyz/,/^endmodule/{/madguy-xyz-/a 123" -e "}" file
Using any POSIX awk in any shell on every Unix box, the following will work for the sunny day case in your question and all rainy day cases such as the ones I mentioned in my comment and more:
$ cat tst.awk
{ print }
$1 == "endmodule" {
inMod = 0
}
inMod && (index($1,old) == 1) {
sub(/[^[:space:]].*/,"")
print $0 new
}
($1 == "module") && ($2 == mod) {
inMod = 1
}
$ awk -v mod='xyz' -v old='madguy-xyz-' -v new='123' -f tst.awk file
module abc
njkenjkfvsfd
madguy-xyz-mafdvnskjfvn
enfvjkesn
endmodule
module xyz
njkenjkfvsfd
madguy-xyz-mafdvnskjfvn
123
enfvjkesn
endmodule

How can I extract using sed or awk between newlines after a specific pattern?

I like to check if there is other alternatives where I can print using other bash commands to get the range of IPs under #Hiko other than the below sed, tail and head which I actually figured out to get what I needed from my hosts file.
I'm just curious and keen in learning more on bash, hope I could gain more knowledge from the community.
:D
$ sed -n '/#Hiko/,/#Pico/p' /etc/hosts | tail -n +3 | head -n -2
/etc/hosts
#Tito
192.168.1.21
192.168.1.119
#Hiko
192.168.1.243
192.168.1.125
192.168.1.94
192.168.1.24
192.168.1.242
#Pico
192.168.1.23
192.168.1.93
192.168.1.121
1st solution: With shown samples could you please try following. Written and tested in GNU awk.
awk -v RS= '/#Pico/{exit} /#Hiko/{found=1;next} found' Input_file
Explanation:
awk -v RS= ' ##Starting awk program from here.
/#Pico/{ ##Checking condition if line has #Pico then do following.
exit ##exiting from program.
}
/#Hiko/{ ##Checking condition if line has #Hiko is present in line.
found=1 ##Setting found to 1 here.
next ##next will skip all further statements from here.
}
found ##Checking condition if found is SET then print the line.
' Input_file ##mentioning Input_file name here.
2nd solution: Without using RS function try following.
awk '/#Pico/{exit} /#Hiko/{found=1;next} NF && found' Input_file
3rd solution: You could look for record #Hiko and then could print its next record and come out with shown samples.
awk -v RS= '/#Hiko/{found=1;next} found{print;exit}' Input_file
NOTE: These all solutions above check if string #Hiko or #Pico are present in anywhere in line, in case you want to look exact string then change above only /#Hiko/ and /#Pico/ part to /^#Hiko$/ and /^#Pico$/ respectively.
With sed (checked with GNU sed, syntax might differ for other implementations)
$ sed -n '/#Hiko/{n; :a n; /^$/q; p; ba}' /etc/hosts
192.168.1.243
192.168.1.125
192.168.1.94
192.168.1.24
192.168.1.242
-n turn off automatic printing of pattern space
/#Hiko/ if line contains #Hiko
n get next line (assuming there's always an empty line)
:a label a
n get next line (using n will overwrite any previous content in the pattern space, so only single line content is present in this case)
/^$/q if the current line is empty, quit
p print the current line
ba branch to label a
You can use
awk -v RS= '/^#Hiko$/{getline;print;exit}' file
awk -v RS= '$0 == "#Hiko"{getline;print;exit}' file
Which means:
RS= - make awk read the file paragraph by paragraph
/^#Hiko$/ or '$0 == "#Hiko" - finds a paragraph that is equal to #Hiko
{getline;print;exit} - gets the next paragraph, prints it and exits.
See the online demo.
You may use:
awk -v RS= 'p && NR == p + 1; $1 == "#Hiko" {p = NR}' /etc/hosts
192.168.1.243
192.168.1.125
192.168.1.94
192.168.1.24
192.168.1.242
This might work for you (GNU sed):
sed -n '/^#/h;G;/^[0-9].*\n#Hiko/P' file
Copy the header to the hold buffer.
Append the hold buffer to each line.
If the line begins with a digit and contains the required header, print the first line in the pattern space.

extract and print all occurrences of disk file (.img) from a configuration file

I have vm configuration files from which I need to print all the disks (26 alphanumeric characters followed by .img) existing within each file.
here is an extract of one of the files
[root#~]# cat demo_vm.cfg
disk = ['file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb0000120000a17dfe12ac74818f.img,xvda,w', 'file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb0000120000e66ace31dac64d98.img,xvdb,w', 'file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb000012000082fbb45a02e24096.img,xvdd,w']
I want to extract the below (all references of 26alphanum.img in the file) :
0004fb0000120000a17dfe12ac74818f.img
0004fb0000120000e66ace31dac64d98.img
0004fb000012000082fbb45a02e24096.img
some files have 3 disks some have only one for which I usually run this and have what I want but in case of multiple occurrences I can only print the first one.
# awk -F [/,] '/disk/ { print $6}' demo_vm.cfg
0004fb0000120000a17dfe12ac74818f.img
Thanks in advance I spent hours trying splits and regex patterns without conclusive result.
This is my first question in SOverflow.
EDIT
here are the 3 types of content put in separate files (1= one 26[alnum].img occurrence, 2= two 26[alnum].img occurrences , 3= three 26[alnum].img occurrences )
# cat demo_vm_1.cfg
disk = ['file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb000012000065a82a4df5e7112b.img,xvda,w']
[root ~]# cat demo_vm_2.cfg
disk = ['file:/OVS/Repositories/0004fb0000030000a079ca25909e5455/VirtualDisks/0004fb0000120000822cb8b0602ee042.img,xvda,w', 'file:/OVS/Repositories/0004fb0000030000a079ca25909e5455/VirtualDisks/0004fb000012000073d5fd864a0ba6b1.img,xvdb,w']
# cat demo_vm_3.cfg
disk = ['file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb0000120000a17dfe12ac74818f.img,xvda,w', 'file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb0000120000e66ace31dac64d98.img,xvdb,w', 'file:/OVS/Repositories/0004fb00000300007b8afb76a3377693/VirtualDisks/0004fb000012000082fbb45a02e24096.img,xvdd,w']
Initial script
my initial script that creates the remove commands for the .cfg files and the pointed images inside each of them had a problem when the cfg had more than one disk reference. I guess I can adapt it now to use grep -Eo instead of awk
strings=(`find /vm_backup/VirtualMachines/*/vm.cfg`)
for i in "${strings[#]}"; do
echo "rm -f $i" >> drop_vm_final.sh
awk -F [/,] '/disk/ { print $6}' "$i" | awk '{print "rm -f /vm_backup/VirtualDisks/"$0}' >>drop_vm_bkp_final.sh
done
$ grep -Eo '[[:alnum:]]{26}\.img' file
0000120000a17dfe12ac74818f.img
0000120000e66ace31dac64d98.img
000012000082fbb45a02e24096.img
If that's not all you need then edit your question to provide more truly representative sample input/output that that doesn't work for.
Could you please try following based on your shown samples.
awk '
match($0,/[[:alnum:]]{26}\.img/){
print substr($0,RSTART,RLENGTH)
}
' Input_file
OR to get all matched values in a single line try following.
awk '
{
while(match($0,/[[:alnum:]]{26}\.img/)){
print substr($0,RSTART,RLENGTH)
$0=substr($0,RSTART+RLENGTH)
}
}' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
{
while(match($0,/[[:alnum:]]{26}\.img/)){ ##Running while loop to match alpha numerics 26 in number followed by .img if this match found then do following.
print substr($0,RSTART,RLENGTH) ##Printing matched sub string of that matched regex from current line.
$0=substr($0,RSTART+RLENGTH) ##Saving rest of the line(after matched string) to current line here.
}
}' Input_file ##mentioning Input_file name here.
Based on your code
awk -F [/,] '/disk/ { print $6}' demo_vm.cfg
you can complete the print adding $14 and $22
awk -F [/,] '{ print $6,$14,$22}' OFS='\n' demo_vm.cfg
0004fb0000120000a17dfe12ac74818f.img
0004fb0000120000e66ace31dac64d98.img
0004fb000012000082fbb45a02e24096.img

Replace a letter with another from the last word from the last two lines of a text file

How could I possibly replace a character with another, selecting the last word from the last two lines of a text file in shell, using only a single command? In my case, replacing every occurrence of a with E from the last word only.
Like, from a text file containing this:
tree;apple;another
mango.banana.half
monkey.shelf.karma
to this:
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
I tried using sed -n 'tail -2 'mytext.txt' -r 's/[a]+/E/*$//' but it doesn't work (my error: sed expression #1, char 10: unknown option to 's).
Could you please try following, tac + awk solution. Completely based on OP's samples only.
tac Input_file |
awk 'FNR<=2{if(/;/){FS=OFS=";"};if(/\./){FS=OFS="."};gsub(/a/,"E",$NF)} 1' |
tac
Output with shown samples is:
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
NOTE: Change gsub to sub in case you want to substitute only very first occurrence of character a in last field.
This might work for you (GNU sed):
sed -E 'N;${:a;s/a([^a.]*)$/E\1/mg;ta};P;D' file
Open a two line window throughout the length of the file by using the N to append the next line to the previous and the P and D commands to print then delete the first of these. Thus at the end of the file, signified by the $ address the last two lines will be present in the pattern space.
Using the m multiline flag on the substitution command, as well as the g global flag and a loop between :a and ta, replace any a in the last word (delimited by .) by an E.
Thus the first pass of the substitution command will replace the a in half and the last a in karma. The next pass will match nothing in the penultimate line and replace the a in karmE. The third pass will match nothing and thus the ta command will fail and the last two lines will printed with the required changes.
If you want to use Sed, here's a solution:
tac input_file | sed -E '1,2{h;s/.*[^a-zA-Z]([a-zA-Z]+)/\1/;s/a/E/;x;s/(.*[^a-zA-Z]).*/\1/;G;s/\n//}' | tac
One tiny detail. In your question you say you want to replace a letter, but then you transform karma in kErme, so what is this? If you meant to write kErma, then the command above will work; if you meant to write kErmE, then you have to change it just a bit: the s/a/E/ should become s/a/E/g.
With tac+perl
$ tac ip.txt | perl -pe 's/\w+\W*$/$&=~tr|a|E|r/e if $.<=2' | tac
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
\w+\W*$ match last word in the line, \W* allows any possible trailing non-word characters to be matched as well. Change \w and \W accordingly if numbers and underscores shouldn't be considered as word characters - for ex: [a-zA-Z]+[^a-zA-Z]*$
$&=~tr|a|E|r change all a to E only for the matched portion
e flag to enable use of Perl code in replacement section instead of string
To do it in one command, you can slurp the entire input as single string (assuming this'll fit available memory):
perl -0777 -pe 's/\w+\W*$(?=(\n.*)?\n\z)/$&=~tr|a|E|r/gme'
Using GNU awk forsplit() 4th arg since in the comments of another solution the field delimiter is every sequence of alphanumeric and numeric characters:
$ gawk '
BEGIN {
pc=2 # previous counter, ie how many are affected
}
{
for(i=pc;i>=1;i--) # buffer to p hash, a FIFO
if(i==pc && (i in p)) # when full, output
print p[i]
else if(i in p) # and keep filling
p[i+1]=p[i] # above could be done using mod also
p[1]=$0
}
END {
for(i=pc;i>=1;i--) {
n=split(p[i],t,/[^a-zA-Z0-9\r]+/,seps) # split on non alnum
gsub(/a/,"E",t[n]) # replace
for(j=1;j<=n;j++) {
p[i]=(j==1?"":p[i] seps[j-1]) t[j] # pack it up
}
print p[i] # output
}
}' file
Output:
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
Would this help you ? on GNU awk
$ cat file
tree;apple;another
mango.banana.half
monkey.shelf.karma
$ tac file | awk 'NR<=2{s=gensub(/(.*)([.;])(.*)$/,"\\3",1);gsub(/a/,"E",s); print gensub(/(.*)([.;])(.*)$/,"\\1\\2",1) s;next}1' | tac
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
Better Readable version :
$ tac file | awk 'NR<=2{
s=gensub(/(.*)([.;])(.*)$/,"\\3",1);
gsub(/a/,"E",s);
print gensub(/(.*)([.;])(.*)$/,"\\1\\2",1) s;
next
}1' | tac
With GNU awk you can set FS with the two separators, then gsub for the replacement in $3, the third field, if NR>1
awk -v FS=";|[.]" 'NR>1 {gsub("a", "E",$3)}1' OFS="." file
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
With GNU awk for the 3rd arg to match() and gensub():
$ awk -v n=2 '
NR>n { print p[NR%n] }
{ p[NR%n] = $0 }
END {
for (i=0; i<n; i++) {
match(p[i],/(.*[^[:alnum:]])(.*)/,a)
print a[1] gensub(/a/,"E","g",a[2])
}
}
' file
tree;apple;another
mango.banana.hElf
monkey.shelf.kErmE
or with any awk:
awk -v n=2 '
NR>n { print p[NR%n] }
{ p[NR%n] = $0 }
END {
for (i=0; i<n; i++) {
match(p[i],/.*[^[:alnum:]]/)
lastWord = substr(p[i],1+RLENGTH)
gsub(/a/,"E",lastWord )
print substr(p[i],1,RLENGTH) lastWord
}
}
' file
If you want to do it for the last 50 lines of a file instead of the last 2 lines just change -v n=2 to -v n=50.
The above assumes there are at least n lines in your input.
You can let sed repeat changing an a into E only for the last word with a label.
tac mytext.txt| sed -r ':a; 1,2s/a(\w*)$/E\1/; ta' | tac

sed match a pattern and insert newline followed by replacement text

Let's say my input file es-service has the following lines:
# Comment 1
key1=value1
# Comment 3
key3=value3
If the pattern key2=value2 is not present in the above file, then add it after key1=value1
Hence, the file should now have:
# Comment 1
key1=value1
# Comment 2
key2=value2
# Comment 3
key3=value3
I came up with the following to achieve it:
if ! grep -qxF 'key2=value2' es-service;
then sed -i "/key1/a \n# Comment 2\nkey2=value2" es-service
fi
The problem is the first \n after /a doesn't insert a new-line. Hence I end-up getting the below:
key1=value1
n# Comment 2
key2=value2
instead of
key1=value1
# Comment 2
key2=value2
Edit:
I eventually solved it by adding one more sed to match Comment 2 and add a newline before it by using option i.
if ! grep -qxF 'key2=value2' es-service;
then sed -i "/key1/a \n# Comment 2\nkey2=value2" es-service; sed -i '/# Comment 2/i\ ' es-service
fi
All in a awk using loop
awk '/key2=/ {f=1} /key1=/ {n=NR} {a[NR]=$0} END {for(i=1;i<=NR;i++) {print a[i];if(i==n && !f) print "\n# Comment 2\nkey2=value2"}}' file
# Comment 1
key1=value1
# Comment 2
key2=value2
# Comment 3
key3=value3
/key2=/ {f=1} if key2= is found set flag f to prevent double insertion.
/key1=/ {n=NR} if key1 is found, store the line number in n
a[NR]=$0 store all line in array a
END After file is rund, do:
for(i=1;i<=NR;i++) loop trough all line, then
print a[i] print the line and
if(i==n && !f) if line is where key=1 is found and flag f is not true, do:
print "\n# Comment 2\nkey2=value2" print extra information.
Could you please try following, this code will take care of any missing key(if keys are NOT continuous in their sequence and add them with comment number too).
awk '
BEGIN{
FS="="
}
!NF{
print
next
}
/^# Comment/{
val=$0
next
}
/^key/{
first_col=$1
sub(/[a-zA-Z]+/,"",first_col)
while(first_col!=prev+1){
prev++
print "# Comment "prev ORS "key"prev"=value"prev ORS
}
prev=first_col
print val ORS $0
}
' Input_file
A gnu awk solution without loop
awk -v RS= -v ORS='\n\n' 'NR>1 && a~/key1/ && !/key2/ {print "# Comment 2\nkey2=value2"} 1; {a=$0}' file
# Comment 1
key1=value1
# Comment 2
key2=value2
# Comment 3
key3=value3
-v RS= -v ORS='\n\n' Set Record selector to nothing and output record selector to two new line
NR>1 && a~/key1/ && !/key2/ skip first block and test if previous block contains key1 and current line does not contain key2, then
print "# Comment 2\nkey2=value2" add new block
1; is always true, so it will print all line.
a=$0 store line in variable a to use for test in next line
a and i are tough to inline.
So this just uses s/// replacement and & for the match data. In other words, s/.*/&\n.../ where ... is your appended strings.
sed -i '/key1/s/.*/&\n# Comment 2\nkey2=value2/' es-service
Alternately:
You can use s///e to construct a shell command to generate output to be placed in the stream.
sed -i '/key1/s/.*/printf "&\n# Comment 2\nkey2=value2\n"/e' es-service
So I'm replacing .* with printf "&\n followed by what you'd like to insert.
e then executes the printf and sticks the output in the stream. I thought e was GNU-sed-only, but it's working for me with --posix.